ep-engine.git
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' 39/57139/1 4.1.0-MP1 v4.1.0
Manu Dhundi [Wed, 18 Nov 2015 01:45:17 +0000 (17:45 -0800)]
Merge remote-tracking branch 'couchbase/3.0.x'

couchbase/3.0.x:
  MB-16836: Reset the stat 'ep_bg_fetched' to 0 on 'cbstats reset' command

Change-Id: I0ad9b7fdcd4b37cbffa38361ec33ccc4f3eec78b

5 years agoMB-16836: Reset the stat 'ep_bg_fetched' to 0 on 'cbstats reset' command 29/57129/2
Manu Dhundi [Tue, 17 Nov 2015 22:55:18 +0000 (14:55 -0800)]
MB-16836: Reset the stat 'ep_bg_fetched' to 0 on 'cbstats reset' command

Change-Id: I444bd6c76265788d4366061d4d2b25e3c5e60518
Reviewed-on: http://review.couchbase.org/57129
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
5 years agoMB-16357: Create a variable to get correct locking scope 77/56977/2
Jim Walker [Thu, 12 Nov 2015 12:21:41 +0000 (12:21 +0000)]
MB-16357: Create a variable to get correct locking scope

A mistake in 495e00acc24 means that no variable is
created for the ReaderLockHolder, the compiler either
optimises away the lock constructor/destructor or the lock
scope is wrong.

Either way we need to create a variable.

Change-Id: I156fcf6b0df7aee99b2c33c0e6e80d3c6ef7e546
Reviewed-on: http://review.couchbase.org/56977
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
5 years agoMB-16357: Create a variable to get correct locking scope 76/56976/2
Jim Walker [Thu, 12 Nov 2015 12:04:56 +0000 (12:04 +0000)]
MB-16357: Create a variable to get correct locking scope

A mistake in 495e00acc24 means that no variable is
created for the ReaderLockHolder, the compiler either
optimises away the lock constructor/destructor or the lock
scope is wrong.

Either way we need to create a variable.

Change-Id: I642ac64d71b73d3d78207ff50d33539a06ce0e7e
Reviewed-on: http://review.couchbase.org/56976
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock' 68/56568/1
abhinavdangeti [Fri, 30 Oct 2015 19:01:06 +0000 (12:01 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock'

couchbase/3.0.x:
* 8939f83 MB-16686: Remove sanity check while adding TAP over DCP
* 0f1ff18 MB-15171: [BP] Initialize dcpConnMap_ to NULL in engine constructor
* 44bd072 MB-14825: [BP] While trying to stream next checkpoint item, check if vbucket is valid

Change-Id: I9ec7c046aa16aeeb0999438ef0186109828f3044

5 years agoMB-16686: Remove sanity check while adding TAP over DCP 64/56564/3 v3.1.2
abhinavdangeti [Fri, 30 Oct 2015 17:11:46 +0000 (10:11 -0700)]
MB-16686: Remove sanity check while adding TAP over DCP

This check isn't accurate as certain TAP messages from
the producer carry no vbucket information - initialized to
zero (expected), as they aren't vbucket specific operations.
In such a scenario, if the TAP consumer needs to be created,
it wouldn't be allowed to if a DCP passive stream exists
for vbucket 0. This would break an online upgrade.

Change-Id: I310b9cf4dbaf652c233cba02de7ca72469efa89d
Reviewed-on: http://review.couchbase.org/56564
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-16574: Log when cluster config is changed 19/56219/2
Dave Rigby [Mon, 19 Oct 2015 11:41:49 +0000 (12:41 +0100)]
MB-16574: Log when cluster config is changed

Change-Id: I0b234e6330ab3e5c0cde84c14977aff8034000d6
Reviewed-on: http://review.couchbase.org/56219
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15171: [BP] Initialize dcpConnMap_ to NULL in engine constructor 71/56171/3
Sriram Ganesan [Thu, 28 May 2015 01:51:23 +0000 (18:51 -0700)]
MB-15171: [BP] Initialize dcpConnMap_ to NULL in engine constructor

Not initializing this variable to NULL can cause access to an
invalid pointer during engine destroy.

Change-Id: Icc5d848f7826bb6331deb40b4832efcf64622dea
Reviewed-on: http://review.couchbase.org/51492
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-on: http://review.couchbase.org/56171

5 years agoMB-14825: [BP] While trying to stream next checkpoint item, check if vbucket is valid 70/56170/3
Manu Dhundi [Thu, 7 May 2015 01:30:24 +0000 (18:30 -0700)]
MB-14825: [BP] While trying to stream next checkpoint item, check if vbucket is valid

If a vbucket is deleted in middle of a DCP connection streaming a checkpoint
item, we should handle such a scenario in a graceful manner.

Change-Id: I24fe52adc572f504f492f015f82fc8d5e0325925
Reviewed-on: http://review.couchbase.org/50674
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-on: http://review.couchbase.org/56170
Tested-by: buildbot <build@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' into sherlock 24/56124/1
abhinavdangeti [Wed, 14 Oct 2015 18:29:15 +0000 (11:29 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x' into sherlock

couchbase/3.0.x:
* dec472a MB-16500 [BP]: Address data race in DcpConsumer, by acquiring readyMutex
* 6abf179 MB-16500 [BP]: Removing unnecessary locking in consumer code
* d388995 MB-16500 [BP]: MB-16496 Fix the race on vbucket state between persistVBState() and compactVB()
* 8828d4d MB-16500 [BP]: Address possible data race in checkpoint remover
* 52e40ed MB-16500 [BP]: Address possible data race in item/expiry pagers
* 5305a71 MB-16500 [BP]: Fix potenial deadlock around Connmap::relaseLock / connLock
* b155309 MB-16500 [BP]: Address possible data race in Notifiable: setSuspended
* 3b8a7be MB-16500 [BP]: Address possible data race in ConnHandler: lastWalkTime

Change-Id: Ie72bf67c10b0bed125a89534815474b1309763bb

5 years agoMB-16500 [BP]: Address data race in DcpConsumer, by acquiring readyMutex 65/56065/4
abhinavdangeti [Wed, 7 Oct 2015 21:49:41 +0000 (14:49 -0700)]
MB-16500 [BP]: Address data race in DcpConsumer, by acquiring readyMutex

WARNING: ThreadSanitizer: data race (pid=27652)

  Write of size 8 at 0x7d08000443c0 by main thread (mutexes: write M57876):
    #0 operator delete(void*) <null>:0 (engine_testapp+0x000000050e7b)
    #1 __gnu_cxx::new_allocator<std::_List_node<unsigned short> >::deallocate(std::_List_node<unsigned short>*, unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/ext/new_allocator.h:110 (ep.so+0x00000005d69a)
    #2 DcpConsumer::step(dcp_message_producers*) /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:516 (ep.so+0x00000005c5cc)
    #3 EvpDcpStep(engine_interface*, void const*, dcp_message_producers*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1479 (ep.so+0x0000000b480b)
    #4 mock_dcp_step(engine_interface*, void const*, dcp_message_producers*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:476 (engine_testapp+0x0000000bb055)
    #5 dcp_step(engine_interface*, engine_interface_v1*, void const*) /home/abhinav/couchbase/ep-engine/tests/ep_test_apis.cc:1219 (ep_testsuite.so+0x0000000b61bd)
    #6 test_chk_manager_rollback(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:5526 (ep_testsuite.so+0x0000000809b4)
    #7 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b952c)
    #8 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)

  Previous write of size 8 at 0x7d08000443c0 by thread T16:
    #0 operator new(unsigned long) <null>:0 (engine_testapp+0x00000005090d)
    #1 __gnu_cxx::new_allocator<std::_List_node<unsigned short> >::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/ext/new_allocator.h:104 (ep.so+0x00000005f265)
    #2 PassiveStream::reconnectStream(RCPtr<VBucket>&, unsigned int, unsigned long) /home/abhinav/couchbase/ep-engine/src/dcp/stream.cc:1104 (ep.so+0x000000076f5f)
    #3 DcpConsumer::doRollback(unsigned int, unsigned short, unsigned long) /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:676 (ep.so+0x00000005db67)
    #4 RollbackTask::run() /home/abhinav/couchbase/ep-engine/src/dcp/consumer.cc:574 (ep.so+0x00000005d9d4)
    #5 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000f834c)
    #6 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f7eb5)
    #7 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d71)

Change-Id: I196a78e54bf8014967a51cdb081126597153f77b
Reviewed-on: http://review.couchbase.org/55881
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-on: http://review.couchbase.org/56065
Reviewed-by: Dave Rigby <daver@couchbase.com>
5 years agoMB-16500 [BP]: Removing unnecessary locking in consumer code 80/56080/3
abhinavdangeti [Thu, 13 Aug 2015 18:46:35 +0000 (11:46 -0700)]
MB-16500 [BP]: Removing unnecessary locking in consumer code

streamMutex is to protect the ready list, but not the streams list.

The front end operations: addStream, closeStream, handleResponse, step
- wouldn't race with each other over the streams list, as multiple
memcached threads will not serve a single cookie.

The back end operations: processBufferedMessages (doesn't grab lock any
way), doRollback just read from streams list.

An addstream (front end op) is the only one that updates streams, and
this wouldn't update when a rollback is in progress.

Therefore, renaming the streamMutex lock in DCPConsumer to readyMutex
which is more apt for its operation - guarding the ready list.

Change-Id: Ia342d7243fef4b97b729aa94fdc64ad020711589
Reviewed-on: http://review.couchbase.org/54406
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Reviewed-on: http://review.couchbase.org/56080
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
5 years agoMB-16500 [BP]: MB-16496 Fix the race on vbucket state between persistVBState() and... 68/56068/5
Chiyoung Seo [Fri, 9 Oct 2015 18:39:10 +0000 (11:39 -0700)]
MB-16500 [BP]: MB-16496 Fix the race on vbucket state between persistVBState() and compactVB()

The following data race is reported by thread sanitizer:

WARNING: ThreadSanitizer: data race (pid=29921)
  Write of size 8 at 0x7d680001f580 by thread T5 (mutexes: write M12734):
    #0 VBucket::setPurgeSeqno() ep-engine/src/vbucket.h:215:9 (ep.so+0x000000086204)
    #1 EventuallyPersistentStore::compactVBucket() ep-engine/src/ep.cc:1584 (ep.so+0x000000086204)
    #2 CompactVBucketTask::run() ep-engine/src/tasks.cc:94:12 (ep.so+0x00000012971e)
    #3 ExecutorThread::run() ep-engine/src/executorthread.cc:115:26 (ep.so+0x0000000ea41d)
    #4 launch_executor_thread() ep-engine/src/executorthread.cc:33:9 (ep.so+0x0000000e9fe5)
    #5 platform_thread_wrap platform/src/cb_pthreads.c:23:5 (libplatform.so.0.1.0+0x000000004161)

  Previous read of size 8 at 0x7d680001f580 by thread T7:
    #0 VBucket::getPurgeSeqno() ep-engine/src/vbucket.h:211:16 (ep.so+0x0000000821d3)
    #1 EventuallyPersistentStore::persistVBState() ep-engine/src/ep.cc:1217 (ep.so+0x0000000821d3)
    #2 VBStatePersistTask::run() ep-engine/src/tasks.cc:86:12 (ep.so+0x000000129636)
    #3 ExecutorThread::run() ep-engine/src/executorthread.cc:115:26 (ep.so+0x0000000ea41d)
    #4 launch_executor_thread() ep-engine/src/executorthread.cc:33:9 (ep.so+0x0000000e9fe5)
    #5 platform_thread_wrap platform/src/cb_pthreads.c:23:5 (libplatform.so.0.1.0+0x000000004161)

  Location is heap block of size 1392 at 0x7d680001f200 allocated by main thread:
    #0 operator new() <null> (engine_testapp+0x00000045cded)
    #1 EventuallyPersistentStore::setVBucketState() ep-engine/src/ep.cc:1300:30 (ep.so+0x000000082b1a)
    #2 EventuallyPersistentEngine::setVBucketState() ep-engine/src/ep_engine.h:718:16 (ep.so+0x0000000ca308)
    #3 setVBucket()) ep-engine/src/ep_engine.cc:884 (ep.so+0x0000000ca308)
    #4 processUnknownCommand()) ep-engine/src/ep_engine.cc:1178 (ep.so+0x0000000ca308)
    #5 EvpUnknownCommand()) ep-engine/src/ep_engine.cc:1389:38 (ep.so+0x0000000aafc8)
    #6 mock_unknown_command()) memcached/programs/engine_testapp/engine_testapp.cc:380:19 (engine_testapp+0x0000004c56b9)
    #7 set_vbucket_state() ep-engine/tests/ep_test_apis.cc:607:9 (ep_testsuite.so+0x0000000a3a4b)
    #8 test_setup() ep-engine/tests/ep_testsuite_common.cc:146:28 (ep_testsuite.so+0x00000009cdda)
    #9 execute_test() memcached/programs/engine_testapp/engine_testapp.cc:1085:47 (engine_testapp+0x0000004c4103)
    #10 main memcached/programs/engine_testapp/engine_testapp.cc:1439 (engine_testapp+0x0000004c4103)

To address the above issue, vbucket states should be read after grabbing
the vbucket writer lock in EPStore::persistVBState().

Change-Id: I5a42b3e15a1cf5c941d399897bc68d6f35746ff3
Reviewed-on: http://review.couchbase.org/55972
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-on: http://review.couchbase.org/56068
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-16500 [BP]: Address possible data race in checkpoint remover 67/56067/4
abhinavdangeti [Thu, 8 Oct 2015 22:14:25 +0000 (15:14 -0700)]
MB-16500 [BP]: Address possible data race in checkpoint remover

WARNING: ThreadSanitizer: data race (pid=102986)

  Read of size 1 at 0x7d180000c298 by thread T17:
    #0 ClosedUnrefCheckpointRemoverTask::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/checkpoint_remover.cc:139 (ep.so+0x00000003d4a6)
    #1 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000f9c4c)
    #2 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f9815)
    #3 platform_thread_wrap /home/couchbase/.ccache/tmp/cb_pthread.tmp.9515862b2292.83537.i:0 (libplatform.so.0.1.0+0x0000000041b1)

  Previous write of size 1 at 0x7d180000c298 by thread T18:
    #0 CheckpointVisitor::complete() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/checkpoint_remover.cc:69 (ep.so+0x00000003e0a1)
    #1 VBCBAdaptor::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep.cc:3789 (ep.so+0x00000009d64a)
    #2 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000f9c4c)
    #3 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f9815)
    #4 platform_thread_wrap /home/couchbase/.ccache/tmp/cb_pthread.tmp.9515862b2292.83537.i:0 (libplatform.so.0.1.0+0x0000000041b1)

Change-Id: I8579d5af259490e41028f57f302e547b0826fa61
Reviewed-on: http://review.couchbase.org/55937
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-on: http://review.couchbase.org/56067

5 years agoMB-16500 [BP]: Address possible data race in item/expiry pagers 66/56066/4
abhinavdangeti [Thu, 8 Oct 2015 22:05:58 +0000 (15:05 -0700)]
MB-16500 [BP]: Address possible data race in item/expiry pagers

WARNING: ThreadSanitizer: data race (pid=102450)

  Write of size 1 at 0x7d180000c2f8 by thread T17:
    #0 PagingVisitor::complete() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/item_pager.cc:175 (ep.so+0x000000106c57)
    #1 VBCBAdaptor::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep.cc:3789 (ep.so+0x00000009d64a)
    #2 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000f9c4c)
    #3 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f9815)
    #4 platform_thread_wrap /home/couchbase/.ccache/tmp/cb_pthread.tmp.9515862b2292.83537.i:0 (libplatform.so.0.1.0+0x0000000041b1)

  Previous read of size 1 at 0x7d180000c2f8 by thread T18:
    #0 ExpiredItemPager::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/item_pager.cc:334 (ep.so+0x0000001053c6)
    #1 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000f9c4c)
    #2 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f9815)
    #3 platform_thread_wrap /home/couchbase/.ccache/tmp/cb_pthread.tmp.9515862b2292.83537.i:0 (libplatform.so.0.1.0+0x0000000041b1)

Change-Id: Iebfe280c95847ee80b2d80d08b0eb340f40663d9
Reviewed-on: http://review.couchbase.org/55935
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-on: http://review.couchbase.org/56066

5 years agoMB-16500 [BP]: Fix potenial deadlock around Connmap::relaseLock / connLock 64/56064/2
Dave Rigby [Wed, 7 Oct 2015 11:23:39 +0000 (11:23 +0000)]
MB-16500 [BP]: Fix potenial deadlock around Connmap::relaseLock / connLock

As reported by ThreadSanitizer (see below), we have a lock inversion
creating a potential deadlock in Connmap, related to how we shutdown
connections:

There exists a cycle in lock order graph:

 M2176 (Connmap::releaseLock in ConnMap::notifyPausedConnection() connmap.cc:235) =>
   M128093 (mock_server::conn_struct::mutex) =>
     M2177 (Connmap::connsLock in TapConnMap::shutdownAllConnections(), connmap.cc:770) =>
       M2176 (Connmap::releaseLock in TapConnMap::shutdownAllConnections(), connmap.cc:777) DEADLOCK!

The problem appears to be that in TapConnMap::shutdownAllConnections()
we first acquire {connsLock}, then acquire {releaseLock}; all while
holding the cookie lock.

Fix is to drop releaseLock in shutdownAllConnections() once we've
released any references, *then* acquire the connLock to actually clear
out the connection map.

ThreadSanitizer report follows (irrelevent parts removed):

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=1087)

  Cycle in lock order graph: M2176 (0x7d840001b810) => M128093 (0x7d280000efa0) => M2177 (0x7d840001b850) => M2176

  Mutex M128093 acquired here while holding mutex M2176 in thread T10:
    <cut>

  Mutex M2176 previously acquired by the same thread here:
    #0 pthread_mutex_lock <null> ()
    ...
    #5 ConnMap::notifyPausedConnection() ep-engine/src/connmap.cc:235 ()
    <cut>

  Mutex M2177 acquired here while holding mutex M128093 in main thread:
    #0 pthread_mutex_lock <null> ()
    ...
    #5 TapConnMap::newProducer() ep-engine/src/connmap.cc:378 ()
    #6 EventuallyPersistentEngine::createTapQueue() ep-engine/src/ep_engine.cc:2663:23 ()
    <cut>

  Mutex M128093 previously acquired by the same thread here:
    #0 pthread_mutex_lock <null> ()
    #1 cb_mutex_enter platform/src/cb_pthreads.c:115:14 ()
    #2 lock_mock_cookie memcached/programs/engine_testapp/mock_server.c:436:4 ()
    #3 test_tap_stream() ep-engine/tests/ep_testsuite.cc:6751:5 ()
    #4 execute_test() memcached/programs/engine_testapp/engine_testapp.cc:1090:19 ()
    #5 main memcached/programs/engine_testapp/engine_testapp.cc:1439 ()

  Mutex M2176 acquired here while holding mutex M2177 in main thread:
    #0 pthread_mutex_lock <null> ()
    ...
    #5 TapConnMap::shutdownAllConnections() ep-engine/src/connmap.cc:777 ()
    #6 EventuallyPersistentEngine::destroy() ep-engine/src/ep_engine.cc:2190:9 ()
    <cut>

  Mutex M2177 previously acquired by the same thread here:
    #0 pthread_mutex_lock <null> ()
    ...
    #5 TapConnMap::shutdownAllConnections() ep-engine/src/connmap.cc:770 ()
    #6 EventuallyPersistentEngine::destroy() ep-engine/src/ep_engine.cc:2190:9 ()
    <cut>

Change-Id: Ic9a4f028b202277729df025333ce630be056903d
Reviewed-on: http://review.couchbase.org/55863
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-on: http://review.couchbase.org/56064

5 years agoMB-16500 [BP]: Address possible data race in Notifiable: setSuspended 63/56063/2
abhinavdangeti [Tue, 6 Oct 2015 00:47:23 +0000 (17:47 -0700)]
MB-16500 [BP]: Address possible data race in Notifiable: setSuspended

WARNING: ThreadSanitizer: data race (pid=19078)

  Write of size 1 at 0x7d600000f078 by thread T10 (mutexes: write M21717):
    #0 Notifiable::setSuspended(bool) /home/abhinav/couchbase/ep-engine/src/tapconnection.h:764 (ep.so+0x00000005fe6a)
    #1 TapProducer::suspendedConnection_UNLOCKED(bool) /home/abhinav/couchbase/ep-engine/src/tapconnection.cc:717 (ep.so+0x00000013b24e)
    #2 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f89f6)
    #3 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f8595)
    #4 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)

  Previous read of size 1 at 0x7d600000f078 by main thread (mutexes: write M21715):
    #0 Notifiable::isSuspended() /home/abhinav/couchbase/ep-engine/src/tapconnection.h:768 (ep.so+0x0000000dfdf2)
    #1 EventuallyPersistentEngine::walkTapQueue(void const*, void**, void**, unsigned short*, unsigned char*, unsigned short*, unsigned int*, unsigned short*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:2531 (ep.so+0x0000000b7ecc)
    #2 EvpTapIterator(engine_interface*, void const*, void**, void**, unsigned short*, unsigned char*, unsigned short*, unsigned int*, unsigned short*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1440 (ep.so+0x0000000d5ad7)
    #3 mock_tap_iterator(engine_interface*, void const*, void**, void**, unsigned short*, unsigned char*, unsigned short*, unsigned int*, unsigned short*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:81 (engine_testapp+0x0000000bbda2)
    #4 test_tap_ack_stream(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:7353 (ep_testsuite.so+0x0000000504a7)
    #5 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
    #6 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)

Change-Id: I596c93c048767911b052861193822ca17270a5dd
Reviewed-on: http://review.couchbase.org/55792
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-on: http://review.couchbase.org/56063

5 years agoMB-16500 [BP]: Address possible data race in ConnHandler: lastWalkTime 62/56062/2
abhinavdangeti [Mon, 5 Oct 2015 22:11:24 +0000 (15:11 -0700)]
MB-16500 [BP]: Address possible data race in ConnHandler: lastWalkTime

WARNING: ThreadSanitizer: data race (pid=26185)

  Write of size 4 at 0x7d4c0000a154 by main thread:
    #0 ConnHandler::setLastWalkTime() /home/abhinav/couchbase/ep-engine/src/tapconnection.h:356 (ep.so+0x000000065641)
    #1 EvpDcpStep(engine_interface*, void const*, dcp_message_producers*) /home/abhinav/couchbase/ep-engine/src/ep_engine.cc:1478 (ep.so+0x0000000b4d5b)
    #2 mock_dcp_step(engine_interface*, void const*, dcp_message_producers*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:476 (engine_testapp+0x0000000baf95)
    #3 dcp_stream(engine_interface*, engine_interface_v1*, char const*, void const*, unsigned short, unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, int, int, int, int, bool, bool, unsigned char, bool, unsigned long*, bool) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:4164 (ep_testsuite.so+0x0000000990df)
    #4 test_dcp_producer_stream_req_dgm(engine_interface*, engine_interface_v1*) /home/abhinav/couchbase/ep-engine/tests/ep_testsuite.cc:4564 (ep_testsuite.so+0x000000077634)
    #5 execute_test(test, char const*, char const*) /home/abhinav/couchbase/memcached/programs/engine_testapp/engine_testapp.cc:1090 (engine_testapp+0x0000000b946c)
    #6 __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 (libc.so.6+0x000000021ec4)

  Previous read of size 4 at 0x7d4c0000a154 by thread T10 (mutexes: write M1367):
    #0 ConnHandler::getLastWalkTime() /home/abhinav/couchbase/ep-engine/src/tapconnection.h:360 (ep.so+0x000000049cbe)
    #1 ConnManager::run() /home/abhinav/couchbase/ep-engine/src/connmap.cc:150 (ep.so+0x00000005031e)
    #2 ExecutorThread::run() /home/abhinav/couchbase/ep-engine/src/executorthread.cc:112 (ep.so+0x0000000f8746)
    #3 launch_executor_thread(void*) /home/abhinav/couchbase/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f82e5)
    #4 platform_thread_wrap /home/abhinav/couchbase/platform/src/cb_pthreads.c:23 (libplatform.so.0.1.0+0x000000003d31)

Change-Id: I2c5024afde6cb749aad901572bfd68734f6d7d5d
Reviewed-on: http://review.couchbase.org/55780
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-on: http://review.couchbase.org/56062

5 years agoMerge remote-tracking branch 'couchbase/3.0.x' into 'sherlock' 33/55933/1
abhinavdangeti [Thu, 8 Oct 2015 21:10:14 +0000 (14:10 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x' into 'sherlock'

couchbase/3.0.x:
* MB-16357: Interlock expiry and vbucket state changes

Change-Id: Ie4ba4138d23a4b811bd43ee182a641c99b5896ad

5 years agoMB-16357: Interlock expiry and vbucket state changes 46/55646/8
Jim Walker [Thu, 1 Oct 2015 16:12:19 +0000 (17:12 +0100)]
MB-16357: Interlock expiry and vbucket state changes

Expiry should only occur whilst the vbucket is active.
Background tasks performing expiry deletion must stop
driving deletions when the vb changes status to !active.

Using a reader/writer lock the core deleteExpiredItem
function which is used by both compaction driven expiry
and the item pager are now interlocked with vbucket::setState()

Change-Id: I19d30c3d7855778613ccb4534a042c0daf627b8c
Reviewed-on: http://review.couchbase.org/55646
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' into sherlock 40/55740/1
abhinavdangeti [Fri, 2 Oct 2015 20:36:21 +0000 (13:36 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x' into sherlock

couchbase/3.0.x:
* MB-16434: In consumer stream get bytes cleared atomically.
* MB-16434: In setDead release streamMutex before cleaning up stream buffer
* [BP] MB-14382 Increase the default number of ht_locks by a factor of approx. 10
* MB-16421: BGFetch to restore an item that is non-resident
* MB-16402:  Ensure objectregistry has an engine when deleting the VBucketMemoryDeletionTask.

Change-Id: I56bf61ddb324f2472682f3b3ac7f86e81e3f6636

5 years agoMB-16434: In consumer stream get bytes cleared atomically. 08/55708/2
Manu Dhundi [Thu, 1 Oct 2015 22:38:14 +0000 (15:38 -0700)]
MB-16434: In consumer stream get bytes cleared atomically.

When a comsumer stream is set to dead we clear the consumer buffer and
notify the producer of the number of bytes cleared. Clearing the
consumer buffer and the accounting of the bytes cleared should be done
atomically

Change-Id: I602d5307c6c2bbd3dc7f03f1d6c43cbe294389ac
Reviewed-on: http://review.couchbase.org/55708
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-16434: In setDead release streamMutex before cleaning up stream buffer 07/55707/5
Manu Dhundi [Thu, 1 Oct 2015 22:35:47 +0000 (15:35 -0700)]
MB-16434: In setDead release streamMutex before cleaning up stream buffer

To avoid lock order inversion in DCP passive stream we must release streamMutex
before acquiring bufMutex. This is because while processing incoming mutations
on dcp consumer we acquire bufMutex first and then streamMutex.

Change-Id: I129d014dc3a7ec91416af04851419782b89cea23
Reviewed-on: http://review.couchbase.org/55707
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years ago[BP] MB-14382 Increase the default number of ht_locks by a factor of approx. 10 05/55705/3
Chiyoung Seo [Fri, 27 Mar 2015 21:58:27 +0000 (14:58 -0700)]
[BP] MB-14382 Increase the default number of ht_locks by a factor of approx. 10

This change is backported from sherlock.

From the recent perf test results, we observed the lock contention on hash table
buckets when a hash table scanning task (i.e., expiry pager, item pager,
accecss scanner, etc.) is running.

As a long term solution, we may need to implement resizing the number of
hash table locks dynamically at runtime.

Change-Id: Ic7b9f951f58fe7190d8d0d23fb62979057e545ac
Reviewed-on: http://review.couchbase.org/48869
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-on: http://review.couchbase.org/55705
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-16421: BGFetch to restore an item that is non-resident 80/55680/3
abhinavdangeti [Thu, 1 Oct 2015 00:40:14 +0000 (17:40 -0700)]
MB-16421: BGFetch to restore an item that is non-resident

In a full eviction scenario, a bgfetch is to restore an item
not just if the hash table item is a temp-initial item, but
if the hash table item is non-resident as well.

Change-Id: I127b0cbe7034133a656b046cd4022635be23aedd
Reviewed-on: http://review.couchbase.org/55680
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-16402: Ensure objectregistry has an engine when 94/55594/2
Jim Walker [Tue, 29 Sep 2015 09:04:31 +0000 (10:04 +0100)]
MB-16402:  Ensure objectregistry has an engine when
deleting the VBucketMemoryDeletionTask.

Ensure the VBucketMemoryDeletionTask is finished before shutting down
to avoid the vbucket deletion occuring on the task with no engine pointer
in thread-local storage (for ObjectRegistry).

This is a backport to 3.0.x of MB-14041

Change-Id: I63caf59bce0e89ed166bffcbd2d0965a91656725
Reviewed-on: http://review.couchbase.org/55594
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15914: During init only let RW instances of kvstore to remove temp files 70/55470/3
abhinavdangeti [Fri, 21 Aug 2015 18:38:34 +0000 (11:38 -0700)]
MB-15914: During init only let RW instances of kvstore to remove temp files

(This is a backport from master)

Do not let RO instances of the couchkvstore even attempt to remove
stale or temporary files during initialization. This would ensure
that there's no spamming of the logs with this message during init:

"A read-only instance of the underlying store was not allowed to delete
a temporary file:.."

Change-Id: I027af9443ec14905608b1e1eab2d817f27b1de0f
Reviewed-on: http://review.couchbase.org/54744
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-on: http://review.couchbase.org/55470
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' 04/55404/1
Manu Dhundi [Mon, 21 Sep 2015 18:12:56 +0000 (11:12 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x'

Change-Id: I0e85a6f9231524ebe4ca944d08f62fca1196a6fe

5 years agoMB-16310: Use metaKeyIndex for the chk_pt meta items 01/55401/2
Manu Dhundi [Mon, 21 Sep 2015 17:32:11 +0000 (10:32 -0700)]
MB-16310: Use metaKeyIndex for the chk_pt meta items

Fixing the bug in http://review.couchbase.org/#/c/55324/

Change-Id: I67f6c70e5d497a12ff70a836215d44a2f908d261
Reviewed-on: http://review.couchbase.org/55401
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' 65/55365/1
Manu Dhundi [Sat, 19 Sep 2015 00:26:36 +0000 (17:26 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x'

* MB-16310

Change-Id: Idec5d6da3653f435715dcc8456700cbe05539e8e

5 years agoMB-16310: Use separate key index for chk_pt meta keys and application keys 24/55324/4
Manu Dhundi [Fri, 18 Sep 2015 18:04:34 +0000 (11:04 -0700)]
MB-16310: Use separate key index for chk_pt meta keys and application keys

Checkpoint uses meta items "dummy_key", "checkpoint_start", "checkpoint_end".
If application tries to store keys with the same names, then we get runtime
errors. This solution addresses the problem by maintaining separate
key indices for the checkpoint meta keys and application keys.

Change-Id: I38aa22ac007bcfe4c9064b73930d08827604a923
Reviewed-on: http://review.couchbase.org/55324
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' 82/55182/1
Manu Dhundi [Fri, 11 Sep 2015 00:55:16 +0000 (17:55 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x'

* MB-16093

Change-Id: I74735d2f80af5380cb6bbe1bdd8277e79dfbe45f

5 years agoMB-16093: Add logs for closing passive stream and stream end status 46/55146/3 v3.1.1
Manu Dhundi [Fri, 11 Sep 2015 00:07:08 +0000 (17:07 -0700)]
MB-16093: Add logs for closing passive stream and stream end status

Change-Id: I7afd17e6a86c90ffae302c13f3269a587da674ca
Reviewed-on: http://review.couchbase.org/55146
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' 01/55001/1
Manu Dhundi [Wed, 2 Sep 2015 23:01:34 +0000 (16:01 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x'

Change-Id: I69c56a71aaf55e45d9dbc83baf986b891dcadcbc

5 years agoMB-16159: DCP consumer to explicitly notify memcached to get flow ctl buffer ack 96/54996/2
Manu Dhundi [Wed, 2 Sep 2015 20:47:19 +0000 (13:47 -0700)]
MB-16159: DCP consumer to explicitly notify memcached to get flow ctl buffer ack

In DCP consumer, when sufficient bytes are drained it is good to notify
memcached to get the flow control out. Previous method of waiting for Connection
Manager daemon task to notify memcached will cause delays while streaming
large items.

Change-Id: If71c9186f3062755d5c301817ec76f9f7eca5dc7
Reviewed-on: http://review.couchbase.org/54996
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock' 97/54997/1
abhinavdangeti [Wed, 2 Sep 2015 20:47:55 +0000 (13:47 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock'

couchbase/3.0.x:
* MB-16178: Removing sanity check while adding DCP on TAP

Change-Id: I8b62a647734128ad9efd00f4f40396500ad01d6c

5 years agoMB-16178: Removing sanity check while adding DCP on TAP 64/54964/7
abhinavdangeti [Wed, 2 Sep 2015 17:47:03 +0000 (10:47 -0700)]
MB-16178: Removing sanity check while adding DCP on TAP

Removing the sanity check which would prevent:
- Adding a DCP passive stream for a vbucket when a TAP consumer is
still connected.

Removing this sanity check as there is no real threat to the server
of it going into an inconsistent state during the upgrade.

Change-Id: If7643b2ebc21404dd4edb984718b322e411e28bc
Reviewed-on: http://review.couchbase.org/54964
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
5 years agoMB-16155: [BP] MB-15374 Cancel all tasks if force flag set during destroy 28/54828/3
abhinavdangeti [Fri, 21 Aug 2015 19:48:46 +0000 (12:48 -0700)]
MB-16155: [BP] MB-15374 Cancel all tasks if force flag set during destroy

Override the blockShutdown flag for tasks if force is set
during shutdown/destroy.

Reviewed-on: http://review.couchbase.org/54746

Change-Id: Ic95807a5d8781455a6d56c61ae053bda47dbf4de
Reviewed-on: http://review.couchbase.org/54828
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
5 years agoMB-16145: Move all backfills from pending to active when possible 97/54797/2
abhinavdangeti [Mon, 24 Aug 2015 22:27:51 +0000 (15:27 -0700)]
MB-16145: Move all backfills from pending to active when possible

Move all backfills from the pending queue to the active queue
to ensure that some of the backfill tasks don't have to wait
longer than they have to before they're processed.

Change-Id: If79da6523162e5a88c532f3fc84fb1d6e07d476a
Reviewed-on: http://review.couchbase.org/54797
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock' 16/54716/1
abhinavdangeti [Thu, 20 Aug 2015 22:48:50 +0000 (15:48 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock'

couchbase/3.0.x:
* MB-16042: Do not process invalid snapshot markers
* [BP] MB-16045: Dcp Mutations/Deletions with seq number 0 are invalid
* MB-16042: [BP] Handling erroneous DCP snapshot markers
* [BP] MB-16044: Handling erroneous DCP mutations/deletions
* MB-16131: [BP] Initialize last_seqno for a passive stream accurately
* MB-16125: Do not wait for certain tasks to shutdown

Change-Id: Ib0ed086ab14d57c7007dd299efa56603343e76c3

5 years agoMB-16042: Do not process invalid snapshot markers 12/54712/5
abhinavdangeti [Thu, 20 Aug 2015 19:09:49 +0000 (12:09 -0700)]
MB-16042: Do not process invalid snapshot markers

Snapshot markers with start > end, are to be
considered as INVALID.

Change-Id: Ibe1922dc388992b830cec7687e0010e5fd26e982
Reviewed-on: http://review.couchbase.org/54712
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years ago[BP] MB-16045: Dcp Mutations/Deletions with seq number 0 are invalid 11/54711/5
abhinavdangeti [Thu, 20 Aug 2015 00:19:58 +0000 (17:19 -0700)]
[BP] MB-16045: Dcp Mutations/Deletions with seq number 0 are invalid

If mutations or deletions are received at a DCP consumer whose
sequence numbers are ZERO (malicious), they need to be dropped
and the error code returned is to be EINVAL.

Change-Id: I920bf969027fae912a5e86164d235d1110f7688b
Reviewed-on: http://review.couchbase.org/54711
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
5 years agoMB-16042: [BP] Handling erroneous DCP snapshot markers 10/54710/5
abhinavdangeti [Fri, 14 Aug 2015 20:24:05 +0000 (13:24 -0700)]
MB-16042: [BP] Handling erroneous DCP snapshot markers

A snapshot marker whose start seqno and end seqno
are both lesser than the last received mutation's
sequence number then, we're better off dropping it.

Change-Id: Ic33abae37eb164f212d4306f99c9029535dcb42c
Reviewed-on: http://review.couchbase.org/54710
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
5 years ago[BP] MB-16044: Handling erroneous DCP mutations/deletions 09/54709/5
abhinavdangeti [Mon, 17 Aug 2015 17:47:45 +0000 (10:47 -0700)]
[BP] MB-16044: Handling erroneous DCP mutations/deletions

Adding some sanity-check code which ensures that
erroneous mutations and deletions are dropped when
sent in from an autonomous producer which could
inject assertions otherwise.

Change-Id: I48b68783314133e3cf3e1e5b77a61ee043e73115
Reviewed-on: http://review.couchbase.org/54709
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-16131: [BP] Initialize last_seqno for a passive stream accurately 08/54708/4
abhinavdangeti [Fri, 14 Aug 2015 17:42:28 +0000 (10:42 -0700)]
MB-16131: [BP] Initialize last_seqno for a passive stream accurately

last_seqno for a passive stream is to point to the
vbucket high_seqno to ensure that erroneous packets
are handled correctly when received at the consumer.

Change-Id: I077ad5b2ca08c3d4bfb81237b46f259a60d3c4dc
Reviewed-on: http://review.couchbase.org/54708
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-16125: Do not wait for certain tasks to shutdown 06/54706/2
abhinavdangeti [Thu, 20 Aug 2015 18:08:09 +0000 (11:08 -0700)]
MB-16125: Do not wait for certain tasks to shutdown

- Access scanner
- Vbucket compaction

+ Additional refactoring in tasks.h for a parameter
to indicate its meaning - completeBeforeShutdown

Change-Id: I68ac8364177733559926f0ee87acd3d2852e3585
Reviewed-on: http://review.couchbase.org/54706
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoFix compilation warnings while printing variables in log messages 10/54610/2
abhinavdangeti [Thu, 20 Aug 2015 01:10:24 +0000 (18:10 -0700)]
Fix compilation warnings while printing variables in log messages

opaque --> uint32_t
last_read_seqno --> uint64_t

Change-Id: I1c5011b4ee6ff5efad2f6c55c4e8358257818e66
Reviewed-on: http://review.couchbase.org/54610
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
5 years agoFix incorrect arguments to log message in DcpConsumer 09/54609/2
abhinavdangeti [Thu, 20 Aug 2015 01:03:46 +0000 (18:03 -0700)]
Fix incorrect arguments to log message in DcpConsumer

Change-Id: Ic90703caff388523d3acafd4a57195feb76b2d12
Reviewed-on: http://review.couchbase.org/54609
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock' 08/54608/1
abhinavdangeti [Thu, 20 Aug 2015 00:33:04 +0000 (17:33 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock'

couchbase/3.0.x:
* MB-15926: Do not add passive stream if tap consumer exists & vice-versa

Change-Id: I22cb175a7e7230f303a78b4386782b260c616128

5 years agoMB-16045: Dcp Mutations/Deletions with seq number 0 are invalid 04/54604/4
abhinavdangeti [Thu, 20 Aug 2015 00:19:58 +0000 (17:19 -0700)]
MB-16045: Dcp Mutations/Deletions with seq number 0 are invalid

If mutations or deletions are received at a DCP consumer whose
sequence numbers are ZERO (malicious), they need to be dropped
and the error code returned is to be EINVAL.

Change-Id: I920bf969027fae912a5e86164d235d1110f7688b
Reviewed-on: http://review.couchbase.org/54604
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
5 years agoMB-15926: Do not add passive stream if tap consumer exists & vice-versa 56/54556/6
abhinavdangeti [Wed, 19 Aug 2015 21:28:58 +0000 (14:28 -0700)]
MB-15926: Do not add passive stream if tap consumer exists & vice-versa

Do not allow the creation of a DCP passive stream for a vbucket
even if a tap consumer is still connected. Similarly, do not allow
the creation of a tap consumer if a DCP passive stream is still
connected for the vbucket.

Change-Id: Ie7ea059cb512ac550fece437a6526081a4ee3fdd
Reviewed-on: http://review.couchbase.org/54556
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
5 years agoMB-16056: Remove unnecessary asserts in DCP 27/54527/8
Manu Dhundi [Tue, 18 Aug 2015 20:51:14 +0000 (13:51 -0700)]
MB-16056: Remove unnecessary asserts in DCP

Change-Id: Ifcacae87c6d025293087739ab4de995e7cb03bfd
Reviewed-on: http://review.couchbase.org/54527
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Manu Dhundi <manu@couchbase.com>
5 years agoMB-16044: Handling erroneous DCP snapshot markers 54/54454/3
abhinavdangeti [Fri, 14 Aug 2015 20:24:05 +0000 (13:24 -0700)]
MB-16044: Handling erroneous DCP snapshot markers

A snapshot marker whose start seqno and end seqno
are both lesser than the last received mutation's
sequence number then, we're better off dropping it.

Change-Id: Ic33abae37eb164f212d4306f99c9029535dcb42c
Reviewed-on: http://review.couchbase.org/54454
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
5 years agoMB-16044: Handling erroneous DCP mutations/deletions 53/54453/3
abhinavdangeti [Mon, 17 Aug 2015 17:47:45 +0000 (10:47 -0700)]
MB-16044: Handling erroneous DCP mutations/deletions

Adding some sanity-check code which ensures that
erroneous mutations and deletions are dropped when
sent in from an autonomous producer which could
inject assertions otherwise.

Change-Id: I48b68783314133e3cf3e1e5b77a61ee043e73115
Reviewed-on: http://review.couchbase.org/54453
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
5 years agoInitialize last_seqno for a passive stream accurately 34/54434/5
abhinavdangeti [Fri, 14 Aug 2015 17:42:28 +0000 (10:42 -0700)]
Initialize last_seqno for a passive stream accurately

last_seqno for a passive stream is to point to the
vbucket high_seqno to ensure that erroneous packets
are handled correctly when received at the consumer.

Change-Id: I077ad5b2ca08c3d4bfb81237b46f259a60d3c4dc
Reviewed-on: http://review.couchbase.org/54434
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15983: During warmup, add lastSnapEnd in vb failover logs if possible. 18/54318/4 v4.0.0
Manu Dhundi [Mon, 10 Aug 2015 22:47:29 +0000 (15:47 -0700)]
MB-15983: During warmup, add lastSnapEnd in vb failover logs if possible.

In case of a hard reboot, during warmup if vbHighSeqno == lastSnapEnd we should
add lastSnapEnd as seqno in the failover log.

Change-Id: If0bea734c5fe4c75b1f2a4fcc556fec4773b51f6
Reviewed-on: http://review.couchbase.org/54318
Tested-by: buildbot <build@couchbase.com>
Tested-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
5 years agoRevert "MB-15926: Revert "Do not allow multiple passive streams for the same vbucket"" 01/54301/2
Chiyoung Seo [Mon, 10 Aug 2015 16:13:40 +0000 (09:13 -0700)]
Revert "MB-15926: Revert "Do not allow multiple passive streams for the same vbucket""

This reverts commit 2a88979c79d33d7bfd86c9561ca49c243e06acb2.

Change-Id: I3a199dba9bc4dca5ae04d8df2d8215785051260e
Reviewed-on: http://review.couchbase.org/54301
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15990: KVShard::setLowPriorityVbSnapshotFlag not working as expected. 90/54290/2
Daniel Owen [Fri, 7 Aug 2015 12:17:16 +0000 (13:17 +0100)]
MB-15990: KVShard::setLowPriorityVbSnapshotFlag not working as expected.

The flag is not set because on the compare_exchange if flag equals
!newvalue, then flag is set to original value, as opposed to the
newvalue.

Change-Id: I029a4d90d601ff53a41926e2806cf9440783ac5c
Reviewed-on: http://review.couchbase.org/54218
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-on: http://review.couchbase.org/54290
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15893: Waking up the executor thread correctly 33/54233/3
abhinavdangeti [Sat, 8 Aug 2015 02:58:17 +0000 (19:58 -0700)]
MB-15893: Waking up the executor thread correctly

When the backfill manager task is to be woken up,
rather than setting the task snooze time to zero
(which wouldn't effectively wake up the executor
thread immediately), wake up the executor thread
for the task.

Change-Id: I64a10166a0f3a6f01d3aed9fc8ad071a0caeba4e
Reviewed-on: http://review.couchbase.org/54233
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
5 years agoMB-15926: Revert "Do not allow multiple passive streams for the same vbucket" 30/54230/3
Manu Dhundi [Fri, 7 Aug 2015 18:10:13 +0000 (11:10 -0700)]
MB-15926: Revert "Do not allow multiple passive streams for the same vbucket"

Reverting http://review.couchbase.org/#/c/54154
Just commenting out portions of code as this is a temporary revert

Change-Id: I1a14c8e45340f33dcfd1db62a32fa5d51d3df684
Reviewed-on: http://review.couchbase.org/54230
Tested-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock' 57/54157/2
abhinavdangeti [Thu, 6 Aug 2015 04:22:42 +0000 (21:22 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock'

couchbase/3.0.x:
* MB-15926: Do not allow multiple passive streams for the same vbucket
* MB-15914: File deletions to be handled only by RW instances
* MB-15914: [Backport] Don't find files in couchkvstore lazily

Change-Id: Ie0c48a2128c786cb29465307cd561172e438b025

5 years agoMB-15926: Do not allow multiple passive streams for the same vbucket 54/54154/4
Manu Dhundi [Thu, 6 Aug 2015 01:44:44 +0000 (18:44 -0700)]
MB-15926: Do not allow multiple passive streams for the same vbucket

If there are multiple requests (across different DCP consumer conns)
to add stream for a vbucket, we honor only one (first) request
Change-Id: I488e23d69174f20f4913d072484420bc450f4168
Reviewed-on: http://review.couchbase.org/54154
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15914: File deletions to be handled only by RW instances 50/54150/6
abhinavdangeti [Wed, 5 Aug 2015 23:17:40 +0000 (16:17 -0700)]
MB-15914: File deletions to be handled only by RW instances

Adding sanity checks to ensure that file deletions are
handled by only read-write instances of the underlying
store and not the read-only instances. Log warnings
when a read-only instance is denied permission to delete
a file.

Change-Id: I166e8a5f2664b7d40fc184ef70573a027c07715a
Reviewed-on: http://review.couchbase.org/54150
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15893 Group AUXIO task priorities together. 20/54120/2
Chiyoung Seo [Tue, 4 Aug 2015 23:16:05 +0000 (16:16 -0700)]
MB-15893 Group AUXIO task priorities together.

Change-Id: I7d9688a4af0a74942296f2401dab129f46844b14
Reviewed-on: http://review.couchbase.org/54120
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
5 years agoMB-15914: [Backport] Don't find files in couchkvstore lazily 45/53945/3
Mike Wiederhold [Fri, 10 Oct 2014 22:51:11 +0000 (15:51 -0700)]
MB-15914: [Backport] Don't find files in couchkvstore lazily

We no longer need to do this because we now have full control of
the engine and this code path is much more deterministic. This
will also fix potential races in updating from dbFileRevMap.

Change-Id: I5c4b0552f279b1e7d0a695071ae503f464891b32
Reviewed-on: http://review.couchbase.org/43118
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-on: http://review.couchbase.org/53945
Tested-by: buildbot <build@couchbase.com>
5 years agoMB-15910: Fix potential segfaults on WIN32 due to incorrect log format strings 24/53924/3
Dave Rigby [Thu, 30 Jul 2015 09:23:34 +0000 (10:23 +0100)]
MB-15910: Fix potential segfaults on WIN32 due to incorrect log format strings

Fix mismatch between LOG() formatting strings and the arguments. The
common pattern here is format strings which:

1) contain a char* (%s) formatter
2) preceded by a mis-sized integer formatter.

This results in printf() on 32bit Windows reading too much (or too
little) from the stack for the integer argument; meaning that when it
attempts to handle the following %s formatter and read the char* it
dereferences an invalid pointer.

Change-Id: I9e8bcddc2afc2f920d1db892da37cd588378e2a1
Reviewed-on: http://review.couchbase.org/53924
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15894: Fix incorrect printf call in ActiveStream::endStream on 32bit 68/53868/2
Dave Rigby [Wed, 29 Jul 2015 14:46:38 +0000 (15:46 +0100)]
MB-15894: Fix incorrect printf call in ActiveStream::endStream on 32bit

A log message using printf-style formatters was incorrectly assuming that
size_t is always unsigned long long (64bit), when it's 32bit on WIN32 (and
other 32bit platforms).

Change-Id: Ic787d41d178ec113cac68ba1b87b856b177c8028
Reviewed-on: http://review.couchbase.org/53868
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' into sherlock 38/53838/1
Chiyoung Seo [Tue, 28 Jul 2015 23:52:11 +0000 (16:52 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x' into sherlock

* couchbase/3.0.x:
  MB-15837 Fix to tap_notify_set_vbucket_state unit test.
  MB-15837 Generate a new vbucket UUID for TAP-based VBucket takeover.

Change-Id: I7fceedd75a861250d2a3a2d56b0a3535cdef1d84

5 years agoMB-15837 Fix to tap_notify_set_vbucket_state unit test. 34/53834/3
Chiyoung Seo [Tue, 28 Jul 2015 23:22:18 +0000 (16:22 -0700)]
MB-15837 Fix to tap_notify_set_vbucket_state unit test.

Change-Id: Ia7af2243a41b86987a76f06b1b1e30dda6a479b0
Reviewed-on: http://review.couchbase.org/53834
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15837 Generate a new vbucket UUID for TAP-based VBucket takeover. 84/53784/4
Chiyoung Seo [Tue, 28 Jul 2015 05:27:10 +0000 (22:27 -0700)]
MB-15837 Generate a new vbucket UUID for TAP-based VBucket takeover.

A new vbucket UUID should be generated for each TAP-based vbucket takeover
completion. Otherwise, it can cause potential data loss after fully
switching from TAP to DCP, because TAP-based replication doesn't synchronize
sequence numbers between active and replica vbuckets.

For more details, please refer to
https://issues.couchbase.com/browse/MB-15837

Change-Id: Id8931bd110417065b244f10c71e18d0b5d47f6d2
Reviewed-on: http://review.couchbase.org/53784
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' 75/53775/1
Chiyoung Seo [Tue, 28 Jul 2015 00:20:29 +0000 (17:20 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x'

* couchbase/3.0.x:
  MB-15826: Handle lower values of snap_st_seqno by DCP client more effectively

Change-Id: I49d616ccf9bae07c1c1bc711290297b9fd9fe372

5 years agoMB-15874: Have sufficient size for dcp stream stats buffer 73/53773/2
Manu Dhundi [Mon, 27 Jul 2015 23:33:54 +0000 (16:33 -0700)]
MB-15874: Have sufficient size for dcp stream stats buffer

Increase the dcp stream stats buffer size from 128 bytes to 1024 bytes.

Change-Id: I979f599c5f915b14b895fff798e4cdf5bddc233c
Reviewed-on: http://review.couchbase.org/53773
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15826: Handle lower values of snap_st_seqno by DCP client more effectively 35/53635/6
Manu Dhundi [Sat, 25 Jul 2015 01:18:29 +0000 (18:18 -0700)]
MB-15826: Handle lower values of snap_st_seqno by DCP client more effectively

If a DCP client passes snap_st_seqno < start_seqno when
start_seqno == snap_end_seqno, then DCP can be more efficient by setting
snap_start_seqno = start_seqno.
Change-Id: Ie59cfed23e9e3855ef0eca6d3b609a53db65c36f
Reviewed-on: http://review.couchbase.org/53635
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15757: Upon merging checkpoints, update snapshot start 65/53565/4
abhinavdangeti [Wed, 22 Jul 2015 21:28:27 +0000 (14:28 -0700)]
MB-15757: Upon merging checkpoints, update snapshot start

After merging the previous checkpoint to the current
checkpoint, the snapshot start of the current checkpoint
needs to be updated with the first item of the current
checkpoint as new items are added into this checkpoint
from the previous checkpoint.

Change-Id: Ieae30e45f3733a8f0bcc3563c14ee1566a9b395c
Reviewed-on: http://review.couchbase.org/53565
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' 49/53549/2
Chiyoung Seo [Wed, 22 Jul 2015 16:30:57 +0000 (09:30 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x'

* couchbase/3.0.x:
  MB-15609 Don't create a new checkpoint on replica vbucket for each snapshot

Change-Id: I5c00c18d55b0b92d5736373dc15b885d9287456f

5 years agoMB-15609 Don't create a new checkpoint on replica vbucket for each snapshot 33/53433/3 v3.1.0
Chiyoung Seo [Mon, 20 Jul 2015 19:18:25 +0000 (12:18 -0700)]
MB-15609 Don't create a new checkpoint on replica vbucket for each snapshot

There was a regression on the disk write queue size that was caused by
the fix (http://review.couchbase.org/#/c/51682/). The main reason was that
less deduplication in the disk write queue was incurred by more frequent
checkpoint creations in replica vbuckets. To resolve this regression,
this change makes sure that a new checkpoint in a replica vbucket should be
created when the memory usage is getting under pressure.

Change-Id: I9db5e1336c9950387f9d492b864fc3c88333c379
Reviewed-on: http://review.couchbase.org/53433
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15769: In notifyStreamReady() check if bufferlog is full within queueLock 41/53441/2
Manu Dhundi [Mon, 20 Jul 2015 22:28:23 +0000 (15:28 -0700)]
MB-15769: In notifyStreamReady() check if bufferlog is full within queueLock

Change-Id: I02436648a6c9df1c832ee1d8dce78c12f0693d2f
Reviewed-on: http://review.couchbase.org/53441
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15740: Lock maxActiveSnoozingBackfills when updating it due to config change 82/53382/2
Manu Dhundi [Sat, 18 Jul 2015 01:44:52 +0000 (18:44 -0700)]
MB-15740: Lock maxActiveSnoozingBackfills when updating it due to config change

updateMaxActiveSnoozingBackfills() is called when value of max_data_size is
changed. Need to lock when updating maxActiveSnoozingBackfills in the function.

Change-Id: Ied793eb86fd24fba9cf3a3a12ea09d21a61d825f
Reviewed-on: http://review.couchbase.org/53382
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15699: Log before we assert due to invalid lastBySeqno in queueDirty() 14/53314/2
Manu Dhundi [Fri, 17 Jul 2015 04:10:15 +0000 (21:10 -0700)]
MB-15699: Log before we assert due to invalid lastBySeqno in queueDirty()

This would be helpful to debug corner cases causing this assertion

Change-Id: I55659e30e5d20e116dfb7abf1b1df2b4c5357c8e
Reviewed-on: http://review.couchbase.org/53314
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15699: Assert that chkpt is not closed for replica vb until snapshot end. 11/53311/8
Manu Dhundi [Fri, 17 Jul 2015 01:33:11 +0000 (18:33 -0700)]
MB-15699: Assert that chkpt is not closed for replica vb until snapshot end.

When we are processing streamed items for a replica vbucket, checkpoint
should not be closed until all the items in the snapshot are processed.
In this commit we are asserting the above condition in queueDirty()
func in checkpointManager and also logging a message.

Change-Id: I6d14bc0df57d685faaa3ee002012090a6e67c2d9
Reviewed-on: http://review.couchbase.org/53311
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15265: minor corrections to cbcompact help text 52/53252/3
Sundar Sridharan [Wed, 15 Jul 2015 18:23:57 +0000 (11:23 -0700)]
MB-15265: minor corrections to cbcompact help text

Change-Id: I53ea4d16341a4bf79b206da24fe7b233df4650ec
Reviewed-on: http://review.couchbase.org/53252
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15479: Add reason in log of endStream; add logHeader in backfill logs 94/53194/3
Manu Dhundi [Tue, 14 Jul 2015 22:25:51 +0000 (15:25 -0700)]
MB-15479: Add reason in log of endStream; add logHeader in backfill logs

Change-Id: I6454a80048597a20ed6622d7ecf0fb898c918f24
Reviewed-on: http://review.couchbase.org/53194
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15680: BackfillManager task to be an AUXIO task 80/53180/2
abhinavdangeti [Tue, 14 Jul 2015 16:46:02 +0000 (09:46 -0700)]
MB-15680: BackfillManager task to be an AUXIO task

BackfillManager task scans vbucket files and
dumps data into the backfill buffers, needs to
be an auxio task.

Change-Id: I9d5af46ec4d5f0edc1be88bb654578c89974e60a
Reviewed-on: http://review.couchbase.org/53180
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
5 years agoMB-15634: Fix memory leak in setFlushParam 83/53083/4
Dave Rigby [Fri, 10 Jul 2015 17:06:33 +0000 (17:06 +0000)]
MB-15634: Fix memory leak in setFlushParam

The msg parameter used to return an error message isn't freed by the
caller, as it is expected to only point to a statically-allocated
string.

Change-Id: I8d098a28f9e36d0741cae634d93cc6d39390a3ab
Reviewed-on: http://review.couchbase.org/53083
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
5 years agoMB-15635: Import json directly 86/53086/2
Bin Cui [Fri, 10 Jul 2015 18:59:15 +0000 (11:59 -0700)]
MB-15635: Import json directly

Since we use pyexe to generate executable on windows, it will include
all dependent libraries during compiling time instead of runtime. So
simplejson library won't be included for cbstats during runtime for
cbstats.

Change-Id: I3da3de3a242345866bb6d61ee992853d247474bf
Reviewed-on: http://review.couchbase.org/53086
Reviewed-by: Bin Cui <bin.cui@gmail.com>
Tested-by: Bin Cui <bin.cui@gmail.com>
5 years agoMB-14425: Populate ep_version with commit SHA 57/52957/3
Chris Hillery [Wed, 8 Jul 2015 08:20:16 +0000 (01:20 -0700)]
MB-14425: Populate ep_version with commit SHA

Change-Id: I8c65ce557e65de4ec4a772750cbeb521d0e98162
Reviewed-on: http://review.couchbase.org/52957
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Dave Finlay <dave.finlay@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
5 years agoMB-15519: Clear backfilled items when a stream is marked as DEAD 89/52789/5
abhinavdangeti [Thu, 2 Jul 2015 19:59:00 +0000 (12:59 -0700)]
MB-15519: Clear backfilled items when a stream is marked as DEAD

Clear out backfilled buffered items, when a stream is marked
as dead while in the backfilling state. This will ensure that
buffered items from this stream do not use up the backfill
buffer blocking other streams that use the same buffer space.

Change-Id: I2f38d99f2a1a30c8dae6b91155471ab8a307d024
Reviewed-on: http://review.couchbase.org/52789
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
5 years agoMB-15470: Add vb_state field in CMD_GET_ALL_VB_SEQNOS request. 84/52684/8
Manu Dhundi [Tue, 30 Jun 2015 19:03:50 +0000 (12:03 -0700)]
MB-15470: Add vb_state field in CMD_GET_ALL_VB_SEQNOS request.

We give the caller an option to specify the state of the vbucket
for which the vb sequence numbers is requested.

Change-Id: I006d248531e88dac93772dcf7ca6894ca08785f9
Reviewed-on: http://review.couchbase.org/52684
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
5 years ago MB-15474: import either simplejson or json 98/52598/2
Bin Cui [Fri, 26 Jun 2015 18:30:12 +0000 (11:30 -0700)]
 MB-15474: import either simplejson or json

Change-Id: Id8b42527404fc02c02695ad9bcf6a6b11ed7f160
Reviewed-on: http://review.couchbase.org/52598
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Bin Cui <bin.cui@gmail.com>
5 years agoMB-14760: Cbepctl help for exp_pager_stime 86/52686/2
abhinavdangeti [Mon, 29 Jun 2015 22:34:42 +0000 (15:34 -0700)]
MB-14760: Cbepctl help for exp_pager_stime

Indicate that a value of 0 for exp_pager_stime
will disabled the expiry pager.

Change-Id: Icc47bd0c58d7a6f5120ef903b9829f2c4bcab7d2
Reviewed-on: http://review.couchbase.org/52686
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' 13/52613/1
Manu Dhundi [Sat, 27 Jun 2015 00:04:22 +0000 (17:04 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x'

Change-Id: Iefa90a90f00301a9349664c703df66e5318908af

5 years agoMB-15404: Update stat call in ep testsuite to track the items in DCP readyQ. 02/52602/4
Manu Dhundi [Fri, 26 Jun 2015 23:16:34 +0000 (16:16 -0700)]
MB-15404: Update stat call in ep testsuite to track the items in DCP readyQ.

Change-Id: Idc10ef1ba39d86062597307daf0c5db800407946
Reviewed-on: http://review.couchbase.org/52602
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Tested-by: Manu Dhundi <manu@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' 00/52600/2
Manu Dhundi [Fri, 26 Jun 2015 21:30:19 +0000 (14:30 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x'

Change-Id: I2e82456549b29f24359df7ca7d9d412ccfa0db05

5 years agoMB-15404: Add stat to track the items in DCP readyQ. 17/52517/8
Manu Dhundi [Thu, 25 Jun 2015 23:50:52 +0000 (16:50 -0700)]
MB-15404: Add stat to track the items in DCP readyQ.

To better debug memory used by DCP, add stats to debug DCP readyQ.
Also add "lastReadSeqNo" (from the disk) to stats.
Change-Id: If0dbb64944549d846084219acc6a793218b4ad23
Reviewed-on: http://review.couchbase.org/52517
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15367: Use simplejson instead of json for backward compatibility 16/52516/2
Bin Cui [Thu, 18 Jun 2015 16:45:47 +0000 (09:45 -0700)]
MB-15367: Use simplejson instead of json for backward compatibility

Change-Id: I602e04ba1e71f6a325ac8fec1684b67025b8e230
Reviewed-on: http://review.couchbase.org/52234
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-on: http://review.couchbase.org/52516
Reviewed-by: Bin Cui <bin.cui@gmail.com>
Tested-by: Bin Cui <bin.cui@gmail.com>
5 years agoMB-15367: Use simplejson instead of json for backward compatibility 34/52234/3
Bin Cui [Thu, 18 Jun 2015 16:45:47 +0000 (09:45 -0700)]
MB-15367: Use simplejson instead of json for backward compatibility

Change-Id: I602e04ba1e71f6a325ac8fec1684b67025b8e230
Reviewed-on: http://review.couchbase.org/52234
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
5 years agoMB-14859: Handle quick successive BG Fetch of a key interleaved with exp pager 45/52445/4
Manu Dhundi [Tue, 23 Jun 2015 20:38:28 +0000 (13:38 -0700)]
MB-14859: Handle quick successive BG Fetch of a key interleaved with exp pager

If two bgfetch are scheduled for a non existing key, and one bgfetch completes
and marks the key as non existant in the hash table, and subsequently expiry
pager removes it from the hash table before the second bgfetch completes,
we need to handle the case appropriately in the complete bgfetch code as
notify the memcached with appropriate return value.
Change-Id: I8eaf54319014ea4039c74d2cbfab21ef275939fe
Reviewed-on: http://review.couchbase.org/52445
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-14889: Don't send the user to membase docs page 23/52423/2
Trond Norbye [Tue, 23 Jun 2015 09:15:18 +0000 (11:15 +0200)]
MB-14889: Don't send the user to membase docs page

Unfortunately this file is shared between the server version
and the old link sent you directly to the 1.8 server page
(which might not be the correct version). The new link takes
you to our docs portal where you may select the right server
version.

Change-Id: I15d9c381b76e296cade062e7efde54569d05be2b
Reviewed-on: http://review.couchbase.org/52423
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock' 07/52407/1
abhinavdangeti [Tue, 23 Jun 2015 01:15:18 +0000 (18:15 -0700)]
Merge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock'

couchbase/3.0.x:
* MB-15413: Consumer to handle snapshot end correctly
* Set the logging level to WARNING if memUsed is too high
* MB-15292: Fix issue in module_tests/atomic_test

Change-Id: Idf8ef76720fcafaecbc4f240fba62458347a71f8

5 years agoMB-15413: Consumer to handle snapshot end correctly 02/52402/2
abhinavdangeti [Tue, 23 Jun 2015 00:33:08 +0000 (17:33 -0700)]
MB-15413: Consumer to handle snapshot end correctly

The DCPConsumer should handle the snapshot end correctly,
and create a new checkpoint only if the last mutation or
deletion returned an ENGINE_SUCCESS.

Change-Id: I52c437d6cf28fd9a8150de6770885a4a2308b34d
Reviewed-on: http://review.couchbase.org/52402
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>