5 years agoMB-15914: [Backport] Don't find files in couchkvstore lazily 16/55416/2 3.1.0-MP1
Mike Wiederhold [Fri, 10 Oct 2014 22:51:11 +0000 (15:51 -0700)]
MB-15914: [Backport] Don't find files in couchkvstore lazily

We no longer need to do this because we now have full control of
the engine and this code path is much more deterministic. This
will also fix potential races in updating from dbFileRevMap.

Change-Id: I5c4b0552f279b1e7d0a695071ae503f464891b32
Reviewed-on: http://review.couchbase.org/43118
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-on: http://review.couchbase.org/53945
Tested-by: buildbot <build@couchbase.com>
Reviewed-on: http://review.couchbase.org/55416
Tested-by: abhinav dangeti <abhinav@couchbase.com>
5 years agoMB-15926: Do not add passive stream if tap consumer exists & vice-versa 20/54720/2
abhinavdangeti [Wed, 19 Aug 2015 21:28:58 +0000 (14:28 -0700)]
MB-15926: Do not add passive stream if tap consumer exists & vice-versa

Do not allow the creation of a DCP passive stream for a vbucket
even if a tap consumer is still connected. Similarly, do not allow
the creation of a tap consumer if a DCP passive stream is still
connected for the vbucket.

Change-Id: Ie7ea059cb512ac550fece437a6526081a4ee3fdd
Reviewed-on: http://review.couchbase.org/54556
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-on: http://review.couchbase.org/54720
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15926: Do not allow multiple passive streams for the same vbucket 19/54719/2
Manu Dhundi [Thu, 6 Aug 2015 01:44:44 +0000 (18:44 -0700)]
MB-15926: Do not allow multiple passive streams for the same vbucket

If there are multiple requests (across different DCP consumer conns)
to add stream for a vbucket, we honor only one (first) request
Change-Id: I488e23d69174f20f4913d072484420bc450f4168
Reviewed-on: http://review.couchbase.org/54154
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-on: http://review.couchbase.org/54719
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15609 Don't create a new checkpoint on replica vbucket for each snapshot 33/53433/3 v3.1.0
Chiyoung Seo [Mon, 20 Jul 2015 19:18:25 +0000 (12:18 -0700)]
MB-15609 Don't create a new checkpoint on replica vbucket for each snapshot

There was a regression on the disk write queue size that was caused by
the fix (http://review.couchbase.org/#/c/51682/). The main reason was that
less deduplication in the disk write queue was incurred by more frequent
checkpoint creations in replica vbuckets. To resolve this regression,
this change makes sure that a new checkpoint in a replica vbucket should be
created when the memory usage is getting under pressure.

Change-Id: I9db5e1336c9950387f9d492b864fc3c88333c379
Reviewed-on: http://review.couchbase.org/53433
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15404: Update stat call in ep testsuite to track the items in DCP readyQ. 02/52602/4
Manu Dhundi [Fri, 26 Jun 2015 23:16:34 +0000 (16:16 -0700)]
MB-15404: Update stat call in ep testsuite to track the items in DCP readyQ.

Change-Id: Idc10ef1ba39d86062597307daf0c5db800407946
Reviewed-on: http://review.couchbase.org/52602
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Tested-by: Manu Dhundi <manu@couchbase.com>
5 years agoMB-15404: Add stat to track the items in DCP readyQ. 17/52517/8
Manu Dhundi [Thu, 25 Jun 2015 23:50:52 +0000 (16:50 -0700)]
MB-15404: Add stat to track the items in DCP readyQ.

To better debug memory used by DCP, add stats to debug DCP readyQ.
Also add "lastReadSeqNo" (from the disk) to stats.
Change-Id: If0dbb64944549d846084219acc6a793218b4ad23
Reviewed-on: http://review.couchbase.org/52517
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15367: Use simplejson instead of json for backward compatibility 16/52516/2
Bin Cui [Thu, 18 Jun 2015 16:45:47 +0000 (09:45 -0700)]
MB-15367: Use simplejson instead of json for backward compatibility

Change-Id: I602e04ba1e71f6a325ac8fec1684b67025b8e230
Reviewed-on: http://review.couchbase.org/52234
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-on: http://review.couchbase.org/52516
Reviewed-by: Bin Cui <bin.cui@gmail.com>
Tested-by: Bin Cui <bin.cui@gmail.com>
5 years agoMB-15413: Consumer to handle snapshot end correctly 02/52402/2
abhinavdangeti [Tue, 23 Jun 2015 00:33:08 +0000 (17:33 -0700)]
MB-15413: Consumer to handle snapshot end correctly

The DCPConsumer should handle the snapshot end correctly,
and create a new checkpoint only if the last mutation or
deletion returned an ENGINE_SUCCESS.

Change-Id: I52c437d6cf28fd9a8150de6770885a4a2308b34d
Reviewed-on: http://review.couchbase.org/52402
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoSet the logging level to WARNING if memUsed is too high 65/52265/4
abhinavdangeti [Thu, 18 Jun 2015 23:06:45 +0000 (16:06 -0700)]
Set the logging level to WARNING if memUsed is too high

Context: DCPBackfill

Change-Id: Ica0f730c6fea98f684f8809f194896ea937e2b1c
Reviewed-on: http://review.couchbase.org/52265
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
5 years agoMB-15292: Fix issue in module_tests/atomic_test 61/52261/2
abhinavdangeti [Thu, 18 Jun 2015 22:32:07 +0000 (15:32 -0700)]
MB-15292: Fix issue in module_tests/atomic_test

Change-Id: I8ef42c7bfd971260f36f2aaa13e40686a7b3dd82
Reviewed-on: http://review.couchbase.org/52261
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
5 years agoMB-15292: Make CouchbaseAtomic::store() atomic 73/51973/4
Dave Rigby [Thu, 11 Jun 2015 18:56:49 +0000 (11:56 -0700)]
MB-15292: Make CouchbaseAtomic::store() atomic

As identified by ThreadSantizer:

WARNING: ThreadSanitizer: data race (pid=59833)
  Write of size 8 at 0x7d240000d3f0 by thread T8:
    #0 CouchbaseAtomic<unsigned long>::store(unsigned long const&, memory_order) /root/couchbase-3.0/ep-engine/src/atomic.h:84 (ep.so+0x0000000414ef)
    #1 CouchbaseAtomic<unsigned long>::operator=(unsigned long const&) /root/couchbase-3.0/ep-engine/src/atomic.h:105 (ep.so+0x0000000401f5)
    #2 Warmup::scheduleEstimateDatabaseItemCount() /root/couchbase-3.0/ep-engine/src/warmup.cc:500 (ep.so+0x000000277991)
    #3 Warmup::step() /root/couchbase-3.0/ep-engine/src/warmup.cc:816 (ep.so+0x000000275124)
    #4 Warmup::transition(int, bool) /root/couchbase-3.0/ep-engine/src/warmup.cc:853 (ep.so+0x0000002754ff)
    #5 Warmup::createVBuckets(unsigned short) /root/couchbase-3.0/ep-engine/src/warmup.cc:491 (ep.so+0x00000027785f)
    #6 WarmupCreateVBuckets::run() /root/couchbase-3.0/ep-engine/src/warmup.h:234 (ep.so+0x00000028cde9)
    #7 ExecutorThread::run() /root/couchbase-3.0/ep-engine/src/executorthread.cc:110 (ep.so+0x0000001a2581)
    #8 launch_executor_thread(void*) /root/couchbase-3.0/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001a1a5a)
    #9 platform_thread_wrap /root/couchbase-3.0/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x000000002f14)

  Previous atomic write of size 8 at 0x7d240000d3f0 by main thread (mutexes: write M670470284968504712):
    #0 __tsan_atomic64_fetch_add <null>:0 (engine_testapp+0x00000008cb48)
    #1 CouchbaseAtomic<unsigned long>::load(memory_order) const /root/couchbase-3.0/ep-engine/src/atomic.h:77 (ep.so+0x0000000446b4)
    #2 CouchbaseAtomic<unsigned long>::operator unsigned long() const /root/couchbase-3.0/ep-engine/src/atomic.h:101 (ep.so+0x000000044575)
    #3 Warmup::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) const /root/couchbase-3.0/ep-engine/src/warmup.cc:901 (ep.so+0x00000027d4ea)
    #4 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /root/couchbase-3.0/ep-engine/src/ep_engine.cc:4405 (ep.so+0x0000001155a9)
    #5 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /root/couchbase-3.0/ep-engine/src/ep_engine.cc:214 (ep.so+0x0000000fa102)
    #6 mock_get_stats /root/couchbase-3.0/memcached/programs/engine_testapp/engine_testapp.c:194 (engine_testapp+0x0000000aeecd)
    #7 wait_for_warmup_complete(engine_interface*, engine_interface_v1*) /root/couchbase-3.0/ep-engine/tests/ep_test_apis.cc:864 (ep_perfsuite.so+0x00000001b1cb)
    #8 test_setup(engine_interface*, engine_interface_v1*) /root/couchbase-3.0/ep-engine/tests/ep_testsuite_common.cc:128 (ep_perfsuite.so+0x00000000dc03)
    #9 execute_test /root/couchbase-3.0/memcached/programs/engine_testapp/engine_testapp.c:1037 (engine_testapp+0x0000000ab85a)
    #10 main /root/couchbase-3.0/memcached/programs/engine_testapp/engine_testapp.c:1296 (engine_testapp+0x0000000a9a19)

Change-Id: I260942712ca471c0d2e0fa3ebc4793d694f9b237
Reviewed-on: http://review.couchbase.org/51973
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15292: Make CouchbaseAtomic::load() atomic 68/51968/4
Dave Rigby [Thu, 11 Jun 2015 18:36:44 +0000 (11:36 -0700)]
MB-15292: Make CouchbaseAtomic::load() atomic

Identified by ThreadSanitizer running ep_perfsuite.so:

WARNING: ThreadSanitizer: data race (pid=51118)
  Atomic write of size 1 at 0x7d4400009d58 by main thread (mutexes: write M5599):
    #0 __tsan_atomic8_compare_exchange_val <null>:0 (engine_testapp+0x000000093f50)
    #1 CouchbaseAtomic<bool>::compare_exchange_strong(bool&, bool, memory_order) /root/couchbase-3.0/ep-engine/src/atomic.h:92 (ep.so+0x00000005575d)
    #2 Flusher::notifyFlushEvent() /root/couchbase-3.0/ep-engine/src/flusher.h:88 (ep.so+0x0000000c6fec)
    #3 EventuallyPersistentStore::queueDirty(RCPtr<VBucket>&, StoredValue*, LockHolder*, bool, bool, bool) /root/couchbase-3.0/ep-engine/src/ep.cc:2826 (ep.so+0x00000009c796)
    #4 EventuallyPersistentStore::add(Item const&, void const*) /root/couchbase-3.0/ep-engine/src/ep.cc:728 (ep.so+0x00000009f673)
    #5 EventuallyPersistentEngine::store(void const*, void*, unsigned long*, ENGINE_STORE_OPERATION, unsigned short) /root/couchbase-3.0/ep-engine/src/ep_engine.cc:2135 (ep.so+0x000000100980)
    #6 EvpStore(engine_interface*, void const*, void*, unsigned long*, ENGINE_STORE_OPERATION, unsigned short) /root/couchbase-3.0/ep-engine/src/ep_engine.cc:229 (ep.so+0x0000000fa21d)
    #7 mock_store /root/couchbase-3.0/memcached/programs/engine_testapp/engine_testapp.c:227 (engine_testapp+0x0000000ade2e)
    #8 storeCasVb11(engine_interface*, engine_interface_v1*, void const*, ENGINE_STORE_OPERATION, char const*, char const*, unsigned long, unsigned int, void**, unsigned long, unsigned short, unsigned int, unsigned char) /root/couchbase-3.0/ep-engine/tests/ep_test_apis.cc:654 (ep_perfsuite.so+0x000000018ec3)
    #9 perf_latency(engine_interface*, engine_interface_v1*, char const*) /root/couchbase-3.0/ep-engine/tests/ep_perfsuite.cc:104 (ep_perfsuite.so+0x0000000097e2)
    #10 perf_latency_baseline(engine_interface*, engine_interface_v1*) /root/couchbase-3.0/ep-engine/tests/ep_perfsuite.cc:169 (ep_perfsuite.so+0x0000000090b7)
    #11 execute_test /root/couchbase-3.0/memcached/programs/engine_testapp/engine_testapp.c:1042 (engine_testapp+0x0000000ab933)
    #12 main /root/couchbase-3.0/memcached/programs/engine_testapp/engine_testapp.c:1296 (engine_testapp+0x0000000a9a19)

  Previous read of size 1 at 0x7d4400009d58 by thread T20:
    #0 CouchbaseAtomic<bool>::load(memory_order) const /root/couchbase-3.0/ep-engine/src/atomic.h:79 (ep.so+0x00000005288c)
    #1 Flusher::canSnooze() /root/couchbase-3.0/ep-engine/src/flusher.h:104 (ep.so+0x00000018e92b)
    #2 Flusher::computeMinSleepTime() /root/couchbase-3.0/ep-engine/src/flusher.cc:239 (ep.so+0x00000018dc87)
    #3 Flusher::step(GlobalTask*) /root/couchbase-3.0/ep-engine/src/flusher.cc:187 (ep.so+0x00000018cb35)

Change-Id: Ie32ca8fa4e662e1244362cbdb0cb2573f80665e2
Reviewed-on: http://review.couchbase.org/51968
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-14256: Add method to get all sequence numbers 58/51858/4
Trond Norbye [Wed, 15 Apr 2015 08:55:28 +0000 (10:55 +0200)]
MB-14256: Add method to get all sequence numbers

This is a backport of commit cb1bd68b8b771b86d7da310f5b42a9ca417570d0 [1]
from 4.x to 3.x which makes `getAllVBucketSequenceNumbers` C++03 compatible.

[1]: https://github.com/couchbase/ep-engine/commit/cb1bd68b8b771b86d7da310f5b42a9ca417570d0


Change-Id: I1c3181b249035f75f9afe891b049304114e9e496
Reviewed-on: http://review.couchbase.org/51858
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-15213: Disable DCP flow control if buffer size passed by client is zero 06/51706/2
Manu Dhundi [Wed, 3 Jun 2015 01:21:42 +0000 (18:21 -0700)]
MB-15213: Disable DCP flow control if buffer size passed by client is zero

We have documented that if DCP client sets flow control buffer size to zero
the DCP producer would not do flow control. So if the flow control buffer
size is set to zero, the producer does not start flow control or disables the
flow control setup before.

Change-Id: I8746c7b65e82f59c268ed4aa6081b35d04571006
Reviewed-on: http://review.couchbase.org/51706
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Manu Dhundi <manu@couchbase.com>
5 years agoMB-15206: Check and add new checkpoint upon receiving snapshot end 82/51682/3
abhinavdangeti [Tue, 2 Jun 2015 18:44:14 +0000 (11:44 -0700)]
MB-15206: Check and add new checkpoint upon receiving snapshot end

In the DCP consumer, upon receiving a snapshot end message,
check and add a new checkpoint for the replica vbucket, to
ensure that items that are in the current checkpoint do not
take up a lot of memory. Any old unreferenced checkpoints
will be removed by the checkpoint-remover and item-pager

Change-Id: I9eb61fb9e71661e4245de9f92f595a9300abffb9
Reviewed-on: http://review.couchbase.org/51682
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
5 years agoMB-14496 Fix to the type mismatch bug in a file's rev number. 90/50390/2
Chiyoung Seo [Wed, 29 Apr 2015 17:18:39 +0000 (10:18 -0700)]
MB-14496 Fix to the type mismatch bug in a file's rev number.

Change-Id: I3e455d6cee9ac24651510badc116a8284418b118
Reviewed-on: http://review.couchbase.org/50390
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years ago[Backport] MB-14374: Properly find the latest rollback point 10/50010/3
Mike Wiederhold [Fri, 10 Apr 2015 19:06:20 +0000 (12:06 -0700)]
[Backport] MB-14374: Properly find the latest rollback point

If we have received a full snapshot on disk then we want to use the
snapshot end sequence number, but if we are in the middle of a
snapshot then we want to use the snapshot start sequence number. We
can figure out what to use by checking the last sequence number

Change-Id: I4da5f8078e5021c22ba28ca5c8ff8f1ece44731e
Reviewed-on: http://review.couchbase.org/50010
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-14003: Set the bySeqno of the HashTable item correctly 60/48560/6
abhinavdangeti [Fri, 20 Mar 2015 20:31:52 +0000 (13:31 -0700)]
MB-14003: Set the bySeqno of the HashTable item correctly

Context: addTAPBackfillItem

Change-Id: I825635cb50b4130dca311ee247cf157c09a76d92
Reviewed-on: http://review.couchbase.org/48560
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
5 years ago[BP] MB-13359: Get DCP next snapshot if there are meta items, but no mutations 62/47962/2
Mike Wiederhold [Fri, 13 Feb 2015 02:23:54 +0000 (18:23 -0800)]
[BP] MB-13359: Get DCP next snapshot if there are meta items, but no mutations

If we only get a snapshot end message when we call getItemsForCursor
then we will consider the snapshot to be empty and pause the stream.
Since a snapshot end message can only be in a closed checkpoint this
means that it's possible that we are pausing the stream for no

Change-Id: I29b8603287b41401fd6f5c1e4d4f185611d5b583
Reviewed-on: http://review.couchbase.org/47087
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-on: http://review.couchbase.org/47962
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
5 years agoMB-13757: Acquire snapshot lock before updating highSeq & snapshot seqs 11/47811/5
abhinavdangeti [Fri, 6 Mar 2015 01:24:09 +0000 (17:24 -0800)]
MB-13757: Acquire snapshot lock before updating highSeq & snapshot seqs

Acquire snapshot lock before updating highSeqno and then the snapshot
sequence numbers for TAP, to avoid the flusher racing with queueDirty
where only highSeqno is updated but not the snapshot sequence numbers.

Change-Id: I2d7c5932c1d4bda316047236218f4ca9336946a3
Reviewed-on: http://review.couchbase.org/47811
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
5 years agoMB-13792: Acquire lock before changing checkpointManager's lastBySeqno 65/47665/5
abhinavdangeti [Thu, 5 Mar 2015 20:23:19 +0000 (12:23 -0800)]
MB-13792: Acquire lock before changing checkpointManager's lastBySeqno

Change-Id: I55e381418b0a5b89704f2e9912caabfa4df8d15c
Reviewed-on: http://review.couchbase.org/47665
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
5 years agoMerge remote-tracking branch 'gerrit/3.0.1' into 3.0.3 48/47248/2
abhinavdangeti [Tue, 24 Feb 2015 02:20:49 +0000 (18:20 -0800)]
Merge remote-tracking branch 'gerrit/3.0.1' into 3.0.3

+ MB-13386: Ensure that purging the highSeqno doesn't happen

Change-Id: Ieacb1d86d747a275181bb238145fd206a14be140

5 years agoMB-13386: Ensure that purging the highSeqno doesn't happen 41/47241/5 3.0.1
abhinavdangeti [Tue, 24 Feb 2015 01:04:19 +0000 (17:04 -0800)]
MB-13386: Ensure that purging the highSeqno doesn't happen

+ The highest seqno is to not be purged for DCP.
+ This change ensures that items are still queued
for deletion if found to be expired.

Change-Id: I8102d5f61989523efc4f39b70b225c05cdd1b128
Reviewed-on: http://review.couchbase.org/47241
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
5 years agoFix build breakage on ubuntu 07/47107/2
abhinavdangeti [Fri, 20 Feb 2015 02:38:14 +0000 (18:38 -0800)]
Fix build breakage on ubuntu

Change-Id: Ic8d7b9c17086a612b55e2c84a26259081d29c8d4
Reviewed-on: http://review.couchbase.org/47107
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
5 years agoMB-13479: Indicate rollback to DCP clients if purge_seqno > snap_start_seqno 94/47094/4
Manu Dhundi [Thu, 19 Feb 2015 23:10:31 +0000 (15:10 -0800)]
MB-13479: Indicate rollback to DCP clients if purge_seqno > snap_start_seqno

The replica may not get all the items if there are purged items in the active
node. Hence this solution proposes to indicate a rollback to seqno 0 in case
the purge_seqno > snap_start_seqno.
Note: Since the occurrence of this scenario is pretty rare, rolling back to 0
may not be a bad thing.

Change-Id: I5c8403115110be136df5d4cb4e2704edc2a4c9e4
Reviewed-on: http://review.couchbase.org/47094
Tested-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
5 years agoMerge remote-tracking branch 'couchbase/3.0.1' into couchbase/3.0.3 41/47041/1
Manu Dhundi [Thu, 19 Feb 2015 01:11:30 +0000 (17:11 -0800)]
Merge remote-tracking branch 'couchbase/3.0.1' into couchbase/3.0.3

Change-Id: I3d214e5b9b7a60831dcb685a6ef85a4ee0996f3e

5 years agoMB-13386: Do not purge the item with highest sequence number from db. 35/47035/2
Manu Dhundi [Wed, 18 Feb 2015 23:07:18 +0000 (15:07 -0800)]
MB-13386: Do not purge the item with highest sequence number from db.

When the highest seq number that DCP is supposed to read from
the db is the last seq number in the db and is purged,
DCP backfill does not know about it and hence waits for it.
This results in DCP connection hang.
To solve this problem, during compaction we do not purge the
item with last(highest) sequence number in the db.

Change-Id: Ib83c335da3f7c0a952e4b760309276f73bff4ccf
Reviewed-on: http://review.couchbase.org/47035
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Manu Dhundi <manu@couchbase.com>
5 years agoMB-10370: Replications' set/delWithMetas to use replication threshold 07/46907/3
abhinavdangeti [Mon, 16 Feb 2015 23:55:42 +0000 (15:55 -0800)]
MB-10370: Replications' set/delWithMetas to use replication threshold

The setWithMetas and deleteWithMetas issue by consumers for
intra-cluster replication will need to work on tapThrottleThreshold
as opposed to mutation_memory_threshold.

Change-Id: I576c9f9961e03e70430d58a192854c6faa14156d
Reviewed-on: http://review.couchbase.org/46907
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
5 years agoMB-11527: Allow tuning of compaction_write_queue_cap 42/46642/3
abhinavdangeti [Wed, 11 Feb 2015 01:02:21 +0000 (17:02 -0800)]
MB-11527: Allow tuning of compaction_write_queue_cap

Configuration parameter: compaction_write_queue_cap
Can be set in runtime through cbepctl

Change-Id: Id35899865509f5de13a9565802aeec7f84a71f3d
Reviewed-on: http://review.couchbase.org/46642
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
5 years agoMB-13205: Setting appropriate memory thresholds 39/46639/4
abhinavdangeti [Tue, 10 Feb 2015 03:16:25 +0000 (19:16 -0800)]
MB-13205: Setting appropriate memory thresholds

mutation_mem_threshold: 93%
backfill_mem_threshold: 96%
replication_mem_threhold: 99%

Change-Id: Ic2abb80266c1e32d55e21e5359fe0ce99e53551f
Reviewed-on: http://review.couchbase.org/46639
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
5 years agoConditionally delete expired items during compaction 38/46638/4
abhinavdangeti [Tue, 10 Feb 2015 19:59:00 +0000 (11:59 -0800)]
Conditionally delete expired items during compaction

Delete expired items during compaction if and only if
memory usage is lesser than threshold
(compaction_exp_mem_threshold) and diskqueue size is
lesser than tap_throttle_queue_cap.

Change-Id: I256b127b32050dc0a1e395cacb369353a2fe0565
Reviewed-on: http://review.couchbase.org/46638
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
5 years ago[Backport] Disable the disk write queue size cap for replication backoff. 14/46614/3
Chiyoung Seo [Wed, 4 Feb 2015 06:44:34 +0000 (22:44 -0800)]
[Backport] Disable the disk write queue size cap for replication backoff.

The disk write queue size cap for replication backoff was determined in
the very early versions of Couchbase Server that were deployed in
small-to-medium sized cluster with spining disks.

In our recent benchmark results, it turns out that disabling or setting
the disk write queue cap to a large value shows much better performance
behaviors in large-scale clusters with SSDs. As Couchbase Server is
deployed in such environments more and more, this change disables the
write queue size cap, but still supports a replication backoff by
checking the memory usage.

Note that the disk write queue size cap is still configurable at runtime.

Change-Id: Iedf711ad1c3bef61ca954f83f802b4a647b9ec88
Reviewed-on: http://review.couchbase.org/46367
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-on: http://review.couchbase.org/46614
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
5 years ago[Backport] MB-13286: New DcpProducer to have paused status set to true 11/46511/2
abhinavdangeti [Wed, 4 Feb 2015 19:15:52 +0000 (11:15 -0800)]
[Backport] MB-13286: New DcpProducer to have paused status set to true

DcpOpen will create a new DcpProducer and if one with a
similar name already exists (e.g - during rebalance),
this new producer will replace the older one, and the
paused status will need to be set to true, to let the
notification for the new connection to be sent to

Change-Id: I40c22601a7d29141741608339c58caa486a698b2
Reviewed-on: http://review.couchbase.org/46374
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-on: http://review.couchbase.org/46511

6 years agoMB-12673: Check for new checkpoint items after takeoverSend 18/43918/2 v3.0.2
abhinavdangeti [Wed, 3 Dec 2014 03:35:25 +0000 (19:35 -0800)]
MB-12673: Check for new checkpoint items after takeoverSend

Check for any new checkpoint items when the readyQ is found to
be empty in the takeoverSendPhase, before setting the old
vbucket to dead and the new vbucket to active.

Change-Id: I2a6ddacc711f5db42a1e3c575ae18d0b2b3126bd
Reviewed-on: http://review.couchbase.org/43918
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-12751: Return KEY EEXISTS for locked objects 25/43625/2
Trond Norbye [Mon, 24 Nov 2014 21:57:29 +0000 (22:57 +0100)]
MB-12751: Return KEY EEXISTS for locked objects

Change-Id: I5eb8a24337a81fed861cc8140bbdcf1a132acd90
Reviewed-on: http://review.couchbase.org/43625
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
6 years agoMB-12647: Ensure CAS value will always be unique 71/43271/3
Sriram Ganesan [Fri, 14 Nov 2014 22:45:58 +0000 (14:45 -0800)]
MB-12647: Ensure CAS value will always be unique

In windows, the gethrtime() returns the same timestamp when 2 requests
are made in a very short interval, thus causing the same CAS to return.
This can race conditions with 2 consecutive requests, resulting in
data corruption.

Change-Id: I4f396de3f14129504ca406ebb8d4c7a9f3a89bd8
Reviewed-on: http://review.couchbase.org/43271
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-12576: Limit writer threads to 4 60/43060/7
Sundar Sridharan [Tue, 11 Nov 2014 06:59:44 +0000 (22:59 -0800)]
MB-12576: Limit writer threads to 4

Having more than 4 writers increases bgfetch latencies in DGM
This change selectively reverts 32a166c511d7b242433011a875402e1278300add
Change-Id: Icdb996622237747e759c52751f2c8e613c9ba262
Reviewed-on: http://review.couchbase.org/43060
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-12562: Avoid terminating all writer threads 80/42880/6
abhinavdangeti [Thu, 6 Nov 2014 05:10:29 +0000 (21:10 -0800)]
MB-12562: Avoid terminating all writer threads

With incremental writer thread scheduling, we
need to make sure that all writer threads aren't
deleted before deleting all buckets.

- maxWriters is 4
- create bucket 1, numWriters=1
- create bucket 2, numWriters=2
- create bucket 3, numWriters=3
- create bucket 4, numWriters=4
- create bucket 5, numWriters=4
- delete bucket 5, numWriters=3 =>with change, 4
- delete bucket 4, numWriters=2 =>with change, 3
- delete bucket 3, numWriters=1 =>with change, 2
- delete bucket 2, numWriters=0 =>with change, 1
- delete bucket 1, numWriters=0 =>with change, 0

Change-Id: Ib9d45f7acb9177924612547538aa74ca9dd49c20
Reviewed-on: http://review.couchbase.org/42880
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-12551: Add all arguments to noop log message 62/42862/2
Mike Wiederhold [Thu, 6 Nov 2014 00:58:19 +0000 (16:58 -0800)]
MB-12551: Add all arguments to noop log message

Change-Id: Id6e8aac2bf9e398dc8c929eb122651884d27b7c1
Reviewed-on: http://review.couchbase.org/42862
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12483: Make sure vbucket_state initializes all variables 29/42729/3
Mike Wiederhold [Mon, 3 Nov 2014 22:00:15 +0000 (14:00 -0800)]
MB-12483: Make sure vbucket_state initializes all variables

We need to make sure that this data is always completely filled
out so that we don't accidentally write garbage data to disk.

Change-Id: I196e7ca9f5bada8e0df90ddb01b6e952650bed56
Reviewed-on: http://review.couchbase.org/42729
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12483: Remove the default constructor in the vbucket_state struct 28/42728/2
Mike Wiederhold [Mon, 3 Nov 2014 20:19:04 +0000 (12:19 -0800)]
MB-12483: Remove the default constructor in the vbucket_state struct

The default constructor can leave uninitialized fields and we persist
this structure to disk. In oreder to prevent garbage from being written
we should remove the default constructor.

Change-Id: I028fac1dc112bb454779a30f695eb180278455df
Reviewed-on: http://review.couchbase.org/42728
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12483: Don't allow recreation during vbucket deletion 77/42677/2
Mike Wiederhold [Fri, 31 Oct 2014 18:44:12 +0000 (11:44 -0700)]
MB-12483: Don't allow recreation during vbucket deletion

When we delete a vbucket we have an option to recreate the file
immediately. Doing this is incorrect because we will not know
what the failover log of the new vbucket looks like until we
actually create it in memory. This can lead to a situation where
there is no failover log and as a result the local doc json will
be invalid. If the server is shutdown right after this happens and
then is restarted the vbuckets might be created with garbage values
in some of their fields.

Change-Id: I70e6335af68746aeac49a336da5e33b70dfcfe0e
Reviewed-on: http://review.couchbase.org/42677
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12279: Incrementally spawn writer threads for buckets 52/42552/3
Sundar Sridharan [Tue, 28 Oct 2014 23:02:16 +0000 (16:02 -0700)]
MB-12279: Incrementally spawn writer threads for buckets

This change is needed to mitigate the high bgfetch latency
observed in heavy Data-Greater-than-Memory scenarios because
having high number of writing threads slows down disk read
Also mitigates MB-11143 slowdown in single HDD case

NOTE: This may slow down disk persistence in fast SSDs by default
NOTE: cbepctl can still be used to dynamically tune writers at runtime

Change-Id: Iddf0d3094f38b66ba8c0e09d6d6a307d15b38d56
Reviewed-on: http://review.couchbase.org/42552
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-12305: Don't update the snap start/end seqno in tap mutation 03/42503/3
Mike Wiederhold [Thu, 23 Oct 2014 17:57:48 +0000 (10:57 -0700)]
MB-12305: Don't update the snap start/end seqno in tap mutation

We do this in the queueDirty function so this code is not needed. It
also appeared to cause a race in updating the snapshot start and end
sequence numbers so removing it should solve the bug linked above.

Change-Id: Ia8fa36df958be9147ea208ba9ebd78496048ebb4
Reviewed-on: http://review.couchbase.org/42503
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12398: log incorrect timeout values 87/42387/2
Trond Norbye [Thu, 23 Oct 2014 12:31:12 +0000 (14:31 +0200)]
MB-12398: log incorrect timeout values

The loglevel DEBUG won't appear in any logs, so customers using
this in production won't get the information anywhere (the server
just change their requested lock time without telling the user).

I guess a better behavior here would be to return ERANGE in
this case, but that would potentially break user applications
and isn't something we should do in a patch revision.

Change-Id: Ib6a37deceb2755b9fea53b0542a13fb2e16a3261
Reviewed-on: http://review.couchbase.org/42387
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoHide the -a argument in cbstats 97/42297/4
Mike Wiederhold [Mon, 20 Oct 2014 20:51:10 +0000 (13:51 -0700)]
Hide the -a argument in cbstats

The -a argument is used by cbcollectinfo so we can't just
get rid of it.

Change-Id: I0f04c2d454c117271b44176df904e90a938965e9
Reviewed-on: http://review.couchbase.org/42297
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12226: Handle ENOENT error during unlink 73/42173/3
Sriram Ganesan [Wed, 15 Oct 2014 21:12:06 +0000 (14:12 -0700)]
MB-12226: Handle ENOENT error during unlink

If the file is not found during an unlink, it shouldn't be added
to the pending file deletions queue.

Change-Id: Ief306277bbbc946ae18e39dd4819f811f12ea76c
Reviewed-on: http://review.couchbase.org/42173
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sriram Ganesan <sriram@couchbase.com>
6 years agoMB-12271: Set the default dcp producer noop interval to 20 seconds 36/42136/4
Mike Wiederhold [Tue, 14 Oct 2014 18:10:21 +0000 (11:10 -0700)]
MB-12271: Set the default dcp producer noop interval to 20 seconds

This is a fix to support backwards compatibility between 3.0 and
3.0.1+ versions of Couchbase. The problem is that 3.0 has a noop
interval of 20 seconds by default and 3.0.1 has a default noop
interval of 200 seconds. In 3.0.1 the consumer explicitly sets
the noop interval of the producer so in 3.0.1+ clusters the 20
second default will be overriden by the consumer, but in the
scenario when a 3.0.1+ node connects to a 3.0 node the 20 second
default will remain.

Change-Id: I2e18e9ad68037f3a82abe5167f2bca89f381f318
Reviewed-on: http://review.couchbase.org/42136
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11454: Add an option to output json in cbstats 84/42084/2
Mike Wiederhold [Mon, 13 Oct 2014 17:56:14 +0000 (10:56 -0700)]
MB-11454: Add an option to output json in cbstats

Change-Id: I0adcb16adde82ddf3671e4edc0118061d684f2d4
Reviewed-on: http://review.couchbase.org/42084
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12271: Set noop interval individually for dcp connections 49/42049/3
Mike Wiederhold [Sat, 11 Oct 2014 00:00:53 +0000 (17:00 -0700)]
MB-12271: Set noop interval individually for dcp connections

We need to do this because it might be the case that two servers
have their respective noop intervals set to different values. If
they are this can cause the connections to be disconnected because
each side is expecting to see a noop at different times.

Change-Id: I6ff475ccba407547e7285fa431b86ad9bf9cdc24
Reviewed-on: http://review.couchbase.org/42049
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoAccount for incorrect entries through cbepctl 83/42083/3
abhinavdangeti [Mon, 13 Oct 2014 17:44:14 +0000 (10:44 -0700)]
Account for incorrect entries through cbepctl

Context: access_scanner_enabled
Relates to: http://review.couchbase.org/#/c/40884/

Change-Id: I35134c91ed03f6ba6093cfd71270484beca1a4cd
Reviewed-on: http://review.couchbase.org/42083
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11642: Change the priority based on the type of dcp connection 58/41758/3
Mike Wiederhold [Tue, 23 Sep 2014 23:06:48 +0000 (16:06 -0700)]
MB-11642: Change the priority based on the type of dcp connection

Change-Id: I1b9a6846879385308bee3920bfa182fc41e39b4f
Reviewed-on: http://review.couchbase.org/41758
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agouse optparse instead of arg_parse cbvdiff for compatibility 52/41752/4
Sundar Sridharan [Mon, 29 Sep 2014 21:32:52 +0000 (14:32 -0700)]
use optparse instead of arg_parse cbvdiff for compatibility

Change-Id: I604354823c71b91a167646c01e2bb6ea9d8c8822
Reviewed-on: http://review.couchbase.org/41752
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-12268: Adding a debug log, when clusterConfig is updated 46/41746/2
abhinavdangeti [Mon, 29 Sep 2014 18:38:23 +0000 (11:38 -0700)]
MB-12268: Adding a debug log, when clusterConfig is updated

Change-Id: I0cbb70789f84bddc644bdc4ee59c1e86e1821220
Reviewed-on: http://review.couchbase.org/41746
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-12160: Use appropriate CAS for comparison with locked items 94/41594/8
Sriram Ganesan [Tue, 23 Sep 2014 23:27:16 +0000 (16:27 -0700)]
MB-12160: Use appropriate CAS for comparison with locked items

In the case of setWithMeta/deleteWithMeta commands, the locked item's
CAS value needs to be compared with the incoming mutation's CAS.

Change-Id: Id12a3c4717b18bc41c3d4b7dded99ea215179e9d
Reviewed-on: http://review.couchbase.org/41594
Tested-by: Sriram Ganesan <sriram@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoAccount for keylength while allocating buffer for allKeys 95/41595/3
abhinavdangeti [Wed, 24 Sep 2014 00:17:21 +0000 (17:17 -0700)]
Account for keylength while allocating buffer for allKeys

While increasing the buffersize (when needed) when the
AllKeysAPI is invoked, we will need to account for two
additional bytes while allocating the key, for its length.

Change-Id: Iba68c4ae7bccae20f97d4e98350d5105093c3487
Reviewed-on: http://review.couchbase.org/41595
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-12117: Release hashtable locks before disk IO 72/41572/4
abhinavdangeti [Wed, 24 Sep 2014 18:29:10 +0000 (11:29 -0700)]
MB-12117: Release hashtable locks before disk IO

While in access log generation, we'll need to release
all hashtable partition locks before we create new
entries in the mutation log.

Change-Id: Ic3dd0a02452b51ee742e30a0f268b86f9ab6205b
Reviewed-on: http://review.couchbase.org/41572
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11999: Preferential loading of active/replica vbuckets 68/41468/3
abhinavdangeti [Thu, 18 Sep 2014 22:44:32 +0000 (15:44 -0700)]
MB-11999: Preferential loading of active/replica vbuckets

During warmup, order the vbucket list such that
active vbuckets get 60% preference while replica
vbuckets get 40% preference.

In a 4 node cluster (DGM), approx. resident ratios:
1. Before warmup
    active:     36%     34%     34%     41%
    replica:    33%     35%     35%     27%
2. After warmup
    active:     40%     42%     41%     46%
    replica:    31%     29%     29%     24%

Change-Id: I60e0427bca58530247086d730135ebb4be70bb84
Reviewed-on: http://review.couchbase.org/41468
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMerge remote-tracking branch 'gerrit/2.5.0' into 3.0.1 60/41460/2
Mike Wiederhold [Wed, 17 Sep 2014 19:32:20 +0000 (12:32 -0700)]
Merge remote-tracking branch 'gerrit/2.5.0' into 3.0.1

Change-Id: I59c45877fc783e30c86dcfd7d303c731cde433fc

6 years agoFix deadlock in checkpoint persistence command 63/41363/2
Mike Wiederhold [Thu, 11 Sep 2014 18:46:00 +0000 (11:46 -0700)]
Fix deadlock in checkpoint persistence command

We need to release the hpChkMutex before notifying memcached in
order to prevent a deadlock. The deadlock occurs when the flusher
trys to notify memcached of checkpoint persistence at the same
time a memcached worker thread is trying to add a new checkpoint
persistence request.

Change-Id: Ida313f5b39ef0e063dee9882410cd0a19ce55292
Reviewed-on: http://review.couchbase.org/41363
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMerge remote-tracking branch 'gerrit/3.0' into 3.0.1 18/41318/1
Mike Wiederhold [Wed, 10 Sep 2014 18:34:38 +0000 (11:34 -0700)]
Merge remote-tracking branch 'gerrit/3.0' into 3.0.1

Change-Id: I195452c3d0684dff5ff673efd00a18b9d70da3cf

6 years agoMB-12137: Don't update the current snapshot during persistence 89/41289/2 3.0 v3.0.0
Mike Wiederhold [Tue, 9 Sep 2014 21:31:07 +0000 (14:31 -0700)]
MB-12137: Don't update the current snapshot during persistence

If we update the current snapshot only when we persist items to
disk then we may miss out on updates to the current snapshot
when there are vbucket state changes.

Change-Id: Iea3139afb669bdf32e6b4d98e8b3515cafe578dc
Reviewed-on: http://review.couchbase.org/41289
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoFix build breakage on Linux 91/41291/2
Sriram Ganesan [Tue, 9 Sep 2014 23:25:11 +0000 (16:25 -0700)]
Fix build breakage on Linux

Remove usage of constant strings for maintaining file deletion queue

Change-Id: I1992343ff65f923b6d3ffbf7a25932b936e3ff7c
Reviewed-on: http://review.couchbase.org/41291
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Sriram Ganesan <sriram@couchbase.com>
6 years agoHandle unlink/remove failures 68/41268/5
Sriram Ganesan [Fri, 5 Sep 2014 23:01:46 +0000 (16:01 -0700)]
Handle unlink/remove failures

The unlink/remove function can fail if there is another process that
has an open file handle on that file. In such cases, we need to retry
the unlink periodically in the flusher task until we get rid of those

Change-Id: I4bfcf29b3fa866ec4946db658a245c722f3725ce
Reviewed-on: http://review.couchbase.org/41268
Tested-by: Sriram Ganesan <sriram@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoremove warmup wait dead code to prevent accidental deadlock 06/41206/2
Sundar Sridharan [Thu, 4 Sep 2014 21:33:03 +0000 (14:33 -0700)]
remove warmup wait dead code to prevent accidental deadlock

Since we should never have a front-end thread wait for background
thread, having a warmup wait logic will ensure that we do not use
it accidentally in the future and cause a deadlock

Change-Id: I28ec96dceaffa02810415402226df20f8eb44cc7
Reviewed-on: http://review.couchbase.org/41206
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoMB-12063 Don't delete expired items during warmup. 19/41219/3
Chiyoung Seo [Fri, 5 Sep 2014 06:15:38 +0000 (23:15 -0700)]
MB-12063 Don't delete expired items during warmup.

When a node is added back to the cluster, its vbucket states
can be different from its previous states (e.g., active -> replica).
Therefore, we should not remove expired items from active vbuckets
during warmup because the states of those vbuckets can be changed
after the warmup.

Change-Id: Ifec4fda80e424a8afa74bae66e17de77b2e0c36b
Reviewed-on: http://review.couchbase.org/41219
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMerge remote-tracking branch 'gerrit/3.0' into 3.0.1 96/41196/2
Mike Wiederhold [Thu, 4 Sep 2014 17:56:29 +0000 (10:56 -0700)]
Merge remote-tracking branch 'gerrit/3.0' into 3.0.1

Change-Id: I678740458ccc13cbf79be847369cf6561b5b678b

6 years agoMB-12100: Don't expire items for replica vbuckets during warmup 77/41177/2
Mike Wiederhold [Wed, 3 Sep 2014 22:16:31 +0000 (15:16 -0700)]
MB-12100: Don't expire items for replica vbuckets during warmup

Change-Id: I81095beb69c0996d819158e3cadc451a83653d2f
Reviewed-on: http://review.couchbase.org/41177
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoUse the Dcp noop interval for dcp connections 94/40994/2
Mike Wiederhold [Wed, 27 Aug 2014 17:43:03 +0000 (10:43 -0700)]
Use the Dcp noop interval for dcp connections

We were accidentally using the tap noop interval.

Change-Id: Ib7d6615d3128095e8f29a12afa1922961f5aa197
Reviewed-on: http://review.couchbase.org/40994
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoRename tapconnmap to connmap since it handles both dcp and tap 93/40993/2
Mike Wiederhold [Wed, 27 Aug 2014 17:41:49 +0000 (10:41 -0700)]
Rename tapconnmap to connmap since it handles both dcp and tap

Change-Id: I698a2bfe62ab4dec5e64934de845611d49860041
Reviewed-on: http://review.couchbase.org/40993
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoFix build breakage 12/41012/2
Mike Wiederhold [Thu, 28 Aug 2014 07:05:45 +0000 (00:05 -0700)]
Fix build breakage

Change-Id: Ibeac0d57fc1ea3bc94bf488ee5e82cf7e9143d3a
Reviewed-on: http://review.couchbase.org/41012
Reviewed-by: Chris Hillery <ceej@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12063: Properly save the max deleted seqno during vb snapshots 97/40997/3
Mike Wiederhold [Wed, 27 Aug 2014 22:47:51 +0000 (15:47 -0700)]
MB-12063: Properly save the max deleted seqno during vb snapshots

Change-Id: Ife9e3b26956638622de0427cd573cd4d37c7e443
Reviewed-on: http://review.couchbase.org/40997
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoAvoid getting config lock on every unlinkCouchFile 04/41004/2
abhinavdangeti [Thu, 28 Aug 2014 00:18:50 +0000 (17:18 -0700)]
Avoid getting config lock on every unlinkCouchFile

Change-Id: I7a5c3cf81a227ee2ddc02750839cc3b94359fe88
Reviewed-on: http://review.couchbase.org/41004
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-12063: Lock the current snapshot mutex when flushing items 96/40996/2
Mike Wiederhold [Wed, 27 Aug 2014 21:41:08 +0000 (14:41 -0700)]
MB-12063: Lock the current snapshot mutex when flushing items

We need to lock the current snapshot mutex before grabbing items
because it is possible that an itme might be queued, but the
current snapshot might be updated.

Change-Id: Ieff85a4605041e764d6facf055c71c59e7e19955
Reviewed-on: http://review.couchbase.org/40996
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoFix build break on win32 81/40981/2
Trond Norbye [Wed, 27 Aug 2014 18:05:08 +0000 (20:05 +0200)]
Fix build break on win32

Change-Id: Iab0f0cfa7fb61b84aa54f20513f80d4683775073
Reviewed-on: http://review.couchbase.org/40981
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoRemove shutdown check 58/40958/2
Mike Wiederhold [Mon, 4 Aug 2014 18:19:59 +0000 (11:19 -0700)]
Remove shutdown check

We only needed this because we didn't want to notify mccouch.

Change-Id: I63ad9cddbe86930facfd68ceaa3ee87998ba025d
Reviewed-on: http://review.couchbase.org/40958
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoRename all classes/variables with upr to dcp 42/40942/3
Mike Wiederhold [Tue, 26 Aug 2014 21:29:16 +0000 (14:29 -0700)]
Rename all classes/variables with upr to dcp

Change-Id: Iaf644f5034fdb30a14c400ec55def22d8bb0ba87
Reviewed-on: http://review.couchbase.org/40942
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoReplace upr with dcp in all file names 41/40941/3
Mike Wiederhold [Tue, 26 Aug 2014 20:10:12 +0000 (13:10 -0700)]
Replace upr with dcp in all file names

Change-Id: Ia418e96b5bcebf40ef99e3c27ce3fb8f20f68f0e
Reviewed-on: http://review.couchbase.org/40941
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11722: Remove the mcCouch dependency 95/40195/10
abhinavdangeti [Tue, 26 Aug 2014 22:22:44 +0000 (15:22 -0700)]
MB-11722: Remove the mcCouch dependency

- unlinking stale files at the end of compaction
- unlinking files in couchkvstore reset
- unlink files in delVbucket

Change-Id: I000fbd076856b8de1a0dcb82aa673b3316dc745f
Reviewed-on: http://review.couchbase.org/40195
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12057: warmup must not block bucket creation 92/40892/8
Sundar Sridharan [Tue, 26 Aug 2014 21:23:49 +0000 (14:23 -0700)]
MB-12057: warmup must not block bucket creation

Bucket creation is a front end thread and warmup is done by
engine background threads. So if bucket creation waits for
warmup then a deadlock happens when there are few reader threads
Test Infrastructure expects vb0 created by ep-engine init
or warmup. This is not necessary for production, so moving
vb0 creation to test setup. Minor test code bug fixes.
Change-Id: I5eae9c8f36fe39d2a18cb1800dc48a29162eab28
Reviewed-on: http://review.couchbase.org/40892
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMerge remote-tracking branch 'gerrit/3.0' into HEAD 36/40936/2
Mike Wiederhold [Tue, 26 Aug 2014 19:51:51 +0000 (12:51 -0700)]
Merge remote-tracking branch 'gerrit/3.0' into HEAD

Change-Id: Ief314ec953a5847ce4a3f555e9055cae6e2a504d

6 years agoMB-12041: Enable/disable access scanner through cbepctl 84/40884/6
abhinavdangeti [Mon, 25 Aug 2014 20:10:14 +0000 (13:10 -0700)]
MB-12041: Enable/disable access scanner through cbepctl

Enable/disble this way:
./cbepctl localhost:11210 set_flush_param access_scanner_enabled true
./cbepctl localhost:11210 set_flush_param access_scanner_enabled false

access_scanner_task_time will show NOT_SCHEDULED if
access scanner task were disabled.

Change-Id: Ia89c09d55764de38f986db36a4467730534c412c
Reviewed-on: http://review.couchbase.org/40884
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11948: Handle GetLastError() returning 0 condition 65/40865/6
Sriram Ganesan [Fri, 22 Aug 2014 22:41:57 +0000 (15:41 -0700)]
MB-11948: Handle GetLastError() returning 0 condition

In windows, the GetLastError() call sometimes returns 0 instead of
the returning WSAEWOULDBLOCK. Either the error gets reset before
the invocation or it could be undefined behavior.

Change-Id: Ifb8f7d576b447d090b21cc0747d068ead0597182
Reviewed-on: http://review.couchbase.org/40865
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Sriram Ganesan <sriram@couchbase.com>
6 years agoMinor fix which helps Xcode generated project build. 25/40825/3
jim [Fri, 22 Aug 2014 08:21:09 +0000 (09:21 +0100)]
Minor fix which helps Xcode generated project build.

Issue is that the add_custom_command references gencode/genconfig via
a path, yet on Xcode build all output goes into
build/ep-engine/Debug|Release/ thus gencode/genconfig fail dependency

Using just the add_excutable name of gencode/genconfig allows cmake
to use the correct path.

Change-Id: Ie55124487bffe80bc2847b67977738e67d8da855
Reviewed-on: http://review.couchbase.org/40825
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
6 years agoMB-12042 check for existence of vbucket in warmup callback 49/40849/4
Sundar Sridharan [Fri, 22 Aug 2014 20:38:41 +0000 (13:38 -0700)]
MB-12042 check for existence of vbucket in warmup callback

Change-Id: Iab0ad237392d681f28b91c29dc51b7631141e187
Reviewed-on: http://review.couchbase.org/40849
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoMB-11973: Only grab items from the chk manager one chk at a time 65/40765/4
Mike Wiederhold [Wed, 20 Aug 2014 20:36:31 +0000 (13:36 -0700)]
MB-11973: Only grab items from the chk manager one chk at a time

We should only grab a single checkpoint at a time from the checkpoint
manager in order to make sure that the replica vbuckets cannot move
into the open checkpoint. This will also help make sure we don't
copy too make items into an upr stream.

Change-Id: I519353df69cf389e6f32f77ffc6afe97dfba7b36
Reviewed-on: http://review.couchbase.org/40765
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11970: Resident ratio should be 100% when no items are a bucket 56/40756/2
Mike Wiederhold [Wed, 20 Aug 2014 17:50:53 +0000 (10:50 -0700)]
MB-11970: Resident ratio should be 100% when no items are a bucket

Change-Id: I0ef18509fecec56df9c922c44c81ffd5e02eae47
Reviewed-on: http://review.couchbase.org/40756
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12032: Clean up TaskQs after last bucket deletion 41/40741/3
Sundar Sridharan [Wed, 20 Aug 2014 01:48:08 +0000 (18:48 -0700)]
MB-12032: Clean up TaskQs after last bucket deletion

Without this sometimes threads can end up sleeping on
a high priority Q even when none exist.
This can result in minor slowdown rarely in some platforms

Change-Id: I58cfe9234e17b9b5502f780c7e334574f64fb30d
Reviewed-on: http://review.couchbase.org/40741
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoMB-11980: Fix bugs caused from persisting the current snapshot 07/40707/4
Mike Wiederhold [Wed, 20 Aug 2014 01:35:58 +0000 (18:35 -0700)]
MB-11980: Fix bugs caused from persisting the current snapshot

There are two issues fixed here. One is around the current
snapshot not being updated properly during bucket flush. The
second is an issue caused by persisting the wrong information
for the high seqno, snap start seqno, and snap end seqno during
the snapshot vbuckets task. These values should remain the same
as they were after the last vbucket persistence task otherwise
we might persist incorrect values.

Change-Id: Ie4a357270b52bd35b5f29018f24fd3550913f1cc
Reviewed-on: http://review.couchbase.org/40707
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11934: Fix warmup for windows 73/40673/3
Sriram Ganesan [Fri, 15 Aug 2014 21:55:54 +0000 (14:55 -0700)]
MB-11934: Fix warmup for windows

During warmup in windows, the platform library appends a backslash instead
of a forward slash causing a database file name parsing to break.

Change-Id: I4b3d2296c83dcdf4f94698532a2115dedc841fdb
Reviewed-on: http://review.couchbase.org/40673
Tested-by: Sriram Ganesan <sriram@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11876 Safely cancel AccessScanner task on shutdown 69/40669/4
Sundar Sridharan [Sat, 16 Aug 2014 01:13:32 +0000 (18:13 -0700)]
MB-11876 Safely cancel AccessScanner task on shutdown

Change-Id: I769ca2ad562ef2b27447d40b6a66c4e24ff21eb3
Reviewed-on: http://review.couchbase.org/40669
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoCorrectly return ENGINE_ERROR_CODE in processing mutation messages 53/40653/2
Chiyoung Seo [Fri, 15 Aug 2014 17:25:26 +0000 (10:25 -0700)]
Correctly return ENGINE_ERROR_CODE in processing mutation messages

Change-Id: I72cadd9beb9a024be7fab5d54c7ecf8a85a40c79
Reviewed-on: http://review.couchbase.org/40653
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoFix the regression from the change for MB-11943 45/40645/4
Chiyoung Seo [Fri, 15 Aug 2014 05:05:05 +0000 (22:05 -0700)]
Fix the regression from the change for MB-11943

The change for MB-11943 caused the sporadic failure
in "vbucket deletion doesn't affect new data" unit test case.

This change makes sure that we have the same logic in maintaining
cachedVBState instance for a deleted vbucket.

Change-Id: I7fda66a258c46cd04820001e945e0679eed0469b
Reviewed-on: http://review.couchbase.org/40645
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoWindows don't support the %zu formatter 50/40650/4
Trond Norbye [Fri, 15 Aug 2014 15:42:51 +0000 (17:42 +0200)]
Windows don't support the %zu formatter

Change-Id: Ia36bc151f84af9cfd6c069c3d97c494753d449f3
Reviewed-on: http://review.couchbase.org/40650
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11958: Make sure mutations and snapshot ranges are updated together 42/40642/4
Mike Wiederhold [Fri, 15 Aug 2014 07:33:43 +0000 (00:33 -0700)]
MB-11958: Make sure mutations and snapshot ranges are updated together

There is a race condition where an items seqno is updated and the
flusher runs and persists both the seqno and old snapshot range
before the new snapshot range can be updated.

Change-Id: If124e5ab05c45c68a6956d28cc3898f8c18df6a9
Reviewed-on: http://review.couchbase.org/40642
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11890: Active vb shouldn't initiate add Stream req 41/40641/4
abhinavdangeti [Thu, 14 Aug 2014 23:12:37 +0000 (16:12 -0700)]
MB-11890: Active vb shouldn't initiate add Stream req

Change-Id: Ic902e632d52f426a73d7708f9c69854dc7fab874
Reviewed-on: http://review.couchbase.org/40641
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11916 Synchronize the access to thread job logs. 37/40637/2
Chiyoung Seo [Thu, 14 Aug 2014 21:14:05 +0000 (14:14 -0700)]
MB-11916 Synchronize the access to thread job logs.

Change-Id: I9551cd70b931b1ed92f806be1df0325d922a239c
Reviewed-on: http://review.couchbase.org/40637
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11945 Remove the assertion check on sequence number in the flusher. 30/40630/2
Chiyoung Seo [Thu, 14 Aug 2014 19:41:29 +0000 (12:41 -0700)]
MB-11945 Remove the assertion check on sequence number in the flusher.

We sporadically observe the sequence number mismatches between the
flusher and Couchstore. This is not a regression from any recent
changes, and mostly existed much earlier. We will investigate this
issue separately to see if there are any bugs in ep-engine or

Change-Id: Ifac2eae0deae09cad09dd7b5664f63bcbf828b24
Reviewed-on: http://review.couchbase.org/40630
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11299: Don't let replica vbuckets send partial snapshots 46/40346/6
Mike Wiederhold [Wed, 6 Aug 2014 18:34:59 +0000 (11:34 -0700)]
MB-11299: Don't let replica vbuckets send partial snapshots

Replica vbuckets cannot create their own snapshots and must
rely on the snapshots it received from the active vbucket
when forwarding the snapshots to a downstream consumer. In
order to make sure that this happens properly we will only
stream mutations from the closed checkpoint for non-active
vbuckets. This will guarantee that the above requirement is

Change-Id: I58151ce96868e34083f87dca0ce1d18b6a12aedf
Reviewed-on: http://review.couchbase.org/40346
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11943: replace map with vector for cachedVBStates 00/40600/6
Sundar Sridharan [Thu, 14 Aug 2014 05:39:29 +0000 (22:39 -0700)]
MB-11943: replace map with vector for cachedVBStates

std::map can incur thread-unsafe rb balance operations which
may invalidate concurrent iterators causing asserts or crashes
Pre-allocated vectors are thread-safe as long as the size is not
Change-Id: I777b0cc7d9c421cc11425db75506fdf90044a93c
Reviewed-on: http://review.couchbase.org/40600
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>