MB-17461 Resolve remaining clang analyzer errors in tests

Change-Id: I92079cc561e98996f798ce1de9cd4643b4ce3b0d
Reviewed-on: http://review.couchbase.org/59337
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17461 Remove dead assignment in forest-kvstore

Resolve clang analyzer warning

Change-Id: I93a195c8c12cde2c355caad895e8bc50cd89d530
Reviewed-on: http://review.couchbase.org/59607
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

Correct logic in ActiveStream::setVbucketStateAckReceived

Do not set vbucket state to dead twice.

Change-Id: I508ccace1dc2fcfdeb9035ba539123cbac715db0
Reviewed-on: http://review.couchbase.org/59516
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-16262: Log filtering [2/2]: Reduce view engine DCP clients to WARNING

When a DCP client connects whose name starts with 'mapreduce_view' or
'spatial_view' reduce the minumum log level from NOTICE to WARNING.

This significantly reduces the noise in the memcached logs from DCP
messages triggered by the view engine.

Change-Id: I1d577cd3d7b368b45bc114d17d544b67c5367acf
Reviewed-on: http://review.couchbase.org/59272
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

MB-16262: Log filtering [1/2]: Refactor to add Logger class

There are currently a very large number of log messages relating to
DCP. The vasy majority (9x%) are from the view engine, due to the fact
that the view engine repeatedly creates new DCP connections per
vBucket instead of creating one per vBucket at the start (like
replication).

The net effect of this is that a single design document will result in
approximately 4096 log lines every 10 seconds (4 per vBucket).

This is so verbose that the memcached log quickly wraps - for example
it's not uncommon to see it wrap after just 1 day, and in some
deployments with many views buckets in a matter of an hour or so.

To solve this problem we need to filter out the view engine's 'normal'
output (i.e. the 4096 log lines per 10 sec), however we don't want to
simply discard all DCP log message of a given type, as they may be
useful when from other DCP consumers (XDCR, replicaiton).

The solution is to be able to identify DCP streams which are created
by the view engine, and reduce their verbosity by setting a higher
threshold (WARNING instead of NOTICE).

To implement this we add a new Logger class, which has a per-instance
minimum log level. Each connection will instantiate it's own logger,
and in the case of a view engine connection it can set the
min_log_level higher compared to other connections. This requires that
all log messages we want to filter (i.e. DCP) need to log via the new
`logger` object.

This patch (1/2) adds the new Logger class, and updates all
DCP-related log messages to use their own logger instance. No
additional log filtering occurs, so the log output should be identical
before / after this patch.

Change-Id: I5e7e1da65e36afa6d7e122a47244bcd6496ceb3b
Reviewed-on: http://review.couchbase.org/59271
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

MB-17861: Do not close passive streams during takeover

Due to a bug in 3.x memcached, where during takeover
the vbucket state is set incorrectly to replica instead
of pending and then promoted to active, modify the
set_vbucket_state code to handle the scenario of an
online upgrade correctly.

Change-Id: I0d156cc0ee663966d203d58be20e58a8e9202cd7
Reviewed-on: http://review.couchbase.org/59489
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Address data race identified in TOUCH command

Recent changes introduced this data race: a466ace5
Fix: Let go of the stored value lock (as part of queueDirty)
after converting the storedValue object to an item object.

10:50:53 WARNING: ThreadSanitizer: data race (pid=111994)
10:50:53   Read of size 1 at 0x7d1000004af4 by main thread (mutexes: write M20076, read M15175):
10:50:53     #0 StoredValue::toItem(bool, unsigned short) const /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/stored-value.cc:754 (ep.so+0x00000011c1d7)
10:50:53     #1 EventuallyPersistentStore::getAndUpdateTtl(std::string const&, unsigned short, void const*, long) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep.cc:2464 (ep.so+0x0000000985be)
10:50:53     #2 EventuallyPersistentEngine::touch(void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:4938 (ep.so+0x0000000cd151)
10:50:53     #3 processUnknownCommand(EventuallyPersistentEngine*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:1166 (ep.so+0x0000000d6864)
10:50:53     #4 EvpUnknownCommand(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:1365 (ep.so+0x0000000b70f8)
10:50:53     #5 mock_unknown_command(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/memcached/programs/engine_testapp/engine_testapp.cc:382 (engine_testapp+0x0000004ce149)
10:50:53     #6 touch(engine_interface*, engine_interface_v1*, char const*, unsigned short, unsigned int) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/tests/ep_test_apis.cc:872 (ep_testsuite.so+0x000000072284)
10:50:53     #7 test_vbucket_compact(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/tests/ep_testsuite.cc:1306 (ep_testsuite.so+0x00000004453c)
10:50:53     #8 execute_test(test, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/memcached/programs/engine_testapp/engine_testapp.cc:1131 (engine_testapp+0x0000004cc600)
10:50:53     #9 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
10:50:53
10:50:53   Previous write of size 1 at 0x7d1000004af4 by thread T11 (mutexes: write M15713, write M19262):
10:50:53     #0 PersistenceCallback::callback(std::pair<int, bool>&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/stored-value.h:462 (ep.so+0x0000000adf4a)
10:50:53     #1 CouchKVStore::commitCallback(std::vector<CouchRequest*, std::allocator<CouchRequest*> >&, KVStatsCtx&, couchstore_error_t) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/couch-kvstore/couch-kvstore.cc:1947 (ep.so+0x000000179431)
10:50:53     #2 _ZN12CouchKVStore17commit2couchstoreEP8CallbackIJ10KVStatsCtxEE /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/couch-kvstore/couch-kvstore.cc:1761 (ep.so+0x000000170e73)
10:50:53     #3 _ZN12CouchKVStore6commitEP8CallbackIJ10KVStatsCtxEE /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/couch-kvstore/couch-kvstore.cc:1007 (ep.so+0x000000170aa1)
10:50:53     #4 EventuallyPersistentStore::commit(unsigned short) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep.cc:3483 (ep.so+0x00000009e916)
10:50:53     #5 EventuallyPersistentStore::flushVBucket(unsigned short) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep.cc:3431 (ep.so+0x00000009d874)
10:50:53     #6 Flusher::flushVB() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/flusher.cc:293 (ep.so+0x0000000feece)
10:50:53     #7 Flusher::step(GlobalTask*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/flusher.cc:183 (ep.so+0x0000000fd595)
10:50:53     #8 FlusherTask::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/tasks.cc:65 (ep.so+0x000000138ae2)
10:50:53     #9 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000f8af6)
10:50:53     #10 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f86a5)
10:50:53     #11 platform_thread_wrap(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x00000000551b)

Change-Id: I3eaa43f4d5238d04500bb779b0a2ea785c68ed0d
Reviewed-on: http://review.couchbase.org/59408
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Handle case:0 in perfsuite for dcp latency tests

Change-Id: Ib1c00873b9b84bd1ce456d2d1c891bb2626607a5
Reviewed-on: http://review.couchbase.org/59397
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

MB-16262: Remove some DCP log messages which are of minimal value.

Remove two DCP log messages: the 'stream created!' message, and the
explicit 'Backfill task finsihed' message if the completion was normal
(clean).

Before:

  NOTICE (default) DCP (Producer) <stream name> - (vb 842) Creating stream with start seqno 173 and end seqno 180
  NOTICE (default) DCP (Producer) <stream name> - (vb 842) stream created!
  NOTICE (default) DCP (Producer) <stream name> - (vb 842) Sending disk snapshot with start seqno 173 and end seqno 181
  NOTICE (default) DCP (Producer) <stream name> - (vb 842) Backfill complete, 0 items read from disk, 4 from memory, last seqno read: 181
  NOTICE (default) DCP (Producer) <stream name> - (vb 842) Backfill task (174 to 180) finished
  NOTICE (default) DCP (Producer) <stream name> - (vb 842) Stream closing, 4 items sent from backfill phase, 0 items sent from memory phase, 181 was last seqno sent, reason: The stream ended due to all items being streamed

After:

  NOTICE (default) DCP (Producer) <stream name> - (vb 842) Creating stream with start seqno 0 and end seqno 195
  NOTICE (default) DCP (Producer) <stream name> - (vb 842) Sending disk snapshot with start seqno 0 and end seqno 195
  NOTICE (default) DCP (Producer) <stream name> - (vb 842) Backfill complete, 0 items read from disk, 20 from memory, last seqno read: 195
  NOTICE (default) DCP (Producer) <stream name> - (vb 842) Stream closing, 20 items sent from backfill phase, 0 items sent from memory phase, 195 was last seqno sent, reason: The stream ended due to all items being streamed

Change-Id: Ib4ce6a02d8dab5c1f2af40caeef371fbc5a60128
Reviewed-on: http://review.couchbase.org/59270
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

MB-17461 Resolve clang warning about testing a garbage value

Because clang doesn't understand that `get` won't return
EEXISTS in this context the clang analyzer raises warnings about
using the uninitialised `locked`.

Change-Id: I29d0c1246d23a6dd37ee614608766e2fe20389b4
Reviewed-on: http://review.couchbase.org/59338
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17801: Close Passive streams into a vbucket only on a failover

Close passive streams into a vbucket only when a vbucket
transitions from replica state to active.

The bug that is being resolved here has to do with closing
passive streams into a vbucket during takeover when a vbucket
changes state from pending to active.

Change-Id: Ie02bc2363e1ccf7a0c51ba11c9475a967803aad0
Reviewed-on: http://review.couchbase.org/59347
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>

MB-16262: Reduce log level to NOTICE for non-warning DCP messages

Adjust the log level for DCP-related messages to NOTICE for message
which are not actually WARNINGS.

Change-Id: I63ebb484536fc297acf208b96f7af78045b4be4f
Reviewed-on: http://review.couchbase.org/59269
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Revert "MB-17534: Initialize drift counter to zero"

This reverts commit cc59592847a9466d7b33f77d02376f97ad838147.
Reverting this commit as the XDCR initial request was to disable
applying drift to HLC CAS and on the replica

Change-Id: I5d9bb5100715bf9d4e5c3f4f33fbf25cb6b8bdfe
Reviewed-on: http://review.couchbase.org/59293
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17461 Remove dead assignments in DCP

Resolves static analyser warnings
Change-Id: I20e83f46c1aa5d4d021873523488a5ecfedcecb9
Reviewed-on: http://review.couchbase.org/59207
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

MB-17461 Remove dead increments caught by static analyzer

Change-Id: Icfc64a544d214e70f63ab2fa8f940499c5fd70fb
Reviewed-on: http://review.couchbase.org/59206
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17461 Resolve static analyzer warnings for couch-kvstore

Moves size argument to second parameter for calloc and
specifies 1 for count. This semantically makes more sense
(allocate space for 1 'thing' rather than a number of bytes
for the size of one thing) and silences the clang warning.

Change-Id: I7bda0a0a21b1f6ae2e991529d078932407e62d35
Reviewed-on: http://review.couchbase.org/59204
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17461 Fix defragmenter static analysis warnings

* Fix potential memory leak in defragmenter test
* Guard defragementer visitor's progress tracker
from dividing by zero

Change-Id: Ic4ce236435c156df7166b33e0245c6971e75710c
Reviewed-on: http://review.couchbase.org/59203
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

MB-15946: Set correct threshold for adding new replica checkpoint on snap end

We want to create a new replica checkpoint (hence close and release the
memory consumed by the current replica checkpoint) before we reach a
condition where we cannot set new items from the front end load on the node.
Currently the threshold for front end load is 93%. This change sets the
threshold to add new replica checkpoint upon receiving snapshot end to
high watermark (85%).

Note, this however does not completely rule out the deadlock scenario
on buckets with very small mem quota and large item size. This change
is simple and reduces the possibility of deadlock.

To completely remove the deadlock possibility we should close all (or most
until memory usage goes below some limit) replica checkpoints which have
received last item in the snaphot. This is likely to be addressed in
another change.

Change-Id: I80a914186fbaae71f45b20f68547261715a09dc8
Reviewed-on: http://review.couchbase.org/59222
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

MB-17766: Close passive streams on vbucket state change

Upon vbucket state change, close passive streams into
the vbucket as well along with the active streams from
it. This is to avoid a bad scenario (which could
result in an incorrect high seqno or a crash) when
the mutations coming in to the consumer clash with
the front end operations.

Change-Id: I6e701900a1f2c9a65e64dfd903e01a3d3b4080ce
Reviewed-on: http://review.couchbase.org/59290
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17461 Prevent null dereference in failover table parsing

Resolves clang static analysis warnings

Change-Id: I20a03d6af4cf3a4d2768d8bbe6c162136dfe9bd3
Reviewed-on: http://review.couchbase.org/59208
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

aggregateQueueStats: Pass aggregator by reference

aggregateQueueStats() takes a ConnCounter argument to aggregate
results in. Given there must be a valid (non-NULL) object to use this
method, change from passing by pointer to by reference to simplify
code by removing the need for explicit nullptr checks.

Change-Id: I2a0b10864b97c96341139e32da41f0eacb7ea47b
Reviewed-on: http://review.couchbase.org/59268
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17461 Adjust test_tap_rcvr_mutate to not malloc(0)

Resolves clang static analyzer warning

Change-Id: I9b492c6122a250516ea35831c00588d3b4ae3d2e
Reviewed-on: http://review.couchbase.org/59209
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Address datarace with curChkSeqno in DcpStream::ActiveStream

10:39:45 WARNING: ThreadSanitizer: data race (pid=17881)
10:39:45   Write of size 8 at 0x7d480000b970 by thread T15:
10:39:45     #0 ActiveStream::nextCheckpointItemTask() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/stream.cc:716 (ep.so+0x0000000741d9)
10:39:45     #1 ActiveStreamCheckpointProcessorTask::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/stream.cc:652 (ep.so+0x000000073dd8)
10:39:45     #2 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000f69b6)
10:39:45     #3 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f6565)
10:39:45     #4 platform_thread_wrap(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x00000000551b)
10:39:45
10:39:45   Previous read of size 8 at 0x7d480000b970 by main thread (mutexes: write M13331, write M901842698644811512, write M37467):
10:39:45     #0 ActiveStream::addTakeoverStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/stream.cc:588 (ep.so+0x0000000739ce)
10:39:45     #1 DcpProducer::addTakeoverStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*, unsigned short) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/producer.cc:717 (ep.so+0x000000066ee0)
10:39:45     #2 EventuallyPersistentEngine::doDcpVbTakeoverStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), std::string&, unsigned short) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:5745 (ep.so+0x0000000c8bfb)
10:39:45     #3 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:4705 (ep.so+0x0000000c6d6c)
10:39:45     #4 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:221 (ep.so+0x0000000b4d1e)
10:39:45     #5 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/memcached/programs/engine_testapp/engine_testapp.cc:241 (engine_testapp+0x0000004cdfcd)
10:39:45     #6 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/tests/ep_test_apis.cc:999 (ep_testsuite_dcp.so+0x0000000366c2)
10:39:45     #7 wait_for_stat_to_be_lte(engine_interface*, engine_interface_v1*, char const*, int, char const*, long) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/tests/ep_test_apis.cc:1162 (ep_testsuite_dcp.so+0x00000003c961)
10:39:45     #8 dcp_stream(engine_interface*, engine_interface_v1*, char const*, void const*, unsigned short, unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, int, int, int, int, bool, bool, unsigned char, bool, unsigned long*, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/tests/ep_testsuite_dcp.cc:123 (ep_testsuite_dcp.so+0x00000002880c)
10:39:45     #9 test_dcp_agg_stats(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/tests/ep_testsuite_dcp.cc:1340 (ep_testsuite_dcp.so+0x000000011247)
10:39:45     #10 execute_test(test, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/memcached/programs/engine_testapp/engine_testapp.cc:1131 (engine_testapp+0x0000004cc600)
10:39:45     #11 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)

Change-Id: I0023685c5ebc2c89977b8bc835b252591120586e
Reviewed-on: http://review.couchbase.org/59282
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

MB-17461 Guard against calling null object pointer in connmap

Resolves static analyser warning.

Change-Id: I6de973cb8a07be08d972bb7b27a5269057be4211
Reviewed-on: http://review.couchbase.org/59205
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

MB-17273: Update bytes processed correctly in passive str when buffer is cleared

Fixing the bug introduced in http://review.couchbase.org/#/c/58774

Change-Id: Id10cf27056f2b11deec2b0e0df3cd984c0e14915
Reviewed-on: http://review.couchbase.org/59286
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

VBucketMap: remove numShards member variable

This variable duplicates information already held in the shards vector.

Change-Id: I6f7ab007614f8e2631c2b52ef22cdc7fe7cd3429
Reviewed-on: http://review.couchbase.org/59267
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17666: Generate CAS using HLC for deletes

We currently only increment the CAS value for a deleted or expired item.
This doesn't work for last write wins conflict resolution as the CAS
needs to be generated using HLC.

Change-Id: I2349470d65b0c27267b8a128a04070faa0f1d96c
Reviewed-on: http://review.couchbase.org/59124
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Clear any stale DCP data before running dcp_stream

This fix is intended to address a sporadic failure seen
in tests:
- test dcp replica stream backfill
- test dcp replica stream in-memory
- test dcp replica stream all

..where a the ADD_STREAM_RSP (0x51) message is the last op
received by the replica vbucket, and when a producer is set
up at it, if no message has been received from it yet, the
test fails on this error:
    "Test failed: `Unknown DCP operation: Q' (false)"

Change-Id: Id06c91ab70a2d0ba161043856d3f77d3b011856f
Reviewed-on: http://review.couchbase.org/59236
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>

Address data race with replicationThrottle parameters

12:23:35 WARNING: ThreadSanitizer: data race (pid=31345)
12:23:35   Read of size 8 at 0x7d08000066c0 by thread T12:
12:23:35     #0 ReplicationThrottle::adjustWriteQueueCap(unsigned long) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/replicationthrottle.cc:49 (ep.so+0x000000114c7f)
12:23:35     #1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:3100 (ep.so+0x0000000bb424)
12:23:35     #2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:4597 (ep.so+0x0000000c54d5)
12:23:35     #3 EventuallyPersistentStore::snapshotStats() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep.cc:1744 (ep.so+0x000000090f2e)
12:23:35     #4 StatSnap::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/tasks.cc:100 (ep.so+0x000000136dd6)
12:23:35     #5 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000f6966)
12:23:35     #6 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f6515)
12:23:35     #7 platform_thread_wrap(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x00000000551b)
12:23:35
12:23:35   Previous write of size 8 at 0x7d08000066c0 by main thread (mutexes: write M2287262534014660504):
12:23:35     #0 EPStoreValueChangeListener::sizeValueChanged(std::string const&, unsigned long) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/replicationthrottle.h:42 (ep.so+0x0000000b35ee)
12:23:35     #1 Configuration::setParameter(std::string const&, long) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/configuration.cc:225 (ep.so+0x000000197ac5)
12:23:35     #2 Configuration::setReplicationThrottleQueueCap(long const&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/build/ep-engine/src/generated_configuration.cc:459 (ep.so+0x0000001a6fa8)
12:23:35     #3 setTapParam(EventuallyPersistentEngine*, char const*, char const*, std::string&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:323 (ep.so+0x0000000d7f3f)
12:23:35     #4 EvpUnknownCommand(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:1365 (ep.so+0x0000000b4f68)
12:23:35     #5 mock_unknown_command(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/memcached/programs/engine_testapp/engine_testapp.cc:382 (engine_testapp+0x0000004ce149)
12:23:35     #6 set_param(engine_interface*, engine_interface_v1*, protocol_binary_engine_param_t, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/tests/ep_test_apis.cc:597 (ep_testsuite_dcp.so+0x000000038e17)
12:23:35     #7 test_consumer_backoff_stat(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/tests/ep_testsuite_dcp.cc:2184 (ep_testsuite_dcp.so+0x000000017411)
12:23:35     #8 execute_test(test, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/memcached/programs/engine_testapp/engine_testapp.cc:1131 (engine_testapp+0x0000004cc600)
12:23:35     #9 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)

Change-Id: Ie4ff039603f2ddfc5b44d5d7f217544307655d31
Reviewed-on: http://review.couchbase.org/59215
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

Merge "Merge remote-tracking branch 'couchbase/sherlock' into 'couchbase/master'"

MB-14140: Fixup Debian7 build failure in baef8d3

Use std::map::insert() instead of replace().

Change-Id: I69b99fca8e12d6afc3dabc411a1ccd14cc8ef3a6
Reviewed-on: http://review.couchbase.org/59181
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Dave Rigby <daver@couchbase.com>

Merge remote-tracking branch 'couchbase/sherlock' into 'couchbase/master'

couchbase/sherlock:
|\
| *   42c027e Merge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock'
| |\
| | * ae39d7f MB-17502: DCP performance regression fixed.
| * | 7f4d0dd MB-17517: check for invalid CAS value in [set/delete]WithMeta
| * | 5372cfd MB-17517: return EINVAL instead of assert in arithmetic

Change-Id: If60d355fa33641bf206d9d891f418f917cb462b8

MB-14140: Add stats for total Couchstore IO bytes read and written

The existing IO stats (io_{read,write}_bytes only measure the amount
of 'user' data (document, key, value and metadata) written. While
useful for some use-cases, these fail to account for the actual amount
of bytes read / written to disk, which can be very useful to measure
and compare to the output of iostat etc.

Add two new stats to CouchstoreStats which measure the 'total' number
of bytes we request the OS to read & write to/from disks:

* io_total_read_bytes - Number of bytes read from disk (total,
  including Couchstore B-Tree and other overheads).

* io_total_write_bytes - Number of bytes written to disk (total,
  including Couchstore B-Tree and other overheads).

These are exposed via two interfaces:

1. The `cbstats kvstore` interface, with per-shard details:

    rw_0:backend_type:         couchstore
    rw_0:close:                2974
    rw_0:failure_del:          0
    rw_0:failure_get:          0
    rw_0:failure_open:         0
    rw_0:failure_set:          0
    rw_0:failure_vbset:        0
    rw_0:io_num_read:          0
    rw_0:io_num_write:         4998
    rw_0:io_read_bytes:        0
    rw_0:io_total_read_bytes:  16935210    ***
    rw_0:io_total_write_bytes: 2607998     ***
    rw_0:io_write_bytes:       184378
    rw_0:lastCommDocs:         2
    rw_0:numLoadedVb:          0
    rw_0:open:                 2974

2. The `cbstats all` interface; with accumulated (sum across all
shard) values:

    ep_io_total_read_bytes:   113216983
    ep_io_total_write_bytes:    4666002

Change-Id: I0f51bbdb7965731b27814532391b5372bd91cede
Reviewed-on: http://review.couchbase.org/59132
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>

MB-14140: Port kvstore_test to GTest framework

To facilitate testing of new KVStore stats functionality, port the kvstore_test unit tests to GTest

Change-Id: Ib3eba03d0de424f9187688515d27cf49622fac47
Reviewed-on: http://review.couchbase.org/59131
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>

Merge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock'

couchbase/3.0.x:
|\
| * ae39d7f MB-17502: DCP performance regression fixed.

Change-Id: Id3262008479ffe7e350a9a4d88438ed541ac242a

MB-17502: DCP performance regression fixed.

Many patches were added to speed up DCP, however some of that
performance was lost when doing some code tidying without
re-profiling.

With all the DCP performance patches (particuarly) 87869fd39 straight
DCP performance is a touch slower. This is because DCP used to take
one lock and then do work. The new code has more locks, but holds
them for fewer lines of code. This means that DCP is friendlier/fairer
to the other threads interacting with a DCP producer.
The front-end operation threads are no longer stalled for long periods
whilst DCP holds the one lock.

Frontend latency before locking changes:

 === Latency [With background DCP] - 100000 items
                                 Percentile
                   Median     95th     99th  Std Dev
 Add               16.337   34.894   45.241   25.627
 Get                1.226    1.524    1.745    0.435
 Replace           16.311   34.386   42.097    8.435
 Delete            15.636   32.915   41.999    7.408

Frontend latency after locking changes:

 === Latency [With background DCP] - 100000 items
                                 Percentile
                   Median     95th     99th  Std Dev
 Add                3.996   12.159   20.724   11.376
 Get                1.299    1.629    1.730    0.634
 Replace            4.274   12.831   22.988    4.523
 Delete             3.142   10.302   14.292    3.350

The average and 95th/99th are all improved.

Fix details:

The roundRobin/vbReady code has a bufferLog.pauseIfFull call on the
"hot" part of the loop, this is the main cause of the regression.

With that fixed CPU profiling and benchmarking shows that DCP is back
to 3.1.3 levels but highlighted that:

1. DcpProducer::getNextItem was hot (5% of a DcpProducer thread).
2. DcpConsumer::processBufferedItems was hitting SpinLock hard.
   20 to 30% at times was consumed by SpinLock code.
3. snapshot creation was frequently yielding even though it had work todo.

So to address 1. the fix is actually to remove the roundRobin/vbReady
code. It is actually no better and in some cases a little slower than
the orginal. This code is replaces with std:: structures *but* the
Mutex used has a much smaller scope.

Note the DcpProducerReadyQueue has been profiled and proven that having
the std::map powering find() is much faster than searching the list.
This is important because the find method is part of the front-end
operation thread.

To address 2. it was observed that the consumer code is constructing
a passive_stream_t frequently, then testing if there is a pointer.
The construction uses the SpinLock code and can be avoided just by
testing the streams[vb] directly and only then do we construct
a copy of the passive_stream_t. This avoids the SpinLock code on
every iteration of the for loop in the affected function.

To address 3. ensure that the snapshot tasks work queue doesn't have
duplicates, there's no need. Then raise the number of snapshots before
yield. Various rebalances showed that around 250 was enough, so let's go
with 256.

Change-Id: I8fb0bd30f8e07d000192675de425726ad26e403a
Reviewed-on: http://review.couchbase.org/58886
Well-Formed: buildbot <build@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

Set more appropriate timeouts for basic & dcp tests

As seen on valgrind enabled tests, test time for basic tests
are as follows:
ep-engine_full_eviction_basic .......... Passed 425.09 sec
ep-engine_basic_tests .................. Passed 433.73 sec

- Increasing the timeout for these tests from 400 to 600, so
that they may complete when run with valgrind.
- Also reducing the timeout for dcp tests from 1800 to 1200.

Change-Id: Ie2aa79a56e56bbe84a8dca5f1819921509331b58
Reviewed-on: http://review.couchbase.org/59116
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>

MB-17273: Ensure that passive stream buffer is not processed after it is DEAD.

We should not process the DCP passive stream buffer or add new items into it
once the stream state is set to dead. Hence, as per current locking design
in the passive stream, upon setDead call we must hold the buffer lock
until we clear the stream buffer and set the stream to dead state.

Change-Id: Iff4497424191b65963d80eff41d5f38b5a174890
Reviewed-on: http://review.couchbase.org/58774
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17082: Subscribing to the ON_DELETE_BUCKET notification

As part of DELETE_BUCKET, when memcached explicitly
disconnects all the connections, drop the connections
and let go of the reference immediately rather than
holding on to them for some time as in the case of
TAP producers.

Pre-requisite: http://review.couchbase.org/#/c/59045/

Change-Id: I83ccea56f11a44d2c3cb96b4e09b538dd1dc6dfb
Reviewed-on: http://review.couchbase.org/59048
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

MB-17616: Update CAS and rev seqno in the case of a TTL update

A TTL update is effectively a mutation on the item. Thus the item's CAS
value needs to be updated so that this mutation will win in the case of
LWW. Also increment the revision sequence number to ensure that the item
with updated TTL will have higher rev sequence number in a conflict
resolution scenario.

Change-Id: Ia137e99f588bd632b592f80a342bf0d929bf550d
Reviewed-on: http://review.couchbase.org/59067
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Update CMakeLists to run tests in full eviction mode

Also feed dbname information along with the eviction mode
to support the scenario where tests are run in the value-
only eviction mode and the full eviction mode in parallel.

Change-Id: I88a4e4c5275d5b1c89f1fb4e5ab78c4d8774a3d5
Reviewed-on: http://review.couchbase.org/58776
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-16976 Log compaction scheduled as NOTICE

Compaction logging changed to notice so that the duration of
compaction can be inferred from start and finish timestamps and
so that you can tell if compaction is currently running or
stuck.

Change-Id: I90f36a832c262d4b4d5f5306f5fe26fd81a2df75
Reviewed-on: http://review.couchbase.org/59036
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

MB-13521 Improve help for alog_task_time in cbepctl

This makes the help message for alog_task_time less ambiguous by
emphasising that it should be a single number and between 0-23

Change-Id: Icc7d4d29d6d88451882c6e5eb04fec926122cd3d
Reviewed-on: http://review.couchbase.org/59009
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17508 4/4 Consolidate and remove unecessary validation from SET_PARAM

* Adds a string to boolean function `cb_stob`  which will throw an
invalid_argument inherited exception if it cannot be validly converted
to boolean. This is used for any string to boolean conversion needed for
setParam and exposes a more detailed error message on the wire.
* Removes any unnecessary validation that will subsequently be performed
by the configuration validators from the set_param level and moves
validation that can be trivially be moved to the configuration validators.

Change-Id: I1bc39aa8028d4327791e396dea6ef1278e63792d
Reviewed-on: http://review.couchbase.org/58939
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17508 3/4 Add support for float and boundless ranges for configuration

* Creates a float range validator based on whether the type of the config option
is a float rather than if the validation range is not a rounded number.
* Allows for leaving one of the validation range boundaries unspecified which will
result in the validation in that direction being set to maximum.

Change-Id: I023175cdd9b5ef0b6170c8124fd433eaa0788190
Reviewed-on: http://review.couchbase.org/58938
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17508 2/4 Expose detailed range errors to SET_PARAM requests

Exposes detailed validation error messages from the configuration validators
further down the stack out on the body of the response instead of a generic
'out of range' style message.

The primary purpose of this is to expose error reasons when using epctl.
Sample output:

> $ ./cbepctl localhost:12001 set flush_param exp_pager_initial_run_time 25
> setting param: exp_pager_initial_run_time 25
> Error: Validation Error, exp_pager_initial_run_time takes values between -1 and 23 (Got: 25)

Change-Id: Ib19c0056d1f289b43c84185a33efa27d20d8ce24
Reviewed-on: http://review.couchbase.org/58937
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17508 1/4 Add detailed range errors to configuration validation exceptions

This commit improves the exceptions thrown by the configuration's validators to
include a message which more precisely describes why the validation failed.

The purpose of this is to in a subsequent commit expose this message to an end
user modifying the configuration via a SET_PARAM operation.

Change-Id: I96bc339161f8788d30efe9e359c0e88825ffcd0a
Reviewed-on: http://review.couchbase.org/58936
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17517: check for invalid CAS value in [set/delete]WithMeta

In a [set/delete]WithMeta call from either XDCR or from DCP,
a corrupt CAS value for the incoming item should return an
error

Change-Id: Id87627ae35ef711de4704a960b3aa7d1e9b9a48b
Reviewed-on: http://review.couchbase.org/58910
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Removing previously unused parameters from CMakeLists

Change-Id: I6d2d19222cb4f03599172479c33fa93c49f8ed0a
Reviewed-on: http://review.couchbase.org/58947
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Move DCP tests to separate test suite

Change-Id: I34476196748e42b382bcdea627209a01aff146d8
Reviewed-on: http://review.couchbase.org/58742
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

MB-17534: Initialize drift counter to zero

The drift counter was previously set to the largest
negative value allowed for a 48-bit integer. This
was done so as to determine whether the drift counter
was ever set by XDCR or not.
    The drift counter will now be set to 0 so that
when time_sync is enabled, drift counter can be used
as is to determine the adjusted time.

Change-Id: I5623a793b34515735036472add819b5e722c9fc0
Reviewed-on: http://review.couchbase.org/58903
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17517: return EINVAL instead of assert in arithmetic

If a get performed on an item returns a CAS value of zero, then
return EINVAL as opposed to asserting

Change-Id: If3d43c270bcc627029d0954dab0e570c83ddca74
Reviewed-on: http://review.couchbase.org/58868
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17497: PersistenceCallback to delete an item only when required

When the persistence callback for a deleted item is
invoked by the flusher, the item in the hash table
that was soft deleted previously needs to be deleted
iff:
    1. Item that is existent in hash table has deleted flag set to true
    2. Item in hash table has seqno equal to the queued item in the
       persistence callback

Change-Id: Ifa6555e66384051db0483fba46da71c75fb03fbc
Reviewed-on: http://review.couchbase.org/58820
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

MB-17301: Returning vb_uuid, last_seqno for setDriftCounterState

Return vb_uuid and last_seqno for a vbucket as part of response
of the setDriftCounterState API atomically.

Change-Id: I21f08c69e7e4119af35d34f15b1ca87c8ae485dc
Reviewed-on: http://review.couchbase.org/58757
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>

Handle test case for unlockKey in full eviction mode

Context: test_unl

Change-Id: Ieee2f61d1fb011e477c636aba4f4d055a22576a5
Reviewed-on: http://review.couchbase.org/58759
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Merge "Merge remote-tracking branch 'couchbase/sherlock' into 'couchbase/master'"

Merge remote-tracking branch 'couchbase/sherlock' into 'couchbase/master'

couchbase/sherlock:
|\
| *   59a37de Merge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock'
| |\
| | * 783d168 MB-17086: Fix to performance regression.
| | * 9569420 MB-16632: Use a background task to handle snapshot creation
| * | 9a7951d MB-17289: Update existing temp item CAS to incoming CAS in deleteWithMeta
| * | cc2ace7 MB-17231: conditionally delete temp items in get API

Change-Id: I9bf5dbe017c0e40d66369b9515589b68780de201

Tighten up missing key check on unlock test

Change-Id: I0bf71c1d6476266e9c2505f8801f3bf3e8f66914
Reviewed-on: http://review.couchbase.org/58745
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

[ActiveStream] Address potential lock-inversion scenarios

Acquire vbucket state lock only when really necessary
in the ActiveStream context. Also avoid acquiring one
lock within the other wherever possible in the ActiveStream
context again.

This change is to avert potential deadlocks due to
lock inversion that will be induced by upcoming changes,
here are the scenarios:
(i)     Locking between streamsMutex, streamMutex and
        vb_stateLock in the set operation - handle
        response scenario.
        (http://factory.couchbase.com/job/ep-engine-threadsanitizer-master/1225/console)
(ii)    In case of a set operation, vb_stateLock is
        acquired and then streamMutex is acquired for
        notification. During markDiskSnapshot, the
        streamMutex is acquired before the vb_stateLock
        lock is acquired.
        (http://factory.couchbase.com/job/ep-engine-threadsanitizer-master/1268/console)

Change-Id: Iddb518b5e94a4595c62026f797ff17ba790d180c
Reviewed-on: http://review.couchbase.org/58557
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

MB-17225 Fix unlockKey to return a correct status for not my vbucket

Change-Id: I9f4e73ffecb597e82727191429e7d7389f57eb80
Reviewed-on: http://review.couchbase.org/58739
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>

Account for slowness of test case : expiration_on_warmup

Increase expiration time for the stored item.

Change-Id: I7ff1c9cf79e98125686e2a09bba3a3788f728854
Reviewed-on: http://review.couchbase.org/58654
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Address heap use after free issue in Dcp BackfillManager

21:23:11 WARNING: ThreadSanitizer: heap-use-after-free (pid=8561)
21:23:11   Read of size 8 at 0x7d240000d6a8 by thread T15:
21:23:11     #0 BackfillManager::backfill() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/backfill-manager.cc:250 (ep.so+0x00000004f35a)
21:23:11     #1 BackfillManagerTask::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/backfill-manager.cc:43 (ep.so+0x00000004ee6f)
21:23:11     #2 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000f1736)
21:23:11     #3 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f12e5)
21:23:11     #4 platform_thread_wrap(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x00000000551b)
21:23:11
21:23:11   Previous write of size 8 at 0x7d240000d6a8 by thread T15:
21:23:11     #0 operator delete(void*) <null> (engine_testapp+0x00000046485b)
21:23:11     #1 DcpProducer::~DcpProducer() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/producer.cc:167 (ep.so+0x00000006377b)
21:23:11     #2 DcpProducer::~DcpProducer() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/producer.cc:165 (ep.so+0x000000063a45)
21:23:11     #3 ActiveStream::~ActiveStream() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/atomic.h:272 (ep.so+0x00000006ed6d)
21:23:11     #4 ActiveStream::~ActiveStream() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/stream.cc:200 (ep.so+0x00000006f8b5)
21:23:11     #5 BackfillManager::backfill() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/atomic.h:272 (ep.so+0x00000004f345)
21:23:11     #6 BackfillManagerTask::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/dcp/backfill-manager.cc:43 (ep.so+0x00000004ee6f)
21:23:11     #7 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000f1736)
21:23:11     #8 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f12e5)
21:23:11     #9 platform_thread_wrap(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x00000000551b)

Change-Id: I3c63215791c23de49b5304654115fd4c558a3328
Reviewed-on: http://review.couchbase.org/58619
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

ep_testsuite: Move remaining basic tests to ep_testsuite_basic

Following on from the previous patch, move the remaining 'basic' tests
from the generic ep_testsuite to the new ep_testsuite_basic.

There is now ~24s worth of tests originally from ep_testsuite moved
into ep_testsuite_basic (as measured on OS X locally); reducing the
wallclock test runtime by that same amount (as they can now be run
concurrently).

Change-Id: I382b099ffe659044cfa67c4035048a04ab8dfb1c
Reviewed-on: http://review.couchbase.org/58584
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17231: Consolidate options for get API calls

- Provide enumerated options that can be used to pass to get calls
- Don't delete temporary items for arithmetic operations

Change-Id: I807550938e28ae8132d6cc4d49797a6395628759
Reviewed-on: http://review.couchbase.org/58423
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

ep_testsuite: Split basic tests into own suite

Create a new 'basic' testsuite which we will move basic get/set etc
operations into from the large ep_testsuite.

This patch only includes the infrastrucutre changes and a couple of
tests. More will be moved over in subsequent patches.

Change-Id: I342787b2e9e2591578ea1004e62b6c11ddb59645
Reviewed-on: http://review.couchbase.org/57184
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Daniel Owen <owend@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Address data race while accessing list:all in connmap

The stl list 'all' is to be guarded by connsLock, not
releaseLock.

17:01:48 WARNING: ThreadSanitizer: data race (pid=231360)
17:01:48   Write of size 8 at 0x7d0800090580 by thread T17 (mutexes: write M13999):
17:01:48     #0 operator delete(void*) <null> (engine_testapp+0x00000046485b)
17:01:48     #1 TapConnMap::getExpiredConnections_UNLOCKED(std::list<RCPtr<ConnHandler>, std::allocator<RCPtr<ConnHandler> > >&) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/ext/new_allocator.h:110 (ep.so+0x00000004095b)
17:01:48     #2 TapConnMap::manageConnections() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/connmap.cc:487 (ep.so+0x00000003fd02)
17:01:48     #3 ConnManager::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/connmap.cc:149 (ep.so+0x00000004e6fe)
17:01:48     #4 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000f1136)
17:01:48     #5 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000f0ce5)
17:01:48     #6 platform_thread_wrap(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x00000000551b)
17:01:48
17:01:48   Previous read of size 8 at 0x7d0800090580 by main thread (mutexes: write M14061):
17:01:48     #0 std::_List_iterator<RCPtr<ConnHandler> >::operator++() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/stl_list.h:159 (ep.so+0x0000000437f8)
17:01:48     #1 EvpDestroy(engine_interface*, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/ep-engine/src/ep_engine.cc:2199 (ep.so+0x0000000b22bc)
17:01:48     #2 mock_destroy(engine_interface*, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/memcached/programs/engine_testapp/engine_testapp.cc:100 (engine_testapp+0x0000004cd787)
17:01:48     #3 execute_test(test, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-master/memcached/programs/engine_testapp/engine_testapp.cc:997 (engine_testapp+0x0000004cc7e0)
17:01:48     #4 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)

Change-Id: Ic5cd20320936a6355f924127c225754f7991af02
Reviewed-on: http://review.couchbase.org/58611
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

Correct unit for time while waiting for stat

useconds_t => us (microseconds)

Change-Id: Ifa493449121e69b9f945f0c5b4a0580a0b682c62
Reviewed-on: http://review.couchbase.org/58596
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

[Unit Test] Skip running test_get_delete_missing_file for forestdb

This particular test tries to check if a get after deleting an
existing file results in TMP_FAIL. But this use case doesn't
apply for forestdb as backend as the forestdb file handle isn't
opened and closed on every single operation

Change-Id: I2be211d97ba477b9a2dc717b50d6c23da445b924
Reviewed-on: http://review.couchbase.org/58561
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

Unify definitions of dcp smart pointer types.

Move the definitions of the smart pointers to their own
header file to prevent cyclic include problems.

For example stream.h includes producer.h and vice versa, yet
the files need the other files types, which can cause a compilation
failure.

Change-Id: I5f96f8c70c955ce2acf3dd572507982509592e16
Reviewed-on: http://review.couchbase.org/58579
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

[DCP Consumer]: Grab streamMutex when transitioning to 'dead' on DCP_STREAM_END

Passive stream should grab streamMutex lock before transitioning to dead
state upon processing the DCP_STREAM_END packet. Lock needs to be acquired
before any state transitions, as transitionState() function can be called
from multiple threads and all those can change the 'state_' member in the
stream object

Change-Id: If17b41555c26555a6fb3fd983768d758221a239d
Reviewed-on: http://review.couchbase.org/58411
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17301: setDriftCounterState's response to carry vb_uuid

The response for the setDriftCounterState API to carry
vb_uuid along with last_seqno.

        .-------------.----------------.
Format: | vb_uuid(8B) | last_seqno(8B) |
        .-------------.----------------.

Change-Id: I8e09c014ee2e0c62be2c46e92fcac83826060664
Reviewed-on: http://review.couchbase.org/58420
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

Merge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock'

* couchbase/3.0.x:
  MB-17086: Fix to performance regression.
  MB-16632: Use a background task to handle snapshot creation

Change-Id: Ib018040e29c699316b5519a1325fff1476b803eb

[Unit test] Get dbname from configuration

TestCase: test_get_delete_missing_file

If user provided a dbname, use it in the test,
rather than some default value.

Change-Id: If42cab0027cc0aeafd0c903635f68fe035bed832
Reviewed-on: http://review.couchbase.org/58506
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Estimate CacheSize of temp items correctly

Mark temp items as non-resident before calculating
the cache size to account for their size correctly.

Change-Id: I655e72ad177fce5896119a2d43f90c62d77b3a91
Reviewed-on: http://review.couchbase.org/58325
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17289: Update existing temp item CAS to incoming CAS in deleteWithMeta

If the incoming CAS is non-zero, a comparison is made with an
existing item CAS to check if the delete can be done or not.
But in case a temp existing item has to be persisted, then
update the existing temp item CAS to the incoming CAS in order
to bypass the above-mentioned comparison only for forced
deletions

Change-Id: Idca2615a7e3166e98b26c9d447b1161742226436
Reviewed-on: http://review.couchbase.org/58385
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

Adapt test_warmup_conf to run in full_eviction mode

Change-Id: I759573c5d6d0ed594eae05d9026224d1b6938b3a
Reviewed-on: http://review.couchbase.org/58326
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

Replace std::list with more efficient std::vector

Also remove unused:
- variable "ready" from DcpProducer
- function appendQueue from DcpProducer
- function appendQueue from TapProducer
- function setEvents from TapConnMap

Change-Id: I986f74fe3a962690dada17a46bdc4111ec88123a
Reviewed-on: http://review.couchbase.org/58350
Reviewed-by: Jim Walker <jim@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17086: Fix to performance regression.

Revert "MB-16632: As part of queueDirty schedule a DCP connections notifier task"

This reverts commit fa17728e7ca0c637c84a2208b5decfe7ba7e54f1.

Performance testing showed that a regression has been introduced and that
fa17728 was the cause.

The regression was introduced by some fixes made during review that weren't
re-profiled.

Performance can be improved by making some further changes but the investigation
revealed that performance is actually at its best without fa17728.

Change-Id: I7ac3ff49d0b9ce8563f3a932dd337a58d03a0153
Reviewed-on: http://review.couchbase.org/58380
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-16632: Use a background task to handle snapshot creation

Frontend threads are delayed by large snaphots due to the time taken
in processing the items into the readyQ.

Moving this work to a background task frees frontend threads to
do other work.

Change-Id: Ic399ef06be996b7b7e179c4c8934a0f5a74cb8f7
Reviewed-on: http://review.couchbase.org/57148
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17231: conditionally delete temp items in get API

There are some conditions in which there might be a need to retain
a temporary non-existent item. For example, in the context of an
arithmetic API call, a get call in full eviction results in a
temp non-existent item that needs to be retained in order for a
subsequent ADD operation to not perform another background fetch.

Change-Id: I357d28a91b1e7ce56006accf919e48e99a643a2e
Reviewed-on: http://review.couchbase.org/58294
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Log True/False based on value of genBySeqno (bool)

Context: CheckpointManager

Change-Id: I9b79a4c061928a98a95eb995fbf45cd506800709
Reviewed-on: http://review.couchbase.org/58319
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Reduce log priority in conflict resolution to avoid unnecessary warnings

It is quite possible to receive mutations that aren't eligible
for last_write_wins conflict resolution strategy when time sync is
enabled. Therefore, reducing the priority of the log to avoid a
large number of warnings.

Change-Id: I032a91242eed9091b03530ca2f0803e742fd810e
Reviewed-on: http://review.couchbase.org/58290
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

Merge remote-tracking branch 'couchbase/sherlock' into 'couchbase/master'

couchbase/sherlock:
|\
| * 06be9e2 Log DCP stream creation before closing the stream
| *   5abd614 Merge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock'
| |\
| | * e338e1b MB-17220: [BP] Seperate logs for notifying seqno/checkpoint persistence
| | * 02064cd MB-17168: Log lastSentSeqno during takeover state change
| | * 380307d MB-16656: Set the open chkpt id on replica to 0 when disk snapshot is recvd.
| | * 6f869d1 MB-16656: Stream a full (disk+mem) snapshot from DCP producer on replica vb
| * | c28f736 MB-17088: Fix the underflow in certain vb stats.

Change-Id: Ib21ae054587d9c4250130e188d46885252a7de3c

MB-17030: Avoid caching wrong snapshot range during flush vb for an active vb

We cache vbstate, which includes snapshot start and end, when flusher
task runs. (This cached vbstate is persisted onto disk when a VBSnapshotTask task
or a peristVBStatePersistTask is run.) During flush vbucket if a checkpoint has
only chk start meta item then we end up persisting snap start and snap end as
0 which is incorrect. This fix address the bug.

Change-Id: Iea9588a4438a960993d0446a73ba51442e9ca6c8
Reviewed-on: http://review.couchbase.org/57969
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Log DCP stream creation before closing the stream

Th endStream() function emits a log message saying the stream was
closed. This log message should always come after the one that says
the stream was created.

Change-Id: Ibcd259373f5675cf49ea7a95ba3f2276507abe7a
Reviewed-on: http://review.couchbase.org/56615
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

ep_test_apis: Mark abort_msg as NORETURN

abort_msg() prints a message and calls abort() which never
returns. Mark the function prototype as NORETURN to compilers /
analzyers can know that any call to abort_msg() will also not return
(like abort()).

Change-Id: Id2b267bd306e5ed519a2326b84e6bfde3bc16d4a
Reviewed-on: http://review.couchbase.org/58269
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

ep_testsuite: Fix suspicious malloc() calls identified by clang analyzer

Potentially erronious uses of malloc were flagged by Clang static analyzer:

File:        ep-engine/tests/ep_testsuite.cc
Location:    line 5977, column 43
Description: Result of 'malloc' is converted to a pointer of type
             'protocol_binary_response_header', which is incompatible
             with sizeof operand type 'uint8_t'

Note these uses are safe in this instance (we arn't actually expecting
size(uint8_t) to be sufficient to store a
protocol_binary_response_header; however simplify to silence the
error.

Change-Id: Idd048587096a8476abdfe61d35ea6c626f65ae6f
Reviewed-on: http://review.couchbase.org/58268
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

ep_testsuite: Remove dead assignments

As identified by Clang static analyzer:

File:        ep-engine/tests/ep_testsuite.cc
Location:    line 5447, column 9
Description: Value stored to 'stream_opaque' is never read

Change-Id: Ifbc7909da0724c363bc3feb09200964f2d10a22d
Reviewed-on: http://review.couchbase.org/58267
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Merge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock'

couchbase/3.0.x:
|\
| * e338e1b MB-17220: [BP] Seperate logs for notifying seqno/checkpoint persistence
| * 02064cd MB-17168: Log lastSentSeqno during takeover state change
| * 380307d MB-16656: Set the open chkpt id on replica to 0 when disk snapshot is recvd.
| * 6f869d1 MB-16656: Stream a full (disk+mem) snapshot from DCP producer on replica vb

Change-Id: I92464ddfa7a20e1db12d2ab432b4072f6f57c8d8

MB-17220: [BP] Seperate logs for notifying seqno/checkpoint persistence

- Print different logs while notifying completion or timeouts
during seqno persistence and checkpoint persistence.
- Also adding additional information to the logs.

Change-Id: Idf29cab2197f37b180b0295b19f6b46542bdc6b6
Reviewed-on: http://review.couchbase.org/58233
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

MB-17088: Fix the underflow in certain vb stats.

Fix the potential underflow bug in dirtyQueueMem, dirtyQueueAge and
dirtyQueuePendingWrites stats.

Change-Id: If5df08e709404399b4e4595f150f5c4616897278
Reviewed-on: http://review.couchbase.org/58067
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-17168: Log lastSentSeqno during takeover state change

When an active vbucket state is changed to dead as part
of takeover, log a message that would indicate the last
sent seqno for the vbucket on the particular stream and
the vbucket's high seqno.

Change-Id: I7097b79cf41b2c62688ddb9345bc529ac08b2223
Reviewed-on: http://review.couchbase.org/58217
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Create Item with value only if needed in ForestKVStore::scan

In ForestKVStore::scan, if only keys are requested by the
application or in the case of deleted items, then don't
include the value for creating an Item

Change-Id: I1da0960b6fd66ac2136ef7a7f37344f15bd02479
Reviewed-on: http://review.couchbase.org/58071
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

MB-16656: Set the open chkpt id on replica to 0 when disk snapshot is recvd.

Currently due to a bug in 3.0.x the open checkpoint id is not set to 0
when replica receives a disk snapshot from active.

Change-Id: Iffda89b8da713539a52d50aa4acc33458ae7150e
Reviewed-on: http://review.couchbase.org/57834
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>

MB-16656: Stream a full (disk+mem) snapshot from DCP producer on replica vb

A replica vbucket receives items from an active vbucket, and until a full
snapshot is received the data on the replica vbucket is not consistent due
to de-duplication and other reasons. Hence while streaming items to a DCP
client from a replica vbucket we need to combine backfill and in memory
snapshots and send items in one snapshot. A caveat here is the replica vb
might not have received all the items in the latest (memory) snapshot, so the
DCP client streaming from replica will have to wait till the replica gets
all the items in the latest snapshot from the active.

Change-Id: I4db622f967316d120506dc9b125211578194bb60
Reviewed-on: http://review.couchbase.org/57816
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>

Merge remote-tracking branch 'couchbase/sherlock' into 'couchbase/master'

couchbase/sherlock: (discarding the changes from 9b12d0f)
|\
| *   b131bc4 Merge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock'
| |\
| | * 9b12d0f [DcpStream] Removing extra exception/abort that was added recently

Change-Id: I82a6ae4847a4c78418bdfadc05a7552d23eb6a9b

Merge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock'

couchbase/3.0.x:
* 9b12d0f [DcpStream] Removing extra exception/abort that was added recently

Change-Id: Ied21f9fb093c403cbcef44fcc6d2c9229030a2ad

[DcpStream] Removing extra exception/abort that was added recently

Exceptions in 3.0.x are unhandled which makes them pretty
much the same as aborts/asserts.

Although it is impossible for the event where an active stream
enters STREAM_READING state to occur , it may be the better
thing to do - to have the risk of hitting this assertion be ZERO
for the maintainance releases only.

Change-Id: I0a1eff5ab6c8cec8ad6d97e9a1c2201844c25fbd
Reviewed-on: http://review.couchbase.org/57962
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>

Merge remote-tracking branch 'couchbase/sherlock' into 'couchbase/master'

couchbase/sherlock:
|\
| * d92f99e Log destruction of Passive stream if it were live
| *   67256d2 Merge remote-tracking branch 'couchbase/3.0.x' into 'couchbase/sherlock'
| |\
| | * 5ffc61f MB-17051: [DcpProducer] Ensure no un-notified streams are left behind
| | * 5107095 [DcpProducer] Refactor function name to indicate intent

Change-Id: I028c9ff79f0a5e1cb398d97a450c6667065a2f20

Log destruction of Passive stream if it were live

Transition state of passive stream to dead in its
destructor, and log a message if we're destroying
a live stream.

Change-Id: I39651ee022a321048409345b9d987dc7c15003cf
Reviewed-on: http://review.couchbase.org/57861
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: buildbot <build@couchbase.com>