ep-engine.git
6 years ago[Backport] Fix deadlock in checkpoint persistence command 07/42407/2 3.0-refresh
Mike Wiederhold [Thu, 11 Sep 2014 18:46:00 +0000 (11:46 -0700)]
[Backport] Fix deadlock in checkpoint persistence command

We need to release the hpChkMutex before notifying memcached in
order to prevent a deadlock. The deadlock occurs when the flusher
trys to notify memcached of checkpoint persistence at the same
time a memcached worker thread is trying to add a new checkpoint
persistence request.

Change-Id: Ida313f5b39ef0e063dee9882410cd0a19ce55292
Reviewed-on: http://review.couchbase.org/41363
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-on: http://review.couchbase.org/42407
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11948: Handle GetLastError() returning 0 condition 00/42400/2
Sriram Ganesan [Fri, 22 Aug 2014 22:41:57 +0000 (15:41 -0700)]
MB-11948: Handle GetLastError() returning 0 condition

In windows, the GetLastError() call sometimes returns 0 instead of
the returning WSAEWOULDBLOCK. Either the error gets reset before
the invocation or it could be undefined behavior.

Change-Id: Ifb8f7d576b447d090b21cc0747d068ead0597182
Reviewed-on: http://review.couchbase.org/40865
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Sriram Ganesan <sriram@couchbase.com>
Reviewed-on: http://review.couchbase.org/42400
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-12137: Don't update the current snapshot during persistence 89/41289/2 3.0 v3.0.0
Mike Wiederhold [Tue, 9 Sep 2014 21:31:07 +0000 (14:31 -0700)]
MB-12137: Don't update the current snapshot during persistence

If we update the current snapshot only when we persist items to
disk then we may miss out on updates to the current snapshot
when there are vbucket state changes.

Change-Id: Iea3139afb669bdf32e6b4d98e8b3515cafe578dc
Reviewed-on: http://review.couchbase.org/41289
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12063 Don't delete expired items during warmup. 19/41219/3
Chiyoung Seo [Fri, 5 Sep 2014 06:15:38 +0000 (23:15 -0700)]
MB-12063 Don't delete expired items during warmup.

When a node is added back to the cluster, its vbucket states
can be different from its previous states (e.g., active -> replica).
Therefore, we should not remove expired items from active vbuckets
during warmup because the states of those vbuckets can be changed
after the warmup.

Change-Id: Ifec4fda80e424a8afa74bae66e17de77b2e0c36b
Reviewed-on: http://review.couchbase.org/41219
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-12100: Don't expire items for replica vbuckets during warmup 77/41177/2
Mike Wiederhold [Wed, 3 Sep 2014 22:16:31 +0000 (15:16 -0700)]
MB-12100: Don't expire items for replica vbuckets during warmup

Change-Id: I81095beb69c0996d819158e3cadc451a83653d2f
Reviewed-on: http://review.couchbase.org/41177
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoFix build breakage 12/41012/2
Mike Wiederhold [Thu, 28 Aug 2014 07:05:45 +0000 (00:05 -0700)]
Fix build breakage

Change-Id: Ibeac0d57fc1ea3bc94bf488ee5e82cf7e9143d3a
Reviewed-on: http://review.couchbase.org/41012
Reviewed-by: Chris Hillery <ceej@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12063: Properly save the max deleted seqno during vb snapshots 97/40997/3
Mike Wiederhold [Wed, 27 Aug 2014 22:47:51 +0000 (15:47 -0700)]
MB-12063: Properly save the max deleted seqno during vb snapshots

Change-Id: Ife9e3b26956638622de0427cd573cd4d37c7e443
Reviewed-on: http://review.couchbase.org/40997
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12063: Lock the current snapshot mutex when flushing items 96/40996/2
Mike Wiederhold [Wed, 27 Aug 2014 21:41:08 +0000 (14:41 -0700)]
MB-12063: Lock the current snapshot mutex when flushing items

We need to lock the current snapshot mutex before grabbing items
because it is possible that an itme might be queued, but the
current snapshot might be updated.

Change-Id: Ieff85a4605041e764d6facf055c71c59e7e19955
Reviewed-on: http://review.couchbase.org/40996
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12057: warmup must not block bucket creation 92/40892/8
Sundar Sridharan [Tue, 26 Aug 2014 21:23:49 +0000 (14:23 -0700)]
MB-12057: warmup must not block bucket creation

Bucket creation is a front end thread and warmup is done by
engine background threads. So if bucket creation waits for
warmup then a deadlock happens when there are few reader threads
Test Infrastructure expects vb0 created by ep-engine init
or warmup. This is not necessary for production, so moving
vb0 creation to test setup. Minor test code bug fixes.
Change-Id: I5eae9c8f36fe39d2a18cb1800dc48a29162eab28
Reviewed-on: http://review.couchbase.org/40892
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-12042 check for existence of vbucket in warmup callback 49/40849/4
Sundar Sridharan [Fri, 22 Aug 2014 20:38:41 +0000 (13:38 -0700)]
MB-12042 check for existence of vbucket in warmup callback

Change-Id: Iab0ad237392d681f28b91c29dc51b7631141e187
Reviewed-on: http://review.couchbase.org/40849
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoMB-11973: Only grab items from the chk manager one chk at a time 65/40765/4
Mike Wiederhold [Wed, 20 Aug 2014 20:36:31 +0000 (13:36 -0700)]
MB-11973: Only grab items from the chk manager one chk at a time

We should only grab a single checkpoint at a time from the checkpoint
manager in order to make sure that the replica vbuckets cannot move
into the open checkpoint. This will also help make sure we don't
copy too make items into an upr stream.

Change-Id: I519353df69cf389e6f32f77ffc6afe97dfba7b36
Reviewed-on: http://review.couchbase.org/40765
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12032: Clean up TaskQs after last bucket deletion 41/40741/3
Sundar Sridharan [Wed, 20 Aug 2014 01:48:08 +0000 (18:48 -0700)]
MB-12032: Clean up TaskQs after last bucket deletion

Without this sometimes threads can end up sleeping on
a high priority Q even when none exist.
This can result in minor slowdown rarely in some platforms

Change-Id: I58cfe9234e17b9b5502f780c7e334574f64fb30d
Reviewed-on: http://review.couchbase.org/40741
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoMB-11980: Fix bugs caused from persisting the current snapshot 07/40707/4
Mike Wiederhold [Wed, 20 Aug 2014 01:35:58 +0000 (18:35 -0700)]
MB-11980: Fix bugs caused from persisting the current snapshot

There are two issues fixed here. One is around the current
snapshot not being updated properly during bucket flush. The
second is an issue caused by persisting the wrong information
for the high seqno, snap start seqno, and snap end seqno during
the snapshot vbuckets task. These values should remain the same
as they were after the last vbucket persistence task otherwise
we might persist incorrect values.

Change-Id: Ie4a357270b52bd35b5f29018f24fd3550913f1cc
Reviewed-on: http://review.couchbase.org/40707
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11934: Fix warmup for windows 73/40673/3
Sriram Ganesan [Fri, 15 Aug 2014 21:55:54 +0000 (14:55 -0700)]
MB-11934: Fix warmup for windows

During warmup in windows, the platform library appends a backslash instead
of a forward slash causing a database file name parsing to break.

Change-Id: I4b3d2296c83dcdf4f94698532a2115dedc841fdb
Reviewed-on: http://review.couchbase.org/40673
Tested-by: Sriram Ganesan <sriram@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11876 Safely cancel AccessScanner task on shutdown 69/40669/4
Sundar Sridharan [Sat, 16 Aug 2014 01:13:32 +0000 (18:13 -0700)]
MB-11876 Safely cancel AccessScanner task on shutdown

Change-Id: I769ca2ad562ef2b27447d40b6a66c4e24ff21eb3
Reviewed-on: http://review.couchbase.org/40669
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoCorrectly return ENGINE_ERROR_CODE in processing mutation messages 53/40653/2
Chiyoung Seo [Fri, 15 Aug 2014 17:25:26 +0000 (10:25 -0700)]
Correctly return ENGINE_ERROR_CODE in processing mutation messages

Change-Id: I72cadd9beb9a024be7fab5d54c7ecf8a85a40c79
Reviewed-on: http://review.couchbase.org/40653
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoFix the regression from the change for MB-11943 45/40645/4
Chiyoung Seo [Fri, 15 Aug 2014 05:05:05 +0000 (22:05 -0700)]
Fix the regression from the change for MB-11943

The change for MB-11943 caused the sporadic failure
in "vbucket deletion doesn't affect new data" unit test case.

This change makes sure that we have the same logic in maintaining
cachedVBState instance for a deleted vbucket.

Change-Id: I7fda66a258c46cd04820001e945e0679eed0469b
Reviewed-on: http://review.couchbase.org/40645
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoWindows don't support the %zu formatter 50/40650/4
Trond Norbye [Fri, 15 Aug 2014 15:42:51 +0000 (17:42 +0200)]
Windows don't support the %zu formatter

Change-Id: Ia36bc151f84af9cfd6c069c3d97c494753d449f3
Reviewed-on: http://review.couchbase.org/40650
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11958: Make sure mutations and snapshot ranges are updated together 42/40642/4
Mike Wiederhold [Fri, 15 Aug 2014 07:33:43 +0000 (00:33 -0700)]
MB-11958: Make sure mutations and snapshot ranges are updated together

There is a race condition where an items seqno is updated and the
flusher runs and persists both the seqno and old snapshot range
before the new snapshot range can be updated.

Change-Id: If124e5ab05c45c68a6956d28cc3898f8c18df6a9
Reviewed-on: http://review.couchbase.org/40642
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11890: Active vb shouldn't initiate add Stream req 41/40641/4
abhinavdangeti [Thu, 14 Aug 2014 23:12:37 +0000 (16:12 -0700)]
MB-11890: Active vb shouldn't initiate add Stream req

Change-Id: Ic902e632d52f426a73d7708f9c69854dc7fab874
Reviewed-on: http://review.couchbase.org/40641
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11916 Synchronize the access to thread job logs. 37/40637/2
Chiyoung Seo [Thu, 14 Aug 2014 21:14:05 +0000 (14:14 -0700)]
MB-11916 Synchronize the access to thread job logs.

Change-Id: I9551cd70b931b1ed92f806be1df0325d922a239c
Reviewed-on: http://review.couchbase.org/40637
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11945 Remove the assertion check on sequence number in the flusher. 30/40630/2
Chiyoung Seo [Thu, 14 Aug 2014 19:41:29 +0000 (12:41 -0700)]
MB-11945 Remove the assertion check on sequence number in the flusher.

We sporadically observe the sequence number mismatches between the
flusher and Couchstore. This is not a regression from any recent
changes, and mostly existed much earlier. We will investigate this
issue separately to see if there are any bugs in ep-engine or
Couchstore.

Change-Id: Ifac2eae0deae09cad09dd7b5664f63bcbf828b24
Reviewed-on: http://review.couchbase.org/40630
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11299: Don't let replica vbuckets send partial snapshots 46/40346/6
Mike Wiederhold [Wed, 6 Aug 2014 18:34:59 +0000 (11:34 -0700)]
MB-11299: Don't let replica vbuckets send partial snapshots

Replica vbuckets cannot create their own snapshots and must
rely on the snapshots it received from the active vbucket
when forwarding the snapshots to a downstream consumer. In
order to make sure that this happens properly we will only
stream mutations from the closed checkpoint for non-active
vbuckets. This will guarantee that the above requirement is
met.

Change-Id: I58151ce96868e34083f87dca0ce1d18b6a12aedf
Reviewed-on: http://review.couchbase.org/40346
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11943: replace map with vector for cachedVBStates 00/40600/6
Sundar Sridharan [Thu, 14 Aug 2014 05:39:29 +0000 (22:39 -0700)]
MB-11943: replace map with vector for cachedVBStates

std::map can incur thread-unsafe rb balance operations which
may invalidate concurrent iterators causing asserts or crashes
Pre-allocated vectors are thread-safe as long as the size is not
altered
Change-Id: I777b0cc7d9c421cc11425db75506fdf90044a93c
Reviewed-on: http://review.couchbase.org/40600
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11941: Use the latest seqno during tap mutation/deletion cur snapshot 01/40601/2
Mike Wiederhold [Thu, 14 Aug 2014 01:38:39 +0000 (18:38 -0700)]
MB-11941: Use the latest seqno during tap mutation/deletion cur snapshot

We generate the seqno when inserting the item into the hashtable when
using tap. As a reuslt we need to retrieve that seqno when updating
the latest snapshot because the byseqno passed into the mutation and
deletion functions for tap is always 0.

Change-Id: I21b7888207d9fea4f46b3d5918d1c8b60b2ea1ff
Reviewed-on: http://review.couchbase.org/40601
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11928: Fast path for rollback to 0 95/40595/3
Mike Wiederhold [Thu, 14 Aug 2014 00:25:29 +0000 (17:25 -0700)]
MB-11928: Fast path for rollback to 0

Right now we run through all of the headers even if we have to
rollback to 0. This change provides a fast path to rollback to
zero by immediately resetting the vbucket.

Change-Id: I8e7b62ec711838894c4eda92eaff78704bd94d50
Reviewed-on: http://review.couchbase.org/40595
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoIncorrect parameter for scheduleVBDeletion 93/40593/3
abhinavdangeti [Wed, 13 Aug 2014 23:46:38 +0000 (16:46 -0700)]
Incorrect parameter for scheduleVBDeletion

This caused completeBeforeShutdown to be set
to false, in resetVBucket/deleteVBucket
scenarios.

Change-Id: I72a1f07f475af49a7f368695a8e83cf2c5df9e28
Reviewed-on: http://review.couchbase.org/40593
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11933: Only set cur snapshot when receving the first snapshot item 83/40583/3
Mike Wiederhold [Wed, 13 Aug 2014 19:50:03 +0000 (12:50 -0700)]
MB-11933: Only set cur snapshot when receving the first snapshot item

We were setting the current snapshot when recieving the snapshot
marker and this is incorrect because if we disconnect before recieving
any items for that snapshot then the next time the replica vbucket
tries to connect to the active the high seqno will be less than the
last snapshot start seqno and this will cause the stream request to
fail with invalid arguments.

Change-Id: I75e1f555656c24221b2697011df878c579dce70e
Reviewed-on: http://review.couchbase.org/40583
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11941: Update current vbucket snapshot when receiving tap items 82/40582/2
Mike Wiederhold [Wed, 13 Aug 2014 19:35:39 +0000 (12:35 -0700)]
MB-11941: Update current vbucket snapshot when receiving tap items

We need to update the vbucket snapshot in order to support upgrades
because we initially have a tap stream and then move to a dcp stream.
Since tap doesn't support snapshots we can just set the current snapshot
to be the same as the seqno for that item.

Change-Id: I9da32df9366508276619507cf6cb9a66557e0919
Reviewed-on: http://review.couchbase.org/40582
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11952 Lower the default max number of readers. 79/40579/2
Chiyoung Seo [Wed, 13 Aug 2014 18:15:43 +0000 (11:15 -0700)]
MB-11952 Lower the default max number of readers.

Change-Id: I43dbcc4d606eea07276327ec7b9ed4f9bf3fc0e4
Reviewed-on: http://review.couchbase.org/40579
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11642 Batch load dcp mutations if replication 92/40392/10
Sundar Sridharan [Wed, 13 Aug 2014 00:40:21 +0000 (17:40 -0700)]
MB-11642 Batch load dcp mutations if replication

Instead of loading 1 mutation per vbucket send out 10
dcp mutations per dcp step() if it is a intra-cluster
replication stream.
This is done so that intra-cluster replication is always
faster than XDCR and views.
Noticeable  improvement seen in intra-cluster replication
Change-Id: I15fc0081852e8e11027a20febe3224acdf21b661
Reviewed-on: http://review.couchbase.org/40392
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoMB-11299: Allow rollbacks to a partial snapshot 40/40340/5
Mike Wiederhold [Wed, 6 Aug 2014 16:47:34 +0000 (09:47 -0700)]
MB-11299: Allow rollbacks to a partial snapshot

If we roll back to a partial snapshot we need to make sure to get
the snapshot start and end seqnos for the partial snapshot in order
to make sure that we don't accidentally miss any items when
reconnecting.

Change-Id: If95d0476d8d582d6b57786387212f03b2aafc7ba
Reviewed-on: http://review.couchbase.org/40340
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11912 Monitor workload pattern for dynamic compaction scheduling. 20/40520/5
Chiyoung Seo [Tue, 12 Aug 2014 02:22:46 +0000 (19:22 -0700)]
MB-11912 Monitor workload pattern for dynamic compaction scheduling.

This change allows us to monitor workload pattern (i.e.,
read_heavy, write_heavy, mixed) at runtime and use the monitored
workload pattern to adjust the number of concurrent compaction
tasks dynamically at runtime. If the workload pattern is read-heavy,
then only one compaction task is executed at a time.

Note that we plan to keep improving this area for better I/O
scheduling.

Change-Id: Ibbb27da0a7e9f3951571ad8068570ebbf9abb318
Reviewed-on: http://review.couchbase.org/40520
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoAdd id to the logs of checkpoint persistence 13/40513/5
Sriram Ganesan [Tue, 12 Aug 2014 01:27:44 +0000 (18:27 -0700)]
Add id to the logs of checkpoint persistence

Add the id to the logs of checkpoint persistence so that we will have the
sequence number/checkpoint id in case a timeout occurs on checkpoint
persistence

Change-Id: Ifd3c70c9d40de43f0b3f1fdc1cf495ee0dc95d60
Reviewed-on: http://review.couchbase.org/40513
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sriram Ganesan <sriram@couchbase.com>
6 years agoMB-11299: Use the snapshot start seqno when creating a failover entry 41/40341/3
Mike Wiederhold [Wed, 6 Aug 2014 17:03:33 +0000 (10:03 -0700)]
MB-11299: Use the snapshot start seqno when creating a failover entry

Change-Id: I7a2c0adf51a761a491d0f12720f2abef8127e5c3
Reviewed-on: http://review.couchbase.org/40341
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11887: Properly send the snapshot start and end seqnos 76/40276/10
Mike Wiederhold [Thu, 7 Aug 2014 18:58:30 +0000 (11:58 -0700)]
MB-11887: Properly send the snapshot start and end seqnos

Currently we always set the snapshot start and end seqno to be the
same as the start seqno. This is incorrect if we recently were
disconnected and need to resume from a partial snapshot.

Change-Id: I7c9430e0b24ef01cc427cb03b518f4c41a55d875
Reviewed-on: http://review.couchbase.org/40276
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoAfter APPEND/PREPEND to json, set datatype to binary 90/40390/3
abhinavdangeti [Fri, 8 Aug 2014 17:53:15 +0000 (10:53 -0700)]
After APPEND/PREPEND to json, set datatype to binary

This is because appending or prepending anything
to a JSON document breaks the JSON structure.

Change-Id: I0acd99b0c43a54e0b34ce21f3a69c41709e3817e
Reviewed-on: http://review.couchbase.org/40390
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11741: Better scheduling of checkpointRemoverTask 55/40355/6
abhinavdangeti [Thu, 7 Aug 2014 02:48:24 +0000 (19:48 -0700)]
MB-11741: Better scheduling of checkpointRemoverTask

Waking up the checkpoint Remover Task once the cursors
complete parsing a checkpoint till the end, helps
prevent retaining blob instances for a longer time
than necessary.

Change-Id: Ieb1435dafac2e9b9984664218d6761c6c807333a
Reviewed-on: http://review.couchbase.org/40355
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agofix copy paste bug to set max non-io at runtime 17/40317/3
Sundar Sridharan [Wed, 6 Aug 2014 00:17:56 +0000 (17:17 -0700)]
fix copy paste bug to set max non-io at runtime

Change-Id: I097e5fc5d926387dff8874427d373e41e383f64e
Reviewed-on: http://review.couchbase.org/40317
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11872: Compile failure on (some?) Debian 7 systems 97/40297/3
Trond Norbye [Tue, 5 Aug 2014 12:22:05 +0000 (14:22 +0200)]
MB-11872: Compile failure on (some?) Debian 7 systems

Building from a Debian 7 setup under SmartOS works out of the box,
but one of our builders does not work for some reason. From looking
at the preprocessed file it doesn't define linux like our other
systems do. Extend the test to work for __linux__ and __linux as
well.

Change-Id: I4a7cf304ab8bfddb6cc117e9064165e1eb16b66e
Reviewed-on: http://review.couchbase.org/40297
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chris Hillery <ceej@couchbase.com>
6 years agoMB-11884: Consider min_item_threshold in full eviction 08/40308/4
abhinavdangeti [Tue, 5 Aug 2014 23:09:41 +0000 (16:09 -0700)]
MB-11884: Consider min_item_threshold in full eviction

In this case, as warmed up keys always matches warmed
up values, we can mark the min item threshold using
warmup's estimated item count.

Change-Id: Ibd1d58edd15be877c24ef794615fe0d56f9fc675
Reviewed-on: http://review.couchbase.org/40308
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11847: [Warmup] Estimated value count with full eviction 04/40304/4
abhinavdangeti [Tue, 5 Aug 2014 18:18:09 +0000 (11:18 -0700)]
MB-11847: [Warmup] Estimated value count with full eviction

In loadingKVPairs phase, set estimated value count equal
to the estimated item count, as no keys have been warmed
up yet, when in full eviction mode.

Change-Id: Id74237c96e0c144d9a9b94e0670f4975739e7914
Reviewed-on: http://review.couchbase.org/40304
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoHave at least 1 unit test with high priority bucket 13/40313/2
Sundar Sridharan [Tue, 5 Aug 2014 23:14:59 +0000 (16:14 -0700)]
Have at least 1 unit test with high priority bucket

Change-Id: Icc92c9455c13ca1af21eb285c72b8a8f3555cf6b
Reviewed-on: http://review.couchbase.org/40313
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoMB-11876 Check for new priority buckets after wakeup 11/40311/3
Sundar Sridharan [Tue, 5 Aug 2014 21:49:19 +0000 (14:49 -0700)]
MB-11876 Check for new priority buckets after wakeup

If threads do not check for the presence of newly registered
buckets, then a rare race condition can happen where a thread
is unaware of the presence of a low priority bucket and keeps
continously checking the high priority Q which has no work.
This would block normal shutdown

Change-Id: Ic59ccbed69c205251e7897f349445d8982ec7be5
Reviewed-on: http://review.couchbase.org/40311
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years ago_unlink as opposed to unlink for windows 80/40280/2
abhinavdangeti [Mon, 4 Aug 2014 23:19:06 +0000 (16:19 -0700)]
_unlink as opposed to unlink for windows

Change-Id: I40a74c0942594cf48dfbbd3406dc935a4dbfbd35
Reviewed-on: http://review.couchbase.org/40280
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11870 Avoid polling starvation in mixed priorities 42/40242/3
Sundar Sridharan [Sun, 3 Aug 2014 22:05:18 +0000 (15:05 -0700)]
MB-11870 Avoid polling starvation in mixed priorities

1) Wake up threads from the correct Q in mixed priorities.
2) If a thread has just woken up off a high priority Q
then make sure it polls the low priority Q next.
Not doing this causes starvation in the low prioirty Q
resuling in rebalance hangs.
3) Avoid sleeping threads for longer than 2 seconds to avoid
missing posts from other threads
Change-Id: I6cef6153285098e2f77639c50b194c9a9541dd24
Reviewed-on: http://review.couchbase.org/40242
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agodebug for MB-11870 39/40239/4
Sundar Sridharan [Sun, 3 Aug 2014 21:38:35 +0000 (14:38 -0700)]
debug for MB-11870

print engine name in dispatcher taskLog
move engine unregistration into WARNING
Change-Id: Ide1206642396f7013017f933a4b5b35cb741c19c
Reviewed-on: http://review.couchbase.org/40239
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11779: Switch the engine context before we update the task log 01/40201/3
Mike Wiederhold [Fri, 1 Aug 2014 23:43:17 +0000 (16:43 -0700)]
MB-11779: Switch the engine context before we update the task log

If we don't do this then we lose 48 bytes from the bucket mem_used
each time the executor thread runs.

Change-Id: Iecf216e3c262573a3b36911f80f44b92851d2a5f
Reviewed-on: http://review.couchbase.org/40201
Tested-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11847: Set estimatedValueCount in loadKVPairs phase 00/40200/2
abhinavdangeti [Fri, 1 Aug 2014 23:05:31 +0000 (16:05 -0700)]
MB-11847: Set estimatedValueCount in loadKVPairs phase

- Otherwise, ep_warmup_estimated_value count would show
"unknown" after warmup completes in some scenarios.
- Also remove unused functions.

Change-Id: I2e2fedf58dcd49a6f572f830c4566b0d8946033b
Reviewed-on: http://review.couchbase.org/40200
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMinor refactoring on checkpoint cursor registration function. 92/40192/2
Chiyoung Seo [Fri, 1 Aug 2014 18:46:07 +0000 (11:46 -0700)]
Minor refactoring on checkpoint cursor registration function.

Change-Id: I7fdffc233664e26a434e0c757e72416e567d1eb2
Reviewed-on: http://review.couchbase.org/40192
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-9131 Warmup should check for couchstore dbfile before open 78/40078/5
D Liao [Wed, 30 Jul 2014 23:44:54 +0000 (16:44 -0700)]
MB-9131 Warmup should check for couchstore dbfile before open

When creating dbFileRevMap during warmup, we now check for the
existence of couchstore db files so that we don't open non-existing
files so to avoid generating warning logs.

Change-Id: I8dd7b423f01240e8958b88fce2d58b429f0c9590
Reviewed-on: http://review.couchbase.org/40078
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoUse the metadata disk size stat for replica vbucket. 58/40158/2
Chiyoung Seo [Fri, 1 Aug 2014 03:04:03 +0000 (20:04 -0700)]
Use the metadata disk size stat for replica vbucket.

Change-Id: Ied887dd0ac0e3ac0ed309c554d27d95775c9874b
Reviewed-on: http://review.couchbase.org/40158
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11864 Initialize max_workers variables before creating threads. 56/40156/2
Chiyoung Seo [Fri, 1 Aug 2014 01:50:22 +0000 (18:50 -0700)]
MB-11864 Initialize max_workers variables before creating threads.

Change-Id: Ie7853d8bd874117e976c74708172a36cf8c8e7c0
Reviewed-on: http://review.couchbase.org/40156
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11863 DaemonVBSnapshot task must complete before shutdown 55/40155/3
Sundar Sridharan [Fri, 1 Aug 2014 00:58:14 +0000 (17:58 -0700)]
MB-11863 DaemonVBSnapshot task must complete before shutdown

Change-Id: Iacb7852525a5055bac16a61d02e4ed65daf17319
Reviewed-on: http://review.couchbase.org/40155
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agocorrectly decrement numToWake to avoid waking more 54/40154/3
Sundar Sridharan [Thu, 31 Jul 2014 23:33:59 +0000 (16:33 -0700)]
correctly decrement numToWake to avoid waking more

Change-Id: I2a0e5f2bb38a0d8831a05028b89bee98fa48c22a
Reviewed-on: http://review.couchbase.org/40154
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11861 Remove incorrect assertion from the flusher. 51/40151/2
Chiyoung Seo [Thu, 31 Jul 2014 23:09:47 +0000 (16:09 -0700)]
MB-11861 Remove incorrect assertion from the flusher.

As VBucket memory deletion can be interleaved with the flusher,
the assertion check on the high sequence number is not required.

Change-Id: Iac80a68f47ec315d01cb18133c0282396e6268cb
Reviewed-on: http://review.couchbase.org/40151
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agofix compiler error in vbucket.cc metaDataDisk += 50/40150/2
Sundar Sridharan [Thu, 31 Jul 2014 22:54:21 +0000 (15:54 -0700)]
fix compiler error in vbucket.cc metaDataDisk +=

Change-Id: I5b2992d3298d5b65f959f48cb078ed1c1f7a01b1
Reviewed-on: http://review.couchbase.org/40150
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoMB-11689 Added a stat ep_meta_data_disk 38/40038/5
D Liao [Wed, 30 Jul 2014 01:24:43 +0000 (18:24 -0700)]
MB-11689 Added a stat ep_meta_data_disk

This gives approx disk space used to store item meta data by couchstore.

Change-Id: I7f1642279c9aca84b02c885fd621cfdf50c20e98
Reviewed-on: http://review.couchbase.org/40038
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11582: Increase mutation memory threshold to 98% of the bucket mem quota 26/40126/6
abhinavdangeti [Thu, 31 Jul 2014 21:31:25 +0000 (14:31 -0700)]
MB-11582: Increase mutation memory threshold to 98% of the bucket mem quota

Backfill memory threshold still at 95% of the bucket memory quota, but
configurable at runtime.

Change-Id: I453b9d0bd7ac122ea21d6882ed88c010997413d3
Reviewed-on: http://review.couchbase.org/40126
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoDon't schedule backfill if cursor isn't registered at first chk item 45/40145/3
Mike Wiederhold [Thu, 31 Jul 2014 21:15:29 +0000 (14:15 -0700)]
Don't schedule backfill if cursor isn't registered at first chk item

If the cursor is not registered at the first item in the checkpoint
then we do not need to schedule backfill. Since items get
de-duplicated in the checkpoint queues the current logic might
choose to schedule a backfill unnecessarily. This change stops the
scheduling of unnecessary backfills.

Change-Id: I11188d302be57035cfac5f1c105cbc6d7179bde6
Reviewed-on: http://review.couchbase.org/40145
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11405 Spawn at least 10 global threads 84/40084/10
Sundar Sridharan [Thu, 31 Jul 2014 21:09:27 +0000 (14:09 -0700)]
MB-11405 Spawn at least 10 global threads

Make the number of threads per task group configurable both
at start time as well as runtime for performance tuning
Change-Id: I4dcbb4de79cfa39375602e498f321d596145b81d
Reviewed-on: http://review.couchbase.org/40084
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11405 Dedicated Threads that poll & sleep on own Q only 80/40080/6
Sundar Sridharan [Thu, 31 Jul 2014 00:47:56 +0000 (17:47 -0700)]
MB-11405 Dedicated Threads that poll & sleep on own Q only

Threads only poll their own Q type toggling between high and low
priority queues and sleep on their own queue type only.
Even if there are more tasks ready, do not wake threads from
other queues.
This is to reduce kernel CPU context switching overhead

Change-Id: I6d77930d1c8f4f6b363fbcf5cdabcf4db0cf6b06
Reviewed-on: http://review.couchbase.org/40080
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11821 rename upr to dcp in stats and logs 98/39898/9
Sundar Sridharan [Mon, 28 Jul 2014 18:17:29 +0000 (11:17 -0700)]
MB-11821 rename upr to dcp in stats and logs

Change-Id: Ib2cf48190b2bf274f953b8da1b1cd1373de190c2
Reviewed-on: http://review.couchbase.org/39898
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoFix leak while checking memory thresholds during warmup 85/40085/3
abhinavdangeti [Thu, 31 Jul 2014 02:32:51 +0000 (19:32 -0700)]
Fix leak while checking memory thresholds during warmup

Change-Id: I13afce24fe325d334575d7d8e315ebe46aa23db7
Reviewed-on: http://review.couchbase.org/40085
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoFix to unit test failure: test_item_pager 81/40081/3
abhinavdangeti [Thu, 31 Jul 2014 00:47:48 +0000 (17:47 -0700)]
Fix to unit test failure: test_item_pager

Change-Id: I186e6d631ff6723fbb2833cff450e9fceaa69037
Reviewed-on: http://review.couchbase.org/40081
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11582: Wake up the itemPagertask(if snoozing) upon tmpOOMs 37/40037/6
abhinavdangeti [Wed, 30 Jul 2014 22:11:58 +0000 (15:11 -0700)]
MB-11582: Wake up the itemPagertask(if snoozing) upon tmpOOMs

Change-Id: I48de57e2c2cc0446228b8ac9463f35d8e02fa0de
Reviewed-on: http://review.couchbase.org/40037
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11830 unless recreating don't resetVBucket on delVBucket 39/40039/5
Sundar Sridharan [Wed, 30 Jul 2014 21:21:00 +0000 (14:21 -0700)]
MB-11830 unless recreating don't resetVBucket on delVBucket

This change avoids writing an empty json failover table which
can cause tools like cbtransfer to fail
Change-Id: I658a08222800a4dc35b2c0b469de6cfbbbb8e9a4
Reviewed-on: http://review.couchbase.org/40039
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11237: Add backoff stats to dcp and dcpagg stats 61/40061/5
Mike Wiederhold [Wed, 30 Jul 2014 18:27:08 +0000 (11:27 -0700)]
MB-11237: Add backoff stats to dcp and dcpagg stats

This stat will be incremented each time the consumer is unable
to process items due to memory usage being too high.

Change-Id: Ie79c4c57f8b0c364f6075f9ba1be776f5bcc490a
Reviewed-on: http://review.couchbase.org/40061
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11383: When thresholds are met, set warmup to complete 62/40062/8
abhinavdangeti [Wed, 30 Jul 2014 20:28:50 +0000 (13:28 -0700)]
MB-11383: When thresholds are met, set warmup to complete

With parallelization, it is possible that other warmup
threads would not respect the completion state that is
marked by one thread. This fix, would ensure that other
threads will also respect the completion state set by
one thread as best possible.

Change-Id: Ie3223e5a95ac5f3c3281e18184c0cebf63fe993f
Reviewed-on: http://review.couchbase.org/40062
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11042 print correct usage help for cbstats 69/40069/3
Sundar Sridharan [Wed, 30 Jul 2014 20:50:52 +0000 (13:50 -0700)]
MB-11042 print correct usage help for cbstats

Shows the mandatory options and optional arguments
Also fixes cbvbucketctl, cbepctl and any user of clitool.py

Change-Id: I554f39d295d16b19d5cbf675c43c142f43ba2f7d
Reviewed-on: http://review.couchbase.org/40069
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoMB-11801: Seqno persist command should return when persisted seqno == seqno 63/40063/3
Mike Wiederhold [Wed, 30 Jul 2014 19:10:10 +0000 (12:10 -0700)]
MB-11801: Seqno persist command should return when persisted seqno == seqno

If the seqno that we are waiting for is already persisted we should return
immediately. This will significantly reduce empty rebalance times.

Change-Id: I7e7437b8a3a370a985149a3d85f6ba250fd2385f
Reviewed-on: http://review.couchbase.org/40063
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11799 Let the flusher update the last persisted checkpoint id. 59/40059/3
Chiyoung Seo [Wed, 30 Jul 2014 18:14:10 +0000 (11:14 -0700)]
MB-11799 Let the flusher update the last persisted checkpoint id.

This change makes sure that only the flusher should update the
last persisted checkpoint id. This is required to prevent the
checkpoint remover task from removing checkpoints that are not
persisted yet.

Change-Id: I7577928b6fad7033738ce2ac8092c90e7774b87d
Reviewed-on: http://review.couchbase.org/40059
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoprevent task state change from overriding task cancel 60/40060/2
Sundar Sridharan [Wed, 30 Jul 2014 18:08:33 +0000 (11:08 -0700)]
prevent task state change from overriding task cancel

task states of snooze and running must not override task cancel
otherwise it can cause a hang while shutting down bucket
use atomic compare and swap for safe lockless operation.

Change-Id: Ibfc3c7745d0c40893c577d3d94775f523ecc2861
Reviewed-on: http://review.couchbase.org/40060
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11799 Throttle vbucket compaction with the disk write queue size. 43/40043/3
Chiyoung Seo [Wed, 30 Jul 2014 01:56:08 +0000 (18:56 -0700)]
MB-11799 Throttle vbucket compaction with the disk write queue size.

If a disk write queue size is above the threshold, it is better to
throttle vbucket compaction tasks, so that the minimum number of
compaction tasks can be running at the same time.

As another commit, we will make both the write queue size threshold and
the minimum number of compaction tasks configurable at runtime.

Change-Id: I70d6c3f02507895ed767060870e9e16265cd60ac
Reviewed-on: http://review.couchbase.org/40043
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoRemove unnecessary lock from Task class. 42/40042/4
Chiyoung Seo [Tue, 29 Jul 2014 20:46:38 +0000 (13:46 -0700)]
Remove unnecessary lock from Task class.

Task::snooze is always invoked by only one thread at any time.

Change-Id: If0921eadafb562cd124d8ce29b154f47f77861c5
Reviewed-on: http://review.couchbase.org/40042
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoFix log message for remaining bytes to send to mccouch 44/40044/2
Trond Norbye [Wed, 30 Jul 2014 05:42:05 +0000 (07:42 +0200)]
Fix log message for remaining bytes to send to mccouch

In addition to improving the log message, backoff trying
to send the data to allow the other side to drain their
end.

Change-Id: Ibddfd9f87c1025d3256ad80f5e49516d56222e69
Reviewed-on: http://review.couchbase.org/40044
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
6 years agoassert if mock file remove fails. Use env for test dir 81/39981/4
Sundar Sridharan [Tue, 29 Jul 2014 22:14:56 +0000 (15:14 -0700)]
assert if mock file remove fails. Use env for test dir

Change-Id: I88640746259cd7d954d07e7d760deb1dc504272b
Reviewed-on: http://review.couchbase.org/39981
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
6 years agoflush multi vbucket unit test fix - acquire vbucket lock for flushall 29/40029/3
Sundar Sridharan [Tue, 29 Jul 2014 21:36:13 +0000 (14:36 -0700)]
flush multi vbucket unit test fix - acquire vbucket lock for flushall

failure to acquire this lock can cause flusher and
flushall tasks to race and hit the assert highSeqno <= vb->getHighSeqno()

Change-Id: Ifd78642a27c76f07a1c94aabb2b7963d84efc053
Reviewed-on: http://review.couchbase.org/40029
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoAcquire vbucket lock on rollback operation with retry 19/40019/4
Sundar Sridharan [Tue, 29 Jul 2014 20:09:48 +0000 (13:09 -0700)]
Acquire vbucket lock on rollback operation with retry

rollback which writes a header to the vbucket file needs
synchronization with the flusher.

Change-Id: I05b2474e3bdcc5160c1b4e194ec2c2a2625533e3
Reviewed-on: http://review.couchbase.org/40019
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoFix to the unit test failure in vbucket deletion / recreation. 23/40023/3
Chiyoung Seo [Tue, 29 Jul 2014 20:37:15 +0000 (13:37 -0700)]
Fix to the unit test failure in vbucket deletion / recreation.

VBucket deletion shouldn't be rescheduled, but instead wait on
the lock for the serialization.

Change-Id: I7f678927a4689e1c301730bed18970fcaf6ae5ee
Reviewed-on: http://review.couchbase.org/40023
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoUse the read-write KV instance to get the last persisted seqno. 22/40022/3
Chiyoung Seo [Mon, 28 Jul 2014 23:02:43 +0000 (16:02 -0700)]
Use the read-write KV instance to get the last persisted seqno.

The read-write KV instance is always synchronized with the last
persisted seqno.

Change-Id: I312f5af31f247d9520621caa02f3467021dc8bd6
Reviewed-on: http://review.couchbase.org/40022
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11725: Place rollback task on the writer dispatcher 17/40017/2
Mike Wiederhold [Tue, 29 Jul 2014 19:42:45 +0000 (12:42 -0700)]
MB-11725: Place rollback task on the writer dispatcher

The rollback task writes a header on success and as a result we need
to make sure that this doesn't race with other writer tasks. We also
need to keep in mind that the cached vbucket states in couch kvstore
are only valid for the rw kvstore.

Change-Id: Ib1edefb50817ac8240165d93d35e5dc89f4db042
Reviewed-on: http://review.couchbase.org/40017
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agounit test fix (95) 18/40018/2
abhinavdangeti [Tue, 29 Jul 2014 19:53:37 +0000 (12:53 -0700)]
unit test fix (95)

Change-Id: I1f1d0e944f982410081ca4a78e389be586bec5d3
Reviewed-on: http://review.couchbase.org/40018
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11735: Rename total_free_bytes -> total_free_mapped_bytes; add total_free_unmapped... 74/39874/6
Dave Rigby [Fri, 25 Jul 2014 13:25:27 +0000 (14:25 +0100)]
MB-11735: Rename total_free_bytes -> total_free_mapped_bytes; add total_free_unmapped_bytes

Change-Id: I9efa357adbeff05d993c0602ad49877b3c0bce98
Reviewed-on: http://review.couchbase.org/39874
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
6 years agoMB-11629: Set engineSpecific to NULL when notify_failed_conns 82/39982/3
abhinavdangeti [Tue, 29 Jul 2014 00:54:56 +0000 (17:54 -0700)]
MB-11629: Set engineSpecific to NULL when notify_failed_conns

Context: While notifying pending connections failed

Change-Id: I5e4085794a70a068f7a95735242b1e841cc6cf8f
Reviewed-on: http://review.couchbase.org/39982
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11830 cached failover table even if empty 67/39967/3
Sundar Sridharan [Mon, 28 Jul 2014 22:05:29 +0000 (15:05 -0700)]
MB-11830 cached failover table even if empty

needed for cbtransfer which needs to parse table in json
even if it is empty

Change-Id: I47c909ce32f1b8e137220bb0f20e0964ad921ba8
Reviewed-on: http://review.couchbase.org/39967
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoMB-11799 re-use existing task for rescheduling vb snapshots 58/39958/5
Sundar Sridharan [Mon, 28 Jul 2014 21:54:38 +0000 (14:54 -0700)]
MB-11799 re-use existing task for rescheduling vb snapshots

we need not create new task on vb snapshot failure,
re-use the task by returning true from the run method.
Change-Id: I8f3b9988e2c55d95bf23cecad9653c87cdcebe0e
Reviewed-on: http://review.couchbase.org/39958
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoAdd vbucket-seqno stats to vbucket-details 68/39968/3
Mike Wiederhold [Mon, 28 Jul 2014 22:19:37 +0000 (15:19 -0700)]
Add vbucket-seqno stats to vbucket-details

We should include these stats in the vbucket details stats as well
since vbucket-seqno is really only mean to be used internally for
clients that only want to get the bare minimum needed to make dcp
connections.

Change-Id: Ib95b80f8ec7afb37b146d6932e7c722da9d917c4
Reviewed-on: http://review.couchbase.org/39968
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11827: Check the checkpoint id instead of backfill state 60/39960/2
Mike Wiederhold [Mon, 28 Jul 2014 19:09:49 +0000 (12:09 -0700)]
MB-11827: Check the checkpoint id instead of backfill state

I've noticed that in the past these values haven't always been
kept in sync and that it is much safer to just check the checkpoint
id.

Change-Id: Icf51588d4ec411bc507bea8cff7505b6642e8c0b
Reviewed-on: http://review.couchbase.org/39960
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11725: Add an assert to make sure that the disk seqno <= mem seqno 59/39959/2
Mike Wiederhold [Fri, 25 Jul 2014 22:50:08 +0000 (15:50 -0700)]
MB-11725: Add an assert to make sure that the disk seqno <= mem seqno

This assert will cause a crash if we try to set the disk seqno
greater than the memory seqno. It will allow use to better debug the
issue where backfilling doesn't read any items. It is also a condition
that if not handled properly can result in data loss.

Change-Id: Ic5935649af4a837a21188013fe02a4dde2b7fb61
Reviewed-on: http://review.couchbase.org/39959
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11731 Reduce the lock overhead among flusher, VB deletion, and compaction 10/39910/3
Chiyoung Seo [Sat, 26 Jul 2014 00:55:15 +0000 (17:55 -0700)]
MB-11731 Reduce the lock overhead among flusher, VB deletion, and compaction

This change reduces the lock blocking overhead among flusher, vbucket
deletion, compaction, and snapshot tasks, by skipping the rest of
tasks if a lock is already grabbed by one task.

Change-Id: I643d7fe74150b5ead5c1368d660deb3c64dd1d01
Reviewed-on: http://review.couchbase.org/39910
Reviewed-by: David Liao <david.liao@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11731 Add the daemon vbucket state snapshot task. 07/39907/3
Chiyoung Seo [Sat, 26 Jul 2014 00:00:45 +0000 (17:00 -0700)]
MB-11731 Add the daemon vbucket state snapshot task.

This new daemon task will be scheduled periodically and persist
any vbucket state changes with a lower priority.

Change-Id: I2aa7dc5dda0153aac74314e239a85ad3f444c4d8
Reviewed-on: http://review.couchbase.org/39907
Reviewed-by: David Liao <david.liao@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11731 Replace the shard-level vbstate snapshot with individual tasks. 06/39906/3
Chiyoung Seo [Fri, 25 Jul 2014 23:03:18 +0000 (16:03 -0700)]
MB-11731 Replace the shard-level vbstate snapshot with individual tasks.

As we replace a shard-level lock with a vbucket-level lock to synchronize
the flusher, vbucket deletion, vbucket compaction, vbucket state
persistence tasks, this changes also replaces the shard-level vbstate
snapshot task with individual vbstate persistence tasks.

Change-Id: Ief7138a4b63d388cd786b966f68fcd8ab6ce2722
Reviewed-on: http://review.couchbase.org/39906
Reviewed-by: David Liao <david.liao@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoRemove unused snapshotVBState flag variable. 05/39905/3
Chiyoung Seo [Fri, 25 Jul 2014 20:24:08 +0000 (13:24 -0700)]
Remove unused snapshotVBState flag variable.

Change-Id: I1f8e471dc91ac8b3a6f027b518d587567e7a650a
Reviewed-on: http://review.couchbase.org/39905
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoExpand stats `allocator` buffer size 73/39873/3
Dave Rigby [Thu, 17 Jul 2014 17:39:24 +0000 (18:39 +0100)]
Expand stats `allocator` buffer size

jemalloc prints longer detailed stats than tcmalloc, so increase the
buffer size to match.

Change-Id: Ic40fbac3cb06ed459eaa7140dce39aac81492625
Reviewed-on: http://review.couchbase.org/39873
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11786: Always create a new checkpoint after backfill phase 96/39896/4
Mike Wiederhold [Fri, 25 Jul 2014 20:43:57 +0000 (13:43 -0700)]
MB-11786: Always create a new checkpoint after backfill phase

Even if we don't get a chk flag in the snapshot marker we still
need to create a new checkpoint if we were previously in backfill
phase.

Change-Id: I147273d23950afd8fdfea98346760075862f39d9
Reviewed-on: http://review.couchbase.org/39896
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11809: Don't use last_by_seqno when checking for sending marker ack 94/39894/4
Mike Wiederhold [Fri, 25 Jul 2014 20:03:08 +0000 (13:03 -0700)]
MB-11809: Don't use last_by_seqno when checking for sending marker ack

The last_by_seqno variable is set when an item is received and
buffered. As a result it is possible that the last_by_seqno might
be different than the mutation seqno we are surrenty processing. This
can lead to no sending an ack when we should.

Change-Id: I0fe43e60b51d2050ab5c8bea016622a7ac0affc7
Reviewed-on: http://review.couchbase.org/39894
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11822 correct bucket low priority to be <= 5 91/39891/2
Sundar Sridharan [Fri, 25 Jul 2014 19:16:50 +0000 (12:16 -0700)]
MB-11822 correct bucket low priority to be <= 5

This is so that it conforms to the expectations in the 3.0
upgrade documentation.

Change-Id: I020dc1a8af097362e596c5808cad623e1cf05ab3
Reviewed-on: http://review.couchbase.org/39891
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoNo need to pass the vbucket object to vb_delete_task 62/39862/3
abhinavdangeti [Fri, 25 Jul 2014 03:00:23 +0000 (20:00 -0700)]
No need to pass the vbucket object to vb_delete_task

Change-Id: I0ca93f00f5d11fac435e63b6ad7f95549f170c17
Reviewed-on: http://review.couchbase.org/39862
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoDecrement session counter if compaction fails 61/39861/3
abhinavdangeti [Fri, 25 Jul 2014 03:04:13 +0000 (20:04 -0700)]
Decrement session counter if compaction fails

Change-Id: Ia4379958e262161f2611097833b8de03b9494a8d
Reviewed-on: http://review.couchbase.org/39861
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>