ep-engine.git
5 years agoMB-13386: Ensure that purging the highSeqno doesn't happen 41/47241/5 3.0.1
abhinavdangeti [Tue, 24 Feb 2015 01:04:19 +0000 (17:04 -0800)]
MB-13386: Ensure that purging the highSeqno doesn't happen

+ The highest seqno is to not be purged for DCP.
+ This change ensures that items are still queued
for deletion if found to be expired.

Change-Id: I8102d5f61989523efc4f39b70b225c05cdd1b128
Reviewed-on: http://review.couchbase.org/47241
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
5 years agoMB-13386: Do not purge the item with highest sequence number from db. 35/47035/2
Manu Dhundi [Wed, 18 Feb 2015 23:07:18 +0000 (15:07 -0800)]
MB-13386: Do not purge the item with highest sequence number from db.

When the highest seq number that DCP is supposed to read from
the db is the last seq number in the db and is purged,
DCP backfill does not know about it and hence waits for it.
This results in DCP connection hang.
To solve this problem, during compaction we do not purge the
item with last(highest) sequence number in the db.

Change-Id: Ib83c335da3f7c0a952e4b760309276f73bff4ccf
Reviewed-on: http://review.couchbase.org/47035
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Manu Dhundi <manu@couchbase.com>
5 years ago[Backport] Disable the disk write queue size cap for replication backoff. 14/46614/3
Chiyoung Seo [Wed, 4 Feb 2015 06:44:34 +0000 (22:44 -0800)]
[Backport] Disable the disk write queue size cap for replication backoff.

The disk write queue size cap for replication backoff was determined in
the very early versions of Couchbase Server that were deployed in
small-to-medium sized cluster with spining disks.

In our recent benchmark results, it turns out that disabling or setting
the disk write queue cap to a large value shows much better performance
behaviors in large-scale clusters with SSDs. As Couchbase Server is
deployed in such environments more and more, this change disables the
write queue size cap, but still supports a replication backoff by
checking the memory usage.

Note that the disk write queue size cap is still configurable at runtime.

Change-Id: Iedf711ad1c3bef61ca954f83f802b4a647b9ec88
Reviewed-on: http://review.couchbase.org/46367
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-on: http://review.couchbase.org/46614
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
5 years ago[Backport] MB-13286: New DcpProducer to have paused status set to true 11/46511/2
abhinavdangeti [Wed, 4 Feb 2015 19:15:52 +0000 (11:15 -0800)]
[Backport] MB-13286: New DcpProducer to have paused status set to true

DcpOpen will create a new DcpProducer and if one with a
similar name already exists (e.g - during rebalance),
this new producer will replace the older one, and the
paused status will need to be set to true, to let the
notification for the new connection to be sent to
memcached.

Change-Id: I40c22601a7d29141741608339c58caa486a698b2
Reviewed-on: http://review.couchbase.org/46374
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-on: http://review.couchbase.org/46511

6 years agoMB-12673: Check for new checkpoint items after takeoverSend 18/43918/2 v3.0.2
abhinavdangeti [Wed, 3 Dec 2014 03:35:25 +0000 (19:35 -0800)]
MB-12673: Check for new checkpoint items after takeoverSend

Check for any new checkpoint items when the readyQ is found to
be empty in the takeoverSendPhase, before setting the old
vbucket to dead and the new vbucket to active.

Change-Id: I2a6ddacc711f5db42a1e3c575ae18d0b2b3126bd
Reviewed-on: http://review.couchbase.org/43918
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-12751: Return KEY EEXISTS for locked objects 25/43625/2
Trond Norbye [Mon, 24 Nov 2014 21:57:29 +0000 (22:57 +0100)]
MB-12751: Return KEY EEXISTS for locked objects

Change-Id: I5eb8a24337a81fed861cc8140bbdcf1a132acd90
Reviewed-on: http://review.couchbase.org/43625
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
6 years agoMB-12647: Ensure CAS value will always be unique 71/43271/3
Sriram Ganesan [Fri, 14 Nov 2014 22:45:58 +0000 (14:45 -0800)]
MB-12647: Ensure CAS value will always be unique

In windows, the gethrtime() returns the same timestamp when 2 requests
are made in a very short interval, thus causing the same CAS to return.
This can race conditions with 2 consecutive requests, resulting in
data corruption.

Change-Id: I4f396de3f14129504ca406ebb8d4c7a9f3a89bd8
Reviewed-on: http://review.couchbase.org/43271
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-12576: Limit writer threads to 4 60/43060/7
Sundar Sridharan [Tue, 11 Nov 2014 06:59:44 +0000 (22:59 -0800)]
MB-12576: Limit writer threads to 4

Having more than 4 writers increases bgfetch latencies in DGM
This change selectively reverts 32a166c511d7b242433011a875402e1278300add
Change-Id: Icdb996622237747e759c52751f2c8e613c9ba262
Reviewed-on: http://review.couchbase.org/43060
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-12562: Avoid terminating all writer threads 80/42880/6
abhinavdangeti [Thu, 6 Nov 2014 05:10:29 +0000 (21:10 -0800)]
MB-12562: Avoid terminating all writer threads

With incremental writer thread scheduling, we
need to make sure that all writer threads aren't
deleted before deleting all buckets.

Scenario
- maxWriters is 4
- create bucket 1, numWriters=1
- create bucket 2, numWriters=2
- create bucket 3, numWriters=3
- create bucket 4, numWriters=4
- create bucket 5, numWriters=4
- delete bucket 5, numWriters=3 =>with change, 4
- delete bucket 4, numWriters=2 =>with change, 3
- delete bucket 3, numWriters=1 =>with change, 2
- delete bucket 2, numWriters=0 =>with change, 1
- delete bucket 1, numWriters=0 =>with change, 0

Change-Id: Ib9d45f7acb9177924612547538aa74ca9dd49c20
Reviewed-on: http://review.couchbase.org/42880
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-12551: Add all arguments to noop log message 62/42862/2
Mike Wiederhold [Thu, 6 Nov 2014 00:58:19 +0000 (16:58 -0800)]
MB-12551: Add all arguments to noop log message

Change-Id: Id6e8aac2bf9e398dc8c929eb122651884d27b7c1
Reviewed-on: http://review.couchbase.org/42862
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12483: Make sure vbucket_state initializes all variables 29/42729/3
Mike Wiederhold [Mon, 3 Nov 2014 22:00:15 +0000 (14:00 -0800)]
MB-12483: Make sure vbucket_state initializes all variables

We need to make sure that this data is always completely filled
out so that we don't accidentally write garbage data to disk.

Change-Id: I196e7ca9f5bada8e0df90ddb01b6e952650bed56
Reviewed-on: http://review.couchbase.org/42729
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12483: Remove the default constructor in the vbucket_state struct 28/42728/2
Mike Wiederhold [Mon, 3 Nov 2014 20:19:04 +0000 (12:19 -0800)]
MB-12483: Remove the default constructor in the vbucket_state struct

The default constructor can leave uninitialized fields and we persist
this structure to disk. In oreder to prevent garbage from being written
we should remove the default constructor.

Change-Id: I028fac1dc112bb454779a30f695eb180278455df
Reviewed-on: http://review.couchbase.org/42728
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12483: Don't allow recreation during vbucket deletion 77/42677/2
Mike Wiederhold [Fri, 31 Oct 2014 18:44:12 +0000 (11:44 -0700)]
MB-12483: Don't allow recreation during vbucket deletion

When we delete a vbucket we have an option to recreate the file
immediately. Doing this is incorrect because we will not know
what the failover log of the new vbucket looks like until we
actually create it in memory. This can lead to a situation where
there is no failover log and as a result the local doc json will
be invalid. If the server is shutdown right after this happens and
then is restarted the vbuckets might be created with garbage values
in some of their fields.

Change-Id: I70e6335af68746aeac49a336da5e33b70dfcfe0e
Reviewed-on: http://review.couchbase.org/42677
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12279: Incrementally spawn writer threads for buckets 52/42552/3
Sundar Sridharan [Tue, 28 Oct 2014 23:02:16 +0000 (16:02 -0700)]
MB-12279: Incrementally spawn writer threads for buckets

This change is needed to mitigate the high bgfetch latency
observed in heavy Data-Greater-than-Memory scenarios because
having high number of writing threads slows down disk read
performance.
Also mitigates MB-11143 slowdown in single HDD case

NOTE: This may slow down disk persistence in fast SSDs by default
NOTE: cbepctl can still be used to dynamically tune writers at runtime

Change-Id: Iddf0d3094f38b66ba8c0e09d6d6a307d15b38d56
Reviewed-on: http://review.couchbase.org/42552
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-12305: Don't update the snap start/end seqno in tap mutation 03/42503/3
Mike Wiederhold [Thu, 23 Oct 2014 17:57:48 +0000 (10:57 -0700)]
MB-12305: Don't update the snap start/end seqno in tap mutation

We do this in the queueDirty function so this code is not needed. It
also appeared to cause a race in updating the snapshot start and end
sequence numbers so removing it should solve the bug linked above.

Change-Id: Ia8fa36df958be9147ea208ba9ebd78496048ebb4
Reviewed-on: http://review.couchbase.org/42503
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12398: log incorrect timeout values 87/42387/2
Trond Norbye [Thu, 23 Oct 2014 12:31:12 +0000 (14:31 +0200)]
MB-12398: log incorrect timeout values

The loglevel DEBUG won't appear in any logs, so customers using
this in production won't get the information anywhere (the server
just change their requested lock time without telling the user).

I guess a better behavior here would be to return ERANGE in
this case, but that would potentially break user applications
and isn't something we should do in a patch revision.

Change-Id: Ib6a37deceb2755b9fea53b0542a13fb2e16a3261
Reviewed-on: http://review.couchbase.org/42387
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoHide the -a argument in cbstats 97/42297/4
Mike Wiederhold [Mon, 20 Oct 2014 20:51:10 +0000 (13:51 -0700)]
Hide the -a argument in cbstats

The -a argument is used by cbcollectinfo so we can't just
get rid of it.

Change-Id: I0f04c2d454c117271b44176df904e90a938965e9
Reviewed-on: http://review.couchbase.org/42297
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12226: Handle ENOENT error during unlink 73/42173/3
Sriram Ganesan [Wed, 15 Oct 2014 21:12:06 +0000 (14:12 -0700)]
MB-12226: Handle ENOENT error during unlink

If the file is not found during an unlink, it shouldn't be added
to the pending file deletions queue.

Change-Id: Ief306277bbbc946ae18e39dd4819f811f12ea76c
Reviewed-on: http://review.couchbase.org/42173
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sriram Ganesan <sriram@couchbase.com>
6 years agoMB-12271: Set the default dcp producer noop interval to 20 seconds 36/42136/4
Mike Wiederhold [Tue, 14 Oct 2014 18:10:21 +0000 (11:10 -0700)]
MB-12271: Set the default dcp producer noop interval to 20 seconds

This is a fix to support backwards compatibility between 3.0 and
3.0.1+ versions of Couchbase. The problem is that 3.0 has a noop
interval of 20 seconds by default and 3.0.1 has a default noop
interval of 200 seconds. In 3.0.1 the consumer explicitly sets
the noop interval of the producer so in 3.0.1+ clusters the 20
second default will be overriden by the consumer, but in the
scenario when a 3.0.1+ node connects to a 3.0 node the 20 second
default will remain.

Change-Id: I2e18e9ad68037f3a82abe5167f2bca89f381f318
Reviewed-on: http://review.couchbase.org/42136
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11454: Add an option to output json in cbstats 84/42084/2
Mike Wiederhold [Mon, 13 Oct 2014 17:56:14 +0000 (10:56 -0700)]
MB-11454: Add an option to output json in cbstats

Change-Id: I0adcb16adde82ddf3671e4edc0118061d684f2d4
Reviewed-on: http://review.couchbase.org/42084
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12271: Set noop interval individually for dcp connections 49/42049/3
Mike Wiederhold [Sat, 11 Oct 2014 00:00:53 +0000 (17:00 -0700)]
MB-12271: Set noop interval individually for dcp connections

We need to do this because it might be the case that two servers
have their respective noop intervals set to different values. If
they are this can cause the connections to be disconnected because
each side is expecting to see a noop at different times.

Change-Id: I6ff475ccba407547e7285fa431b86ad9bf9cdc24
Reviewed-on: http://review.couchbase.org/42049
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoAccount for incorrect entries through cbepctl 83/42083/3
abhinavdangeti [Mon, 13 Oct 2014 17:44:14 +0000 (10:44 -0700)]
Account for incorrect entries through cbepctl

Context: access_scanner_enabled
Relates to: http://review.couchbase.org/#/c/40884/

Change-Id: I35134c91ed03f6ba6093cfd71270484beca1a4cd
Reviewed-on: http://review.couchbase.org/42083
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11642: Change the priority based on the type of dcp connection 58/41758/3
Mike Wiederhold [Tue, 23 Sep 2014 23:06:48 +0000 (16:06 -0700)]
MB-11642: Change the priority based on the type of dcp connection

Change-Id: I1b9a6846879385308bee3920bfa182fc41e39b4f
Reviewed-on: http://review.couchbase.org/41758
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agouse optparse instead of arg_parse cbvdiff for compatibility 52/41752/4
Sundar Sridharan [Mon, 29 Sep 2014 21:32:52 +0000 (14:32 -0700)]
use optparse instead of arg_parse cbvdiff for compatibility

Change-Id: I604354823c71b91a167646c01e2bb6ea9d8c8822
Reviewed-on: http://review.couchbase.org/41752
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-12268: Adding a debug log, when clusterConfig is updated 46/41746/2
abhinavdangeti [Mon, 29 Sep 2014 18:38:23 +0000 (11:38 -0700)]
MB-12268: Adding a debug log, when clusterConfig is updated

Change-Id: I0cbb70789f84bddc644bdc4ee59c1e86e1821220
Reviewed-on: http://review.couchbase.org/41746
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-12160: Use appropriate CAS for comparison with locked items 94/41594/8
Sriram Ganesan [Tue, 23 Sep 2014 23:27:16 +0000 (16:27 -0700)]
MB-12160: Use appropriate CAS for comparison with locked items

In the case of setWithMeta/deleteWithMeta commands, the locked item's
CAS value needs to be compared with the incoming mutation's CAS.

Change-Id: Id12a3c4717b18bc41c3d4b7dded99ea215179e9d
Reviewed-on: http://review.couchbase.org/41594
Tested-by: Sriram Ganesan <sriram@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoAccount for keylength while allocating buffer for allKeys 95/41595/3
abhinavdangeti [Wed, 24 Sep 2014 00:17:21 +0000 (17:17 -0700)]
Account for keylength while allocating buffer for allKeys

While increasing the buffersize (when needed) when the
AllKeysAPI is invoked, we will need to account for two
additional bytes while allocating the key, for its length.

Change-Id: Iba68c4ae7bccae20f97d4e98350d5105093c3487
Reviewed-on: http://review.couchbase.org/41595
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-12117: Release hashtable locks before disk IO 72/41572/4
abhinavdangeti [Wed, 24 Sep 2014 18:29:10 +0000 (11:29 -0700)]
MB-12117: Release hashtable locks before disk IO

While in access log generation, we'll need to release
all hashtable partition locks before we create new
entries in the mutation log.

Change-Id: Ic3dd0a02452b51ee742e30a0f268b86f9ab6205b
Reviewed-on: http://review.couchbase.org/41572
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11999: Preferential loading of active/replica vbuckets 68/41468/3
abhinavdangeti [Thu, 18 Sep 2014 22:44:32 +0000 (15:44 -0700)]
MB-11999: Preferential loading of active/replica vbuckets

During warmup, order the vbucket list such that
active vbuckets get 60% preference while replica
vbuckets get 40% preference.

Example:
In a 4 node cluster (DGM), approx. resident ratios:
1. Before warmup
    active:     36%     34%     34%     41%
    replica:    33%     35%     35%     27%
2. After warmup
    active:     40%     42%     41%     46%
    replica:    31%     29%     29%     24%

Change-Id: I60e0427bca58530247086d730135ebb4be70bb84
Reviewed-on: http://review.couchbase.org/41468
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMerge remote-tracking branch 'gerrit/2.5.0' into 3.0.1 60/41460/2
Mike Wiederhold [Wed, 17 Sep 2014 19:32:20 +0000 (12:32 -0700)]
Merge remote-tracking branch 'gerrit/2.5.0' into 3.0.1

Change-Id: I59c45877fc783e30c86dcfd7d303c731cde433fc

6 years agoFix deadlock in checkpoint persistence command 63/41363/2
Mike Wiederhold [Thu, 11 Sep 2014 18:46:00 +0000 (11:46 -0700)]
Fix deadlock in checkpoint persistence command

We need to release the hpChkMutex before notifying memcached in
order to prevent a deadlock. The deadlock occurs when the flusher
trys to notify memcached of checkpoint persistence at the same
time a memcached worker thread is trying to add a new checkpoint
persistence request.

Change-Id: Ida313f5b39ef0e063dee9882410cd0a19ce55292
Reviewed-on: http://review.couchbase.org/41363
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMerge remote-tracking branch 'gerrit/3.0' into 3.0.1 18/41318/1
Mike Wiederhold [Wed, 10 Sep 2014 18:34:38 +0000 (11:34 -0700)]
Merge remote-tracking branch 'gerrit/3.0' into 3.0.1

Change-Id: I195452c3d0684dff5ff673efd00a18b9d70da3cf

6 years agoMB-12137: Don't update the current snapshot during persistence 89/41289/2 3.0 v3.0.0
Mike Wiederhold [Tue, 9 Sep 2014 21:31:07 +0000 (14:31 -0700)]
MB-12137: Don't update the current snapshot during persistence

If we update the current snapshot only when we persist items to
disk then we may miss out on updates to the current snapshot
when there are vbucket state changes.

Change-Id: Iea3139afb669bdf32e6b4d98e8b3515cafe578dc
Reviewed-on: http://review.couchbase.org/41289
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoFix build breakage on Linux 91/41291/2
Sriram Ganesan [Tue, 9 Sep 2014 23:25:11 +0000 (16:25 -0700)]
Fix build breakage on Linux

Remove usage of constant strings for maintaining file deletion queue

Change-Id: I1992343ff65f923b6d3ffbf7a25932b936e3ff7c
Reviewed-on: http://review.couchbase.org/41291
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Sriram Ganesan <sriram@couchbase.com>
6 years agoHandle unlink/remove failures 68/41268/5
Sriram Ganesan [Fri, 5 Sep 2014 23:01:46 +0000 (16:01 -0700)]
Handle unlink/remove failures

The unlink/remove function can fail if there is another process that
has an open file handle on that file. In such cases, we need to retry
the unlink periodically in the flusher task until we get rid of those
files.

Change-Id: I4bfcf29b3fa866ec4946db658a245c722f3725ce
Reviewed-on: http://review.couchbase.org/41268
Tested-by: Sriram Ganesan <sriram@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoremove warmup wait dead code to prevent accidental deadlock 06/41206/2
Sundar Sridharan [Thu, 4 Sep 2014 21:33:03 +0000 (14:33 -0700)]
remove warmup wait dead code to prevent accidental deadlock

Since we should never have a front-end thread wait for background
thread, having a warmup wait logic will ensure that we do not use
it accidentally in the future and cause a deadlock

Change-Id: I28ec96dceaffa02810415402226df20f8eb44cc7
Reviewed-on: http://review.couchbase.org/41206
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoMB-12063 Don't delete expired items during warmup. 19/41219/3
Chiyoung Seo [Fri, 5 Sep 2014 06:15:38 +0000 (23:15 -0700)]
MB-12063 Don't delete expired items during warmup.

When a node is added back to the cluster, its vbucket states
can be different from its previous states (e.g., active -> replica).
Therefore, we should not remove expired items from active vbuckets
during warmup because the states of those vbuckets can be changed
after the warmup.

Change-Id: Ifec4fda80e424a8afa74bae66e17de77b2e0c36b
Reviewed-on: http://review.couchbase.org/41219
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMerge remote-tracking branch 'gerrit/3.0' into 3.0.1 96/41196/2
Mike Wiederhold [Thu, 4 Sep 2014 17:56:29 +0000 (10:56 -0700)]
Merge remote-tracking branch 'gerrit/3.0' into 3.0.1

Change-Id: I678740458ccc13cbf79be847369cf6561b5b678b

6 years agoMB-12100: Don't expire items for replica vbuckets during warmup 77/41177/2
Mike Wiederhold [Wed, 3 Sep 2014 22:16:31 +0000 (15:16 -0700)]
MB-12100: Don't expire items for replica vbuckets during warmup

Change-Id: I81095beb69c0996d819158e3cadc451a83653d2f
Reviewed-on: http://review.couchbase.org/41177
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoUse the Dcp noop interval for dcp connections 94/40994/2
Mike Wiederhold [Wed, 27 Aug 2014 17:43:03 +0000 (10:43 -0700)]
Use the Dcp noop interval for dcp connections

We were accidentally using the tap noop interval.

Change-Id: Ib7d6615d3128095e8f29a12afa1922961f5aa197
Reviewed-on: http://review.couchbase.org/40994
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoRename tapconnmap to connmap since it handles both dcp and tap 93/40993/2
Mike Wiederhold [Wed, 27 Aug 2014 17:41:49 +0000 (10:41 -0700)]
Rename tapconnmap to connmap since it handles both dcp and tap

Change-Id: I698a2bfe62ab4dec5e64934de845611d49860041
Reviewed-on: http://review.couchbase.org/40993
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoFix build breakage 12/41012/2
Mike Wiederhold [Thu, 28 Aug 2014 07:05:45 +0000 (00:05 -0700)]
Fix build breakage

Change-Id: Ibeac0d57fc1ea3bc94bf488ee5e82cf7e9143d3a
Reviewed-on: http://review.couchbase.org/41012
Reviewed-by: Chris Hillery <ceej@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12063: Properly save the max deleted seqno during vb snapshots 97/40997/3
Mike Wiederhold [Wed, 27 Aug 2014 22:47:51 +0000 (15:47 -0700)]
MB-12063: Properly save the max deleted seqno during vb snapshots

Change-Id: Ife9e3b26956638622de0427cd573cd4d37c7e443
Reviewed-on: http://review.couchbase.org/40997
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoAvoid getting config lock on every unlinkCouchFile 04/41004/2
abhinavdangeti [Thu, 28 Aug 2014 00:18:50 +0000 (17:18 -0700)]
Avoid getting config lock on every unlinkCouchFile

Change-Id: I7a5c3cf81a227ee2ddc02750839cc3b94359fe88
Reviewed-on: http://review.couchbase.org/41004
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-12063: Lock the current snapshot mutex when flushing items 96/40996/2
Mike Wiederhold [Wed, 27 Aug 2014 21:41:08 +0000 (14:41 -0700)]
MB-12063: Lock the current snapshot mutex when flushing items

We need to lock the current snapshot mutex before grabbing items
because it is possible that an itme might be queued, but the
current snapshot might be updated.

Change-Id: Ieff85a4605041e764d6facf055c71c59e7e19955
Reviewed-on: http://review.couchbase.org/40996
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoFix build break on win32 81/40981/2
Trond Norbye [Wed, 27 Aug 2014 18:05:08 +0000 (20:05 +0200)]
Fix build break on win32

Change-Id: Iab0f0cfa7fb61b84aa54f20513f80d4683775073
Reviewed-on: http://review.couchbase.org/40981
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoRemove shutdown check 58/40958/2
Mike Wiederhold [Mon, 4 Aug 2014 18:19:59 +0000 (11:19 -0700)]
Remove shutdown check

We only needed this because we didn't want to notify mccouch.

Change-Id: I63ad9cddbe86930facfd68ceaa3ee87998ba025d
Reviewed-on: http://review.couchbase.org/40958
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoRename all classes/variables with upr to dcp 42/40942/3
Mike Wiederhold [Tue, 26 Aug 2014 21:29:16 +0000 (14:29 -0700)]
Rename all classes/variables with upr to dcp

Change-Id: Iaf644f5034fdb30a14c400ec55def22d8bb0ba87
Reviewed-on: http://review.couchbase.org/40942
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoReplace upr with dcp in all file names 41/40941/3
Mike Wiederhold [Tue, 26 Aug 2014 20:10:12 +0000 (13:10 -0700)]
Replace upr with dcp in all file names

Change-Id: Ia418e96b5bcebf40ef99e3c27ce3fb8f20f68f0e
Reviewed-on: http://review.couchbase.org/40941
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11722: Remove the mcCouch dependency 95/40195/10
abhinavdangeti [Tue, 26 Aug 2014 22:22:44 +0000 (15:22 -0700)]
MB-11722: Remove the mcCouch dependency

Now,
- unlinking stale files at the end of compaction
- unlinking files in couchkvstore reset
- unlink files in delVbucket

Change-Id: I000fbd076856b8de1a0dcb82aa673b3316dc745f
Reviewed-on: http://review.couchbase.org/40195
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12057: warmup must not block bucket creation 92/40892/8
Sundar Sridharan [Tue, 26 Aug 2014 21:23:49 +0000 (14:23 -0700)]
MB-12057: warmup must not block bucket creation

Bucket creation is a front end thread and warmup is done by
engine background threads. So if bucket creation waits for
warmup then a deadlock happens when there are few reader threads
Test Infrastructure expects vb0 created by ep-engine init
or warmup. This is not necessary for production, so moving
vb0 creation to test setup. Minor test code bug fixes.
Change-Id: I5eae9c8f36fe39d2a18cb1800dc48a29162eab28
Reviewed-on: http://review.couchbase.org/40892
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMerge remote-tracking branch 'gerrit/3.0' into HEAD 36/40936/2
Mike Wiederhold [Tue, 26 Aug 2014 19:51:51 +0000 (12:51 -0700)]
Merge remote-tracking branch 'gerrit/3.0' into HEAD

Change-Id: Ief314ec953a5847ce4a3f555e9055cae6e2a504d

6 years agoMB-12041: Enable/disable access scanner through cbepctl 84/40884/6
abhinavdangeti [Mon, 25 Aug 2014 20:10:14 +0000 (13:10 -0700)]
MB-12041: Enable/disable access scanner through cbepctl

Enable/disble this way:
./cbepctl localhost:11210 set_flush_param access_scanner_enabled true
./cbepctl localhost:11210 set_flush_param access_scanner_enabled false

access_scanner_task_time will show NOT_SCHEDULED if
access scanner task were disabled.

Change-Id: Ia89c09d55764de38f986db36a4467730534c412c
Reviewed-on: http://review.couchbase.org/40884
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11948: Handle GetLastError() returning 0 condition 65/40865/6
Sriram Ganesan [Fri, 22 Aug 2014 22:41:57 +0000 (15:41 -0700)]
MB-11948: Handle GetLastError() returning 0 condition

In windows, the GetLastError() call sometimes returns 0 instead of
the returning WSAEWOULDBLOCK. Either the error gets reset before
the invocation or it could be undefined behavior.

Change-Id: Ifb8f7d576b447d090b21cc0747d068ead0597182
Reviewed-on: http://review.couchbase.org/40865
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Sriram Ganesan <sriram@couchbase.com>
6 years agoMinor fix which helps Xcode generated project build. 25/40825/3
jim [Fri, 22 Aug 2014 08:21:09 +0000 (09:21 +0100)]
Minor fix which helps Xcode generated project build.

Issue is that the add_custom_command references gencode/genconfig via
a path, yet on Xcode build all output goes into
build/ep-engine/Debug|Release/ thus gencode/genconfig fail dependency
checks.

Using just the add_excutable name of gencode/genconfig allows cmake
to use the correct path.

Change-Id: Ie55124487bffe80bc2847b67977738e67d8da855
Reviewed-on: http://review.couchbase.org/40825
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
6 years agoMB-12042 check for existence of vbucket in warmup callback 49/40849/4
Sundar Sridharan [Fri, 22 Aug 2014 20:38:41 +0000 (13:38 -0700)]
MB-12042 check for existence of vbucket in warmup callback

Change-Id: Iab0ad237392d681f28b91c29dc51b7631141e187
Reviewed-on: http://review.couchbase.org/40849
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoMB-11973: Only grab items from the chk manager one chk at a time 65/40765/4
Mike Wiederhold [Wed, 20 Aug 2014 20:36:31 +0000 (13:36 -0700)]
MB-11973: Only grab items from the chk manager one chk at a time

We should only grab a single checkpoint at a time from the checkpoint
manager in order to make sure that the replica vbuckets cannot move
into the open checkpoint. This will also help make sure we don't
copy too make items into an upr stream.

Change-Id: I519353df69cf389e6f32f77ffc6afe97dfba7b36
Reviewed-on: http://review.couchbase.org/40765
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11970: Resident ratio should be 100% when no items are a bucket 56/40756/2
Mike Wiederhold [Wed, 20 Aug 2014 17:50:53 +0000 (10:50 -0700)]
MB-11970: Resident ratio should be 100% when no items are a bucket

Change-Id: I0ef18509fecec56df9c922c44c81ffd5e02eae47
Reviewed-on: http://review.couchbase.org/40756
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-12032: Clean up TaskQs after last bucket deletion 41/40741/3
Sundar Sridharan [Wed, 20 Aug 2014 01:48:08 +0000 (18:48 -0700)]
MB-12032: Clean up TaskQs after last bucket deletion

Without this sometimes threads can end up sleeping on
a high priority Q even when none exist.
This can result in minor slowdown rarely in some platforms

Change-Id: I58cfe9234e17b9b5502f780c7e334574f64fb30d
Reviewed-on: http://review.couchbase.org/40741
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoMB-11980: Fix bugs caused from persisting the current snapshot 07/40707/4
Mike Wiederhold [Wed, 20 Aug 2014 01:35:58 +0000 (18:35 -0700)]
MB-11980: Fix bugs caused from persisting the current snapshot

There are two issues fixed here. One is around the current
snapshot not being updated properly during bucket flush. The
second is an issue caused by persisting the wrong information
for the high seqno, snap start seqno, and snap end seqno during
the snapshot vbuckets task. These values should remain the same
as they were after the last vbucket persistence task otherwise
we might persist incorrect values.

Change-Id: Ie4a357270b52bd35b5f29018f24fd3550913f1cc
Reviewed-on: http://review.couchbase.org/40707
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11934: Fix warmup for windows 73/40673/3
Sriram Ganesan [Fri, 15 Aug 2014 21:55:54 +0000 (14:55 -0700)]
MB-11934: Fix warmup for windows

During warmup in windows, the platform library appends a backslash instead
of a forward slash causing a database file name parsing to break.

Change-Id: I4b3d2296c83dcdf4f94698532a2115dedc841fdb
Reviewed-on: http://review.couchbase.org/40673
Tested-by: Sriram Ganesan <sriram@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11876 Safely cancel AccessScanner task on shutdown 69/40669/4
Sundar Sridharan [Sat, 16 Aug 2014 01:13:32 +0000 (18:13 -0700)]
MB-11876 Safely cancel AccessScanner task on shutdown

Change-Id: I769ca2ad562ef2b27447d40b6a66c4e24ff21eb3
Reviewed-on: http://review.couchbase.org/40669
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoCorrectly return ENGINE_ERROR_CODE in processing mutation messages 53/40653/2
Chiyoung Seo [Fri, 15 Aug 2014 17:25:26 +0000 (10:25 -0700)]
Correctly return ENGINE_ERROR_CODE in processing mutation messages

Change-Id: I72cadd9beb9a024be7fab5d54c7ecf8a85a40c79
Reviewed-on: http://review.couchbase.org/40653
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoFix the regression from the change for MB-11943 45/40645/4
Chiyoung Seo [Fri, 15 Aug 2014 05:05:05 +0000 (22:05 -0700)]
Fix the regression from the change for MB-11943

The change for MB-11943 caused the sporadic failure
in "vbucket deletion doesn't affect new data" unit test case.

This change makes sure that we have the same logic in maintaining
cachedVBState instance for a deleted vbucket.

Change-Id: I7fda66a258c46cd04820001e945e0679eed0469b
Reviewed-on: http://review.couchbase.org/40645
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoWindows don't support the %zu formatter 50/40650/4
Trond Norbye [Fri, 15 Aug 2014 15:42:51 +0000 (17:42 +0200)]
Windows don't support the %zu formatter

Change-Id: Ia36bc151f84af9cfd6c069c3d97c494753d449f3
Reviewed-on: http://review.couchbase.org/40650
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11958: Make sure mutations and snapshot ranges are updated together 42/40642/4
Mike Wiederhold [Fri, 15 Aug 2014 07:33:43 +0000 (00:33 -0700)]
MB-11958: Make sure mutations and snapshot ranges are updated together

There is a race condition where an items seqno is updated and the
flusher runs and persists both the seqno and old snapshot range
before the new snapshot range can be updated.

Change-Id: If124e5ab05c45c68a6956d28cc3898f8c18df6a9
Reviewed-on: http://review.couchbase.org/40642
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11890: Active vb shouldn't initiate add Stream req 41/40641/4
abhinavdangeti [Thu, 14 Aug 2014 23:12:37 +0000 (16:12 -0700)]
MB-11890: Active vb shouldn't initiate add Stream req

Change-Id: Ic902e632d52f426a73d7708f9c69854dc7fab874
Reviewed-on: http://review.couchbase.org/40641
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11916 Synchronize the access to thread job logs. 37/40637/2
Chiyoung Seo [Thu, 14 Aug 2014 21:14:05 +0000 (14:14 -0700)]
MB-11916 Synchronize the access to thread job logs.

Change-Id: I9551cd70b931b1ed92f806be1df0325d922a239c
Reviewed-on: http://review.couchbase.org/40637
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11945 Remove the assertion check on sequence number in the flusher. 30/40630/2
Chiyoung Seo [Thu, 14 Aug 2014 19:41:29 +0000 (12:41 -0700)]
MB-11945 Remove the assertion check on sequence number in the flusher.

We sporadically observe the sequence number mismatches between the
flusher and Couchstore. This is not a regression from any recent
changes, and mostly existed much earlier. We will investigate this
issue separately to see if there are any bugs in ep-engine or
Couchstore.

Change-Id: Ifac2eae0deae09cad09dd7b5664f63bcbf828b24
Reviewed-on: http://review.couchbase.org/40630
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11299: Don't let replica vbuckets send partial snapshots 46/40346/6
Mike Wiederhold [Wed, 6 Aug 2014 18:34:59 +0000 (11:34 -0700)]
MB-11299: Don't let replica vbuckets send partial snapshots

Replica vbuckets cannot create their own snapshots and must
rely on the snapshots it received from the active vbucket
when forwarding the snapshots to a downstream consumer. In
order to make sure that this happens properly we will only
stream mutations from the closed checkpoint for non-active
vbuckets. This will guarantee that the above requirement is
met.

Change-Id: I58151ce96868e34083f87dca0ce1d18b6a12aedf
Reviewed-on: http://review.couchbase.org/40346
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11943: replace map with vector for cachedVBStates 00/40600/6
Sundar Sridharan [Thu, 14 Aug 2014 05:39:29 +0000 (22:39 -0700)]
MB-11943: replace map with vector for cachedVBStates

std::map can incur thread-unsafe rb balance operations which
may invalidate concurrent iterators causing asserts or crashes
Pre-allocated vectors are thread-safe as long as the size is not
altered
Change-Id: I777b0cc7d9c421cc11425db75506fdf90044a93c
Reviewed-on: http://review.couchbase.org/40600
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11941: Use the latest seqno during tap mutation/deletion cur snapshot 01/40601/2
Mike Wiederhold [Thu, 14 Aug 2014 01:38:39 +0000 (18:38 -0700)]
MB-11941: Use the latest seqno during tap mutation/deletion cur snapshot

We generate the seqno when inserting the item into the hashtable when
using tap. As a reuslt we need to retrieve that seqno when updating
the latest snapshot because the byseqno passed into the mutation and
deletion functions for tap is always 0.

Change-Id: I21b7888207d9fea4f46b3d5918d1c8b60b2ea1ff
Reviewed-on: http://review.couchbase.org/40601
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11928: Fast path for rollback to 0 95/40595/3
Mike Wiederhold [Thu, 14 Aug 2014 00:25:29 +0000 (17:25 -0700)]
MB-11928: Fast path for rollback to 0

Right now we run through all of the headers even if we have to
rollback to 0. This change provides a fast path to rollback to
zero by immediately resetting the vbucket.

Change-Id: I8e7b62ec711838894c4eda92eaff78704bd94d50
Reviewed-on: http://review.couchbase.org/40595
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoIncorrect parameter for scheduleVBDeletion 93/40593/3
abhinavdangeti [Wed, 13 Aug 2014 23:46:38 +0000 (16:46 -0700)]
Incorrect parameter for scheduleVBDeletion

This caused completeBeforeShutdown to be set
to false, in resetVBucket/deleteVBucket
scenarios.

Change-Id: I72a1f07f475af49a7f368695a8e83cf2c5df9e28
Reviewed-on: http://review.couchbase.org/40593
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11933: Only set cur snapshot when receving the first snapshot item 83/40583/3
Mike Wiederhold [Wed, 13 Aug 2014 19:50:03 +0000 (12:50 -0700)]
MB-11933: Only set cur snapshot when receving the first snapshot item

We were setting the current snapshot when recieving the snapshot
marker and this is incorrect because if we disconnect before recieving
any items for that snapshot then the next time the replica vbucket
tries to connect to the active the high seqno will be less than the
last snapshot start seqno and this will cause the stream request to
fail with invalid arguments.

Change-Id: I75e1f555656c24221b2697011df878c579dce70e
Reviewed-on: http://review.couchbase.org/40583
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11941: Update current vbucket snapshot when receiving tap items 82/40582/2
Mike Wiederhold [Wed, 13 Aug 2014 19:35:39 +0000 (12:35 -0700)]
MB-11941: Update current vbucket snapshot when receiving tap items

We need to update the vbucket snapshot in order to support upgrades
because we initially have a tap stream and then move to a dcp stream.
Since tap doesn't support snapshots we can just set the current snapshot
to be the same as the seqno for that item.

Change-Id: I9da32df9366508276619507cf6cb9a66557e0919
Reviewed-on: http://review.couchbase.org/40582
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11952 Lower the default max number of readers. 79/40579/2
Chiyoung Seo [Wed, 13 Aug 2014 18:15:43 +0000 (11:15 -0700)]
MB-11952 Lower the default max number of readers.

Change-Id: I43dbcc4d606eea07276327ec7b9ed4f9bf3fc0e4
Reviewed-on: http://review.couchbase.org/40579
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11642 Batch load dcp mutations if replication 92/40392/10
Sundar Sridharan [Wed, 13 Aug 2014 00:40:21 +0000 (17:40 -0700)]
MB-11642 Batch load dcp mutations if replication

Instead of loading 1 mutation per vbucket send out 10
dcp mutations per dcp step() if it is a intra-cluster
replication stream.
This is done so that intra-cluster replication is always
faster than XDCR and views.
Noticeable  improvement seen in intra-cluster replication
Change-Id: I15fc0081852e8e11027a20febe3224acdf21b661
Reviewed-on: http://review.couchbase.org/40392
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoMB-11299: Allow rollbacks to a partial snapshot 40/40340/5
Mike Wiederhold [Wed, 6 Aug 2014 16:47:34 +0000 (09:47 -0700)]
MB-11299: Allow rollbacks to a partial snapshot

If we roll back to a partial snapshot we need to make sure to get
the snapshot start and end seqnos for the partial snapshot in order
to make sure that we don't accidentally miss any items when
reconnecting.

Change-Id: If95d0476d8d582d6b57786387212f03b2aafc7ba
Reviewed-on: http://review.couchbase.org/40340
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11912 Monitor workload pattern for dynamic compaction scheduling. 20/40520/5
Chiyoung Seo [Tue, 12 Aug 2014 02:22:46 +0000 (19:22 -0700)]
MB-11912 Monitor workload pattern for dynamic compaction scheduling.

This change allows us to monitor workload pattern (i.e.,
read_heavy, write_heavy, mixed) at runtime and use the monitored
workload pattern to adjust the number of concurrent compaction
tasks dynamically at runtime. If the workload pattern is read-heavy,
then only one compaction task is executed at a time.

Note that we plan to keep improving this area for better I/O
scheduling.

Change-Id: Ibbb27da0a7e9f3951571ad8068570ebbf9abb318
Reviewed-on: http://review.couchbase.org/40520
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoAdd id to the logs of checkpoint persistence 13/40513/5
Sriram Ganesan [Tue, 12 Aug 2014 01:27:44 +0000 (18:27 -0700)]
Add id to the logs of checkpoint persistence

Add the id to the logs of checkpoint persistence so that we will have the
sequence number/checkpoint id in case a timeout occurs on checkpoint
persistence

Change-Id: Ifd3c70c9d40de43f0b3f1fdc1cf495ee0dc95d60
Reviewed-on: http://review.couchbase.org/40513
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sriram Ganesan <sriram@couchbase.com>
6 years agoMB-11299: Use the snapshot start seqno when creating a failover entry 41/40341/3
Mike Wiederhold [Wed, 6 Aug 2014 17:03:33 +0000 (10:03 -0700)]
MB-11299: Use the snapshot start seqno when creating a failover entry

Change-Id: I7a2c0adf51a761a491d0f12720f2abef8127e5c3
Reviewed-on: http://review.couchbase.org/40341
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoMB-11887: Properly send the snapshot start and end seqnos 76/40276/10
Mike Wiederhold [Thu, 7 Aug 2014 18:58:30 +0000 (11:58 -0700)]
MB-11887: Properly send the snapshot start and end seqnos

Currently we always set the snapshot start and end seqno to be the
same as the start seqno. This is incorrect if we recently were
disconnected and need to resume from a partial snapshot.

Change-Id: I7c9430e0b24ef01cc427cb03b518f4c41a55d875
Reviewed-on: http://review.couchbase.org/40276
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
6 years agoAfter APPEND/PREPEND to json, set datatype to binary 90/40390/3
abhinavdangeti [Fri, 8 Aug 2014 17:53:15 +0000 (10:53 -0700)]
After APPEND/PREPEND to json, set datatype to binary

This is because appending or prepending anything
to a JSON document breaks the JSON structure.

Change-Id: I0acd99b0c43a54e0b34ce21f3a69c41709e3817e
Reviewed-on: http://review.couchbase.org/40390
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11741: Better scheduling of checkpointRemoverTask 55/40355/6
abhinavdangeti [Thu, 7 Aug 2014 02:48:24 +0000 (19:48 -0700)]
MB-11741: Better scheduling of checkpointRemoverTask

Waking up the checkpoint Remover Task once the cursors
complete parsing a checkpoint till the end, helps
prevent retaining blob instances for a longer time
than necessary.

Change-Id: Ieb1435dafac2e9b9984664218d6761c6c807333a
Reviewed-on: http://review.couchbase.org/40355
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agofix copy paste bug to set max non-io at runtime 17/40317/3
Sundar Sridharan [Wed, 6 Aug 2014 00:17:56 +0000 (17:17 -0700)]
fix copy paste bug to set max non-io at runtime

Change-Id: I097e5fc5d926387dff8874427d373e41e383f64e
Reviewed-on: http://review.couchbase.org/40317
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11872: Compile failure on (some?) Debian 7 systems 97/40297/3
Trond Norbye [Tue, 5 Aug 2014 12:22:05 +0000 (14:22 +0200)]
MB-11872: Compile failure on (some?) Debian 7 systems

Building from a Debian 7 setup under SmartOS works out of the box,
but one of our builders does not work for some reason. From looking
at the preprocessed file it doesn't define linux like our other
systems do. Extend the test to work for __linux__ and __linux as
well.

Change-Id: I4a7cf304ab8bfddb6cc117e9064165e1eb16b66e
Reviewed-on: http://review.couchbase.org/40297
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chris Hillery <ceej@couchbase.com>
6 years agoMB-11884: Consider min_item_threshold in full eviction 08/40308/4
abhinavdangeti [Tue, 5 Aug 2014 23:09:41 +0000 (16:09 -0700)]
MB-11884: Consider min_item_threshold in full eviction

In this case, as warmed up keys always matches warmed
up values, we can mark the min item threshold using
warmup's estimated item count.

Change-Id: Ibd1d58edd15be877c24ef794615fe0d56f9fc675
Reviewed-on: http://review.couchbase.org/40308
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11847: [Warmup] Estimated value count with full eviction 04/40304/4
abhinavdangeti [Tue, 5 Aug 2014 18:18:09 +0000 (11:18 -0700)]
MB-11847: [Warmup] Estimated value count with full eviction

In loadingKVPairs phase, set estimated value count equal
to the estimated item count, as no keys have been warmed
up yet, when in full eviction mode.

Change-Id: Id74237c96e0c144d9a9b94e0670f4975739e7914
Reviewed-on: http://review.couchbase.org/40304
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoHave at least 1 unit test with high priority bucket 13/40313/2
Sundar Sridharan [Tue, 5 Aug 2014 23:14:59 +0000 (16:14 -0700)]
Have at least 1 unit test with high priority bucket

Change-Id: Icc92c9455c13ca1af21eb285c72b8a8f3555cf6b
Reviewed-on: http://review.couchbase.org/40313
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
6 years agoMB-11876 Check for new priority buckets after wakeup 11/40311/3
Sundar Sridharan [Tue, 5 Aug 2014 21:49:19 +0000 (14:49 -0700)]
MB-11876 Check for new priority buckets after wakeup

If threads do not check for the presence of newly registered
buckets, then a rare race condition can happen where a thread
is unaware of the presence of a low priority bucket and keeps
continously checking the high priority Q which has no work.
This would block normal shutdown

Change-Id: Ic59ccbed69c205251e7897f349445d8982ec7be5
Reviewed-on: http://review.couchbase.org/40311
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years ago_unlink as opposed to unlink for windows 80/40280/2
abhinavdangeti [Mon, 4 Aug 2014 23:19:06 +0000 (16:19 -0700)]
_unlink as opposed to unlink for windows

Change-Id: I40a74c0942594cf48dfbbd3406dc935a4dbfbd35
Reviewed-on: http://review.couchbase.org/40280
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMB-11870 Avoid polling starvation in mixed priorities 42/40242/3
Sundar Sridharan [Sun, 3 Aug 2014 22:05:18 +0000 (15:05 -0700)]
MB-11870 Avoid polling starvation in mixed priorities

1) Wake up threads from the correct Q in mixed priorities.
2) If a thread has just woken up off a high priority Q
then make sure it polls the low priority Q next.
Not doing this causes starvation in the low prioirty Q
resuling in rebalance hangs.
3) Avoid sleeping threads for longer than 2 seconds to avoid
missing posts from other threads
Change-Id: I6cef6153285098e2f77639c50b194c9a9541dd24
Reviewed-on: http://review.couchbase.org/40242
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agodebug for MB-11870 39/40239/4
Sundar Sridharan [Sun, 3 Aug 2014 21:38:35 +0000 (14:38 -0700)]
debug for MB-11870

print engine name in dispatcher taskLog
move engine unregistration into WARNING
Change-Id: Ide1206642396f7013017f933a4b5b35cb741c19c
Reviewed-on: http://review.couchbase.org/40239
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11779: Switch the engine context before we update the task log 01/40201/3
Mike Wiederhold [Fri, 1 Aug 2014 23:43:17 +0000 (16:43 -0700)]
MB-11779: Switch the engine context before we update the task log

If we don't do this then we lose 48 bytes from the bucket mem_used
each time the executor thread runs.

Change-Id: Iecf216e3c262573a3b36911f80f44b92851d2a5f
Reviewed-on: http://review.couchbase.org/40201
Tested-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11847: Set estimatedValueCount in loadKVPairs phase 00/40200/2
abhinavdangeti [Fri, 1 Aug 2014 23:05:31 +0000 (16:05 -0700)]
MB-11847: Set estimatedValueCount in loadKVPairs phase

- Otherwise, ep_warmup_estimated_value count would show
"unknown" after warmup completes in some scenarios.
- Also remove unused functions.

Change-Id: I2e2fedf58dcd49a6f572f830c4566b0d8946033b
Reviewed-on: http://review.couchbase.org/40200
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
6 years agoMinor refactoring on checkpoint cursor registration function. 92/40192/2
Chiyoung Seo [Fri, 1 Aug 2014 18:46:07 +0000 (11:46 -0700)]
Minor refactoring on checkpoint cursor registration function.

Change-Id: I7fdffc233664e26a434e0c757e72416e567d1eb2
Reviewed-on: http://review.couchbase.org/40192
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-9131 Warmup should check for couchstore dbfile before open 78/40078/5
D Liao [Wed, 30 Jul 2014 23:44:54 +0000 (16:44 -0700)]
MB-9131 Warmup should check for couchstore dbfile before open

When creating dbFileRevMap during warmup, we now check for the
existence of couchstore db files so that we don't open non-existing
files so to avoid generating warning logs.

Change-Id: I8dd7b423f01240e8958b88fce2d58b429f0c9590
Reviewed-on: http://review.couchbase.org/40078
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoUse the metadata disk size stat for replica vbucket. 58/40158/2
Chiyoung Seo [Fri, 1 Aug 2014 03:04:03 +0000 (20:04 -0700)]
Use the metadata disk size stat for replica vbucket.

Change-Id: Ied887dd0ac0e3ac0ed309c554d27d95775c9874b
Reviewed-on: http://review.couchbase.org/40158
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
6 years agoMB-11864 Initialize max_workers variables before creating threads. 56/40156/2
Chiyoung Seo [Fri, 1 Aug 2014 01:50:22 +0000 (18:50 -0700)]
MB-11864 Initialize max_workers variables before creating threads.

Change-Id: Ie7853d8bd874117e976c74708172a36cf8c8e7c0
Reviewed-on: http://review.couchbase.org/40156
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>