ep-engine.git
4 years agoGeneralise cli_auth_utils 79/76779/3
James Harrison [Thu, 13 Apr 2017 15:19:33 +0000 (16:19 +0100)]
Generalise cli_auth_utils

Some minor modifications to allow arbitrary kwargs to be passed through
the cmd decorator - permits use in cbcompact

Change-Id: Ib506b29d1aa8b8cabc28ddad91024294632c0a63
Reviewed-on: http://review.couchbase.org/76779
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23906: Delete-with-value: Fix count of deleted items 20/76120/15
Dave Rigby [Fri, 31 Mar 2017 10:46:13 +0000 (11:46 +0100)]
MB-23906: Delete-with-value: Fix count of deleted items

Prior to Spock, the valid state transitions for StoredValues (relating
to deletion) were:

    /---\
    |   |
    |   V
    Alive --------> Deleted
          <--------

With the advent of Deleted Bodies for system XATTRs, Deleted SVs can
optionally have a value (Blob) associated with them, which can be
changed (or removed) without going back to Alive:

    /---\              /---\
    |   |              |   |
    \   V              \   V
    Alive -------->   Deleted   ----------> Deleted
      ^   <--------  with-value <---------- no-value
      |                                         |
      |                                         |
      \-----------------------------------------/

Given that Deleted-with-Value and Alive both are moved-to using
updateStoredValue(), we need to handle increasing numDeletedItems in
additional code paths - specifically whenever the value changes.

Fix this in HashTable::unlocked_updateStoredValue, and add tests for
these cases to the Delete-with-value tests.

Change-Id: Id9224ed0e66d0298c0f73621fedba3af1fe62e86
Reviewed-on: http://review.couchbase.org/76120
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23906: Implement delete-with-value with store() instead of delete() 19/76119/17
Dave Rigby [Wed, 29 Mar 2017 15:15:03 +0000 (16:15 +0100)]
MB-23906: Implement delete-with-value with store() instead of delete()

During investigation of issues with 'fully' deleting documents
(i.e. discarding their bodies) which are in state Delete-with-value,
it was found that there's two bugs which interact to allow some
transitions between Deleted states, but not others. As a result
Deletes are not always handled correctly.

Prior to Spock, the valid state transitions for StoredValues (relating
to deletion) were:

              /---\
              |   |
              |   V
              Alive -----------------\
                ^                    |
                |                    V
                \---------------- Deleted
                                 (no value)

With the advent of Deleted Bodies for system XATTRs, Deleted SVs can
optionally have a value (Blob) associated with them, which can be
changed (or removed) without going back to Alive:

              /---\
              |   |
              |   V
              Alive -----------------\
                ^                    |
                |                    V
                \---------------- Deleted <------------\
                           (with-value or no-value)    |
                                     |                 |
                                     \-----------------/

While the above transitions should be supported, StoredValue was not
fully updated to allow transitioning between the two Deleted variants.
Specifically, if an item had the `deleted` flag set, then calling
StoreValue::del() while in the Deleted-with-Value state became a no-op
- the value was not removed:

    void del(HashTable &ht) {
        if (isDeleted()) {
            return;
        }
        ...
        resetValue();  // <-- Value removed here

This means the Deleted-with-value -> Deleted-no-value transition is
broken - the deleted value was not removed.

If we want to fix this, and handle 'deleting-a-deleted-value' in
HashTable, then we need to allow the HashTable to re-delete an item
(essentially remove the early exit in StoredValue:del(). However in
doing so, it breaks *setting* the deleted value, due to a second bug
in VBucket::processSoftDelete():-

  VBucket::processSoftDelete() (which is the function which handles
  both Delete-with-value and Delete-no-value) assumes the deletion
  should have no value (it calls softDeleteStoredValue() with
  onlyMarkDeleted==false), and hence the item should have its body
  discarded. However due to the aforementioned 'bug' (early-exit) in
  StoredValue::del(), the body was kept (and Delete-with-Value ->
  Delete-with-Value transition currently works).

To resolve this, it seems simpler and cleaner to implement
Delete-with-Value as a Mutation - i.e. call down the EvpStore path
instead of EvpItemDelete. As EvpStore already passes an Item object
all the way down the call-chain to updateStoredValue(), we can simply
use the existing Item object to convey the the `del` op instead of a
`set`.

Change-Id: I3b9223da101b38e3e25aef237d0e52ee55bae623
Reviewed-on: http://review.couchbase.org/76119
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
4 years agoMove OSV::stale back to StoredValue 74/76774/5
Dave Rigby [Thu, 13 Apr 2017 13:01:56 +0000 (14:01 +0100)]
Move OSV::stale back to StoredValue

As part of MB-23795 it was necessary to move the OrderedStoredValue
{stale} flag out of the packed bitfields in StoredValue so it didn't
share its byte with any other data not guarded by writeLock. This was
to prevent any data races, as C++/x86_64 only exposes byte-level
addressing and if it remained in the bitfield we'd see races when
other elements in the bitfield (e.g. deleted) were changed.

While fixing the correctness issue, this had the consequence of
bloating the size of OrderedStoredValue by 8 bytes; as OSV was
previously an exact multiple of 8 bytes in size, and adding one more
byte for the bitfield forced the object size to increase by 8 bytes to
maintain ABI alignment rules.

However, we actually /do/ have a spare byte in StoredValue after the
bitfields. If we move the stale flag back to SV, but in its own byte
then we recover the size bloat of OSV - we are back to the original SV
and OSV sizes before MB-23795.

Change-Id: I15d299dcdd0107915c8b50c717305e2ecb9960a4
Reviewed-on: http://review.couchbase.org/76774
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
4 years agoFix cbstats timings 78/76778/3
James Harrison [Thu, 13 Apr 2017 15:15:45 +0000 (16:15 +0100)]
Fix cbstats timings

histograms was unexpectedly decorated with @cmd - despite not being a
top level cmd called by clitool.

This lead to an erroneous auth failure, no matter what.

Change-Id: I367d8869db8b151376916e726694fa8758b636d9
Reviewed-on: http://review.couchbase.org/76778
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23863: Make DCP backfill read deleted documents 09/76709/4
Jim Walker [Tue, 18 Apr 2017 09:12:31 +0000 (10:12 +0100)]
MB-23863: Make DCP backfill read deleted documents

recordDbDump is invoked for KVStore::scan (backfill) and was coded to
skip opening of the document if the docinfo says deleted.

This commit then removes the error case where
couchstore_open_doc_with_docinfo returns -5
(COUCHSTORE_ERROR_DOC_NOT_FOUND) as that just means the document has
no-value and we still need to continue to creating the Item.

Change-Id: I6e2e563ef68f9bc4404c5e59480f8c6fb2dd36e4
Reviewed-on: http://review.couchbase.org/76709
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoAdd WithMeta unit tests 81/76381/14
Jim Walker [Wed, 5 Apr 2017 15:20:00 +0000 (16:20 +0100)]
Add WithMeta unit tests

Create a test-suite using the SingleThreaded store test to drive
the with_meta commands.

The test-suite so far tries to exercise many of the error and success
paths.

The intent is to give a starting point for more comprehensive and easy
to understand *with_meta tests as opposed to the extension of the
testapp tests.

Change-Id: I4bdfec9879de3a9671fed820f9b5aab0ac7d2c71
Reviewed-on: http://review.couchbase.org/76381
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoRCPtr: Add dynamic_pointer_cast 52/76552/2
Dave Rigby [Wed, 5 Apr 2017 16:46:08 +0000 (17:46 +0100)]
RCPtr: Add dynamic_pointer_cast

Allows dynamic cast from one RCPtr instantiation to another. Based on
same function for std::shared_ptr.

Change-Id: Idc8723ea90dfd2843a3c5b602f5e42fad7a36613
Reviewed-on: http://review.couchbase.org/76552
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: James Harrison <james.harrison@couchbase.com>
Reviewed-by: Daniel Owen <owend@couchbase.com>
4 years agoFix typo in MockBasicLinkedList::getWriteLock 77/76777/4
Dave Rigby [Thu, 13 Apr 2017 16:14:32 +0000 (16:14 +0000)]
Fix typo in MockBasicLinkedList::getWriteLock

Change-Id: I7f7ceda0195549b89e3b317224c2eeebf80153b4
Reviewed-on: http://review.couchbase.org/76777
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23767: Notify vb high priority requests in a separate task 59/76659/12
Manu Dhundi [Thu, 13 Apr 2017 16:31:19 +0000 (09:31 -0700)]
MB-23767: Notify vb high priority requests in a separate task

Vbucket high priority requests are made by ns server during rebalance
wherein it expects a notification when the vbucket receives upto a
particular seqno.

We must send this notification in a separate task rather than in
front end thread to avoid deadlocks due to lock inversion.
That is, a front end thread generally makes call down the stack and
acquires locks in the order top to down (in our software stack).
But a notification path is up the software stack and hence must be
done as a separate task.

This commit creates a daemon task per Ephemeral Bucket to notify
vb high priority reuqests. When the task runs it notifies all the
connections to be notified as goes to sleep. When a new connection
is to be notified the task is woken again.

The current deadlock that was hit is captured in MB-23767.

Change-Id: Id114bedb5cd4de8b493ae6885734c3d440c8bea3
Reviewed-on: http://review.couchbase.org/76659
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23719: Add dynamic change of ephemeral_metadata_purge_age 41/76541/7
Dave Rigby [Mon, 10 Apr 2017 08:53:34 +0000 (09:53 +0100)]
MB-23719: Add dynamic change of ephemeral_metadata_purge_age

Add ephemeral_metadata_purge_age to setFlushParam() so its value can
be changed dynamically via SET_PARAM.

Also add a requirement on bucket_type==ephemeral, as it's not valid
for other bucket types.

Change-Id: I0fecb5ced449c672dd95a4b196bdab9f6aaa7347
Reviewed-on: http://review.couchbase.org/76541
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23795: Ephemeral Tombstone purging Task 85/76485/16
Dave Rigby [Fri, 7 Apr 2017 17:20:52 +0000 (18:20 +0100)]
MB-23795: Ephemeral Tombstone purging Task

Expands on the previous patch to implement a Task which performs
Tombstone purging. This Task is scheduled periodically (see
ep_ephemeral_metadata_purge_interval), and when run it visits all
vBuckets and purges all applicable OSVs.

Task can be reconfigured dynamically via epctl parameters:

- ephemeral_metadata_purge_age: Age in seconds after which deleted
  items are purged.

- ephemeral_metadata_purge_interval: How often should Tombstone
  purging task run to check for items to be purged.

Example output:

    NOTICE (eph) Ephemeral Tombstone Purger starting with purge age:60
    NOTICE (eph) Ephemeral Tombstone Purger completed. Purged 39000 items. Took 104ms. Sleeping for 60 seconds.

Change-Id: I126c74f2e82c0a31a2843240303548a24af2e90f
Reviewed-on: http://review.couchbase.org/76485
Reviewed-by: Jim Walker <jim@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23795: Ephemeral Tombstone purging (VBucket-level) 08/76408/25
Dave Rigby [Thu, 6 Apr 2017 16:14:42 +0000 (17:14 +0100)]
MB-23795: Ephemeral Tombstone purging (VBucket-level)

Add Ephemeral Tombstone purging for Ephemeral VBuckets. This patch
adds two purge operations - one for the HashTable, and one for the
sequenceList:

a) The HashTable visitor (EphemeralVBucket::HTTombstonePurger) visits
   all items in the HashTable, and identifies any deletes which are
   old enough to be purged (age > ephemeral_metadata_purge_age). Such
   items are marked as stale, and transferred from the HashTable to
   the SequenceList.

b) The SequenceList purger (BasicLinkedList::purgeTombstones()) then
   iterates over the sequencelist, and hard-deletes any items marked
   stale - both Alive and Deleted items. It is at this stage that the
   OSVs are actually deleted.

   Note the SequenceList purging is somewhat delicate, to ensure
   correctness while not blocking front-end writes. See the inline
   comments in BasicLinkedList::purgeTombstones() for further details.

Note this isn't yet driven by any tasks - only by the unit
tests. Subsequent patches will connect into tasks.

Change-Id: I937ed23317826c84cbdd0bb0b3749a99ff446497
Reviewed-on: http://review.couchbase.org/76408
Reviewed-by: Jim Walker <jim@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23795: Move StoredValue::stale to OSV; guard with SeqList::writeLock 15/76715/7
Dave Rigby [Wed, 12 Apr 2017 14:00:53 +0000 (15:00 +0100)]
MB-23795: Move StoredValue::stale to OSV; guard with SeqList::writeLock

The stale flag in (Ordered)StoredValue needs to be read by the
tombstone purger when iterating the sequenceList; however at this
point there is no HashTable lock available - the OSV has been released
from the hashTable and hence we have no way to determine it's bucket
(without reading the key to recalculate - and that's not possible
without first acquiring the lock!).

To allow us to safely access the stale flag, move it from being
protected by the HT locks to the SequenceList::writeLock, and move to
OSV so it doesn't share with the other bitfields in the same
byte. While on the face of it this might seen like a serious
performance degredation having to serialise on a single lock, it
actually doesn't appear to be /that/ bad:

1) The stale flag is only accessed rarely:

   a) Set to false when a StoredValue is initially created (and this
      doesn't need the lock as the item isn't in the SeqList yet)

   b) Set to true when an item becomes stale (result of concurrent
      update & rangeRead, or deleted item is aged out and marked Stale).

   c) Read during tombStone purging.

2) Even when we *do* need to access the stale flag, we have already
   acquired the writeLock in the hot path
   (updateStoredValue/markItemStale).

Note: This will increase contention during a tombstone purge, as it
will need to acquire and release the writeLock for every seqList
element; this is an area which should be looked at for future
performance improvement - for example can the stale flag be made
atomic, or can we add lightweight, per-OSV lock?

Note(2): This increases the size of OSV by 8 bytes, as moving Stale to
it pushes up the alignment requirements (it was previously aligned to
8 bytes), although we've only added 1 bit to it. Better solution would
be to pack this into the boost_list hook, // or introduce per-OSV
microlock to guard the whole object.

Change-Id: If814b966a10c64fada204998410dafc5ad255351
Reviewed-on: http://review.couchbase.org/76715
Reviewed-by: Jim Walker <jim@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
4 years agoMB-23530: Prevent replace with CAS on deleted item 61/76261/9
James Harrison [Tue, 4 Apr 2017 16:32:15 +0000 (17:32 +0100)]
MB-23530: Prevent replace with CAS on deleted item

Against an ephemeral bucket or full eviction persistent bucket, a
replace operation could erroneously succeed if the correct CAS value was
given. A replace with a CAS resolves internally to OPERATION_CAS, which
is treated in the same manner as OPERATION_SET.

Prior to this change, we checked if an item was previously deleted if
the CAS did _not_ match, but even if the CAS is /correct/ a replace with
a CAS should fail if the old item is deleted.

The set should only be rejected if the new item is /not/ deleted, to
avoid breaking the delete implementation described in
http://review.couchbase.org/#/c/76119.

Change-Id: Id1eeaeb5326b7a143c7213f07c774f02623bff8d
Reviewed-on: http://review.couchbase.org/76261
Reviewed-by: Daniel Owen <owend@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoMinor cbstats tasks improvements 34/76634/4
James Harrison [Tue, 11 Apr 2017 14:55:13 +0000 (15:55 +0100)]
Minor cbstats tasks improvements

Altered sort order to by default sort durations in descending order.

Small preparations for adding a "-r" flag to reverse.

Change-Id: I6a32d064c8bf38f677b512137bed0b0724fc8e67
Reviewed-on: http://review.couchbase.org/76634
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-22738: Move stripping of value from DCP stream back into ep-engine 18/76418/31
Daniel Owen [Thu, 6 Apr 2017 21:12:28 +0000 (22:12 +0100)]
MB-22738: Move stripping of value from DCP stream back into ep-engine

A revert of the http://review.couchbase.org/#/c/72398/
with the addition that determining whether whether to retrieve only
is made on the connection level, (as opposed to the stream level).

Change-Id: I641978c2be6c67e6a9d96c0a229ff21688c74055
Reviewed-on: http://review.couchbase.org/76418
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>
4 years agoMB-23530: Add test for CAS set after delete 26/76326/7
James Harrison [Wed, 5 Apr 2017 14:37:36 +0000 (15:37 +0100)]
MB-23530: Add test for CAS set after delete

It is erroneously possible to set an item with CAS after deleting it on
an ephemeral bucket or a full eviction persistent bucket. This breaks
the expected behaviour.

This test should identify this failure and as such can not be enabled
for these types until after the issue is resolved.

Change-Id: I5b382d10a2fa6f955645d8305282a494b5f8ba6b
Reviewed-on: http://review.couchbase.org/76326
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoMB-23267: Prevent changing inapplicable options from cbepctl 99/76399/12
James Harrison [Thu, 6 Apr 2017 13:30:22 +0000 (14:30 +0100)]
MB-23267: Prevent changing inapplicable options from cbepctl

Check requirements in setFlushParam and setTapParam for the following
configuration parameters:

 "tap_keepalive"
 "access_scanner_enabled"
 "alog_sleep_time"
 "alog_task_time"
 "ephemeral_full_policy"

this will prevent them being set if their requirements are not met.

Change-Id: Ie70d062e5333393e12771d325d22438f5e865bdf
Reviewed-on: http://review.couchbase.org/76399
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoMake set*Param methods members to allow use in tests 02/76702/3
James Harrison [Wed, 12 Apr 2017 10:55:04 +0000 (11:55 +0100)]
Make set*Param methods members to allow use in tests

Moved set*Param methods to be members of EventuallyPersistentEngine to
simplify testing the logic therein. They were previously static methods
and were not callable directly from tests.

Change-Id: I263ed94149a0142de5838556eb34799ff02c9049
Reviewed-on: http://review.couchbase.org/76702
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-22695: Add stats for datatypes 17/76617/3
olivermd [Mon, 6 Mar 2017 15:37:22 +0000 (15:37 +0000)]
MB-22695: Add stats for datatypes

This commit adds counts for datatypes of documents whose Storedvalue is
resident in memory.

Adds the datatype stats to the 'all' engine stat call. This means that
they will be added to the ETS tables in ns_server.

Introduces a regression test for MB-23892.

This patch was originally e4606e8 and was reverted by 6501c72 due to
MB-23892.

Change-Id: Ic0961af1a3c18362004369db0ff0fa7a9eeba22c
Reviewed-on: http://review.couchbase.org/76617
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoMB-23267: Hide irrelavant config options 62/75962/14
James Harrison [Wed, 29 Mar 2017 09:42:38 +0000 (10:42 +0100)]
MB-23267: Hide irrelavant config options

Hides from stats and prevents changing through cbepctl of:
 * item_eviction_policy on ephemeral buckets
 * ep_alog_* on ephemeral buckets
 * ephemeral_full_policy on persistent buckets
 * ep_tap_* if tap is disabled

Change-Id: I70ed89b033b845ac6d02f965fd5dda9ce884185c
Reviewed-on: http://review.couchbase.org/75962
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoIntroduce testing exception for rescheduled dead tasks 51/76551/7
James Harrison [Fri, 7 Apr 2017 14:21:32 +0000 (15:21 +0100)]
Introduce testing exception for rescheduled dead tasks

In http://review.couchbase.org/#/c/76394/ the ability to reschedule a
cancelled (and now in state TASK_DEAD) GlobalTask was fixed.

it does not appear that any tasks other than the ItemPager for ephemeral
buckets are rescheduled once dead, but to avoid changing existing
behaviour this introduces an exception if any task other than the
ItemPager is rescheduled, to bring them to our attention.

NB: This patch will be reverted to remove this exception for Spock.
(MB-23797)

Change-Id: If44b7cf8a71c3dc4d85fba98d65c4f608d449460
Reviewed-on: http://review.couchbase.org/76551
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23750: Fix toggling ephemeral ejection policy 94/76394/11
James Harrison [Thu, 6 Apr 2017 12:23:38 +0000 (13:23 +0100)]
MB-23750: Fix toggling ephemeral ejection policy

If going

auto_delete -> fail_new_data -> auto_delete

the itemPager task was rescheduled, but as it had previously been
cancelled the state remained TASK_DEAD, and so ExecutorThread::run
behaves as if the task has just been cancelled and needs cleaning up.
By resetting scheduled tasks back to TASK_RUNNING if they are dead, this
can be avoided.

Change-Id: Id007a15fdaeb80a79828da0cade031a424e653cf
Reviewed-on: http://review.couchbase.org/76394
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoMove defragmenter benchmarks to benchmarks 76/76476/5
olivermd [Fri, 7 Apr 2017 14:01:11 +0000 (15:01 +0100)]
Move defragmenter benchmarks to benchmarks

This moves the defragmenter benchmarks to ep_engine_benchmarks because
they can take ~18 seconds under ASAN. It also makes sense to group all
the benchmarks together.

To enable this move, this commit also puts the defragmenter test in to a
header file so that it can continued to be used by the defragmenter
tests which are not benchmarks.

Change-Id: I69f8da02c07e6469b9c80fbe06507c80d866ac7f
Reviewed-on: http://review.couchbase.org/76476
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoRefactor: Replace RCPtr<VBucket> with VBucketPtr 56/76556/3
Jim Walker [Mon, 10 Apr 2017 14:03:34 +0000 (15:03 +0100)]
Refactor: Replace RCPtr<VBucket> with VBucketPtr

Declare a named type for shared pointers to VBuckets.

Change-Id: I93e121f86199617c1651c5896efc7df7cd99ea83
Reviewed-on: http://review.couchbase.org/76556
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23829: Revert "MB-22695: Add stats for datatypes" 75/76575/3
Dave Rigby [Mon, 10 Apr 2017 17:45:26 +0000 (17:45 +0000)]
MB-23829: Revert "MB-22695: Add stats for datatypes"

Reverting to due null pointer dereference when replacing a non-resident item:

       297      if (v.getDatatype() != itm.getDataType()) {
    -> 298          --datatypeCounts[v.getValue()->getDataType()];
       299          ++datatypeCounts[itm.getDataType()];
       300      }
    (lldb) p v
    (StoredValue) $0 = {
      value = {
        value = 0x0000000000000000
      }

This reverts commit e4606e8f50797e40d3a9f7931c1e45a070f82002.

Change-Id: I5998e2eaadedf897192d0fb8aeb184ac85c4bf8f
Reviewed-on: http://review.couchbase.org/76575
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23796: Wait for flusher to avoid stat race 48/76548/2
olivermd [Mon, 10 Apr 2017 11:52:44 +0000 (12:52 +0100)]
MB-23796: Wait for flusher to avoid stat race

Under full eviction, the stat used for curr items is different to that
used under value eviction. Aditionally, under full eviction, the flusher
updates the relevant stat. This can lead to a race in the test between
the flusher and the test code. For example to test code to get the stats
may run before the flusher can, meaning that the previous operation is
not reflected in the stats, leading to the test failing a check.

Change-Id: I3ae6617fec0e0d076438bd839feaaea7633e23fd
Reviewed-on: http://review.couchbase.org/76548
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoMB-23556: Add response stats to cbstats 55/76555/4
olivermd [Mon, 10 Apr 2017 13:33:41 +0000 (14:33 +0100)]
MB-23556: Add response stats to cbstats

Example output:

$ ./management/cbstats localhost:12000 responses
 SUCCESS: 14410

$ ./ep-engine/management/cbstats localhost:12000 responses all
 AUTH_CONTINUE:                   0
 AUTH_ERROR:                      0
 AUTH_STALE:                      0
 DELTA_BADVAL:                    0
 E2BIG:                           0
 EACCESS:                         0
 EBUSY:                           0
 EINTERNAL:                       0
 EINVAL:                          0
 ENOMEM:                          0
 ERANGE:                          0
 ETMPFAIL:                        0
 KEY_EEXISTS:                     0
 KEY_ENOENT:                      0
 LOCKED:                          0
 NOT_INITIALIZED:                 0
 NOT_MY_VBUCKET:                  0
 NOT_STORED:                      0
 NOT_SUPPORTED:                   0
 NO_BUCKET:                       0
 ROLLBACK:                        0
 SUBDOC_DELTA_EINVAL:             0
 SUBDOC_DOC_NOTJSON:              0
 SUBDOC_INVALID_COMBO:            0
 SUBDOC_MULTI_PATH_FAILURE:       0
 SUBDOC_NUM_ERANGE:               0
 SUBDOC_PATH_E2BIG:               0
 SUBDOC_PATH_E2DEEP:              0
 SUBDOC_PATH_EEXISTS:             0
 SUBDOC_PATH_EINVAL:              0
 SUBDOC_PATH_ENOENT:              0
 SUBDOC_PATH_MISMATCH:            0
 SUBDOC_SUCCESS_DELETED:          0
 SUBDOC_VALUE_CANTINSERT:         0
 SUBDOC_VALUE_ETOODEEP:           0
 SUBDOC_XATTR_INVALID_FLAG_COMBO: 0
 SUBDOC_XATTR_INVALID_KEY_COMBO:  0
 SUBDOC_XATTR_UNKNOWN_MACRO:      0
 SUCCESS:                         24909
 UNKNOWN_COLLECTION:              0
 UNKNOWN_COMMAND:                 0
 XATTR_EINVAL:                    0

Change-Id: Id75d6f19d4302a62c4c4d13274ba09a6bf8a0743
Reviewed-on: http://review.couchbase.org/76555
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoMB-23618: Revert 25ce75aa1 bgFetch changes 40/76540/2
Jim Walker [Mon, 10 Apr 2017 08:52:17 +0000 (09:52 +0100)]
MB-23618: Revert 25ce75aa1 bgFetch changes

Revert the bgFetch changes from 25ce75aa1 as they're not required
anymore. The original commit added conditional "isMeta" parameter
changes to withMeta functions because the datatype of the document
use to be part of the body. Now that this is part of the meta data
the changes are no longer required.

This has the side-effect of fixing MB-23618, so there was also
something wrong with the bgFetch changes.

Change-Id: If3743b766b5da1b51c00e3b5ee84707f2ba4262e
Reviewed-on: http://review.couchbase.org/76540
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoReorganise EPStore tests 25/76325/5
James Harrison [Wed, 5 Apr 2017 13:48:08 +0000 (14:48 +0100)]
Reorganise EPStore tests

Separating out tests which are applicable to all of:

 EP full eviction
 EP value eviction
 Ephemeral

from those only applicable to the two former.

Change-Id: If1580b5d20b60778dd0a7d88260c80a172251b07
Reviewed-on: http://review.couchbase.org/76325
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23267: Add ability to hide inapplicable config options 61/75961/10
James Harrison [Tue, 28 Mar 2017 12:46:12 +0000 (13:46 +0100)]
MB-23267: Add ability to hide inapplicable config options

This is expressed by declaring the requirements for a particular config
option in configuration.json e.g.,

"ephemeral_full_policy": {
    ...
    "requirements": {
        "bucket_type": "ephemeral"
    }
}

This example prevents "ephemeral_full_policy" from being listed in stats
if "bucket_type" is not "ephemeral".

Change-Id: I4c85132612f55a4edb7c5497c9744ef63efbd206
Reviewed-on: http://review.couchbase.org/75961
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoMB-23719: Add ephemeral_metadata_purge_age setting 07/76407/6
Dave Rigby [Thu, 6 Apr 2017 16:39:51 +0000 (17:39 +0100)]
MB-23719: Add ephemeral_metadata_purge_age setting

Add the new config option "ephemeral_metadata_purge_age" to
ep-engine's configuration, along with listeners so changes in it's value
are propogated to the bucket.

Note the default value of 60s, and -1 for disabled exists mostly for
component testing - don't expect that to be exposed to
users. Similarly the unit of seconds is used to aid in testing - I'm
aware the UI is limited to a minimum of 1 hour.

This setting will be dynamic - i.e. can be changed via SET_PARAM
without restarting the bucket.

As per the name, this option is only applicable to Ephemeral buckets -
it will be ignored by Persistent Buckets.

  Implementation Note: The actual Tombstone Purger Tasks do not yet
  exist, so there's currenlty placeholder code for enable/disabling
  the task.

Change-Id: I78726a1bce8c870c70c916cae6f174ea86ef97bb
Reviewed-on: http://review.couchbase.org/76407
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
4 years agoStoredValue: record deleted time for OSVs 06/76406/6
Dave Rigby [Thu, 6 Apr 2017 16:18:35 +0000 (17:18 +0100)]
StoredValue: record deleted time for OSVs

For OrderedStoredValues, record the time when they are deleted, so
deleted OSVs can later be purged (completely deleted) when they reach
a certain age.

To avoid adding another 4bytes to OSV, the lock_expiry member is
re-used to also be used for the deleted time (deleted items cannot be
locked and hence this is otherwise unused once items enter that
state).

Change-Id: I1a6b9c5e8be4d3df1dca5c1e91be6219889686da
Reviewed-on: http://review.couchbase.org/76406
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
4 years agoMB-23712: Check for invalid cas before setting value for deleted body 52/76252/7
Dave Rigby [Tue, 4 Apr 2017 13:31:52 +0000 (14:31 +0100)]
MB-23712: Check for invalid cas before setting value for deleted body

Before setting the value for a deleted item, check to see if the
incoming cas matches the existing cas

Change-Id: If61e47f0c29ede41778f0f3d817082a83ec8f851
Reviewed-on: http://review.couchbase.org/76252
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Sriram Ganesan <sriram@couchbase.com>
4 years agoEPEngine::destroy: explain why stats are snapshotted 66/76466/3
Dave Rigby [Fri, 7 Apr 2017 08:21:44 +0000 (09:21 +0100)]
EPEngine::destroy: explain why stats are snapshotted

Change-Id: Iddf82827efc7810c2fb6abd459d31e777fcc7392
Reviewed-on: http://review.couchbase.org/76466
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoRestore correct clitools support for variable argument count 84/76484/2
James Harrison [Fri, 7 Apr 2017 17:18:14 +0000 (18:18 +0100)]
Restore correct clitools support for variable argument count

This was broken by failing to account for defaults being provided to
arguments.

Change-Id: I8058b12052c3befe5fe32684decaa88c659695d4
Reviewed-on: http://review.couchbase.org/76484
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>
4 years ago[Ephemeral]: Rollback to 0 explicitly and improve rollback testing 44/76444/4
Manu Dhundi [Fri, 7 Apr 2017 15:35:10 +0000 (08:35 -0700)]
[Ephemeral]: Rollback to 0 explicitly and improve rollback testing

In ephemeral buckets we currently rollback to 0 even if the request
is to rollback to an intermediate point. This is because we don't have
the luxury of disk snapshots to do an intermediate rollback.

This commit does an explicit rollback to 0, rather than as default.

Also adds a test case that requests a rollback to an intermediate
point.

Change-Id: Ifea06562efaae4ef9b995c718a1b1adc026c4e02
Reviewed-on: http://review.couchbase.org/76444
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoChange some EPStore tests to also run against Ephemeral 24/76324/4
James Harrison [Wed, 5 Apr 2017 11:31:56 +0000 (12:31 +0100)]
Change some EPStore tests to also run against Ephemeral

For tests which do not reference eviction it is reasonable to also run
them for ephemeral buckets

Change-Id: I2819ba466119b32bb9651e027ae6e024dc31eab5
Reviewed-on: http://review.couchbase.org/76324
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoep_testsuite: Expand Delete-with-value test cases 10/76110/16
Dave Rigby [Thu, 30 Mar 2017 11:28:52 +0000 (12:28 +0100)]
ep_testsuite: Expand Delete-with-value test cases

Expand the ep_testsuite tests for Deletes with a value. Ensure that
all possible state transitions (between Alive, Deleted-with-value &
Deleted-no-value) are handled.

Note that the Delete-with-value -> Delete-no-value transition cannot
be driven correctly from the engine API:-

a) Calling EvpDelete() will fail, as it requires the item is in the
   Alive state.

b) Calling EvpStore(<zero-length-item>, DocState:Deleted) will
   succeed, but results in a memory leak from couchstore when fetching
   the zero-length body. Note: this is a latent bug - see MB-23697.

As such this transition isn't currently handled.

Change-Id: I9ed79e5dd1bc0387fa39a84cd5574a38d696576a
Reviewed-on: http://review.couchbase.org/76110
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>
4 years agoIntroduce basic tests for python tools 51/76251/5
James Harrison [Tue, 4 Apr 2017 12:49:11 +0000 (13:49 +0100)]
Introduce basic tests for python tools

Adds tests which run cbstats and cbepctl to confirm they are both
valid Python 2.6 code. These are /very/ minimal tests, and essentially
test only for correct syntax and resonable functioning of the "--help"
code path. They do not assert anything beyond syntactic correctness for
code outside this coverage.

Change-Id: I89a991c972fef30a881e3b278507cac4d0df62af
Reviewed-on: http://review.couchbase.org/76251
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoImprove cbstats and cbepctl usage text 11/76411/3
James Harrison [Thu, 6 Apr 2017 17:48:26 +0000 (18:48 +0100)]
Improve cbstats and cbepctl usage text

Corrects exit code when calling with "--help" - this is a successful
execution and should exit(0). This facilitates simply executing these
tools with "--help" as a very basic test.

Also improves some error messages, making better use of optparse.

Change-Id: I4449e8e711072f156de1c700992d0d98b5bebddd
Reviewed-on: http://review.couchbase.org/76411
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoHashTable: Use NonNegativeCounter<> for num{...}Items 47/76047/9
Dave Rigby [Tue, 28 Mar 2017 15:35:10 +0000 (16:35 +0100)]
HashTable: Use NonNegativeCounter<> for num{...}Items

Use cb::NonNegativeCounter<> for counts of items in the HashTable,
instead of std::atomic and manually checking for underflow.

Change-Id: I3b8182d6b443971c6d23f0e55e42a78825e147c2
Reviewed-on: http://review.couchbase.org/76047
Reviewed-by: Oliver Downard <oliver.downard@couchbase.com>
Reviewed-by: Daniel Owen <owend@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoStoredValue: Add const variant of toOrderedStoredValue() 05/76405/3
Dave Rigby [Thu, 6 Apr 2017 10:56:10 +0000 (11:56 +0100)]
StoredValue: Add const variant of toOrderedStoredValue()

Also tidy up some outdated comments about 'small item types' (these
haven't existed for a long time).

Change-Id: I40fded0ccd8baed1faeca6689c8f3185b693823c
Reviewed-on: http://review.couchbase.org/76405
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23637: Extra logging for dcp-vbtakeover stat & refactor stream type 62/76162/9
Daniel Owen [Sat, 1 Apr 2017 14:00:18 +0000 (15:00 +0100)]
MB-23637: Extra logging for dcp-vbtakeover stat & refactor stream type

The dcp-vbtakeover stat is critical to the success of rebalance.
Therefore it is useful for debugging future rebalance failures to log
the unexpected paths and error paths.

We want to print the stream type in the logging output.  Therefore the
stream type has been refactored into a enum class and a to_string
method provided to print a textual representation of the type.

Change-Id: I72ce7c6bd1f923f3cfa7557f1b150f97aa5fa7f9
Reviewed-on: http://review.couchbase.org/76162
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoCoverity:CID165843: Unitialised member in SystemEventConsumerMessage 93/76393/3
Jim Walker [Thu, 6 Apr 2017 12:05:04 +0000 (13:05 +0100)]
Coverity:CID165843: Unitialised member in SystemEventConsumerMessage

vbid member was not initialised

Change-Id: I0e685cf3008e2bc446ced39a460b94e024596073
Reviewed-on: http://review.couchbase.org/76393
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23573: Add full eviction defragment benchmarks 71/75971/5
olivermd [Wed, 29 Mar 2017 15:43:56 +0000 (16:43 +0100)]
MB-23573: Add full eviction defragment benchmarks

Enables the defragmenter benchmarks for full eviction. This required
changing the populate phase to use processSet rather than processAdd as
processAdd would check respond with bgFetch needed.

As a result of changing to processSet, when testing mapped memory we
need to clear the checkpointManager so that the memory gets released. I
also found that I was able to reduce the fuzz factor from 1.3 to 1.05.
Change-Id: I1cb01d9bb4501201caedd67c7da2e2fc1353b21a
Reviewed-on: http://review.couchbase.org/75971
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>
4 years ago[Refactor]: Change configuration.cc to use boost::variant 10/75210/19
James Harrison [Tue, 14 Mar 2017 15:04:07 +0000 (15:04 +0000)]
[Refactor]: Change configuration.cc to use boost::variant

Using variant simplifies some aspects of configuration.cc. It also paved
the way to templating a number of methods, reducing code repetition.

Change-Id: I625c84dd1daa905d38f6ab0e2d7ee31bf998657b
Reviewed-on: http://review.couchbase.org/75210
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoMB-23713: increment the rev seqno while updating a deleted item 15/76315/6
Sriram Ganesan [Wed, 5 Apr 2017 09:50:26 +0000 (17:50 +0800)]
MB-23713: increment the rev seqno while updating a deleted item

When setting the value for a deleted item, the rev seqno also
needs to be updated

Change-Id: I563e7ac896acbb9319db5fc3516d98ebc28d4d33
Reviewed-on: http://review.couchbase.org/76315
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoKVStore tests: Use unique name for data_dir 90/76190/3
Dave Rigby [Mon, 3 Apr 2017 13:22:35 +0000 (14:22 +0100)]
KVStore tests: Use unique name for data_dir

The CouchKVStoreTest, CouchKVStoreErrorInjectionTest and
CouchstoreTest suites are using a fixed path ("/tmp/kvstore-test") for
the data directory. This could collide with other instances.

Instead use a uniquely-generated name, and move the common setup code
into a SetUp() function.

Change-Id: I07b8169b66e1c0c55af538b1fd10de83d932f92c
Reviewed-on: http://review.couchbase.org/76190
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
4 years agoMB-23590: Update del_with_meta to accept a value (xattrs) 56/76256/5
Jim Walker [Tue, 4 Apr 2017 09:37:40 +0000 (10:37 +0100)]
MB-23590: Update del_with_meta to accept a value (xattrs)

Clients need to be able to perform del_with_meta with an xattr value
i.e. delete the document, but leave the xattrs.

This commit adds the simple functionality of setting a value but does
not add more extensive functionality where we need to consider pruning
user xattrs from the deleted document.

This commit is also flawed in that a subsequent set with cas following
the delete may succeed when it should not as the del_with_meta(xattr)
leaves the StoredValue in the HT marked as isDeleted where a
del_with_meta would remove it from the HT, giving a different result
to the subsequent set.

Change-Id: I3f6d50ab96d60724a24d33abe405f2f4df069480
Reviewed-on: http://review.couchbase.org/76256
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoRefactor setWithMeta/delWithMeta in EventuallyPersistentEngine 55/76255/5
Jim Walker [Tue, 4 Apr 2017 12:50:06 +0000 (13:50 +0100)]
Refactor setWithMeta/delWithMeta in EventuallyPersistentEngine

The refactoring separates the new object (Item/ExtendedMetaData)
allocation from the packet decoding/validation layer.

Change-Id: Iddcbd8d60bd743f396631043765a4052ffc3a74e
Reviewed-on: http://review.couchbase.org/76255
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
4 years agoMB-23664: Use max possible limit for Ephemeral backfill memory mgmt 94/76294/4
Manu Dhundi [Wed, 5 Apr 2017 17:16:51 +0000 (10:16 -0700)]
MB-23664: Use max possible limit for Ephemeral backfill memory mgmt

In-memory backfilling is currently not memory managed. Therefore set the
scan buffer (per backfill buffer for managing backfill memory usage)
byte limit, item limit to max. Also, for the same reason as above set
the overall DCP backfill limit to max.

Unlike disk backfills, where there is luxury of surplus disk memory,
we cannot retain in-memory snapshots for long in case of a memory
pressure during backfill. This is because the snapshots themselves
use memory.

Memory management can be done, but will be considered in a future
task (MB-23734).

Change-Id: I796d1b055b6948ba8bfde01556db8c16e3e16622
Reviewed-on: http://review.couchbase.org/76294
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Premkumar <premkumar.thangamani@couchbase.com>
4 years agoMB-23519: Add stats and logs for high priority vbucket requests 04/76304/6
Manu Dhundi [Wed, 5 Apr 2017 17:13:38 +0000 (10:13 -0700)]
MB-23519: Add stats and logs for high priority vbucket requests

These stats and logs aid in debugging rebalance related issues.

Change-Id: I03c68c5ed987c8281b14b3a48140a18de2e709fd
Reviewed-on: http://review.couchbase.org/76304
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Premkumar <premkumar.thangamani@couchbase.com>
4 years agoMB-22695: Add stats for datatypes 74/74974/21
olivermd [Mon, 6 Mar 2017 15:37:22 +0000 (15:37 +0000)]
MB-22695: Add stats for datatypes

This commit adds counts for datatypes of documents whose Storedvalue is
resident in memory.

Adds the datatype stats to the 'all' engine stat call. This means that
they will be added to the ETS tables in ns_server.

Change-Id: I321f81d00e4fb63604aeec84df4933f6d368fdb7
Reviewed-on: http://review.couchbase.org/74974
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoMB-23704: Fix Python2.6 incompatibilities 40/76240/5
James Harrison [Tue, 4 Apr 2017 08:16:03 +0000 (09:16 +0100)]
MB-23704: Fix Python2.6 incompatibilities

cbepctl used dictionary comprehension which requires Python>=2.7. In 2.6
it is not valid syntax. Replaced with equivalent 2.6 code.

cbstats used an OrderedDict, and formatting without field names/indexes
i.e., `"{}".format(foo)`.

Change-Id: Icf3362db4fd8095c6a9b5e1b6fa2bacae01f0dca
Reviewed-on: http://review.couchbase.org/76240
Tested-by: James Harrison <james.harrison@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years ago[Ephemeral]: Serialize CMD_SEQNO_PERSISTENCE with vBucket seqno generation 95/76195/5
Manu Dhundi [Mon, 3 Apr 2017 20:32:08 +0000 (13:32 -0700)]
[Ephemeral]: Serialize CMD_SEQNO_PERSISTENCE with vBucket seqno generation

This commit serializes the CMD_SEQNO_PERSISTENCE request w.r.t seqno
generation in ephemeral vbuckets because we must not queue up stale
requests. In regular couchbase vbuckets this is not a problem because
of cold runs of flusher task periodically. But in ephemeral we might end
up in a scenario wherein stale requests may never get notified as
there is no new sequence number triggering the notification.

It is not easy to write a test case for this race condition as we will
need 2 racing threads (writer thread and CMD_SEQNO_PERSISTENCE thread)
to be scheduled in a particular way inside the ep-engine.

Change-Id: Ie7ef1c1312b2047e834922f7c1dcb76491bd1588
Reviewed-on: http://review.couchbase.org/76195
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years ago[Ephemeral]: Don't support CMD_CHECKPOINT_PERSISTENCE 60/76160/5
Manu Dhundi [Mon, 3 Apr 2017 17:29:12 +0000 (10:29 -0700)]
[Ephemeral]: Don't support CMD_CHECKPOINT_PERSISTENCE

In ephemeral buckets we don't need to handle CMD_CHECKPOINT_PERSISTENCE
as there is no persistence. Hence we return not supported in the
response.

Change-Id: Ibc1d86b9de5078367dc60b23189f2815795d6ef9
Reviewed-on: http://review.couchbase.org/76160
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoUpdate isDatatypeSupported to take a value 50/75950/9
Jim Walker [Wed, 29 Mar 2017 06:48:26 +0000 (07:48 +0100)]
Update isDatatypeSupported to take a value

Memcached function now takes a parameter, individual bits of the
datatype may or may not be supported.

Also update places where we used the now renamed memcached
functions and types (compression->snappy).

Change-Id: I1170dcf369a7378bdd136f374d4989495a8b90c9
Reviewed-on: http://review.couchbase.org/75950
Reviewed-by: James Harrison <james.harrison@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23505: Add test for notification upon receiving a backfill item 52/76152/3
Manu Dhundi [Mon, 3 Apr 2017 15:33:40 +0000 (08:33 -0700)]
MB-23505: Add test for notification upon receiving a backfill item

Test case for commit 78a65deb6a14150829c44b06fcd8d53d5389f1ba.

It tests if replica vbucket notifies pending high priority requests
upon receiving backfill items.

Change-Id: Ief9e14c265081fab70cc2823a410d6f0c84cd477
Reviewed-on: http://review.couchbase.org/76152
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoImprove No access error message in cbstats and cbepctl 83/75783/5
James Harrison [Mon, 27 Mar 2017 13:27:36 +0000 (14:27 +0100)]
Improve No access error message in cbstats and cbepctl

Change-Id: If52292eac105d66cce24dc06458da7f287327a63
Reviewed-on: http://review.couchbase.org/75783
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: James Harrison <james.harrison@couchbase.com>
4 years agoMB-23637: findByName should not find connections marked as dead 21/76121/8
Daniel Owen [Fri, 31 Mar 2017 13:35:41 +0000 (14:35 +0100)]
MB-23637: findByName should not find connections marked as dead

When we attempt to make a connection with the same name we mark the
first as needing to disconnect and create a new connection with the
same name.  When manageConnections is executed the connection marked as
needing to disconnect will be closed.

However if findByName is called before manageConnections is run then it
could return either the new connection or the connection about to be
closed.

This patch changes findByName to only return the connection which is
not marked as needing to disconnect.

Change-Id: I02b77f30b3d48ca13e37fd5f07580c8fa3ea17e9
Reviewed-on: http://review.couchbase.org/76121
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-16181: Collections DcpProducer 03/75603/6
Jim Walker [Tue, 14 Feb 2017 11:44:19 +0000 (11:44 +0000)]
MB-16181: Collections DcpProducer

1. Enable SystemEvent Items in "Item::shouldReplicate"
2. Update DcpProducer so that it grabs the various bits of collection
   data for the transferred events and calls through the engine's dcp
   system_event callback

An update synchrounous engine test now links a MockDcpConsumer to
a MockDcpProducer and drives events from one to the other.

Change-Id: I83052aa956afbe7495c15420889c84d920a5b7fc
Reviewed-on: http://review.couchbase.org/75603
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-22018: Add tasks stats to cbstats 10/74510/28
James Harrison [Fri, 24 Feb 2017 16:47:33 +0000 (16:47 +0000)]
MB-22018: Add tasks stats to cbstats

Initial work on adding a "top" like tool showing tasks running and
sleeping in a cluster.

This can be viewed by running

    cbstats host:port tasks

This shows the information about the tasks such as the state, next
runtime, description, type etc.

Future additions to the tool are intended to add features such as load
averages for task types (of the style shown by `uptime`) and percentage
utilisation over a period of time per thread.

Change-Id: I005a2589a35b4c4364ba46cf2b44b71061ba06b3
Reviewed-on: http://review.couchbase.org/74510
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-22018: Simplify task scheduling 76/75776/8
James Harrison [Mon, 27 Mar 2017 11:14:38 +0000 (12:14 +0100)]
MB-22018: Simplify task scheduling

Tasks are now associated with a particular type, so the queue type no
longer needs to be specified when scheduling.

Change-Id: I53a4d76a6b0395957abd5391780aa1cf32a59444
Reviewed-on: http://review.couchbase.org/75776
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-22018: Add method mapping TaskId to task_type_t 75/75775/7
James Harrison [Fri, 24 Mar 2017 16:36:34 +0000 (16:36 +0000)]
MB-22018: Add method mapping TaskId to task_type_t

Allows identification of the task type without needing to know what
queue it is scheduled in. At the moment this is not /guaranteed/ to be
entirely true, but another patch will alter scheduling to use this,
ensuring consistency.

Change-Id: Icf03d9f459f0db3b480df61734c390580954bd6d
Reviewed-on: http://review.couchbase.org/75775
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoAdd basic xerror support to mc_bin_client.py 07/75607/15
James Harrison [Thu, 23 Mar 2017 10:21:39 +0000 (10:21 +0000)]
Add basic xerror support to mc_bin_client.py

A very na├»ve implementation of the xerror support - for now just adds
the mapped error name and description to the printed error. This is not
overly useful and would require further work.

Also adds subclasses for each current exception, to allow except-ing a
specific error.

Change-Id: I6d55b90eb46e628f6c148b9ab434ac0caae75308
Reviewed-on: http://review.couchbase.org/75607
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: James Harrison <james.harrison@couchbase.com>
4 years agoMB-22451: Don't schedule backfill until previous is complete 48/76048/6
Daniel Owen [Wed, 29 Mar 2017 14:58:09 +0000 (15:58 +0100)]
MB-22451: Don't schedule backfill until previous is complete

Previously in the handleSlowStream function if we are in the
Backfilling state and backfillTask is not running then we called
scheduleBackfill_UNLOCKED.

However although the backfillTask is not running there could still
be items in the Ready Queue.

As we are still in the Backfilling state, regardless as to whether
the backfillTask is running, we should drop the current cursor
and set the pendingBackfill flag.

Change-Id: I7d8c19041b9cec10640c4ef72c5d62cd73985ea4
Reviewed-on: http://review.couchbase.org/76048
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoMB-23639: Fix potential overflow of persisted_seqno in CMD_SEQNO_PERSISTENCE 40/76140/3
Manu Dhundi [Fri, 31 Mar 2017 18:00:22 +0000 (11:00 -0700)]
MB-23639: Fix potential overflow of persisted_seqno in CMD_SEQNO_PERSISTENCE

Using uint16_t for persisted_seqno can cause overflow and might lead
to rebalance failures in Ephemeral buckets and slower rebalance in
couchbase buckets.

Rebalance hang: if indeed persisted_seqno == seqno_waited_on
(high_seqno), but not seen as such due to overflow, then ns_server
will wait for a notification which will never get triggered as there
would be no new mutation (as the vb has already reached high_seqno).

In Couchbase buckets, flusher task will sleep for less than
DEFAULT_MAX_SLEEP_TIME and every time it runs, it will notify any high
priority vb requests.

In Ephemeral buckets, we do not have flusher task and hence can result
in rebalance hang forever.

This commit changes persisted_seqno type from uint16_t to uint64_t.

Change-Id: Ib0caa33b5536744bece64c2e8ab76efa2b841c72
Reviewed-on: http://review.couchbase.org/76140
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
4 years agoMB-23454: Fix data race in createTapQueue 27/76127/4
James Harrison [Fri, 31 Mar 2017 14:30:27 +0000 (15:30 +0100)]
MB-23454: Fix data race in createTapQueue

Appears to be caused by the same buggy string COW as MB-19688 and
MB-18510.

Upon updating the name of a tap connection, the old value was deleted.
This was considered by ThreadSanitizer to be racy with changes made in
one of multiple tasks.

Resolved in the same manner as MB-19688 - forcing an explicit
copy of `std::string name` in `ConnHandler::setName()`.

Change-Id: Iffee9ce2a695e97248e6b4f8aa640f4b1f4f3dfa
Reviewed-on: http://review.couchbase.org/76127
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
4 years agoMake cbstats and cbepctl hello 06/75606/13
James Harrison [Thu, 23 Mar 2017 10:21:02 +0000 (10:21 +0000)]
Make cbstats and cbepctl hello

Change-Id: I797304de75220c661ea7a77b1fc8818e8b61340f
Reviewed-on: http://review.couchbase.org/75606
Tested-by: James Harrison <james.harrison@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoRefactor: Move shared logic out of cbstats and cbepctl 05/75605/10
James Harrison [Wed, 22 Mar 2017 12:32:41 +0000 (12:32 +0000)]
Refactor: Move shared logic out of cbstats and cbepctl

Change-Id: Ibd3fb6b8b294e394fbd06a3eb58abc1c7ae1c590
Reviewed-on: http://review.couchbase.org/75605
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: James Harrison <james.harrison@couchbase.com>
4 years agoMB-23505: Add seqno received in backfills correctly for notifies 05/76105/2
Manu Dhundi [Fri, 31 Mar 2017 07:35:51 +0000 (00:35 -0700)]
MB-23505: Add seqno received in backfills correctly for notifies

The notify context must be populated with correct seqno even when
the item is received as a backfill item on a replica vbucket. This
helps notify any pending vb high priority rebalance requests
correctly.

Change-Id: I828cbf4fcfef0afd034d1fa026addcbb0a0a1c6f
Reviewed-on: http://review.couchbase.org/76105
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoFix warning reported by '-Wformat=' 26/76126/2
Trond Norbye [Fri, 31 Mar 2017 14:28:54 +0000 (16:28 +0200)]
Fix warning reported by '-Wformat='

format '%d' expects argument of type 'int', but argument 5 has type 'DcpResponse::Event' [-Wformat=]

Change-Id: I2c2979d8a1c539b72db8f772ecfd95227246c245
Reviewed-on: http://review.couchbase.org/76126
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoRefactor: Use DocStateFilter for engine->get() 15/76115/5
Trond Norbye [Fri, 31 Mar 2017 11:27:02 +0000 (13:27 +0200)]
Refactor: Use DocStateFilter for engine->get()

Change-Id: Ied4fa5cdd9520a449eac4ccd33be51cdbacc6f37
Reviewed-on: http://review.couchbase.org/76115
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23640: Fix check for document state in EvpGet 07/76107/8
Trond Norbye [Fri, 31 Mar 2017 08:41:44 +0000 (10:41 +0200)]
MB-23640: Fix check for document state in EvpGet

Change-Id: I975d7aa8196df16684b40f54895f0cd96113b0ff
Reviewed-on: http://review.couchbase.org/76107
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years ago[Ephemeral]: Add VBucket level basic unit tests 04/76004/4
Manu Dhundi [Thu, 30 Mar 2017 22:29:18 +0000 (15:29 -0700)]
[Ephemeral]: Add VBucket level basic unit tests

This commit adds VBucket level unit tests which test add, update,
backill and soft delete of items. It also tests updates and soft
deletes when a simulated backill is running on the linked list
in the ephemeral vbucket.

Change-Id: I393b0491db7d8c4211c1ac2148ea00975c727e7d
Reviewed-on: http://review.couchbase.org/76004
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoLet DCP log messages look like memcached log messages 71/75771/9
Trond Norbye [Mon, 27 Mar 2017 09:25:13 +0000 (11:25 +0200)]
Let DCP log messages look like memcached log messages

The log messages include the connection identifiers first to
allow it to be easily mapped to the events logged by the
core.

example:

2017-03-30T17:53:25.450557+02:00 NOTICE 40: (bucket) DCP (Producer) eq_dcpq:dcpsend - (vb 1023) Creating stream with start seqno 0 and end seqno 18446744073709551615

Change-Id: I84628c182ce314b371b3c8de00d3b771f2d6ac76
Reviewed-on: http://review.couchbase.org/75771
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoDefragmenterTest: Fix missing populate phase 72/75872/3
Dave Rigby [Tue, 28 Mar 2017 14:16:10 +0000 (15:16 +0100)]
DefragmenterTest: Fix missing populate phase

During previous migration of the Defragmenter benchmark tests to
GTest, we inadvertantly lost the actual population of the data they
need to act on. As such the various tests (Visit, DefragAlways,
DefragAge10, DefragAge10_20ms) essentially were invalid.

Fix this by restoring a populateVbucket() call.

Change-Id: Ib2c1d4b4385c4aa5b4611e034d80fffa0d7aaf54
Reviewed-on: http://review.couchbase.org/75872
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23562: [Ephemeral] Don't permit deleted items to be pagedOut 74/75874/9
Dave Rigby [Tue, 28 Mar 2017 14:41:37 +0000 (15:41 +0100)]
MB-23562: [Ephemeral] Don't permit deleted items to be pagedOut

Given that soft-deleted items are held in HashTable for Ephemeral
buckets, we should not allow a StoredValue which have already been
deleted to be pagedOut again, as we end up counting the delete twice.

Change-Id: I105c01da48c776bebadc679f9b00cd5dd8cbdf19
Reviewed-on: http://review.couchbase.org/75874
Reviewed-by: Manu Dhundi <manu@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoTests: Refactor to move helpers from MockEPVBucket to VBucketTest 71/75871/7
Dave Rigby [Thu, 30 Mar 2017 09:38:45 +0000 (10:38 +0100)]
Tests: Refactor to move helpers from MockEPVBucket to VBucketTest

Previously we accessed protected members of VBucket by inheriting from
it (MockEPVBucket), and adding public methods to the mock which called
the protected ones. This was fine when there was just one type of
VBucket to test, but this doesn't work well when we also want to test
other VBucket subclasses - e.g. EphemeralVBucket.

Similar to the changes made in 915ba08, change to making the test
fixture (VBucketTest) a friend of VBucket. This will allow the
soon-to-be-added EphemeralVBucketTest to inherit from VBucketTest, and
hence make use of all the same helper methods.

This also refactors the DefragmenterTest fixture so it inherits from
VBucketTest, as DefragmenterTest also used to use MockEPBucket and
hence needs updating to be able to setup its vbucket for testing.

Change-Id: I24da4098ea9da212284cd44a683a5e0e2e397fff
Reviewed-on: http://review.couchbase.org/75871
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Jim Walker <jim@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
4 years agoMB-23293: Remove seqno from mutation log 14/75614/4
olivermd [Thu, 23 Mar 2017 14:52:05 +0000 (14:52 +0000)]
MB-23293: Remove seqno from mutation log

This patch:
        - Removes the rowid(seqno) from the mutation log (v2)
        - Removes storage of the seqno from the access scanner
        - Changes the storage of rows in the access scanner to a vector
          from a linked list

Use of the sequence number stored in the access log by warmup was
removed by MB-20623 as it was unused. As such it makes sense not to
store it in the access log or the access scanner.

As the access scanner is the only other component that makes use of the
mutation log, the sequence number was removed from V2 of
MutationLogEntry since it is unused.

As MutationLogEntryV2 as yet doesn't appear in a release "in the wild"
it seems unnecessary to increment MutationLogEntry to V3.

This commit reduces the memory overhead per item stored in the access
scanner as can be seen below in the before and after comparison.

Before:
$ ./ep_engine_benchmarks
Run on (8 X 2200 MHz CPU s)
2017-03-29 09:50:07
------------------------------------------------------------------------------------------------------------------
Benchmark                                                           Time           CPU Iterations
 UserCounters...------------------------------------------------------------------------------------------------------------------
AccessLogBenchEngine/MemoryOverhead/0/32768/min_time:0.000      31682 ns       6000 ns          1 MaxBytesAllocatedPerItem=0 Control
AccessLogBenchEngine/MemoryOverhead/1/32768/min_time:0.000   28544445 ns   22181000 ns          1 MaxBytesAllocatedPerItem=106 AccessScanner
AccessLogBenchEngine/MemoryOverhead/0/65536/min_time:0.000       7550 ns       8000 ns          1 MaxBytesAllocatedPerItem=0 Control
AccessLogBenchEngine/MemoryOverhead/1/65536/min_time:0.000   57640028 ns   49224000 ns          1 MaxBytesAllocatedPerItem=104 AccessScanner

After:
$ ./ep_engine_benchmarks
Run on (8 X 2200 MHz CPU s)
2017-03-29 09:50:48
------------------------------------------------------------------------------------------------------------------
Benchmark                                                           Time           CPU Iterations
 UserCounters...------------------------------------------------------------------------------------------------------------------
AccessLogBenchEngine/MemoryOverhead/0/32768/min_time:0.000      39719 ns       6000 ns          1 MaxBytesAllocatedPerItem=0 Control
AccessLogBenchEngine/MemoryOverhead/1/32768/min_time:0.000   21993107 ns   17858000 ns          1 MaxBytesAllocatedPerItem=82 AccessScanner
AccessLogBenchEngine/MemoryOverhead/0/65536/min_time:0.000       7347 ns       7000 ns          1 MaxBytesAllocatedPerItem=0 Control
AccessLogBenchEngine/MemoryOverhead/1/65536/min_time:0.000   38534385 ns   31009000 ns          1 MaxBytesAllocatedPerItem=80 AccessScanner

Change-Id: I4becdc56a54518cb914298ce99709b8886fe6f35
Reviewed-on: http://review.couchbase.org/75614
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23604: set_with_meta XATTR handling 60/75960/6
Jim Walker [Wed, 29 Mar 2017 08:34:17 +0000 (09:34 +0100)]
MB-23604: set_with_meta XATTR handling

For xattr writes, the checkUTF8Json function is now invoked on the user
data, not the entire blob.

Change-Id: Ia6398a9b96303fadda9d560ba8d39eabb07818ef
Reviewed-on: http://review.couchbase.org/75960
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoModify access scanner memory benchmark 58/75958/2
olivermd [Wed, 29 Mar 2017 08:44:19 +0000 (09:44 +0100)]
Modify access scanner memory benchmark

This change changes, slightly, what we are measuring with the benchmark.
We now record the maximum overhead per item actually stored in the
access scanner at any one time (the alog_max_stored_items) rather than
than the maximum overhead per item for all the items set during the
test. This is so we can more closely monitor the changes in memory
overhead for each item stored.

Change-Id: I8b5a67ca734fe59dc2f03a6e8c6b1fb11cd388e3
Reviewed-on: http://review.couchbase.org/75958
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoAdd RBAC support to cbstats and cbepctl 04/75604/7
James Harrison [Wed, 22 Mar 2017 11:45:59 +0000 (11:45 +0000)]
Add RBAC support to cbstats and cbepctl

Change-Id: I40ba8afaf68f3b6db2024fdcb9aab424a5dff3de
Reviewed-on: http://review.couchbase.org/75604
Tested-by: James Harrison <james.harrison@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoUse the connection logger for dcp disconnect 93/75793/4
Trond Norbye [Mon, 27 Mar 2017 16:02:39 +0000 (18:02 +0200)]
Use the connection logger for dcp disconnect

Change-Id: Ibe36fbcd3d74941e0b455c077fa005ee989573ac
Reviewed-on: http://review.couchbase.org/75793
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoRefactor: get rid of default case in enum switch 60/75860/4
Trond Norbye [Tue, 28 Mar 2017 07:24:11 +0000 (09:24 +0200)]
Refactor: get rid of default case in enum switch

Change-Id: I735277790fcd463cfe3bce09acc6d02fb4b082e0
Reviewed-on: http://review.couchbase.org/75860
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoCheck return value for couchstore_db_info 59/75859/4
Trond Norbye [Tue, 28 Mar 2017 07:17:10 +0000 (09:17 +0200)]
Check return value for couchstore_db_info

We could end up using uninitialized variables

Change-Id: Id56937841198fbf0b7cbbf041a91644cff5b67cf
Reviewed-on: http://review.couchbase.org/75859
Reviewed-by: Oliver Downard <oliver.downard@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-22613: [Ephemeral] Do not update disk stats during chk clear/merge 14/75914/2
Manu Dhundi [Tue, 28 Mar 2017 22:23:06 +0000 (15:23 -0700)]
MB-22613: [Ephemeral] Do not update disk stats during chk clear/merge

Currently we update disk stats like disk queue size during checkpoint
clear and when checkpoints are merged.

This must not happen in ephemeral buckets as disk stats do not have
any significance.

Change-Id: Ic03a76dcbac71451f7a4e5430d8ea85e82e29908
Reviewed-on: http://review.couchbase.org/75914
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
4 years ago[Ephemeral]: Maintain opsCreate, opsUpdate and opsDelete stats 11/75911/2
Manu Dhundi [Tue, 28 Mar 2017 20:53:34 +0000 (13:53 -0700)]
[Ephemeral]: Maintain opsCreate, opsUpdate and opsDelete stats

'opsCreate', 'opsUpdate' and 'opsDelete' stats are necessary for
testing, debugging purposes.

Change-Id: I9c76d911618c2a3ab2697fb71a53836b67da5aee
Reviewed-on: http://review.couchbase.org/75911
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoMB-23505: Notify high priority rebalance requests in ephemeral buckets 16/75716/7
Manu Dhundi [Tue, 28 Mar 2017 18:06:12 +0000 (11:06 -0700)]
MB-23505: Notify high priority rebalance requests in ephemeral buckets

High priority requests are async requests made by modules like ns-server
during rebalance. The request is for a response from the vbucket when it
'sees' beyond a certain sequence number or when a certain checkpoint is
persisted. Depending on the vbucket type, the meaning 'seeing' a sequence
number changes. That is, it could mean persisted in case of EPVBucket and
added to the sequenced data structure in case of EphemeralVBucket.

In ephemeral buckets we must notify high priority seqno requests as well.
Wwe notify when the requested seqno (or beyond) is put on the sequenced
data structure (say list) by the respective vbucket.

Change-Id: Ie2e255d762ac7fa3d5c2a6c817e62cf394b31406
Reviewed-on: http://review.couchbase.org/75716
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Dave Rigby <daver@couchbase.com>
4 years agoMB-23550: return NULL from initScanContext() on failure 85/75785/3
Dave Rigby [Mon, 27 Mar 2017 14:19:51 +0000 (15:19 +0100)]
MB-23550: return NULL from initScanContext() on failure

The callers of this function treat a NULL return value as an error
(which is handled), however we currenlty throw an exception from it on
error, which isn't caught. Change to use the same method of
propagating the failure and simply return NULL (after logging the
failure).

Change-Id: I7fd7b6cff0cffee45d05ad365bc0e9e73efafd14
Reviewed-on: http://review.couchbase.org/75785
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
4 years agoMB-23505: Do not add items to vb backfill queue in Ephemeral buckets 74/75674/7
Manu Dhundi [Mon, 27 Mar 2017 17:00:31 +0000 (10:00 -0700)]
MB-23505: Do not add items to vb backfill queue in Ephemeral buckets

Vb level backfill queue is for items in a huge snapshot (disk backfill
snapshots from DCP are typically huge) that could not be fit on a
checkpoint. They update all stats, checkpoint seqno, but are not put
on checkpoint and are directly persisted from the queue.

In ephemeral buckets we must not add backfill items from DCP (on
replica vbuckets), to the vb backfill queue because we have put them on
linkedlist already. Also we do not have the flusher task to drain the
items from that queue.
(Unlike checkpoints, the items in this queue is not cleaned up
 in a background cleanup task).

But we must be careful to update certain stats and checkpoint seqno
like in a regular couchbase bucket.

Further, this commit adds a stat 'backfill_queue_size' to track
the items in the vb backfill queue.

Change-Id: I8e513173a345d4c84d2e01778bad92d570d4754b
Reviewed-on: http://review.couchbase.org/75674
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-16181: Refactor buffered backfill accounting. 33/75433/11
Jim Walker [Mon, 20 Mar 2017 15:44:49 +0000 (15:44 +0000)]
MB-16181: Refactor buffered backfill accounting.

The general aim of this commit is to be more abstract in how we
account the resources used during in backfill.

Currently the assumption is that MutationResponses will be the only
object to account for, thus on the read increment we use Item::size
and on the send decrement we access the Item::size by casting to a
MutationResponse.

With Collections, SystemEvents will be part of the backfill and should
be accounted for, but a SystemEvent in the ActiveStream's readyQueue
means we need a cleaner way of asking a DcpResponse for its size, as
opposed to more if (event == ...) {static_cast...}

1. Add a method to the abstract DcpResponse class which by default
   returns 0, so for example a SnapShotMarker can be unconditionally
   applied to the backfill accounting without changing it.

2. Implement in MutationResponse and (upstream) SystemEvent the method
   so it returns a value that better represents the resources that
   DcpResponse is holding, that is the size of the Item + any extra
   resources.

Change-Id: I7c57572e2f09ed80d84a7505b36734def5c75580
Reviewed-on: http://review.couchbase.org/75433
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-16181: Collections DcpConsumer 09/74609/30
Jim Walker [Sun, 5 Mar 2017 14:18:08 +0000 (14:18 +0000)]
MB-16181: Collections DcpConsumer

Add the functional paths that allow a replica VBucket's manifest to
be updated from incoming DCP messages. Note that the DCP protocol
has not been written, but the API defined here shows what data needs
to be transfered over-the-wire.

Add a unit test to use MockDcpConsumer to drive a create and delete

Change-Id: Ida8171b2cde104ccd452fafcd6193a72455b404b
Reviewed-on: http://review.couchbase.org/74609
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoFix configuration valueChanged bug 58/75258/10
James Harrison [Thu, 16 Mar 2017 10:20:12 +0000 (10:20 +0000)]
Fix configuration valueChanged bug

Changing a ssize_t parameter called the validator for size_t, and /not/
for ssize_t.

Change-Id: Iac7cf33c9f6257a45f62f7f9240fa6c1cf306cdd
Reviewed-on: http://review.couchbase.org/75258
Reviewed-by: Dave Rigby <daver@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23435: [Ephemeral] Add auto_delete_count statistics 83/75683/4
Dave Rigby [Fri, 24 Mar 2017 12:33:45 +0000 (12:33 +0000)]
MB-23435: [Ephemeral] Add auto_delete_count statistics

Add per-vBucket and aggregated statistics for the number of documents
which have been automatically deleted by the item pager, when
ephemeral_full_policy is set to 'auto_delete'

Change-Id: I5ffa18a74b1632f75628792ee69d635087e9db4c
Reviewed-on: http://review.couchbase.org/75683
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: David Haikney <david.haikney@couchbase.com>
4 years agoMB-23435: [Ephemeral] Add aggregrated vBucket seqlist stats 24/75624/5
Dave Rigby [Thu, 23 Mar 2017 17:41:57 +0000 (17:41 +0000)]
MB-23435: [Ephemeral] Add aggregrated vBucket seqlist stats

Add aggregated statistics (all active, replica, pending vbuckets) for
Ephemeral buckets, as part of the 'all' stats set:

    vb_active_seqlist_count
    vb_active_seqlist_deleted_count
    vb_active_seqlist_read_range_count
    vb_active_seqlist_stale_count
    vb_active_seqlist_stale_value_bytes
    vb_active_seqlist_stale_metadata_bytes

(Same for 'replica' and 'pending').

Change-Id: I46f0fe0a0486d8c429e04c58df139152cce81f95
Reviewed-on: http://review.couchbase.org/75624
Tested-by: Build Bot <build@couchbase.com>
Reviewed-by: Manu Dhundi <manu@couchbase.com>
4 years agoMB-23435: Refactor - move VBucketCountVisitor to own file 23/75623/3
Dave Rigby [Wed, 22 Mar 2017 15:33:57 +0000 (15:33 +0000)]
MB-23435: Refactor - move VBucketCountVisitor to own file

Move VBucketCountVisitor from ep_engine.{cc,h} to it's own files -
vb_count_visitor.{cc,h}. This is to facilitate derived classes of
VBucketCountVisitor, without such classes having to include all of
ep_engine.h.

Change-Id: I472eba08e0c690a4926d00d87feb63645688af7b
Reviewed-on: http://review.couchbase.org/75623
Reviewed-by: David Haikney <david.haikney@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>
4 years agoMB-23435: Refactor vBucket aggregated stats 22/75622/3
Dave Rigby [Wed, 22 Mar 2017 14:53:25 +0000 (14:53 +0000)]
MB-23435: Refactor vBucket aggregated stats

Move the code for outputting vBucket aggregated statistics from
EventuallyPersistentEngine to KVBucket. This is in preparation for
having the concrete subclasses of KVBucket - specificially
EphemeralBucket - add statistics specific to Ephemeral buckets.

This has no effect on the stats output, just a pure refactor.

Change-Id: I084dd503388b064bd625e41af7bdd248361d1c00
Reviewed-on: http://review.couchbase.org/75622
Reviewed-by: James Harrison <james.harrison@couchbase.com>
Reviewed-by: David Haikney <david.haikney@couchbase.com>
Tested-by: Build Bot <build@couchbase.com>