[BP] MB-18580: Wait for VB state to be persisted before starting tests 98/65298/5
authorDave Rigby <daver@couchbase.com>
Wed, 4 May 2016 09:49:59 +0000 (10:49 +0100)
committerDave Rigby <daver@couchbase.com>
Mon, 4 Jul 2016 08:58:42 +0000 (08:58 +0000)
commit7d9dc5d4bb21cc80cc91b5496a6c30a30c0a5f08
tree924af820a0a7e699e847de66f997468f17a3d6ea
parentc509624bda146dfdc26ebaf8044657ecc1160912
[BP] MB-18580: Wait for VB state to be persisted before starting tests

Intermittent test failures (across multiple tests) have been seen
where we fail to read the number of items in vbucket disk file:

    terminate called after throwing an instance of 'std::invalid_argument'
    what(): CouchKVStore::getDbFileInfo: Failed to open database file for vBucket = 1 rev = 1 with error:no such file

The issue is that we do not correctly wait for the vBucket files to be
created before starting a test. We /attempt/ to wait in test_setup,
waiting for ep_vb_snapshot_total to be non-zero, however this stat is
not updated when vBuckets are written to disk, instead only when the
vb state snapshot occurs.

To fix this, create a new histogram stat - ep_vb_persist_state_total -
which records how long the actual persist takes (and counts then at
the same time). Change test_setup to check for this stat becoming 1
before continuing.

Results in two new stats:

* disk_persist_vbstate - timing histogram of how long vbState
                          operations took.

* ep_persist_vbstate_total - count of how many VBStatePersists have
                             occurred.

Change-Id: Ic24e6cdb51a98ea6fa65005158242bfcf44225d0
Reviewed-on: http://review.couchbase.org/65298
Reviewed-by: Dave Rigby <daver@couchbase.com>
Well-Formed: buildbot <build@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
docs/stats.org
src/ep.cc
src/ep_engine.cc
src/stats.h
tests/ep_testsuite.cc