MB-20623: Warmup: Process the access log in chunks 45/68045/7
authorDave Rigby <daver@couchbase.com>
Tue, 23 Aug 2016 14:28:07 +0000 (15:28 +0100)
committerDave Rigby <daver@couchbase.com>
Fri, 30 Sep 2016 15:39:25 +0000 (15:39 +0000)
commite585bd843cadc0356cdf40e0ec1534ee2f46bee7
treeced6bf913b19773219da80f515165cd36fd6c868
parent65a2ae3ab878c1b604aa8983353c0ffbf6e66bf2
MB-20623: Warmup: Process the access log in chunks

Instead of loading the entire Access log into memory, and then
applying in one (potentially very large) getMulti; process it in
warmup_batch_size chunks (default 10,000 items).

This sigificantly reduces the amount of temporary memory consumed
during warmup, which in turn means this memory can be used for
document values.

Results: in the following workload we manage to load 19.2M items (or
4.2GB) into memory during warmup, where previously only two items
(yes, *two*) were warmed up due to the size of the temporary data
structures - because the temporary data structures consumed ~4.2GB.

* 1 bucket, 10,000MB quota, 1 node.
* 30M items, 300bytes each; giving a residency ratio of ~60%. Dataset
  generated by:

    cbc-pillowfight -U couchbase://localhost:12000 -I 30000000 -m 300 -M 300 -t16

Change-Id: I511b70d5ea9c9c6b9556249a936030a67bf70c02
Reviewed-on: http://review.couchbase.org/68045
Reviewed-by: Daniel Owen <owend@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
configuration.json
src/ep.cc
src/mutation_log.cc
src/mutation_log.h
src/warmup.cc
src/warmup.h
tests/module_tests/mutation_log_test.cc