Aerospike Benchmark Tool

This project contains the files necessary to build the C client benchmarking tool. This program is used to insert data and generate load emulating real-world usage patterns of the database.

Wiki

For more information on how to use the benchmark tool and configure it to your needs, visit the wiki here. Object specs, synthetic data generators and CDT workloads are documented in the sections below.

Get started

Dependencies

Asbackup builds the Aerospike C Client as a submodule. Make sure all the C clients build dependencies are installed before starting the asbackup build.

Additional external dependencies:

• OpenSSL (libssl and libcrypto)
• libyaml-devel
• libev, libuv, or libevent, if an event library is used
• If on macOS install via brew

This project uses git submodules, so you will need to initialize and update submodules before building this project.

$ git submodule update --init --recursive

Build

To build the benchmark tool, run:

make [EVENT_LIB=libev|libuv|libevent]

with EVENT_LIB matching the event library used when the C client was compiled, if one was used (it is necessary to build with an event library to use async commands). If an event library is defined, it must be installed separately. Event libraries usually install into /usr/local/lib. Most operating systems do not search /usr/local/lib by default. Therefore, the following LD_LIBRARY_PATH setting may be necessary:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

Now, the benchmark executable will be located at target/asbench.

The synthetic data dictionaries are generated by scripts/gen_synth_data.py (make synth-data) and committed, so a normal build needs no network access or Python. Regenerate them only when bumping the pinned upstream commits.

Build with static linkage

To statically link the benchmark tool to external libraries, pass <LIB_NAME>_STATIC_PATH, set to the directory containing the respective static libraries, as an argument to make. To get these, you'll likely need to compile the library itself from source.

For example, for static linking against OpenSSL, with libssl.a and libcrypto.a in the directory /usr/local/lib, compile with OPENSSL_STATIC_PATH=/usr/local/lib.

Running Tests and Coverage

To run the unit tests, call

make test [EVENT_LIB=libev|libuv|libevent]

To generate coverage data for the unit tests, run

make coverage [EVENT_LIB=libev|libuv|libevent]

note: this will rerun the tests.

To view coverage data in the console, run

make report

The integration tests start a two node Aerospike cluster in Docker and need python3. Each test session picks a random free port range and its own work directory, waits until both nodes have joined and migrations are done, and removes its containers when it exits. A session that was killed is cleaned up by the next one, so the tests can run next to an Aerospike server already listening on port 3000.

The dictionary generator has its own tests:

python3 -m unittest scripts/test_gen_synth_data.py

Example usage

To run a random read/update workload for 30 seconds, run:

target/asbench --workload RU,50 --duration 30

To:

• Connect to localhost:3000 using namespace "test".
• Read 80% and write 20% of the time using 8 concurrent threads.
• Use 1000000 keys and 1400 length byte array values using a single bin.
• Timeout after 50ms for reads and writes.
• Restrict transactions/second to 2500.

target/asbench -h 127.0.0.1 -p 3000 -n test -k 1000000 -o B1400 -w RU,80 -g 2500 -T 50 -z 8

To:

• Benchmark asynchronous methods using 1 event loop.
• Limit the maximum number of concurrent commands to 50.
• Use and 50% read 50% write pattern.

target/asbench -h 127.0.0.1 -p 3000 -n test -k 1000000 -o S50 -w RU,50 --async --asyncMaxCommands 50 --eventLoops 1

To benchmark a read-based LRU cache, where a read in the last 80% of a record's 1 hour TTL resets it to 1 hour (server 7.1+). The namespace needs a non-zero nsup-period (or allow-ttl-without-nsup true), otherwise the server rejects every write with a TTL:

target/asbench -h 127.0.0.1 -n test -k 1000000 -w RU,90 -e 3600 --read-touch-ttl-percent 80

To insert 1,000,000 realistic customer records with named bins, reproducibly:

target/asbench -h 127.0.0.1 -n test -s customers -w I -k 1000000 --seed 42 \
  -o "id=@uuid, first=@first_name, last=@last_name, email=@email, age=@int(18,90), city=@city, loc=@geojson"

Command line usage can be read with:

target/asbench --help

Object spec

The object spec (-o, or object-spec: in a workload stages file) describes the bins of every record written. It is a comma separated list of bin types:

Any top-level bin can be named with <name>=, for example first=@first_name. Names are 1 to 15 characters of letters, digits, _ or -, start with a letter or _, and must be unique. Unnamed bins keep the <bin>, <bin>_2, ... names derived from -b (default testbin). A repeated named bin such as tags=3*@word produces tags, tags_2 and tags_3. --read-bins and --write-bins still select bins by position (1-based), whatever their names.

Synthetic data generators

Generators produce realistic values from dictionaries compiled into the binary (US English names, cities, streets, companies, products, words, and more). They can be used anywhere a scalar is allowed: as top-level bins, list elements, map values, and map keys when they produce strings or integers.

[name=][N*]@generator[(arg, arg, ...)]

Arguments are integers, doubles or double-quoted strings. @pick also takes weighted items ("a":70).

Notes:

• @date and @timestamp bounds are epoch seconds or "YYYY-MM-DD" dates; both default to 2020-01-01 through 2029-12-31, and the default @date format is %Y-%m-%d.
• @lat, @lon, @geojson and @geo_circle default to the continental US (latitude 24 to 49, longitude -125 to -66), matching the US city and state data. @geo_circle radii default to 100 to 5000 meters.
• @fmt placeholders take any string, integer or double generator with its arguments, e.g. @fmt("SKU-#{int(1000,9999)}"). Quotes inside a template are escaped: @fmt("#{pick(\"x\",\"y\")}"). A #{double(...)} placeholder is rendered into the string with 2 decimals (#{double(1,500)} gives 316.98), so its bounds must be within ±9e16.
• A generator used as a map key must produce strings or integers. Map keys are drawn at random and a duplicate is redrawn until the map has all its entries, so a map cannot ask for more unique keys than the generator has distinct values ({60*@state:I} is rejected because there are 50 states).
• An object spec that contains generators generates a new record for every write, as if -R were given.

Worked examples:

# customer profile
-o "id=@uuid, first=@first_name, last=@last_name, email=@email, phone=@phone, age=@int(18,90), city=@city, state=@state_abbr, zip=@zip, tier=@pick(\"basic\":80,\"gold\":15,\"platinum\":5)"

# e-commerce order with a list of line-item maps
-o "order_id=@uuid, placed=@timestamp(\"2024-01-01\",\"2025-12-31\"), items=[3*{\"sku\":@product,\"qty\":@int(1,9),\"price\":@double(1,500)}]"

# IoT device reading with a native GeoJSON location
-o "device=@uuid, mac=@mac, loc=@geojson(37.70,37.80,-122.50,-122.40), battery=@int(0,100), seen=@now"

Reproducible data (--seed)

With --seed <n>, every record written by insert and update workloads is a pure function of the seed, the record key and the object spec. Each bin gets its own random stream derived from (seed, key, bin number), so:

• the same key always gets the same bins, whatever the thread count, --batch-write-size or --async;
• a partial write with --write-bins produces exactly the values the full record would have in those bins;
• a load can be verified later by regenerating it with the same seed.

@now is the only generator that does not depend on the seed. CDT workloads draw values from a per-thread stream seeded by --seed and the thread number, so they are reproducible with a single thread (-z 1). Without --seed each thread uses its own independently seeded random stream.

GeoJSON

@geojson and @geo_circle write native GeoJSON particles, not strings, so the bins can be indexed and queried directly. They also work inside lists and maps, which can be indexed with LIST or MAPVALUES geo indexes.

asbench -n test -s devices -w I -k 100000 -o "loc=@geojson(37.70,37.80,-122.50,-122.40)"

aql> CREATE INDEX loc_idx ON test.devices (loc) GEO2DSPHERE
aql> SELECT * FROM test.devices WHERE loc WITHIN CAST('{"type":"AeroCircle","coordinates":[[-122.45,37.75],2000]}' AS GEOJSON)

With the Python client:

from aerospike import predicates as p
client.index_geo2dsphere_create("test", "devices", "loc", "loc_idx")
query = client.query("test", "devices")
query.where(p.geo_within_radius("loc", -122.45, 37.75, 2000))
nearby = query.results()

CDT workloads (-w C, CI, CK)

CDT workloads exercise list and map operations on records instead of whole-record puts. There are three variants:

• C (put): each write appends the generated list to list bins and puts the generated map into map bins.
• CI (increment): like C, but map bins get one map_increment per generated entry, for counters and leaderboards.
• CK (key): like C, but each map read fetches one generated key instead of a range, for session and attribute maps.

For every top-level bin of the object spec, one operate() call per key does:

-w C[I|K][,<read percent>[,<cap>[,<read count>]]]

Defaults are 50% reads, cap 0 (no trimming) and a read count of 10. CI needs map values that are integers, doubles, or @int/@double/@timestamp generators, and increments at most 1000 map entries per write. Records are created by the first write, so a preload is optional; for CI skip the preload so maps are created value-ordered. Lists use the unordered policy (appends are O(1)), C and CK maps are key ordered, and CI maps are key-value ordered so rank operations stay cheap.

Without @generators and without -R, the write operations are built once per thread and reused for every transaction, enabling higher asbench throughput. With generators, a fresh payload is generated for every write. Batch sizes, --read-bins and --write-bins are not supported by CDT workloads. With cap 0, collections grow without bound and long runs eventually fail with record-too-big errors, so set a cap for long runs.

Examples:

# activity feed: append one event per write, keep the last 100, read the last 10
asbench -n test -s feed -k 1000000 -w C,80,100,10 -t 60 -z 16 \
  -o "events=[1*{\"ts\":@timestamp,\"type\":@pick(\"view\":70,\"click\":25,\"buy\":5),\"sku\":@product}], updated=@now"

# time series per device: keep one day of minute samples, read the last hour
asbench -n test -s metrics -k 200000 -w C,20,1440,60 -t 120 \
  -o "samples=[1*[@timestamp,@double(0,100)]], updated=@now"

# session / attribute maps: put 3 attributes per write, cap at 50 keys, read one by key
asbench -n test -s sessions -k 5000000 -w CK,90,50 -t 60 \
  -o "attrs={3*@pick(\"theme\",\"lang\",\"tz\",\"plan\",\"region\",\"last_page\"):@word}"

# leaderboards: increment 5 player scores per write, keep the top 1000, read the top 10
asbench -n test -s boards -k 10000 -w CI,30,1000,10 -t 60 -o "scores={5*@username:@int(1,100)}"

# geo-tagged check-ins: a native GeoJSON point in every list element
asbench -n test -s checkins -k 500000 -w C,50,200,20 -t 60 \
  -o "visits=[1*{\"at\":@timestamp,\"loc\":@geojson,\"city\":@city}]"

Examples

Ready to run workload stage files live in examples/synth. Run any of them with:

asbench -h 127.0.0.1 -n test --workload-stages examples/synth/<file>.yaml

Coming from the professional services data synthesizer

The last five files are the asbench versions of the tools/create (datasynth) examples: alltypes.json, maptest.json, simple.json with example2.json, deletes.json and transformer.json. Its data types map to asbench as follows:

Differences to keep in mind: record keys are always integers (put composite ids in a bin with @fmt), stages run one after another rather than concurrently (use RU, RUD and C read/write mixes instead), and each write op has one object spec rather than several randomly chosen bin specs.

Performance

Generation is designed to stay far below network and server cost:

• dictionaries are sorted, NUL separated string pools with offset arrays in read-only memory, and a word is picked with one multiply and shift;
• dictionary words are wrapped without copying, and composed strings (emails, streets, templates) are built in a single allocation that holds both the value and its text;
• every thread has its own random stream, and bin names are resolved once per stage instead of formatting them on every write.

The hidden --gen-bench <n> flag measures generation in-process, without a server. It generates n records from the first stage's object spec through the same code path writes use, then repeats with -z threads and reports the raw dictionary pick cost and the average msgpack payload size:

asbench --gen-bench 3000000 -z 6 -o "id=@uuid, first=@first_name, last=@last_name, email=@email, age=@int(18,90)"

Single thread results on an Apple M3 Pro, release build. Each record includes creating and destroying the as_record, which is about 60 ns of the numbers below.

A raw dictionary pick costs 1.4 ns. With 6 threads (one per performance core) the 5-bin customer record reaches about 21M to 22M records/s, over 100M bins/s. Multi-thread numbers on macOS vary with scheduling between performance and efficiency cores.

Dictionaries and licensing

The dictionaries are derived from the English word lists of gofakeit and Faker, both MIT licensed, at the upstream commits pinned in scripts/gen_synth_data.py. The script keeps printable ASCII entries only, removes duplicates, sorts every table, and writes src/include/synth_data_tables.h and src/main/synth_data_*.c. It covers about 12,000 words in 26 tables, including 3,188 first names, 786 last names, 933 cities, 552 street names, 244 countries, 528 companies and 783 products. That compiles to about 150 KB of read-only data.

make synth-data         # regenerate from the pinned commits
make synth-data-check   # verify the committed tables match the pinned commits

The upstream license texts are included in LICENSE.md.

Limitations

• Values are ASCII only; entries with other characters are dropped from the dictionaries.
• Bin names are limited to 15 characters.
• Address fields are drawn independently, so a city, state and zip in one record do not necessarily match.
• Generators ignore --compression-ratio, which only applies to B bins.
• CDT workloads do not support batch sizes, --read-bins or --write-bins.
• @now depends on the clock, so it is not reproducible with --seed.