devilutionX/docs/benchmarking.md

Benchmarking

On Linux, use the tools/linux_reduced_cpu_variance_run.sh wrapper to reduce CPU variance.

Timedemo:

tools/linux_reduced_cpu_variance_run.sh tools/measure_timedemo_performance.py -n 5 --binary build-rel/devilutionx

Individual benchmarks (built when BUILD_TESTING is ON):

tools/build_and_run_benchmark.py clx_render_benchmark

You can pass arguments to the benchmark binary with --, e.g.:

tools/build_and_run_benchmark.py clx_render_benchmark -- --benchmark_repetitions=5

The tools/build_and_run_benchmark.py script basically does something like this:

{ [ -d build-reld ] || cmake -S. -Bbuild-reld -DCMAKE_BUILD_TYPE=RelWithDebInfo; } && \
cmake --build build-reld --target clx_render_benchmark && \
tools/linux_reduced_cpu_variance_run.sh build-reld/clx_render_benchmark

See tools/build_and_run_benchmark.py --help for more information.

You can also profile your benchmarks.

Comparing benchmark runs

You can use compare.py from Google Benchmark to compare 2 benchmarks.

First, install the tool:

git clone git@github.com:google/benchmark.git ~/google-benchmark
cd ~/google-benchmark/tools
pip3 install -r requirements.txt
cd -

Then, build the 2 binaries that you'd like to compare. For example:

BASELINE=master
BENCHMARK=dun_render_benchmark

git checkout "$BASELINE"
tools/build_and_run_benchmark.py -B "build-reld-${BASELINE}" --no-run "$BENCHMARK"
git checkout -

tools/build_and_run_benchmark.py --no-run "$BENCHMARK"

tools/linux_reduced_cpu_variance_run.sh ~/google-benchmark/tools/compare.py -a benchmarks \
  "build-reld-${BASELINE}/${BENCHMARK}" "build-reld/${BENCHMARK}" \
  --benchmark_repetitions=10