Developer guide

Benchmarking

We use PkgBenchmark.jl to conduct benchmarks, coupled with BenchmarkCI.jl to automate benchmark runs with GitHub Actions on certain PRs. Currently, PRs will not run benchmarks by default. Only PRs with the label run benchmark will trigger benchmarks on push, adding a comment with the benchmark results.

Running benchmarks locally

With PkgBenchmark

To locally run benchmarks, you can use PkgBenchmark.jl with the following code:

# make sure PkgBenchmark and BenchmarkTools
# are installed globally before running it
using Meshes, PkgBenchmark
benchmarkpkg(Meshes)

It will include benchmark/benchmarks.jl and look for a BenchmarkGroup variable named SUITE, that it will run for you with a nice printing.

The full list of options is detailed in the PkgBenchmark documentation.

Manually

Sometimes you may prefer to run the suite manually, especially during interactive development. All you have to do is include the benchmark/benchmarks.jl file, and run the BenchmarkGroup suite:

include("benchmark/benchmarks.jl")
run(SUITE)

Limitations

Note that because GitHub Actions may use different runners between benchmarks, you are likely to see fluctuations and performance changes that may not always be relevant. This makes it somewhat unreliable for tracking regressions.

Benchmark results can be very useful for validation, for example when optimizing or modifying existing features, but you are encouraged to benchmark important changes locally.