PkgJogger

A Benchmarking Framework for Julia

PkgJogger makes benchmarking easy by providing a framework for running BenchmarkTool.jl benchmarks without the boilerplate.

Just write benchmarks

Create a benchmark/bench_*.jl file, define a suite and go!

using Benchmark
using AwesomePkg
suite = BenchmarkGroup()
suite["fast"] = @benchmarkable fast_code()

PkgJogger will wrap each benchmark/bench_*.jl in a module and bundle them into JogAwesomePkg

using AwesomePkg
using PkgJogger

# Creates the JogAwesomePkg module
@jog AwesomePkg

# Warmup, tune, and run all of AwesomePkg's benchmarks
JogAwesomePkg.benchmark()

Benchmark, Revise, and Benchmark Again!

PkgJogger uses Revise.jl to track changes to your benchmark/bench_*.jl files and reload your suite as you edit. No more waiting for benchmarks to precompile!

Tracked Changes:

  • Changing your benchmarked function
  • Changing benchmarking parameters (i.e. seconds or samples)
  • Adding new benchmarks

Current Limitations:

  • New benchmark files are not tracked
  • Deleted benchmarks will stick around
  • Renamed benchmarks will create a new benchmark and retain the old name

To get around the above, run @jog PkgName to get an updated jogger.

Continuous Benchmarking Baked In!

Install PkgJogger, run benchmarks, and save results to a *.json.gz with a one-line command.

julia -e 'using Pkg; Pkg.add("PkgJogger"); using PkgJogger; PkgJogger.ci()'

What gets done:

  • Add the package at pwd() to a temporary environment
  • If found, instantiate benchmark/Project.toml and add to the LOAD_PATH
  • Add PkgJogger to the environment and build JogPkgName for your package
  • Warmup, tune and run all benchmarks
  • Save Benchmarking results and more to a compressed *.json.gz file