Packages that are required by modules created with @jog

Generated modules will access these via Base.loaded_modules

benchmark_dir(pkg::Module)
benchmark_dir(pkg::PackageSpec)
benchmark_dir(project_path::String)

Returns the absolute path of the benchmarks folder for pkg.

Supported Benchmark Directories:

• PKG_DIR/benchmark

build_module(name, file)

Construct a module wrapping the BenchmarkGroup defined by file with name

Sets up an isolated benchmarking environment and then runs the following:

using PkgJogger
using PkgName
jogger = @jog PkgName
result = JogPkgName.benchmark()
filename = JogPkgName.save_benchmarks(result)
@info "Saved benchmarks to $filename"

Where PkgName is the name of the package in the current directory

judge(new, old; metric=Statistics.median, kwargs...)

Compares benchmarking results from new vs old for regressions/improvements using metric as a basis. Additional kwargs are passed to BenchmarkTools.judge

Effectively a convenience wrapper around load_benchmarks and BenchmarkTools.judge

new and old can be any one of the following: - Filename of benchmarking results saved by PkgJogger - A Dict as returned by PkgJogger.load_benchmarks(filename) - A BenchmarkTools.BenchmarkGroup with benchmarking results

load_benchmarks(filename::String)::Dict

Load benchmarking results from filename

locate_benchmarks(pkg::Module)
locate_benchmarks(bench_dir::String)

Returns a dict of name => filename of identified benchmark files

save_benchmarks(filename, results::BenchmarkGroup)

Save benchmarking results to filename.json.gz for later analysis.

File Contents

• Julia Version, Commit and Commit date
• System Information
• Timestamp
• Benchmarking Results
• Git Commit, 'Is Dirty' status and author datetime

File Format:

Results are saved as a gzip compressed JSON file and can be loaded with PkgJogger.load_benchmarks

@jog PkgName

Creates a module named JogPkgName for running benchmarks for PkgName.

Most edits to benchmark files are correctly tracked by Revise.jl. If they are not, re-run @jog PkgName to fully reload JogPkgName.

Methods

• suite Return a BenchmarkGroup of the benchmarks for PkgName
• benchmark Warmup, tune and run the suite
• run Dispatch to BenchmarkTools.run(suite(), args...; kwargs...)
• warmup Dispatch to BenchmarkTools.warmup(suite(), args...; kwargs...)
• save_benchmarks Save benchmarks for PkgName using an unique filename

Isolated Benchmarks

Each benchmark file, is wrapped in it's own module preventing code loaded in one file from being visible in another (unless explicitly included).

Example

using AwesomePkg, PkgJogger
@jog AwesomePkg
results = JogAwesomePkg.benchmark()
file = JogAwesomePkg.save_benchmarks(results)