Packages that are required by modules created with @jog

Generated modules will access these via Base.loaded_modules

benchmark_dir(pkg::Module)
benchmark_dir(pkg::PackageSpec)
benchmark_dir(project_path::String)

Returns the absolute path of the benchmarks folder for pkg.

Supported Benchmark Directories:

• PKG_DIR/benchmark

build_module(s::BenchModule)

Construct a module wrapping the BenchmarkGroup defined by s::BenchModule

Sets up an isolated benchmarking environment and then runs the following:

using PkgJogger
using PkgName
jogger = @jog PkgName
result = JogPkgName.benchmark()
filename = JogPkgName.save_benchmarks(result)
@info "Saved benchmarks to $filename"

Where PkgName is the name of the package in the current directory

judge(new, old; metric=Statistics.median, kwargs...)

Compares benchmarking results from new vs old for regressions/improvements using metric as a basis. Additional kwargs are passed to BenchmarkTools.judge

Effectively a convenience wrapper around load_benchmarks and BenchmarkTools.judge

new and old can be any one of the following: - Filename of benchmarking results saved by PkgJogger - A Dict as returned by PkgJogger.load_benchmarks(filename) - A BenchmarkTools.BenchmarkGroup with benchmarking results

load_benchmarks(filename::String)::Dict

Load benchmarking results from filename

Prior to v0.4 PkgJogger saved results as *.json.gz instead of *.bson.gz. This function supports both formats. However, the *.json.gz format is deprecated, and may not support all features.

locate_benchmarks(pkg::Module)
locate_benchmarks(path::String, name=String[])

Returns a list of BenchModule for identified benchmark files

save_benchmarks(filename, results::BenchmarkGroup)

Save benchmarking results to filename.bson.gz for later analysis.

File Contents

• Julia Version, Commit and Commit date
• System Information
• Timestamp
• Benchmarking Results
• Git Commit, 'Is Dirty' status and author datetime
• PkgJogger Version used to save the file

File Format:

Results are saved as a gzip compressed BSON file and can be loaded with PkgJogger.load_benchmarks

test_benchmarks(s::BenchmarkGroup)

Runs a @testsuite for each benchmark in s once (One evaluation of the benchmark's target) Sub-benchmark groups / benchmarks are recursively wrapped in @testsuites for easy identification.

benchmarks are marked as "passing" if they don't error during evaluation.

tune!(group::BenchmarkGroup, ref::BenchmarkGroup; verbose::Bool=false)

Tunes a BenchmarkGroup, only tunning benchmarks not found in ref, otherwise reuse tuning results from the reference BenchmarkGroup, by copying over all benchmark parameters from ref.

This can reduce benchmarking runtimes significantly by only tuning new benchmarks. But does ignore the following: - Changes to benchmarking parameters (ie. memory_tolerance) between group and ref - Significant changes in performance, such that re-tunning is warranted - Other changes (ie. changing machines), such that re-tunning is warranted

@jog PkgName

Creates a module named JogPkgName for running benchmarks for PkgName.

Most edits to benchmark files are correctly tracked by Revise.jl. If they are not, re-run @jog PkgName to fully reload JogPkgName.

Revise must be loaded before calling @jog PkgName in order for edits to be automatically tracked.

Methods

• suite Return a BenchmarkGroup of the benchmarks for PkgName
• benchmark Warmup, tune and run the suite
• run Dispatch to BenchmarkTools.run(suite(), args...; kwargs...)
• save_benchmarks Save benchmarks for PkgName using an unique filename

Isolated Benchmarks

Each benchmark file, is wrapped in it's own module preventing code loaded in one file from being visible in another (unless explicitly included).

Example

using AwesomePkg, PkgJogger
@jog AwesomePkg
results = JogAwesomePkg.benchmark()
file = JogAwesomePkg.save_benchmarks(results)

@test_benchmarks PkgName

Collects all benchmarks for PkgName, and test that they don't error when ran once.

Example

julia> using PkgJogger, Example

julia> @test_benchmarks Example
│ Test Summary:  | Pass  Total
│ bench_timer.jl |    2      2
[...]

Testing

Each benchmark is wrapped in a @testset, run only once, and marked as passing iff no errors are raised. This provides a fast smoke test for a benchmarking suite, and avoids the usual cost of tunning, warming up and collecting samples accrued when actually benchmarking.

Benchmark Loading

Locating benchmarks for testing is the same as for @jog and can be examined using PkgJogger.locate_benchmarks.