benchpkgplot package_name [-r --rev <arg>] [-i --input-dir <arg>]
                          [-o --output-dir <arg>] [-n --npart <arg>]
                          [--format <arg>]

Plot the benchmarks of a package as created with benchpkg.

Arguments

• package_name: Name of the package.

Options

• -r, --rev <arg>: Revisions to test (delimit by comma).
• -i, --input-dir <arg>: Where the JSON results were saved (default: ".").
• -o, --output-dir <arg>: Where to save the plots results (default: ".").
• -n, --npart <arg>: Max number of plots per page (default: 10).
• --format <arg>: File type to save the plots as (default: "png").

comonicon_install(;kwargs...)

Install the CLI manually. This will use the default configuration in Comonicon.toml, if it exists. For more detailed reference, please refer to Comonicon documentation.

comonicon_install_path(;[yes=false])

Install the PATH and FPATH to your shell configuration file. You can use comonicon_install_path(;yes=true) to skip interactive prompt. For more detailed reference, please refer to Comonicon documentation.

benchpkg [package_name] [-r --rev <arg>]
                        [--url <arg>]
                        [--path <arg>]
                        [-o, --output-dir <arg>]
                        [-e, --exeflags <arg>]
                        [-a, --add <arg>]
                        [-s, --script <arg>]
                        [--bench-on <arg>]
                        [-f, --filter <arg>]
                        [--nsamples-load-time <arg>]
                        [--tune]
                        [--dont-print]

Benchmark a package over a set of revisions.

Arguments

• package_name: Name of the package. If not given, the package is assumed to be the current directory.

Options

• -r, --rev <arg>: Revisions to test (delimit by comma). Use dirty to benchmark the current state of the package at path (and not a git commit). The default is {DEFAULT},dirty, which will attempt to find the default branch of the package.
• --url <arg>: URL of the package.
• --path <arg>: Path of the package. The default is . if other arguments are not given.
• -o, --output-dir <arg>: Where to save the JSON results. The default is ..
• -e, --exeflags <arg>: CLI flags for Julia (default: none).
• -a, --add <arg>: Extra packages needed (delimit by comma).
• -s, --script <arg>: The benchmark script. Default: benchmark/benchmarks.jl downloaded from stable.
• --bench-on <arg>: If the script is not set, this specifies the revision at which to download benchmark/benchmarks.jl from the package.
• -f, --filter <arg>: Filter the benchmarks to run (delimit by comma).
• --nsamples-load-time <arg>: Number of samples to take when measuring load time of the package (default: 5). (This means starting a Julia process for each sample.)
• --dont-print: Don't print the table.

Flags

• --tune: Whether to run benchmarks with tuning (default: false).

comonicon_install(;kwargs...)

Install the CLI manually. This will use the default configuration in Comonicon.toml, if it exists. For more detailed reference, please refer to Comonicon documentation.

comonicon_install_path(;[yes=false])

Install the PATH and FPATH to your shell configuration file. You can use comonicon_install_path(;yes=true) to skip interactive prompt. For more detailed reference, please refer to Comonicon documentation.

benchpkgtable [package_name] [-r --rev <arg>]
                             [-i --input-dir <arg>]
                             [--ratio]
                             [--mode <arg>]
                             [--url <arg>]
                             [--path <arg>]

Print a table of the benchmarks of a package as created with benchpkg.

Arguments

• package_name: Name of the package.

Options

• -r, --rev <arg>: Revisions to test (delimit by comma). The default is {DEFAULT},dirty, which will attempt to find the default branch of the package.
• -i, --input-dir <arg>: Where the JSON results were saved (default: ".").
• --url <arg>: URL of the package. Only used to get the package name.
• --path <arg>: Path of the package. The default is . if other arguments are not given. Only used to get the package name.

Flags

• --ratio: Whether to include the ratio (default: false). Only applies when comparing two revisions.
• --mode: Table mode(s). Valid values are "time" (default), to print the benchmark time, or "memory", to print the allocation and memory usage. Both options can be passed, if delimited by comma.

comonicon_install(;kwargs...)

Install the CLI manually. This will use the default configuration in Comonicon.toml, if it exists. For more detailed reference, please refer to Comonicon documentation.

comonicon_install_path(;[yes=false])

Install the PATH and FPATH to your shell configuration file. You can use comonicon_install_path(;yes=true) to skip interactive prompt. For more detailed reference, please refer to Comonicon documentation.

combined_plots(combined_results::OrderedDict; npart=10)

Create a combined plot of the results loaded from the load_results function. The function partitions the plots into smaller groups of size npart (defaults to 10) and combines the plots in each group vertically. It returns an array of combined plots.

Arguments

• combined_results::OrderedDict: Data to be plotted, obtained from the load_results function.
• npart::Int=10: Max plots to be combined in a single vertical group. Default is 10.

Returns

• Array{Plotly.Plot,1}: An array of combined Plots objects, with each element representing a group of up to npart vertical plots.

create_table(combined_results::OrderedDict; kws...)

Create a markdown table of the results loaded from the load_results function. If there are two results for a given benchmark, will have an additional column for the comparison, assuming the first revision is one to compare against.

The formatter keyword argument generates the column value. It defaults to TableUtils.format_time, which prints the median time ± the interquantile range. TableUtils.format_memory is also available to print the number of allocations and the allocated memory.

benchmark(package_name::String, rev::Union{String,Vector{String}}; output_dir::String=".", script::Union{String,Nothing}=nothing, tune::Bool=false, exeflags::Cmd=``, extra_pkgs::Vector{String}=String[])

Run benchmarks for a given Julia package.

This function runs the benchmarks specified in the script for the package defined by the package_spec. If script is not provided, the function will use the default benchmark script located at {PACKAGE_SRC_DIR}/benchmark/benchmarks.jl.

The benchmarks are run using the SUITE variable defined in the benchmark script, which should be of type BenchmarkTools.BenchmarkGroup. The benchmarks can be run with or without tuning depending on the value of the tune argument.

The results of the benchmarks are saved to a JSON file named results_packagename@rev.json in the specified output_dir.

Arguments

• package_name::String: The name of the package for which to run the benchmarks.
• rev::Union{String,Vector{String}}: The revision of the package for which to run the benchmarks. You can also pass a vector of revisions to run benchmarks for multiple versions of a package.
• output_dir::String=".": The directory where the benchmark results JSON file will be saved (default: current directory).
• script::Union{String,Nothing}=nothing: The path to the benchmark script file. If not provided, the default script at {PACKAGE}/benchmark/benchmarks.jl will be used.
• tune::Bool=false: Whether to run benchmarks with tuning (default: false).
• exeflags::Cmd=``: Additional execution flags for running the benchmark script (default: empty).
• extra_pkgs::Vector{String}=String[]: Additional packages to add to the benchmark environment.
• url::Union{String,Nothing}=nothing: URL of the package.
• path::Union{String,Nothing}=nothing: Path to the package.
• benchmark_on::Union{String,Nothing}=nothing: If the benchmark script file is to be downloaded, this specifies the revision to use.
• filter_benchmarks::Vector{String}=String[]: Filter the benchmarks to run (default: all).
• nsamples_load_time::Int=5: Number of samples to take for the time-to-load benchmark.

benchmark(package_specs::Union{PackageSpec,Vector{PackageSpec}}; output_dir::String=".", script::Union{String,Nothing}=nothing, tune::Bool=false, exeflags::Cmd=``, extra_pkgs::Vector{String}=String[])

Run benchmarks for a given Julia package.

This function runs the benchmarks specified in the script for the package defined by the package_spec. If script is not provided, the function will use the default benchmark script located at {PACKAGE_SRC_DIR}/benchmark/benchmarks.jl.

The benchmarks are run using the SUITE variable defined in the benchmark script, which should be of type BenchmarkTools.BenchmarkGroup. The benchmarks can be run with or without tuning depending on the value of the tune argument.

The results of the benchmarks are saved to a JSON file named results_packagename@rev.json in the specified output_dir.

Arguments

• package::Union{PackageSpec,Vector{PackageSpec}}: The package specification containing information about the package for which to run the benchmarks. You can also pass a vector of package specifications to run benchmarks for multiple versions of a package.
• output_dir::String=".": The directory where the benchmark results JSON file will be saved (default: current directory).
• script::Union{String,Nothing}=nothing: The path to the benchmark script file. If not provided, the default script at {PACKAGE}/benchmark/benchmarks.jl will be used.
• tune::Bool=false: Whether to run benchmarks with tuning (default: false).
• exeflags::Cmd=``: Additional execution flags for running the benchmark script (default: empty).
• extra_pkgs::Vector{String}=String[]: Additional packages to add to the benchmark environment.
• benchmark_on::Union{String,Nothing}=nothing: If the benchmark script file is to be downloaded, this specifies the revision to use.
• filter_benchmarks::Vector{String}=String[]: Filter the benchmarks to run (default: all).
• nsamples_load_time::Int=5: Number of samples to take for the time-to-load benchmark.

Fill in the package name, URL, and path based on the context.

load_results(specs::Vector{PackageSpec}; input_dir::String=".")

Load the results from JSON files for each PackageSpec in the specs vector. The function assumes that the JSON files are located in the input_dir directory and are named as "results_{s}.json" where s is equal to PackageName@Rev.

The function returns a combined OrderedDict, to be input to the combined_plots function.

Arguments

• specs::Vector{PackageSpec}: Vector of each package revision to be loaded (as PackageSpec).
• input_dir::String=".": Directory where the results. Default is current directory.

Returns

• OrderedDict{String,OrderedDict}: Combined results ready to be passed to the combined_plots function.

Fill in the default branch if needed.