TestContext()

A context for test setups. Used to keep track of @testsetup-expanded TestSetups and a TestSetupModules for a given process; used in runtestitem to ensure any setups relied upon by the @testitem are evaluated on the process that will run the test item.

TestItem

A single, independently runnable group of tests. Used to wrap tests that must be run together, similar to a @testset, but encapsulating those test in their own module. Should only be created via the @testitem macro.

TestSetup(name, code)

A module that a TestItem can require to be run before that TestItem is run. Used for declaring code that multiple TestItems rely on. Should only be created via the @testsetup macro.

TestSetupModules()

A set of test setups. Used to keep track of which test setups have been evaluated on a given process.

_redirect_logs(f, target::Union{IO,String})

Redirects stdout and stderr while f is evaluated to target. If target is String it is assumed it is a file path.

print_errors_and_captured_logs(ti::TestItem, run_number::Int; logs=:batched, errors_first=false)

When a testitem doesn't succeed, we print the corresponding error/failure reports from the testset and any logs that we captured while the testitem was eval()'d.

For :eager mode of logs we don't print any logs as they bypass log capture. :batched means we print logs even for passing test items, whereas :issues means we are only printing captured logs if there were any errors or failures.

If errors_first=true, then the test errors are printed first and the logs second. The default errors_first=false, prints the logs firsts.

Nothing is printed when no logs were captures and no failures or errors occured.

ReTestItems.runtests()
ReTestItems.runtests(mod::Module)
ReTestItems.runtests(paths::AbstractString...)

Execute @testitem tests.

Only files ending in _test.jl or _tests.jl will be searched for test items.

If directory or file paths are passed, only those directories and files are searched. If no arguments are passed, the src/ and test/ directories of the current project are searched for @testitems.

Keywords

Filtering @testitems

• name::Union{Regex,AbstractString,Nothing}=nothing: Used to filter @testitems by their name. AbstractString input will only keep the @testitem that exactly matches name, Regex can be used to partially match multiple @testitems. By default, no filtering is applied.
• tags::Union{Symbol,AbstractVector{Symbol},Nothing}=nothing: Used to filter @testitems by their tags. A single tag can be used to match any @testitem that contains it, when multiple tags are provided, only @testitems that contain all of the tags will be run. By default, no filtering is applied.

name and tags filters are applied together and only those @testitems that pass both filters will be run.

Configuring runtests

• testitem_timeout::Real: The number of seconds to wait until a @testitem is marked as failed. Defaults to 30 minutes. Can also be set using the RETESTITEMS_TESTITEM_TIMEOUT environment variable. If a @testitem sets its own timeout keyword, then that takes precedence. Fractional values are rounded up to the nearest second. Note timeouts are currently only applied when nworkers > 0.
• retries::Int=0: The number of times to retry a @testitem if either tests do not pass or, if running with multiple worker processes, the worker fails or hits the timeout limit while running the tests. Can also be set using the RETESTITEMS_RETRIES environment variable. If a @testitem sets its own retries keyword, then the maximum of these two retry numbers will be used as the retry limit for that @testitem. When report=true, the report will contain information for all runs of a @testitem that was retried.
• nworkers::Int: The number of worker processes to use for running @testitems. Default 0. Can also be set using the RETESTITEMS_NWORKERS environment variable.
• nworker_threads::Union{String,Int}: The number of threads to use for each worker process. Defaults to 2. Can also be set using the RETESTITEMS_NWORKER_THREADS environment variable. Interactive threads are supported through a string (e.g. "auto,2").
• worker_init_expr::Expr: an expression that will be run on each worker process before any tests are run. Can be used to load packages or set up the environment. Must be a :block expression.
• test_end_expr::Expr: an expression that will be run after each testitem is run. Can be used to verify that global state is unchanged after running a test. Must be a :block expression.
• memory_threshold::Real: Sets the fraction of memory that can be in use before a worker processes are restarted to free memory. Defaults to 0.99. Only supported with nworkers > 0. For example, if set to 0.8, then when >80% of the available memory is in use, a worker process will be killed and replaced with a new worker before the next testitem is run. The testitem will then be run on the new worker process, regardless of if memory pressure dropped below the threshold. If the memory pressure remains above the threshold, then a worker process will again be replaced before the next testitem is run. Can also be set using the RETESTITEMS_MEMORY_THRESHOLD environment variable. Note: the memory_threshold keyword is experimental and may be removed in future versions.
• report::Bool=false: If true, write a JUnit-format XML file summarising the test results. Can also be set using the RETESTITEMS_REPORT environment variable. The location at which the XML report is saved can be set using the RETESTITEMS_REPORT_LOCATION environment variable. By default the report will be written at the root of the project being tested.
• logs::Symbol: Handles how and when we display messages produced during test evaluation. Can be one of:
  • :eager: Everything is printed to stdout immediately, like in a regular Julia session.
  • :batched: Logs are saved to a file and then printed when the test item is finished.
  • :issues: Logs are saved to a file and only printed if there were any errors or failures.
  For interative sessions, :eager is the default when running with 0 or 1 worker processes, :batched otherwise. For non-interactive sessions, :issues is used by default.
• verbose_results::Bool: If true, the final test report will list each @testitem, otherwise the results are aggregated. Default is false for non-interactive sessions or when logs=:issues, true otherwise.
• validate_paths::Bool=false: If true, runtests will throw an error if any of the paths passed to it cannot contain test files, either because the path doesn't exist or the path points to a file which is not a test file. Default is false. Can also be set using the RETESTITEMS_VALIDATE_PATHS environment variable.
• timeout_profile_wait::Real=0: When non-zero, a worker that times-out will trigger a CPU profile for which we will wait timeout_profile_wait seconds before terminating the worker. Zero means no profile will be taken. Can also be set using the RETESTITEMS_TIMEOUT_PROFILE_WAIT environment variable. See the Profile documentation for more information on triggered profiles. Note you can use worker_init_expr to tweak the profile settings on workers.

@testitem "name" [tags=[] setup=[] retries=0 skip=false default_imports=true] begin
    # code that will be run as tests
end

A single, independently runnable group of tests.

A test item is a standalone block of tests, and cannot access names from the surrounding scope. Multiple test items may run in parallel, executing on distributed processes.

A @testitem can contain a single test:

@tesitem "Arithmetic" begin
    @test 1 + 1 == 2
end

Or it can contain many tests, which can be arranged in @testsets:

@testitem "Arithmetic" begin
    @testset "addition" begin
        @test 1 + 2 == 3
        @test 1 + 0 == 1
    end
    @testset "multiplication" begin
        @test 1 * 2 == 2
        @test 1 * 0 == 0
    end
    @test 1 + 2 * 2 == 5
end

A @testitem is wrapped into a module when run, so must import any additional packages:

@testitem "Arithmetic" begin
    using LinearAlgebra
    @testset "multiplication" begin
        @test dot(1, 2) == 2
    end
end

The test item's code is run as top-level code in a new module, so it can include imports, define new structs or helper functions, and declare tests and testsets.

@testitem "DoCoolStuff" begin
    function do_really_cool_stuff()
        # ...
    end
    @testset "cool stuff doing" begin
        @test do_really_cool_stuff()
    end
end

By default, Test and the package being tested will be loaded into the @testitem. This can be disabled by passing default_imports=false.

A @testitem can use test-specific setup code declared with @testsetup, by passing the name of the test setup module with the setup keyword:

@testsetup module TestIrrationals
    const PI = 3.14159
    const INV_PI = 0.31831
    area(radius) = PI * radius^2
    export PI, INV_PI, area
end
@testitem "Arithmetic" setup=[TestIrrationals] begin
    @test 1 / PI ≈ INV_PI atol=1e-6
end
@testitem "Geometry" setup=[TestIrrationals] begin
    @test area(1) ≈ PI
end

If a @testitem is known to be flaky, i.e. contains tests that sometimes don't pass, then you can set it to automatically retry by passing the retries keyword. If a @testitem passes on retry, then it will be recorded as passing in the test summary.

@testitem "Flaky test" retries=1 begin
    @test rand() < 1e-4
end

If a @testitem should be aborted after a certain period of time, e.g. the test is known to occassionally hang, then you can set a timeout (in seconds) by passing the timeout keyword. Note that timeout currently only works when tests are run with multiple workers.

@testitem "Sometimes too slow" timeout=10 begin
    @test sleep(rand(1:100))
end

If a @testitem needs to be skipped, then you can set the skip keyword. Either pass skip=true to unconditionally skip the test item, or pass skip an expression that returns a Bool to determine if the testitem should be skipped.

@testitem "Skip on old Julia" skip=(VERSION < v"1.9") begin
    v = [1]
    @test 0 == @allocations sum(v)
end

The skip expression is run in its own module, just like a test-item. No code inside a @testitem is run when a test-item is skipped.

@testsetup module MyTestSetup
    # code that can be shared between @testitems
end

A module only used for tests, and which @testitems can depend on.

Useful for setup logic that is used across multiple test items. The setup will run once, before any @testitem that requires it is executed. If running with multiple processes, each test-setup with be run once on each process.

Each test-setup module will live for the lifetime of the tests. Mutable state should be avoided, since the order in which test items run is non-deterministic, and test items may access the same test-setup module concurrently in the same process.

Other than being declared with the @testsetup macro, to make then knowable to @testitems, test-setup modules are just like other modules and can import dependencies and export names.

A @testitem depends on a @testsetup via the setup keyword e.g

@testitem "MyTests1" setup=[MyTestSetup]
    # tests that require MyTestSetup
end