An ExecutableProduct is a Product that represents an executable file.

On all platforms, an ExecutableProduct checks for existence of the file. On non-Windows platforms, it will check for the executable bit being set. On Windows platforms, it will check that the file ends with ".exe", (adding it on automatically, if it is not already present).

A FileProduct represents a file that simply must exist to be satisfied.

A LibraryProduct is a special kind of Product that not only needs to exist, but needs to be dlopen()'able. You must know which directory the library will be installed to, and its name, e.g. to build a LibraryProduct that refers to "/lib/libnettle.so", the "directory" would be "/lib", and the "libname" would be "libnettle". Note that a LibraryProduct can support multiple libnames, as some software projects change the libname based on the build configuration.

OutputCollector

A run() wrapper class that captures subprocess stdout and stderr streams independently, resynthesizing and colorizing the streams appropriately.

OutputCollector(cmd::AbstractCmd; verbose::Bool = false)

Run cmd, and collect the output such that stdout and stderr are captured independently, but with the time of each line recorded such that they can be stored/analyzed independently, but replayed synchronously.

A Product is an expected result after building or installation of a package.

Examples of Products include LibraryProduct, ExecutableProduct and FileProduct. All Product types must define the following minimum set of functionality:

• locate(::Product): given a Product, locate it within the wrapped Prefix returning its location as a string

• satisfied(::Product): given a Product, determine whether it has been successfully satisfied (e.g. it is locateable and it passes all callbacks)

• variable_name(::Product): return the variable name assigned to a Product

• repr(::Product): Return a representation of this Product, useful for auto-generating source code that constructs Products, if that's your thing.

merge(collector::OutputCollector; colored::Bool = false)

Merge the stdout and stderr streams of the OutputCollector on a per-line basis, returning a single string containing all collected lines, interleaved by capture time. If colored is set to true, embeds terminal color codes to print stderr in red.

arch(p::Platform)

Get the architecture for the given Platform object as a Symbol.

Examples

julia> arch(Linux(:aarch64))
:aarch64

julia> arch(MacOS())
:x86_64

bindir(prefix::Prefix)

Returns the binary directory for the given prefix.

call_abi(p::Platform)

Get the calling ABI for the given Platform object as a Symbol.

Examples

julia> call_abi(Linux(:x86_64))
:blank_abi

julia> call_abi(FreeBSD(:armv7l))
:eabihf

choose_download(download_info::Dict, platform::Platform = platform_key_abi())

Given a download_info dictionary mapping platforms to some value, choose the value whose key best matches platform, returning nothing if no matches can be found.

Platform attributes such as architecture, libc, calling ABI, etc... must all match exactly, however attributes such as compiler ABI can have wildcards within them such as :gcc_any which matches any version of GCC.

collect_stderr(collector::OutputCollector)

Returns all stderr lines collected by this collector so far.

collect_stdout(collector::OutputCollector)

Returns all stdout lines collected by this collector so far.

download_verify(url::AbstractString, hash::AbstractString,
                dest::AbstractString; verbose::Bool = false,
                force::Bool = false, quiet_download::Bool = false)

Download file located at url, verify it matches the given hash, and throw an error if anything goes wrong. If dest already exists, just verify it. If force is set to true, overwrite the given file if it exists but does not match the given hash.

This method returns true if the file was downloaded successfully, false if an existing file was removed due to the use of force, and throws an error if force is not set and the already-existent file fails verification, or if force is set, verification fails, and then verification fails again after redownloading the file.

If quiet_download is set to false (the default), this method will print to stdout when downloading a new file. If it is set to true (and verbose is set to false) the downloading process will be completely silent. If verbose is set to true, messages about integrity verification will be printed in addition to messages regarding downloading.

download_verify_unpack(url::AbstractString, hash::AbstractString,
                       dest::AbstractString; tarball_path = nothing,
                       verbose::Bool = false, ignore_existence::Bool = false,
                       force::Bool = false)

Helper method to download tarball located at url, verify it matches the given hash, then unpack it into folder dest. In general, the method install() should be used to download and install tarballs into a Prefix; this method should only be used if the extra functionality of install() is undesired.

If tarball_path is specified, the given url will be downloaded to tarball_path, and it will not be removed after downloading and verification is complete. If it is not specified, the tarball will be downloaded to a temporary location, and removed after verification is complete.

If force is specified, a verification failure will cause tarball_path to be deleted (if it exists), the dest folder to be removed (if it exists) and the tarball to be redownloaded and reverified. If the verification check is failed a second time, an exception is raised. If force is not specified, a verification failure will result in an immediate raised exception.

If ignore_existence is set, the tarball is unpacked even if the destination directory already exists.

Returns true if a tarball was actually unpacked, false if nothing was changed in the destination prefix.

extract_name_version_platform_key(path::AbstractString)

Given the path to a tarball, return the name, platform key and version of that tarball. If any of those things cannot be found, throw an error.

extract_platform_key(path::AbstractString)

Given the path to a tarball, return the platform key of that tarball. If none can be found, prints a warning and return the current platform suffix.

gen_download_cmd(url::AbstractString, out_path::AbstractString)

Return a Cmd that will download resource located at url and store it at the location given by out_path.

This method is initialized by probe_platform_engines(), which should be automatically called upon first import of BinaryProvider.

gen_list_tarball_cmd(tarball_path::AbstractString)

Return a Cmd that will list the files contained within the tarball located at tarball_path. The list will not include directories contained within the tarball.

This method is initialized by probe_platform_engines(), which should be automatically called upon first import of BinaryProvider.

gen_package_cmd(in_path::AbstractString, tarball_path::AbstractString)

Return a Cmd that will package up the given in_path directory into a tarball located at tarball_path.

This method is initialized by probe_platform_engines(), which should be automatically called upon first import of BinaryProvider.

gen_sh_cmd(cmd::Cmd)

Runs a command using sh. On Unices, this will default to the first sh found on the PATH, however on Windows if that is not found it will fall back to the sh provided by the busybox.exe shipped with Julia.

This method is initialized by probe_platform_engines(), which should be automatically called upon first import of BinaryProvider.

gen_unpack_cmd(tarball_path::AbstractString, out_path::AbstractString; excludelist::Union{AbstractString, Nothing} = nothing)

Return a Cmd that will unpack the given tarball_path into the given out_path. If out_path is not already a directory, it will be created. excludlist is an optional file which contains a list of files that is not unpacked This option is mainyl used to exclude symlinks from extraction (see: copyderef)

This method is initialized by probe_platform_engines(), which should be automatically called upon first import of BinaryProvider.

includedir(prefix::Prefix)

Returns the include directory for the given prefix

install(tarball_url::AbstractString,
        hash::AbstractString;
        prefix::Prefix = global_prefix,
        force::Bool = false,
        ignore_platform::Bool = false,
        verbose::Bool = false)

Given a prefix, a tarball_url and a hash, download that tarball into the prefix, verify its integrity with the hash, and install it into the prefix. Also save a manifest of the files into the prefix for uninstallation later.

This will not overwrite any files within prefix unless force=true is set. If force=true is set, installation will overwrite files as needed, and it will also delete any files previously installed for tarball_url as listed in a pre-existing manifest (if any).

By default, this will not install a tarball that does not match the platform of the current host system, this can be overridden by setting ignore_platform.

isinstalled(tarball_url::AbstractString,
            hash::AbstractString;
            prefix::Prefix = global_prefix)

Given a prefix, a tarball_url and a hash, check whether the tarball with that hash has been installed into prefix.

In particular, it checks for the tarball, matching hash file, and manifest installed by install, and checks that the files listed in the manifest are installed and are not older than the tarball.

libc(p::Platform)

Get the libc for the given Platform object as a Symbol.

Examples

julia> libc(Linux(:aarch64))
:glibc

julia> libc(FreeBSD(:x86_64))
:default_libc

libdir(prefix::Prefix, platform = platform_key_abi())

Returns the library directory for the given prefix (note that this differs between unix systems and windows systems).

list_tarball_files(path::AbstractString; verbose::Bool = false)

Given a .tar.gz filepath, list the compressed contents.

list_tarball_symlinks(path::AbstractString; verbose::Bool = false)

Given a .tar.gz filepath, return a dictionary of symlinks in the archive

locate(fp::ExecutableProduct; platform::Platform = platform_key_abi(), verbose::Bool = false, isolate::Bool = false)

If the given executable file exists and is executable, return its path.

On all platforms, an ExecutableProduct checks for existence of the file. On non-Windows platforms, it will check for the executable bit being set. On Windows platforms, it will check that the file ends with ".exe", (adding it on automatically, if it is not already present).

locate(fp::FileProduct; platform::Platform = platformkeyabi(), verbose::Bool = false, isolate::Bool = false)

If the given file exists, return its path. The platform argument is ignored here, but included for uniformity.

locate(lp::LibraryProduct; verbose::Bool = false, platform::Platform = platformkeyabi())

If the given library exists (under any reasonable name) and is dlopen()able, (assuming it was built for the current platform) return its location. Note that the dlopen() test is only run if the current platform matches the given platform keyword argument, as cross-compiled libraries cannot be dlopen()ed on foreign platforms.

logdir(prefix::Prefix)

Returns the logs directory for the given prefix.

manifest_for_file(path::AbstractString; prefix::Prefix = global_prefix)

Returns the manifest file containing the installation receipt for the given path, throws an error if it cannot find a matching manifest.

manifest_from_url(url::AbstractString; prefix::Prefix = global_prefix())

Returns the file path of the manifest file for the tarball located at url.

package(src_dir::AbstractString, tarball_path::AbstractString;
        verbose::Bool = false)

Compress src_dir into a tarball located at tarball_path.

package(prefix::Prefix, output_base::AbstractString,
        version::VersionNumber;
        platform::Platform = platform_key_abi(),
        verbose::Bool = false, force::Bool = false)

Build a tarball of the prefix, storing the tarball at output_base, appending a version number, a platform-dependent suffix and a file extension. If no platform is given, defaults to current platform. Runs an audit() on the prefix, to ensure that libraries can be dlopen()'ed, that all dependencies are located within the prefix, etc... See the audit() documentation for a full list of the audit steps. Returns the full path to and hash of the generated tarball.

parse_7z_list(output::AbstractString)

Given the output of 7z l, parse out the listed filenames. This funciton used by list_tarball_files.

parse_tar_list(output::AbstractString)

Given the output of tar -t, parse out the listed filenames. This funciton used by list_tarball_files.

parse_tarball_listing(output::AbstractString)

Parses the result of gen_list_tarball_cmd() into something useful.

This method is initialized by probe_platform_engines(), which should be automatically called upon first import of BinaryProvider.

platform_dlext(platform::Platform = platform_key_abi())

Return the dynamic library extension for the given platform, defaulting to the currently running platform. E.g. returns "so" for a Linux-based platform, "dll" for a Windows-based platform, etc...

platform_key_abi(machine::AbstractString)

Returns the platform key for the current platform, or any other though the the use of the machine parameter.

satisfied(p::Product; platform::Platform = platform_key_abi(),
          verbose::Bool = false, isolate::Bool = false)

Given a Product, return true if that Product is satisfied, e.g. whether a file exists that matches all criteria setup for that Product. If isolate is set to true, will isolate all checks from the main Julia process in the event that dlopen()'ing a library might cause issues.

tail(collector::OutputCollector; len::Int = 100, colored::Bool = false)

Write out the last len lines, optionally writing colored lines.

tee(c::OutputCollector; colored::Bool = false, stream::IO = stdout)

Spawn a background task to incrementally output lines from collector to the standard output, optionally colored.

temp_prefix(func::Function)

Create a temporary prefix, passing the prefix into the user-defined function so that build/packaging operations can occur within the temporary prefix, which is then cleaned up after all operations are finished. If the path provided exists already, it will be deleted.

Usage example:

out_path = abspath("./libfoo")
temp_prefix() do p
    # <insert build steps here>

    # tarball up the built package
    tarball_path, tarball_hash = package(p, out_path)
end

triplet(platform)

Get the target triplet for the given Platform object as a String.

Examples

julia> triplet(MacOS())
"x86_64-apple-darwin14"

julia> triplet(Windows(:i686))
"i686-w64-mingw32"

julia> triplet(Linux(:armv7l, :default_libc, :default_abi, CompilerABI(:gcc4))
"arm-linux-gnueabihf-gcc4"

uninstall(manifest::AbstractString; verbose::Bool = false)

Uninstall a package from a prefix by providing the manifest_path that was generated during install(). To find the manifest_file for a particular installed file, use manifest_for_file(file_path; prefix=prefix).

unpack(tarball_path::AbstractString, dest::AbstractString;
       verbose::Bool = false)

Unpack tarball located at file tarball_path into directory dest.

valid_dl_path(path::AbstractString, platform::Platform)

Return true if the given path ends in a valid dynamic library filename. E.g. returns true for a path like "usr/lib/libfoo.so.3.5", but returns false for a path like "libbar.so.f.a".

variable_name(p::Product)

Return the variable name associated with this Product as a string

verify(path::AbstractString, hash::AbstractString;
       verbose::Bool = false, report_cache_status::Bool = false)

Given a file path and a hash, calculate the SHA256 of the file and compare it to hash. If an error occurs, verify() will throw an error. This method caches verification results in a "$(path).sha256" file to accelerate re- verification of files that have been previously verified. If no ".sha256" file exists, a full verification will be done and the file will be created, with the calculated hash being stored within the ".sha256" file.. If a ".sha256" file does exist, its contents are checked to ensure that the hash contained within matches the given hash parameter, and its modification time shows that the file located at path has not been modified since the last verification.

If report_cache_status is set to true, then the return value will be a Symbol giving a granular status report on the state of the hash cache, in addition to the true/false signifying whether verification completed successfully.

wordsize(platform)

Get the word size for the given Platform object.

Examples

julia> wordsize(Linux(:arm7vl))
32

julia> wordsize(MacOS())
64

write_deps_file(depsjl_path::AbstractString, products::Vector{Product};
                verbose::Bool = false)

Generate a deps.jl file that contains the variables referred to by the products within products. As an example, running the following code:

fooifier = ExecutableProduct(..., :foo_exe)
libbar = LibraryProduct(..., :libbar)
write_deps_file(joinpath(@__DIR__, "deps.jl"), [fooifier, libbar])

Will generate a deps.jl file that contains definitions for the two variables foo_exe and libbar. If any Product object cannot be satisfied (e.g. LibraryProduct objects must be dlopen()-able, FileProduct objects must exist on the filesystem, etc...) this method will error out. Ensure that you have used install() to install the binaries you wish to write a deps.jl file for.

The result of this method is a deps.jl file containing variables named as defined within the Product objects passed in to it, holding the full path to the installed binaries. Given the example above, it would contain code similar to:

global const foo_exe = "<pkg path>/deps/usr/bin/fooifier"
global const libbar = "<pkg path>/deps/usr/lib/libbar.so"

This deps.jl file is intended to be include()'ed from within the top-level source of your package. Note that all files are checked for consistency on package load time, and if an error is discovered, package loading will fail, asking the user to re-run Pkg.build("package_name").

LineStream(pipe::Pipe)

Given a Pipe that has been initialized by spawn(), create an async Task to read in lines as they come in and annotate the time the line was captured for later replay/merging with other simultaneously captured streams.

wait(collector::OutputCollector)

Wait for the command and all line streams within an OutputCollector to finish their respective tasks and be ready for full merging. Return the success of the underlying process. Prints out the last 10 lines of the process if it does not complete successfully unless the OutputCollector was created as verbose.

withenv(f::Function, prefixes::Vector{Prefix}; julia_libdir::Bool = true)

Wrapper function designed to help executables find dynamic libraries and child binaries by wrapping PATH and (DY)LD_(FALLBACK_)LIBRARY_PATH. If julia_libdir is true, then the private library directory of this Julia distribution will be added on to the end of the LDLIBRARYPATH settings.

alive(s::LineStream)

Returns trueif the task owned by thisLineStreamis still processing output from an underlyingPipe`.

compiler_abi(p::Platform)

Get the compiler ABI object for the given Platform

Examples

julia> compiler_abi(Linux(:x86_64))
CompilerABI(:gcc_any, :cxx_any)

julia> compiler_abi(Linux(:x86_64; compiler_abi=CompilerABI(:gcc7)))
CompilerABI(:gcc7, :cxx_any)

detect_cxx11_string_abi()

Introspects the currently running Julia process to see what version of the C++11 string ABI it was compiled with. (In reality, it checks for symbols within LLVM, but that is close enough for our purposes, as you can't mix linkages between Julia and LLVM if they are not compiled in the same way).

detect_libgfortran_abi(libgfortran_name::AbstractString)

Examines the given libgfortran SONAME to see what version of GCC corresponds to the given libgfortran version.

detect_libgfortran_abi()

If no parameter is given, introspects the current Julia process to determine the version of GCC this Julia was built with.

detect_libstdcxx_abi()

Introspects the currently running Julia process to find out what version of libstdc++ it is linked to (if any), as a proxy for GCC version compatibility. E.g. if we are linked against libstdc++.so.19, binary dependencies built by GCC 8.1.0 will have linker errors. This method returns the maximum GCC abi that we can support.

download(url::AbstractString, dest::AbstractString;
         verbose::Bool = false)

Download file located at url, store it at dest, continuing if dest already exists and the server and download engine support it.

info_onchange(msg, key, location)

This macro is used to gate verbose messages within a function; within functions such as verify(), we want to print out that we successfully verified a file only once per session, so we keep track of which log message we printed within a particular method. This is done by passing a key to @info_onchange as the second parameter, e.g. in the above scenario we might pass the string "verify_$(filepath)". Combined with this key is the file and line the message originates from, which is used to suppress duplicate messages; e.g. if a method with two branches that both log with the same key is called, if the execution path is branch A, A, B, A, B, B, B, A then what will be printed is message A, B, A, B, A. In essence, this method calls info() only when the message generated by this particular method changes.

parse_dl_name_version(path::AbstractString, platform::Platform)

Given a path to a dynamic library, parse out what information we can from the filename. E.g. given something like "lib/libfoo.so.3.2", this function returns "libfoo", v"3.2". If the path name is not a valid dynamic library, this method throws an error. If no soversion can be extracted from the filename, as in "libbar.so" this method returns "libbar", nothing.

platform_name(p::Platform)

Get the "platform name" of the given platform. E.g. returns "Linux" for a Linux object, or "Windows" for a Windows object.

probe_cmd(cmd::Cmd; verbose::Bool = false)

Returns true if the given command executes successfully, false otherwise.

probe_platform_engines!(;verbose::Bool = false)

Searches the environment for various tools needed to download, unpack, and package up binaries. Searches for a download engine to be used by gen_download_cmd() and a compression engine to be used by gen_unpack_cmd(), gen_package_cmd(), gen_list_tarball_cmd() and parse_tarball_listing(), as well as a sh execution engine for gen_sh_cmd(). Running this function will set the global functions to their appropriate implementations given the environment this package is running on.

This probing function will automatically search for download engines using a particular ordering; if you wish to override this ordering and use one over all others, set the BINARYPROVIDER_DOWNLOAD_ENGINE environment variable to its name, and it will be the only engine searched for. For example, put:

ENV["BINARYPROVIDER_DOWNLOAD_ENGINE"] = "fetch"

within your ~/.juliarc.jl file to force fetch to be used over curl. If the given override does not match any of the download engines known to this function, a warning will be printed and the typical ordering will be performed.

Similarly, if you wish to override the compression engine used, set the BINARYPROVIDER_COMPRESSION_ENGINE environment variable to its name (e.g. 7z or tar) and it will be the only engine searched for. If the given override does not match any of the compression engines known to this function, a warning will be printed and the typical searching will be performed.

If verbose is true, print out the various engines as they are searched.

probe_symlink_creation(dest::AbstractString)

Probes whether we can create a symlink within the given destination directory, to determine whether a particular filesystem is "symlink-unfriendly".

readuntil_many(s::IO, delims)

Given a collection of delimiter characters, read from s until one of those delimiters is reached, or we reach the end of s.