The Julia module PythonCall

Installation

This package is in the general registry, so to install just type ] in the Julia REPL and run:

pkg> add PythonCall

Getting started

Import the module with:

julia> using PythonCall

By default this will initialize a conda environment in your Julia environment, install Python into it, load the corresponding Python library and initialize an interpreter.

Now you can interact with Python as follows:

julia> re = pyimport("re")
Python module: <module 're' from '[...]/lib/re.py'>

julia> words = re.findall("[a-zA-Z]+", "PythonCall.jl is very useful!")
Python list: ['PythonCall', 'jl', 'is', 'very', 'useful']

julia> sentence = Py(" ").join(words)
Python str: 'PythonCall jl is very useful'

julia> pyconvert(String, sentence)
"PythonCall jl is very useful"

In this example:

  • We used pyimport to import the re module. Equivalently we could have done @py import re (see @py).
  • We called its findall function on a pair of strings, which were automatically converted to Python strings (see Conversion to Python).
  • We called Py to explicitly convert a string to a Python string, so that we could call its join method. All Python objects are of type Py.
  • We called pyconvert to convert the Python string sentence to a Julia string (see Conversion to Julia).

Read on to find out what else you can do.

Py

PythonCall.@pyconst — Macro
@pyconst ex

Equivalent to Py(ex) but always returns the exact same Julia object.

That is, if foo() = @pyconst ex then foo() === foo().

The expression ex is evaluated the first time the code is run.

If ex is a string literal, the string is interned.

Do not use this macro at the top level of a module. Instead, use pynew() and pycopy!().

The object pybuiltins has all the standard Python builtin objects as its properties. Hence you can access pybuiltins.None and pybuiltins.TypeError.

@py

PythonCall.@py — Macro
@py expr

Evaluate the given expression using Pythonic semantics.

For example:

  • f(x, y) is translated to pycall(f, x, y)
  • x + y is translated to pyadd(x, y)
  • x === y is translated to pyis(x, y)
  • x.foo is translated to pygetattr(x, "foo")

Compound statements such as begin, if, while and for are supported.

See the online documentation for more details.

Python functions

Most of the functions in this section are essentially Python builtins with a py prefix. For example pyint(x) converts x to a Python int and is equivalent to int(x) in Python when x is a Python object.

Notable exceptions are:

  • pyconvert to convert a Python object to a Julia object.
  • pyimport to import a Python module.
  • pyjl to directly wrap a Julia object as a Python object.
  • pyclass to construct a new class.
  • pywith to emulate the Python with statement.

If a Julia value is passed as an argument to one of these functions, it is converted to a Python value using the rules documented here.

Constructors

These functions construct Python objects of builtin types from Julia values.

PythonCall.pycomplex — Function
pycomplex(x=0.0)
pycomplex(re, im)

Convert x to a Python complex, or create one from given real and imaginary parts.

PythonCall.pytuple — Function
pytuple(x=())

Convert x to a Python tuple.

If x is a Python object, this is equivalent to tuple(x) in Python. Otherwise x must be iterable.

PythonCall.pylist — Function
pylist(x=())

Convert x to a Python list.

If x is a Python object, this is equivalent to list(x) in Python. Otherwise x must be iterable.

PythonCall.pycollist — Function
pycollist(x::AbstractArray)

Create a nested Python list-of-lists from the elements of x. For matrices, this is a list of columns.

PythonCall.pyrowlist — Function
pyrowlist(x::AbstractArray)

Create a nested Python list-of-lists from the elements of x. For matrices, this is a list of rows.

PythonCall.pyset — Function
pyset(x=())

Convert x to a Python set.

If x is a Python object, this is equivalent to set(x) in Python. Otherwise x must be iterable.

PythonCall.pyfrozenset — Function
pyfrozenset(x=())

Convert x to a Python frozenset.

If x is a Python object, this is equivalent to frozenset(x) in Python. Otherwise x must be iterable.

PythonCall.pydict — Function
pydict(x)
pydict(; x...)

Convert x to a Python dict. In the second form, the keys are strings.

If x is a Python object, this is equivalent to dict(x) in Python. Otherwise x must iterate over key-value pairs.

PythonCall.pyslice — Function
pyslice([start], stop, [step])

Construct a Python slice. Unspecified arguments default to None.

PythonCall.pyrange — Function
pyrange([[start], [stop]], [step])

Construct a Python range. Unspecified arguments default to None.

PythonCall.pyclass — Function
pyclass(name, bases=(); members...)

Construct a new Python type with the given name, bases and members.

Equivalent to pytype(name, bases, members).

Builtins

These functions mimic the Python builtin functions or keywords of the same name.

PythonCall.pyimport — Function
pyimport(m)
pyimport(m => k)
pyimport(m => (k1, k2, ...))
pyimport(m1, m2, ...)

Import a module m, or an attribute k, or a tuple of attributes.

If several arguments are given, return the results of importing each one in a tuple.

PythonCall.pywith — Function
pywith(f, o, d=nothing)

Equivalent to with o as x: f(x) in Python, where x is a Py.

On success, the value of f(x) is returned.

If an exception occurs but is suppressed then d is returned.

PythonCall.pyis — Function
pyis(x, y)

True if x and y are the same Python object. Equivalent to x is y in Python.

PythonCall.pygetattr — Function
pygetattr(x, k, [d])

Equivalent to getattr(x, k) or x.k in Python.

If d is specified, it is returned if the attribute does not exist.

PythonCall.pysetattr — Function
pysetattr(x, k, v)

Equivalent to setattr(x, k, v) or x.k = v in Python.

PythonCall.pycall — Function
pycall(f, args...; kwargs...)

Call the Python object f with the given arguments.

PythonCall.pylen — Function
pylen(x)

The length of x. Equivalent to len(x) in Python, converted to an Integer.

PythonCall.pysetitem — Function
pysetitem(x, k, v)

Equivalent to setitem(x, k, v) or x[k] = v in Python.

PythonCall.pyissubclass — Function
pyissubclass(s, t)

Test if s is a subclass of t. Equivalent to issubclass(s, t) in Python.

PythonCall.pyisinstance — Function
pyisinstance(x, t)

Test if x is of type t. Equivalent to isinstance(x, t) in Python.

PythonCall.pyhash — Function
pyhash(x)

Equivalent to hash(x) in Python, converted to an Integer.

Conversion to Julia

These functions convert Python values to Julia values, using the rules documented here.

PythonCall.pyconvert — Function
pyconvert(T, x, [d])

Convert the Python object x to a T.

If d is specified, it is returned on failure instead of throwing an error.

PythonCall.@pyconvert — Macro
@pyconvert(T, x, [onfail])

Convert the Python object x to a T.

On failure, evaluates to onfail, which defaults to return pyconvert_unconverted() (mainly useful for writing conversion rules).

Wrap Julia values

These functions explicitly wrap Julia values into Python objects, documented here.

As documented here, Julia values are wrapped like this automatically on conversion to Python, unless the value is immutable and has a corresponding Python type.

PythonCall.pyjl — Function
pyjl([t], x)

Create a Python object wrapping the Julia object x.

If x is mutable, then mutating the returned object also mutates x, and vice versa.

Its Python type is normally inferred from the type of x, but can be specified with t.

For example if x is an AbstractVector then the object will have type juliacall.VectorValue. This object will satisfy the Python sequence interface, so for example uses 0-up indexing.

PythonCall.pyjlraw — Function
pyjlraw(v)

Create a Python object wrapping the Julia object x.

It has type juliacall.RawValue. This has a much more rigid "Julian" interface than pyjl(v). For example, accessing attributes or calling this object will always return a RawValue.

PythonCall.pyisjl — Function
pyisjl(x)

Test whether x is a wrapped Julia value, namely an instance of juliacall.ValueBase.

PythonCall.pybinaryio — Function
pybinaryio(io::IO)

Wrap io as a Python binary IO object.

This is the default behaviour of Py(io).

Arithmetic

These functions are equivalent to the corresponding Python arithmetic operators.

Note that the equivalent Julia operators are overloaded to call these when all arguments are Py (or Number). Hence the following are equivalent: Py(1)+Py(2), Py(1)+2, pyadd(1, 2), pyadd(Py(1), Py(2)), etc.

PythonCall.pypow — Function
pypow(x, y, z=None)

Equivalent to x ** y or pow(x, y, z) in Python.

PythonCall.pyiadd — Function
pyiadd(x, y)

In-place add. x = pyiadd(x, y) is equivalent to x += y in Python.

PythonCall.pyisub — Function
pyisub(x, y)

In-place subtract. x = pyisub(x, y) is equivalent to x -= y in Python.

PythonCall.pyimul — Function
pyimul(x, y)

In-place multiply. x = pyimul(x, y) is equivalent to x *= y in Python.

PythonCall.pyimatmul — Function
pyimatmul(x, y)

In-place matrix multiply. x = pyimatmul(x, y) is equivalent to x @= y in Python.

PythonCall.pyipow — Function
pyipow(x, y, z=None)

In-place power. x = pyipow(x, y) is equivalent to x **= y in Python.

PythonCall.pyifloordiv — Function
pyifloordiv(x, y)

In-place floor divide. x = pyifloordiv(x, y) is equivalent to x //= y in Python.

PythonCall.pyitruediv — Function
pyitruediv(x, y)

In-place true division. x = pyitruediv(x, y) is equivalent to x /= y in Python.

PythonCall.pyimod — Function
pyimod(x, y)

In-place subtraction. x = pyimod(x, y) is equivalent to x %= y in Python.

PythonCall.pyilshift — Function
pyilshift(x, y)

In-place left shift. x = pyilshift(x, y) is equivalent to x <<= y in Python.

PythonCall.pyirshift — Function
pyirshift(x, y)

In-place right shift. x = pyirshift(x, y) is equivalent to x >>= y in Python.

PythonCall.pyiand — Function
pyiand(x, y)

In-place and. x = pyiand(x, y) is equivalent to x &= y in Python.

PythonCall.pyixor — Function
pyixor(x, y)

In-place xor. x = pyixor(x, y) is equivalent to x ^= y in Python.

PythonCall.pyior — Function
pyior(x, y)

In-place or. x = pyior(x, y) is equivalent to x |= y in Python.

Logic

These functions are equivalent to the corresponding Python logical operators.

Note that the equivalent Julia operators are overloaded to call these when all arguments are Py (or Number). Hence the following are equivalent: Py(1) < Py(2), Py(1) < 2, pylt(1, 2), pylt(Py(1), Py(2)), etc.

Note that the binary operators by default return Py (not Bool) since comparisons in Python do not necessarily return bool.

PythonCall.pytruth — Function
pytruth(x)

The truthyness of x. Equivalent to bool(x) in Python, converted to a Bool.

PythonCall.pynot — Function
pynot(x)

The falsyness of x. Equivalent to not x in Python, converted to a Bool.

PythonCall.pyeq — Function
pyeq(x, y)
pyeq(Bool, x, y)

Equivalent to x == y in Python. The second form converts to Bool.

PythonCall.pyne — Function
pyne(x, y)
pyne(Bool, x, y)

Equivalent to x != y in Python. The second form converts to Bool.

PythonCall.pyle — Function
pyle(x, y)
pyle(Bool, x, y)

Equivalent to x <= y in Python. The second form converts to Bool.

PythonCall.pylt — Function
pylt(x, y)
pylt(Bool, x, y)

Equivalent to x < y in Python. The second form converts to Bool.

PythonCall.pyge — Function
pyge(x, y)
pyge(Bool, x, y)

Equivalent to x >= y in Python. The second form converts to Bool.

PythonCall.pygt — Function
pygt(x, y)
pygt(Bool, x, y)

Equivalent to x > y in Python. The second form converts to Bool.

Managing Python dependencies

PythonCall manages its Python dependencies using Conda. A Conda environment is automatically created in your active Julia environment when PythonCall is loaded, is initialised with at least python and pip, and is activated.

If your project requires more Python dependencies, use the mechanisms below to ensure they are automatically installed.

We strongly recommend that you specify Conda dependencies if possible, instead of pip or script dependencies. This is because Conda can account for all inter-dependencies between packages and so prevent incompatible combinations of packages from being installed.

PythonCallDeps.toml

If you put a file called PythonCallDeps.toml in a project/package/environment which depends on PythonCall, then the dependencies therein will be automatically installed into the Conda environment.

Here is an example (all parts are optional):

[conda]
packages = ["python>=3.6", "scikit-learn"]
channels = ["conda-forge"]

[pip]
packages = ["numpy>=1.21"]
# indexes = [...]

[script]
# expr = "some_julia_expression()"
# file = "/path/to/julia/script.jl"

When PythonCall starts, it will ensure the Conda environment has the given Conda and pip packages installed, and will run the script if specified.

The Deps submodule

Instead of manually editing PythonCallDeps.toml, you can use the submodule PythonCall.Deps to manage the Python dependencies of the current Julia project.

PythonCall.Deps.add — Function
add(...)

Add Python dependencies to the current Julia project.

Keyword arguments (all optional):

  • conda_channels: An iterable of conda channels to use.
  • conda_packages: An iterable of conda packages to install.
  • pip_indexes: An iterable of pip indexes to use.
  • pip_packages: An iterable of pip packages to install.
  • script_expr: An expression to evaluate in the Deps module.
  • script_file: The path to a Julia file to evaluate in the Deps module.
  • resolve=true: When true, immediately resolve the dependencies. Otherwise, the dependencies are not resolved until you call resolve or load PythonCall in a new Julia session.
  • create=false: When true, creates the environment from scratch when resolving.

The conda and pip packages can include version specifiers, such as python>=3.6.

PythonCall.Deps.rm — Function
rm(...)

Remove Python dependencies from the current Julia project.

Keyword arguments (all optional):

  • conda_channels: An iterable of conda channels to remove.
  • conda_packages: An iterable of conda packages to remove.
  • pip_indexes: An iterable of pip indexes to remove.
  • pip_packages: An iterable of pip packages to remove.
  • script_expr=false: When true, remove the script expression.
  • script_file=false: When true, remove the script file.
  • resolve=true: When true, immediately resolve the dependencies. Otherwise, the dependencies are not resolved until you call resolve or load PythonCall in a new Julia session.
  • create=true: When true, creates the environment from scratch when resolving.
PythonCall.Deps.resolve — Function
resolve(; create=true, force=false)

Resolve all Python dependencies.

If create=true then a new Conda environment is created and activated. Otherwise, the existing one is updated.

By default, if no dependencies have actually changed, then resolving them is skipped. Specify force=true to skip this check and force resolving dependencies.

The Python interpreter

By default, python is automatically installed into the Conda environment mentioned above.

To use a different interpreter, you can set the environment variable JULIA_PYTHONCALL_EXE to its path before importing PythonCall. You can set it to python if it is in your PATH.

You can also set it to the special value "@PyCall" which will use the same interpreter as PyCall.

Note that using a non-default interpreter will disable all dependency management: no Conda environment will be created and no packages will be automatically installed. It is up to the user to ensure any required packages are installed.

Writing packages which depend on PythonCall

Example

See [https://github.com/cjdoris/Faiss.jl] for an example package which wraps the Python FAISS package.

Precompilation

You may not interact with Python during module precompilation. Therefore, instead of

module MyModule
  using PythonCall
  const foo = pyimport("foo")
  bar() = foo.bar() # will crash when called
end

you must do

module MyModule
  using PythonCall
  const foo = PythonCall.pynew() # initially NULL
  function __init__()
    PythonCall.pycopy!(foo, pyimport("foo"))
  end
  bar() = foo.bar() # now ok
end

Dependencies

If your package depends on some Python packages, you must write a PythonCallDeps.toml file. See Managing Python dependencies.

Low-level API

The functions here are not exported. They are mostly unsafe in the sense that you can crash Julia by using them incorrectly.

PythonCall.pynew — Function
pynew([ptr])

A new Py representing the Python object at ptr (NULL by default).

If ptr is given and non-NULL, this function steals a reference to the Python object it points at, i.e. the new Py object owns a reference.

Note that NULL Python objects are not safe in the sense that most API functions will probably crash your Julia session if you pass a NULL argument.

PythonCall.pycopy! — Function
pycopy!(dst::Py, src)

Copy the Python object src into dst, so that they both represent the same object.

This function exists to support module-level constant Python objects. It is illegal to call most PythonCall API functions at the top level of a module (i.e. before __init__() has run) so you cannot do const x = pything() at the top level. Instead do const x = pynew() at the top level then pycopy!(x, pything()) inside __init__().

Assumes dst is NULL, otherwise a memory leak will occur.

PythonCall.getptr — Function
getptr(x)

Get the underlying pointer from the Python object x.

PythonCall.pydel! — Function
pydel!(x::Py)

Delete the Python object x.

DANGER! Use this function ONLY IF the Julia object x could have been garbage-collected anyway, i.e. was about to become unreachable. This means you MUST KNOW that no other part of the program has the Julia object x.

This decrements the reference count, sets the pointer to NULL and appends x to a cache of unused objects (PYNULL_CACHE).

This is an optimization to avoid excessive allocation and deallocation in Julia, which can be a significant source of slow-down in code which uses a lot of Python objects. It allows pynew() to pop an item from PYNULL_CACHE instead of allocating one, and avoids calling the relatively slow finalizer on x.

PythonCall.unsafe_pynext — Function
unsafe_pynext(x)

Return the next item in the iterator x. When there are no more items, return NULL.