The Julia module PythonCall

Installation

This package is in the general registry, so to install just type ] in the Julia REPL and run:

pkg> add PythonCall

Getting started

Import the module with:

julia> using PythonCall

By default this will initialize a conda environment in your Julia environment, install Python into it, load the corresponding Python library and initialize an interpreter. See here to configure which Python to use.

Now you can interact with Python as follows:

julia> re = pyimport("re")
Python module: <module 're' from '[...]/lib/re.py'>

julia> words = re.findall("[a-zA-Z]+", "PythonCall.jl is very useful!")
Python list: ['PythonCall', 'jl', 'is', 'very', 'useful']

julia> sentence = Py(" ").join(words)
Python str: 'PythonCall jl is very useful'

julia> pyconvert(String, sentence)
"PythonCall jl is very useful"

In this example:

We used pyimport to import the re module.
We called its findall function on a pair of strings, which were automatically converted to Python strings (see Conversion to Python).
We called Py to explicitly convert a string to a Python string, so that we could call its join method. All Python objects are of type Py.
We called pyconvert to convert the Python string sentence to a Julia string (see Conversion to Julia).

The values re, words and sentence in the example are all Python objects, which have type Py in Julia. As we have seen, these objects support attribute access (e.g. re.findall) and function calls (e.g. join(words)). They also support indexing, comparison and arithmetic:

julia> x = pylist([3, 4, 5])
Python list: [3, 4, 5]

julia> x[2] == 5
Python bool: True

julia> x[pyslice(0,2)] + pylist([1,2])
Python list: [3, 4, 1, 2]

We have just seen the functions pylist (for constructing a Python list) and pyslice (for constructing a Python slice). There are many such functions, mirroring most of the Python builtin functions and types. The API Reference documents them all.

Most of these functions are essentially Python builtins with a py prefix. For example pyint(x) converts x to a Python int and is equivalent to int(x) in Python when x is a Python object.

Notable exceptions are:

pyconvert to convert a Python object to a Julia object.
pyimport to import a Python module.
pyjl to directly wrap a Julia object as a Python object.
pywith to emulate the Python with statement.

To access the Python builtins directly, you can access the fields of pybuiltins:

julia> pybuiltins.None
Python None

julia> pybuiltins.True
Python bool: True

julia> pybuiltins.ValueError("some error")
Python ValueError: ValueError('some error')

With the functions introduced so far, you have access to the vast majority of Python's functionality.

Conversion between Julia and Python

A Julia object can be converted to a Python one either explicitly (such as Py(x)) or implicitly (such as the arguments when calling a Python function). Either way, it follows the default conversion rules here.

Most operations involving Python objects will return a Py and are not automatically converted to another Julia type. Instead, you can explicitly convert using pyconvert:

julia> x = pylist([3.4, 5.6])
Python list: [3.4, 5.6]

julia> pyconvert(Vector, x)
2-element Vector{Float64}:
 3.4
 5.6

julia> pyconvert(Vector{Float32}, x)
2-element Vector{Float32}:
 3.4
 5.6

julia> pyconvert(Any, x)
2-element PyList{Py}:
 3.4
 5.6

In the above example, we converted a Python list to a Julia vector in three ways.

pyconvert(Vector, x) returned a Vector{Float64} since all the list items are floats.
pyconvert(Vector{Float32}, x) specified the element type, so the floats were converted to Float32.
pyconvert(Any, x) returned a PyList{Py} which is a no-copy wrapper around the original list x, viewing it as a AbstractVector{Py}. Since it is a wrapper, mutating it mutates x and vice-versa.

See here for the rules regarding how pyconvert(T, x) works. If x is an immutable scalar type (such as an int or str) then pyconvert(Any, x) may return the corresponding Julia object (such as an Integer or String). Otherwise it will typically return either a wrapper type (such as PyList{Py} in the above example) or will fall back to returning a Py.

Wrapper types

A wrapper is a type which wraps a Python object but provides it with the semantics of some other Julia type.

Since it is merely wrapping a Python object, if you mutate the wrapper you also mutate the wrapped object, and vice versa.

See here for details of all the wrapper types provided by PythonCall.

We have already seen PyList. It wraps any Python sequence (such as a list) as a Julia vector:

julia> x = pylist([3,4,5])
Python list: [3, 4, 5]

julia> y = PyList{Union{Int,Nothing}}(x)
3-element PyList{Union{Nothing, Int64}}:
 3
 4
 5

julia> push!(y, nothing)
4-element PyList{Union{Nothing, Int64}}:
 3
 4
 5
  nothing

julia> append!(y, 1:2)
6-element PyList{Union{Nothing, Int64}}:
 3
 4
 5
  nothing
 1
 2

julia> x
Python list: [3, 4, 5, None, 1, 2]

There are wrappers for other container types, such as PyDict and PySet.

The wrapper PyArray provides a Julia array view of any Python array, i.e. anything satisfying either the buffer protocol or the numpy array interface. This includes things like bytes, bytearray, array.array and numpy.ndarray:

julia> x = pyimport("array").array("i", [3, 4, 5])
Python array: array('i', [3, 4, 5])

julia> y = PyArray(x)
3-element PyArray{Int32, 1, true, true, Int32}:
 3
 4
 5

julia> sum(y)
12

julia> y[1] = 0
0

julia> x
Python array: array('i', [0, 4, 5])

It directly wraps the underlying data buffer, so array operations such as indexing are about as fast as for an ordinary Array.

The PyIO wrapper type views a Python file object as a Julia IO object:

julia> x = pyimport("io").StringIO()
Python StringIO: <_io.StringIO object at 0x000000006579BC70>

julia> y = PyIO(x)
PyIO(<py _io.StringIO object at 0x000000006579BC70>, false, true, false, 4096, UInt8[], 4096, UInt8[])

julia> println(y, "Hello, world!")

julia> flush(y)

julia> x.seek(0)
Python int: 0

julia> x.read()
Python str: 'Hello, world!\n'

Configuration

By default, PythonCall uses CondaPkg.jl to manage its dependencies. This will install Conda and use it to create a Conda environment specific to your current Julia project containing Python and any required Python packages.

If you already have Conda, Mamba or MicroMamba

ENV["JULIA_CONDAPKG_BACKEND"] = "System"
ENV["JULIA_CONDAPKG_EXE"] = "/path/to/conda"  # optional

The System backend to CondaPkg will use your preinstalled Conda implementation instead of downloading one.

If conda, mamba or micromamba is not in your PATH you will also need to set JULIA_CONDAPKG_EXE to its path.

If you don't want to use Conda

ENV["JULIA_PYTHONCALL_EXE"] = "/path/to/python"
ENV["JULIA_PYTHONCALL_EXE"] = "@PyCall" # optional
ENV["JULIA_CONDAPKG_BACKEND"] = "Null"  # optional

You will need to install Python and any Python packages required yourself, and ensure that they are at compatible versions.

If you also use PyCall, you can set JULIA_PYTHONCALL_EXE=@PyCall to use the same Python interpreter.

You may also set JULIA_CONDAPKG_BACKEND=Null to completely disable CondaPkg. This should not be necessary because PythonCall will not use CondaPkg if the above settings are used.

Installing Python packages

Assuming you haven't opted out, PythonCall uses CondaPkg.jl to automatically install any required Python packages.

This is as simple as

julia> using CondaPkg

julia> # press ] to enter the Pkg REPL

pkg> conda add some_package

This creates a CondaPkg.toml file in the active project specifying the dependencies, just like a Project.toml specifies Julia dependencies. Commit this file along with the rest of the project so that dependencies are automatically installed for everyone using it.

To add dependencies to a Julia package, just ensure the package project is activated first.

See the CondaPkg.jl documentation.

module MyModule
  using PythonCall
  const foo = pyimport("foo")
  bar() = foo.bar() # will crash when called
end

you must do

module MyModule
  using PythonCall
  const foo = PythonCall.pynew() # initially NULL
  function __init__()
    PythonCall.pycopy!(foo, pyimport("foo"))
  end
  bar() = foo.bar() # now ok
end

Dependencies

If your package depends on some Python packages, you must generate a CondaPkg.toml file. See Installing Python packages.