EXPR represents Julia expressions overlaying a span of bytes in the source text. The full span starts at the first syntactically significant token and includes any trailing whitespace/comments.

Iterating or directly indexing EXPR results in a sequence of child EXPR in source order, including most syntax trivia but not including whitespace, comments and semicolons.

The fields of EXPR are:

• head represents the type of the expression

  • For internal tree nodes it usually matches the associated Expr's head field. But not always because there's some additional heads, for example :brackets for grouping parentheses, :globalrefdoc, :quotenode, etc
  • For leaf nodes (ie, individual tokens), it's capitalized. Eg, :INTEGER for integer tokens, :END for end, :LPAREN for [, etc.
  • For syntactic operators such as = and <: (which have the operator itself as the expression head in normal Expr), the head is an EXPR.

• args are the significant subexpressions, in the order used by Base.Expr. For leaf nodes, this is nothing.

• trivia are any nontrivial tokens which are trivial after parsing.

  • This includes things like the parentheses in (1 + 2), and the keywords in begin x end
  • Whitespace and comments are not included in trivia

• fullspan is the total number of bytes of text covered by this expression, including any trailing whitespace or comment trivia.

• span is the number of bytes of text covered by the syntactically relevant part of this expression (ie, not including trailing whitespace or comment trivia).

• val is the source text covered by span

• parent is the parent node in the expression tree, or Nothing for the root.

• meta contains metadata. This includes some ad-hoc information supplied by the parser. (But can also be used downstream in linting or static analysis.)

Whitespace, comments and semicolons are not represented explicitly. Rather, they're tacked onto the end of leaf tokens in args or trivia, in the last fullspan-span bytes of the token.

_do_kw_convert(ps::ParseState, a::EXPR)

Should a be converted to a keyword-argument expression?

_kw_convert(a::EXPR, blockwrap = false)

Converted an assignment expression to a keyword-argument expression.

can_become_chain(x::EXPR, op::EXPR)

Is x a binary call for + or * that can be extended to include more arguments?

can_become_comparison(x::EXPR)

Is x a binary comparison call (e.g. a < b) that can be extended to include more arguments?

check_span(x, neq = [])

Recursively checks whether the span of an expression equals the sum of the span of its components. Returns a vector of failing expressions.

closer(ps::ParseState)

A magical function determining whether the parsing of an expression should continue or stop.

compare(x,y)

Recursively checks whether two Base.Expr are the same. Returns unequal sub- expressions.

convertsigtotuple(sig::EXPR)

When parsing a function or macro signature, should it be converted to a tuple?

disallowednumberjuxt(ret::EXPR)

Does this number literal end in a decimal and so cannot precede a paren for implicit multiplication?

docable(head)

When parsing a block of expressions, can documentation be attached? Prefixed docs at the top-level are handled within parse(ps::ParseState, cont = false).

find_arg_at(x, i)

Returns the index of the node of x within which the byte offset i falls.

firstdiff(s0::AbstractString, s1::AbstractString)

Returns the last byte index, i, for which s0 and s1 are the same such that: s0[1:i] == s1[1:i]

get_sig(x)

Returns the full signature of function, macro and datatype definitions. Should only be called when has_sig(x) == true.

has_error(ps::ParseState)
has_error(x::EXPR)

Determine whether a parsing error occured while processing text with the given ParseState, or exists as a (sub) expression of x.

is_getfield(x::EXPR)

Is this an expression of the form a.b.

is_range(x::EXPR)

Is x a valid iterator for use in for loops or generators?

is_wrapped_assignment(x::EXPR)

Is x an assignment expression, ignoring any surrounding parentheses.

parse(str, cont = false)

Parses the passed string. If cont is true then will continue parsing until the end of the string returning the resulting expressions in a TOPLEVEL block.

parse_array(ps)

Having hit '[' return either:

• A vect
• A vcat
• A ncat
• A comprehension
• An array (vcat of hcats)

Continue parsing statements until an element of closers is hit (usually end). Statements are grouped in a Block EXPR.

parse_blockexpr(ps::ParseState, head)

General function for parsing block expressions comprised of a series of statements terminated by an end.

parse_blockexpr_sig(ps::ParseState, head)

Utility function to parse the signature of a block statement (i.e. any statement preceding the main body of the block). Returns nothing in some cases (e.g. begin end)

parse_call(ps, ret)

Parses a function call. Expects to start before the opening parentheses and is passed the expression declaring the function name, ret.

Parses a comma separated list, optionally allowing for conversion of assignment (=) expressions to Kw.

parse_compound(ps::ParseState, ret::EXPR)

Attempts to parse a compound expression given the preceding expression ret.

parse_curly(ps, ret)

Parses the juxtaposition of ret with an opening brace. Parses a comma seperated list.

parse_doc(ps::ParseState)

Used for top-level parsing - attaches documentation (such as this) to expressions.

Helper function for parsing import/using statements.

parse_expression(ps)

Parses an expression until closer(ps) == true. Expects to enter the ParseState the token before the the beginning of the expression and ends on the last token.

Acceptable starting tokens are:

• A keyword
• An opening parentheses or brace.
• An operator.
• An instance (e.g. identifier, number, etc.)
• An @.

parse_filter(ps::ParseState, arg)

Parse a conditional filter following a generator.

parse_generator(ps)

Having hit for not at the beginning of an expression return a generator. Comprehensions are parsed as SQUAREs containing a generator.

parse_if(ps, nested=false)

Parse an if block.

Parses an iterator, allowing for the preceding keyword outer. Returns an error expression if an invalid expression is parsed (anything other than =, in, ∈).

parse_iterators(ps::ParseState, allowfilter = false)

Parses a group of iterators e.g. used in a for loop or generator. Can allow for a succeeding Filter expression.

parse_kw(ps::ParseState)

Dispatch function for when the parser has reached a keyword.

parse_macrocall(ps)

Parses a macro call. Expects to start on the @.

parse_parameters(ps::ParseState, args::Vector{EXPR}, args1::Vector{EXPR} = EXPR[]; usekw = true)

Parses parameter arguments for a function call (e.g. following a semicolon).

parse_paren(ps, ret)

Parses an expression starting with a (.

parse_prefixed_string_cmd(ps::ParseState, ret::EXPR)

Parse prefixed strings and commands such as pre"text".

parse_ref(ps, ret)

Handles cases where an expression - ret - is followed by [. Parses the following bracketed expression and modifies it's .head appropriately.

parsestringor_cmd(ps)

When trying to make an INSTANCE from a string token we must check for interpolating operators.

parse_tuple(ps, ret)

ret is followed by a comma so tries to parse the rest of the tuple.

parse_unary(ps)

Having hit a unary operator at the start of an expression return a call.

lex_ws_comment(l::Lexer, c)

Having hit an initial whitespace/comment/semicolon continues collecting similar Chars until they end. Returns a WS token with an indication of newlines/ semicolons. Indicating a semicolons takes precedence over line breaks as the former is equivalent to the former in most cases.

remlineinfo!(x)

Removes line info expressions. (i.e. Expr(:line, 1))

revfirstdiff(s0::AbstractString, s1::AbstractString)

Reversed version of firstdiff but returns two indices, one for each string.

str_value(x)

Attempt to get a string representation of a nodeless expression.

to_codeobject(x::EXPR)

Convert an EXPR into the object that Meta.parse would have produced from the original string, which could e.g. be an Expr, Symbol, or literal.

@closer ps rule body

Continues parsing closing on rule.

@default ps body

Parses the next expression using default closure rules.

@nocloser ps rule body

Continues parsing not closing on rule.

@precedence ps prec body

Continues parsing binary operators until it hits a more loosely binding operator (with precdence lower than prec).