CSTParser.EXPRType

EXPR represents Julia expressions overlaying a span of bytes in the source text. The full span starts at the first syntactically significant token and includes any trailing whitespace/comments.

Iterating or directly indexing EXPR results in a sequence of child EXPR in source order, including most syntax trivia but not including whitespace, comments and semicolons.

The fields of EXPR are:

  • head represents the type of the expression

    • For internal tree nodes it usually matches the associated Expr's head field. But not always because there's some additional heads, for example :brackets for grouping parentheses, :globalrefdoc, :quotenode, etc
    • For leaf nodes (ie, individual tokens), it's capitalized. Eg, :INTEGER for integer tokens, :END for end, :LPAREN for [, etc.
    • For syntactic operators such as = and <: (which have the operator itself as the expression head in normal Expr), the head is an EXPR.
  • args are the significant subexpressions, in the order used by Base.Expr. For leaf nodes, this is nothing.

  • trivia are any nontrivial tokens which are trivial after parsing.

    • This includes things like the parentheses in (1 + 2), and the keywords in begin x end
    • Whitespace and comments are not included in trivia
  • fullspan is the total number of bytes of text covered by this expression, including any trailing whitespace or comment trivia.

  • span is the number of bytes of text covered by the syntactically relevant part of this expression (ie, not including trailing whitespace or comment trivia).

  • val is the source text covered by span

  • parent is the parent node in the expression tree, or Nothing for the root.

  • meta contains metadata. This includes some ad-hoc information supplied by the parser. (But can also be used downstream in linting or static analysis.)

Whitespace, comments and semicolons are not represented explicitly. Rather, they're tacked onto the end of leaf tokens in args or trivia, in the last fullspan-span bytes of the token.

CSTParser._do_kw_convertMethod
_do_kw_convert(ps::ParseState, a::EXPR)

Should a be converted to a keyword-argument expression?

CSTParser._kw_convertMethod
_kw_convert(ps::ParseState, a::EXPR)

Converted an assignment expression to a keyword-argument expression.

CSTParser.can_become_chainMethod
can_become_chain(x::EXPR, op::EXPR)

Is x a binary call for + or * that can be extended to include more arguments?

CSTParser.can_become_comparisonMethod
can_become_comparison(x::EXPR)

Is x a binary comparison call (e.g. a < b) that can be extended to include more arguments?

CSTParser.check_spanFunction

check_span(x, neq = [])

Recursively checks whether the span of an expression equals the sum of the span of its components. Returns a vector of failing expressions.

CSTParser.closerMethod

closer(ps::ParseState)

A magical function determining whether the parsing of an expression should continue or stop.

CSTParser.compareMethod
compare(x,y)

Recursively checks whether two Base.Expr are the same. Returns unequal sub- expressions.

CSTParser.convertsigtotupleMethod
convertsigtotuple(sig::EXPR)

When parsing a function or macro signature, should it be converted to a tuple?

CSTParser.disallowednumberjuxtMethod
disallowednumberjuxt(ret::EXPR)

Does this number literal end in a decimal and so cannot precede a paren for implicit multiplication?

CSTParser.docableMethod
docable(head)

When parsing a block of expressions, can documentation be attached? Prefixed docs at the top-level are handled within parse(ps::ParseState, cont = false).

CSTParser.find_arg_atMethod
find_arg_at(x, i)

Returns the index of the node of x within which the byte offset i falls.

CSTParser.firstdiffMethod
firstdiff(s0::AbstractString, s1::AbstractString)

Returns the last byte index, i, for which s0 and s1 are the same such that: s0[1:i] == s1[1:i]

CSTParser.get_sigMethod
get_sig(x)

Returns the full signature of function, macro and datatype definitions. Should only be called when has_sig(x) == true.

CSTParser.has_errorMethod
has_error(ps::ParseState)
has_error(x::EXPR)

Determine whether a parsing error occured while processing text with the given ParseState, or exists as a (sub) expression of x.

CSTParser.is_rangeMethod
is_range(x::EXPR)

Is x a valid iterator for use in for loops or generators?

CSTParser.parseFunction
parse(str, cont = false)

Parses the passed string. If cont is true then will continue parsing until the end of the string returning the resulting expressions in a TOPLEVEL block.

CSTParser.parse_arrayFunction
parse_array(ps)

Having hit '[' return either:

  • A vect
  • A vcat
  • A ncat
  • A comprehension
  • An array (vcat of hcats)
CSTParser.parse_blockFunction

Continue parsing statements until an element of closers is hit (usually end). Statements are grouped in a Block EXPR.

CSTParser.parse_blockexprMethod
parse_blockexpr(ps::ParseState, head)

General function for parsing block expressions comprised of a series of statements terminated by an end.

CSTParser.parse_blockexpr_sigMethod
parse_blockexpr_sig(ps::ParseState, head)

Utility function to parse the signature of a block statement (i.e. any statement preceding the main body of the block). Returns nothing in some cases (e.g. begin end)

CSTParser.parse_callFunction
parse_call(ps, ret)

Parses a function call. Expects to start before the opening parentheses and is passed the expression declaring the function name, ret.

CSTParser.parse_comma_sepFunction

Parses a comma separated list, optionally allowing for conversion of assignment (=) expressions to Kw.

CSTParser.parse_compoundMethod
parse_compound(ps::ParseState, ret::EXPR)

Attempts to parse a compound expression given the preceding expression ret.

CSTParser.parse_curlyMethod

parse_curly(ps, ret)

Parses the juxtaposition of ret with an opening brace. Parses a comma seperated list.

CSTParser.parse_docMethod
parse_doc(ps::ParseState)

Used for top-level parsing - attaches documentation (such as this) to expressions.

CSTParser.parse_expressionFunction
parse_expression(ps)

Parses an expression until closer(ps) == true. Expects to enter the ParseState the token before the the beginning of the expression and ends on the last token.

Acceptable starting tokens are:

  • A keyword
  • An opening parentheses or brace.
  • An operator.
  • An instance (e.g. identifier, number, etc.)
  • An @.
CSTParser.parse_filterMethod

parse_filter(ps::ParseState, arg)

Parse a conditional filter following a generator.

CSTParser.parse_generatorMethod

parse_generator(ps)

Having hit for not at the beginning of an expression return a generator. Comprehensions are parsed as SQUAREs containing a generator.

CSTParser.parse_iteratorFunction

Parses an iterator, allowing for the preceding keyword outer. Returns an error expression if an invalid expression is parsed (anything other than =, in, ).

CSTParser.parse_iteratorsMethod
parse_iterators(ps::ParseState, allowfilter = false)

Parses a group of iterators e.g. used in a for loop or generator. Can allow for a succeeding Filter expression.

CSTParser.parse_kwMethod
parse_kw(ps::ParseState)

Dispatch function for when the parser has reached a keyword.

CSTParser.parse_parametersFunction
parse_parameters(ps::ParseState, args::Vector{EXPR}, args1::Vector{EXPR} = EXPR[]; usekw = true)

Parses parameter arguments for a function call (e.g. following a semicolon).

CSTParser.parse_refMethod
parse_ref(ps, ret)

Handles cases where an expression - ret - is followed by [. Parses the following bracketed expression and modifies it's .head appropriately.

CSTParser.parse_string_or_cmdFunction

parsestringor_cmd(ps)

When trying to make an INSTANCE from a string token we must check for interpolating operators.

CSTParser.parse_tupleFunction

parse_tuple(ps, ret)

ret is followed by a comma so tries to parse the rest of the tuple.

CSTParser.parse_unaryMethod
parse_unary(ps)

Having hit a unary operator at the start of an expression return a call.

CSTParser.read_ws_commentMethod
lex_ws_comment(l::Lexer, c)

Having hit an initial whitespace/comment/semicolon continues collecting similar Chars until they end. Returns a WS token with an indication of newlines/ semicolons. Indicating a semicolons takes precedence over line breaks as the former is equivalent to the former in most cases.

CSTParser.revfirstdiffMethod
revfirstdiff(s0::AbstractString, s1::AbstractString)

Reversed version of firstdiff but returns two indices, one for each string.

CSTParser.str_valueMethod
str_value(x)

Attempt to get a string representation of a nodeless expression.

CSTParser.to_codeobjectMethod
to_codeobject(x::EXPR)

Convert an EXPR into the object that Meta.parse would have produced from the original string, which could e.g. be an Expr, Symbol, or literal.

CSTParser.@defaultMacro
@default ps body

Parses the next expression using default closure rules.

CSTParser.@precedenceMacro
@precedence ps prec body

Continues parsing binary operators until it hits a more loosely binding operator (with precdence lower than prec).