Data

read_data(config) -> data

Pulls in data found in files listed in the config, and stores into data.

Calls the following functions:

• read_data_files!(config, data) - read in the data from files
• modify_raw_data!(config, data) - Gives Modifications a chance to modify the raw data before the data gets setup.
• setup_data!(config, data) - Sets up the data, modifying/adding to the tables as needed.
• modify_setup_data!(config, data) - Gives Modifications a chance to modify the setup data before the model is built.

read_data_files!(config, data)

Loads in the data files presented in the config.

modify_raw_data!(config, data)

Allows Modifications to modify the raw data - calls modify_raw_data!(mod, config, data)

setup_data!(config, data)

Sets up the data, modifying, adding to, or combining the tables as needed. New generators built in the setup_gen_table! function.

modify_setup_data!(config, data)

Allows Modifications to modify the raw data - calls modify_setup_data!(mod, config, data)

read_summary_table!(config, data)

Loads in the summary table for each of the other tables.

setup_table!(config, data, table_name)

Sets up the data[:table_name]. Calls setup_table!(config, data, Val(table_name)), if defined.

get_table_summary(data, table_name) -> summary::SubDataFrame

Returns a summary of table_name, read in from summarize_table and read_summary_table!.

get_table_names(data) -> table_list

Returns a list of all the tables in data.

get_table(data, table_name, conditions...) -> subtable::SubDataFrame

Return a subset of the table table_name for which the row passes the conditions. Conditions are Pairs generally consisting of <column name> => value. Here are some examples of supported conditions:

• :genfuel => "ng" - All rows for which row.genfuel == "ng"
• "bus_idx" => 1 - All rows for which row.bus_idx == 1. Note that the column name can be String or Symbol
• :bus_idx => "1" - All rows for which row.bus_idx == 1. Note that this is a String but it will get converted to the eltype of table.bus_idx for the comparison
• :year_on => ("y2022", "y2030") - All rows for which row.year_on is between "y2022" and "y2030", inclusive. Also works for fractional years.
• :genfuel => ["ng", "solar", "wind"] - All rows for which row.genfuel is either "ng", "solar", or "wind"
• :emis_co2 => f::Function - All rows for which f(row.emis_co2) returns true. For example >(0), or x->(0<x<=0.5)

get_table(data, table_name::Symbol) -> table::DataFrame

Retrieves data[table_name], enforcing that it is a DataFrame. See get_table_names for a list of available tables.

get_table_row_idxs(data, table_name, conditions...) -> row_idxs::Vector{Int64}

Gets the row indices for data[table_name] for which the conditions hold true. See get_table for a description of possible conditions

get_table_val(data, table_name, col_name, row_idx) -> val

Returns the value of the table at column col_name and row row_idx

Related functions:

• get_table_num(data, table_name, col_name, row_idx, yr_idx, hr_idx): retrieves the Float64 from data[variable_name], indexing by year and hour.
• get_num(data, name, yr_idx, hr_idx): retrieves a Float64 from data, indexing by year and hour.
• get_val(data, variable_name): retrieves the value from data[variable_name] regardless of type, not indexed by row, year or hour.

get_table_num(data, table_name, col_name, row_idx, yr_idx, hr_idx) -> num::Float64

Retrieves a Float64 from table[row_idx, col_idx][yr_idx, hr_idx]. This indexes into Containers as needed and will still work for Float64 columns.

Related functions:

• get_table_val(data, table_name, col_name, row_idx): retrieves the raw value from the table (without indexing by year/hour).
• get_num(data, name, yr_idx, hr_idx): retrieves a Float64 from data, indexing by year and hour.

get_table_col(data, table_name, col_name) -> col::Vector

get_table_col_type(data, table_name, column_name) -> ::Type

get_table_col_unit(data, table_name, column_name) -> unit::Type{<:Unit}

get_table_col_description(data, table_name, column_name) -> description

get_row_idxs(table, conditions) -> row_idxs

Returns row indices of the passed-in table that correspond to conditions, where conditions can be:

• ::Colon - all rows
• ::Int64 - a single row
• ::AbstractVector{Int64} - a list of rows
• p::Pair - returns a Vector containing the index of each row for which comparison(p[2], typeof(row[p[1]]))(row[p[1]]) is true. See comparison
• pairs, an iterator of Pairs - returns a Vector containing the indices which satisfy all the pairs as above.

Some possible pairs to filter by:

• :nation => "narnia": checks if the nation column is equal to the string "narnia"
• :emis_co2 => >=(0.1): checks if the emis_co2 column is greater than or equal to 0.1
• :age => (2,10): checks if the age column is between 2, and 10, inclusive. To be exclusive, use different values like (2.0001, 9.99999) for clarity
• :state => in(("alabama", "arkansas")): checks if the state column is either "alabama" or "arkansas"

Used in get_table and get_table_row_idxs.

get_year_idxs(data, year_idxs) -> idxs

Converts year_idxs into a usable set of indices that can index into get_years(data). year_idxs can be any of the following types:

• Colon
• Int64
• AbstractVector{Int64}
• AbstractString - Representing the year, i.e. "y2020"
• AbstractVector{<:AbstractString} - a vector of strings representing the year, i.e. "y2020"
• Tuple{<:AbstractString, <:AbstractString}
• Function - a function of the year string that returns a boolean. I.e. <=("y2030")

get_hour_idxs(data, hour_idxs)

Converts hour_idxs into a usable set of indices that can index into hourly data. hour_idxs can be any of the following types:

• Colon
• Int64
• AbstractVector{Int64}

has_table(data, table_name) -> Bool

get_num(data, variable_name, yr_idx, hr_idx) -> num::Float64

get_num(table, col_name, row_idx, yr_idx, hr_idx) -> num::Float64

Retrieves a Float64 from data[variable_name], indexing by year and hour. Works for Containers and Numbers.

Related functions:

• get_table_val(data, table_name, col_name, row_idx): retrieves the raw value from the table (without indexing by year/hour).
• get_table_num(data, table_name, col_name, row_idx, yr_idx, hr_idx): retrieves the Float64 from data[variable_name], indexing by year and hour.
• get_val(data, variable_name): retrieves the value from data[variable_name] regardless of type, not indexed by row, year or hour.

parse_comparison(s) -> comp

Parses the string, s for a comparison with which to filter a table.

Possible examples of strings s to parse:

• "nation=>narnia" - All rows for which row.nation=="narnia"
• "bus_idx=>5" - All rows for which row.bus_idx==5
• "year_on=>(y2002,y2030)" - All rows for which row.year_on is between 2002 and 2030, inclusive.
• "emis_co2=>(0.0,4.99)" - All rows for which row.emis_co2 is between 0.0 and 4.99, inclusive. (Works for integers and negatives too)
• "emis_co2=> >(0)" - All rows for which row.emis_co2 is greater than 0 (Works for integers and negatives too)
• "year_on=> >(y2002) - All rows for which row.year_on is greater than "y2002" (works for fractional years too, such as "y2002.4")
• "genfuel=>[ng, wind, solar]" - All rows for which row.genfuel is "ng", "wind", or "solar". Works for Ints and Floats too.
• "genfuel=>![ng, coal, biomass]" - All rows for which row.genfuel is not "ng", "coal", or "biomass". Works for Ints and Floats too.

parse_comparisons(row::DataFrameRow) -> pairs

Returns a set of pairs to be used in filtering rows of another table. Looks for the following properties in the row:

• area, subarea - if the row has a non-empty area and subarea, it will parse the comparison row.area=>row.subarea
• filter_ - if the row has any non-empty filter_ (i.e. filter1, filter2) values, it will parse the comparison via parse_comparison
• genfuel - if the row has a non-empty genfuel, it will add an comparion that checks that each row's genfuel equals this value
• gentype - if the row has a non-empty gentype, it will add an comparion that checks that each row's gentype equals this value
• load_type - if the row has a non-empty load_type, it will add an comparion that checks that each row's load_type equals this value

parse_comparisons(d::AbstractDict) -> pairs

Returns a set of pairs to be used in filtering rows of another table, where each value d

parse_year_idxs(s::AbstractString) -> comparisons

Parse a year comparison. Could take the following forms:

• "y2020" - year 2020 only
• "" - All years, returns (:)
• "1" - year index 1
• "[1,2,3]"

parse_hour_idxs(s::AbstractString) -> comparisons

Parse a year comparison. Could take the following forms:

• "1" - hour 1 only
• "" - All hours, returns (:)
• "season=>winter" - returns "season"=>"winter"

comparison(value, v) -> comp::Function

Returns the appropriate comparison function for value to be compared to each member of v.

comparison(value, ::Type) -> comp::Function

Returns the appropriate comparison function for value to be compared to the 2nd argument type. Here are a few options:

• comparison(f::Function, ::Type) -> f
• comparison(s::String, ::Type{<:AbstractString}) -> ==(s)