Post Processing

Often, it is necessary to perform post-processing and combine the results from multiple simulations into one. e4st_post attempts to do that in a streamlined way, only deserializing data a single time for each of the runs, and only storing one full data dictionary in memory at a time.

E4ST.e4st_post — Function

e4st_post(post_config)

Runs post-processing on post_config. See read_post_config for information about the post_config. The general outline of post processing is:

post_data = extract_results(post_config): Create a post_data file and extract the results from each simulation from each mod into it.
combine_results(post_config, post_data): allow the mod to combine anything into a single result, plot, etc. Store any output into get_out_path(post_config).

E4ST.read_post_config — Function

read_post_config(filenames...; create_out_path = true, kwargs...) -> post_config

Reads a config file for e4st_post. See summarize_post_config for the required fields.

E4ST.summarize_post_config — Function

summarize_post_config() -> summary

Summarizes the post_config, with columns for:

name - the property name, i.e. key
required - whether or not the property is required
default - default value of this property
description

name	required	default	description
`sim_paths`	true	nothing	The paths to the desired simulation outputs.
`sim_names`	false	nothing	The names of each of the sims. This will get used in post processing. If none given, `e4st_post` will check the configs in each of the `sim_paths` to see if there is a `name` given.
`base_sim_name`	false		The name of the base simulation to use for comparisons. Used by Modifications
`out_path`	true	nothing	The path to the desired output path for the results of postprocessing.
`mods`	false	OrderedCollections.OrderedDict{Symbol, E4ST.Modification}()	A list of `Modification`s specifying changes for how `e4st_post` runs. See `extract_results` and `combine_results`.

E4ST.extract_results — Function

extract_results(m::LeftJoinCols, config, data) ->

extract_results(post_config) -> post_data::OrderedDict{Symbol, Any}

Initializes post_data, and extracts results for each modification in post_config[:mods], for each of the simulations in post_config[:sim_paths] and post_config[:sim_names].

Calls extract_results(post_mod, config, data) for each post_mod, where config is the simulation config, and data is the simulation data, read in from read_processed_results.

extract_results(post_mod::Modification, config, data) -> results

One of the main steps in e4st_post. Extract results (or compute new ones) from data (the full set of data deserialized from an E4ST run). This will get stored into an OrderedDict mapping sim_name to results. See combine_results for the next step.

Note that we do this to prevent excessive memory usage. If e4st_post is run with a large number of simulations, storing the entire set of data in memory for all of them may cost too much RAM.

E4ST.combine_results — Function

combine_results(post_config, post_data) -> nothing

Combine results for each of the mods in post_config[:mods]. Calls combine_results(post_mod, post_config, post_mod_data), where post_mod_data is post_data[mod_name].

combine_results(post_mod::Modification, post_config, post_mod_data)

Combine results and probably save them to the out path specified in post_config.

post_mod_data is the porton of post_data for this particular post_mod, an OrderedDict mapping sim_name to the extracted results from extract_results.

E4ST.join_sim_tables — Function

join_sim_tables(post_mod_data, keep_col)

Joins tables for multiple sims stored in post_mod_data, with keep_col as the column to keep, and the remaining columns as the joining columns.

replace_missing - replaces missing values in res after the tables are joined with the value of the kw arg. To keep missing values, set replace_missing = missing.