Post Processing

Often, it is necessary to perform post-processing and combine the results from multiple simulations into one. e4st_post attempts to do that in a streamlined way, only deserializing data a single time for each of the runs, and only storing one full data dictionary in memory at a time.

summarize_post_config() -> summary

Summarizes the post_config, with columns for:

  • name - the property name, i.e. key
  • required - whether or not the property is required
  • default - default value of this property
  • description
sim_pathstruenothingThe paths to the desired simulation outputs.
sim_namesfalsenothingThe names of each of the sims. This will get used in post processing. If none given, e4st_post will check the configs in each of the sim_paths to see if there is a name given.
base_sim_namefalseThe name of the base simulation to use for comparisons. Used by Modifications
out_pathtruenothingThe path to the desired output path for the results of postprocessing.
modsfalseOrderedCollections.OrderedDict{Symbol, E4ST.Modification}()A list of Modifications specifying changes for how e4st_post runs. See extract_results and combine_results.
extract_results(post_config) -> post_data::OrderedDict{Symbol, Any}

Initializes post_data, and extracts results for each modification in post_config[:mods], for each of the simulations in post_config[:sim_paths] and post_config[:sim_names].

Calls extract_results(post_mod, config, data) for each post_mod, where config is the simulation config, and data is the simulation data, read in from read_processed_results.

extract_results(post_mod::Modification, config, data) -> results

One of the main steps in e4st_post. Extract results (or compute new ones) from data (the full set of data deserialized from an E4ST run). This will get stored into an OrderedDict mapping sim_name to results. See combine_results for the next step.

Note that we do this to prevent excessive memory usage. If e4st_post is run with a large number of simulations, storing the entire set of data in memory for all of them may cost too much RAM.

combine_results(post_config, post_data) -> nothing

Combine results for each of the mods in post_config[:mods]. Calls combine_results(post_mod, post_config, post_mod_data), where post_mod_data is post_data[mod_name].

combine_results(post_mod::Modification, post_config, post_mod_data)

Combine results and probably save them to the out path specified in post_config.

post_mod_data is the porton of post_data for this particular post_mod, an OrderedDict mapping sim_name to the extracted results from extract_results.

join_sim_tables(post_mod_data, keep_col)

Joins tables for multiple sims stored in post_mod_data, with keep_col as the column to keep, and the remaining columns as the joining columns.

  • replace_missing - replaces missing values in res after the tables are joined with the value of the kw arg. To keep missing values, set replace_missing = missing.