Package 'hubValidations'

Description

Capture a condition of the result of validation check.

Usage

capture_check_cnd(
  check,
  file_path,
  msg_subject,
  msg_attribute,
  msg_verbs = c("is", "must be"),
  error = FALSE,
  details = NULL,
  ...
)

Arguments

Details

Arguments msg_subject, msg_attribute, msg_verbs and details accept text that can interpreted and formatted by cli::format_inline().

Value

Depending on whether validation has succeeded and the value of the error argument, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Examples

capture_check_cnd(
  check = TRUE, file_path = "test/file.csv",
  msg_subject = "{.var round_id}", msg_attribute = "valid.", error = FALSE
)
capture_check_cnd(
  check = FALSE, file_path = "test/file.csv",
  msg_subject = "{.var round_id}", msg_attribute = "valid.", error = FALSE,
  details = "Must be one of 'A' or 'B', not 'C'"
)
capture_check_cnd(
  check = FALSE, file_path = "test/file.csv",
  msg_subject = "{.var round_id}", msg_attribute = "valid.", error = TRUE,
  details = "Must be one of {.val {c('A', 'B')}}, not {.val C}"
)

Description

Capture a simple info message condition. Useful for communicating when a check is ignored or skipped.

Usage

capture_check_info(file_path, msg, call = rlang::caller_call())

Arguments

Value

A ⁠<message/check_info>⁠ condition class object. Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Capture an execution error condition. Useful for communicating when a check execution has failed. Usually used in conjunction with try.

Usage

capture_exec_error(file_path, msg, call = NULL)

Arguments

Value

A ⁠<error/check_exec_error>⁠ condition class object. Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Capture an execution warning condition. Useful for communicating when a check execution has failed. Usually used in conjunction with try.

Usage

capture_exec_warning(file_path, msg, call = NULL)

Arguments

Value

A ⁠<warning/check_exec_warn>⁠ condition class object. Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Capture a warning about the validation process. Unlike check results (success/failure/error), validation warnings are informational messages about the validation process itself rather than validation outcomes. They do not cause validation to fail — check_for_errors() only aborts on check_failure, check_error, check_exec_error or check_exec_warn objects.

Usage

capture_validation_warning(msg, where = NULL, call = rlang::caller_call(), ...)

Arguments

Details

Validation warnings can be attached at two levels:

• Validation-level: Stored as an attribute on hub_validations objects, printed prominently by default.

• Check-level: Stored in a warnings field on individual check results, printed only when verbose = TRUE.

Value

A ⁠<warning/validation_warning>⁠ condition class object.

Examples

# Simple warning
capture_validation_warning(
  msg = "Configuration files were modified"
)

# Warning with location and additional structured data
config_files <- c("tasks.json", "admin.json")
capture_validation_warning(
  msg = "Config files modified: {.path {config_files}}",
  where = "hub-config",
  config_files = config_files
)

Description

Checks that admin and tasks configuration files in directory hub-config are valid.

Usage

check_config_hub_valid(hub_path)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check file exists at the file path specified

Usage

check_file_exists(
  file_path,
  hub_path = ".",
  subdir = c("model-output", "model-metadata", "hub-config", "target-data")
)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check file format is accepted by hub.

Usage

check_file_format(file_path, hub_path, round_id)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Checks that the model_id metadata in the file name matches the directory name the file is being submitted to.

Usage

check_file_location(file_path)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check number of files submitted per round does not exceed the allowed number of submissions per team.

Usage

check_file_n(file_path, hub_path, allowed_n = 1L)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check a model output file name can be correctly parsed.

Usage

check_file_name(file_path)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check file can be read successfully

Usage

check_file_read(file_path, hub_path = ".")

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Checks validation objects for errors and raises conditions if any are found. Works with hub_validations and hub_validations_collection objects, as well as their subclasses (target_validations and target_validations_collection). Can be used in CI workflows to signal validation failures, or locally to summarise validation results.

Usage

check_for_errors(x, verbose = FALSE, show_warnings = FALSE)

Arguments

Details

For more details on these classes, see article on ⁠<hub_validations>⁠ S3 class objects.

Value

An error if one of the elements of x is of class check_failure, check_error, check_exec_error or check_exec_warning. TRUE invisibly otherwise.

See Also

validate_submission(), validate_pr(), validate_target_submission(), validate_target_pr()

Description

Check whether a metadata schema file exists

Usage

check_metadata_file_exists(hub_path = ".", file_path)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Checks that the model_id metadata in the file name matches the directory name the file is being submitted to.

Usage

check_metadata_file_ext(file_path)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check that the metadata file is being submitted to the correct folder

Usage

check_metadata_file_location(file_path)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check whether the file name of a metadata file matches the model_id or combination of team_abbr and model_abbr specified within the metadata file

Usage

check_metadata_file_name(file_path, hub_path = ".")

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check whether a metadata file matches the schema provided by the hub

Usage

check_metadata_matches_schema(file_path, hub_path = ".")

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check whether a metadata schema file exists

Usage

check_metadata_schema_exists(hub_path = ".", file_path)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check whether a metadata file for the given model exists

Usage

check_submission_metadata_file_exists(file_path, hub_path = ".")

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Checks submission is within the valid submission window for a given round.

Usage

check_submission_time(
  hub_path,
  file_path,
  ref_date_from = c("file", "file_path")
)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check target dataset can be detected for a given target type

Usage

check_target_dataset_exists(
  hub_path,
  target_type = c("time-series", "oracle-output")
)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check that all files of a given target type share a single unique file format

Usage

check_target_dataset_file_ext_unique(
  hub_path,
  target_type = c("time-series", "oracle-output")
)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check that there are no duplicate rows in a target dataset. Function designed to be used as part of overall target data integrity check.

Usage

check_target_dataset_rows_unique(
  target_type = c("time-series", "oracle-output"),
  na = c("NA", ""),
  date_col = NULL,
  output_type_id_datatype = c("from_config", "auto", "character", "double", "integer",
    "logical", "Date"),
  hub_path
)

Arguments

Details

If datasets are versioned, multiple observations are allowed in time-series target data, so long as they have different as_of values. The as_of column is therefore included when determining duplicates. In oracle-output data, there should be only a single observation, regardless of the as_of value so the column it is not be included when determining duplicates.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check that a single unique target dataset exists for a given target type.

Usage

check_target_dataset_unique(
  hub_path,
  target_type = c("time-series", "oracle-output")
)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Note that files which are part of a hive partitioned dataset must have parquet file extension only.

Usage

check_target_file_ext_valid(file_path)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check that a hive-partitioned target data file name can be correctly parsed.

Usage

check_target_file_name(file_path)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check target file can be read successfully

Usage

check_target_file_read(file_path, hub_path = ".")

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check that a target data file has the correct column names according to target type

Usage

check_target_tbl_colnames(
  target_tbl,
  target_type = c("time-series", "oracle-output"),
  file_path,
  hub_path,
  config_target_data = NULL,
  date_col = NULL
)

Arguments

Details

Column name validation depends on whether a target-data.json configuration file is provided:

With target-data.json config: Expected columns are determined directly from the configuration. The target table must contain exactly the columns defined in the config.

Without target-data.json config (inference mode): Expected columns are inferred from the task ID configuration in tasks.json, allowed columns according to the target type, and expectations based on the detected output types in the target data. Additional optional columns (e.g., as_of) are allowed for time-series data.

Note on date columns: Target data always contains a date column (e.g., target_end_date) representing when observations occurred. However, in horizon-based forecast hubs, task IDs may only define origin_date and horizon (with target dates calculated from these). In such cases, provide date_col to enable deterministic validation of the date column when it is not a valid task ID. Validation of date column existence and type is performed by check_target_tbl_coltypes().

Inference mode validation for time-series data is limited. For robust validation, create a target-data.json config file. See target-data.json schema # nolint: line_length_linter. for more information on the json schema scpecifics.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check that a target data file has the correct column types according to target type

Usage

check_target_tbl_coltypes(
  target_tbl,
  target_type = c("time-series", "oracle-output"),
  date_col = NULL,
  na = c("NA", ""),
  output_type_id_datatype = c("from_config", "auto", "character", "double", "integer",
    "logical", "Date"),
  file_path,
  hub_path
)

Arguments

Details

Column type validation depends on whether a target-data.json configuration file is provided:

With target-data.json config: Expected column types are determined directly from the schema defined in the configuration. Validation is performed against the schema specifications in target-data.json.

Without target-data.json config: Expected column types are determined from the dataset itself and validated for internal consistency across files, which mainly applies to partitioned datasets.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

This check is only performed when the target data file contains an output_type_id column and cdf or pmf output types. It verifies that distributional output type (cdf and pmf) oracle values meet the following criteria:

• oracle_value values are either 0 or 1.

• pmf oracle values sum to 1 for each observation unit.

• cdf oracle values are non-decreasing for each observation unit when sorted by the output_type_id set defined in the hub config.

Usage

check_target_tbl_oracle_value(
  target_tbl,
  target_type = c("oracle-output", "time-series"),
  file_path,
  hub_path,
  config_target_data = NULL
)

Arguments

Details

When validating oracle values, data is grouped by observation unit to check PMF sums and CDF monotonicity within each unit.

With target-data.json config: Observable unit is determined from the config's observable_unit specification.

Without target-data.json config: Observable unit is inferred from task ID columns present in the data.

The as_of column is NOT included in the grouping. Oracle data is designed to contain a single version per observable unit with a one-to-one mapping to model output data.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

This check is only performed when the target data file contains an output_type_id column. It verifies that non-distributional output types have all NA output type IDs, and that distributional output types (cdf, pmf) include the complete output_type_id set defined in the hub config.

Usage

check_target_tbl_output_type_ids(
  target_tbl_chr,
  target_type = c("oracle-output", "time-series"),
  file_path,
  hub_path,
  config_target_data = NULL
)

Arguments

Details

When checking for completeness of distributional output types, data is grouped by observation unit to verify each unit has the complete set of output_type_id values.

With target-data.json config: Observable unit is determined from the config's observable_unit specification.

Without target-data.json config: Observable unit is inferred from task ID columns present in the data.

The as_of column is NOT included in the grouping. Oracle data is designed to contain a single version per observable unit with a one-to-one mapping to model output data.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check that there are no duplicate rows in target data files being validated.

Usage

check_target_tbl_rows_unique(
  target_tbl,
  target_type = c("time-series", "oracle-output"),
  file_path,
  hub_path,
  config_target_data = NULL
)

Arguments

Details

Row uniqueness is determined by checking for duplicate combinations of key columns (excluding value columns).

With target-data.json config: Columns to check are determined from the config's observable_unit specification. For oracle-output data with output type IDs, the output_type and output_type_id columns are also included in the uniqueness check.

Without target-data.json config: For time-series data, if versioned, multiple observations are allowed so long as they have different as_of values. The as_of column is therefore included when determining duplicates. For oracle-output data, there should be only a single observation, regardless of the as_of value, so the column is not included when determining duplicates.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check is only performed when the target data type is time-series. When the target task ID is not specified in the config (i.e. hub has single target and target_keys = NULL), the validity of the target is only checked through the config file. Otherwise, the values in the target task ID column of target_tbl are checked. Note that valid time-series targets must be step ahead and their target type must be one of "continuous", "discrete", "binary" or "compositional". If the hub contains no valid time-series targets, no time-series target data should be present and validation of such data will be skipped.

Usage

check_target_tbl_ts_targets(
  target_tbl,
  target_type = c("time-series", "oracle-output"),
  file_path,
  hub_path
)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check is only performed when the target data file contains columns that map onto task IDs or output types defined in the hub configuration.

Usage

check_target_tbl_values(
  target_tbl_chr,
  target_type = c("time-series", "oracle-output"),
  file_path,
  hub_path,
  date_col = NULL,
  allow_extra_dates = FALSE,
  config_target_data = NULL
)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check that model output data column datatypes conform to those define in the hub config.

Usage

check_tbl_col_types(
  tbl,
  file_path,
  hub_path,
  output_type_id_datatype = c("from_config", "auto", "character", "double", "integer",
    "logical", "Date")
)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Checks that a tibble/data.frame of data read in from the file being validated contains the expected task ID and standard column names according the round configuration being validated against.

Usage

check_tbl_colnames(tbl, round_id, file_path, hub_path = ".")

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

This check is used to validate that values in any derived task ID columns matches accepted values for each derived task ID in the config. Given the dependence of derived task IDs on the values of other values, it ignores the combinations of derived task ID values with those of other task IDs and focuses only on identifying values that do not match the accepted values.

Usage

check_tbl_derived_task_id_vals(
  tbl,
  round_id,
  file_path,
  hub_path,
  derived_task_ids = get_hub_derived_task_ids(hub_path, round_id)
)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

If no derived_task_ids are specified, the check is skipped and a ⁠<message/check_info>⁠ condition class object is retuned.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check model output data tbl round ID matches submission round ID.

Usage

check_tbl_match_round_id(tbl, file_path, hub_path, round_id_col = NULL)

Arguments

Details

This check only applies to files being submitted to rounds where round_id_from_variable: true or where a round_id_col name is explicitly provided. Skipped otherwise.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

If round_id_from_variable: false and no round_id_col name is provided, check is skipped and a ⁠<message/check_info>⁠ condition class object is returned. If no valid round_id_col name is provided or can extracted from config (check through check_valid_round_id_col), a ⁠<message/check_error>⁠ condition class object is returned and the rest of the check skipped.

Description

Checks that combinations of task ID, output type and output type ID value combinations are unique, by checking that there are no duplicate rows across all tbl columns excluding the value column.

Usage

check_tbl_rows_unique(tbl, file_path, hub_path)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

This check detects the compound task ID sets of samples, implied by the output_type_id and task ID values, and checks them for internal consistency and compliance with the compound_taskid_set defined for each round modeling task in the tasks.json config.

Usage

check_tbl_spl_compound_taskid_set(
  tbl,
  round_id,
  file_path,
  hub_path,
  derived_task_ids = get_hub_derived_task_ids(hub_path)
)

Arguments

Details

If the check fails, the output of the check includes an errors element, a list of items, one for each modeling task failing validation. The structure depends on the reason the check failed.

If the check failed because more that a single unique compound_taskid_set was found for a given model task, the errors object will be a list with one element for each compound_taskid_set detected and will have the following structure:

• tbl_comp_tids: a compound task id set detected in the the tbl.

• output_type_ids: The output type ID of the sample that does not contain a single, unique value for each compound task ID.

If the check failed because task IDs which is not allowed in the config, were identified as compound task ID (i.e. samples describe "finer" compound modeling tasks) for a given model task, the errors object will be a list with the structure described above as well as the additional following elements:

• config_comp_tids: the allowed compound_taskid_set defined in the modeling task config.

• invalid_tbl_comp_tids: the names of invalid compound task IDs.

The name of each element is the index identifying the config modeling task the sample is associated with mt_id. See hubverse documentation on samples for more details.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check model output data tbl samples contain single unique values for each compound task ID within individual samples

Usage

check_tbl_spl_compound_tid(
  tbl,
  round_id,
  file_path,
  hub_path,
  compound_taskid_set = NULL,
  derived_task_ids = get_hub_derived_task_ids(hub_path, round_id)
)

Arguments

Details

Output of the check includes an errors element, a list of items, one for each sample failing validation, with the following structure:

• mt_id: Index identifying the config modeling task the sample is associated with.

• output_type_id: The output type ID of the sample that does not contain a single, unique value for each compound task ID.

• values: The unique values of each compound task ID. See hubverse documentation on samples for more details.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check that individual sample output_type_ids do not span multiple model tasks

Usage

check_tbl_spl_mt_unique(
  tbl,
  round_id,
  file_path,
  hub_path,
  derived_task_ids = get_hub_derived_task_ids(hub_path, round_id)
)

Arguments

Details

Different model tasks can have different sample configurations (compound_taskid_set, min/max_samples_per_task, etc.), so samples should be entirely independent across model tasks. This check verifies that no sample output_type_id appears in more than one model task.

Output of the check includes an errors element, a list with the following structure:

• mt_ids: Integer vector of model task indices the overlapping samples span.

• output_type_ids: Character vector of sample output_type_ids that appear in multiple model tasks.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check model output data tbl samples contain the appropriate number of samples for a given compound idx.

Usage

check_tbl_spl_n(
  tbl,
  round_id,
  file_path,
  hub_path,
  compound_taskid_set = NULL,
  derived_task_ids = get_hub_derived_task_ids(hub_path, round_id)
)

Arguments

Details

Output of the check includes an errors element, a list of items, one for each compound_idx failing validation, with the following structure:

• compound_idx: the compound idx that failed validation of number of samples.

• n: the number of samples counted for the compound idx.

• min_samples_per_task: the minimum number of samples required for the compound idx.

• max_samples_per_task: the maximum number of samples required for the compound idx.

• compound_idx_tbl: a tibble of the expected structure for samples belonging to the compound idx. See hubverse documentation on samples for more details.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check model output data tbl samples contain single unique combination of non-compound task ID values across all samples

Usage

check_tbl_spl_non_compound_tid(
  tbl,
  round_id,
  file_path,
  hub_path,
  compound_taskid_set = NULL,
  derived_task_ids = get_hub_derived_task_ids(hub_path, round_id)
)

Arguments

Details

Output of the check includes an errors element, a list of items, one for each modeling task containing samples failing validation, with the following structure:

• mt_id: Index identifying the config modeling task the samples are associated with.

• output_type_ids: The output type IDs of samples that do not match the most frequent non-compound task ID value combination across all samples in the modeling task.

• frequent: The most frequent non-compound task ID value combination across all samples in the modeling task to which all samples were compared. See hubverse documentation on samples for more details.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check model output data tbl contains a single unique round ID.

Usage

check_tbl_unique_round_id(tbl, file_path, hub_path, round_id_col = NULL)

Arguments

Details

This check only applies to files being submitted to rounds where round_id_from_variable: true or where a round_id_col name is explicitly provided. Skipped otherwise.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

If round_id_from_variable: false and no round_id_col name is provided, check is skipped and a ⁠<message/check_info>⁠ condition class object is returned. If no valid round_id_col name is provided or can extracted from config (check through check_valid_round_id_col), a ⁠<message/check_error>⁠ condition class object is returned and the rest of the check skipped.

Description

Checks that values in the value column of a tibble/data.frame of data read in from the file being validated conform to the configuration for each output type of the appropriate model task.

Usage

check_tbl_value_col(
  tbl,
  round_id,
  file_path,
  hub_path,
  derived_task_ids = get_hub_derived_task_ids(hub_path, round_id)
)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Checks that values in the value column for quantile and cdf output type data for each unique task ID/output type combination are non-descending when arranged by increasing output_type_id order. Check only performed if tbl contains quantile or cdf output type data. If not, the check is skipped and a ⁠<message/check_info>⁠ condition class object is returned.

Usage

check_tbl_value_col_ascending(
  tbl,
  file_path,
  hub_path,
  round_id,
  derived_task_ids = get_hub_derived_task_ids(hub_path)
)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Checks that values in the value column of pmf output type data for each unique task ID combination sum to 1. Check only performed if tbl contains pmf output type data. If not, the check is skipped and a ⁠<message/check_info>⁠ condition class object is returned.

Usage

check_tbl_value_col_sum1(tbl, file_path)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check model output data tbl contains valid value combinations

Usage

check_tbl_values(
  tbl,
  round_id,
  file_path,
  hub_path,
  derived_task_ids = get_hub_derived_task_ids(hub_path, round_id)
)

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check all required task ID/output type/output type ID value combinations present in model data.

Usage

check_tbl_values_required(
  tbl,
  round_id,
  file_path,
  hub_path,
  derived_task_ids = get_hub_derived_task_ids(hub_path)
)

Arguments

Details

Note that it is necessary for derived_task_ids to be specified if any of the task IDs with required values have dependent derived task IDs. If this is the case and derived task IDs are not specified, the dependent nature of derived task ID values will result in false validation errors when validating required values.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check whether the round_id determined for the submission is valid

Usage

check_valid_round_id(round_id, file_path, hub_path = ".")

Arguments

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_error>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check that any round_id_col name provided or extracted from the hub config is valid.

Usage

check_valid_round_id_col(tbl, file_path, hub_path, round_id_col = NULL)

Arguments

Details

This check only applies to files being submitted to rounds where round_id_from_variable: true or where a round_id_col name is explicitly provided. Skipped otherwise.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

If round_id_from_variable: false and no round_id_col name is provided, check is skipped and a ⁠<message/check_info>⁠ condition class object is returned. Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Combines multiple validation objects of the same class into one. Works with both single-file validation objects (hub_validations, target_validations) and multi-file collection objects (hub_validations_collection, target_validations_collection). For more details on these classes, see article on ⁠<hub_validations>⁠ S3 class objects.

Usage

combine(...)

## S3 method for class 'hub_validations_collection'
combine(...)

Arguments

Details

For hub_validations objects, all inputs must share the same where attribute (i.e., be validations for the same subject).

For hub_validations_collection objects, the individual hub_validations objects from all collections are extracted and grouped by their where attribute, combining validation results for the same subject.

Subclasses (e.g., target_validations, target_validations_collection) are preserved.

Value

An object of the same class as the inputs, or NULL if no valid inputs provided.

See Also

new_hub_validations(), new_hub_validations_collection(), new_target_validations(), new_target_validations_collection()

Description

Create a custom validation check function template file.

Usage

create_custom_check(
  name,
  hub_path = ".",
  r_dir = "src/validations/R",
  error = FALSE,
  conditional = FALSE,
  error_object = FALSE,
  config = FALSE,
  extra_args = FALSE,
  overwrite = FALSE
)

Arguments

Details

See the article on writing custom check functions for more.

Value

Invisible TRUE if the custom check function file is created successfully.

Examples

withr::with_tempdir({
  # Create the custom check file with default settings.
  create_custom_check("check_default")
  cat(readLines("src/validations/R/check_default.R"), sep = "\n")

  # Create fully featured custom check file.
  create_custom_check("check_full",
    error = TRUE, conditional = TRUE,
    error_object = TRUE, config = TRUE,
    extra_args = TRUE
  )
  cat(readLines("src/validations/R/check_full.R"), sep = "\n")
})

Description

Create expanded grid of valid task ID and output type value combinations

Usage

expand_model_out_grid(
  config_tasks,
  round_id,
  required_vals_only = FALSE,
  force_output_types = FALSE,
  all_character = FALSE,
  output_type_id_datatype = c("from_config", "auto", "character", "double", "integer",
    "logical", "Date"),
  as_arrow_table = FALSE,
  bind_model_tasks = TRUE,
  include_sample_ids = FALSE,
  compound_taskid_set = NULL,
  output_types = NULL,
  derived_task_ids = get_config_derived_task_ids(config_tasks, round_id)
)

Arguments

Details

When a round is set to round_id_from_variable: true, the value of the task ID from which round IDs are derived (i.e. the task ID specified in round_id property of config_tasks) is set to the value of the round_id argument in the returned output.

When sample output types are included in the output and include_sample_ids = TRUE, the output_type_id column contains example sample indexes which are useful for identifying the compound task ID structure of multivariate sampling distributions in particular, i.e. which combinations of task ID values represent individual samples.

Value

If bind_model_tasks = TRUE (default) a tibble or arrow table containing all possible task ID and related output type ID value combinations. If bind_model_tasks = FALSE, a list containing a tibble or arrow table for each round modeling task.

Columns are coerced to data types according to the hub schema, unless all_character = TRUE. If all_character = TRUE, all columns are returned as character which can be faster when large expanded grids are expected. If required_vals_only = TRUE, values are limited to the combinations of required values only.

Note that if required_vals_only = TRUE and an optional output type is requested through output_types, a zero row grid will be returned. If all output types are requested however (i.e. when output_types = NULL) and they are all optional, a grid of required task ID values only will be returned. However, whenever force_output_types = TRUE, all output types are treated as required.

Examples

hub_con <- hubData::connect_hub(
  system.file("testhubs/flusight", package = "hubUtils")
)
config_tasks <- attr(hub_con, "config_tasks")
expand_model_out_grid(config_tasks, round_id = "2023-01-02")
expand_model_out_grid(
  config_tasks,
  round_id = "2023-01-02",
  required_vals_only = TRUE
)
# Specifying a round in a hub with multiple round configurations.
hub_con <- hubData::connect_hub(
  system.file("testhubs/simple", package = "hubUtils")
)
config_tasks <- attr(hub_con, "config_tasks")
expand_model_out_grid(config_tasks, round_id = "2022-10-01")
# Later round_id maps to round config that includes additional task ID 'age_group'.
expand_model_out_grid(config_tasks, round_id = "2022-10-29")
# Coerce all columns to character
expand_model_out_grid(config_tasks,
  round_id = "2022-10-29",
  all_character = TRUE
)
# Return arrow table
expand_model_out_grid(config_tasks,
  round_id = "2022-10-29",
  all_character = TRUE,
  as_arrow_table = TRUE
)
# Hub with sample output type
config_tasks <- read_config_file(system.file("config", "tasks.json",
  package = "hubValidations"
))
expand_model_out_grid(config_tasks,
  round_id = "2022-12-26"
)
# Include sample IDS
expand_model_out_grid(config_tasks,
  round_id = "2022-12-26",
  include_sample_ids = TRUE
)
# Hub with sample output type and compound task ID structure
config_tasks <- read_config_file(
  system.file("config", "tasks-comp-tid.json", package = "hubValidations")
)
expand_model_out_grid(config_tasks,
  round_id = "2022-12-26",
  include_sample_ids = TRUE
)
# Override config compound task ID set
# Create coarser compound task ID set for the first modeling task which contains
# samples
expand_model_out_grid(config_tasks,
  round_id = "2022-12-26",
  include_sample_ids = TRUE,
  compound_taskid_set = list(
    c("forecast_date", "target"),
    NULL
  )
)
expand_model_out_grid(config_tasks,
  round_id = "2022-12-26",
  include_sample_ids = TRUE,
  compound_taskid_set = list(
    NULL,
    NULL
  )
)
# Subset output types
config_tasks <- read_config(
  system.file("testhubs", "samples", package = "hubValidations")
)
expand_model_out_grid(config_tasks,
  round_id = "2022-10-29",
  include_sample_ids = TRUE,
  bind_model_tasks = FALSE,
  output_types = c("sample", "pmf"),
)
expand_model_out_grid(config_tasks,
  round_id = "2022-10-29",
  include_sample_ids = TRUE,
  bind_model_tasks = TRUE,
  output_types = "sample",
)
# Ignore derived task IDs
expand_model_out_grid(config_tasks,
  round_id = "2022-10-29",
  include_sample_ids = TRUE,
  bind_model_tasks = FALSE,
  output_types = "sample",
  derived_task_ids = "target_end_date"
)
# Return only required values
hub_path <- system.file("testhubs", "v4", "simple", package = "hubUtils")
config_tasks <- read_config(hub_path)
# Return required output types and output_types_ids only
expand_model_out_grid(
  config_tasks = config_tasks,
  round_id = "2022-10-22",
  required_vals_only = TRUE
)
# Force all output types to be required
expand_model_out_grid(
  config_tasks = config_tasks,
  round_id = "2022-10-22",
  required_vals_only = TRUE,
  force_output_types = TRUE
)
# Sub-setting for an optional output type returns an empty data frame
expand_model_out_grid(
  config_tasks = config_tasks,
  round_id = "2022-10-22",
  output_types = "mean",
  required_vals_only = TRUE
)
# force_output_types on an optional output type forces all output_type_id values
# to be required
expand_model_out_grid(
  config_tasks = config_tasks,
  round_id = "2022-10-22",
  output_types = "mean",
  required_vals_only = TRUE,
  force_output_types = TRUE
)
# Ignore derived task IDs
hub_path <- system.file("testhubs", "v4", "flusight", package = "hubUtils")
config_tasks <- read_config(hub_path)
# Defaults to using derived_task_ids from config
expand_model_out_grid(config_tasks, round_id = "2023-05-08")
# Can be overridden by argument derived_task_ids
expand_model_out_grid(config_tasks,
  round_id = "2023-05-08",
  derived_task_ids = NULL
)

Description

Get hub configuration fields from a ⁠<config>⁠ class object

Usage

get_config_derived_task_ids(config_tasks, round_id = NULL)

Arguments

Value

• get_config_derived_task_ids: character vector of hub or round level derived task ID names. If round_id is NULL or the round does not have a round level derived_tasks_ids setting, returns the hub level derived_tasks_ids setting.

Functions

• get_config_derived_task_ids(): Get the hub or round level derived_tasks_ids

Examples

hub_path <- system.file("testhubs/v4/flusight", package = "hubUtils")
config_tasks <- read_config(hub_path)
get_config_derived_task_ids(config_tasks)
get_config_derived_task_ids(config_tasks, round_id = "2023-05-08")

Description

Retrieves the unique target task ID by extracting all target metadata and extracting the names of their target_keys. For valid config files these should be the same across all rounds and model tasks.

Usage

get_target_task_id(config_tasks)

Arguments

Value

A character vector of unique target task ID names. Post v5.0.0 this should be a single task ID name.

Description

Detect the compound_taskid_set for a tbl for each modeling task in a given round.

Usage

get_tbl_compound_taskid_set(
  tbl,
  config_tasks,
  round_id,
  compact = TRUE,
  error = TRUE,
  derived_task_ids = get_config_derived_task_ids(config_tasks, round_id)
)

Arguments

Value

A list of vectors of compound task IDs detected in the tbl, one for each modeling task in the round. If compact is TRUE, modeling tasks returning NULL elements will be removed.

Examples

hub_path <- system.file("testhubs/samples", package = "hubValidations")
file_path <- "flu-base/2022-10-22-flu-base.csv"
round_id <- "2022-10-22"
tbl <- read_model_out_file(
  file_path = file_path,
  hub_path = hub_path,
  coerce_types = "chr"
)
config_tasks <- read_config(hub_path, "tasks")
get_tbl_compound_taskid_set(tbl, config_tasks, round_id)
get_tbl_compound_taskid_set(tbl, config_tasks, round_id,
  compact = FALSE
)

Description

Get status of a hub check

Usage

is_success(x)

is_failure(x)

is_error(x)

is_info(x)

not_pass(x)

is_exec_error(x)

is_exec_warn(x)

is_any_error(x)

Arguments

Value

Logical. Is given status of check TRUE?

Functions

• is_success(): Is check success?

• is_failure(): Is check failure?

• is_error(): Is check error?

• is_info(): Is check info?

• not_pass(): Did check not pass?

• is_exec_error(): Is exec error?

• is_exec_warn(): Is exec warning?

• is_any_error(): Is error or exec error?

Description

Split and match model output tbl data to their corresponding model tasks in config_tasks. Useful for performing model task specific checks on model output. For v3 samples, the output_type_id column is set to NA for sample outputs.

Usage

match_tbl_to_model_task(
  tbl,
  config_tasks,
  round_id,
  output_types = NULL,
  derived_task_ids = get_config_derived_task_ids(config_tasks, round_id),
  all_character = TRUE
)

Arguments

Value

A list containing a tbl_df of model output data matched to a model task with one element per round model task.

Examples

hub_path <- system.file("testhubs/samples", package = "hubValidations")
tbl <- read_model_out_file(
  file_path = "flu-base/2022-10-22-flu-base.csv",
  hub_path, coerce_types = "chr"
)
config_tasks <- read_config(hub_path, "tasks")
match_tbl_to_model_task(tbl, config_tasks, round_id = "2022-10-22")
match_tbl_to_model_task(tbl, config_tasks,
  round_id = "2022-10-22",
  output_types = "sample"
)

Description

A hub_validations object contains validation results for a single validation subject. Depending on context, this could be a file, a configuration directory (hub-config), or a target dataset type (time-series, oracle-output). All checks must have the same ⁠$where⁠ value, which is extracted and stored as the where attribute.

Usage

new_hub_validations(...)

as_hub_validations(x)

Arguments

Value

an S3 object of class ⁠<hub_validations>⁠ with a where attribute.

Functions

• new_hub_validations(): Create new ⁠<hub_validations>⁠ S3 class object

• as_hub_validations(): Convert list to ⁠<hub_validations>⁠ S3 class object

Examples

new_hub_validations()

hub_path <- system.file("testhubs/simple", package = "hubValidations")
file_path <- "team1-goodmodel/2022-10-08-team1-goodmodel.csv"
new_hub_validations(
  file_exists = check_file_exists(file_path, hub_path),
  file_name = check_file_name(file_path)
)
x <- list(
  file_exists = check_file_exists(file_path, hub_path),
  file_name = check_file_name(file_path)
)
as_hub_validations(x)

Description

A hub_validations_collection is a container for validation results from multiple validation subjects. It is a named list where each element is a hub_validations object. Names are automatically extracted from the where attribute of each hub_validations object. If multiple hub_validations objects have the same where value, they are merged using combine(). Empty hub_validations objects are ignored.

Usage

new_hub_validations_collection(...)

as_hub_validations_collection(x)

Arguments

Value

an S3 object of class ⁠<hub_validations_collection>⁠. Elements are named by their where attribute (e.g., collection[["path/to/file.csv"]]).

Functions

• new_hub_validations_collection(): Create new ⁠<hub_validations_collection>⁠ S3 class object

• as_hub_validations_collection(): Convert list to ⁠<hub_validations_collection>⁠ S3 class object

Examples

new_hub_validations_collection()

hub_path <- system.file("testhubs/simple", package = "hubValidations")

# Create validations for two different files
file_path_1 <- "team1-goodmodel/2022-10-08-team1-goodmodel.csv"
validations_1 <- new_hub_validations(
  file_exists = check_file_exists(file_path_1, hub_path),
  file_name = check_file_name(file_path_1)
)

file_path_2 <- "team1-goodmodel/2022-10-15-team1-goodmodel.csv"
validations_2 <- new_hub_validations(
  file_exists = check_file_exists(file_path_2, hub_path),
  file_name = check_file_name(file_path_2)
)

# Combine into a collection
collection <- new_hub_validations_collection(validations_1, validations_2)

# Print the collection
collection

# Get file paths (element names)
names(collection)

# Access validations for a specific file
collection[[file_path_1]]

# Access validations for a specific file and check
collection$`team1-goodmodel/2022-10-08-team1-goodmodel.csv`$file_exists

Description

A target_validations object contains validation results for a single validation subject. Depending on context, this could be a target data file, the hub configuration directory (hub-config), or a target dataset type (time-series, oracle-output). All checks must have the same ⁠$where⁠ value, which is extracted and stored as the where attribute.

Usage

new_target_validations(...)

as_target_validations(x)

Arguments

Value

an S3 object of class ⁠<target_validations>⁠ with a where attribute.

Functions

• new_target_validations(): Create new ⁠<target_validations>⁠ S3 class object

• as_target_validations(): Convert list to ⁠<target_validations>⁠ S3 class object

Examples

new_target_validations()

hub_path <- system.file("testhubs/v5/target_file", package = "hubUtils")
file_path <- "time-series.csv"
new_target_validations(
  target_file_name = check_target_file_name(file_path),
  target_file_ext_valid = check_target_file_ext_valid(file_path)
)
x <- list(
  target_file_name = check_target_file_name(file_path),
  target_file_ext_valid = check_target_file_ext_valid(file_path)
)
as_target_validations(x)
file_path <- "time-series/target=flu_hosp_rate/part-0.parquet"
new_target_validations(
  target_file_name = check_target_file_name(file_path),
  target_file_ext_valid = check_target_file_ext_valid(file_path)
)

Description

A target_validations_collection is a container for target validation results from multiple validation subjects. It is a named list where each element is a target_validations object. Names are automatically extracted from the where attribute of each target_validations object. If multiple target_validations objects have the same where value, they are merged using combine(). Empty target_validations objects are ignored.

Usage

new_target_validations_collection(...)

as_target_validations_collection(x)

Arguments

Value

an S3 object of class ⁠<target_validations_collection>⁠. Elements are named by their where attribute (e.g., collection[["path/to/file.csv"]]).

Functions

• new_target_validations_collection(): Create new ⁠<target_validations_collection>⁠ S3 class object

• as_target_validations_collection(): Convert list to ⁠<target_validations_collection>⁠ S3 class object

Examples

new_target_validations_collection()

# Create validations for two different files
file_path_1 <- "time-series.csv"
validations_1 <- new_target_validations(
  target_file_name = check_target_file_name(file_path_1),
  target_file_ext_valid = check_target_file_ext_valid(file_path_1)
)

file_path_2 <- "other-data.csv"
validations_2 <- new_target_validations(
  target_file_name = check_target_file_name(file_path_2),
  target_file_ext_valid = check_target_file_ext_valid(file_path_2)
)

# Combine into a collection
collection <- new_target_validations_collection(validations_1, validations_2)

# Print the collection
collection

# Get file paths (element names)
names(collection)

# Access validations for a specific file
collection[[file_path_1]]

# Access a specific check within a file's validations
collection[["time-series.csv"]]$target_file_name

Description

Check that submitting team does not exceed maximum number of allowed models per team

Usage

opt_check_metadata_team_max_model_n(file_path, hub_path, n_max = 2L)

Arguments

Details

Should be deployed as part of validate_model_metadata optional checks.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check time difference between values in two date columns equal a defined period.

Usage

opt_check_tbl_col_timediff(
  tbl,
  file_path,
  hub_path,
  t0_colname,
  t1_colname,
  timediff = lubridate::weeks(2),
  output_type_id_datatype = c("from_config", "auto", "character", "double", "integer",
    "logical", "Date")
)

Arguments

Details

Should be deployed as part of validate_model_data optional checks.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Check that predicted values per location are less than total location population.

Usage

opt_check_tbl_counts_lt_popn(
  tbl,
  file_path,
  hub_path,
  targets = NULL,
  popn_file_path = "auxiliary-data/locations.csv",
  popn_col = "population",
  location_col = "location"
)

Arguments

Details

Should only be applied to rows containing count predictions. Use argument targets to filter tbl data to appropriate count target rows.

Should be deployed as part of validate_model_data optional checks.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Examples

hub_path <- system.file("testhubs/flusight", package = "hubValidations")
file_path <- "hub-ensemble/2023-05-08-hub-ensemble.parquet"
tbl <- hubValidations::read_model_out_file(file_path, hub_path)
# Single target key list
targets <- list("target" = "wk ahead inc flu hosp")
opt_check_tbl_counts_lt_popn(tbl, file_path, hub_path, targets = targets)

Description

Check time difference between values in two date columns equals a defined time period defined by values in a horizon column

Usage

opt_check_tbl_horizon_timediff(
  tbl,
  file_path,
  hub_path,
  t0_colname,
  t1_colname,
  horizon_colname = "horizon",
  timediff = lubridate::weeks(),
  output_type_id_datatype = c("from_config", "auto", "character", "double", "integer",
    "logical", "Date")
)

Arguments

Details

Should be deployed as part of validate_model_data optional checks.

Value

Depending on whether validation has succeeded, one of:

• ⁠<message/check_success>⁠ condition class object.

• ⁠<error/check_failure>⁠ condition class object.

Returned object also inherits from subclass ⁠<hub_check>⁠.

Description

Parse model output file metadata from file name

Usage

parse_file_name(file_path, file_type = c("model_output", "model_metadata"))

Arguments

Details

File names are allowed to contain the following compression extension prefixes: .snappy, .gzip, .gz, .brotli, .zstd, .lz4, .lzo, .bz2. These extension prefixes are now extracted when parsing the file name and returned as compression_ext element if present.

Value

A list with the following elements:

• round_id: The round ID the model output is associated with (NA for model metadata files.)

• team_abbr: The team responsible for the model.

• model_abbr: The name of the model.

• model_id: The unique model ID derived from the concatenation of ⁠<team_abbr>-<model_abbr>⁠.

• ext: The file extension.

• compression_ext: optional. The compression extension if present.

Examples

parse_file_name("hub-baseline/2022-10-15-hub-baseline.csv")
parse_file_name("hub-baseline/2022-10-15-hub-baseline.gzip.parquet")

Description

Prints a formatted summary of validation results. Validation-level warnings (attached to the hub_validations object) are always displayed prominently in a box at the top. Check-level warnings (attached to individual checks) are only shown when show_check_warnings = TRUE.

Usage

## S3 method for class 'hub_validations'
print(x, show_check_warnings = FALSE, ...)

Arguments

Value

Returns x invisibly.

Examples

hub_path <- system.file("testhubs/simple", package = "hubValidations")
v <- validate_submission(
  hub_path,
  file_path = "team1-goodmodel/2022-10-08-team1-goodmodel.csv"
)

# Default print
print(v)

# Show check-level warnings (if any)
print(v, show_check_warnings = TRUE)

# Example with validation-level warning
v_with_warning <- v
attr(v_with_warning, "warnings") <- list(
  capture_validation_warning(
    msg = "Example validation-level warning message.",
    where = "example"
  )
)
print(v_with_warning)

Description

Prints a formatted summary of validation results from multiple files. Each file's validations are printed under a header showing the file path. Validation-level warnings (attached to the collection object) are displayed prominently in a box at the top.

Usage

## S3 method for class 'hub_validations_collection'
print(x, show_check_warnings = FALSE, ...)

Arguments

Value

Returns x invisibly.

Description

Read a model output file

Usage

read_model_out_file(
  file_path,
  hub_path = ".",
  coerce_types = c("hub", "chr", "none"),
  output_type_id_datatype = c("from_config", "auto", "character", "double", "integer",
    "logical", "Date")
)

Arguments

Value

a tibble of contents of the model output file.

Description

Read a single target data file

Usage

read_target_file(
  target_file_path,
  hub_path,
  coerce_types = c("target", "chr", "none"),
  date_col = NULL,
  na = c("NA", "")
)

Arguments

Value

a tibble of contents of the target data file.

Examples

# download example hub
hub_path <- system.file("testhubs/v5/target_file",
  package = "hubUtils"
)
# read in time-series file
read_target_file("time-series.csv", hub_path)
read_target_file("time-series.csv", hub_path, coerce_types = "chr")
# read in oracle-output file
read_target_file("oracle-output.csv", hub_path)
read_target_file("oracle-output.csv", hub_path, coerce_types = "chr")

Description

Create a model output submission file template

Usage

submission_tmpl(
  path,
  round_id,
  required_vals_only = FALSE,
  force_output_types = FALSE,
  complete_cases_only = TRUE,
  compound_taskid_set = NULL,
  output_types = NULL,
  derived_task_ids = NULL,
  hub_con = deprecated(),
  config_tasks = deprecated()
)

Arguments

Details

For task IDs where all values are optional, by default, columns are created as columns of NAs when required_vals_only = TRUE. When such columns exist, the function returns a tibble with zero rows, as no complete cases of required value combinations exists. (Note that determination of complete cases does excludes valid NA output_type_id values in "mean" and "median" output types). To return a template of incomplete required cases, which includes NA columns, use complete_cases_only = FALSE.

To include output types that are optional in the submission template when required_vals_only = TRUE and complete_cases_only = FALSE, use force_output_types = TRUE. Use this in combination with sub-setting for output types you plan to submit via argument output_types to create a submission template customised to your submission plans. Tip: to ensure you create a template with all required output types, it's a good idea to first run the functions without subsetting or forcing output types and examing the unique values in output_type to check which output types are required.

When sample output types are included in the output, the output_type_id column contains example sample indexes which are useful for identifying the compound task ID structure of multivariate sampling distributions in particular, i.e. which combinations of task ID values represent individual samples.

When a round is set to round_id_from_variable: true, the value of the task ID from which round IDs are derived (i.e. the task ID specified in round_id property of config_tasks) is set to the value of the round_id argument in the returned output.

Value

a tibble template containing an expanded grid of valid task ID and output type ID value combinations for a given submission round and output type. If required_vals_only = TRUE, values are limited to the combination of required values only.

Examples

hub_path <- system.file("testhubs/flusight", package = "hubUtils")
submission_tmpl(hub_path, round_id = "2023-01-02")
# Return required values only
submission_tmpl(
  hub_path,
  round_id = "2023-01-02",
  required_vals_only = TRUE
)
submission_tmpl(
  hub_path,
  round_id = "2023-01-02",
  required_vals_only = TRUE,
  complete_cases_only = FALSE
)
# Specify a round in a hub with multiple rounds
hub_path <- system.file("testhubs/simple", package = "hubUtils")
submission_tmpl(hub_path, round_id = "2022-10-01")
submission_tmpl(hub_path, round_id = "2022-10-29")
# Subset for a specific output type
hub_path <- system.file("testhubs", "samples", package = "hubValidations")
submission_tmpl(
  hub_path,
  round_id = "2022-12-17",
  output_types = "sample"
)
# Create a template from the path to a tasks config file
config_path <- system.file("config", "tasks.json",
  package = "hubValidations"
)
submission_tmpl(
  config_path,
  round_id = "2022-12-26"
)
# Hub with sample output type and compound task ID structure
config_path <- system.file("config", "tasks-comp-tid.json",
  package = "hubValidations"
)
submission_tmpl(
  config_path,
  round_id = "2022-12-26",
  output_types = "sample"
)
# Override config compound task ID set
# Create coarser compound task ID set for the first modeling task which contains
# samples
submission_tmpl(
  config_path,
  round_id = "2022-12-26",
  output_types = "sample",
  compound_taskid_set = list(
    c("forecast_date", "target"),
    NULL
  )
)
# Derive a template with ignored derived task ID. Useful to avoid creating
# a template with invalid derived task ID value combinations.
hub_path <- system.file("testhubs", "flusight", package = "hubValidations")
submission_tmpl(
  hub_path,
  round_id = "2022-12-12",
  output_types = "pmf",
  derived_task_ids = "target_end_date",
  complete_cases_only = FALSE
)
# Force optional output type, in this case "mean".
submission_tmpl(
  hub_path,
  round_id = "2022-12-12",
  required_vals_only = TRUE,
  output_types = c("pmf", "quantile", "mean"),
  force_output_types = TRUE,
  derived_task_ids = "target_end_date",
  complete_cases_only = FALSE
)
# Create a template from a URL to fully configured hub repository on GitHub
submission_tmpl(
  path = "https://github.com/hubverse-org/example-simple-forecast-hub",
  round_id = "2022-11-28",
  output_types = "quantile"
)
# Create a template from a URL to the raw contents of a tasks.json file on
# GitHub
config_raw_url <- paste0(
  "https://raw.githubusercontent.com/hubverse-org/",
  "example-simple-forecast-hub/refs/heads/main/hub-config/tasks.json"
)
submission_tmpl(
  path = config_raw_url,
  round_id = "2022-11-28",
  output_types = "quantile"
)

# Create submission file using config file from AWS S3 bucket hub
# Use `s3_bucket()` to create a path to the hub's root directory
s3_hub_path <- arrow::s3_bucket("hubverse/hubutils/testhubs/simple/")
submission_tmpl(
  path = s3_hub_path,
  round_id = "2022-10-01",
  output_types = "quantile"
)
# Use `path()` method to create a path to the tasks.json file relative to the
# the S3 cloud hub's root directory
s3_config_path <- s3_hub_path$path("hub-config/tasks.json")
submission_tmpl(
  path = s3_config_path,
  round_id = "2022-10-01",
  output_types = "quantile"
)

Description

Wrap check expression in try to capture check execution errors

Usage

try_check(expr, file_path)

Arguments

Value

If expr executes correctly, the output of expr is returned. If execution fails, and object of class ⁠<error/check_exec_error>⁠ is returned. The execution error message is attached as attribute msg.

Description

Validate the contents of a submitted model data file

Usage

validate_model_data(
  hub_path,
  file_path,
  round_id_col = NULL,
  output_type_id_datatype = c("from_config", "auto", "character", "double", "integer",
    "logical", "Date"),
  validations_cfg_path = NULL,
  derived_task_ids = NULL
)

Arguments

Details

Note that it is necessary for derived_task_ids to be specified if any task IDs with required values have dependent derived task IDs. If this is the case and derived task IDs are not specified, the dependent nature of derived task ID values will result in false validation errors when validating required values.

Details of checks performed by validate_model_data()

Value

An object of class hub_validations. Each named element contains a hub_check class object reflecting the result of a given check. Function will return early if a check returns an error. The where attribute is set to file_path.

For more details on the structure of ⁠<hub_validations>⁠ objects, including how to access more information on individual checks, see article on ⁠<hub_validations>⁠ S3 class objects.

Examples

hub_path <- system.file("testhubs/simple", package = "hubValidations")
file_path <- "team1-goodmodel/2022-10-08-team1-goodmodel.csv"
validate_model_data(hub_path, file_path)

Description

Valid file level properties of a submitted model output file.

Usage

validate_model_file(hub_path, file_path, validations_cfg_path = NULL)

Arguments

Details

Details of checks performed by validate_model_file()

Value

An object of class hub_validations. Each named element contains a hub_check class object reflecting the result of a given check. Function will return early if a check returns an error. The where attribute is set to file_path.

For more details on the structure of ⁠<hub_validations>⁠ objects, including how to access more information on individual checks, see article on ⁠<hub_validations>⁠ S3 class objects.

Examples

hub_path <- system.file("testhubs/simple", package = "hubValidations")
validate_model_file(hub_path,
  file_path = "team1-goodmodel/2022-10-08-team1-goodmodel.csv"
)
validate_model_file(hub_path,
  file_path = "team1-goodmodel/2022-10-15-team1-goodmodel.csv"
)

Description

Valid properties of a metadata file.

Usage

validate_model_metadata(
  hub_path,
  file_path,
  round_id = "default",
  validations_cfg_path = NULL
)

Arguments

Details

Details of checks performed by validate_model_metadata()

Value

An object of class hub_validations. Each named element contains a hub_check class object reflecting the result of a given check. Function will return early if a check returns an error. The where attribute is set to file_path.

For more details on the structure of ⁠<hub_validations>⁠ objects, including how to access more information on individual checks, see article on ⁠<hub_validations>⁠ S3 class objects.

Examples

hub_path <- system.file("testhubs/simple", package = "hubValidations")
validate_model_metadata(hub_path,
  file_path = "hub-baseline.yml"
)
validate_model_metadata(hub_path,
  file_path = "team1-goodmodel.yaml"
)

Description

Validates model output and model metadata files in a Pull Request.

Usage

validate_pr(
  hub_path = ".",
  gh_repo,
  pr_number,
  round_id_col = NULL,
  output_type_id_datatype = c("from_config", "auto", "character", "double", "integer",
    "logical", "Date"),
  validations_cfg_path = NULL,
  skip_submit_window_check = FALSE,
  file_modification_check = c("error", "failure", "warn", "message", "none"),
  allow_submit_window_mods = TRUE,
  submit_window_ref_date_from = c("file", "file_path"),
  derived_task_ids = NULL
)

Arguments

Details

Only model output and model metadata files are individually validated using validate_submission() or validate_model_metadata() respectively although as part of checks, hub config files are also validated. Any other files included in the PR are ignored but flagged in a message.

By default, modifications (which include renaming) and deletions of previously submitted model output files and deletions or renaming of previously submitted model metadata files are not allowed and return a ⁠<error/check_error>⁠ condition class object for each applicable modified/deleted file. This behaviour can be modified through arguments file_modification_check, which controls whether modification/deletion checks are performed and what is returned if modifications/deletions are detected, and allow_submit_window_mods, which controls whether modifications/deletions of model output files are allowed within their submission windows.

When modification/deletion checks are enabled, each affected file creates an entry in the returned collection named by the file's path. The check within each entry is named valid_file_status (reflecting that we validate the file's git status). For example, to access the check for a deleted file: collection[["team1-goodmodel/2022-10-15-team1-goodmodel.csv"]][["valid_file_status"]].

Note that to establish relative submission windows when performing modification/deletion checks and allow_submit_window_mods is TRUE, the reference date is taken as the round_id extracted from the file path (i.e. submit_window_ref_date_from is always set to "file_path"). This is because we cannot extract dates from columns of deleted files. If hub submission window reference dates do not match round IDs in file paths, currently allow_submit_window_mods will not work correctly and is best set to FALSE. This only relates to hubs/rounds where submission windows are determined relative to a reference date and not when explicit submission window start and end dates are provided in the config.

Finally, note that it is necessary for derived_task_ids to be specified if any task IDs with required values have dependent derived task IDs. If this is the case and derived task IDs are not specified, the dependent nature of derived task ID values will result in false validation errors when validating required values.

Checks on model output files

Details of checks performed by validate_submission()

Checks on model metadata files

Details of checks performed by validate_model_metadata()

Value

An object of class hub_validations_collection, a collection of validation results. The collection includes entries for hub config validation ("hub-config") and file-specific validations (named by file path).

Examples

## Not run: 
validate_pr(
  hub_path = ".",
  gh_repo = "hubverse-org/ci-testhub-simple",
  pr_number = 3
)

## End(Not run)

Description

Checks both file level properties like file name, extension, location etc as well as model output data, i.e. the contents of the file.

Usage

validate_submission(
  hub_path,
  file_path,
  round_id_col = NULL,
  validations_cfg_path = NULL,
  output_type_id_datatype = c("from_config", "auto", "character", "double", "integer",
    "logical", "Date"),
  skip_submit_window_check = FALSE,
  skip_check_config = FALSE,
  submit_window_ref_date_from = c("file", "file_path"),
  derived_task_ids = NULL
)

Arguments

Details

Note that it is necessary for derived_task_ids to be specified if any task IDs with required values have dependent derived task IDs. If this is the case and derived task IDs are not specified, the dependent nature of derived task ID values will result in false validation errors when validating required values.

Details of checks performed by validate_submission()