Validation utilities for transformation pipelines

[A-Alaa]

• Each transformation in a pipeline, which may contain +10 transformations, expects the input with a set of properties, and returns output with set of properties, potentially different after the transformation.
• Define a minimal set of properties e.g.:
  • indexing_property: [{"indexed", "not_indexed"}],
  • time_property: [{"relative_time", "absolute_time"}],
  • code_scheme_property: [{"raw_code", "intermediate_code", "final_code"}],
  • uom_property: [{"harmonized_uom", "raw_uom"}],
  • timestamp_type_property: {"timestamp_type", "maybe_timestamp_type"}]
  • etc.
• The validation ensures a sound sequence of transformations before running time-consuming data processing. It will allow early warning or early aborting the program if the user provides a buggy pipeline of transformation.
• The validation algorithm will retrieve the input and output property of each transformation and make sure that each transformation gets the expected properties.