diff --git a/docs/advanced_guidance/package_documentation/entity_hierarchy.md b/docs/advanced_guidance/package_documentation/entity_hierarchy.md new file mode 100644 index 0000000..4080dcb --- /dev/null +++ b/docs/advanced_guidance/package_documentation/entity_hierarchy.md @@ -0,0 +1,5 @@ +::: dve.core_engine.configuration.v1.hierarchy + handler: python + options: + show_root_heading: true + heading_level: 2 diff --git a/docs/user_guidance/entity_relationships.md b/docs/user_guidance/entity_relationships.md new file mode 100644 index 0000000..5029aa7 --- /dev/null +++ b/docs/user_guidance/entity_relationships.md @@ -0,0 +1,62 @@ +--- +title: Entity Relationships +tags: + - Linkage + - Relationships + - Missing + - Parent + - Group + - Rejections +--- + +Sometimes a user may choose to use the file transformation stage to `normalise` a heavily nested dataset into separate entities during the initial reading of data. This would be done by specifying different entities in the dataset section of the contract configuration in the `dischema` file. This allows for easier interaction when customising errors in the data contract or writing transformations in the business rules. + +`Normalising` assets can lead to more complex validations being required. For example in the flights dataset: + +```mermaid +erDiagram + COUNTRY ||--|{ AIRPORT : "" + AIRPORT ||--o{ FLIGHT : "" + FLIGHT ||--o{ PASSENGER : "" + AIRPORT ||--|{ STAFF_MEMBER : "" +``` + +### Missing Parent Records + +It could be that an airport record is deemed invalid and removed. Due to this, any flight records that linked to the now removed airport record are themselves invalid - a situation we refer to as a `missing_parent` issue, but are now existing in an entirely different entity. + +### No Valid Mandatory Records + +It could also be the case that staff records are a mandatory field for airport records. If all staff records for a particular airport record are removed during validation, this itself would invalidate the airport record - a situation we refer to as `no_valid_records` issue - but again the invalid airport record is in a different entity. + +### Dischema + +In order to perform these validations, how to link normalised entities needs to be provided. This can be specified in the `entity_relationships` section of the `dischema`. + +## Entity Relationships Content + +To allow the DVE to link between normalised assets, the following information should be provided (per linkable entity): + +- parent_entity: the immediate parent of the entity +- join_fields: how to join the entity with its parent in dictionary form (parent_field_name: child_field_name) +- mandatory: whether the child entity is a mandatory field in the immediate parent + +There is also the functionality to customise errors related to either missing parent or group rejections: + +- missing_parent_id_error_code: the error code to display if a record is rejected as it has no valid parent record +- missing_parent_id_error_message: the error message to display if a record is rejected as it hs no valid parent record +- no_valid_records_error_code: the error code to display if parent records are removed due to no valid children in a mandatory field +- no_valid_records_error_message: the error message to display if parent records are removed due to no valid children in a mandatory field + +!!! note + For root entities, you don't need to specify entity relationships - this will be inferred based on their absence. + But you may wish to so that error codes and messages can be customised. Ensure that for root entities the parent_entity + abd join_fields values are left blank. + + +## Entity Hierarchy Object + + + +The details provided in the entity_relationships section of the dischema are used to create an EntityHierarchy object. +Please refer to [Advanced User Guidance: Entity Hierarchy](../advanced_guidance/package_documentation/entity_hierarchy.md). diff --git a/docs/user_guidance/getting_started.md b/docs/user_guidance/getting_started.md index e938c2d..fbda5ae 100644 --- a/docs/user_guidance/getting_started.md +++ b/docs/user_guidance/getting_started.md @@ -68,6 +68,8 @@ Within the example above, there are two parent keys - `schemas` and `datasets`. !!! note The "splitting" of entities is considerably more useful in situtations where you want to normalise/de-normalise your data. If you're unfamiliar with this concept, you can read more about it [here](https://en.wikipedia.org/wiki/Database_normalization). However, you should keep in mind potential performance impacts of doing this. If you have rules that requires fields from different entities, you will have to perform a `join` between the split entities to be able to perform the rule. +To support with the application of more complex validation relating to parent and child records within normalised data, the [entity_relationships](entity_relationships.md) section of the `dischema` enables users to specify parent-child relationships and to customise error codes related to missing parent and group level validation issues. + For each dataset definition, you will need to provide a `reader_config` which describes how to load the data during the [File Transformation](file_transformation.md) stage. So, in the example above, we expect `movies` to come in as a `JSON` file. However, you can add more readers if you have the same data in different data formats (e.g. `csv`, `xml`, `json`). Regardless of what file format, the [File Transformation](file_transformation.md) stage will convert the submitted data into a "stringified" parquet format which is a requirement for the subsequent stages. To learn more about how you can construct your Data Contract please read [here](data_contract.md). diff --git a/zensical.toml b/zensical.toml index 064cb26..0141b6d 100644 --- a/zensical.toml +++ b/zensical.toml @@ -25,6 +25,7 @@ nav = [ {"File Transformation" = "user_guidance/file_transformation.md"}, {"Data Contract" = "user_guidance/data_contract.md"}, {"Business Rules" = "user_guidance/business_rules.md"}, + {"Entity Relationships" = "user_guidance/entity_relationships.md"} ]}, {"Backend Implementations" = [ {"DuckDB" = "user_guidance/implementations/duckdb.md"}, @@ -56,6 +57,9 @@ nav = [ {"Refdata" = [ {"Refdata Types" = "advanced_guidance/package_documentation/refence_data_types.md"}, {"Refdata Loaders" = "advanced_guidance/package_documentation/refdata_loaders.md"}, + ]}, + {"Entity Relationships" = [ + {"Entity Hierarchy" = "advanced_guidance/package_documentation/entity_hierarchy.md"} ]} ]}, {"Feedback" = [ @@ -193,6 +197,9 @@ options.custom_icons = ["overrides/.icons"] auto_append = ["includes/jargon_and_acronyms.md"] [project.markdown_extensions.pymdownx.superfences] +custom_fences = [ + { name = "mermaid", class = "mermaid", format = "pymdownx.superfences.fence_code_format" }, +] [project.markdown_extensions.pymdownx.tabbed] alternate_style = true