From c0199015cc47f06ec5d24e053eac19e003a090e4 Mon Sep 17 00:00:00 2001 From: stevenhsd <56357022+stevenhsd@users.noreply.github.com> Date: Mon, 21 Sep 2026 08:08:44 +0100 Subject: [PATCH 1/3] docs: add entity relationships detail to docs --- .../package_documentation/entity_hierarchy.md | 5 +++ docs/user_guidance/entity_relationships.md | 32 +++++++++++++++++++ docs/user_guidance/getting_started.md | 2 ++ 3 files changed, 39 insertions(+) create mode 100644 docs/advanced_guidance/package_documentation/entity_hierarchy.md create mode 100644 docs/user_guidance/entity_relationships.md diff --git a/docs/advanced_guidance/package_documentation/entity_hierarchy.md b/docs/advanced_guidance/package_documentation/entity_hierarchy.md new file mode 100644 index 0000000..4080dcb --- /dev/null +++ b/docs/advanced_guidance/package_documentation/entity_hierarchy.md @@ -0,0 +1,5 @@ +::: dve.core_engine.configuration.v1.hierarchy + handler: python + options: + show_root_heading: true + heading_level: 2 diff --git a/docs/user_guidance/entity_relationships.md b/docs/user_guidance/entity_relationships.md new file mode 100644 index 0000000..a7e970a --- /dev/null +++ b/docs/user_guidance/entity_relationships.md @@ -0,0 +1,32 @@ +--- +title: Entity Relationships +tags: + - Linkage + - Relationships + - Missing + - Parent + - Group + - Rejections +--- + +Sometimes a user may choose to use the file transformation stage to `normalise` a heavily nested dataset into separate entities during the initial reading of data. This would be done by specifying different entities in the dataset section of the contract configuration in the `dischema` file. This allows for easier interaction when customising errors in the data contract or writing transformations in the business rules. However, if the dataset being processed requires more complex validation, for example removing orphaned records or implementing group rejections, then how to link normalised entities needs to be provided. This can be provided in the `entity_relationships` section of the `dischema` + +## Entity Relationships Content + +To allow the DVE to link between normalised assets, the following information should be provided (per linkable entity): + +- parent_entity: the immediate parent of the entity +- join_fields: how to join the entity with its parent in dictionary form (parent_field_name: child_field_name) +- mandatory: whether the child entity is a mandatory field in the immediate parent + +There is also the functionality to customise errors related to either missing parent or group rejections: + +- missing_parent_id_error_code: the error code to display if a record is rejected as it hs no valid parent record +- missing_parent_id_error_message: the error message to display if a record is rejected as it hs no valid parent record +- no_valid_records_error_code: the error code to display if parent records are removed due to no valid children in a mandatory field +- no_valid_records_error_message: the error message to display if parent records are removed due to no valid children in a mandatory field + +## Entity Hierarchy Object + +The details provided in the entity_relationships section of the dischema are used to create an EntityHierarchy object. +Please refer to [Advanced User Guidance: Entity Hierarchy](../advanced_guidance/package_documentation/entity_hierarchy.md). diff --git a/docs/user_guidance/getting_started.md b/docs/user_guidance/getting_started.md index e938c2d..fbda5ae 100644 --- a/docs/user_guidance/getting_started.md +++ b/docs/user_guidance/getting_started.md @@ -68,6 +68,8 @@ Within the example above, there are two parent keys - `schemas` and `datasets`. !!! note The "splitting" of entities is considerably more useful in situtations where you want to normalise/de-normalise your data. If you're unfamiliar with this concept, you can read more about it [here](https://en.wikipedia.org/wiki/Database_normalization). However, you should keep in mind potential performance impacts of doing this. If you have rules that requires fields from different entities, you will have to perform a `join` between the split entities to be able to perform the rule. +To support with the application of more complex validation relating to parent and child records within normalised data, the [entity_relationships](entity_relationships.md) section of the `dischema` enables users to specify parent-child relationships and to customise error codes related to missing parent and group level validation issues. + For each dataset definition, you will need to provide a `reader_config` which describes how to load the data during the [File Transformation](file_transformation.md) stage. So, in the example above, we expect `movies` to come in as a `JSON` file. However, you can add more readers if you have the same data in different data formats (e.g. `csv`, `xml`, `json`). Regardless of what file format, the [File Transformation](file_transformation.md) stage will convert the submitted data into a "stringified" parquet format which is a requirement for the subsequent stages. To learn more about how you can construct your Data Contract please read [here](data_contract.md). From 2baba2e12b67378ee3105c87770d64b8058947ec Mon Sep 17 00:00:00 2001 From: stevenhsd <56357022+stevenhsd@users.noreply.github.com> Date: Wed, 23 Sep 2026 12:19:12 +0100 Subject: [PATCH 2/3] docs: address review comments --- docs/user_guidance/entity_relationships.md | 34 ++++++++++++++++++++-- zensical.toml | 4 +++ 2 files changed, 36 insertions(+), 2 deletions(-) diff --git a/docs/user_guidance/entity_relationships.md b/docs/user_guidance/entity_relationships.md index a7e970a..a9d8eb8 100644 --- a/docs/user_guidance/entity_relationships.md +++ b/docs/user_guidance/entity_relationships.md @@ -9,7 +9,29 @@ tags: - Rejections --- -Sometimes a user may choose to use the file transformation stage to `normalise` a heavily nested dataset into separate entities during the initial reading of data. This would be done by specifying different entities in the dataset section of the contract configuration in the `dischema` file. This allows for easier interaction when customising errors in the data contract or writing transformations in the business rules. However, if the dataset being processed requires more complex validation, for example removing orphaned records or implementing group rejections, then how to link normalised entities needs to be provided. This can be provided in the `entity_relationships` section of the `dischema` +Sometimes a user may choose to use the file transformation stage to `normalise` a heavily nested dataset into separate entities during the initial reading of data. This would be done by specifying different entities in the dataset section of the contract configuration in the `dischema` file. This allows for easier interaction when customising errors in the data contract or writing transformations in the business rules. + +`Normalising` assets can lead to more complex validations being required. For example in the dataset: + +```mermaid +erDiagram + COUNTRY ||--o{ AIRPORT : "" + AIRPORT ||--o{ FLIGHT : "" + FLIGHT ||--o{ PASSENGER : "" + AIRPORT ||--o{ STAFF_MEMBER : "" +``` + +### Missing Parent Records + +It could be that an airport record is deemed invalid and removed. Due to this, any flight records that linked to the now removed airport record are themselves invalid - a situation we refer to as a `missing_parent` issue, but are now existing in an entirely different entity. + +### No Valid Mandatory Records + +It could also be the case that staff records are a mandatory field for airport records. If all staff records for a particular airport record are removed during validation, this itself would invalidate the airport record - a situation we refer to as `no_valid_records` issue - but again the invalid airport record is in a different entity. + +### Dischema + +In order to perform these validations, how to link normalised entities needs to be provided. This can be specified in the `entity_relationships` section of the `dischema`. ## Entity Relationships Content @@ -21,12 +43,20 @@ To allow the DVE to link between normalised assets, the following information sh There is also the functionality to customise errors related to either missing parent or group rejections: -- missing_parent_id_error_code: the error code to display if a record is rejected as it hs no valid parent record +- missing_parent_id_error_code: the error code to display if a record is rejected as it has no valid parent record - missing_parent_id_error_message: the error message to display if a record is rejected as it hs no valid parent record - no_valid_records_error_code: the error code to display if parent records are removed due to no valid children in a mandatory field - no_valid_records_error_message: the error message to display if parent records are removed due to no valid children in a mandatory field +!!! note + For root entities, you don't need to specify entity relationships - this will be inferred based on their absence. + But you may wish to so that error codes and messages can be customised. Ensure that for root entities the parent_entity + abd join_fields values are left blank. + + ## Entity Hierarchy Object + + The details provided in the entity_relationships section of the dischema are used to create an EntityHierarchy object. Please refer to [Advanced User Guidance: Entity Hierarchy](../advanced_guidance/package_documentation/entity_hierarchy.md). diff --git a/zensical.toml b/zensical.toml index 064cb26..15c04c4 100644 --- a/zensical.toml +++ b/zensical.toml @@ -25,6 +25,7 @@ nav = [ {"File Transformation" = "user_guidance/file_transformation.md"}, {"Data Contract" = "user_guidance/data_contract.md"}, {"Business Rules" = "user_guidance/business_rules.md"}, + {"Entity Relationships" = "user_guidance/entity_relationships.md"} ]}, {"Backend Implementations" = [ {"DuckDB" = "user_guidance/implementations/duckdb.md"}, @@ -193,6 +194,9 @@ options.custom_icons = ["overrides/.icons"] auto_append = ["includes/jargon_and_acronyms.md"] [project.markdown_extensions.pymdownx.superfences] +custom_fences = [ + { name = "mermaid", class = "mermaid", format = "pymdownx.superfences.fence_code_format" }, +] [project.markdown_extensions.pymdownx.tabbed] alternate_style = true From 04be773dade5e7318a1c380f35c44af862dafd24 Mon Sep 17 00:00:00 2001 From: stevenhsd <56357022+stevenhsd@users.noreply.github.com> Date: Wed, 23 Sep 2026 20:29:45 +0100 Subject: [PATCH 3/3] docs: further tweaks following review --- docs/user_guidance/entity_relationships.md | 6 +++--- zensical.toml | 3 +++ 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/user_guidance/entity_relationships.md b/docs/user_guidance/entity_relationships.md index a9d8eb8..5029aa7 100644 --- a/docs/user_guidance/entity_relationships.md +++ b/docs/user_guidance/entity_relationships.md @@ -11,14 +11,14 @@ tags: Sometimes a user may choose to use the file transformation stage to `normalise` a heavily nested dataset into separate entities during the initial reading of data. This would be done by specifying different entities in the dataset section of the contract configuration in the `dischema` file. This allows for easier interaction when customising errors in the data contract or writing transformations in the business rules. -`Normalising` assets can lead to more complex validations being required. For example in the dataset: +`Normalising` assets can lead to more complex validations being required. For example in the flights dataset: ```mermaid erDiagram - COUNTRY ||--o{ AIRPORT : "" + COUNTRY ||--|{ AIRPORT : "" AIRPORT ||--o{ FLIGHT : "" FLIGHT ||--o{ PASSENGER : "" - AIRPORT ||--o{ STAFF_MEMBER : "" + AIRPORT ||--|{ STAFF_MEMBER : "" ``` ### Missing Parent Records diff --git a/zensical.toml b/zensical.toml index 15c04c4..0141b6d 100644 --- a/zensical.toml +++ b/zensical.toml @@ -57,6 +57,9 @@ nav = [ {"Refdata" = [ {"Refdata Types" = "advanced_guidance/package_documentation/refence_data_types.md"}, {"Refdata Loaders" = "advanced_guidance/package_documentation/refdata_loaders.md"}, + ]}, + {"Entity Relationships" = [ + {"Entity Hierarchy" = "advanced_guidance/package_documentation/entity_hierarchy.md"} ]} ]}, {"Feedback" = [