From 55c8a5cf5280d3cc6e999fb15d301e2360f7e326 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Wed, 30 Sep 2026 12:50:38 -0500 Subject: [PATCH 1/7] docs: state the tier axis, and reserve "ingest" for the tier that ingests Two of the tier names describe opposite axes, and each names the axis the other means: Manual names the writer but means the origin, Imported names the origin but means the writer. A reader reasoning from the names alone lands in the wrong tier, and the recognizable result is an Imported table with no make() and a permanent allow_direct_insert=True. - State the axis where the tiers are introduced: a tier answers what puts rows in this table, never who caused them to be written. - Add a grid crossing where rows come from with what writes them, showing why only four combinations exist, and say why Part is not a fifth tier. - Say plainly that Manual does not mean hand-filled and Imported does not describe the data's origin. - Add the common-mistake callout, with allow_direct_insert as its tell. - Fix the Manual row, which listed "imports" as a way its rows arrive while Imported is the tier that imports. Terminology: "ingestion script" described the Manual tier in nine places while "ingestion routine" described an Imported make() elsewhere, pointing one word at both tiers. An entry script writes to an entry-point table; "ingest" is left to the tier that fetches for itself. Also corrects two statements that taught the misreading (installation.md, 01-first-pipeline.ipynb), and replaces illustrative filler in versioning.md that read as a real deprecation notice for allow_direct_insert, which the reference documents as current. Notebook edits are markdown cells only; no code cells or outputs change. --- src/about/versioning.md | 2 +- src/explanation/relational-workflow-model.md | 49 ++++++++++++++++--- src/how-to/define-tables.md | 6 +-- src/how-to/insert-data.md | 2 +- src/how-to/installation.md | 2 +- src/reference/specs/table-declaration.md | 2 +- src/tutorials/advanced/sql-comparison.ipynb | 2 +- src/tutorials/basics/01-first-pipeline.ipynb | 2 +- src/tutorials/basics/02-schema-design.ipynb | 22 +++++---- .../calcium-imaging/calcium-imaging.ipynb | 2 +- src/tutorials/examples/blob-detection.ipynb | 2 +- 11 files changed, 66 insertions(+), 27 deletions(-) diff --git a/src/about/versioning.md b/src/about/versioning.md index 365618c2..6a49c115 100644 --- a/src/about/versioning.md +++ b/src/about/versioning.md @@ -58,7 +58,7 @@ Version admonitions are used for features introduced **after 2.0** (i.e., versio !!! version-deprecated "Deprecated in 2.1, removed in 3.0" - The `allow_direct_insert` parameter is deprecated. Use `dj.config['safemode']` instead. + The `legacy_option` parameter is deprecated. Use `replacement_option` instead. **Note:** Features deprecated at the 2.0 baseline (coming from pre-2.0) are documented in the [Migration Guide](../how-to/migrate-to-v20.md) rather than with admonitions, since this documentation assumes 2.0 as the baseline. diff --git a/src/explanation/relational-workflow-model.md b/src/explanation/relational-workflow-model.md index fac1d7f7..364a4403 100644 --- a/src/explanation/relational-workflow-model.md +++ b/src/explanation/relational-workflow-model.md @@ -145,19 +145,56 @@ work in practice. ### Workflow steps and table tiers -Tables are classified into tiers by data-entry mode: +A table's tier answers one question: **what puts rows in this table?** -| Tier | Role | `make()` | +| Tier | What puts rows in it | `make()` | |------|------|----------| -| **Manual** | Rows entered at runtime from outside the pipeline (people, forms, instruments, imports) | No | -| **Lookup** | Reference rows defined in the schema itself via `contents` | No | -| **Imported** | Reach out to data sources outside DataJoint (instruments, ELNs, external databases) | Yes | -| **Computed** | Derive their contents entirely from upstream DataJoint tables | Yes | +| **Lookup** | The code, through the table's own `contents` | No | +| **Manual** | A writer outside the table — a person, an instrument, an entry script | No | +| **Imported** | The table itself, through `make()`, reading an external source | Yes | +| **Computed** | The table itself, through `make()`, deriving from other DataJoint tables | Yes | + +The axis is *what writes the rows*, never *who caused them to be written*. Two +of the names suggest otherwise, so it is worth saying plainly: + +- **Manual** does not mean hand-entered. It means the rows arrive from outside + the table, by whatever means — a technician typing into a form, a LIMS feed, an + instrument, or a nightly entry script. An automated feed still writes to a + Manual table. +- **Imported** does not say where the data came from. It says what the table + does: its own `make()` reaches out and fetches. A table filled *by something + else* is Manual, however far away the data originated. + +Crossing where the rows come from with what writes them separates the two +questions, and shows why only four combinations exist: + +| Rows come from | Written from outside the table | Written by the table's own `make()` | +|---|---|---| +| The committed schema | **Lookup** | — | +| Outside the pipeline | **Manual** | **Imported** | +| Other DataJoint tables | — | **Computed** | + +`Part` is absent because it is not a fifth answer to the same question. A part +table is a structural role: it inherits its master's tier and is written in the +same transaction as its master. Any tier can serve as a master. Imported and Computed tables define computations via `make()` methods. The `make()` method specifies how each entity is derived — declared within the table definition, not in an external workflow file. +!!! warning "The mistake the names invite" + A designer building an automated feed reasons: *this table is not filled by + hand, so it is not Manual; the data comes from outside, so it is Imported* — + and declares `dj.Imported`. But an Imported table is populated by its own + `make()`, and this one has none, so the insert is refused. The fix that + presents itself is `allow_direct_insert=True`, which silences a guard + reporting a real modeling error. + + The result is recognizable: an auto-populated table with **no `make()`**, a + permanent `allow_direct_insert=True`, and `populate()` that silently does + nothing — with the pipeline's boundary drawn one table away from where it + actually sits. Such a table is a **Manual** table. + #### Manual vs. Lookup Manual and Lookup tables are both **entry points** — their rows are entered diff --git a/src/how-to/define-tables.md b/src/how-to/define-tables.md index 7fb5f850..a0833adc 100644 --- a/src/how-to/define-tables.md +++ b/src/how-to/define-tables.md @@ -30,7 +30,7 @@ class MyTable(dj.Manual): | Type | Base Class | Purpose | |------|------------|---------| -| Manual | `dj.Manual` | Data inserted directly from outside the pipeline (forms, instruments, ingestion scripts) | +| Manual | `dj.Manual` | Data inserted directly from outside the pipeline (forms, instruments, entry scripts) | | Lookup | `dj.Lookup` | Reference data defined in the schema via `contents` | | Imported | `dj.Imported` | Populated by `make()` from an external source | | Computed | `dj.Computed` | Populated by `make()` from other tables | @@ -266,8 +266,8 @@ data management?** process, not a runtime insert. - Use **`dj.Manual`** when the rows are **populated at runtime** and their quality is guaranteed by the **data-management process** — validation, curation, and - access control at ingest, not code review. Subjects, sessions, samples, or - anything typed into a form, ingested from a file, or read from an instrument. + access control at entry, not code review. Subjects, sessions, samples, or + anything typed into a form, loaded from a file, or read from an instrument. This resolves the case that "where a row comes from" leaves ambiguous: a controlled vocabulary that is **populated at runtime** — gene symbols loaded from an external diff --git a/src/how-to/insert-data.md b/src/how-to/insert-data.md index fa9cff1b..221c7274 100644 --- a/src/how-to/insert-data.md +++ b/src/how-to/insert-data.md @@ -59,7 +59,7 @@ Subject.insert(rows, ignore_extra_fields=True) The sections above target **Manual** tables — the tier whose rows are inserted directly, from outside the DataJoint pipeline (whether entered by hand through a -form or GUI, or loaded by an automated ingestion tool). Other tiers get their +form or GUI, or loaded by an automated entry script). Other tiers get their rows a different way, and inserting into them directly breaks reproducibility: - **Computed and Imported tables** — their rows are produced only by `make()` diff --git a/src/how-to/installation.md b/src/how-to/installation.md index b3a78aa5..8ba8d0ab 100644 --- a/src/how-to/installation.md +++ b/src/how-to/installation.md @@ -159,7 +159,7 @@ Session.insert1({'subject_id': 1, 'session_idx': 1, 'session_date': '2026-01-06' SessionAnalysis.populate() ``` -`Subject` and `Session` are entered by hand; `SessionAnalysis` derives from `Session` and fills +`Subject` and `Session` are written from outside the pipeline; `SessionAnalysis` derives from `Session` and fills itself when you call `populate()`. That dependency — declared with `->` — is the whole of the [Relational Workflow Model](../explanation/relational-workflow-model.md) in miniature. diff --git a/src/reference/specs/table-declaration.md b/src/reference/specs/table-declaration.md index 6bbb9833..0d994c14 100644 --- a/src/reference/specs/table-declaration.md +++ b/src/reference/specs/table-declaration.md @@ -23,7 +23,7 @@ class TableName(dj.Manual): | Tier | Base Class | Table Prefix | Purpose | |------|------------|--------------|---------| -| Manual | `dj.Manual` | (none) | Data inserted at runtime from outside the pipeline (users, instruments, ingestion scripts); quality assured by the data-management process | +| Manual | `dj.Manual` | (none) | Data inserted at runtime from outside the pipeline (users, instruments, entry scripts); quality assured by the data-management process | | Lookup | `dj.Lookup` | `#` | Reference data defined in the schema via `contents`; quality assured by code review | | Imported | `dj.Imported` | `_` | Populated by `make()` from an external source | | Computed | `dj.Computed` | `__` | Derived from other tables | diff --git a/src/tutorials/advanced/sql-comparison.ipynb b/src/tutorials/advanced/sql-comparison.ipynb index 68dbd121..a2fc5fcd 100644 --- a/src/tutorials/advanced/sql-comparison.ipynb +++ b/src/tutorials/advanced/sql-comparison.ipynb @@ -2067,7 +2067,7 @@ "| Tier | Purpose | SQL Equivalent |\n", "|------|---------|----------------|\n", "| `Lookup` | Reference data, parameters | Regular table |\n", - "| `Manual` | Data inserted directly from outside the pipeline (people or ingestion scripts) | Regular table |\n", + "| `Manual` | Data inserted directly from outside the pipeline (people or entry scripts) | Regular table |\n", "| `Imported` | Data from external sources (files, instruments, databases), populated by `make()` | Regular table + trigger |\n", "| `Computed` | Derived results | Materialized view + trigger |\n", "\n", diff --git a/src/tutorials/basics/01-first-pipeline.ipynb b/src/tutorials/basics/01-first-pipeline.ipynb index 031d3f31..c5d83802 100644 --- a/src/tutorials/basics/01-first-pipeline.ipynb +++ b/src/tutorials/basics/01-first-pipeline.ipynb @@ -13,7 +13,7 @@ "- Use the four core operations: restriction, projection, join, aggregation\n", "- Understand the schema diagram\n", "\n", - "We'll work with **Manual tables** only—tables where you enter data directly. Later tutorials introduce automated computation.\n", + "We'll work with **Manual tables** only—tables whose rows are written from outside the table, here by you. A Manual table is not necessarily hand-filled; an instrument or an entry script writes to one the same way. Later tutorials introduce automated computation.\n", "\n", "> **Database Backend:** All tutorials work identically on **MySQL** and **PostgreSQL** (PostgreSQL support added in DataJoint Python 2.1). The examples shown here were executed on PostgreSQL, but you can follow along using either backend.\n", "\n", diff --git a/src/tutorials/basics/02-schema-design.ipynb b/src/tutorials/basics/02-schema-design.ipynb index b93b334a..7097ffc9 100644 --- a/src/tutorials/basics/02-schema-design.ipynb +++ b/src/tutorials/basics/02-schema-design.ipynb @@ -48,20 +48,22 @@ "source": [ "## Table Tiers\n", "\n", - "DataJoint has four table tiers, each serving a different purpose:\n", + "DataJoint has four table tiers. A table's tier answers one question: **what puts rows in this table?**\n", "\n", - "| Tier | Class | Purpose | Data Entry |\n", - "|------|-------|---------|------------|\n", - "| **Manual** | `dj.Manual` | Core experimental data | Inserted directly by operators, instruments, or ingestion scripts |\n", - "| **Lookup** | `dj.Lookup` | Reference/configuration data | Pre-populated, rarely changes |\n", - "| **Imported** | `dj.Imported` | Data from external sources (files, instruments, databases) | Auto-populated via `make()` |\n", - "| **Computed** | `dj.Computed` | Derived/processed data | Auto-populated via `make()` |\n", + "| Tier | Class | What puts rows in it | Typical use |\n", + "|------|-------|----------------------|-------------|\n", + "| **Manual** | `dj.Manual` | A writer outside the table — an operator, an instrument, an entry script | Core experimental data |\n", + "| **Lookup** | `dj.Lookup` | The code, through the table's own `contents` | Reference and configuration data |\n", + "| **Imported** | `dj.Imported` | The table's own `make()`, reading an external source | Files, instruments, external databases |\n", + "| **Computed** | `dj.Computed` | The table's own `make()`, deriving from other tables | Derived and processed data |\n", "\n", - "**Manual** tables are not necessarily populated by hand—they contain data entered into the pipeline by operators, instruments, or ingestion scripts using `insert` commands. In contrast, **Imported** and **Computed** tables are auto-populated by calling the `.populate()` method, which invokes the `make()` callback for each missing entry.\n", + "The axis is *what writes the rows*, not *who caused them to be written*. **Manual** does not mean hand-filled: an instrument or a nightly entry script writes to a Manual table exactly as a person does, with `insert`. **Imported** and **Computed** are the two the framework fills for you, when you call `.populate()`, which invokes `make()` for each missing entry.\n", + "\n", + "See [the Relational Workflow Model](../../../explanation/relational-workflow-model/#workflow-steps-and-table-tiers) for the full axis, including why `Part` is not a fifth tier.\n", "\n", "### Manual Tables\n", "\n", - "Manual tables store data that is inserted directly—the starting point of your pipeline." + "Manual tables store rows written from outside the table—the starting point of your pipeline." ] }, { @@ -1420,7 +1422,7 @@ "- Keep keys minimal but sufficient for uniqueness\n", "\n", "### 2. Use Appropriate Table Tiers\n", - "- **Manual**: Data inserted directly — by operators, instruments, or ingestion scripts\n", + "- **Manual**: Data inserted directly — by operators, instruments, or entry scripts\n", "- **Lookup**: Configuration, parameters, reference data\n", "- **Imported**: Data read from external sources (recordings, images, instruments)\n", "- **Computed**: Derived analyses and summaries\n", diff --git a/src/tutorials/domain/calcium-imaging/calcium-imaging.ipynb b/src/tutorials/domain/calcium-imaging/calcium-imaging.ipynb index 288648cd..ea642dcc 100644 --- a/src/tutorials/domain/calcium-imaging/calcium-imaging.ipynb +++ b/src/tutorials/domain/calcium-imaging/calcium-imaging.ipynb @@ -1517,7 +1517,7 @@ "metadata": {}, "source": [ "**Legend:**\n", - "- **Green rounded boxes**: Manual tables (data entered at runtime — by a person, an instrument, or an ingestion script)\n", + "- **Green rounded boxes**: Manual tables (data entered at runtime — by a person, an instrument, or an entry script)\n", "- **Gray rounded boxes**: Lookup tables (parameters, part of the schema definition)\n", "- **Blue ellipses**: Imported tables (data from files)\n", "- **Orange ellipses**: Computed tables (derived from other tables)\n", diff --git a/src/tutorials/examples/blob-detection.ipynb b/src/tutorials/examples/blob-detection.ipynb index e0c6d643..8cbfd375 100644 --- a/src/tutorials/examples/blob-detection.ipynb +++ b/src/tutorials/examples/blob-detection.ipynb @@ -637,7 +637,7 @@ "metadata": {}, "source": [ "The diagram shows:\n", - "- **Green** = Manual tables (data inserted directly, manually or via ingestion)\n", + "- **Green** = Manual tables (data inserted directly, manually or by an entry script)\n", "- **Gray** = Lookup tables (reference data)\n", "- **Red** = Computed tables (derived data)\n", "- **Edges** = Dependencies (foreign keys), always flow top-to-bottom" From d167bf7975d6eede00ee953ab1f754384dcce800 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Fri, 2 Oct 2026 10:04:18 -0500 Subject: [PATCH 2/7] docs: hone the tier-axis prose A `/hone --focus all` pass over what this PR adds. - "so it is worth saying plainly" announced the point instead of making it. `worth` is the named marker for the emphatic appositive, and the colon was already doing the work. - "Crossing ... separates the two questions, and shows why only four combinations exist" described what the table would do. The table does it; the sentence now makes the claim. - A part table is not a structural role, it fills one. And "written in the same transaction as its master" repeated the master named one clause earlier. - "Imported and Computed tables define computations via `make()` methods" was the third telling of what both new tables already say. Cut; the sentence after it carries the claim that is not elsewhere -- derivation declared in the table definition rather than an external workflow file. - In the tutorial, "are the two the framework fills for you, when you call `.populate()`, which invokes" spliced three clauses onto a dangling "the two". Also "not" -> "never", matching the axis as the explanation page states it. --- src/explanation/relational-workflow-model.md | 13 ++++++------- src/tutorials/basics/02-schema-design.ipynb | 2 +- 2 files changed, 7 insertions(+), 8 deletions(-) diff --git a/src/explanation/relational-workflow-model.md b/src/explanation/relational-workflow-model.md index 364a4403..1d923286 100644 --- a/src/explanation/relational-workflow-model.md +++ b/src/explanation/relational-workflow-model.md @@ -155,7 +155,7 @@ A table's tier answers one question: **what puts rows in this table?** | **Computed** | The table itself, through `make()`, deriving from other DataJoint tables | Yes | The axis is *what writes the rows*, never *who caused them to be written*. Two -of the names suggest otherwise, so it is worth saying plainly: +of the names suggest otherwise: - **Manual** does not mean hand-entered. It means the rows arrive from outside the table, by whatever means — a technician typing into a form, a LIMS feed, an @@ -165,8 +165,8 @@ of the names suggest otherwise, so it is worth saying plainly: does: its own `make()` reaches out and fetches. A table filled *by something else* is Manual, however far away the data originated. -Crossing where the rows come from with what writes them separates the two -questions, and shows why only four combinations exist: +Cross where the rows come from with what writes them, and only four +combinations exist: | Rows come from | Written from outside the table | Written by the table's own `make()` | |---|---|---| @@ -175,11 +175,10 @@ questions, and shows why only four combinations exist: | Other DataJoint tables | — | **Computed** | `Part` is absent because it is not a fifth answer to the same question. A part -table is a structural role: it inherits its master's tier and is written in the -same transaction as its master. Any tier can serve as a master. +table fills a structural role: it inherits its master's tier and is written in +the same transaction. Any tier can serve as a master. -Imported and Computed tables define computations via `make()` methods. The -`make()` method specifies how each entity is derived — declared within the +The `make()` method specifies how each entity is derived — declared within the table definition, not in an external workflow file. !!! warning "The mistake the names invite" diff --git a/src/tutorials/basics/02-schema-design.ipynb b/src/tutorials/basics/02-schema-design.ipynb index 7097ffc9..34d5fdde 100644 --- a/src/tutorials/basics/02-schema-design.ipynb +++ b/src/tutorials/basics/02-schema-design.ipynb @@ -57,7 +57,7 @@ "| **Imported** | `dj.Imported` | The table's own `make()`, reading an external source | Files, instruments, external databases |\n", "| **Computed** | `dj.Computed` | The table's own `make()`, deriving from other tables | Derived and processed data |\n", "\n", - "The axis is *what writes the rows*, not *who caused them to be written*. **Manual** does not mean hand-filled: an instrument or a nightly entry script writes to a Manual table exactly as a person does, with `insert`. **Imported** and **Computed** are the two the framework fills for you, when you call `.populate()`, which invokes `make()` for each missing entry.\n", + "The axis is *what writes the rows*, never *who caused them to be written*. **Manual** does not mean hand-filled: an instrument or a nightly entry script writes to a Manual table exactly as a person does, with `insert`. **Imported** and **Computed** are the tiers the framework fills: calling `.populate()` invokes `make()` for each missing entry.\n", "\n", "See [the Relational Workflow Model](../../../explanation/relational-workflow-model/#workflow-steps-and-table-tiers) for the full axis, including why `Part` is not a fifth tier.\n", "\n", From 3bd48b6d3233222481acd33e2f8cc887ebc742fa Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Fri, 2 Oct 2026 10:11:24 -0500 Subject: [PATCH 3/7] docs: state the tier roles, drop the critique of the names MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The section argued with the names — "Manual does not mean hand-entered", "Imported does not say where the data came from", and a callout titled "The mistake the names invite". That is meta-discussion, not role documentation, and it reads as distraction in a page a reader came to for the model. It is also short-lived. `dj.Entry`, `dj.Ingest` and `dj.Compute` ship in 2.3.4, so prose explaining how the old names mislead would be stale as soon as the new names become primary. Both tables stay: they state each tier's role positively, and crossing where the rows come from with what writes them still shows why there are four. Net 24 lines removed. --- src/explanation/relational-workflow-model.md | 28 ++------------------ src/tutorials/basics/02-schema-design.ipynb | 2 +- 2 files changed, 3 insertions(+), 27 deletions(-) diff --git a/src/explanation/relational-workflow-model.md b/src/explanation/relational-workflow-model.md index 1d923286..961313dc 100644 --- a/src/explanation/relational-workflow-model.md +++ b/src/explanation/relational-workflow-model.md @@ -154,19 +154,8 @@ A table's tier answers one question: **what puts rows in this table?** | **Imported** | The table itself, through `make()`, reading an external source | Yes | | **Computed** | The table itself, through `make()`, deriving from other DataJoint tables | Yes | -The axis is *what writes the rows*, never *who caused them to be written*. Two -of the names suggest otherwise: - -- **Manual** does not mean hand-entered. It means the rows arrive from outside - the table, by whatever means — a technician typing into a form, a LIMS feed, an - instrument, or a nightly entry script. An automated feed still writes to a - Manual table. -- **Imported** does not say where the data came from. It says what the table - does: its own `make()` reaches out and fetches. A table filled *by something - else* is Manual, however far away the data originated. - -Cross where the rows come from with what writes them, and only four -combinations exist: +Two things vary independently: where the rows come from, and what writes them. +Crossing them gives four combinations, and those are the four tiers: | Rows come from | Written from outside the table | Written by the table's own `make()` | |---|---|---| @@ -181,19 +170,6 @@ the same transaction. Any tier can serve as a master. The `make()` method specifies how each entity is derived — declared within the table definition, not in an external workflow file. -!!! warning "The mistake the names invite" - A designer building an automated feed reasons: *this table is not filled by - hand, so it is not Manual; the data comes from outside, so it is Imported* — - and declares `dj.Imported`. But an Imported table is populated by its own - `make()`, and this one has none, so the insert is refused. The fix that - presents itself is `allow_direct_insert=True`, which silences a guard - reporting a real modeling error. - - The result is recognizable: an auto-populated table with **no `make()`**, a - permanent `allow_direct_insert=True`, and `populate()` that silently does - nothing — with the pipeline's boundary drawn one table away from where it - actually sits. Such a table is a **Manual** table. - #### Manual vs. Lookup Manual and Lookup tables are both **entry points** — their rows are entered diff --git a/src/tutorials/basics/02-schema-design.ipynb b/src/tutorials/basics/02-schema-design.ipynb index 34d5fdde..bcc5f447 100644 --- a/src/tutorials/basics/02-schema-design.ipynb +++ b/src/tutorials/basics/02-schema-design.ipynb @@ -57,7 +57,7 @@ "| **Imported** | `dj.Imported` | The table's own `make()`, reading an external source | Files, instruments, external databases |\n", "| **Computed** | `dj.Computed` | The table's own `make()`, deriving from other tables | Derived and processed data |\n", "\n", - "The axis is *what writes the rows*, never *who caused them to be written*. **Manual** does not mean hand-filled: an instrument or a nightly entry script writes to a Manual table exactly as a person does, with `insert`. **Imported** and **Computed** are the tiers the framework fills: calling `.populate()` invokes `make()` for each missing entry.\n", + "Rows reach a **Manual** table from outside it, through `insert` — a person typing, an instrument, or a nightly entry script all write the same way. **Imported** and **Computed** are the tiers the framework fills: calling `.populate()` invokes `make()` for each missing entry.\n", "\n", "See [the Relational Workflow Model](../../../explanation/relational-workflow-model/#workflow-steps-and-table-tiers) for the full axis, including why `Part` is not a fifth tier.\n", "\n", From 2673530ebafb3dc1b8ef5e22adafabb36497d0a3 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Fri, 2 Oct 2026 10:13:17 -0500 Subject: [PATCH 4/7] docs: state the tier axis as a sentence, not a bolded question MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "A table's tier answers one question: **what puts rows in this table?**" wedged a bolded question inside a declarative. It is now a plain sentence, and it matches the column header of the table it introduces. The axis stays "what puts rows in them" rather than data origin, because origin alone does not classify: Manual and Imported share one — both sit in the "Outside the pipeline" row of the second table — and are separated by what writes the rows. The writer alone does not classify either, since Imported and Computed are both written by the table's own `make()`. Only the two together give four cells, which is what that table shows. The Lookup row now says what its rows are rather than who writes them: the schema definition itself, committed as `contents`. That matches how the "Manual vs. Lookup" section below already describes it. --- src/explanation/relational-workflow-model.md | 4 ++-- src/tutorials/basics/02-schema-design.ipynb | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/explanation/relational-workflow-model.md b/src/explanation/relational-workflow-model.md index 961313dc..4a575708 100644 --- a/src/explanation/relational-workflow-model.md +++ b/src/explanation/relational-workflow-model.md @@ -145,11 +145,11 @@ work in practice. ### Workflow steps and table tiers -A table's tier answers one question: **what puts rows in this table?** +Tables are classified into tiers by what puts rows in them. | Tier | What puts rows in it | `make()` | |------|------|----------| -| **Lookup** | The code, through the table's own `contents` | No | +| **Lookup** | The schema definition itself — the table's committed `contents` | No | | **Manual** | A writer outside the table — a person, an instrument, an entry script | No | | **Imported** | The table itself, through `make()`, reading an external source | Yes | | **Computed** | The table itself, through `make()`, deriving from other DataJoint tables | Yes | diff --git a/src/tutorials/basics/02-schema-design.ipynb b/src/tutorials/basics/02-schema-design.ipynb index bcc5f447..2f8fc670 100644 --- a/src/tutorials/basics/02-schema-design.ipynb +++ b/src/tutorials/basics/02-schema-design.ipynb @@ -48,12 +48,12 @@ "source": [ "## Table Tiers\n", "\n", - "DataJoint has four table tiers. A table's tier answers one question: **what puts rows in this table?**\n", + "DataJoint has four table tiers, classified by what puts rows in them.\n", "\n", "| Tier | Class | What puts rows in it | Typical use |\n", "|------|-------|----------------------|-------------|\n", "| **Manual** | `dj.Manual` | A writer outside the table — an operator, an instrument, an entry script | Core experimental data |\n", - "| **Lookup** | `dj.Lookup` | The code, through the table's own `contents` | Reference and configuration data |\n", + "| **Lookup** | `dj.Lookup` | The schema definition itself — the table's committed `contents` | Reference and configuration data |\n", "| **Imported** | `dj.Imported` | The table's own `make()`, reading an external source | Files, instruments, external databases |\n", "| **Computed** | `dj.Computed` | The table's own `make()`, deriving from other tables | Derived and processed data |\n", "\n", From ea24a55efea9e6ed15aa057b268bbdc650e5dc01 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Fri, 2 Oct 2026 10:16:08 -0500 Subject: [PATCH 5/7] docs: one tier table, with origin as a column The second table put **Lookup** under "written from outside the table", which is false: a Lookup's `contents` are part of its own declaration. The column grouped Lookup with Manual under a label only Manual satisfied. Its other axis was already the first table's `make()` column, so the fix is a column rather than a relabel. Origin joins the tier table, and Manual and Imported now sit one above the other with the same origin and different writers -- which is the distinction the matrix existed to draw. What the matrix did carry, that two empty cells explained why there are four tiers, goes with it. The one sentence worth keeping says it directly. --- src/explanation/relational-workflow-model.md | 33 +++++++++----------- 1 file changed, 14 insertions(+), 19 deletions(-) diff --git a/src/explanation/relational-workflow-model.md b/src/explanation/relational-workflow-model.md index 4a575708..1a6d9ae1 100644 --- a/src/explanation/relational-workflow-model.md +++ b/src/explanation/relational-workflow-model.md @@ -147,25 +147,20 @@ work in practice. Tables are classified into tiers by what puts rows in them. -| Tier | What puts rows in it | `make()` | -|------|------|----------| -| **Lookup** | The schema definition itself — the table's committed `contents` | No | -| **Manual** | A writer outside the table — a person, an instrument, an entry script | No | -| **Imported** | The table itself, through `make()`, reading an external source | Yes | -| **Computed** | The table itself, through `make()`, deriving from other DataJoint tables | Yes | - -Two things vary independently: where the rows come from, and what writes them. -Crossing them gives four combinations, and those are the four tiers: - -| Rows come from | Written from outside the table | Written by the table's own `make()` | -|---|---|---| -| The committed schema | **Lookup** | — | -| Outside the pipeline | **Manual** | **Imported** | -| Other DataJoint tables | — | **Computed** | - -`Part` is absent because it is not a fifth answer to the same question. A part -table fills a structural role: it inherits its master's tier and is written in -the same transaction. Any tier can serve as a master. +| Tier | Rows come from | What puts them there | `make()` | +|------|----------------|----------------------|----------| +| **Lookup** | The committed schema | The table's own `contents`, versioned with the code | No | +| **Manual** | Outside the pipeline | A writer outside the table — a person, an instrument, an entry script | No | +| **Imported** | Outside the pipeline | The table itself, fetching through `make()` | Yes | +| **Computed** | Other DataJoint tables | The table itself, deriving through `make()` | Yes | + +**Manual** and **Imported** share an origin and differ in the writer: the data +comes from outside the pipeline either way, and what changes is whether the +table fetches it itself. + +`Part` is absent because it is not a tier of its own. A part table fills a +structural role: it inherits its master's tier and is written in the same +transaction. Any tier can serve as a master. The `make()` method specifies how each entity is derived — declared within the table definition, not in an external workflow file. From 3659b8a45b2c3faaaac88fd89361dea1486e064f Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Fri, 2 Oct 2026 10:20:14 -0500 Subject: [PATCH 6/7] docs: name the three new tier names, without arguing about the old ones The section now states the renaming as a fact: `dj.Entry`, `dj.Ingest` and `dj.Compute` arrive in 2.3.4 as second names, each pair is one class, both names are permanent, and the new ones become primary in 2.4 (#1546). No claim about what the current names suggest. A reader who wants the distinction has the table above it. --- src/explanation/relational-workflow-model.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/explanation/relational-workflow-model.md b/src/explanation/relational-workflow-model.md index 1a6d9ae1..e2185002 100644 --- a/src/explanation/relational-workflow-model.md +++ b/src/explanation/relational-workflow-model.md @@ -158,6 +158,16 @@ Tables are classified into tiers by what puts rows in them. comes from outside the pipeline either way, and what changes is whether the table fetches it itself. +!!! version-added "New in 2.3.4" + + Three tiers gain a second name: **`dj.Entry`** for `dj.Manual`, + **`dj.Ingest`** for `dj.Imported`, and **`dj.Compute`** for `dj.Computed`. + Each pair is one class, so either name declares the same table, and both + names are permanent. `dj.Lookup` and `dj.Part` are unchanged. + + These pages use the original names. The new ones become primary in 2.4 + ([datajoint-python#1546](https://github.com/datajoint/datajoint-python/issues/1546)). + `Part` is absent because it is not a tier of its own. A part table fills a structural role: it inherits its master's tier and is written in the same transaction. Any tier can serve as a master. From a902a9f0f8102f7b1dafdbbb3609c873dbd77b66 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Fri, 2 Oct 2026 10:27:55 -0500 Subject: [PATCH 7/7] docs: what separates Entry from Ingest is initiation, and the key that follows The paragraph said they "share an origin and differ in the writer", which restates the table and teaches nothing. The real difference is in two parts, and the second follows from the first: An external process initiates a write to a Manual table, on its own schedule. An Imported table is filled automatically -- `populate()` works through the keys its parents already hold. So `populate()` has to know which entity it is working on before `make()` runs, which is why every attribute of an Imported table's primary key must arrive through a foreign key. Both halves are enforced, with error messages that say so: `key_source` refuses a table with no primary-key parents, and `_validate_pk_attributes` refuses a non-FK key attribute. A Manual table carries neither constraint -- no parent required, and it may introduce key attributes of its own. That is what "a new entity enters here" means, and it ties the tier back to the new entity type the page's intro already defines. --- src/explanation/relational-workflow-model.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/src/explanation/relational-workflow-model.md b/src/explanation/relational-workflow-model.md index e2185002..72bc2964 100644 --- a/src/explanation/relational-workflow-model.md +++ b/src/explanation/relational-workflow-model.md @@ -154,9 +154,19 @@ Tables are classified into tiers by what puts rows in them. | **Imported** | Outside the pipeline | The table itself, fetching through `make()` | Yes | | **Computed** | Other DataJoint tables | The table itself, deriving through `make()` | Yes | -**Manual** and **Imported** share an origin and differ in the writer: the data -comes from outside the pipeline either way, and what changes is whether the -table fetches it itself. +The table shows **Manual** and **Imported** drawing on the same origin. What +separates them is who initiates the write. + +A Manual table is written by an external process, on that process's schedule. +An Imported table is filled automatically: `populate()` works through the keys +its parents already hold and calls `make()` for each one still missing. + +Their primary keys follow from that. `populate()` has to know which entity it +is working on before `make()` runs, so every attribute of an Imported table's +primary key arrives through a foreign key. A Manual table carries no such +constraint — it may sit at the head of the pipeline with no parent at all, and +may introduce primary-key attributes of its own. That is what makes it the +place a new entity enters. !!! version-added "New in 2.3.4"