Repository navigation
Conversation
The readiness profile, the record of the constants a job resolved, tag properties and `export tag`, the property vocabulary with UCUM checks and JSON-LD, and a Great Expectations example. Read with web-agents-common.md and ccdb-2.5.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d data One SQL count per number over the whole database, nothing sampled: columns with units, described tables, commented versions, open-ended run ranges, versions by anonymous, stale and empty tables, deprecations, shared and unused constant sets, instance properties. The staleness cutoff is the start of the UTC day minus stale_years (5 by default), echoed in `config`, so two profiles of one day compare line by line. unit_test_readiness checks every number against plain SQL on SQLite and MySQL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ess profile The text form is one line per count with its share of the total; --json prints the profile dict. The command only reads. A negative window exits 1. Tested in integ_test_cli_manager and run by cli-smoke.yml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…an hour The same dict as `ccdb db readiness --json`; `stale_years` from 0 to 100, 5 by default. The answer is cached for an hour (HOURLY), since the counts move slowly and cost seconds on a large instance. The API ruler requests it with and without a window and with a refused one; golden.py checks twelve counts against SQL on the fixture and the cache header. OpenAPI document updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The same values as /api/v1/readiness; the tool list grows to 14, all read-only. The server instructions point agents to it, and the recorded session in docs/apis/mcp.md starts with it: the agent checks what the instance documents before it reads constants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Metadata, history, data and instance properties as cards, each count with its share of the total. Each process keeps a profile for an hour and the page carries the same Cache-Control, since the counts cost seconds on a large instance. The Documentation page links to it. The GUI ruler requests it with and without a window and with a refused one; golden.py checks nine counts against SQL on the fixture (the golden file gains the key, every other value is unchanged). Browser check: light and dark, desktop and phone, clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…I pages concepts/ai-readiness.md maps readiness requirements onto CCDB features and defines every field of the profile, with the Hall D profile as the example. cli.md shows the text form; http.md lists /readiness and its hourly cache. The page joins the sidebar. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…geRecord), C++ and Python Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
C++: Calibration::EnableUsageRecord(bool), IsUsageRecordEnabled(), GetResolvedAssignments(), ClearResolvedAssignments(). While on, every distinct resolution (table, run, variation, time or tag, assignment) is kept once in request order, with the SHA-256 of the data the program received; a table read per event adds one entry. Python: AlchemyProvider usage_record_enabled (False), resolved_assignments with the same fields, filled by get_assignment, and clear_resolved_assignments(). Tests: Catch2 [usage] on SQLite, MySQL and MariaDB, cache on and off; an [http] case equal to reading the file; unit_test_usage_record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the usage record At startup CCDB_service logs the connection (password masked) with the variation and time, or the tag, and warns when neither ccdb:time nor ccdb:tag is set: such a job reads the newest constants, which its parameters can't repeat. ccdb:provenance=PATH turns on the usage record of every Calibration the service hands out and writes it as JSON when the service ends (or on WriteUsageRecord()): a header with the CCDB version, connection, variation, time or tag, and per run every distinct resolution with its assignment id and SHA-256. The file is written whole and renamed. [jana2] tests for both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t the database For every entry of a record (the JANA2 service's ccdb:provenance file, or AlchemyProvider.resolved_assignments as JSON): the assignment exists, belongs to the table named, and its stored SHA-256 equals the one the job received; the command also shows what the same request resolves to today. --tag lists the tags that give every entry's assignment. Exit code 1 when an entry fails. unit_test_provenance and a cli-smoke step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jana.md: the startup log and its warning, ccdb:provenance, the record's format. cpp.md: EnableUsageRecord and GetResolvedAssignments. python.md: usage_record_enabled and resolved_assignments. cli.md: `ccdb provenance`. The AI readiness page gains the provenance-of-use row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, API, MCP and GUI A tag is a property owner (owner type 'tag'; no schema change), named with @ as in a request: prop set|get|ls|rm @name. Standard keys for a tag: description, software, container, data, rucio_did, doi, citation. `tag NAME`, `info @NAME` and `tag -l` print them; /api/v1/tags and /tags/{name} and the MCP tool list_tags answer them; the GUI tag page lists them. Removing a tag removes its properties, as for other owners. One more query on the GUI tag page; byte-identical GUI pages on the fixture, which has no tag properties. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s hash and a DataCite record ccdb.tag_export builds the file with the run-bundle builder over every run: every table, the tag's variation and its parents, every assignment the tag sees, the tag, and the properties of the instance, directories, tables, columns, those variations and the tag. NAME.sha256 is in sha256sum form; NAME.datacite.json is a DataCite 4.5 record from the instance and tag properties (creator, publisher, license, URL, software, container, data, Rucio DID), with the previous export's DOI as IsNewVersionOf. CCDB contacts no outside service; an export is never overwritten. unit_test_export_tag compares requests through the tag in the file and the source and validates the record against the DataCite 4.5 schema, vendored under scripts/docs/datacite (Invenio, Revised BSD License). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cpp-tests.yml exports tag_b of the time resolution database and the C++ library reads it: the tag, any run, and the newest value of the file all give the tag's constants. The exporter silences SQLAlchemy's reflection warnings about SQLite column types, which said nothing to a user. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… record Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
concepts/tags.md: "Tag properties" and "Publishing a tag" (export, upload, record the DOI). cli.md: @tag in `prop`, the tag names, `tag -l`, and "Export of a tag" with the output of a Hall D tag export (42 s, 1.8 GB). http.md, mcp.md and the AI readiness page name the tag properties and the export. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ccdb.ucum.is_ucum checks a unit string against UCUM's grammar (products,
quotients, exponents, prefixes on metric atoms, parentheses, 10*N, {annota-
tions}) with a table of the atoms calibration tables use; no dependency, no
network. mktbl warns on units UCUM can't read (Mev, nsec, GeV/c) and stores
them as written. AlchemyProvider.set_column_units sets or clears the units
of a column of an existing table, with a log record: `edit` has no option for
it while #125 rewrites that command. The readiness profile fills
units.columns_with_ucum_units. unit_test_ucum.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s, /api/v1/properties/vocabulary ccdb.vocabulary holds the standard names per owner type with their meanings (instance gains publisher and creator, used by the tag export). `prop set` warns on another name and sets it; --free silences the warning; the instance property property_names (comma-separated) adds the names an experiment agreed on. The API serves the vocabulary. The tables in docs/cli.md are generated by scripts/docs/vocabulary_table.py, and a docs test fails when page and code differ. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lock on the GUI card page queries.card_jsonld builds the Dataset from the card: name and identifier (the table path), description, url, creator (the authors of its versions), dateModified, license and citation, keywords, the catalog of the experiment, and one variableMeasured per column with its units. The API serves it with application/ld+json; the GUI card page embeds it through tojson, which escapes <, > and &, so a hostile table name cannot end the block. The API ruler gains the JSON-LD and a refused format; golden checks the columns and units against SQL (its table-info filter now skips every card URL). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fair.md: F1 (tag export for a DOI), F2 (vocabulary), F4 (JSON-LD cards), I2 (UCUM and the vocabulary) and R1.2 (the job record) move to today; the data card section shows the JSON-LD form. cli.md: the UCUM check of mktbl and set_column_units for old tables. http.md: /properties/vocabulary and the card's jsonld form. The AI readiness page: the units field, and the tools that fix each gap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
example_expectations_parquet.py exports /test/test_vars/test_table and validates the file: the column set, float64 values between 0 and 100, rows 0-1 once per version, and the row count of every version. It turns off Great Expectations' usage analytics and progress bars. python-examples.yml runs it first, on the standard test content, after installing the export extra and great_expectations>=1,<2; tested here with 1.8.1 on Python 3.9 and 1.24.1 on Python 3.12. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndex analysis.md shows example_expectations_parquet.py as it is, with its output; a docs test keeps page and example equal and the example in CI. index.md names the readiness profile, the record of what a job used, and tags that name a pass and export for a DOI, and the schema.org form of the data cards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…written Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it runs without SQLAlchemy The documentation job installs no CCDB; importing the ccdb package needs SQLAlchemy. The vocabulary module imports nothing of the package. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a tag The JSON-LD of a data card took its url from the request, so the API answered other bytes behind nginx than in process (api-container check). The url is now the card page under the instance property `url`, the published GUI address, and absent without it. mysql-tests.yml exports tag_b of the MySQL time database for the [export] test, which CI requires: the export now runs from a MySQL source in CI too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…(arXiv:2609.34987) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AI readiness for CCDB 2.5: the plan
ccdb-ai-ready.md(phases A0–A5), built on 2.5.0 as merged in #122. Nothing changes the database schema; everything that worked in 2.5.0 keeps working.What it adds
ccdb db readiness [--stale-years N] [--json]counts what an instance documents, over the whole database: units (and UCUM units), described tables, commented versions, open-ended run ranges, anonymous versions, stale and empty tables, shared and unused sets, instance license/citation/contact. The same profile at/api/v1/readiness(cached an hour), as the MCP toolreadiness, and on the GUI page/readiness. On a copy of the Hall D database it equals a hand count by SQL on every field (7 s).Calibration::EnableUsageRecord(true)in C++,usage_record_enabledin Python. It keeps each distinct resolution once, with the assignment id and the SHA-256 of the data received. The JANA2 service writes it as JSON with-Pccdb:provenance=FILE, logs at startup what it pins, and warns when a job pins neither a time nor a tag.ccdb provenance FILE [--tag]checks a record against any copy of the database.ccdb prop set @pass1 software ..., shown bytag,info @NAME, the API, MCP and the GUI.ccdb export tag @NAME -o DIRwrites the tag's constants for every run as one SQLite file, its SHA-256, and a DataCite 4.5 record; no network call. A Hall D production tag: 42 s, 1.8 GB.prop setwarns on names outside a vocabulary (--free;property_namesextends it;/api/v1/properties/vocabulary).mktblwarns on units UCUM can't read (ccdb.ucum, no dependency).AlchemyProvider.set_column_unitsgives units to old tables. Data cards as schema.orgDatasetJSON-LD (/card?format=jsonld, and a block on the GUI card page).python-examples.yml.docs/concepts/ai-readiness.md, and the AI-readiness sections of the CLI, C++, Python, JANA2, HTTP, MCP, analysis, tags, FAIR and index pages.Deviations from the plan
editdoes not check units: on v2-main it is a stub, and CLI: edit -a ID --comment replaces the comment of a version (#56) #125 rewrites it. Units of existing tables are set withset_column_units.ccdb provenancereads them by id.Tests
Local gate after every phase: Python on SQLite, MySQL 8.3 and MariaDB 10.5 (433 → 479 tests), C++ on all three engines (55 → 58 cases),
[http],[jana2]and DD4hep in eic-shell, the API and GUI rulers (golden 1127 / 574), browser check in Chromium and Firefox (712 visits), docs links, MCP transcript, the demo stack. Every new test fails without its change.🤖 Generated with Claude Code