From 06adeb344f698b469f70418e1317d75d302b31b0 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 21 Oct 2026 12:54:17 -0400 Subject: [PATCH 1/4] fix: close the 1.1.0 release review findings before rc.7 Fixes everything the 1.1.0 release review found, plus the adopter-found `eject --list` defect, and closes the test gaps it named. - Decimals: one canonical string on the wire in every port (W1-W4), a shared decimal-wire corpus and an api-contract decimal/ sub-corpus, exact filter operands, and 400 validation past @precision/@scale. The stronger assertions found the TypeScript TPH polymorphic read sending a subtype-only decimal uncanonicalized; fixed in the generator. - Report lanes: every port runs the report scenarios on SQLite and MySQL (Java OMDB SQLiteDriver, Python SQLite/MySQL drivers), the Java report lane is database-backed, and the Cube lane queries every exporter shape. - Loader: four more report shapes refused (D5, M10, R10, R11), an integral float accepted where an int is declared, the requirement package and ERR_UNRESOLVED_SUPER hint fixed. - eject --list reports owned companion files; JVM and Python naming discovery; TS forms step/inputMode; config unknown keys warn. - Breaking (recorded in CHANGELOG and the 1.0-to-1.1 upgrade guide): C# scalar and enum arrays are IList; a Java POJO redeclaring an inherited field with another subtype is standalone; Kotlin gains per-package MetaDecimalWire.kt and MetaPortableColumnTypes.kt. --- .github/workflows/publish-csharp.yml | 17 +- .github/workflows/publish-java.yml | 5 + .github/workflows/publish-npm.yml | 5 + .github/workflows/publish-python.yml | 5 + AGENTS.md | 2 +- CHANGELOG.md | 240 ++++++++-- README.md | 3 +- .../metaobjects-codegen/references/java.md | 7 +- .../metaobjects-codegen/references/kotlin.md | 7 +- .../packages/react/src/use-entity-form.tsx | 4 + .../test/use-entity-form-fractional.test.tsx | 56 +++ docs/1.0-readiness.md | 2 +- docs/CONFORMANCE.md | 79 ++-- docs/README.md | 6 +- docs/RELEASING.md | 24 +- docs/compatibility-policy.md | 45 +- docs/features/api-contract.md | 35 +- docs/features/cube-export.md | 51 ++- docs/features/field-types.md | 11 +- .../migrations/docs-are-generators.md | 9 +- docs/features/migrations/physical-naming.md | 7 +- .../migrations/upgrading-1.0-to-1.1.md | 189 ++++++++ .../migrations/upgrading-within-1.0.md | 4 + docs/features/own-your-codegen.md | 14 +- docs/features/reporting.md | 133 +++--- docs/features/requirements.md | 29 +- docs/ports/csharp.md | 22 +- docs/ports/java.md | 21 +- docs/ports/kotlin.md | 18 +- docs/ports/python.md | 19 +- docs/ports/typescript-client.md | 2 +- docs/ports/typescript.md | 14 + ...-05-26-fr-009-filter-operators-per-port.md | 9 +- .../codegen/generators/routes.ts | 17 +- .../codegen/runtime/decimal-wire.ts | 143 +++++- .../drizzle-fastify/filter-allowlist.ts | 6 +- .../runtime/drizzle-fastify/filter-parser.ts | 31 +- .../drizzle-fastify/filter-value-format.ts | 4 + .../codegen/runtime/drizzle-fastify/index.ts | 14 +- .../runtime/drizzle-fastify/mount-m2m.ts | 5 +- .../drizzle-fastify/mount-read-only.ts | 15 +- .../codegen/templates/decimal-wire-helper.ts | 47 ++ .../codegen/templates/entity-constants.ts | 2 + .../codegen/templates/field-meta.ts | 36 ++ .../codegen/templates/filter-allowlist.ts | 1 + .../codegen/templates/zod-validators.ts | 34 +- .../.metaobjects/.gen-state/.hashes.json | 4 +- .../showcase/codegen/generators/routes.ts | 17 +- .../showcase/codegen/runtime/decimal-wire.ts | 143 +++++- .../drizzle-fastify/filter-allowlist.ts | 6 +- .../runtime/drizzle-fastify/filter-parser.ts | 31 +- .../drizzle-fastify/filter-value-format.ts | 4 + .../codegen/runtime/drizzle-fastify/index.ts | 14 +- .../runtime/drizzle-fastify/mount-m2m.ts | 5 +- .../drizzle-fastify/mount-read-only.ts | 15 +- .../codegen/templates/decimal-wire-helper.ts | 47 ++ .../codegen/templates/entity-constants.ts | 2 + .../showcase/codegen/templates/field-meta.ts | 36 ++ .../codegen/templates/filter-allowlist.ts | 1 + .../codegen/templates/zod-validators.ts | 34 +- .../generated/csharp/SubscriberRoutes.g.cs | 6 +- .../generated/python/subscriber_router.py | 56 ++- examples/showcase/site-payload.json | 4 +- .../references/surface-review-axes.md | 3 +- .../metaobjects-codegen/references/java.md | 7 +- .../metaobjects-codegen/references/kotlin.md | 7 +- .../references/surface-review-axes.md | 3 +- .../metaobjects-codegen/references/java.md | 7 +- .../references/surface-review-axes.md | 3 +- .../references/surface-review-axes.md | 3 +- .../references/surface-review-axes.md | 3 +- fixtures/api-contract-conformance/README.md | 10 +- .../decimal/README.md | 57 +++ .../decimal/meta.json | 25 ++ .../decimal-filter-sort-exact-15.yaml | 44 ++ .../scenarios/decimal-filter-sort-exact.yaml | 97 ++++ .../scenarios/decimal-read-exact-15.yaml | 28 ++ .../decimal/scenarios/decimal-read-exact.yaml | 53 +++ .../scenarios/decimal-report-exact-15.yaml | 34 ++ .../scenarios/decimal-report-exact.yaml | 53 +++ .../scenarios/decimal-write-exact-15.yaml | 72 +++ .../scenarios/decimal-write-exact.yaml | 91 ++++ .../decimal/schema.mysql.sql | 23 + .../decimal/schema.postgres.sql | 25 ++ .../decimal/schema.sqlite.sql | 23 + .../decimal/seed.json | 25 ++ .../projection/README.md | 5 +- .../projection/scenarios/filter-decimal.yaml | 24 +- .../api-contract-conformance/report/README.md | 33 +- .../api-contract-conformance/report/meta.json | 20 + .../filter-array-and-map-dimension.yaml | 29 ++ .../report/scenarios/filter-on-measure.yaml | 9 +- .../report/scenarios/list-totals.yaml | 9 +- .../sort-array-and-map-dimension.yaml | 31 ++ .../report/schema.mysql.sql | 14 + .../report/schema.postgres.sql | 16 + .../report/schema.sqlite.sql | 14 + .../api-contract-conformance/tph/README.md | 6 +- .../tph-per-subtype-list-and-create.yaml | 39 +- .../tph-polymorphic-list-and-get.yaml | 2 +- .../attr-int-integral-float/expected.json | 53 +++ .../input/meta.shop.json | 54 +++ .../attr-int-integral-float/providers.json | 1 + .../expected-errors.json | 15 + .../input/meta.shop.json | 33 ++ .../error-attr-int-fractional/providers.json | 1 + .../expected-errors.json | 15 + .../input/meta.events.json | 75 ++++ .../providers.json | 1 + .../expected-errors.json | 15 + .../input/meta.shop.json | 39 ++ .../providers.json | 1 + .../expected-errors.json | 4 + .../expected-errors.json | 15 + .../input/meta.shop.json | 321 ++++++++++++++ .../providers.json | 1 + .../expected-errors.json | 15 + .../input/meta.shop.json | 321 ++++++++++++++ .../providers.json | 1 + .../expected-errors.json | 15 + .../input/meta.events.json | 76 ++++ .../error-measure-over-object/providers.json | 1 + .../expected-errors.json | 15 + .../input/meta.events.json | 90 ++++ .../error-report-abstract/providers.json | 1 + .../expected-errors.json | 15 + .../input/meta.events.json | 89 ++++ .../providers.json | 1 + .../expected-errors.json | 15 + .../input/meta.events.json | 89 ++++ .../providers.json | 1 + .../expected-errors.json | 15 + .../input/meta.events.json | 89 ++++ .../providers.json | 1 + .../expected-errors.json | 15 + .../input/meta.requirements.json | 16 + .../providers.json | 1 + .../expected.json | 332 ++++++++++++++ .../input/meta.shop.json | 320 ++++++++++++++ .../providers.json | 1 + .../expected.json | 50 +++ .../input/meta.a-order.json | 27 ++ .../input/meta.b-requirement.json | 16 + .../input/meta.c-overlay.json | 27 ++ .../providers.json | 1 + .../expected.json | 16 + .../input/meta.requirements.json | 16 + .../providers.json | 1 + .../conformance/yaml-twin-exclusions.json | 1 + fixtures/cube-model/README.md | 4 +- .../canonical/expected/model/cubes/Entry.yml | 48 ++ .../canonical/expected/model/cubes/Gadget.yml | 14 + .../expected/model/cubes/GadgetEvent.yml | 25 ++ .../expected/model/cubes/Program.yml | 25 +- fixtures/cube-model/canonical/seed.mysql.sql | 20 + fixtures/cube-model/canonical/seed.sql | 20 + .../expected/model/cubes/Program.yml | 14 +- .../cube-model/report-sourceless/meta.json | 41 -- fixtures/decimal-wire/README.md | 18 + fixtures/decimal-wire/vectors.json | 73 +++ fixtures/persistence-conformance/README.md | 44 +- .../canonical/meta.fitness.json | 46 ++ .../canonical/schema.mysql.sql | 39 ++ .../canonical/schema.postgres.sql | 45 ++ .../canonical/schema.sqlite.sql | 39 ++ .../bootstrap-canonical-from-empty.yaml | 5 + .../queries/decimal-exact-15.yaml | 55 +++ .../queries/decimal-exact.yaml | 112 +++++ .../queries/report-decimal-exact-15.yaml | 23 + .../queries/report-decimal-exact.yaml | 32 ++ .../queries/report-via-redeclared-field.yaml | 26 ++ .../report-shapes.json | 84 ++++ .../requirement-check-conformance/README.md | 3 + .../expected.json | 15 + .../input/meta.a-shop.json | 36 ++ .../input/meta.b-overlay.json | 28 ++ .../expected.json | 15 + .../input/meta.a-shop.json | 36 ++ .../input/meta.b-overlay.json | 27 ++ .../expected.json | 27 ++ .../input/meta.a-shop.json | 27 ++ .../input/meta.b-requirements.json | 18 + .../source-resolution-conformance/README.md | 54 +-- scripts/check-release-tag.mjs | 99 +++++ scripts/ci-local.sh | 4 +- scripts/eject-release-gate.test.ts | 3 +- scripts/lib/changelog-promote.mjs | 31 ++ scripts/regen-showcase.ts | 52 ++- scripts/release.mjs | 25 +- scripts/test-changelog-promote.mjs | 34 ++ scripts/test-check-release-tag.mjs | 45 ++ server/csharp/Directory.Build.props | 2 + .../CodegenCompileConformanceTests.cs | 2 +- .../DecimalWireTests.cs | 276 ++++++++++++ .../EntityGeneratorTests.cs | 4 + ...ndsAbstractFieldInheritanceCodegenTests.cs | 2 +- .../ExtractorCodegenTests.cs | 12 +- .../Issue195NativeTypingCodegenTests.cs | 2 +- .../OrphanSweepTests.cs | 79 ++++ .../ReportRowCodegenTests.cs | 78 ++-- .../ScalarArrayCodegenTests.cs | 16 +- .../ApiDocs/CSharpApiModelBuilder.cs | 7 +- .../Generators/EntityGenerator.cs | 330 +++++++++++++- .../Generators/RoutesGenerator.cs | 82 +++- .../MetaObjects.Codegen/HelperRuntime.cs | 5 +- .../MetaObjects.Codegen.csproj | 2 + .../csharp/MetaObjects.Codegen/ReportRows.cs | 6 +- .../Runtime/DecimalRangeEndpointFilter.cs | 51 +++ .../Runtime/DecimalWire.cs | 272 ++++++++++++ .../Runtime/EfCoreFilterDispatch.cs | 41 ++ .../ConformanceAdapter.cs | 5 +- .../ConformanceTests.cs | 8 + .../DataConverterTests.cs | 16 + .../OperationScript.cs | 14 +- .../ReportShapeTests.cs | 2 +- .../ReportingValidationTests.cs | 9 +- .../Api/ApiContractAssertions.cs | 7 +- .../ApiContractAssertionsExactnessTests.cs | 51 +++ .../Api/ApiContractConformanceTest.cs | 2 +- .../Api/ApiContractCorpusPaths.cs | 10 + .../Api/ApiContractDecimalConformanceTest.cs | 108 +++++ .../ApiContractGeneratedConformanceTest.cs | 2 +- .../Api/ApiContractJsonbConformanceTest.cs | 2 +- .../Api/ApiContractM2mConformanceTest.cs | 2 +- .../ApiContractProjectionConformanceTest.cs | 2 +- .../Api/ApiContractReportConformanceTest.cs | 2 +- .../Api/ApiContractScenarios.cs | 33 +- .../Api/ApiContractTphConformanceTest.cs | 2 +- .../ApiContractWriteThroughConformanceTest.cs | 2 +- .../Api/AuthorApiServer.cs | 2 +- .../Api/DecimalArrayReportRowTests.cs | 53 +++ .../Api/DecimalWireGeneratedApiTests.cs | 151 +++++++ .../Api/EntityArraySqliteTests.cs | 67 +++ .../Api/GeneratedRoutesServer.cs | 121 +++++ .../Api/JsonbReferenceServer.cs | 2 +- .../Api/ReportFixture.cs | 18 +- .../Api/ReportGeneratedServerFactory.cs | 2 +- .../Api/TphReferenceServer.cs | 80 +++- .../Generated/AllTypes.g.cs | 1 + .../Generated/AllTypesBreakdown.g.cs | 1 + .../Generated/AppDbContext.g.cs | 9 + .../Generated/CopayAuth.g.cs | 1 + .../Generated/DecimalWire.g.cs | 272 ++++++++++++ .../Generated/Entry.g.cs | 27 ++ .../Generated/EntryNames.g.cs | 42 ++ .../Generated/EntryTotals.g.cs | 38 ++ .../Generated/FitnessTotals.g.cs | 1 + .../Generated/FitnessTotalsFilled.g.cs | 1 + .../Generated/Gadget.g.cs | 19 + .../Generated/GadgetEvent.g.cs | 19 + .../Generated/GadgetEventNames.g.cs | 39 ++ .../Generated/GadgetEventQueries.g.cs | 24 + .../Generated/GadgetEventsByLabel.g.cs | 17 + .../Generated/GadgetNames.g.cs | 30 ++ .../Generated/LabeledNames.g.cs | 31 ++ .../Generated/Measurement.g.cs | 1 + .../Generated/MeasurementStats.g.cs | 3 + .../Generated/MeasurementStatsFilled.g.cs | 2 + .../Generated/ProgramMinutes.g.cs | 2 + .../Generated/ProgramRoster.g.cs | 2 + .../Generated/ProgramsEmptyScope.g.cs | 1 + .../Generated/ProgramsOverTime.g.cs | 1 + .../QueryScenarioMySqlTests.cs | 4 +- .../QueryScenarioSqliteTests.cs | 4 +- .../QueryScenarioTests.cs | 5 +- .../Runner/CorpusPaths.cs | 11 +- .../Runner/DbContextAdapter.cs | 6 + .../Runner/DecimalGate.cs | 85 ++++ .../Runner/QueryScenarioRunner.cs | 3 + .../Runner/ReportViewsDbContext.cs | 6 +- .../Runner/Scenario.cs | 16 +- .../Runner/ScenarioLoader.cs | 16 + .../Runner/WriteCoercion.cs | 5 +- .../Core/Requirement/RequirementCheck.cs | 22 +- server/csharp/MetaObjects/DataConverter.cs | 10 +- .../MetaObjects/Loader/MetaDataLoader.cs | 2 +- .../Loader/ValidationPasses.Reporting.cs | 67 ++- server/csharp/MetaObjects/NamingRefs.cs | 25 +- server/csharp/MetaObjects/Parser.cs | 2 +- server/csharp/README.md | 9 +- server/java/MIGRATION.md | 6 + server/java/README.md | 25 +- server/java/RELEASE_NOTES.md | 11 +- .../direct/object/BaseObjectCodeWriter.java | 34 +- .../object/javacode/PojoAwareCodeWriter.java | 2 +- .../javacode/ValueObjectCodeWriter.java | 2 +- .../exposed1x/Exposed1xCodegenCompileTest.kt | 22 +- server/java/codegen-kotlin/pom.xml | 1 + .../generator/kotlin/GeneratorRegistry.kt | 7 +- .../kotlin/KotlinDecimalWireEmitter.kt | 155 +++++++ .../generator/kotlin/KotlinEntityGenerator.kt | 36 +- .../kotlin/KotlinExposedTableGenerator.kt | 118 ++++- .../kotlin/KotlinFilterAllowlistGenerator.kt | 10 +- .../kotlin/KotlinSpringControllerGenerator.kt | 96 +++- .../generator/kotlin/KotlinTypeMapper.kt | 45 +- ...tlinAbstractFieldExtendsInheritanceTest.kt | 2 +- .../generator/kotlin/KotlinDecimalWireTest.kt | 258 +++++++++++ .../kotlin/KotlinExposedTableGeneratorTest.kt | 58 ++- .../kotlin/KotlinReportRestSurfaceTest.kt | 65 ++- .../kotlin/KotlinReportTableGeneratorTest.kt | 64 +-- .../generator/kotlin/KotlinTypeMapperTest.kt | 4 +- .../kotlin/KotlinUuidPrimaryKeyCodegenTest.kt | 8 +- .../acme/blog/AuthorController.kt | 5 + .../acme/blog/MetaDecimalWire.kt | 124 ++++++ .../acme/catalog/MetaDecimalWire.kt | 124 ++++++ .../acme/catalog/MetaPortableColumnTypes.kt | 51 +++ .../acme/catalog/Product.kt | 4 + .../acme/catalog/ProductController.kt | 11 +- .../acme/catalog/ProductTable.kt | 2 +- .../entity-with-tph/acme/auth/Auth.kt | 4 + .../acme/auth/AuthController.kt | 32 +- .../acme/auth/AuthFilterAllowlist.kt | 2 + .../acme/auth/AuthLineController.kt | 5 + .../entity-with-tph/acme/auth/AuthTable.kt | 2 +- .../acme/auth/MetaDecimalWire.kt | 124 ++++++ .../acme/auth/MetaPortableColumnTypes.kt | 51 +++ .../acme/report/MetaDecimalWire.kt | 124 ++++++ server/java/codegen-spring/pom.xml | 11 +- .../generator/GeneratorRegistry.java | 10 +- .../spring/SpringControllerGenerator.java | 26 +- .../generator/spring/SpringDtoGenerator.java | 82 +++- .../SpringFilterAllowlistGenerator.java | 39 +- .../spring/SpringGeneratorSupport.java | 62 +++ .../spring/SpringValueObjectGenerator.java | 19 +- .../spring/runtime/CanonicalDecimal.java | 143 ++++++ .../runtime/CanonicalDecimalDeserializer.java | 89 ++++ .../runtime/CanonicalDecimalSerializer.java | 29 ++ .../spring/runtime/DeclaredDecimal.java | 29 ++ .../spring/runtime/FilterParser.java | 34 +- .../spring/runtime/FilterPredicate.java | 3 +- .../spring/runtime/UnreadableBody.java | 39 ++ .../SpringDecimalWireCompileRunTest.java | 171 +++++++ .../SpringFilterAllowlistGeneratorTest.java | 37 ++ .../spring/SpringReportRestSurfaceTest.java | 52 ++- .../spring/runtime/DecimalWireTest.java | 150 +++++++ .../runtime/FilterParserDecimalTest.java | 55 +++ .../spring/DatabaseDriverResolver.java | 8 +- .../spring/MetaDataLoaderConfiguration.java | 57 ++- .../spring/ObjectManagerProperties.java | 2 +- .../spring/DatabaseDriverResolverTest.java | 5 +- ...oaderConfigurationNamingDiscoveryTest.java | 107 +++++ .../SpringRuntimeLiteralNamingTest.java | 96 ++++ server/java/integration-tests-kotlin/pom.xml | 8 + .../KotlinCodegenMatchesReferenceTest.kt | 56 ++- .../kotlin/QueryScenarioConformanceTest.kt | 2 +- .../kotlin/QueryScenarioMySqlTest.kt | 2 +- .../integration/kotlin/QueryScenarioRunner.kt | 33 +- .../kotlin/QueryScenarioSqliteTest.kt | 2 +- .../integration/kotlin/ScenarioEngine.kt | 2 +- .../integration/kotlin/ScenarioLoader.kt | 14 + .../integration/kotlin/Scenarios.kt | 11 +- .../kotlin/api/ApiContractAssertions.kt | 12 +- .../kotlin/api/ApiContractConformanceTest.kt | 2 +- .../kotlin/api/ApiContractScenarioLoader.kt | 13 + .../kotlin/api/ApiContractScenarios.kt | 7 +- .../integration/kotlin/api/TomcatHost.kt | 6 +- ...imalGeneratedApiContractConformanceTest.kt | 114 +++++ .../ApiContractGeneratedConformanceTest.kt | 2 +- .../jsonb/JsonbApiContractConformanceTest.kt | 2 +- ...sonbGeneratedApiContractConformanceTest.kt | 2 +- .../api/m2m/M2mApiContractConformanceTest.kt | 2 +- ...tionGeneratedApiContractConformanceTest.kt | 2 +- ...portGeneratedApiContractConformanceTest.kt | 61 +-- .../GeneratedReportControllerHarness.kt | 154 ++++--- .../api/tph/TphFullSuiteCompilesTest.kt | 2 +- .../TphGeneratedApiContractConformanceTest.kt | 2 +- ...oughGeneratedApiContractConformanceTest.kt | 2 +- .../spring/ClasspathNamingDiscoveryTest.kt | 84 ++++ .../kotlin/tables/AllTypesBreakdownView.kt | 2 +- .../kotlin/tables/AllTypesTable.kt | 2 +- .../integration/kotlin/tables/AuthTable.kt | 2 +- .../integration/kotlin/tables/EntryTable.kt | 19 + .../kotlin/tables/EntryTotalsView.kt | 21 + .../kotlin/tables/FitnessTotalsFilledView.kt | 2 +- .../kotlin/tables/FitnessTotalsView.kt | 2 +- .../kotlin/tables/GadgetEventsByLabelView.kt | 15 + .../tables/MeasurementStatsFilledView.kt | 4 +- .../kotlin/tables/MeasurementStatsView.kt | 6 +- .../kotlin/tables/MeasurementTable.kt | 2 +- .../tables/PortableDecimalColumnType.kt | 31 ++ .../kotlin/tables/ProgramMinutesView.kt | 4 +- .../kotlin/tables/ProgramRosterView.kt | 4 +- .../kotlin/tables/ProgramStatView.kt | 2 +- .../kotlin/tables/ProgramsEmptyScopeView.kt | 2 +- .../kotlin/tables/ProgramsOverTimeView.kt | 2 +- .../kotlin/tables/ReferenceDecimal.kt | 28 ++ .../integration/kotlin/tables/ReportViews.kt | 3 + server/java/integration-tests/pom.xml | 9 + .../LiteralNamingDiscoveryTest.java | 126 ++++++ .../integration/ObjectManagerDbAdapter.java | 16 + .../integration/QueryScenarioMySqlTests.java | 4 +- .../integration/QueryScenarioRunner.java | 7 +- .../integration/QueryScenarioSqliteTests.java | 59 +++ .../integration/QueryScenarioTests.java | 7 +- .../integration/RoundtripWriter.java | 25 ++ .../integration/ScenarioEngine.java | 24 +- .../integration/ScenarioLoader.java | 21 + .../metaobjects/integration/Scenarios.java | 1 + .../integration/SqliteDatabase.java | 36 ++ .../integration/SqliteDecimalReadTest.java | 118 +++++ .../integration/SqliteObjectManagerTest.java | 195 ++++++++ .../api/ApiContractAssertions.java | 14 +- ...ApiContractAssertionsExactNumbersTest.java | 46 ++ .../api/ApiContractConformanceTest.java | 2 +- .../api/ApiContractScenarioLoader.java | 24 + .../integration/api/ApiContractScenarios.java | 1 + .../integration/api/DecimalCorpus.java | 53 +++ .../api/DecimalOmdbApiContractLane.java | 185 ++++++++ ...alOmdbMySqlApiContractConformanceTest.java | 24 + ...mdbPostgresApiContractConformanceTest.java | 24 + ...lOmdbSqliteApiContractConformanceTest.java | 24 + .../api/JsonbApiContractConformanceTest.java | 2 +- .../integration/api/ReportCorpus.java | 50 +-- ...rtGeneratedApiContractConformanceTest.java | 84 ---- .../api/ReportOmdbApiContractLane.java | 185 ++++++++ ...rtOmdbMySqlApiContractConformanceTest.java | 24 + ...mdbPostgresApiContractConformanceTest.java | 24 + ...tOmdbSqliteApiContractConformanceTest.java | 24 + .../integration/api/TomcatHost.java | 6 +- .../GeneratedDecimalControllerHarness.java | 194 ++++++++ .../GeneratedReportControllerHarness.java | 76 ++-- .../InMemoryProjectionRepositorySource.java | 9 +- .../InMemoryReportRepositorySource.java | 200 --------- .../generated/OmdbEntryRepositorySource.java | 215 +++++++++ .../generated/OmdbReportRepositorySource.java | 194 ++++++++ .../m2m/M2mApiContractConformanceTest.java | 2 +- .../metaobjects/mojo/MetaDataEjectMojo.java | 86 +++- .../mojo/EjectRuntimeRoundTripTest.java | 46 +- .../metaobjects/mojo/EjectSupportTest.java | 7 +- .../EntityExtractorEjectRoundTripTest.java | 6 +- .../mojo/MetaDataEjectMojoTest.java | 17 + .../RequirementTestsGeneratorMojoTest.java | 97 +++- .../com/metaobjects/attr/IntAttribute.java | 19 +- .../com/metaobjects/attr/LongAttribute.java | 17 + .../metaobjects/config/SourceResolver.java | 25 ++ .../metaobjects/loader/MetaDataLoader.java | 41 +- .../loader/ReportingValidation.java | 77 +++- .../metaobjects/loader/ValidationPhase.java | 36 +- .../parser/json/CanonicalJsonParser.java | 7 +- .../reporting/ReportReadModel.java | 5 +- .../metaobjects/config/NeutralConfigTest.java | 7 + .../conformance/ConformanceTest.java | 14 +- .../metaobjects/conformance/FixtureLint.java | 23 +- .../loader/FromUrisPhysicalNamingTest.java | 116 +++++ .../reporting/ReportReadModelTest.java | 17 +- server/java/omdb/pom.xml | 7 + .../manager/db/driver/GenericSQLDriver.java | 23 +- .../manager/db/driver/MySQLDriver.java | 10 + .../manager/db/driver/SQLiteDriver.java | 280 ++++++++++++ .../db/driver/SQLiteDriverAutoIdTest.java | 47 ++ .../db/driver/SQLiteDriverPagingTest.java | 50 +++ .../driver/SQLiteDriverValueBindingTest.java | 69 +++ server/java/pom.xml | 6 +- server/python/README.md | 14 +- server/python/pyproject.toml | 12 +- server/python/src/metaobjects/cli.py | 26 +- .../python/src/metaobjects/codegen/eject.py | 35 +- .../generators/filter_allowlist_generator.py | 7 + .../codegen/generators/router_generator.py | 166 +++++-- .../metaobjects/codegen/runtime/__init__.py | 2 +- .../codegen/runtime/decimal_wire.py | 184 ++++++++ .../src/metaobjects/config/neutral_config.py | 12 + .../metaobjects/loader/meta_data_loader.py | 69 ++- .../metaobjects/loader/validate_reporting.py | 65 ++- .../metaobjects/meta/core/attr/meta_attr.py | 8 + .../python/src/metaobjects/meta/meta_data.py | 5 + server/python/src/metaobjects/naming_refs.py | 34 +- server/python/src/metaobjects/parser.py | 4 +- .../src/metaobjects/requirement_check.py | 6 +- .../src/metaobjects/runtime/__init__.py | 11 +- .../python/src/metaobjects/runtime/dialect.py | 298 +++++++++++++ .../python/src/metaobjects/runtime/drivers.py | 44 ++ .../src/metaobjects/runtime/object_manager.py | 349 ++++++++++----- .../python/src/metaobjects/super_resolve.py | 5 +- .../codegen/test_cli_help_removed_flags.py | 15 + .../python/tests/codegen/test_eject_alone.py | 2 +- .../tests/codegen/test_eject_runtime.py | 12 +- .../tests/codegen/test_project_import.py | 23 + .../tests/codegen/test_report_router.py | 29 ++ .../tests/codegen/test_router_generator.py | 7 +- .../test_router_generator_m2m_target_tph.py | 2 +- .../codegen/test_router_generator_tph_m2m.py | 6 +- .../test_router_generator_tph_m2m_shadowed.py | 6 +- .../python/tests/codegen/test_tph_codegen.py | 2 +- .../config/test_neutral_config_unknown_key.py | 22 + .../tests/conformance/conformance_adapter.py | 9 +- .../tests/conformance/test_conformance.py | 6 + .../integration/api_contract_assertions.py | 26 +- server/python/tests/integration/conftest.py | 22 + server/python/tests/integration/engines.py | 87 ++++ .../integration/generated_projection_app.py | 30 +- .../tests/integration/generated_report_app.py | 257 ++++++----- .../tests/integration/generated_router_app.py | 3 + .../tests/integration/generated_tph_app.py | 23 +- .../tests/integration/mysql_container.py | 176 ++++++++ .../python/tests/integration/normalization.py | 25 +- .../python/tests/integration/query_runner.py | 112 ++++- .../python/tests/integration/report_lane.py | 131 ++++++ server/python/tests/integration/scenarios.py | 27 ++ .../tests/integration/test_api_contract.py | 2 +- .../integration/test_api_contract_decimal.py | 139 ++++++ .../test_api_contract_generated.py | 2 +- .../integration/test_api_contract_jsonb.py | 2 +- .../integration/test_api_contract_m2m.py | 2 +- .../test_api_contract_m2m_generated.py | 2 +- .../test_api_contract_projection.py | 2 +- .../integration/test_api_contract_report.py | 107 +++-- .../test_api_contract_tph_generated.py | 2 +- .../test_generated_decimal_wire.py | 192 ++++++++ .../tests/integration/test_normalization.py | 12 + .../tests/integration/test_query_scenarios.py | 55 ++- .../tests/loader/test_naming_discovery.py | 104 +++++ .../python/tests/runtime/test_decimal_wire.py | 132 ++++++ .../runtime/test_literal_naming_sqlite.py | 57 +++ .../runtime/test_object_manager_dialects.py | 417 ++++++++++++++++++ server/python/tests/test_report_shape.py | 4 +- ...test_reporting_validation_spine_default.py | 14 +- server/python/uv.lock | 17 +- server/typescript/packages/cli/README.md | 4 +- .../packages/cli/src/commands/eject.ts | 41 +- .../packages/cli/src/commands/init.ts | 6 +- .../packages/cli/src/commands/migrate.ts | 81 ++-- .../packages/cli/src/commands/verify.ts | 2 +- .../cli/src/lib/agent-context-staleness.ts | 14 +- .../cli/src/lib/decimal-precision-advisory.ts | 50 +++ .../packages/cli/src/lib/docs-removed.ts | 3 +- .../cli/src/lib/load-metaobjects-config.ts | 3 +- .../packages/cli/src/lib/requirement-check.ts | 11 +- .../cli/test/eject-template-handover.test.ts | 44 +- .../cli/test/gen-docs-generators.test.ts | 11 +- .../test/migrate-decimal-precision.test.ts | 71 +++ .../cli/test/migrate-dry-run-no-slug.test.ts | 80 ++++ .../cli/test/migrate-emit-notices.test.ts | 70 +++ .../test/migrate-load-failure-exit.test.ts | 34 ++ .../verify-requirements-scope-no-deps.test.ts | 29 +- .../src/templates/form-file.ts | 23 +- .../test/nested-value-object-form.test.ts | 12 + .../packages/codegen-ts/src/api-surface.ts | 35 +- .../codegen-ts/src/cube/build-cube-model.ts | 4 +- .../codegen-ts/src/cube/cube-members.ts | 9 +- .../codegen-ts/src/cube/cube-reports.ts | 40 +- .../src/generators/agent-docs-file.ts | 3 +- .../src/generators/agent-ui-page.ts | 6 +- .../packages/codegen-ts/src/index.ts | 4 +- .../codegen-ts/src/metaobjects-config.ts | 3 +- .../codegen-ts/src/reference/agent-docs.ts | 3 +- .../codegen-ts/src/reference/routes-hono.ts | 9 +- .../codegen-ts/src/reference/routes.ts | 17 +- .../codegen-ts/src/requirement-walk.ts | 3 +- .../src/templates/decimal-wire-helper.ts | 47 ++ .../src/templates/entity-constants.ts | 2 + .../src/templates/entity-ui-descriptor.ts | 18 +- .../codegen-ts/src/templates/field-meta.ts | 36 ++ .../src/templates/filter-allowlist.ts | 1 + .../src/templates/routes-file-hono.ts | 8 +- .../codegen-ts/src/templates/routes-file.ts | 16 +- .../src/templates/zod-validators.ts | 34 +- .../test/agent-ui-html-type-cell.test.ts | 82 ++++ .../test/cube/build-cube-model.test.ts | 24 +- .../test/cube/cube-model-canonical.test.ts | 3 + .../codegen-ts/test/cube/cube-yaml.test.ts | 27 +- .../test/cube/report-rollups.test.ts | 20 +- .../codegen-ts/test/decimal-wire-e2e.test.ts | 119 +++++ .../test/decimal-write-schema.test.ts | 72 +++ .../test/form-fractional-input.test.ts | 76 ++++ .../test/metaobjects-config.test.ts | 5 +- .../test/physical-naming-refusals.test.ts | 5 +- .../projection/build-projection-views.test.ts | 11 - .../report-via-redeclared-field.test.ts | 76 ++++ .../test/projection/routes-file.test.ts | 29 +- .../test/reference-byte-identical.test.ts | 5 +- .../test/templates/tph-tier2-routes.test.ts | 9 + .../packages/conformance/src/adapter.ts | 6 + .../conformance/src/operation-script.ts | 20 +- .../packages/conformance/src/runner.ts | 9 + .../packages/conformance/test/runner.test.ts | 50 +++ .../cube-live/cube-live-support.ts | 195 +++++++- .../cube-live/cube-model-mysql-cases.live.ts | 388 ++++++++++++++++ .../cube-live/cube-model-mysql.live.ts | 57 +-- .../cube-live/cube-model-production.live.ts | 338 ++++++++++++++ .../cube-live/cube-model-shapes.live.ts | 357 +++++++++++++++ .../cube-live/cube-model.live.ts | 66 +-- .../cube-live/cube-prod-stack.ts | 151 +++++++ .../packages/integration-tests/package.json | 3 +- .../src/api-contract-decimal-schema.ts | 37 ++ .../src/api-contract-decimal-server.ts | 201 +++++++++ .../api-contract-report-generated-server.ts | 10 + .../src/api-contract-report-mysql-schema.ts | 5 + .../src/api-contract-report-mysql-server.ts | 8 + .../src/api-contract-report-sqlite-server.ts | 9 + .../src/api-contract-scenario.ts | 44 +- .../src/api-contract-tph-server.ts | 29 +- .../src/canonical-schema-mysql.ts | 15 + .../integration-tests/src/engine-wire.ts | 55 ++- .../src/gen-api-contract-decimal-schema.ts | 28 ++ .../packages/integration-tests/src/paths.ts | 14 +- .../integration-tests/src/query-scenario.ts | 49 +- .../integration-tests/src/scenario.ts | 31 +- .../test/api-contract-decimal.test.ts | 83 ++++ .../test/api-contract-generated.test.ts | 4 +- .../test/api-contract-jsonb.test.ts | 4 +- .../test/api-contract-m2m.test.ts | 4 +- .../test/api-contract-projection.test.ts | 4 +- .../test/api-contract-report-corpus.test.ts | 14 +- .../test/api-contract-report.test.ts | 4 +- .../test/api-contract-scenario.test.ts | 34 ++ .../test/api-contract-tph.test.ts | 4 +- .../test/api-contract-write-through.test.ts | 4 +- .../test/api-contract.test.ts | 4 +- .../test/engine-wire.test.ts | 14 +- .../test/mysql-generated-app.test.ts | 2 +- .../integration-tests/test/query-d1.test.ts | 4 +- .../test/query-mysql.test.ts | 4 +- .../test/query-sqlite.test.ts | 4 +- .../integration-tests/test/query.test.ts | 4 +- .../test/report-shapes-artifact.test.ts | 14 + .../test/report-views-pg.test.ts | 3 +- .../test/runtime-return-types.test.ts | 9 +- .../test/view-lifecycle-pg.test.ts | 3 +- .../test/view-recreate-drift-pg.test.ts | 3 +- .../test/view-recreate-grants-pg.test.ts | 146 ++++++ .../test/view-replace-probe-pg.test.ts | 3 +- .../typescript/packages/metadata/package.json | 5 + .../metadata/src/core/attr/meta-attr.ts | 9 +- .../src/core/reporting/report-describe.ts | 14 +- .../src/core/reporting/report-read-model.ts | 18 +- .../src/core/requirement/resolve-claim.ts | 20 + .../packages/metadata/src/docs-ref.ts | 41 ++ .../typescript/packages/metadata/src/index.ts | 9 +- .../metadata/src/loader/meta-data-loader.ts | 3 +- .../src/loader/reporting-validation.ts | 95 +++- .../packages/metadata/src/naming-refs.ts | 24 +- .../packages/metadata/src/naming.ts | 15 +- .../packages/metadata/src/release-docs.ts | 48 ++ .../metadata/test/conformance/adapter.ts | 4 +- .../packages/metadata/test/docs-ref.test.ts | 29 ++ .../test/report-array-map-dimension.test.ts | 47 ++ .../metadata/test/report-describe.test.ts | 10 +- .../metadata/test/reporting-refusals.test.ts | 86 ++++ .../test/reporting-validation.test.ts | 74 ++-- .../test/unresolved-super-hint.test.ts | 31 ++ .../packages/migrate-ts/src/emit/postgres.ts | 165 ++++++- .../packages/migrate-ts/src/index.ts | 2 + .../packages/migrate-ts/src/types.ts | 6 + ...emit-postgres-view-grants-recreate.test.ts | 77 ++++ .../test/emit-postgres-view-grants.test.ts | 44 ++ .../integration/pg-adopt-view-239.test.ts | 3 +- .../packages/runtime-ts/src/decimal-wire.ts | 143 +++++- .../src/drizzle-fastify/filter-allowlist.ts | 6 +- .../src/drizzle-fastify/filter-parser.ts | 31 +- .../drizzle-fastify/filter-value-format.ts | 4 + .../runtime-ts/src/drizzle-fastify/index.ts | 14 +- .../src/drizzle-fastify/mount-m2m.ts | 5 +- .../src/drizzle-fastify/mount-read-only.ts | 15 +- .../packages/runtime-ts/src/hono/index.ts | 14 +- .../runtime-ts/src/hono/mount-read-only.ts | 15 +- .../packages/runtime-ts/src/object-manager.ts | 3 +- .../packages/runtime-ts/src/type-coercer.ts | 51 ++- .../test/decimal-wire-vectors.test.ts | 80 ++++ .../runtime-ts/test/decimal-wire.test.ts | 11 +- .../test/filter-parser-decimal.test.ts | 52 +++ .../test/object-manager-naming.test.ts | 5 +- .../test/object-manager-report.test.ts | 4 +- .../sdk/src/agent-context/assemble.ts | 15 +- .../sdk/src/agent-context/scaffold.ts | 91 +++- .../typescript/packages/sdk/src/collection.ts | 18 +- server/typescript/packages/sdk/src/config.ts | 55 ++- server/typescript/packages/sdk/src/index.ts | 2 +- .../sdk/test/agent-context/assemble.test.ts | 14 + .../sdk/test/agent-context/scaffold.test.ts | 66 ++- .../packages/sdk/test/config.test.ts | 41 +- .../test-generators/src/generators/routes.ts | 17 +- .../src/templates/decimal-wire-helper.ts | 47 ++ .../src/templates/entity-constants.ts | 2 + .../src/templates/field-meta.ts | 36 ++ .../src/templates/filter-allowlist.ts | 1 + .../src/templates/zod-validators.ts | 34 +- spec/conformance-tests.md | 4 +- .../ADR-0019-runtime-return-type-contract.md | 72 +++ spec/roadmap.md | 4 +- tsconfig.scripts.json | 4 +- 681 files changed, 23885 insertions(+), 2453 deletions(-) create mode 100644 client/web/packages/react/test/use-entity-form-fractional.test.tsx create mode 100644 docs/features/migrations/upgrading-1.0-to-1.1.md create mode 100644 examples/advanced-modeling/codegen/templates/decimal-wire-helper.ts create mode 100644 examples/showcase/codegen/templates/decimal-wire-helper.ts create mode 100644 fixtures/api-contract-conformance/decimal/README.md create mode 100644 fixtures/api-contract-conformance/decimal/meta.json create mode 100644 fixtures/api-contract-conformance/decimal/scenarios/decimal-filter-sort-exact-15.yaml create mode 100644 fixtures/api-contract-conformance/decimal/scenarios/decimal-filter-sort-exact.yaml create mode 100644 fixtures/api-contract-conformance/decimal/scenarios/decimal-read-exact-15.yaml create mode 100644 fixtures/api-contract-conformance/decimal/scenarios/decimal-read-exact.yaml create mode 100644 fixtures/api-contract-conformance/decimal/scenarios/decimal-report-exact-15.yaml create mode 100644 fixtures/api-contract-conformance/decimal/scenarios/decimal-report-exact.yaml create mode 100644 fixtures/api-contract-conformance/decimal/scenarios/decimal-write-exact-15.yaml create mode 100644 fixtures/api-contract-conformance/decimal/scenarios/decimal-write-exact.yaml create mode 100644 fixtures/api-contract-conformance/decimal/schema.mysql.sql create mode 100644 fixtures/api-contract-conformance/decimal/schema.postgres.sql create mode 100644 fixtures/api-contract-conformance/decimal/schema.sqlite.sql create mode 100644 fixtures/api-contract-conformance/decimal/seed.json create mode 100644 fixtures/api-contract-conformance/report/scenarios/filter-array-and-map-dimension.yaml create mode 100644 fixtures/api-contract-conformance/report/scenarios/sort-array-and-map-dimension.yaml create mode 100644 fixtures/conformance/attr-int-integral-float/expected.json create mode 100644 fixtures/conformance/attr-int-integral-float/input/meta.shop.json create mode 100644 fixtures/conformance/attr-int-integral-float/providers.json create mode 100644 fixtures/conformance/error-attr-int-fractional/expected-errors.json create mode 100644 fixtures/conformance/error-attr-int-fractional/input/meta.shop.json create mode 100644 fixtures/conformance/error-attr-int-fractional/providers.json create mode 100644 fixtures/conformance/error-dimension-over-object/expected-errors.json create mode 100644 fixtures/conformance/error-dimension-over-object/input/meta.events.json create mode 100644 fixtures/conformance/error-dimension-over-object/providers.json create mode 100644 fixtures/conformance/error-dimension-via-base-lacks-field/expected-errors.json create mode 100644 fixtures/conformance/error-dimension-via-base-lacks-field/input/meta.shop.json create mode 100644 fixtures/conformance/error-dimension-via-base-lacks-field/providers.json create mode 100644 fixtures/conformance/error-measure-default-on-count-integral-float/expected-errors.json create mode 100644 fixtures/conformance/error-measure-default-on-count-integral-float/input/meta.shop.json create mode 100644 fixtures/conformance/error-measure-default-on-count-integral-float/providers.json create mode 100644 fixtures/conformance/error-measure-default-on-count-non-numeric/expected-errors.json create mode 100644 fixtures/conformance/error-measure-default-on-count-non-numeric/input/meta.shop.json create mode 100644 fixtures/conformance/error-measure-default-on-count-non-numeric/providers.json create mode 100644 fixtures/conformance/error-measure-over-object/expected-errors.json create mode 100644 fixtures/conformance/error-measure-over-object/input/meta.events.json create mode 100644 fixtures/conformance/error-measure-over-object/providers.json create mode 100644 fixtures/conformance/error-report-abstract/expected-errors.json create mode 100644 fixtures/conformance/error-report-abstract/input/meta.events.json create mode 100644 fixtures/conformance/error-report-abstract/providers.json create mode 100644 fixtures/conformance/error-report-source-kind-materialized-view/expected-errors.json create mode 100644 fixtures/conformance/error-report-source-kind-materialized-view/input/meta.events.json create mode 100644 fixtures/conformance/error-report-source-kind-materialized-view/providers.json create mode 100644 fixtures/conformance/error-report-source-kind-stored-proc/expected-errors.json create mode 100644 fixtures/conformance/error-report-source-kind-stored-proc/input/meta.events.json create mode 100644 fixtures/conformance/error-report-source-kind-stored-proc/providers.json create mode 100644 fixtures/conformance/error-report-source-kind-table-function/expected-errors.json create mode 100644 fixtures/conformance/error-report-source-kind-table-function/input/meta.events.json create mode 100644 fixtures/conformance/error-report-source-kind-table-function/providers.json create mode 100644 fixtures/conformance/error-requirement-level-fractional/expected-errors.json create mode 100644 fixtures/conformance/error-requirement-level-fractional/input/meta.requirements.json create mode 100644 fixtures/conformance/error-requirement-level-fractional/providers.json create mode 100644 fixtures/conformance/measure-default-integral-float/expected.json create mode 100644 fixtures/conformance/measure-default-integral-float/input/meta.shop.json create mode 100644 fixtures/conformance/measure-default-integral-float/providers.json create mode 100644 fixtures/conformance/overlay-explicit-package-not-applied-to-target/expected.json create mode 100644 fixtures/conformance/overlay-explicit-package-not-applied-to-target/input/meta.a-order.json create mode 100644 fixtures/conformance/overlay-explicit-package-not-applied-to-target/input/meta.b-requirement.json create mode 100644 fixtures/conformance/overlay-explicit-package-not-applied-to-target/input/meta.c-overlay.json create mode 100644 fixtures/conformance/overlay-explicit-package-not-applied-to-target/providers.json create mode 100644 fixtures/conformance/requirement-level-integral-float/expected.json create mode 100644 fixtures/conformance/requirement-level-integral-float/input/meta.requirements.json create mode 100644 fixtures/conformance/requirement-level-integral-float/providers.json create mode 100644 fixtures/cube-model/canonical/expected/model/cubes/Entry.yml create mode 100644 fixtures/cube-model/canonical/expected/model/cubes/Gadget.yml create mode 100644 fixtures/cube-model/canonical/expected/model/cubes/GadgetEvent.yml create mode 100644 fixtures/decimal-wire/README.md create mode 100644 fixtures/decimal-wire/vectors.json create mode 100644 fixtures/persistence-conformance/queries/decimal-exact-15.yaml create mode 100644 fixtures/persistence-conformance/queries/decimal-exact.yaml create mode 100644 fixtures/persistence-conformance/queries/report-decimal-exact-15.yaml create mode 100644 fixtures/persistence-conformance/queries/report-decimal-exact.yaml create mode 100644 fixtures/persistence-conformance/queries/report-via-redeclared-field.yaml create mode 100644 fixtures/requirement-check-conformance/package-overlay-child-from-other-package-file/expected.json create mode 100644 fixtures/requirement-check-conformance/package-overlay-child-from-other-package-file/input/meta.a-shop.json create mode 100644 fixtures/requirement-check-conformance/package-overlay-child-from-other-package-file/input/meta.b-overlay.json create mode 100644 fixtures/requirement-check-conformance/package-overlay-child-from-packageless-file/expected.json create mode 100644 fixtures/requirement-check-conformance/package-overlay-child-from-packageless-file/input/meta.a-shop.json create mode 100644 fixtures/requirement-check-conformance/package-overlay-child-from-packageless-file/input/meta.b-overlay.json create mode 100644 fixtures/requirement-check-conformance/package-packageless-file-beside-packaged/expected.json create mode 100644 fixtures/requirement-check-conformance/package-packageless-file-beside-packaged/input/meta.a-shop.json create mode 100644 fixtures/requirement-check-conformance/package-packageless-file-beside-packaged/input/meta.b-requirements.json create mode 100644 scripts/check-release-tag.mjs create mode 100644 scripts/lib/changelog-promote.mjs create mode 100644 scripts/test-changelog-promote.mjs create mode 100644 scripts/test-check-release-tag.mjs create mode 100644 server/csharp/MetaObjects.Codegen.Tests/DecimalWireTests.cs create mode 100644 server/csharp/MetaObjects.Codegen/Runtime/DecimalRangeEndpointFilter.cs create mode 100644 server/csharp/MetaObjects.Codegen/Runtime/DecimalWire.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Api/ApiContractAssertionsExactnessTests.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Api/ApiContractDecimalConformanceTest.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Api/DecimalArrayReportRowTests.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Api/DecimalWireGeneratedApiTests.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Api/EntityArraySqliteTests.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Api/GeneratedRoutesServer.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Generated/DecimalWire.g.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Generated/Entry.g.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Generated/EntryNames.g.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Generated/EntryTotals.g.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Generated/Gadget.g.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEvent.g.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEventNames.g.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEventQueries.g.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEventsByLabel.g.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Generated/GadgetNames.g.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Generated/LabeledNames.g.cs create mode 100644 server/csharp/MetaObjects.IntegrationTests/Runner/DecimalGate.cs create mode 100644 server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinDecimalWireEmitter.kt create mode 100644 server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinDecimalWireTest.kt create mode 100644 server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-controller/acme/blog/MetaDecimalWire.kt create mode 100644 server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/MetaDecimalWire.kt create mode 100644 server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/MetaPortableColumnTypes.kt create mode 100644 server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/MetaDecimalWire.kt create mode 100644 server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/MetaPortableColumnTypes.kt create mode 100644 server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-view/acme/report/MetaDecimalWire.kt create mode 100644 server/java/codegen-spring/src/main/java/com/metaobjects/generator/spring/runtime/CanonicalDecimal.java create mode 100644 server/java/codegen-spring/src/main/java/com/metaobjects/generator/spring/runtime/CanonicalDecimalDeserializer.java create mode 100644 server/java/codegen-spring/src/main/java/com/metaobjects/generator/spring/runtime/CanonicalDecimalSerializer.java create mode 100644 server/java/codegen-spring/src/main/java/com/metaobjects/generator/spring/runtime/DeclaredDecimal.java create mode 100644 server/java/codegen-spring/src/main/java/com/metaobjects/generator/spring/runtime/UnreadableBody.java create mode 100644 server/java/codegen-spring/src/test/java/com/metaobjects/generator/spring/SpringDecimalWireCompileRunTest.java create mode 100644 server/java/codegen-spring/src/test/java/com/metaobjects/generator/spring/runtime/DecimalWireTest.java create mode 100644 server/java/codegen-spring/src/test/java/com/metaobjects/generator/spring/runtime/FilterParserDecimalTest.java create mode 100644 server/java/core-spring/src/test/java/com/metaobjects/spring/MetaDataLoaderConfigurationNamingDiscoveryTest.java create mode 100644 server/java/core-spring/src/test/java/com/metaobjects/spring/SpringRuntimeLiteralNamingTest.java create mode 100644 server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/decimal/DecimalGeneratedApiContractConformanceTest.kt create mode 100644 server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/spring/ClasspathNamingDiscoveryTest.kt create mode 100644 server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/EntryTable.kt create mode 100644 server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/EntryTotalsView.kt create mode 100644 server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/GadgetEventsByLabelView.kt create mode 100644 server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/PortableDecimalColumnType.kt create mode 100644 server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ReferenceDecimal.kt create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/LiteralNamingDiscoveryTest.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioSqliteTests.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/SqliteDatabase.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/SqliteDecimalReadTest.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/SqliteObjectManagerTest.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractAssertionsExactNumbersTest.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalCorpus.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbApiContractLane.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbMySqlApiContractConformanceTest.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbPostgresApiContractConformanceTest.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbSqliteApiContractConformanceTest.java delete mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportGeneratedApiContractConformanceTest.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbApiContractLane.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbMySqlApiContractConformanceTest.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbPostgresApiContractConformanceTest.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbSqliteApiContractConformanceTest.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/GeneratedDecimalControllerHarness.java delete mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/InMemoryReportRepositorySource.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/OmdbEntryRepositorySource.java create mode 100644 server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/OmdbReportRepositorySource.java create mode 100644 server/java/metadata/src/test/java/com/metaobjects/loader/FromUrisPhysicalNamingTest.java create mode 100644 server/java/omdb/src/main/java/com/metaobjects/manager/db/driver/SQLiteDriver.java create mode 100644 server/java/omdb/src/test/java/com/metaobjects/manager/db/driver/SQLiteDriverAutoIdTest.java create mode 100644 server/java/omdb/src/test/java/com/metaobjects/manager/db/driver/SQLiteDriverPagingTest.java create mode 100644 server/java/omdb/src/test/java/com/metaobjects/manager/db/driver/SQLiteDriverValueBindingTest.java create mode 100644 server/python/src/metaobjects/codegen/runtime/decimal_wire.py create mode 100644 server/python/src/metaobjects/runtime/dialect.py create mode 100644 server/python/src/metaobjects/runtime/drivers.py create mode 100644 server/python/tests/codegen/test_cli_help_removed_flags.py create mode 100644 server/python/tests/config/test_neutral_config_unknown_key.py create mode 100644 server/python/tests/integration/conftest.py create mode 100644 server/python/tests/integration/engines.py create mode 100644 server/python/tests/integration/mysql_container.py create mode 100644 server/python/tests/integration/report_lane.py create mode 100644 server/python/tests/integration/test_api_contract_decimal.py create mode 100644 server/python/tests/integration/test_generated_decimal_wire.py create mode 100644 server/python/tests/loader/test_naming_discovery.py create mode 100644 server/python/tests/runtime/test_decimal_wire.py create mode 100644 server/python/tests/runtime/test_literal_naming_sqlite.py create mode 100644 server/python/tests/runtime/test_object_manager_dialects.py create mode 100644 server/typescript/packages/cli/src/lib/decimal-precision-advisory.ts create mode 100644 server/typescript/packages/cli/test/migrate-decimal-precision.test.ts create mode 100644 server/typescript/packages/cli/test/migrate-dry-run-no-slug.test.ts create mode 100644 server/typescript/packages/cli/test/migrate-emit-notices.test.ts create mode 100644 server/typescript/packages/cli/test/migrate-load-failure-exit.test.ts create mode 100644 server/typescript/packages/codegen-ts/src/templates/decimal-wire-helper.ts create mode 100644 server/typescript/packages/codegen-ts/test/agent-ui-html-type-cell.test.ts create mode 100644 server/typescript/packages/codegen-ts/test/decimal-wire-e2e.test.ts create mode 100644 server/typescript/packages/codegen-ts/test/decimal-write-schema.test.ts create mode 100644 server/typescript/packages/codegen-ts/test/form-fractional-input.test.ts create mode 100644 server/typescript/packages/codegen-ts/test/projection/report-via-redeclared-field.test.ts create mode 100644 server/typescript/packages/integration-tests/cube-live/cube-model-mysql-cases.live.ts create mode 100644 server/typescript/packages/integration-tests/cube-live/cube-model-production.live.ts create mode 100644 server/typescript/packages/integration-tests/cube-live/cube-model-shapes.live.ts create mode 100644 server/typescript/packages/integration-tests/cube-live/cube-prod-stack.ts create mode 100644 server/typescript/packages/integration-tests/src/api-contract-decimal-schema.ts create mode 100644 server/typescript/packages/integration-tests/src/api-contract-decimal-server.ts create mode 100644 server/typescript/packages/integration-tests/src/gen-api-contract-decimal-schema.ts create mode 100644 server/typescript/packages/integration-tests/test/api-contract-decimal.test.ts create mode 100644 server/typescript/packages/integration-tests/test/api-contract-scenario.test.ts create mode 100644 server/typescript/packages/integration-tests/test/view-recreate-grants-pg.test.ts create mode 100644 server/typescript/packages/metadata/src/docs-ref.ts create mode 100644 server/typescript/packages/metadata/src/release-docs.ts create mode 100644 server/typescript/packages/metadata/test/docs-ref.test.ts create mode 100644 server/typescript/packages/metadata/test/report-array-map-dimension.test.ts create mode 100644 server/typescript/packages/metadata/test/reporting-refusals.test.ts create mode 100644 server/typescript/packages/metadata/test/unresolved-super-hint.test.ts create mode 100644 server/typescript/packages/migrate-ts/test/emit-postgres-view-grants-recreate.test.ts create mode 100644 server/typescript/packages/migrate-ts/test/emit-postgres-view-grants.test.ts create mode 100644 server/typescript/packages/runtime-ts/test/decimal-wire-vectors.test.ts create mode 100644 server/typescript/packages/runtime-ts/test/filter-parser-decimal.test.ts create mode 100644 server/typescript/packages/test-generators/src/templates/decimal-wire-helper.ts diff --git a/.github/workflows/publish-csharp.yml b/.github/workflows/publish-csharp.yml index 9d7eee6eb..6145a8fa8 100644 --- a/.github/workflows/publish-csharp.yml +++ b/.github/workflows/publish-csharp.yml @@ -41,6 +41,13 @@ jobs: steps: - uses: actions/checkout@v4 + # The tag must name the version the manifest commits: these workflows publish the + # manifest's version whatever tag started them (scripts/check-release-tag.mjs). + - name: Tag matches the manifest version + env: + VERSION_INPUT: ${{ github.event.inputs.version }} + run: node scripts/check-release-tag.mjs csharp "$VERSION_INPUT" + # A final version is not published while the eject pending list has entries # (ADR-0034 Amendment 8). A release candidate passes, which is why the `version` # input is read first. @@ -75,9 +82,17 @@ jobs: with: user: ${{ secrets.NUGET_USER }} + # --skip-duplicate lets a re-run finish a push that failed part way. A run that pushed + # NOTHING, because every package was already on nuget.org, is a re-run of a published + # version and fails rather than reporting green. - name: NuGet push run: | + set -o pipefail dotnet nuget push "artifacts/*.nupkg" \ --api-key ${{ steps.login.outputs.NUGET_API_KEY }} \ --source https://api.nuget.org/v3/index.json \ - --skip-duplicate + --skip-duplicate | tee push.log + if ! grep -q "Your package was pushed" push.log; then + echo "::error::nothing was pushed: every package is already on nuget.org at this version" + exit 1 + fi diff --git a/.github/workflows/publish-java.yml b/.github/workflows/publish-java.yml index d52469fd6..7415377fb 100644 --- a/.github/workflows/publish-java.yml +++ b/.github/workflows/publish-java.yml @@ -33,6 +33,11 @@ jobs: steps: - uses: actions/checkout@v4 + # The tag must name the version the manifest commits: these workflows publish the + # manifest's version whatever tag started them (scripts/check-release-tag.mjs). + - name: Tag matches the manifest version + run: node scripts/check-release-tag.mjs java + # A final version is not published while the eject pending list has entries # (ADR-0034 Amendment 8). A release candidate passes. - name: Eject release gate diff --git a/.github/workflows/publish-npm.yml b/.github/workflows/publish-npm.yml index f0ebef191..8f3926814 100644 --- a/.github/workflows/publish-npm.yml +++ b/.github/workflows/publish-npm.yml @@ -39,6 +39,11 @@ jobs: steps: - uses: actions/checkout@v4 + # The tag must name the version the manifest commits: these workflows publish the + # manifest's version whatever tag started them (scripts/check-release-tag.mjs). + - name: Tag matches the manifest version + run: node scripts/check-release-tag.mjs npm + # A final version is not published while the eject pending list has entries # (ADR-0034 Amendment 8). A release candidate passes. - name: Eject release gate diff --git a/.github/workflows/publish-python.yml b/.github/workflows/publish-python.yml index 87c702f82..515571e05 100644 --- a/.github/workflows/publish-python.yml +++ b/.github/workflows/publish-python.yml @@ -28,6 +28,11 @@ jobs: steps: - uses: actions/checkout@v4 + # The tag must name the version the manifest commits: these workflows publish the + # manifest's version whatever tag started them (scripts/check-release-tag.mjs). + - name: Tag matches the manifest version + run: node scripts/check-release-tag.mjs python + # A final version is not published while the eject pending list has entries # (ADR-0034 Amendment 8). A release candidate passes. - name: Eject release gate diff --git a/AGENTS.md b/AGENTS.md index 3bb8bc101..c6773b526 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -72,7 +72,7 @@ PyPI has had no product change since `0.25.0` — nothing is broken. - **Kotlin** — `codegen-kotlin` (KotlinPoet on JVM): entity + Exposed table + Spring controller + payload + relations + filter allowlist + validator + stored-proc + output-parser generators. `integration-tests-kotlin` runs the persistence-conformance corpus through Exposed against Testcontainers Postgres. **Cross-port conformance corpora** (every port runs the shared corpus): -- Metamodel: `fixtures/conformance/` (395 fixtures; 29 shared corpora in total — per-corpus counts + the corpus x port matrix live in `docs/CONFORMANCE.md`). TS / C# / Java / Python all green. +- Metamodel: `fixtures/conformance/` (410 fixtures; 30 shared corpora in total — per-corpus counts + the corpus x port matrix live in `docs/CONFORMANCE.md`). TS / C# / Java / Python all green. - Render: `fixtures/render-conformance/`. TS / C# / Java / Kotlin / Python byte-identical. - Persistence: `fixtures/persistence-conformance/`. **Query** scenarios run on every port (TS / C# / Java / Kotlin / Python), each provisioning its test DB by executing the committed, TS-produced `canonical/schema.postgres.sql` (Postgres only — Derby dropped for the cross-port query corpus, ADR-0015; the FR-044 `report-*` scenarios also run on SQLite, MySQL and D1 where a port has the engine, see [docs/CONFORMANCE.md](docs/CONFORMANCE.md)). The **migration** scenarios are exercised by **TS only** (TS owns schema migrations). **The corpus now gates WRITES, not just reads (SP-H):** an `op: roundtrip` scenario type INSERTs through each port's runtime/ORM write codec (NOT raw SQL), reads the row back, and asserts the wire-normalized value. The `AllTypes` entity (`roundtrip-all-types.yaml`) carries one field of **every** persistable `field.*` subtype — string/int/long/double/float/decimal/boolean/date/time/timestamp(+tz)/currency/enum/uuid/object — plus an **array-of-VO** `field.object @isArray @storage:jsonb` column (`labels`, written as 2-element / empty-`[]` / single-element arrays across the three rows) — so every subtype write+read (incl. the array-of-value-object jsonb codec) round-trips through every port against Testcontainers PG. (`field.byte`/`field.short`/`field.class` were cut as non-functional registration-only stubs — the matrix tracks only genuinely-supported subtypes; see `fixtures/registry-conformance/README.md` → "Per-subtype write-round-trip matrix".) - API-contract: `fixtures/api-contract-conformance/`. TS / C# / Java / Kotlin / Python all green — each port runs **two lanes**: a hand-rolled reference server AND its **generated** API artifact booted over HTTP (the deployed controller/routes; TS+C# full-stack vs Testcontainers PG, Java/Kotlin/Python generated controller + in-memory repo behind the consumer seam). The generated fan-out found 10 real deployment bugs golden snapshots missed. Three sub-corpora run the **generated lane only, on all five ports** — `write-through/`, `projection/` (F22: a view-only `object.projection` serves GET list + GET by id and answers every write verb with `405 {"error": "method_not_allowed"}`) and `report/` (FR-044: a view-backed `object.report` serves GET list, answers `POST` with that same `405`, and mounts no `/{id}` route). That is deliberate, not a gap: what is under test is whether a port's GENERATOR emits those routes, and a hand-rolled reference server would answer every scenario by construction. The `m2m/` sub-corpus also gates **TPH x M:N together** (base-declared, subtype-declared, abstract-mid-declared, a non-subtype source onto a subtype TARGET, and the cross-subtype source id answering `200 []`) — the two corpora were originally built disjoint (`tph/` had no relationships, `m2m/` no discriminators), which is precisely how that defect class survived. diff --git a/CHANGELOG.md b/CHANGELOG.md index 15737cc7d..5d405a93d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,14 +14,18 @@ here.** _npm `1.1.0` (all 14 `@metaobjectsdev/*` packages in lockstep), PyPI `1.1.0`, NuGet `1.1.0` and Maven Central `8.1.0`. **`metamodelVersion` moves `1.0` to `1.1`.** The vocabulary is additive: -a model that uses none of the new names loads as it did under 1.0, and what loads changes only -where the Java and Python parser fixes under **Fixed** bring those ports into line with -TypeScript. What it generates is NOT unchanged in every port: in C#, Java, Kotlin's runtime and +a model that uses none of the new names loads as it did under 1.0, with two exceptions: a bare +`extends` naming an object in another package is refused in TypeScript, Python and C# (see +**Breaking** below), and the Java and Python parser fixes under **Fixed** bring those ports into +line with TypeScript. What it generates is NOT unchanged in every port: in C#, Java, Kotlin's runtime and Python the default for column and table names flips from the field name to `snake_case`, so a 1.0 estate whose columns are named like its fields must add `"naming": { "columns": "literal" }` (see **Breaking** below). Each entry says which models it affects. This is a MINOR because the -metamodel moved and reporting is a new feature, not because the package surface broke._ +metamodel moved and reporting is a new feature. It also removes commands and exports (the docs +commands and the types behind them, under **Removed**): that break rides this MINOR by the +maintainer's ruling in ADR-0034 Amendment 9. One page walks a 1.0.x project through every step: +[upgrading-1.0-to-1.1.md](docs/features/migrations/upgrading-1.0-to-1.1.md)._ **Breaking, C#, Java, Kotlin's runtime and Python: the default physical names are `snake_case`.** Column and table names the metadata does not give now come from one `naming` setting in @@ -73,7 +77,35 @@ package.** It now fails with `ERR_UNRESOLVED_SUPER`, as Java already did and ADR bare name binds the declaring file's own package, then the root. Qualify the reference: `extends: other::Base`. The entry is under **Changed** (ADR-0059). -**Five diffs to expect when you upgrade.** +**Breaking, in every port: a `field.decimal` crosses the API as one canonical string +(ADR-0019 Amendment 1).** Every decimal a generated route sends, on an entity, a projection or a +report row (an `avg`, a ratio, a `sum` of a decimal), is a JSON string in plain notation with no +exponent and no trailing fraction zeros: `12.50` is sent `"12.5"`, zero is `"0"`. Before 1.1.0 +each port sent its own form (TypeScript a string, the others a number), so a client that read a +number from C#, Java, Kotlin or Python now reads a string. Display formatting, such as two digits +for an amount of money, is a view's job, never the wire's. On input a route takes a string in +JSON-number grammar or a JSON number, read exactly; anything else is `400 validation`, and so is +a value with more fraction digits than `@scale` or more integer digits than +`@precision - @scale` (nothing is rounded). A decimal filter operand uses the same grammar, is +compared exactly, and a bad one is `400 invalid_filter_value`. Postgres and MySQL keep the +declared precision; SQLite and D1, which have no decimal type, keep 15 significant digits, and +`meta migrate` warns for a decimal there declared with `@precision` above 15. An ejected routes +generator keeps its old decimal handling until you resync it. The per-port entries are under +**Changed**. + +**Nine diffs to expect when you upgrade.** + +- **Generated files no generator emits any more are deleted.** The first full `gen` after the + upgrade, in every port, deletes a file the committed hash manifest records writing, that no + generator emitted in that run and that still hashes to what was recorded (a renamed value + object's old file, an api-docs page that moved). An edited such file is refused by name and + kept. Review the deletions in the diff. The entry is under **Fixed**. +- **An inherited reference means what it meant where it was written (ADR-0059 Decisions 3 + and 7).** A reference an object inherits through `extends` resolves in the package of the + node that declares it, not the inheriting object's package. A model whose base and subtype + live in different packages and that relied on a bare name meaning the subtype package's + object now binds the one in the base's package (or the root-level one). Qualify the + reference if you meant the other. - **References are written fully qualified (ADR-0059).** `meta export` and canonical serialization write every reference as `pkg::Name`, a root-level target referenced from a @@ -101,6 +133,13 @@ bare name binds the declaring file's own package, then the root. Qualify the ref `SpringRepositoryGenerator`, or subclasses one of them, still compiles too: the 1.0.x helpers are still there, marked `@Deprecated`, and forward to `SpringGeneratorSupport`. A generator you never ejected needs nothing. +- **Decimal handling is generated.** Every port's generated routes gain their decimal + wire code (the canonical writer, the strict reader and the exact filter operand), and the + allowlists gain the decimal field set. An ejected routes or allowlist generator needs a + resync to get it; the **Changed** entries name the files per port. +- **C# array properties are `IList`.** A generated scalar or enum array property was + `ICollection`, which EF Core 8 cannot read on SQLite; code that assigns a `HashSet` to + one assigns a `List`. The entry is under **Changed**. - **A keyless projection regenerates differently (G12).** `meta gen` rewrites the read-only surface of a keyless projection (no `/:id` route, no by-id function, and no write verbs in the API docs), and `meta verify --codegen` reports the affected pages as stale until they are @@ -117,7 +156,8 @@ bare name binds the declaring file's own package, then the root. Qualify the ref Every other inline enum, in every port, keeps its previous name and bytes. Two build settings follow from it. Java: `SpringDtoGenerator` owns the shared enum files, so a pom that runs the value-object or output-parser generator WITHOUT it, on a model where one of - their objects reaches a shared enum, must set `emitSharedEnums=true` on one of them. Kotlin: + their objects reaches a shared enum, must set `emitSharedEnums=true` (an `` entry on the + `SpringValueObjectGenerator` or `SpringOutputParserGenerator` ``) on one of them. Kotlin: every generator that types an enum now needs `providedEnumNamespace` / `providedEnumPackages` for a model with a `@provided` enum; set them in the plugin's ``. An owned (ejected) Kotlin generator keeps compiling and keeps its previous @@ -821,10 +861,9 @@ generator catalog and the Atlas evaluation are 1.2 work. integer. A measure's @default is a whole number (for example 0).", while C#, Java and Python refuse it through their existing integer-attribute check with that check's message. No port changed its general `attr.int` check, so a fraction in a validator's `@min` or `@max` loads - where it did. Two edge cases have no fixture: `@default: 0.0` (or `1.0`) is read as an integer - by TypeScript and refused by C#, Java and Python, and `@default: "zero"` on a `count` gives two - errors in TypeScript, C# and Python and one in Java. Write a whole number with no decimal - point. TypeScript lowers it: the view selects `FROM` the spine entity's table with `LEFT OUTER + where it did. `@default: 0.0` (or `1.0`) loads as that integer in every port, and a + non-number `@default` reports that one error (both under **Fixed**, with fixtures). + TypeScript lowers it: the view selects `FROM` the spine entity's table with `LEFT OUTER JOIN`s back to `@from`, and a defaulted measure is `COALESCE(…, n)` (SQLite writes `n.0` for a `decimal`, `double` or `float` measure and for a ratio, so the column keeps one storage class; MySQL reads a defaulted ratio of zero as `0.0000`). `meta migrate` refuses a derived `@spine` @@ -872,6 +911,24 @@ generator catalog and the Atlas evaluation are 1.2 work. against it. Read files through `ctx.inputRoot`; `runGen` takes it as `inputRoot`. The docs generators read a project's template overrides and owned site theme through it. +- **Python: SQLite and MySQL runtime drivers.** `metaobjects.runtime` exports `SqliteDriver` + (the standard library's `sqlite3`) and `MySqlDriver` (PyMySQL, through the new `mysql` extra: + `pip install "metaobjects[mysql]"`) beside the Postgres one, so `ObjectManager` reads and writes + those engines, report views included. The persistence and api-contract corpora run on both. +- **Java: an OMDB SQLite driver.** OMDB gains a `SQLiteDriver`, and `core-spring` resolves the + dialect `sqlite` to it. The persistence corpus, report views included, and the generated + report and decimal controllers run on Postgres, SQLite and MySQL. +- **A `decimal-wire` corpus.** `fixtures/decimal-wire/vectors.json` holds the shared vectors of + ADR-0019 Amendment 1 (the canonical format, the input grammar, filter operands and the + 15-digit read), and every port's decimal wire runs them. The api-contract corpus gains a + `decimal/` sub-corpus (8 scenarios, generated lane, Postgres, SQLite and MySQL, and D1 in + TypeScript), and the persistence corpus `decimal-exact` and its 15-digit twin. A scenario's + new `engines:` key names exactly the engines that run it. +- **The Cube lane queries every exporter shape.** A fifth live file queries the alias cubes, a + TPH subtype's `sql`, a one-to-one join from both ends, the int-backed enum's `CASE` and a + two-hop `@spine` report, and compares each with plain SQL on the same database. These were + compile-checked only before. + ### Changed **References mean the same thing in YAML and JSON, in every port (ADR-0059).** Every source now @@ -884,9 +941,13 @@ loaded, then rewrites each to its canonical spelling before the tree is frozen. object in another package (TypeScript, Python, C#; Java already refused it): qualify it. - A loaded model's reference values are canonical (`pkg::Name`, or `::Name` for a root-level target from a package). Code that reads a reference attribute and compares it to an object's - short name must compare the target instead: `refTarget(node, attr)` (TypeScript) returns the - object the loader bound, and `lookupCanonical(root, value)` resolves a canonical value exactly - (a bare canonical `Name` is the root-level object). `MetaRoot.findObject` binds a qualified + short name must compare the target instead: `refTarget(node, attr)` returns the object the + loader bound, and `lookupCanonical(root, value)` resolves a canonical value exactly (a bare + canonical `Name` is the root-level object). The same two exist in every port: TypeScript + `refTarget` / `lookupCanonical`, Java `com.metaobjects.loader.References.refTarget` / + `.lookupCanonical`, C# `MetaObjects.NamingRefs.RefTarget` / `.LookupCanonical`, Python + `metaobjects.naming_refs.ref_target` / `.lookup_canonical` (Kotlin generators call + `KotlinGenUtil.lookupCanonicalObject`). `MetaRoot.findObject` binds a qualified value exactly, but a bare name still matches the first object of that short name in any package, so do not pass it a reference value. - TypeScript `RenderContext.resolveValueObjectName` takes the field (`resolveValueObjectName(field)`) @@ -998,6 +1059,51 @@ until you regenerate. field. The value Drizzle reads from a `numeric` view column was always a string, so the old schema disagreed with the row it parsed. Code that treated the parsed field as a `number` stops typechecking. +- **Every port: generated routes carry the canonical decimal wire (ADR-0019 Amendment 1).** + The rule is under **Breaking** at the top; this is what each port generates for it. Every + port also rejects an input with more digits than the field's `@precision` and `@scale` as + `400 validation`, and reads a decimal on SQLite and D1 at 15 significant digits. + - TypeScript: a generated model module's write schema carries a local `decimalWrite` + function (generated code still imports nothing from MetaObjects), and the mounts in + `@metaobjectsdev/runtime-ts` write a decimal and read its filter operand by the same rule + (`runtime-ts/src/decimal-wire.ts`). An owned `entity` generator needs the new + `codegen/templates/decimal-wire-helper.ts` renderer, which `meta eject --list` reports as + differing, and owned runtime copies need a re-sync. `meta migrate` warns for a decimal with + `@precision` above 15 on SQLite or D1. A single-table base's polymorphic list and get + send a decimal that only a subtype declares as its canonical string too (it came back + `"12.50"`); an owned `routes` generator needs the resync to pick that up. + - C#: each namespace gets a generated `DecimalWire.g.cs`; a decimal property carries its + converter and, with `@precision`, `[DecimalWire.Declared(precision, scale)]`. A filter + operand that cannot be the field's type (decimal, int, date, Guid, enum) answers + `400 invalid_filter_value` instead of throwing, and a stored value the CLR `decimal` cannot + hold is `500 decimal_out_of_range`. An owned `routes` generator copies `DecimalWire.cs` and + `DecimalRangeEndpointFilter.cs`. On SQLite a decimal list sorts by its double, the engine + having no decimal type. + - Java: the Spring DTOs, Patch and value-object records serialize through + `CanonicalDecimalSerializer` and read through a contextual `CanonicalDecimalDeserializer` + that checks `@DeclaredDecimal(precision, scale)`; allowlists gain `DECIMAL_FIELDS`, and + `FilterParser.parse(rawQuery, fields, opsByField, decimalFields)` hands a repository + `BigDecimal` operands. An unreadable body is `400 validation`. Re-eject owned `routes`, + `dto`, `value-object` and `filter-allowlist` generators. OMDB binds a MySQL decimal operand + as `CAST(? AS DECIMAL(65,30))`. + - Kotlin: each package that needs it gets a generated `MetaDecimalWire.kt` (the writer, the + strict reader, the declared-digit check and the unreadable-body envelope), so generated + Kotlin imports nothing from MetaObjects. Entities serialize decimals through it, TPH + controllers filter decimals, and the controller checks the declared digits on create and + PATCH. The Exposed tables use a portable decimal and uuid column type + (`MetaPortableColumnTypes.kt`, per package): a uuid binds as text off Postgres, which fixes + a MySQL `500` and a SQLite no-match, and a decimal binds as `REAL` on SQLite, which fixes its + filters. `@precision` without `@scale` is scale 0, as in Java. + - Python: the routers use `metaobjects.codegen.runtime.decimal_wire` and emit the declared + limits of each decimal field; re-eject an owned `routes` generator to get them. A malformed + or non-object JSON body is `400 validation` (FastAPI answered `422`). +- **Breaking, C#: generated scalar and enum array properties are `IList`.** They were + `ICollection`, which EF Core 8 cannot materialize on SQLite, so an entity, TPH subtype, + value object or report row with an array field answered `500` there. Code that assigns a + `HashSet` (or another non-list collection) to one of these properties assigns a `List` + instead. `DecimalWire.DecimalCollectionJsonConverter` is replaced by + `DecimalListJsonConverter`; re-eject owned `entity` and `routes` generators. Arrays of value + objects and navigation collections stay `ICollection`. - **TypeScript API docs: a read-only object's page documents only what is generated.** In the `apiDocsFile()` pages, a read-only projection's page no longer lists `create`, `update` or `delete` functions, write REST verbs or Insert/Update schemas, none of which were ever @@ -1030,15 +1136,13 @@ until you regenerate. `coerceDouble`. Before, a list filter on such a column threw `ClassCastException`, a `500` on a request the allowlist had admitted, because the handler cast the coerced `Double` to the column's `BigDecimal` or `Float`. A single-table-inheritance (TPH) controller - changes for a float field only (on the base or a subtype); its filter pipeline already left - decimals out. `field.double` is unchanged, so a model with no decimal and no float field + changes for a float field only (on the base or a subtype); its decimals become filterable too + (see the decimal entry below). `field.double` is unchanged, so a model with no decimal and no float field keeps its bytes. Separately, a read-only controller or row mapper over a field named after a member of Exposed's `Table` (such as `source`) now reads the suffixed column property (`sourceColumn`) and compiles; every other field name keeps its bytes. -### Changed - - **TypeScript: every page of documentation is a generator in `generators` (ADR-0034 Amendment 9).** `meta gen` writes it and `meta verify --codegen` checks it, with the code. - The factories, from `@metaobjectsdev/codegen-ts/generators`: `docsFile()` (model pages), @@ -1194,6 +1298,23 @@ until you regenerate. prompt, apart from one dropped phrase ("in this file"). Its maintainer notes moved to `docs/superpowers/specs/2026-07-12-metaobjects-fit-assessment-maintainer-notes.md`, and the skill's `scaffold: false` marker is now `metadata: { scaffold: false }` (no code reads it). +- **Smaller public API changes in C#, Java and Python.** These touch an owned generator or a + library caller, not a project that only runs the CLI or the plugin. Each followed from + references resolving once (ADR-0059) or the one naming setting (ADR-0058). + - C#: removed `YamlDesugar.ExpandRef` (the YAML source no longer rewrites references), + `M2MDerivation.ResolveEntity` and `ExtractDelegateEmitter.FindObject` (read the bound target + with `NamingRefs.RefTarget`). The `ColumnNamingStrategy` enum moved from `MetaObjects.Codegen` + to `MetaObjects.Meta`, and the default of `CSharpNaming.Column`, `ResolveObjectNames` and + `GenConfig.ColumnNamingStrategy` is `SnakeCase`. `ResolvedMetadata` and `PayloadField` take + extra positional parameters. + - Java: removed `BaseMetaDataParser.getFullyQualifiedSuperMetaDataName`. + `ColumnNaming.DEFAULT` is now `snake_case` (it was `literal`). `PayloadField` gained + `isArray`; the two-argument constructor is kept. The HTML doc writer base classes changed. + - Python: removed `naming_refs.is_relative_ref`. `find_inbound.response_shape` takes only the + template (it was `(root, template, referrer_pkg)`), and `MetaSource.physical_name` takes an + optional `table_naming`. `router_generator.emits_router` and + `entity_model.has_literal_default` are no longer imported by the modules that used to + import them from there. ### Deprecated @@ -1359,6 +1480,65 @@ until you regenerate. ### Fixed +- **Found by the 1.1.0 release review, in every port unless named.** + - **The loader refuses four more report shapes**, with the cross-port code and a message + naming the fix: a dimension or measure over a `field.object` or `@objectRef` field + (D5, M10: `ERR_INVALID_DIMENSION` / `ERR_INVALID_MEASURE`; a `field.map` with `@valueType` + stays legal), a report whose read-only source is not `@kind: view` (R10), and an abstract + report (R11, both `ERR_INVALID_REPORT`). Each used to load and fail later, in + `meta migrate` or at the first request. A `@via` dimension's `@of` must name a field the + owner itself has: one only a subtype declares is `ERR_INVALID_DIMENSION`, naming the + subtype to write. When the entity the `@via` reaches redeclares that field (another + subtype or column), the report groups by the redeclared field, in every port. + - **A report dimension over an array or a map is neither filterable nor sortable**: a filter + on one is `400 invalid_filter_field` and a sort `400 invalid_sort`. A Java report map + dimension keeps its `@valueType`. + - **An integer attribute written `4.0` loads as `4`**, and a fraction is + `ERR_BAD_ATTR_VALUE`, in every port (a requirement's `level`, a measure's `@default`). A + bad `@default` reports that one error, not a second rule error beside it. + - **A requirement's package is its own, else the nearest enclosing declaration, else the + file's**, and an overlay that declares a package gives it to the children it adds. Coverage + counts every non-abstract project entity except one imported from a dependency; the + project's `scope` filters generated output, not coverage. + - **`ERR_UNRESOLVED_SUPER` names the fix**: the same-named objects in other packages and the + qualified `extends` to write. + - **The JVM runtimes read the `naming` setting.** Java and Kotlin outside Spring: + `MetaDataLoader.fromUris` and `fromDirectory` read the `.metaobjects/config.json` above the + metadata. Spring: `classpath:.metaobjects/config.json`, or the location + `metaobjects.physical-naming-config` names (a `classpath:` location or a file path; a named + location that is missing fails startup). Each logs once when it falls back to the + `snake_case` default. Before, a runtime that was not handed the rules used the default + while codegen used the project's, and queried columns that did not exist. + - **`eject --list` compares what an owned generator owns, not just its own file.** TypeScript + compares each renderer under `codegen/templates/` it imports (text: + `[owned — DIFFERS: generator identical, N renderer(s) differ]` and a line per renderer; + JSON: `templates`), Python each ejected runtime module, and `dotnet meta gen --list` and + `mvn metaobjects:eject -Dlist=true` fold the companions into the verdict. A generator + reported identical while its renderers lagged kept 1.0 logic silently. + - **TypeScript: generated forms submit fractional numbers.** A `field.double` or + `field.float` input carries `step="any"` (the browser's default step of 1 refused `12.5`), + and a `field.decimal` is a `type="text" inputMode="decimal"` input, since a number input + validates through a double. Both the `.meta.ts` descriptor (new `step` and + `inputMode` keys, which `useEntityForm` spreads) and the nested sub-forms; `agent/ui.md` + prints them. + - **TypeScript: an unknown top-level key in `.metaobjects/config.json` is a warning**, as it + is ignored in the other ports, so a key a later release adds does not fail an older CLI. + - **TypeScript: `meta migrate --dry-run` needs no `--slug`**, and the staleness nudge links + the guide for the minor you are on, pinned to the release tag (the links come from the new + `@metaobjectsdev/metadata/release-docs` entry). + - **Python: the loaders read `naming` from the project.** `from_directory`, `from_uris` and + `from_string(project_dir=...)` find `.metaobjects/config.json`; pass + `discover_naming=False` to skip it. With no config they warn once (`UserWarning`) that + `snake_case` is the default. + - **TypeScript: the agent-context files carry a content hash**, so a refresh tells an edited + file from a stale one by its bytes, not its version line. + - **Java: an object that redeclares an inherited field with another subtype gets a POJO that + compiles.** The class is generated standalone (every effective field, extending + `PojoObject`) instead of `extends` the super, so `instanceof` against the super no longer + holds for that class only. + - **Cube: a report served at several grains gets one rollup per grain** + (`_`), so each grain is answered from its own rollup. + - **References resolve the same way from every source (ADR-0059).** - YAML `::Name` reaches the root-level object even when the package declares the same name. It used to bind the local object silently. @@ -1784,15 +1964,15 @@ until you regenerate. `"paidShare": 0.4` where Postgres answers `"paidShare": "0.4"`, and the row type said `string` for both, so `.toFixed(1)` compiled on neither and ran on one. The route is now corrected, not the type. **Behaviour note:** on SQLite a report's `avg`, ratio and - `sum` of a decimal field arrive as strings (`"0.4"`, `"1.6666666666666667"`, null stays + `sum` of a decimal field arrive as strings (`"0.4"`, `"1.66666666666667"`, null stays null); convert with `Number(x)` where you did arithmetic or formatting on the number. The generated report route passes the decimal field names to the mount as `decimalColumns` (`mountReadOnlyCrudRoutes` in the Fastify and Hono adapters, new optional option), which sends a number under one of them as its string. Filtering and sorting on the column are unchanged - and stay numeric, because the view column is still a REAL. The digits are JavaScript's - shortest spelling of the double, not Postgres' fixed scale, and a decimal's digits stay - outside the cross-port contract. Postgres and MySQL are untouched, and so is a projection on - SQLite (released behaviour; its decimals still arrive as numbers). **Upgrading an owned + and stay numeric, because the view column is still a REAL. The digits are the double read at + 15 significant digits in the canonical form of ADR-0019 Amendment 1, the same in every port, + and a projection's or entity's decimal on SQLite is a string by the same rule (see the decimal + entry under **Breaking**). **Upgrading an owned `routes` or `routes-hono` generator:** a copy ejected before this release passes no `decimalColumns`, so a SQLite report keeps answering a number until you resync it (`meta eject routes --force`, `meta eject routes-hono --force`, which also refreshes the @@ -1950,13 +2130,15 @@ until you regenerate. its model lists. It now is for every table the definition in the database names as well: one the new definition no longer reads, and one an `@sql` body joins. The D1 foreign-key cascade does the same for every table it rebuilds, including a view the migration drops. - - **A drop and a create discards the privileges granted on the view.** `DROP VIEW` removes the - view's grants with it, and `meta migrate` writes no `GRANT`, so a role that had `SELECT` on a - projection's view loses it, with nothing reported. It needs no `--allow` and applies cleanly. - In an offline plan every changed Postgres view takes this path, including an appended field - that was a `CREATE OR REPLACE VIEW` before. A `DROP VIEW` also fails at apply if another - object depends on the view, and says which. To keep grants and dependents across the change, - plan with `--from-db`; otherwise grant the privileges again after applying. + - **A Postgres drop and create keeps the privileges granted on the view.** `DROP VIEW` removes + the view's grants with it, so the migration copies them into a temporary table before the + drop and grants each one again after the create (`WITH GRANT OPTION` where it was + grantable), and `meta migrate` names each view it recreated: "view .: dropped + and recreated; its grants are carried over." In an offline plan every changed Postgres view + takes this path, including an appended field that was a `CREATE OR REPLACE VIEW` before. + The down migration's recreate does the same, and column-level grants are carried for the + columns the new view still has. A `DROP VIEW` also fails at apply if another object depends on the view, and says which; plan + with `--from-db` to keep dependents. - **Adopting a view (`--allow adopt-view`) follows the same rule**, for a derived view and for an `@sql` one. A hand-written or pre-fingerprint view with the same columns is still replaced in place when the probe proves it. It becomes a drop and a create when the columns differ, or diff --git a/README.md b/README.md index 86892af00..8c7659fa3 100644 --- a/README.md +++ b/README.md @@ -127,6 +127,7 @@ first-week wedge plan — and `meta init` picks up from there. - Want a worked, non-toy model? → [`examples/advanced-modeling/`](examples/advanced-modeling/) (projections, value objects, TPH, prompt payloads on one runnable spine). - Want the cross-language contract? → [`fixtures/`](fixtures/) (the conformance corpora are the oracle). +- Upgrading from 1.0.x to 1.1? → [`docs/features/migrations/upgrading-1.0-to-1.1.md`](docs/features/migrations/upgrading-1.0-to-1.1.md). ## Languages @@ -263,7 +264,7 @@ metaobjects/ ├── CLAUDE.md # project instructions for Claude ├── spec/ # canonical metamodel docs, ADRs, roadmap ├── fixtures/ # 25 cross-language conformance corpora — the oracle -│ ├── conformance/ # metamodel (loader + serializer + navigation), 395 fixtures +│ ├── conformance/ # metamodel (loader + serializer + navigation), 410 fixtures │ ├── yaml-conformance/ # YAML authoring desugar │ ├── render-conformance/ # FR-004 byte-identical render oracle │ ├── verify-conformance/ # FR-004 template-drift gate diff --git a/agent-context/skills/metaobjects-codegen/references/java.md b/agent-context/skills/metaobjects-codegen/references/java.md index 3f211a60d..2e3872643 100644 --- a/agent-context/skills/metaobjects-codegen/references/java.md +++ b/agent-context/skills/metaobjects-codegen/references/java.md @@ -41,9 +41,10 @@ field names; a database whose columns are named exactly like the fields declares There is no per-generator or runtime naming: the `columnNaming` generator arg fails `mvn metaobjects:generate`, and `SimpleMappingHandlerDB.setColumnNaming` throws. A Spring -application that loads its metadata from the classpath points `metaobjects.physical-naming-config` -at the project's `.metaobjects/config.json` packaged as a resource (for example -`classpath:.metaobjects/config.json`). Details and the 1.0 to 1.1 steps: +application reads `classpath:.metaobjects/config.json` when the project packages that file as a +resource; `metaobjects.physical-naming-config` points elsewhere (a `classpath:` location or a file +path). `MetaDataLoader.fromUris` and `fromDirectory` find the file above the metadata. With no file +found, the loader logs one WARN and uses snake_case. Details and the 1.0 to 1.1 steps: . ## Plugin config in `pom.xml` diff --git a/agent-context/skills/metaobjects-codegen/references/kotlin.md b/agent-context/skills/metaobjects-codegen/references/kotlin.md index 3a047c508..6a80fafdd 100644 --- a/agent-context/skills/metaobjects-codegen/references/kotlin.md +++ b/agent-context/skills/metaobjects-codegen/references/kotlin.md @@ -59,9 +59,10 @@ field names; a database whose columns are named exactly like the fields declares There is no per-generator or runtime naming: the `columnNaming` generator arg fails `mvn metaobjects:generate`, and `SimpleMappingHandlerDB.setColumnNaming` throws (the Exposed tables and the names generator already read the setting). A Spring -application that loads its metadata from the classpath points `metaobjects.physical-naming-config` -at the project's `.metaobjects/config.json` packaged as a resource (for example -`classpath:.metaobjects/config.json`). Details and the 1.0 to 1.1 steps: +application reads `classpath:.metaobjects/config.json` when the project packages that file as a +resource; `metaobjects.physical-naming-config` points elsewhere (a `classpath:` location or a file +path). `MetaDataLoader.fromUris` and `fromDirectory` find the file above the metadata. With no file +found, the loader logs one WARN and uses snake_case. Details and the 1.0 to 1.1 steps: . ## Plugin config in `pom.xml` diff --git a/client/web/packages/react/src/use-entity-form.tsx b/client/web/packages/react/src/use-entity-form.tsx index 8389a72de..0d77b7df6 100644 --- a/client/web/packages/react/src/use-entity-form.tsx +++ b/client/web/packages/react/src/use-entity-form.tsx @@ -40,6 +40,8 @@ export interface EntityFieldMeta { readonly label: string; readonly view: string; readonly htmlType?: string; + readonly step?: string; + readonly inputMode?: string; readonly placeholder?: string; readonly helpText?: string; readonly rules?: RegisterOptions; @@ -126,6 +128,8 @@ export function useEntityForm, (field.rules ?? {}) as any); const props: BoundInputProps = { ...registered, "aria-label": field.label }; if (field.htmlType !== undefined) props.type = field.htmlType; + if (field.step !== undefined) props.step = field.step; + if (field.inputMode !== undefined) props.inputMode = field.inputMode; if (field.placeholder !== undefined) props.placeholder = field.placeholder; acc[key] = props; } diff --git a/client/web/packages/react/test/use-entity-form-fractional.test.tsx b/client/web/packages/react/test/use-entity-form-fractional.test.tsx new file mode 100644 index 000000000..26641e3a2 --- /dev/null +++ b/client/web/packages/react/test/use-entity-form-fractional.test.tsx @@ -0,0 +1,56 @@ +import "./setup.js"; +import { describe, test, expect, afterEach } from "bun:test"; +import { render, cleanup } from "@testing-library/react"; +import React from "react"; +import { z } from "zod"; +import { useEntityForm } from "../src/use-entity-form.js"; + +afterEach(() => { cleanup(); }); + +// What `.meta.ts` carries for a double, a decimal and an int field. +const Item = { + $entity: "Item", + $table: "items", + $path: "/items", + weight: { name: "weight", label: "Weight", view: "number", htmlType: "number", step: "any" }, + price: { name: "price", label: "Price", view: "number", htmlType: "text", inputMode: "decimal" }, + qty: { name: "qty", label: "Qty", view: "number", htmlType: "number" }, +} as const; + +function Form() { + const form = useEntityForm(Item, z.object({ weight: z.string(), price: z.string(), qty: z.string() })); + return ( +
+ + + +
+ ); +} + +describe("useEntityForm — fractional inputs", () => { + test("a double's input has step=any, so the browser accepts 12.5", () => { + const { getByTestId } = render(
); + const weight = getByTestId("weight") as HTMLInputElement; + expect(weight.type).toBe("number"); + expect(weight.getAttribute("step")).toBe("any"); + weight.value = "12.5"; + expect(weight.validity.stepMismatch).toBe(false); + }); + + test("an int keeps the default step, so 12.5 is still refused", () => { + const { getByTestId } = render(); + const qty = getByTestId("qty") as HTMLInputElement; + expect(qty.hasAttribute("step")).toBe(false); + qty.value = "12.5"; + expect(qty.validity.stepMismatch).toBe(true); + }); + + test("a decimal is a text input with a decimal keyboard, not a number input", () => { + const { getByTestId } = render(); + const price = getByTestId("price") as HTMLInputElement; + expect(price.type).toBe("text"); + expect(price.getAttribute("inputmode")).toBe("decimal"); + expect(price.hasAttribute("step")).toBe(false); + }); +}); diff --git a/docs/1.0-readiness.md b/docs/1.0-readiness.md index 0efcc352d..ced552c54 100644 --- a/docs/1.0-readiness.md +++ b/docs/1.0-readiness.md @@ -2,7 +2,7 @@ > **CLOSED 2026-09-09 — the cut shipped. This is history, not a live gate list.** > npm / PyPI / NuGet are at `1.0.0`, Maven Central at `8.0.0`, and `metamodelVersion` is -> `1.0` (it moves to `1.1` on `main` with the FR-044 reporting vocabulary). The declared blocking path this doc named in its own footer — **G3d, then +> `1.0` (it moved to `1.1` in 1.1.0, with the FR-044 reporting vocabulary). The declared blocking path this doc named in its own footer — **G3d, then > G1/G2/G4** — is closed end to end: G3d on nine adopter estates that ran the release > candidate with their own drift gates enforced (not a bare `meta verify`), and G1/G2/G4 at > the cut itself. diff --git a/docs/CONFORMANCE.md b/docs/CONFORMANCE.md index feffbed3b..c0e94907e 100644 --- a/docs/CONFORMANCE.md +++ b/docs/CONFORMANCE.md @@ -1,6 +1,6 @@ # Conformance coverage -The MetaObjects standard ships **29 shared conformance corpora** under +The MetaObjects standard ships **30 shared conformance corpora** under [`fixtures/`](../fixtures/). Every port runs every corpus that is *applicable to it* and asserts the same expected behaviour against the same fixtures. **This page is the inverse index**: fixture → feature doc + per-port pass status, and it is the @@ -26,15 +26,16 @@ regenerate with `ls -d fixtures//*/ | wc -l` for directory-shaped corpor | Corpus | Fixtures | TS | Java | Kotlin | C# | Python | |---|---|---|---|---|---|---| -| [`fixtures/conformance/`](../fixtures/conformance/) (metamodel) | 395 | ✓ | ✓ | inherits via `metadata-ktx` | ✓ | ✓ | +| [`fixtures/conformance/`](../fixtures/conformance/) (metamodel) | 410 | ✓ | ✓ | inherits via `metadata-ktx` | ✓ | ✓ | | [`fixtures/yaml-conformance/`](../fixtures/yaml-conformance/) | 16 | 16 / 16 | 15 / 16 (1 ledgered: `yaml-quoted-leading-zero` — Java pipeline strips quotes off `"007"`) | inherits via Java | 15 / 16 (1 ledgered: `error-yaml-coerced-hex-in-string` — YamlDotNet doesn't coerce `0xFF`) | 16 / 16 | | [`fixtures/verify-conformance/`](../fixtures/verify-conformance/) | 37 | ✓ | ✓ | inherits via Java | ✓ | ✓ | | [`fixtures/verify-strict-conformance/`](../fixtures/verify-strict-conformance/) | 1 | ✓ | — | — | — | ✓ | | [`fixtures/render-conformance/`](../fixtures/render-conformance/) | 16 | ✓ | ✓ | inherits via Java | ✓ | ✓ | | [`fixtures/extract-conformance/`](../fixtures/extract-conformance/) | 48 | ✓ | ✓ | inherits the shared JVM engine | ✓ | ✓ | | [`fixtures/output-prompt-conformance/`](../fixtures/output-prompt-conformance/) | 17 | ✓ | ✓ | ✓ | ✓ | ✓ | -| [`fixtures/persistence-conformance/`](../fixtures/persistence-conformance/) | 54 (48 query + 6 migration) | all 54 (the 21 `report-*` also on SQLite, MySQL and D1) | 48 query (migrations TS-only, ADR-0015); the 21 `report-*` also on MySQL | 48 query (via Exposed); the 21 `report-*` also on SQLite and MySQL | 48 query; the 21 `report-*` also on SQLite and MySQL | 48 query (Postgres only: pg8000) | -| [`fixtures/api-contract-conformance/`](../fixtures/api-contract-conformance/) | 91 (31 core + 10 tph + 9 m2m + 2 jsonb + 2 write-through + 12 projection + 25 report) | ✓ (Fastify reference + generated lane); `report/` also on SQLite, MySQL and D1 | ✓ (embedded HTTP + JDBC) | ✓ (embedded HTTP + Exposed) | ✓ (HttpListener + Npgsql); `report/` also on SQLite and MySQL | ✓ (FastAPI + pg8000) | +| [`fixtures/persistence-conformance/`](../fixtures/persistence-conformance/) | 59 (53 query + 6 migration) | all 59; the 24 `report-*` and the `-15` decimal scenarios also on SQLite, MySQL and D1, `decimal-exact` also on MySQL (a scenario's `engines:` key is exact) | 53 query (migrations TS-only, ADR-0015); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 53 query (via Exposed); the `report-*` and `engines:` scenarios also on SQLite and MySQL | 53 query; the `report-*` and `engines:` scenarios also on SQLite and MySQL | 53 query (pg8000); the `report-*` and `engines:` scenarios also on SQLite (`sqlite3`) and MySQL (PyMySQL) | +| [`fixtures/api-contract-conformance/`](../fixtures/api-contract-conformance/) | 101 (31 core + 10 tph + 9 m2m + 2 jsonb + 2 write-through + 12 projection + 27 report + 8 decimal) | ✓ (Fastify reference + generated lane); `report/` and `decimal/` also on SQLite, MySQL and D1 | ✓ (embedded HTTP + JDBC); `report/` and `decimal/` generated controllers over OMDB on Postgres, SQLite and MySQL | ✓ (embedded HTTP + Exposed); `report/` and `decimal/` also on SQLite and MySQL | ✓ (HttpListener + Npgsql); `report/` and `decimal/` also on SQLite and MySQL | ✓ (FastAPI + pg8000); `report/` and `decimal/` also on SQLite and MySQL | +| [`fixtures/decimal-wire/`](../fixtures/decimal-wire/) | 1 vector file (format, parse, filter operand, 15-digit read) | ✓ (`runtime-ts` and the generated helper) | ✓ (`CanonicalDecimal`) | ✓ (the generated wire file) | ✓ (`DecimalWire`) | ✓ (`decimal_wire`) | | [`fixtures/validation-conformance/`](../fixtures/validation-conformance/) | 42 cases | ✓ (generated Zod + run-time `runValidators`) | ✓ | ✓ | ✓ | ✓ (generated Pydantic + run-time `run_validators`) | | [`fixtures/registry-conformance/`](../fixtures/registry-conformance/) | 1 canonical manifest | ✓ (reference emitter) | ✓ | ✓ | ✓ | ✓ | | [`fixtures/object-model-conformance/`](../fixtures/object-model-conformance/) | 1 shared metadata fixture (per-port scenarios) | ✓ | ✓ | ✓ | ✓ | ✓ | @@ -111,9 +112,10 @@ Amendment 8). A gate, not a corpus: it has no fixtures of its own, and reads `cli/test/eject-contract.test.ts`. C#: `EjectContractTests`. Java: `EjectContractTest` in `maven-plugin`. Kotlin: `EjectContractTest` in `codegen-kotlin`. Python: `tests/conformance/test_eject_contract.py`. -- *Not yet true everywhere, and the shared file says where.* `eject.pending` lists every - generator that still fails an assertion. A port's test fails when an unlisted gap appears - and when a listed one closes without its entry being removed. +- *A gap is listed, never silent.* `eject.pending` lists every generator that still fails an + assertion; it is empty (`[]`) for 1.1.0, so every reference passes every assertion. A port's + test fails when an unlisted gap appears and when a listed one closes without its entry being + removed, and a final release is refused while the list has entries (ADR-0034 Amendment 8). - *Not gated: an adopter's owned copy.* The contract is about what the reference hands over. **Capability requirements** ([features/requirements.md](features/requirements.md)): @@ -149,7 +151,7 @@ manifest contains without moving these boundaries. canonical model's golden. `codegen-ts/test/cube/cube-model-corpus.test.ts` and `cube-model-canonical.test.ts` run them. No other port has an exporter (spec R6), so no other port makes a claim the corpus would have to check. -- *Cube accepts the output — queried on Postgres, compiled in both dialects.* The `cube` lane +- *Cube accepts the output — queried on Postgres and MySQL, in development and production mode.* The `cube` lane (`scripts/ci-local.sh --only cube`, which needs Docker) loads the canonical golden and the 36 cases that hold a tree into a pinned Cube (`cubejs/cube:v1.7.43`) and requires each to compile, the three MySQL cases included: compiling a model runs no SQL, so the lane's Postgres @@ -160,10 +162,18 @@ manifest contains without moving these boundaries. generated with `dialect: "mysql"` is loaded into the same pinned Cube over a real MySQL 8.4 holding the report views `buildReportViews` lowers (`fixtures/cube-model/canonical/seed.mysql.sql`), and each of the twenty-three served reports' Cube query is compared with `SELECT *` from its view, with the - pre-aggregation used asserted. That is the smaller of the two honest options: the three MySQL - *cases* (`dimension-time-mysql`, `measure-mysql`, `measure-boolean-mysql`) are still compiled, not executed, and the goldens hold - their SQL. The lane runs Cube in development mode, so Cube's production mode with a separate - Cube Store is not gated. + pre-aggregation used asserted. A third file (`cube-model-mysql-cases.live.ts`) executes the + three MySQL *cases* (`dimension-time-mysql`, `measure-mysql`, `measure-boolean-mysql`) on a + real MySQL 8.4 with seeded tables, comparing 14 queries with rows worked out by hand from the + seed (the cases have no report view to compare with). A fourth (`cube-model-production.live.ts`) + runs Cube in production mode (`CUBEJS_DEV_MODE=false`, a signed token, a separate + `cubejs/cubestore:v1.7.43` container), on Postgres and on MySQL: it builds every rollup through + the pre-aggregations jobs API, requires each served report with a rollup to be answered from + Cube Store with the view's rows, and requires every rollup the model declares to answer at least + one query, so a rollup no query can match fails the lane. A fifth + (`cube-model-shapes.live.ts`) queries the shapes the canonical model does not carry (alias + cubes, a TPH subtype, a one-to-one join, the int-backed enum's `CASE`, a two-hop `@spine` + report) from their `fixtures/cube-model/` trees and compares the rows with plain SQL. - *The lane is a gate, not a second corpus.* Like the codegen-compile gate, it reuses `fixtures/persistence-conformance/canonical/meta.fitness.json` and the committed `schema.postgres.sql` rather than adding a model beside them; only the seed rows are its own @@ -267,7 +277,7 @@ unit-test runners (`bun test`, `dotnet test`, `pytest`, `mvn test`) pull Docker. ## Fixture-to-doc mapping -### `fixtures/conformance/` — metamodel loader + canonical serializer (395) +### `fixtures/conformance/` — metamodel loader + canonical serializer (410) | Fixture prefix | Feature doc | |---|---| @@ -318,17 +328,17 @@ All 37 fixtures → [features/migrations-and-drift.md](features/migrations-and-d may carry `"isArray": true`; with `fields` it is a list of objects, without it a list of scalars. The six `verify-positional-accessor-*` cases pin where `-last` is available. -### `fixtures/persistence-conformance/` (54 — 48 query + 6 migration) +### `fixtures/persistence-conformance/` (59 — 53 query + 6 migration) - `migrations/*` (6) → [features/migrations-and-drift.md](features/migrations-and-drift.md) (schema migration section) -- `queries/*` (48) → [features/source-kinds.md](features/source-kinds.md) (query semantics against `source.rdb`) - -**Engines beyond Postgres (FR-044 reports).** The twenty-one `queries/report-*.yaml` scenarios also -run on SQLite and MySQL 8.4 wherever a port's runtime has the engine, and on D1's local runtime -in TypeScript (no cloud account): TypeScript on all four, C# and Kotlin on SQLite and MySQL, Java -on MySQL. **Stated limitations, not gaps:** Java's OMDB has no SQLite driver, and Python's runtime -ships a Postgres driver only (pg8000), so Python runs the corpus on Postgres alone. Adding a -driver to run a view-reading test is not worth the surface. Each engine executes an artifact +- `queries/*` (53) → [features/source-kinds.md](features/source-kinds.md) (query semantics against `source.rdb`) + +**Engines beyond Postgres (FR-044 reports, decimals).** The twenty-four `queries/report-*.yaml` +scenarios also run on SQLite and MySQL 8.4 in every port, and on D1's local runtime in +TypeScript (no cloud account). A scenario that declares `engines:` runs on exactly those engines +(`decimal-exact-15` on all four, `decimal-exact` on Postgres and MySQL). Java reads SQLite +through OMDB's `SQLiteDriver`, and Python through `SqliteDriver` (`sqlite3`) and `MySqlDriver` +(PyMySQL). Each engine executes an artifact TypeScript produced (`canonical/schema.sqlite.sql` from `meta migrate --dialect sqlite`; `canonical/schema.mysql.sql` from the adopter-written tables plus `buildReportViews`), held equal to its generator by a test, so no port writes view SQL. The `expect` blocks stay the @@ -337,9 +347,9 @@ REAL, a MySQL `DECIMAL` keeps its scale, a MySQL `DATETIME` is the UTC wall cloc `report-relative-date` carries a per-engine seed (`seed-data-engine`). `op: relate` and the write ops stay Postgres-only. See [features/reporting.md](features/reporting.md#what-the-corpus-gates). -### `fixtures/api-contract-conformance/` (91) +### `fixtures/api-contract-conformance/` (101) -All 91 scenarios → [features/api-contract.md](features/api-contract.md) (cross-port +All 101 scenarios → [features/api-contract.md](features/api-contract.md) (cross-port REST API URL grammar + JSON wire format). Verifies every backend's emitted CRUD routes answer identically over HTTP — list / get / create / patch+put / delete, plus pagination (`limit`/`offset`), sort (`sort=field:dir`), the `withCount=1` @@ -350,25 +360,26 @@ The corpus also covers the 9 cross-port filter operators (`eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `like`, `isNull`) plus the implicit-AND combinator and 2 error shapes (`invalid_filter_field` / `invalid_filter_op`) under the URL grammar `?filter[][]=` (FR-009). On top of the 31 core -scenarios the corpus carries six sub-corpora — `tph/` (10, single-table -inheritance), `m2m/` (9 — 3 plain, 5 gating TPH x M:N together, the combination +scenarios the corpus carries seven sub-corpora — `tph/` (10, single-table +inheritance; a subtype decimal filters and refuses extra digits), `m2m/` (9 — 3 plain, 5 gating TPH x M:N together, the combination each corpus alone could not reach, and 1 pinning the collection-URL spelling), `jsonb/` (2, typed value-object columns), `write-through/` (2, table-write + view-read entities), `projection/` (12, a read-only view answers reads and refuses writes with 405; a projection with no declared identity has no item route; a projection keyed on a field not named `id` is addressed by -it, whether its identity names `@fields` or derives them; decimal and float fields filter) and `report/` (25, +it, whether its identity names `@fields` or derives them; decimal and float fields filter), `decimal/` (8, ADR-0019 Amendment 1: a decimal is one +canonical string, read exactly, filtered exactly and refused past its declared digits) and `report/` (27, FR-044: a view-backed `object.report` is listed, filtered, sorted and paged on its derived fields, answers `POST` with 405 and mounts no `/{id}`; a `@spine` report keeps a row with no facts, and a measure with `@default` filters and sorts as its default). All 5 ports — TS, Java, Kotlin, C#, Python — run it in BOTH lanes: a hand-rolled reference server and the port's own GENERATED API artifact booted over HTTP. `write-through/`, -`projection/` and `report/` run the generated lane only, on all five ports: what +`projection/`, `decimal/` and `report/` run the generated lane only, on all five ports: what they test is whether a port's generator emits the routes. The `report/` sub-corpus is the one -that reads real views, so it also runs on SQLite and MySQL (`report/schema.sqlite.sql`, -`report/schema.mysql.sql`, both TypeScript-produced) in TypeScript and C#, and on D1's local -runtime in TypeScript. Java, Kotlin and Python serve that corpus from seeded rows behind their -repository seam, so no engine reaches them and the extra engines are not applicable there. +that reads real views, so it runs full stack on Postgres, SQLite and MySQL +(`report/schema.sqlite.sql`, `report/schema.mysql.sql`, both TypeScript-produced) in every +port, and on D1's local runtime in TypeScript. The `decimal/` sub-corpus does the same over +decimal columns. ### `fixtures/source-resolution-conformance/` (25 cases) @@ -542,8 +553,8 @@ files into a real Cube. ## Orphaned fixtures (tested but not yet documented) -The fixtures in the nine corpora mapped above (metamodel 395 + yaml 16 + verify 37 -+ render 16 + persistence 54 + api-contract 91 + source-resolution 25 + scope 10 + +The fixtures in the nine corpora mapped above (metamodel 410 + yaml 16 + verify 37 ++ render 16 + persistence 59 + api-contract 101 + source-resolution 25 + scope 10 + dependency 23) each map to a feature doc, and so do the two requirement corpora and the `cube-model` corpus, whose case lists live in their own READMEs. None are orphaned today. The remaining corpora in the totals table gate tooling contracts (registry manifests, provider diff --git a/docs/README.md b/docs/README.md index d0ee9dcab..d922143df 100644 --- a/docs/README.md +++ b/docs/README.md @@ -10,6 +10,10 @@ files under [`features/`](features/) are the source of truth — each file is self-contained and shows the same `Author` entity across all five ports so you can copy-paste cross-port without context-switching. +**Upgrading from 1.0.x to 1.1?** Start at +[`features/migrations/upgrading-1.0-to-1.1.md`](features/migrations/upgrading-1.0-to-1.1.md): +the steps in order and every change that needs action, per port. + ## Layout ``` @@ -26,7 +30,7 @@ docs/ │ ├── templates-and-payloads.md │ ├── loaders.md │ ├── migrations-and-drift.md -│ ├── migrations/ # per-version migration guides (e.g. 0.x → 1.0) +│ ├── migrations/ # per-version migration guides (start at upgrading-1.0-to-1.1.md for 1.0.x → 1.1) │ ├── api-contract.md # cross-port REST contract (the universal browser client speaks it) │ ├── abstracts-and-inheritance.md # abstract: true + extends: — author-side shape reuse │ ├── extending-with-providers.md # custom metamodel subtypes/attrs via MetaDataTypeProvider diff --git a/docs/RELEASING.md b/docs/RELEASING.md index db1638410..eca9b640f 100644 --- a/docs/RELEASING.md +++ b/docs/RELEASING.md @@ -279,7 +279,7 @@ contracts on two numbers: - **Package version** (npm/PyPI/NuGet `1.x`, Maven `8.x`) promises the SOFTWARE surface — exports, CLI flags, generated-code shape, runtime helpers. A break here is `2.0.0` / `9.0.0`. -- **`metamodelVersion`** (`"1.0"` at the cut; the current value is the first key of the +- **`metamodelVersion`** (`"1.0"` at the cut, `"1.1"` from 1.1.0; the current value is the first key of the byte-gated `expected-registry.json`) promises the METADATA contract — registered vocabulary, canonical/interchange format, wire contract. A break here moves ITS major, and **does not force a package major.** @@ -620,7 +620,8 @@ How to publish the `MetaObjects*` C# packages to nuget.org. We use **Trusted Pub ## What gets published -Four packages, version-locked at the C# port version (currently `1.0.13`): +Four packages, version-locked at the C# port version (the `` in +[`server/csharp/Directory.Build.props`](../server/csharp/Directory.Build.props)): | Package | Contents | |---|---| @@ -720,8 +721,8 @@ a pre-release you have not committed. ## What gets published -One package, `metaobjects` (version in [`server/python/pyproject.toml`](../server/python/pyproject.toml), -currently `1.0.13`), as an **sdist + a universal `py3-none-any` wheel** (pure Python). +One package, `metaobjects` (version in [`server/python/pyproject.toml`](../server/python/pyproject.toml)), +as an **sdist + a universal `py3-none-any` wheel** (pure Python). ## How we publish: a tag-triggered workflow (API token) @@ -782,7 +783,7 @@ Project `metaobjects` → **Settings → Publishing → Add a new GitHub publish The 14 `com.metaobjects:*` modules (15 artifacts, counting the parent POM) ship to **Maven Central via the Sonatype Central Portal**, versioned on its own major line — npm major + 7, so `7.x` while npm was `0.x` and `8.x` from the -1.0 cut (currently `8.0.13`) — in the parent + module poms. Signed with the maintainer's GPG key. +1.0 cut (the reactor version in [`server/java/pom.xml`](../server/java/pom.xml)) — in the parent + module poms. Signed with the maintainer's GPG key. ## Procedure @@ -868,11 +869,14 @@ four ports at once. A context stamped by a **strictly newer** release is no long Anything not orderable as a plain `N.N.N` still nudges — prereleases, build metadata, and the `0.0.0` unresolved-install sentinel — so the "any drift nudges" property is intact. -The opposite failure is **narrowed, not closed**: equal coordinates still assert in-sync, so a port -parked at `24.4` across several npm releases will not be told its context has moved. Settling that -needs the shipped context hashed rather than its version compared, and the JVM ships no -agent-context content — only the manifest reader. Live limitation, recorded here rather than in an -issue nobody reads. +**Content, not version, decides in TypeScript (1.1.0).** `meta init` stamps a `contentHash` of the +shipped agent-context tree (and the docs ref its links are pinned to) into the manifest, and the +TypeScript check compares that hash: a context whose content differs from what this CLI ships +nudges even at an equal version, and one whose content is identical is silent at any version. A +manifest written before 1.1.0 has no hash and falls back to the version rule above. The C#, +Python and JVM checks keep the version rule, because they ship no agent-context content to hash: +`meta agent-docs` (npm) is the one scaffolder. A context stamped at that port's own version is +the context that release documents, so the version rule is the right answer there. ## Public-repo hygiene diff --git a/docs/compatibility-policy.md b/docs/compatibility-policy.md index 911b7dc96..6efcd3d59 100644 --- a/docs/compatibility-policy.md +++ b/docs/compatibility-policy.md @@ -31,7 +31,7 @@ different parts of your project, and each has its own number | Number | Promises | A break moves | |---|---|---| | **Package version** (npm/PyPI/NuGet `1.x`, Maven `8.x`) | the SOFTWARE surface — your **build** depends on it | the package major (`2.0.0` / `9.0.0`) | -| **`metamodelVersion`** (`"1.0"` at the cut) | the METADATA contract — your **model** depends on it | the metamodel major (Metamodel 2.0) | +| **`metamodelVersion`** (`"1.0"` at the cut, `"1.1"` from 1.1.0) | the METADATA contract — your **model** depends on it | the metamodel major (Metamodel 2.0) | Reading it the other way round: a package major means *your imports, CLI invocations or generated-code shape may need work*. A metamodel major means *your metadata may need @@ -177,6 +177,36 @@ offer the first two is not eligible for this exception. > version gate as a finding that nothing broke. +## What a metamodel MINOR may refuse + +The section above is about a PATCH. A metamodel MINOR (`1.0` → `1.1`) adds vocabulary, and +it may also do one thing a PATCH may not: **refuse input that the written grammar never +allowed, even where some port accepted it**, provided three things hold: + +1. **The grammar is written down and the input contradicts it.** For references that is + [ADR-0042](../spec/decisions/ADR-0042-bare-references-are-package-local.md): the rule a port + departed from, not a rule written to justify the refusal. +2. **The error names the fix.** The message gives the exact edit: the candidates found and + the qualified spelling to write. A conformance fixture asserts those strings in every port + (`messageIncludes`, `spec/conformance-tests.md`). +3. **The changelog lists it under Breaking**, with the ports that used to accept it. + +The difference from the PATCH exception is its second test. A MINOR may refuse a form that +did what its author meant in some port, because the release is where adopters are told to +read the Breaking list and regenerate. A refusal that fails the first test is a break and +waits for Metamodel 2.0, however small. + +**Worked example: a bare cross-package `extends` (1.1.0).** ADR-0042 says a bare name binds +the declaring file's own package, then the root. Java followed it; TypeScript, Python and C# +also searched every other package, so `extends: Base` with `Base` in another package loaded +in three ports and failed in one. 1.1.0 (ADR-0059) makes all five follow the ADR. The PATCH +bar does not admit it (it worked in three ports), and the MINOR bar does: the grammar was +written, `ERR_UNRESOLVED_SUPER` now names the object found in the other package and the +qualified spelling to write (`write extends: "other::Base"`) in all five ports, and the 1.1.0 +changelog lists it under Breaking. `meta upgrade` does not make the edit: it works on one file's +text at a time, and choosing the package needs the whole project's model, where a wrong guess +would load and mean something else. + ## Spec-version support across ports All ports that ship a given release implement the **same** Metamodel spec version, @@ -185,11 +215,10 @@ api-contract, registry). A port's package version tells you its own fix/feature its declared `metamodelVersion` tells you the contract it honors. When they differ, the spec version is authoritative for cross-language interop. -## Pre-1.0 (today) +## Before 1.0 -Until the 1.0 cut, the project is in `0.x` (npm/PyPI/NuGet) / `7.x` (Maven) and the -public API is not yet frozen — breaking changes may ship in a minor, as they did -through the `0.14`–`0.15` vocabulary-finalization window. The -[1.0 readiness checklist](1.0-readiness.md) tracks the remaining path; the -[0.x → 1.0 migration guide](features/migrations/0.x-to-1.0.md) consolidates the -breaking changes adopters absorb at the cut. +Before the 1.0 cut the project was in `0.x` (npm/PyPI/NuGet) / `7.x` (Maven) and the public +API was not frozen: breaking changes shipped in a minor, as they did through the +`0.14`–`0.15` vocabulary-finalization window. The +[0.x → 1.0 migration guide](features/migrations/0.x-to-1.0.md) consolidates the breaking +changes adopters absorbed at the cut. diff --git a/docs/features/api-contract.md b/docs/features/api-contract.md index de3e27a07..7159100c1 100644 --- a/docs/features/api-contract.md +++ b/docs/features/api-contract.md @@ -322,7 +322,7 @@ for the one-line change. | `field.int`, `field.long`, `field.double` | number | `long` MAY be string on overflow; defer to per-port docs. | | `field.boolean` | boolean | – | | `field.date` | string | ISO 8601 calendar date (`YYYY-MM-DD`). Asserted literally by the `report/` sub-corpus (a time dimension's bucket), the first corpus to do so. | -| `field.decimal` | **per port** | Not pinned. Each port sends its own decimal form (TypeScript a string, since the driver reads `numeric` as one), and no scenario asserts a decimal's spelling: precision is the engine's. A client reading decimals from more than one backend must accept a string or a number. | +| `field.decimal` | string, canonical | ADR-0019 Amendment 1. **Out:** plain notation, no exponent, no trailing fraction zeros, zero as `"0"`, matching `^-?(0\|[1-9][0-9]*)(\.[0-9]*[1-9])?$` (`12.50` is sent `"12.5"`). **In:** a string in JSON-number grammar or a JSON number, read exactly; anything else is `400 validation`, and so is a value with more fraction digits than `@scale` or more integer digits than `@precision - @scale` (nothing is rounded). A filter operand uses the same grammar, is compared exactly, and a bad one is `400 invalid_filter_value`. Postgres and MySQL are exact to the declared precision; SQLite and D1 to 15 significant digits. Every decimal (entity, projection, report row) in every port; the `decimal/` sub-corpus asserts it. The wire carries the value, not a presentation: showing `12.5` as `12.50`, or with two digits for money, is a view's job, not the API's. | | `field.time` | string | `HH:MM:SS[.fff]`. | | `field.timestamp` | string | Instant, tz-aware by default (ADR-0036 Wave 2) — `YYYY-MM-DDTHH:MM:SS[.fff]Z`, **always UTC**. | | `field.timestamp` (`@localTime: true`) | string | Naive wall clock — `YYYY-MM-DDTHH:MM:SS[.fff]`, **no `Z`**. | @@ -437,7 +437,7 @@ The surface: mount item routes only for a **single-column** identity, while TypeScript and Python bind a composite one to its first field. - Filters apply to a projection's `field.decimal` and `field.float` fields as they do to - an entity's. A decimal's wire spelling is port-specific and is not asserted. + an entity's. A decimal is the canonical string of "Type encodings". - **The api docs list the same routes.** A projection's api page documents `GET `, and `GET /{id}` only when the item route exists. The `405` refusals are not operations a caller can use, so no write verb is documented. Pinned in every port by @@ -480,7 +480,7 @@ the empty rows, `isNull=true` matches nothing, and a sort on it orders the same engine. Gated by [`fixtures/api-contract-conformance/report/`](../../fixtures/api-contract-conformance/report/) -(25 scenarios), **generated lane only, on all five ports**, for the reason `projection/` +(27 scenarios), **generated lane only, on all five ports**, for the reason `projection/` gives. The columns, their types and the per-port generated files are in [reporting.md](reporting.md#how-a-report-is-served). @@ -499,9 +499,10 @@ above. ```java // AuthorController.java +import com.metaobjects.generator.spring.runtime.FilterParser; +import jakarta.servlet.http.HttpServletRequest; import org.springframework.http.ResponseEntity; import org.springframework.web.bind.annotation.*; -import java.util.List; import java.util.Map; @RestController @@ -512,11 +513,17 @@ public class AuthorController { public AuthorController(AuthorRepository repo) { this.repo = repo; } @GetMapping - public Object list(@RequestParam Map qs) { - var page = repo.find(FilterParser.parse(qs, AuthorFilterAllowlist.INSTANCE)); - return qs.containsKey("withCount") + public ResponseEntity list(HttpServletRequest request) { + // The RAW query string: a servlet container drops a parameter with a malformed escape. + var filter = FilterParser.parse(request.getQueryString(), AuthorFilterAllowlist.FIELDS, + AuthorFilterAllowlist.OPS_BY_FIELD, AuthorFilterAllowlist.DECIMAL_FIELDS); + if (filter.error() != null) { + return ResponseEntity.badRequest().body(Map.of("error", filter.error(), "field", filter.field())); + } + var page = repo.find(filter.predicates()); + return ResponseEntity.ok(request.getParameter("withCount") != null ? Map.of("rows", page.rows(), "total", page.total()) - : page.rows(); + : page.rows()); } @GetMapping("/{id}") public Author get(@PathVariable long id) { return repo.findById(id).orElseThrow(() -> new NotFound()); } @@ -530,6 +537,7 @@ public class AuthorController { ```kotlin // AuthorController.kt +import jakarta.servlet.http.HttpServletRequest import org.springframework.http.ResponseEntity import org.springframework.web.bind.annotation.* @@ -538,9 +546,14 @@ import org.springframework.web.bind.annotation.* class AuthorController(private val repo: AuthorRepository) { @GetMapping - fun list(@RequestParam qs: Map): Any { - val page = repo.find(FilterParser.parse(qs, AuthorFilterAllowlist)) - return if ("withCount" in qs) mapOf("rows" to page.rows, "total" to page.total) else page.rows + fun list(request: HttpServletRequest): ResponseEntity<*> { + // Parse the RAW query string, as the generated controller's parseAuthorFilter does. + val filter = parseAuthorFilter(request.queryString) + filter.error?.let { return ResponseEntity.badRequest().body(mapOf("error" to it, "field" to filter.field)) } + val page = repo.find(filter.predicates) + return ResponseEntity.ok( + if (request.getParameter("withCount") != null) mapOf("rows" to page.rows, "total" to page.total) else page.rows, + ) } @GetMapping("/{id}") fun get(@PathVariable id: Long): Author = repo.findById(id) ?: throw NotFound() diff --git a/docs/features/cube-export.md b/docs/features/cube-export.md index db0280cd2..390481306 100644 --- a/docs/features/cube-export.md +++ b/docs/features/cube-export.md @@ -369,13 +369,17 @@ SQL is exact. ### Reports: rollups and scope segments A served report writes into the cube of its `@from` entity, a TPH subtype's cube included. The -rollup is named after the report. +rollup is named after the report. A report that lists one time dimension at several grains gets +one rollup per grain instead, each at a single granularity and named `_` +(`ProgramsOverTime_year`, `_quarter`, `_day`, written coarsest first); with several such +dimensions, one rollup per combination of grains. A Cube query names a time dimension once, at +one granularity, so this is the form Cube can match. | Report part | In the cube | |---|---| | attribute dimensions | `dimensions: [CUBE.d, …]` in listed order. A `@via` dimension is a member of the `@from` cube, so it is listed the same way. | | one time dimension | `time_dimension: CUBE.d` and `granularity: ` | -| two or more time dimensions, including one dimension at two grains | `time_dimensions:`, a block list with one `{ dimension, granularity }` entry for each, in listed order | +| two or more different time dimensions | `time_dimensions:`, a block list with one `{ dimension, granularity }` entry for each, in listed order | | measures | `measures: [CUBE.m, …]` in listed order | | `@segment` | `segments: [CUBE.]` | | `@filter` | a public segment `Scope` on the cube, where `` is the report's name with a lower-case first letter (`RecentRevenue` gives `recentRevenueScope`), whose `sql` is the filter. It is listed after `@segment` in the rollup's `segments`. | @@ -567,7 +571,7 @@ there. | Ad-hoc queries | Cube lets a caller pick any members. | Nothing here checks the numbers for a combination no report declares. | | Rollup freshness | Cube refreshes a rollup on its refresh key, and the exporter writes none, so Cube's default applies. | A rollup can trail the table until it refreshes. The view is never stale. A report with a relative date gets no rollup for this reason. | | Empty scope | A report with no dimension whose scope reads no rows returns one row from the view: the count `0` and each measure's `@default`. Cube answers from the rollup, which is empty, and sums it to null for every measure, the count and the default included. | `ProgramsEmptyScope` is the one canonical report where the rows differ; the live check requires both answers, so the difference stays an executed check. | -| Several grains | A report that lists one time dimension at several grains gets a rollup naming that dimension once per grain. A Cube query names a time dimension once, at one granularity, so Cube 1.7.43 matches no query to that rollup. | The rollup is declared and never used; Cube answers from the source table and the rows equal the view's. `ProgramsOverTime` is the canonical case. | +| Cube Store precision | Cube Store keeps a rolled-up average to 5 decimal places (`2322.11111`); the source database gives up to 16. | An average answered from a rollup in production mode differs from the view's after the fifth place. | | Encodings | Cube's REST API sends numbers as strings and time buckets as `YYYY-MM-DDTHH:MM:SS.sss`. Where a rollup sends `60`, the view sends `60.0000000000000000`, and where Cube sends `2026-05-01T00:00:00.000`, the view sends `2026-05-01`. | Clients normalize them. The live check does, before it compares rows. | ## Checking it @@ -623,6 +627,27 @@ Cube over a private MySQL 8.4 holding the report views `buildReportViews` lowers the Cube query returns the rows of `SELECT * FROM `, from the rollup when the model has one. Checks 4, 5 and 6 are Postgres only. +The three MySQL mapping cases are executed by a third file (`cube-model-mysql-cases.live.ts`): +their generated trees are loaded into Cube over MySQL 8.4 with seeded `events`, `weeks` and +`calls` tables, and 14 queries (date and timestamp buckets, a tuple distinct count with and +without a condition, a ratio over an empty scope, the boolean measures and their defaults) are +compared with rows worked out by hand from the seed. + +Production mode is a fourth file (`cube-model-production.live.ts`), run on Postgres and on +MySQL: `CUBEJS_DEV_MODE=false`, a signed JWT (a request without one gets `403`), and a separate +`cubejs/cubestore:v1.7.43` container. It builds every rollup through the pre-aggregations jobs +API (`CUBEJS_DEFAULT_API_SCOPES=meta,data,jobs`, since that API is off by default) and waits for +each job. Each served report with a rollup must be answered from Cube Store +(`usedPreAggregations` names `prod_pre_aggregations.`, `external: true`) with the view's +rows, each report without one from the source, `ProgramsOverTime` at day, quarter and year from +its own rollup, and every rollup the model declares must answer at least one query. + +A fifth file (`cube-model-shapes.live.ts`) queries the shapes the canonical model does not +carry, each from its `fixtures/cube-model/` tree on Postgres: the alias cubes (each alias and +both together), the TPH subtype's `sql`, a one-to-one join from both ends, the int-backed enum's +`CASE`, and a two-hop `@spine` report, whose rows, the spine rows with no facts included, must +equal the lowered view's. Each query's rows are compared with plain SQL on the same database. + Run it with `scripts/ci-local.sh --only cube`. The full `scripts/ci-local.sh` runs it after the integration suite, and `--quick` and `--no-integration` drop it. The first run pulls the Cube image, which is about 1 GB. With Docker down, the lane records a SKIP behind a banner, never a @@ -631,11 +656,12 @@ pass; under `--strict-toolchains` it is a failure. In CI it is a `cube` entry in ## Rollups in production -A rollup needs Cube Store. Cube's development mode embeds one, and the live check runs in that -mode only. A production deployment runs Cube Store as a separate service you set up and size; -the exporter does not deploy one. It writes no refresh key and no partition, so those follow -Cube's defaults until you set them. The live check has not run in Cube's production mode, so -the table names above are the development-mode form. +A rollup needs Cube Store. Cube's development mode embeds one; a production deployment runs Cube +Store as a separate service you set up and size, and the exporter does not deploy one. It writes +no refresh key and no partition, so those follow Cube's defaults until you set them. The live +check runs both modes (see [Checking it](#checking-it)); in production mode the rollup tables +are `prod_pre_aggregations.
`, and they must be built (by Cube's refresh worker or the +jobs API) before a query can use them. ## Known limits @@ -645,21 +671,12 @@ the table names above are the development-mode form. refuses the model with `ERR_CUBE_AMBIGUOUS_PATH`, naming both. The lossless form needs a nested alias for each path. That is more machinery than a rare model is worth yet, so the exporter errors rather than guess. -- **The three MySQL mapping cases are compiled, never executed.** The corpus pins their SQL byte - for byte and Cube compiles them, but no MySQL database runs them. The MySQL path runs only the - served reports. The tuple distinct count's MySQL form was compared with the view's on mysql:8.4 - by hand when it was built. -- **Some shapes are compile-checked, not query-checked.** Cube accepts the alias cubes, the TPH - subtype's `sql`, one-to-one joins, the int-backed enum's `CASE` and a two-hop `@spine` chain - (the corpus pass), but no live query of the lane crosses them. The two-hop chain's rows were - compared with the view lowering's by hand on 1.7.43 when it was built. - **A `@spine` report has no rollup.** Its view is answered from the tables. A rollup on the spine cube would lose the zero rows (see [Reports with `@spine`](#reports-with-spine-a-cube-view)). - **No `sqlite` or `d1`.** See [Dialects](#wiring-it). - **No `extends`, no `refresh_key`, no partitions, no `format`.** The exporter writes cubes, and a Cube view only for a served `@spine` report. - **No derived measure.** `measure.derived` is not registered, so there is nothing to map. -- **Rollups in production mode are unverified** (see above). ## Compatibility diff --git a/docs/features/field-types.md b/docs/features/field-types.md index 5eb2ae6f0..9641eb26d 100644 --- a/docs/features/field-types.md +++ b/docs/features/field-types.md @@ -31,7 +31,16 @@ across ports — a `field.currency` is integer minor units everywhere; a That is the whole registered vocabulary — 17 concrete subtypes. `field.base` is an abstract registry anchor, never authored (`ERR_ABSTRACT_SUBTYPE_AUTHORED`). -Three rows need a footnote: +Four rows need a footnote: + +- **`field.decimal`** is exact in every port's own type, and crosses JSON as a canonical + string (`"12.5"`, zero as `"0"`; a JSON number is accepted on input). TypeScript's type is + that string. Postgres and MySQL keep the declared `numeric(@precision, @scale)` exactly; + SQLite and D1 have no decimal type and keep 15 significant digits, so `meta migrate` warns + for a decimal there with `@precision` above 15. Trailing fraction zeros are not kept + (`12.50` crosses as `"12.5"`): the wire carries the value, and display formatting, such as + two digits for an amount of money, is a view concern. The rule is ADR-0019 Amendment 1, and + [api-contract.md](api-contract.md) "Type encodings" has the wire detail. - **`field.uri` / `field.inet`** bind a native URL/IP type on every port but TypeScript, which has none (the same reason `field.uuid` is a `string` there). `@lenient: true` diff --git a/docs/features/migrations/docs-are-generators.md b/docs/features/migrations/docs-are-generators.md index 457ee0ba3..3646b411e 100644 --- a/docs/features/migrations/docs-are-generators.md +++ b/docs/features/migrations/docs-are-generators.md @@ -47,7 +47,9 @@ api reference, the requirement ledger and the agent pages) into `docs.outDir`, f ```ts import { defineConfig } from "@metaobjectsdev/cli"; -import { entityFile, routesFile } from "@metaobjectsdev/codegen-ts/generators"; +// 1.0.x had no packaged `entity` / `routes` export: they were owned copies from `meta eject`. +import { entityFile } from "./codegen/generators/entity"; +import { routesFile } from "./codegen/generators/routes"; export default defineConfig({ outDir: "src/generated", @@ -83,11 +85,12 @@ import { apiDocsFile, docsFile, docsSiteFile, - entityFile, metamodelDocsFile, requirementsFile, - routesFile, } from "@metaobjectsdev/codegen-ts/generators"; +// Your owned copies stay as they were. +import { entityFile } from "./codegen/generators/entity"; +import { routesFile } from "./codegen/generators/routes"; export default defineConfig({ outDir: "src/generated", diff --git a/docs/features/migrations/physical-naming.md b/docs/features/migrations/physical-naming.md index bc223026f..76381b896 100644 --- a/docs/features/migrations/physical-naming.md +++ b/docs/features/migrations/physical-naming.md @@ -84,15 +84,16 @@ regenerate. | 1.0 | 1.1.0 | |---|---| | the `columnNaming` generator arg in `pom.xml` | `"naming": { "columns": "" }` in `.metaobjects/config.json`. `mvn metaobjects:generate` fails while any generator still passes it | -| `SimpleMappingHandlerDB.setColumnNaming(...)` | the rule comes from the loaded metadata; the setter throws. A Spring application loading metadata from the classpath points `metaobjects.physical-naming-config` at the project's `.metaobjects/config.json` packaged as a resource (for example `classpath:.metaobjects/config.json`); the loader sets the rules from that file | -| no arg: Java codegen and the JVM runtime used the field names (`literal`) | no setting: snake_case in Java codegen, Kotlin codegen and the runtime alike. Add `"naming": { "columns": "literal" }` to keep 1.0's column names | +| `SimpleMappingHandlerDB.setColumnNaming(...)` | the rule comes from the loaded metadata; the setter throws. A Spring application reads `classpath:.metaobjects/config.json` when the project packages that file as a resource, or the location `metaobjects.physical-naming-config` names (a `classpath:` location or a file path; a named location that does not exist fails startup). `MetaDataLoader.fromUris` and `fromDirectory` find the `.metaobjects/config.json` above the metadata. With no file found, the loader logs one WARN and uses snake_case | +| no arg, Java codegen and the JVM runtime (OMDB, Java and Kotlin alike): the field names (`literal`) | no setting: snake_case. Add `"naming": { "columns": "literal" }` to keep 1.0's column names | +| no arg, Kotlin codegen (the Exposed tables): already snake_case in 1.0 | no setting: snake_case, unchanged. A Kotlin/Exposed project whose tables came from 1.0 Kotlin codegen adds nothing: adding `literal` would rename its columns | ## Python | 1.0 | 1.1.0 | |---|---| | `metaobjects gen --column-naming `, `metaobjects verify --codegen --column-naming ` | `"naming": { "columns": "" }` in `.metaobjects/config.json`. The flag is refused (exit 2) | -| `ObjectManager(..., column_naming="")` | `ObjectManager(...)`. The manager reads the rule from the loaded root; the argument raises. A model loaded outside the project directory gets the project's rules with `attach_naming(root, project_dir)` from `metaobjects.naming`, which reads that project's `.metaobjects/config.json` and returns the root, so `root = attach_naming(root, project_dir)` | +| `ObjectManager(..., column_naming="")` | `ObjectManager(...)`. The manager reads the rule from the loaded root; the argument raises. `MetaDataLoader.from_directory(...)` and `from_uris(...)` find the `.metaobjects/config.json` above the metadata, and warn once and use snake_case when there is none. `from_string(..., project_dir=...)` takes the project explicitly. A model loaded any other way gets the project's rules with `attach_naming(root, project_dir)` from `metaobjects.naming`, which reads that project's `.metaobjects/config.json` and returns the root, so `root = attach_naming(root, project_dir)` | | no flag: `*_names.py` and the `ObjectManager` used the field names (`literal`) | no setting: snake_case. Add `"naming": { "columns": "literal" }` to keep 1.0's column names | `metaobjects verify --codegen` reports drift that another column rule, another table rule, or diff --git a/docs/features/migrations/upgrading-1.0-to-1.1.md b/docs/features/migrations/upgrading-1.0-to-1.1.md new file mode 100644 index 000000000..a6392b1bd --- /dev/null +++ b/docs/features/migrations/upgrading-1.0-to-1.1.md @@ -0,0 +1,189 @@ +# Upgrading from 1.0 to 1.1 (1.0.x → 1.1.0) + +This is the one page for moving a project from any 1.0.x release to 1.1.0 (npm, PyPI and NuGet +`1.1.0`; Maven Central `8.1.0`). It lists the steps in order and every change that needs +something from you, per port, each with its fix. The per-change detail is in `CHANGELOG.md` +under `[1.1.0]`, and the longer guides it links are written for a coding agent to apply. + +1.1.0 is a MINOR: `metamodelVersion` moves from `1.0` to `1.1`, the vocabulary is additive (a +model that uses none of the new names keeps its meaning), and the reporting feature is new. +Five changes still need action in an existing project, and three of them fail nothing when you +skip them, so read the whole list before you regenerate. + +## Before you start + +- **Upgrade every port together.** All five ports read `.metaobjects/config.json`, and a 1.0.x + reader refuses a key it does not know: once you add `naming` (step 3), a 1.0.x TypeScript, + C#, Java, Kotlin or Python tool in the same estate fails with "Unrecognized key(s) in object: + 'naming'". From 1.1.0 on, an unknown key there is a warning, not a failure, so the next key a + release adds is not the same trap. +- **Mixed-version exports.** A 1.1 `meta export` file can hold a root-level reference written + `::Name`, which 1.0.x refuses (`ERR_RELATIVE_REF_IN_CANONICAL`). Read 1.1 exports with 1.1 + tools. +- **Commit first.** Every step below shows its effect as a diff; a clean tree makes the diff + the whole story. + +## The steps + +1. **Bump every MetaObjects package together**: all `@metaobjectsdev/*` at `1.1.0`, + `metaobjects` `1.1.0` (PyPI), `MetaObjects*` `1.1.0` (NuGet), `com.metaobjects:*` `8.1.0` + (Maven, including the `metaobjects-maven-plugin` version in each `pom.xml`). Reinstall + cleanly and read the resolved versions back. +2. **`meta upgrade`**, then `meta upgrade --apply` if it reports anything. Then load the model + (`meta verify`): a bare cross-package `extends` now fails with `ERR_UNRESOLVED_SUPER`, and the + message names the spelling to write (see "References and `extends`"). +3. **Set the physical naming rule** if your database's columns are named like the fields (C#, + Java, the JVM runtime and Python; see "Column and table names"). This is the step that + fails nothing when skipped. +4. **Move documentation into the generator list** if you ran `meta docs`, `dotnet meta docs`, + `metaobjects docs` or the `metaobjects:docs` goal (see "Documentation is generated"). +5. **Java and Kotlin: let `mvn metaobjects:generate` write the hash manifest, then commit + `.metaobjects/.gen-state/.hashes.json`** in every module (see "JVM hash manifest"). +6. **Resync the generators you own.** `meta eject --list` (TypeScript; `metaobjects eject + --list`, `dotnet meta gen --list`, `mvn metaobjects:eject -Dlist=true` in the other ports) + reports each owned copy, and every file it owns beside it (in TypeScript the renderers + under `codegen/templates/`), as identical to the 1.1.0 reference or not. An owned copy keeps the logic it was copied with: it gets no + report output and none of the 1.1 fixes until you resync it. For a copy you never edited, + `meta eject --force` (Java: `mvn metaobjects:eject -Dnames= -Dforce=true`) + takes the new reference; for one you edited, three-way merge it. +7. **Regenerate** (`meta gen`, `dotnet meta gen`, `metaobjects gen`, `mvn metaobjects:generate`), + then build and run your tests. Read the generated diff against "Diffs to expect". +8. **Run the drift gates**: `meta verify --codegen` (and your port's `verify`). They should be + clean; a column-name mismatch is reported naming the `naming` setting. +9. **Refresh the agent context**: `meta init --docs-only --refresh-docs` (or `npx meta + agent-docs`). A file you edited is left alone and the new copy written beside it as + `.new`; merge it by hand and delete the `.new`. + +## Changes that need action + +### Column and table names (C#, Java, the JVM runtime, Python) + +Column and table names the metadata does not give come from one `naming` setting in +`.metaobjects/config.json`, defaulting to `snake_case` in every port. In 1.0 the default in C#, +Java codegen, the JVM runtime (OMDB, Java and Kotlin) and Python was the field name. A database +whose columns are named exactly like the fields adds: + +```json +{ "naming": { "columns": "literal" } } +``` + +The per-port flags that carried the rule are removed and refused with a message naming the +setting: `--column-naming` (C#, Python), the `columnNaming` generator arg (Java/Kotlin pom), +`SimpleMappingHandlerDB.setColumnNaming(...)` (Java), `ObjectManager(column_naming=)` (Python) +and `columnNamingStrategy` (TypeScript, whose default was already `snake_case`). Kotlin +codegen's Exposed tables were already `snake_case` in 1.0: a Kotlin/Exposed project whose tables +came from 1.0 Kotlin codegen adds nothing. + +The runtimes read the same file codegen reads: + +- **Python** — `MetaDataLoader.from_directory(...)` and `from_uris(...)` find + `.metaobjects/config.json` by walking up from the metadata directory (or the first file URI); + `from_string(..., project_dir=...)` takes the project explicitly. A model loaded any other way + (a package resource) takes the project's rules with `attach_naming(root, project_dir)` from + `metaobjects.naming`. +- **Spring** — the runtime reads `classpath:.metaobjects/config.json` when it is on the + classpath (package the project's file as a resource), or the location + `metaobjects.physical-naming-config` names (a `classpath:` location or a file path). +- **Java and Kotlin outside Spring** — `MetaDataLoader.fromUris(...)` and `fromDirectory(...)` + read the `.metaobjects/config.json` above the metadata; set the rules explicitly with + `root.setPhysicalNaming(SourceResolver.resolvePhysicalNaming(projectDir))` for any other + source. +- When a runtime finds no project config it says once that it is using the `snake_case` + default, so a missing file is visible in the log rather than as a query against a column + that does not exist. + +Full table, per port: [physical-naming.md](physical-naming.md). + +### Documentation is generated (every port) + +Every generator, documentation included, is an entry in the project's generator list; one +command runs the list, and nothing is generated that the list does not declare. The docs +commands are removed and exit 2 naming where their work went: `meta docs` and +`meta verify --docs` (TypeScript), `dotnet meta docs` (C#), `metaobjects docs` (Python), the +`metaobjects:docs` goal (Java/Kotlin). A gen command given a 1.0 docs configuration fails before +it writes anything. Before and after configs per port, and the one-time `--baseline=fresh` run +that records the existing pages: [docs-are-generators.md](docs-are-generators.md). + +### JVM hash manifest (Java and Kotlin) + +`mvn metaobjects:generate` records what it writes in `.metaobjects/.gen-state/.hashes.json`, +the committed manifest the other ports already use, and decides overwrites from it. An edited +generated file is refused by name (1.0 overwrote it), and a file no generator emits any more is +deleted. The first run adopts your existing generated files and tells you to commit the +manifest. Details, including the two-executions case: [jvm-codegen-hash-manifest.md](jvm-codegen-hash-manifest.md). + +### References and `extends` (TypeScript, Python, C#) + +A bare `extends` no longer binds an object in another package (ADR-0042, ADR-0059). In 1.0, +TypeScript, Python and C# searched every package; Java did not. Now a bare name binds the +declaring file's own package, then the root, in all five ports. The load error names the fix: + +``` +ERR_UNRESOLVED_SUPER: … An object named "Base" exists in: acme::common::Base. +Qualify it with its package (FQN): write extends: "acme::common::Base". +``` + +Make that edit. When two or more packages declare an object of that name the message lists +them all; pick the one the object meant. In 1.0 the TypeScript, Python and C# loaders bound the +first match in load order, so with one candidate the qualified spelling keeps the meaning it +had. `meta upgrade` does not make this edit: choosing the package needs the whole model. + +### Decimals on the wire (every port) + +A `field.decimal` crosses the generated API as one canonical string in every port (ADR-0019 +Amendment 1): `12.50` is sent `"12.5"`, zero `"0"`, never a number and never an exponent. In +1.0 C#, Java, Kotlin and Python sent a number. A client that reads a decimal from those ports +parses a string now; format it for display with the field's `@scale` (display is a view's job). +On input a string or a JSON number is accepted, read exactly, and a value with more fraction +digits than `@scale` or more integer digits than `@precision - @scale` is `400 validation` +rather than rounded. A decimal filter operand is compared exactly; a bad one is +`400 invalid_filter_value`. On SQLite and D1 a decimal keeps 15 significant digits, and +`meta migrate` warns for one declared with `@precision` above 15. An owned routes generator +keeps its 1.0 decimal handling until you resync it (step 6). + +## Diffs to expect after regenerating + +- **References are written fully qualified (ADR-0059).** `meta export` and canonical + serialization write every reference as `pkg::Name` (a root-level target referenced from a + package as `::Name`), and a model spanning packages gets a `package` on each top-level object. + The neutral template data's `targetRef` carries the same canonical value + (`rel product -> shop::Product`), so a Mustache template that prints it changes. + `meta fmt` and documentation pages keep references as you wrote them. +- **Inherited references resolve in the declaring package (ADR-0059 Decisions 3 and 7).** A reference + an object inherits through `extends` resolves where the base declares it, not where the + inheriting object lives. A model that relied on the 1.0 behaviour (the same bare name meaning + different objects in base and subtype packages) now binds the base's. +- **Orphaned generated files are deleted.** The first full `meta gen` after the upgrade deletes + generated files that no generator in the list emits any more (the hash manifest records what + was generated). Review the deletions in the diff; a file you want to keep, take out of the + manifest or rename. +- **A keyless projection regenerates differently.** No `/:id` route, no by-id function, and no + write verbs in its API docs; `meta verify --codegen` reports those pages stale until you + regenerate. +- **A multi-hop enum field references the shared enum** in every port, instead of getting an + enum type of its own; Java and Kotlin have two more enum diffs and two build settings + (`emitSharedEnums=true`, an `` entry on `SpringValueObjectGenerator` or + `SpringOutputParserGenerator`; Kotlin's `providedEnumNamespace` / `providedEnumPackages` in the + plugin's ``). See the CHANGELOG entry. +- **Decimal wire files.** C# writes `DecimalWire.g.cs` per namespace and Kotlin + `MetaDecimalWire.kt` per package; Java's DTOs and allowlists gain decimal annotations and + `DECIMAL_FIELDS`; TypeScript and Python routes call their decimal wire. See "Decimals on the + wire". +- **C# array properties are `IList`** (scalar and enum arrays on entities, TPH subtypes, + value objects and report rows; they were `ICollection`, which EF Core 8 cannot read on + SQLite). Code that assigns a `HashSet` to one assigns a `List`. +- **Owned Java Spring generators** made before 1.1.0 keep their old rules: re-eject with + `mvn metaobjects:eject -Dnames= -Dforce=true` and merge your edits back from + `git diff`. + +## CLI behaviour that changed + +- `meta init` in a project that is already initialised is a no-op that exits 0 (1.0 exited 1, + "already exists; use --force"). `--force` still re-scaffolds. + +## New in 1.1, nothing to do + +Reporting (dimensions, measures, segments, `object.report`, served read-only routes, the Cube +exporter): [reporting.md](../reporting.md). Requirements (the existence and coverage gate in +every port's `verify`): [requirements.md](../requirements.md). A project that declares neither +sees no change from them. diff --git a/docs/features/migrations/upgrading-within-1.0.md b/docs/features/migrations/upgrading-within-1.0.md index 93e6589b9..28c4dc05e 100644 --- a/docs/features/migrations/upgrading-within-1.0.md +++ b/docs/features/migrations/upgrading-within-1.0.md @@ -1,5 +1,9 @@ # Upgrading within 1.0 (1.0.x → a later 1.0.x) +> **Moving to 1.1?** This page covers 1.0.x patches only. From any 1.0.x to 1.1.0 (Maven +> 8.1.0), follow [upgrading-1.0-to-1.1.md](upgrading-1.0-to-1.1.md), which lists every change +> that needs action, per port. + A 1.0 patch never asks you to change metadata that worked, and `metamodelVersion` has stayed `1.0` since the 1.0 cut. It can still change the code `meta gen` writes, and some of those changes reach your app. This page lists what came up when nine adopter projects moved from diff --git a/docs/features/own-your-codegen.md b/docs/features/own-your-codegen.md index 0be0e3d3e..dd422c441 100644 --- a/docs/features/own-your-codegen.md +++ b/docs/features/own-your-codegen.md @@ -1126,7 +1126,11 @@ Eject never edits an existing pom. It prints the three things to wire: the `` on the codegen module for the module that runs `metaobjects:generate`, and the new `` for each ``. It never overwrites a copy without `-Dforce`, and `-Dlist` marks each owned copy `identical` -or `DIFFERS: N behind, M of your own`. +or `DIFFERS: N behind, M of your own`. A copy is `identical` only when every file it owns is: +the classes copied with it, its templates, the shared support classes it imports and the helper +runtime its output imports. A differing one is named +(`[owned — DIFFERS: 0 behind, 1 of your own; differing: api/index.md.mustache]`), so a resync +that took the generator and left a template behind still shows. A name that is ejectable on both ports needs `-Dport`, unless your project declares a dependency on exactly one of `metaobjects-codegen-spring` and `metaobjects-codegen-kotlin`, @@ -1345,11 +1349,11 @@ on are yours too. Eject copies two kinds of source with a Java generator: | Ejecting | Shared emit rules, beside the generator | Helper runtime its output imports | |---|---|---| -| `routes` | `SpringGeneratorSupport` | `ConstraintErrors`, `FilterParseResult`, `FilterParser`, `FilterPredicate`, `PatchValidationException`, `RecordComponentNames` | -| `dto` | `SpringGeneratorSupport` | `PatchValidationException` | +| `routes` | `SpringGeneratorSupport` | `CanonicalDecimal`, `ConstraintErrors`, `FilterParseResult`, `FilterParser`, `FilterPredicate`, `PatchValidationException`, `RecordComponentNames`, `UnreadableBody` | +| `dto` | `SpringGeneratorSupport` | `CanonicalDecimal`, `CanonicalDecimalDeserializer`, `CanonicalDecimalSerializer`, `DeclaredDecimal`, `PatchValidationException` | | `repository` | `SpringGeneratorSupport` | `FilterPredicate` | -| `value-object` | `SpringGeneratorSupport` | nothing | -| any other Java generator, every Kotlin generator | nothing | nothing: its output imports only core | +| `value-object` | `SpringGeneratorSupport` | `CanonicalDecimal`, `CanonicalDecimalDeserializer`, `CanonicalDecimalSerializer`, `DeclaredDecimal` | +| any other Java generator, every Kotlin generator | nothing | nothing: its output imports only core (Kotlin writes its decimal wire into the output as `MetaDecimalWire.kt`) | **The shared emit rules.** `SpringGeneratorSupport` holds the rules those four generators apply together: which fields a DTO and a patch carry, which of them are stamped for diff --git a/docs/features/reporting.md b/docs/features/reporting.md index 148c984f7..75a8fc72f 100644 --- a/docs/features/reporting.md +++ b/docs/features/reporting.md @@ -23,8 +23,8 @@ need), a rollup pre-aggregation for each served report, and a Cube view for a se with `@spine`. See [Exporting to Cube](#exporting-to-cube) and [cube-export.md](cube-export.md). **What does not exist yet.** No grid, form or other UI-tier output is generated for a report -in any port; TypeScript's list hook is the one client piece (a later plan of FR-044 adds the -rest). There is no dbt MetricFlow exporter (it is built when an adopter asks), no `measure.derived`, +in any port; TypeScript's list hook is the one client piece (the 1.2 `reporting` library +adds the rest). There is no dbt MetricFlow exporter (it is built when an adopter asks), no `measure.derived`, no query-time choice of dimensions or measures (a report is a fixed combination, compiled once), and no time-zone vocabulary: time grains and relative dates are UTC. @@ -231,7 +231,7 @@ and the one every runtime reads. | `source.rdb` with `@kind: view` | A derived view. `meta migrate` creates `CREATE VIEW `, where the name is the source's `@view` (or the legacy `@table`). | | the same, plus `@sql` | Your SQL is the body, exactly as for a projection. The column shape below still defines what the runtime reads. | | the same, plus `@unmanaged: true` | `meta migrate` never creates or drops it (you or a migration tool own the DDL), but the runtime still reads it through the shape below. | -| `@kind: materializedView`, `storedProc` or `tableFunction` | `meta migrate` skips it, as for a projection. | +| `@kind: materializedView`, `storedProc` or `tableFunction` | Refused at load (R10). Declare a view you create yourself as `@kind: view` with `@unmanaged: true`. | A report whose view body is your own SQL (`@sql`) or that you manage yourself (`@unmanaged`) still gets the derived read shape below, so a defaulted measure and a `@spine` key column are @@ -440,8 +440,9 @@ database and in every port's row type. Declare it only where zero is true. A program with no purchases has revenue `0`. An average rating with no ratings is not `0`; null is the honest answer there. -- **It is an integer.** Every case found is zero; a sentinel such as `-1` is legal too. Write - a whole number with no decimal point (see [Known limits](#known-limits)). +- **It is an integer.** Every case found is zero; a sentinel such as `-1` is legal too. A + number with an integral value (`0`, `0.0`, `-1.0`) loads as that integer in every port; a + fraction (`0.5`) is `ERR_BAD_ATTR_VALUE`, as on any integer attribute. - **Where it is legal.** A `sum`, an `avg`, a `min` or `max` over a numeric field, an `any` or `all` (as `0` or `1`, see [Boolean measures](#boolean-measures-any-and-all)), and a ratio. It is refused on a `count`, which is never null, and on a `min` or `max` over a field that is @@ -538,11 +539,11 @@ with an error naming the report and the dimension or measure. ### How a report is served -**Which reports.** One rule in five ports: a report is served when it is not abstract and its -read source has `@kind: view` (with or without `@sql`, with or without `@unmanaged`; the view -is assumed to exist). A report with no `source.*`, an abstract report, and a report whose -read source is a `materializedView`, `storedProc` or `tableFunction` are not served: nothing -is generated for them and nothing is mounted. +**Which reports.** One rule in five ports: a report is served when it has a `source.rdb` with +`@kind: view` (with or without `@sql`, with or without `@unmanaged`; the view is assumed to +exist). A report with no `source.*` is not served: nothing is generated for it and nothing is +mounted. An abstract report and a report source of any other read-only `@kind` do not load +(R10, R11). **The surface.** `` is the report's name, `snake_case`d and then pluralized, the rule every object uses ([api-contract.md](api-contract.md)). @@ -561,8 +562,9 @@ return every row when `limit` is omitted, Java, Kotlin and Python the first 50. **What a caller may filter and sort on.** Every derived field whose type has filter operators (string, enum, uuid, int, long, double, float, decimal, currency, date, time, timestamp, boolean), dimension and measure alike, with exactly the operators that type has on any other -object, and each such field sorts (default direction `asc`). A derived field typed `object` -or `map` is neither filterable nor sortable. The allowlists are the report's **own** derived +object, and each such field sorts (default direction `asc`). A derived field over an array +(`isArray`) or a `map` is neither filterable nor sortable, in every port: filtering or sorting +on one is `400 invalid_filter_field` or `400 invalid_sort`, and the `report/` corpus asserts both. The allowlists are the report's **own** derived fields: a field of the `@from` entity that the report does not expose is refused with `400 invalid_filter_field` or `400 invalid_sort`. No vocabulary was added for this; the generators mark each derived field filterable on the detached read model, and nothing is @@ -578,7 +580,7 @@ encodings"), applied to the column types above: | `currency` | integer minor units | | `date` (a time dimension at `day`, `week`, `month`, `quarter` or `year`) | `YYYY-MM-DD`, the first day of the bucket | | `timestamp` (a time dimension at `hour`) | an instant with `Z`, or naive without for a `@localTime` field | -| `decimal` (`avg`, a ratio, a `sum` of a decimal) | **the port's own decimal spelling.** Not part of the contract: precision is the engine's, and the ports do not agree on one JSON form (TypeScript sends a string, on SQLite as on Postgres: SQLite computes the value as a REAL and the generated route sends it as its string) | +| `decimal` (`avg`, a ratio, a `sum` of a decimal) | a canonical string, as for every decimal ([api-contract.md](api-contract.md), ADR-0019 Amendment 1): `"12.5"`, zero as `"0"`, no exponent. The digits are the engine's: Postgres and MySQL are exact, SQLite and D1 exact to 15 significant digits. | | a null value | `null`, with the key present | **What each port generates for a served report.** `` is the report's name. @@ -648,47 +650,16 @@ emitted. - **A derived report from a TPH subtype is refused.** Declare it from the base with an `@filter` on the discriminator field (see "Which reports lower"). -- **An abstract view-backed report is not served, and gets no C# row class and no Kotlin table - object.** No port generates a route for it. The TypeScript, Java and Python runtimes still - read it. The same holds for a report whose source `@kind` is `materializedView`, `storedProc` - or `tableFunction`: no port's generators emit anything for it, `meta migrate` skips it, and - the three runtimes issue a `SELECT` against whatever relation the source names. That works - for a materialized view you created and is a database error for a stored procedure or a - table function. -- **A report over a `field.object` is not supported across ports.** A dimension whose `@of` is a - `field.object` (or a field carrying `@objectRef`) loads in every port. Java OMDB then refuses - the read, and Java, Kotlin and C# `gen` refuse to generate anything for the served report, - each with an error naming the report and the dimension (the same refusal covers a measure - whose column is typed by such a field). TypeScript and Python read it, serve it and return - the parsed JSON; the field is neither filterable nor sortable. Group by a scalar field. - **A report cannot narrow what is filterable.** Every derived field with filter operators is filterable and sortable. A report declares no fields, so there is nowhere to write `@filterable`, and registering it on a dimension or measure would be new vocabulary. - **The route segment follows the object rule, with no report spelling.** `InvoicesByMonth` is served at `/invoices_by_months`, not `/invoices_by_month`; `StoreTotals` at `/store_totals`. -- **A decimal's JSON spelling differs by port.** An `avg`, a ratio and a `sum` of a decimal - are decimals. Each port sends its own form and the corpus asserts none of them, so a client - that reads one from two backends must parse both a string and a number. -- **An array- or map-valued derived field is not gated, and the ports' sort allowlists differ - for one.** A dimension over an `isArray` field takes its element subtype's filter operators - in every port. For sorting, C# leaves an array out of its sort allowlist; TypeScript, Java, - Kotlin and Python decide by subtype alone and so list it. A `map`-typed field has no filter - operators and is left out of the sort allowlist by TypeScript, C#, Java and Kotlin; Python's - router drops only `object` fields, so it would list a `map` one. The corpus has no array or - map dimension, so none of this is asserted: do not rely on filtering or sorting one across - ports. - **A list hook and nothing else of a client.** A served report has a route, a row type and, in TypeScript, the list hook. A grid, a form or a dashboard over reports is yours to write until the `reporting` library covers it. -- **With `@via`, `@of` must name an entity that has the field** (declared on it or inherited by - it); naming a base of the reached entity for a field only the subtype declares loads and then - fails `meta migrate`. The quiet form of the same rule: a `@via` dimension reads its field from - the entity `@of` names, so when the reached subtype **redeclares** that field and `@of` names - the base, the view selects the base's column and type with no error. Qualify `@of` with the - subtype that declares the field you mean. Without `@via` the field is read from the `@from` - entity itself. -Each of the next five is refused at load today, so admitting it later is additive: +Each of the next four is refused at load today, so admitting it later is additive: - **One spine per report.** Two independent spines (every program against every customer) would be a cross join the size of both tables multiplied. Entities to-one beyond the spine @@ -701,14 +672,6 @@ Each of the next five is refused at load today, so admitting it later is additiv - **No dimension over the reference column itself** (`Purchase.programId`) in a `@spine` report. Declare the dimension over the spine entity's key (`Program.id` via `Purchase.program`) instead. -- **`@default` is an integer, and a whole number is the only portable spelling.** A - fractional `@default` (`0.5`) is refused with `ERR_BAD_ATTR_VALUE` on the measure in every - port. TypeScript says "@default '0.5' is not an integer. A measure's @default is a whole - number (for example 0)."; C#, Java and Python refuse it through their existing integer - attribute check, with that check's own message. Two edge cases have no fixture and differ - by port: `@default: 0.0` (or `1.0`) loads as an integer in TypeScript and is refused by C#, - Java and Python; and `@default: "zero"` on a `count` gives two errors in TypeScript, C# and - Python (the attribute type error and the count rule) and one in Java. Write `0`, not `0.0`. A `@spine` row's facts are reached through a declared reference only. A join on a natural key (events matched to a schedule on several columns), an anti-join ("bought, never started") and @@ -717,11 +680,11 @@ the anti-join by hand, and make two reports. ### What the corpus gates -Twenty-one shared scenarios under `fixtures/persistence-conformance/queries/report-*.yaml` read the +Twenty-four shared scenarios under `fixtures/persistence-conformance/queries/report-*.yaml` read the canonical reports through every port's runtime (list and count, filter, sort, an empty table, the Monday boundary, an hour bucket, a relative window, a `@spine` report with a program that has no weeks, a `@spine` report whose segment leaves a program no facts, and defaulted measures -over an empty table; and, beyond those, every `@agg` kind over int, long, float, double, decimal, currency, date and instant columns, a nullable int-backed enum, a boolean and a uuid as dimensions, `day`, `quarter` and `year` grains together, every filter operator on a measure, a report `@segment` and `@filter` together, a two-hop `@via`, a self-referencing dimension, a report declared from a TPH base, and a negative `@default`). The derived columns are pinned by +over an empty table; and, beyond those, every `@agg` kind over int, long, float, double, decimal, currency, date and instant columns, a nullable int-backed enum, a boolean and a uuid as dimensions, `day`, `quarter` and `year` grains together, every filter operator on a measure, a report `@segment` and `@filter` together, a two-hop `@via`, a self-referencing dimension, a report declared from a TPH base, a negative `@default`, `any` and `all` over a boolean (also on a `@spine` report), exact decimal aggregates, and a `@via` dimension over a field a subtype redeclares). The derived columns are pinned by `fixtures/persistence-conformance/report-shapes.json`, produced by TypeScript and byte-matched by every port. The SQL is produced by TypeScript only, so the other ports read the view the TypeScript migrate engine produced and never lower a report themselves. TypeScript's own value @@ -729,15 +692,17 @@ tests read the `@spine` and `@default` views and their column types on Postgres, MySQL, and on Postgres and SQLite also a fact whose reference is null, an `isNull` condition and a two-hop spine. -The twenty-one report scenarios run on more than Postgres, wherever a port's runtime has the engine: +The twenty-four report scenarios run on every engine a port's runtime has, and so do the +scenarios whose `engines:` key lists more than Postgres (the decimal ones: `decimal-exact` on +Postgres and MySQL, which hold the declared precision, and its `-15` twin everywhere): | | Postgres | SQLite | MySQL 8.4 | D1 (local runtime) | |---|---|---|---|---| | TypeScript | ✓ | ✓ | ✓ | ✓ | | C# (EF Core) | ✓ | ✓ | ✓ (Pomelo) | n/a | | Kotlin (Exposed) | ✓ | ✓ | ✓ | n/a | -| Java (OMDB) | ✓ | not run: OMDB has no SQLite driver | ✓ | n/a | -| Python | ✓ | not run: the runtime ships a Postgres driver only (pg8000) | not run: same | n/a | +| Java (OMDB) | ✓ | ✓ | ✓ | n/a | +| Python (`ObjectManager`) | ✓ (pg8000) | ✓ (`sqlite3`) | ✓ (PyMySQL) | n/a | Each engine reads the artifact TypeScript produced for it: `canonical/schema.sqlite.sql` is the output of `meta migrate --dialect sqlite`, and `canonical/schema.mysql.sql` is the @@ -745,16 +710,13 @@ adopter-written tables plus a `CREATE VIEW` per report from `buildReportViews`. writes view SQL, and a test holds each artifact equal to its generator. The expectations stay the Postgres wire ones, never loosened: the engine's own spelling (a REAL ratio on SQLite, a scale-keeping `DECIMAL` and a zoneless `DATETIME` on MySQL) is mapped on the actual side by -each runner, and `report-relative-date` spells its seed per engine (`seed-data-engine`). D1 -has no separate corpus: it is SQLite behind Cloudflare's binding, so TypeScript runs the same -scenarios and the same twenty-five REST scenarios through Miniflare's local D1 (`meta migrate ---dialect d1`), with no cloud account. +each runner, and `report-relative-date` spells its seed per engine (`seed-data-engine`). A +decimal is exempt from that mapping: every runner requires its port's exact decimal type and +compares the canonical string. D1 has no separate corpus: it is SQLite behind Cloudflare's +binding, so TypeScript runs the same scenarios and the same REST scenarios through Miniflare's +local D1 (`meta migrate --dialect d1`), with no cloud account. -A port without the engine is a stated limitation, not a gap to fill by building a driver: -Java's OMDB and Python's `ObjectManager` ship the drivers they ship, and a view-reading test -does not justify a new one. - -The REST surface is gated by twenty-five scenarios under +The REST surface is gated by twenty-seven scenarios under [`fixtures/api-contract-conformance/report/`](../../fixtures/api-contract-conformance/report/), run in the **generated lane on all five ports**: list (a dimension with a segment-scoped sum that is null for one group), a time dimension at a grain (`YYYY-MM-DD`), a no-dimension @@ -763,15 +725,13 @@ on a measure and an enum dimension, paging over groups, the three field-naming ` `405` on `POST`, `404` on every verb at `/{id}`, and a `@spine` report whose product with no sale keeps its row, with a filter and a sort on its defaulted measure, a dimension reached through two references, one time dimension listed at two grains, a filter on a string (`like`), a boolean and a -uuid dimension, and a filter on a measure that carries its own `@filter`. The corpus model -carries one sourceless report, so a port that serves every report it finds fails. No scenario asserts a decimal's -spelling or a timestamp literal. TypeScript and C# run the scenarios against the real views on -Postgres, SQLite and MySQL (`report/schema.sqlite.sql`, `report/schema.mysql.sql`), and -TypeScript also through local D1; TypeScript also holds that a ratio reaches the wire as a -string on SQLite, as it does on Postgres. Java, Kotlin and Python serve seeded rows behind -their repository seam, so they read no view and no engine reaches them: running that lane on -SQLite or MySQL would test nothing the seam does not already, and a TypeScript test holds -those rows equal to what the views return. +uuid dimension, a filter on a measure that carries its own `@filter`, and a dimension over an +array and one over a map, which refuse a filter (`invalid_filter_field`) and a sort +(`invalid_sort`). The corpus model carries one sourceless report, so a port that serves every +report it finds fails. Every port runs the scenarios through its generated routes against the +real views on Postgres, SQLite and MySQL (`report/schema.sqlite.sql`, `report/schema.mysql.sql`), +and TypeScript also through local D1. A report's decimal columns are asserted as canonical +strings by the `decimal/` sub-corpus, on the same engines. ## Exporting to Cube @@ -813,10 +773,11 @@ a rule differently fails the corpus. | Rule | Refused | |---|---| -| **D1** | An `@of` that is not `Entity.field`, that names the wrong entity, or whose field does not exist. Without `@via`, `Entity` is the owning entity; with `@via`, it is the `@via` terminal, and a bare `Entity` resolves in the terminal's package. | +| **D1** | An `@of` that is not `Entity.field`, that names the wrong entity, or whose field does not exist. Without `@via`, `Entity` is the owning entity; with `@via`, it is the `@via` terminal, and a bare `Entity` resolves in the terminal's package. With `@via`, the field must be declared on, or inherited by, the entity `@of` names: naming a base for a field only the reached subtype declares is refused, and the message names the subtype to write. When the reached subtype redeclares a field the base has, the subtype's declaration (its column and type) is what the view selects. | | **D2** | An `@via` that is not `Owner.hop[.hop...]` starting at the owning entity, or that crosses a hop which is not a `relationship.*` with `@cardinality: one` or an `identity.reference`, or a hop whose target resolves to no object. | | **D3** | A `dimension.time` whose `@of` is not a `field.date` or `field.timestamp`. | | **D4** | A `dimension.time` over a `field.date` that declares `hour` (a date has no hour). | +| **D5** | An `@of` naming a `field.object`, or any field carrying `@objectRef`. A report groups by a scalar: group by a scalar field of the value instead. A `field.map` with `@valueType` is legal. D3, M2, M3 and M4 do not run on a dimension D5 refused. | For D1, D2 and M1, the `Entity` part may name the owning entity **or an entity it extends**, so a member declared on an abstract base and written `BaseEvent.occurredAt` validates under @@ -841,13 +802,15 @@ row, and a `SUM` over the result would double-count. A to-one hop cannot fan out | **M7** | A `@default` on an `@agg: count`. A count is never null, so the default could never apply. | | **M8** | A `@default` on an `@agg: min` or `max` over a field that is not numeric (not one of `int, long, double, float, decimal, currency`). | | **M9** | A `@default` on an `@agg: any` or `all` other than `0` (false) or `1` (true). | +| **M10** | An `@of` naming a `field.object`, or any field carrying `@objectRef`, for any `@agg`. A measure aggregates scalars: aggregate a scalar field of the value instead. | A measure that breaks several of M1 to M4 reports only the first, in that order, so one mistake gives one error. M7, M8 and M9 run only when none of M1 to M4 fired. -**A `@default` must be a whole number** (`ERR_BAD_ATTR_VALUE`, on the measure). A fraction is -refused in every port, and then M7, M8 and M9 are not checked. A ratio's `@default` is checked for -this only. The message and two edge cases differ by port: see [Known limits](#known-limits). +**A `@default` must be a whole number** (`ERR_BAD_ATTR_VALUE`, on the measure), the check +every integer attribute gets. `0.0` loads as `0`; a fraction (`0.5`) or a non-number (`"zero"`) +is refused, and then M7, M8 and M9 are not checked, so the mistake gives one error in every +port. A ratio's `@default` is checked for this only. ### Segments and filters (`ERR_BAD_ATTR_FILTER`) @@ -868,6 +831,8 @@ this only. The message and two edge cases differ by port: see [Known limits](#kn | **R7** | `ERR_INVALID_REPORT` | A report `@segment` that names no segment of `@from`. | | **R8** | `ERR_INVALID_REPORT` | A `@spine` that is not `Owner.hop[.hop...]` with `Owner` resolving to `@from` or an entity it extends (a bare `Owner` resolves in `@from`'s package), or that crosses a hop which is not a `relationship.*` with `@cardinality: one` or an `identity.reference`, or a hop whose target resolves to no object. This is D2's walk, started at `@from`. | | **R9** | `ERR_INVALID_REPORT` | A `@spine` report with no dimension, or a listed dimension whose `@via` does not begin with the spine's hops (no `@via` at all included). Hop names are compared as written; the owner segment is not. | +| **R10** | `ERR_INVALID_REPORT` | A read-only report source whose effective `@kind` is not `view` (`materializedView`, `storedProc`, `tableFunction`), on the `source.rdb`. A report is read through a view; for a view or materialized view you create yourself, declare `@kind: view` with `@unmanaged: true`. | +| **R11** | `ERR_INVALID_REPORT` | A report declared `abstract: true`. Its fields are derived from its own `@dimensions` and `@measures`, so there is nothing for a subtype to add, and no port reads or serves an abstract report. | `purchasedAt:day` and `purchasedAt:week` together are legal, since they derive different fields (`purchasedAtDay`, `purchasedAtWeek`). @@ -914,9 +879,11 @@ load, and a passing load says nothing about a query against it: there is none. All additions are additive. `metamodelVersion` moved `1.0` to `1.1`, a MINOR on the metamodel axis and on every registry ([compatibility-policy.md](../compatibility-policy.md)). -A model that does not use the new names generates exactly as before. What loads changes only -where the same change fixed Java and Python parsing bugs, each toward what TypeScript already -did; the [CHANGELOG](../../CHANGELOG.md) lists them and the models they affect. Until 1.1 +A model that declares no reporting vocabulary gets nothing new from it. Outside reporting, what +loads changes in two places: a bare `extends` naming an object in another package is refused in +TypeScript, Python and C#, and fixes to Java and Python parsing bugs move those ports toward what +TypeScript already did. The [CHANGELOG](../../CHANGELOG.md) lists them and the models they affect, +and [upgrading-1.0-to-1.1.md](migrations/upgrading-1.0-to-1.1.md) the upgrade steps. Until 1.1 ships, `main` carries `metamodelVersion` 1.1, so no 1.0.x PATCH is cut from it. `@spine` and a measure's `@default` joined the same 1.1 change before it shipped, so diff --git a/docs/features/requirements.md b/docs/features/requirements.md index 486b77714..fcf85df6a 100644 --- a/docs/features/requirements.md +++ b/docs/features/requirements.md @@ -405,14 +405,18 @@ gate prints them itself. Its load is strict unless `--lax`, as the drift gates' ### Known differences between ports -- The package a requirement is taken to be in can differ between ports for two multi-file - shapes: a root document with no package loaded beside packaged ones, and a child merged - from a package-less document into a requirement that declares its own package. -- Python applies only the dependency-import rule to decide which of a project's own - entities are counted; TypeScript also applies the project's `scope` patterns. -- A `level` that is not an integer (`4.5`) is refused at load by the Python, Java and C# - loaders (`ERR_BAD_ATTR_VALUE`). The TypeScript loader lets it through, and the gate reports - it as `ERR_REQUIREMENT_BAD_LEVEL`. +None. Three that 1.1.0 release candidates had are settled, each by a shared fixture: + +- **The package a requirement is in** is its own declared package, else the nearest + enclosing node that declares one, else its file's default package. A child merged through + an overlay that declares a package takes that package (`requirement-check-conformance`, + the `package-*` cases). +- **Which entities coverage counts**: every non-abstract entity of the project, except one + imported from a dependency. The project's `scope` patterns filter generated output, not + coverage. +- **A `level` written `4.0`** loads as `4`; a fraction (`4.5`) is refused at load with + `ERR_BAD_ATTR_VALUE`, as on any integer attribute. `ERR_REQUIREMENT_BAD_LEVEL` is for a + whole number outside 1 to 5, or a missing level. ## Requiring implementers @@ -687,11 +691,10 @@ longer matches a fresh run, so the port's codegen drift gate (`metaobjects verif `mvn metaobjects:verify`, `dotnet meta verify --codegen`) fails until you regenerate. That is the prompt to re-read the witness against the new claim. -**A known limit.** In Python, Java, Kotlin and C#, a package that loses its last -requirement leaves its generated test file behind. `gen` does not remove it. The port's -codegen drift gate reports it, and you delete it by hand. TypeScript's generator reconciles -these itself: it removes the stub of a requirement that is gone, and refuses, by name, one -that carries a hand edit. +A package that loses its last requirement loses its generated test file too, in every +port: `gen` deletes a file it recorded and no longer emits, using the committed hash +manifest, and refuses, by name, to delete one that carries a hand edit. TypeScript reconciles +per requirement in the same way, removing the stub of a requirement that is gone. ### Owning the generator diff --git a/docs/ports/csharp.md b/docs/ports/csharp.md index d9ef71757..0781ebb0e 100644 --- a/docs/ports/csharp.md +++ b/docs/ports/csharp.md @@ -167,6 +167,21 @@ The codegen emits: - `AuthorFilterAllowlist.g.cs` — the server-side filter/sort allowlist feeding the generated list handler. +### Decimals on the wire + +A `field.decimal` crosses the API as one canonical string (ADR-0019 Amendment 1): `12.50` is +`"12.5"`, zero is `"0"`, never a JSON number or an exponent. Input may be a string or a number, +is read exactly, and is `400 validation` when it has more fraction digits than `@scale` or more +integer digits than `@precision - @scale` (no `@scale` beside `@precision` means 0). A filter +operand compares exactly; a bad one is `400 invalid_filter_value`. On SQLite a decimal is read at +15 significant digits. Every port runs the shared vectors in `fixtures/decimal-wire/`. + +`dotnet meta gen` writes `DecimalWire.g.cs` once per namespace. A decimal property carries its +JSON converter and, with `@precision`, `[DecimalWire.Declared(precision, scale)]`; a decimal +array is an `IList` with `DecimalListJsonConverter`. A stored value the CLR `decimal` +cannot hold answers `500 {"error": "decimal_out_of_range"}` (`DecimalRangeEndpointFilter`). +An owned `routes` generator copies `DecimalWire.cs` and `DecimalRangeEndpointFilter.cs`. + ### `Names` — the physical names, as constants `names` is opt-in — list `new MetaObjects.Codegen.Generators.NamesGenerator()` in @@ -538,9 +553,10 @@ qualified, as above: an owned copy and the packaged class share a simple name, s | `WarnUncovered` | `true` (default): one warning naming up to five requirements the filter left out. | The hook types, the identity function and the digest stay in the package when you eject; -the owned copy uses them from there. A package that loses its last requirement leaves its -two files behind: `gen` does not remove them, `dotnet meta verify --codegen` reports each as -`committed but a fresh regen would not emit it`, and you delete them. The model and the +the owned copy uses them from there. A package that loses its last requirement loses its +two files: `dotnet meta gen` deletes them through the hash manifest (an edited one is refused by +name), and until then `dotnet meta verify --codegen` reports each as +`committed but a fresh regen would not emit it`. The model and the other ports' spelling are in [Generated requirement tests and witnesses](../features/requirements.md#generated-requirement-tests-and-witnesses). diff --git a/docs/ports/java.md b/docs/ports/java.md index 5c1389ee9..046b9b910 100644 --- a/docs/ports/java.md +++ b/docs/ports/java.md @@ -397,6 +397,22 @@ derived field reads a `field.object`. A report with no view source, or an abstra generates nothing. The contract is in [reporting](../features/reporting.md#how-a-report-is-served). +### Decimals on the wire + +A `field.decimal` crosses the API as one canonical string (ADR-0019 Amendment 1): `12.50` is +`"12.5"`, zero is `"0"`, never a JSON number or an exponent. Input may be a string or a number, +is read exactly, and is `400 validation` when it has more fraction digits than `@scale` or more +integer digits than `@precision - @scale` (no `@scale` beside `@precision` means 0). A filter +operand compares exactly; a bad one is `400 invalid_filter_value`. On SQLite a decimal is read at +15 significant digits. Every port runs the shared vectors in `fixtures/decimal-wire/`. + +The generated DTO, Patch and value-object records write a `BigDecimal` through +`CanonicalDecimalSerializer` and read it through `CanonicalDecimalDeserializer`, which checks +the field's `@DeclaredDecimal(precision, scale)`. A filter allowlist lists its decimal fields in +`DECIMAL_FIELDS`, and `FilterParser.parse(rawQuery, FIELDS, OPS_BY_FIELD, DECIMAL_FIELDS)` +hands the repository `BigDecimal` operands. These classes live in +`com.metaobjects.generator.spring.runtime`. + ### `Names` — the physical names, as constants `SpringNamesGenerator` is opt-in, like every Java generator — there is no @@ -558,8 +574,9 @@ compiling. That signal depends on the annotation: without `@Override` a stale wi ordinary method, compiles, and goes stale silently, so annotate every witness. A planned or retired requirement is `@Disabled` and has no member. The generated files are rewritten whole, so do not edit them; `mvn metaobjects:verify` reports them stale when a claim changes, and a -package that loses its last requirement leaves its two files behind, which `verify` reports as -`[stale-in-repo]` (delete them: `gen` never removes a file). +package that loses its last requirement loses its two files: the full `gen` deletes them, +because the hash manifest recorded them and they are untouched (an edited one is refused by +name), and until then `verify` reports them as `[stale-in-repo]`. | Arg | Meaning | |---|---| diff --git a/docs/ports/kotlin.md b/docs/ports/kotlin.md index ba921accc..96d3cd7bb 100644 --- a/docs/ports/kotlin.md +++ b/docs/ports/kotlin.md @@ -174,7 +174,7 @@ class Witnesses : Requirements_acme_shop_Witnesses { } ``` -A requirement that becomes live adds a failing member (a red test, no compile break); one that is retired or deleted removes its member, so a stale override stops compiling (always, in Kotlin: `override` is mandatory). A planned or retired requirement is `@Disabled` and has no member. Write the witness class in Kotlin. The interface's members have default bodies, and a witness class written in **Java** against it is untested: unless your Kotlin compiler is set to emit JVM default methods for interface members (a compiler setting of your build, which this generator does not control), Java sees every member as abstract and has to implement all of them, which turns each new live requirement into a compile break instead of a red test. The args are `grain` (`concern`, the default, or `member`), `filter` and `renderer` (class names on your project's classpath implementing `RequirementTestFilter` and `RequirementTestRenderer`; a renderer's `source` is Kotlin and its `imports` are Kotlin import names) and `warnUncovered` (`true` by default). The generated files are rewritten whole and import only `org.junit.jupiter.api`, so the test classpath needs `org.junit.jupiter:junit-jupiter-api`. A package that loses its last requirement leaves its two files behind (`gen` never removes a file), and `mvn metaobjects:verify` reports them stale. `mvn metaobjects:eject -Dnames=requirement-tests -Dport=kotlin` copies `KotlinRequirementTestsGenerator.kt`, with its default rendering, into a `codegen/` module under your package; `-Dport` is needed because the name is ejectable on both JVM ports, and the goal infers it only when your project declares a dependency on exactly one of `metaobjects-codegen-kotlin` and `metaobjects-codegen-spring`. The identity function, the digest and the hook types stay in the package. The Java page, [Requirement tests](java.md#requirement-tests--junitrequirementtestsgenerator), has the full arg table. +A requirement that becomes live adds a failing member (a red test, no compile break); one that is retired or deleted removes its member, so a stale override stops compiling (always, in Kotlin: `override` is mandatory). A planned or retired requirement is `@Disabled` and has no member. Write the witness class in Kotlin. The interface's members have default bodies, and a witness class written in **Java** against it is untested: unless your Kotlin compiler is set to emit JVM default methods for interface members (a compiler setting of your build, which this generator does not control), Java sees every member as abstract and has to implement all of them, which turns each new live requirement into a compile break instead of a red test. The args are `grain` (`concern`, the default, or `member`), `filter` and `renderer` (class names on your project's classpath implementing `RequirementTestFilter` and `RequirementTestRenderer`; a renderer's `source` is Kotlin and its `imports` are Kotlin import names) and `warnUncovered` (`true` by default). The generated files are rewritten whole and import only `org.junit.jupiter.api`, so the test classpath needs `org.junit.jupiter:junit-jupiter-api`. A package that loses its last requirement loses its two files: the full `mvn metaobjects:generate` deletes them through the hash manifest (an edited one is refused by name), and until then `mvn metaobjects:verify` reports them stale. `mvn metaobjects:eject -Dnames=requirement-tests -Dport=kotlin` copies `KotlinRequirementTestsGenerator.kt`, with its default rendering, into a `codegen/` module under your package; `-Dport` is needed because the name is ejectable on both JVM ports, and the goal infers it only when your project declares a dependency on exactly one of `metaobjects-codegen-kotlin` and `metaobjects-codegen-spring`. The identity function, the digest and the hook types stay in the package. The Java page, [Requirement tests](java.md#requirement-tests--junitrequirementtestsgenerator), has the full arg table. **Own a generator.** `mvn metaobjects:eject -Dnames=` copies a reference generator into a `codegen/` Maven module under your own package, and prints the module, dependency and `` to wire. See [Own your codegen → Java and Kotlin](../features/own-your-codegen.md#java-and-kotlin-mvn-metaobjectseject). @@ -472,6 +472,22 @@ A column property whose name is a member of Exposed's `Table` gets a `Column` su follows the output mode: `options` and `storageParameters` are `Table` members only in Exposed 1.x, so they are suffixed only with `exposedApi=1`. +### Decimals on the wire + +A `field.decimal` crosses the API as one canonical string (ADR-0019 Amendment 1): `12.50` is +`"12.5"`, zero is `"0"`, never a JSON number or an exponent. Input may be a string or a number, +is read exactly, and is `400 validation` when it has more fraction digits than `@scale` or more +integer digits than `@precision - @scale` (no `@scale` beside `@precision` means 0). A filter +operand compares exactly; a bad one is `400 invalid_filter_value`. On SQLite a decimal is read at +15 significant digits. Every port runs the shared vectors in `fixtures/decimal-wire/`. + +Generated Kotlin imports nothing from MetaObjects, so codegen writes `MetaDecimalWire.kt` into +each package that has a decimal: the canonical writer, the strict reader, the declared-digit +check and the `400 validation` envelope for an unreadable body. The Exposed tables of a package +that has a decimal or uuid column also get `MetaPortableColumnTypes.kt`, which binds them on +SQLite and MySQL as well as Postgres; the exposed-table generator writes it. Ejecting `entity`, +`routes` or `exposed-table` copies `KotlinDecimalWireEmitter.kt` with the generator. + ### `Names` — the physical names, as constants `names` (`KotlinNamesGenerator`) is **not** wired above — it is opt-in, like diff --git a/docs/ports/python.md b/docs/ports/python.md index ad3b30936..4b8a89d17 100644 --- a/docs/ports/python.md +++ b/docs/ports/python.md @@ -267,6 +267,19 @@ The generated `get_repository` raises until you override it. Any object with the `AuthorRepository` methods above works, including an in-memory dict while you try it out. Serve the app with `uvicorn my_app.main:app --reload` (`pip install uvicorn`). +### Decimals on the wire + +A `field.decimal` crosses the API as one canonical string (ADR-0019 Amendment 1): `12.50` is +`"12.5"`, zero is `"0"`, never a JSON number or an exponent. Input may be a string or a number, +is read exactly, and is `400 validation` when it has more fraction digits than `@scale` or more +integer digits than `@precision - @scale` (no `@scale` beside `@precision` means 0). A filter +operand compares exactly; a bad one is `400 invalid_filter_value`. On SQLite a decimal is read at +15 significant digits. Every port runs the shared vectors in `fixtures/decimal-wire/`. + +Generated routers import `metaobjects.codegen.runtime.decimal_wire` (`format_decimal`, +`parse_decimal`, `parse_decimal_operand`, `check_decimal_range`) and pass each decimal field's +declared limits. A malformed or non-object JSON body is `400 validation`. + ### `_names.py` — the physical names, as constants The `names` generator is opt-in — list `names` in a target's `generators` in @@ -467,8 +480,8 @@ an owned generator is imported with, which loads the project's own copy of a mod another copy is already cached and refuses a project package or module whose top-level name is a standard-library module name (`types`, `json`): rename it. `metaobjects verify` goes out of sync when a claim changes, because each test carries a digest of its requirement. A -package that loses its last requirement leaves its file behind: `gen` does not remove it, -`verify` lists it as `extra:`, and you delete it. The model and the other ports' spelling are in +package that loses its last requirement loses its file: `verify` lists it as `extra:`, and a +full `gen` deletes it (an edited one is refused by name), as for any generated file. The model and the other ports' spelling are in [Generated requirement tests and witnesses](../features/requirements.md#generated-requirement-tests-and-witnesses). ### Declarative template-codegen (`templateSpec`) @@ -830,7 +843,7 @@ import lines are stable. | Drift verify | Yes — template / payload drift (`metaobjects.render.verify`); `verify --codegen` regenerates the config's list and diffs each target's `outDir` | | API reference docs | Yes — `api-docs` in a target's `generators`, options in the top-level `apiDocs` block (`apiSubDir`, `projectName`, `modelBaseUrl`); `metaobjects docs` was removed in 1.1.0 ([migration](../features/migrations/docs-are-generators.md#python)) | | Requirement gate + requirement tests | Yes (`metaobjects verify`, on every run; `requirement-tests` emits pytest and is ejectable) | -| Runtime metadata | Yes (`metaobjects.runtime.ObjectManager` — DB-API 2 driver, pg8000/psycopg) + loader API + render engine | +| Runtime metadata | Yes (`metaobjects.runtime.ObjectManager` — DB-API 2 drivers: Postgres via pg8000/psycopg, `SqliteDriver` over the standard library's `sqlite3`, `MySqlDriver` over PyMySQL with the `mysql` extra) + loader API + render engine | ## Conformance status (as of 2026-05-27) diff --git a/docs/ports/typescript-client.md b/docs/ports/typescript-client.md index 10aea1445..86604f0d9 100644 --- a/docs/ports/typescript-client.md +++ b/docs/ports/typescript-client.md @@ -473,7 +473,7 @@ the one **named for it**: | `name` on the view | who reads it | |---|---| -| `form` | `.form.tsx`, and the `` descriptor (`label` / `view` / `htmlType` / `placeholder` / `rules`) | +| `form` | `.form.tsx`, and the `` descriptor (`label` / `view` / `htmlType` / `step` / `inputMode` / `placeholder` / `rules`) | | `grid` | `.columns.tsx` and the Angular grid — the column's `header` and `meta.view` | ```jsonc diff --git a/docs/ports/typescript.md b/docs/ports/typescript.md index 29fc36fd8..0afef639f 100644 --- a/docs/ports/typescript.md +++ b/docs/ports/typescript.md @@ -374,6 +374,20 @@ export default defineConfig({ The mapping, the refusals, the Cube query that reproduces a report and the `cube` lane that checks the output against a real Cube are in [cube-export](../features/cube-export.md). +### Decimals on the wire + +A `field.decimal` crosses the API as one canonical string (ADR-0019 Amendment 1): `12.50` is +`"12.5"`, zero is `"0"`, never a JSON number or an exponent. Input may be a string or a number, +is read exactly, and is `400 validation` when it has more fraction digits than `@scale` or more +integer digits than `@precision - @scale` (no `@scale` beside `@precision` means 0). A filter +operand compares exactly; a bad one is `400 invalid_filter_value`. On SQLite a decimal is read at +15 significant digits. Every port runs the shared vectors in `fixtures/decimal-wire/`. + +A generated model module's write schema carries a local `decimalWrite` function, so generated +code imports nothing from MetaObjects; the route mounts in `@metaobjectsdev/runtime-ts` write +decimals and read filter operands by the same rule. `meta migrate` warns for a decimal declared +with `@precision` above 15 on SQLite or D1. + ### `Names` — the physical names, as constants `meta init` scaffolds a `names` generator, so a new project gets `.names.ts` diff --git a/docs/superpowers/specs/2026-05-26-fr-009-filter-operators-per-port.md b/docs/superpowers/specs/2026-05-26-fr-009-filter-operators-per-port.md index 415e37ecd..95b378d9e 100644 --- a/docs/superpowers/specs/2026-05-26-fr-009-filter-operators-per-port.md +++ b/docs/superpowers/specs/2026-05-26-fr-009-filter-operators-per-port.md @@ -73,11 +73,14 @@ Plus modify `SpringControllerGenerator.list()` to call a shared `FilterParser` r ```java @GetMapping public ResponseEntity list(/* ... existing params, plus: */ - @RequestParam Map allParams + HttpServletRequest request ) { // ... - FilterParseResult filter = FilterParser.parse(allParams, AuthorFilterAllowlist.FIELDS, AuthorFilterAllowlist.OPS_BY_FIELD); - if (filter.error() != null) return ResponseEntity.badRequest().body(Map.of("error", filter.error())); + // The raw query string (since 1.0: a servlet container drops a parameter with a malformed + // escape), plus the allowlist's decimal fields (1.1.0, ADR-0019 Amendment 1). + FilterParseResult filter = FilterParser.parse(request.getQueryString(), AuthorFilterAllowlist.FIELDS, + AuthorFilterAllowlist.OPS_BY_FIELD, AuthorFilterAllowlist.DECIMAL_FIELDS); + if (filter.error() != null) return ResponseEntity.badRequest().body(Map.of("error", filter.error(), "field", filter.field())); List rows = repository.list(actualLimit, actualOffset, sortClause, filter.predicates()); // ... } diff --git a/examples/advanced-modeling/codegen/generators/routes.ts b/examples/advanced-modeling/codegen/generators/routes.ts index c6f8a3d89..d9cc73bbf 100644 --- a/examples/advanced-modeling/codegen/generators/routes.ts +++ b/examples/advanced-modeling/codegen/generators/routes.ts @@ -68,7 +68,8 @@ import { isWriteThrough, isReport, itemRouteField, - reportDecimalColumns, + decimalColumnsLine, + decimalFieldNames, servesReadApi, formatTs, entityOutputPath, @@ -154,8 +155,8 @@ function renderRoutes( // The mount addresses `id` by default. A projection keyed on another field names it // (the view's key for that column, which is the field name), so `GET /:id` reads the // same column the by-id query does. Absent for `id`, which keeps that output's bytes. - // A decimal a SQLite report computes is a REAL; the mount sends it as the string it is elsewhere. - const decimalColumns = reportDecimalColumns(entity, ctx.dialect); + // A decimal is a canonical string on the wire on every dialect (ADR-0019 Amendment 1). + const decimalColumns = decimalFieldNames(entity); const keylessOpts = (indent: string): string => (idField !== undefined && idField !== DEFAULT_ID_FIELD ? `\n${indent}idColumn: ${JSON.stringify(idField)},` @@ -297,7 +298,7 @@ export async function ${handlerName}(fastify: ${FastifyInstanceSym}) { updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${exposeLinePrefixed} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ")}${exposeLinePrefixed} }); ${m2mMountsPrefixed} }, { prefix: ${JSON.stringify(ctx.apiPrefix)} }); } @@ -322,7 +323,7 @@ export async function ${handlerName}(fastify: ${FastifyInstanceSym}) { updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${exposeLineFlat} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ")}${exposeLineFlat} }); ${m2mMountsFlat}} `; @@ -478,7 +479,7 @@ function renderM2mMount( sourceColumn: ${sourceColumn}, targetColumn: ${targetColumn}, targetPkColumn: ${targetPkColumn}, - symmetric: ${entry.symmetric ? "true" : "false"},${discriminatorLine}${sourceDiscriminatorLine} + symmetric: ${entry.symmetric ? "true" : "false"},${decimalColumnsLine(entry.target, " ")}${discriminatorLine}${sourceDiscriminatorLine} });`; } @@ -579,7 +580,7 @@ function renderTphRoutes( updateSchema: ${baseUpdateSym}, filterAllowlist: ${baseFilterSym}, sortAllowlist: ${baseSortSym}, - dialect: ${dialectLit},${polymorphicExposeLine} + dialect: ${dialectLit},${decimalColumnsLine([base, ...plan.subtypes.map((st) => st.entity)], " ")}${polymorphicExposeLine} });`; // FR-018 x FR-017 — M:N traversal inside a TPH hierarchy. This file never consulted @@ -622,7 +623,7 @@ function renderTphRoutes( updateSchema: ${subUpdateSym}, filterAllowlist: ${subFilterSym}, sortAllowlist: ${subSortSym}, - dialect: ${dialectLit}, + dialect: ${dialectLit},${decimalColumnsLine(sub, " ")} discriminator: { column: ${JSON.stringify(discField)}, value: ${JSON.stringify(value)} },${exposeLine(expose, " ")} });`; // This subtype's own M:N navigations, mounted beneath its segment and gated on the diff --git a/examples/advanced-modeling/codegen/runtime/decimal-wire.ts b/examples/advanced-modeling/codegen/runtime/decimal-wire.ts index c71e7034c..fab6c35d5 100644 --- a/examples/advanced-modeling/codegen/runtime/decimal-wire.ts +++ b/examples/advanced-modeling/codegen/runtime/decimal-wire.ts @@ -1,21 +1,120 @@ -// The wire spelling of a field.decimal that a read-only mount SENDS. +// The wire form of a field.decimal — ADR-0019 Amendment 1 (W1-W4). The ONE decimal boundary of +// the TypeScript runtime: every decimal column of an entity, a projection or a report, on every +// dialect, crosses to a client through here, and every decimal a client sends comes in through +// here (filter operands) or through the same grammar (the generated Zod write schemas, which +// carry a local copy of `canonicalDecimal` because generated model code imports nothing from +// MetaObjects; test/decimal-wire.test.ts holds both to fixtures/decimal-wire/vectors.json). // -// A decimal is a string on the wire: it is the string Postgres `numeric` and MySQL `DECIMAL` -// really return, and the string the TypeScript read schema types it as. SQLite has no -// decimal. A decimal field on a table is a `text` column and stays a string, but the value -// a report's view computes (a ratio, an average, a sum of a decimal field) is a REAL, which -// a SQLite driver hands back as a JS number. Sent as it is, the same route answers -// `"paidShare": "0.4"` on Postgres and `"paidShare": 0.4` on SQLite, and the type says -// `string` for both. +// W1 A decimal is a JSON string in canonical form, `^-?(0|[1-9][0-9]*)(\.[0-9]*[1-9])?$`: +// plain notation, no exponent, no trailing fractional zeros, zero as "0". The server never +// formats a scale; a client that wants `12.50` formats with the field's `@scale`. +// W2 Input is a string in the JSON-number grammar (an exponent is allowed on input only) or a +// JSON number. Anything else is refused: `parseDecimalInput` answers `undefined`. +// W3 A value the declared column cannot hold is an error, never a silent rounding +// (`decimalFits`, applied by the generated write schemas). +// W4 SQLite and D1 have no decimal type: a decimal on them is a double, exact to 15 significant +// digits. The read path renders the double with 15 significant digits, then canonicalizes +// it, so `0.1 + 0.2` reads `"0.3"` (`decimalFromDouble15`). // -// So a mount takes the NAMES of its decimal fields (`decimalColumns`, which the generated -// report route passes on SQLite) and sends a number among them as its string. Which keys are -// decimals comes from the caller, never from the shape of a value: an integer column is not -// a decimal because it is a number. A string, a null and anything else pass through. The -// digits are JavaScript's own shortest round-trip spelling of the double (`1.6666666666666667`); -// like every decimal's spelling on the wire, they are the engine's and not part of the contract. - -/** Rewrite a number under any of `columns` to its string. Returns `undefined` when there is nothing to do. */ +// A string is never turned into a number on the way (no `Number()` on a decimal anywhere): an +// operand or a body value stays text, and is bound to SQL as text. + +/** W2: the JSON-number grammar, exponent allowed. No whitespace, no leading `+`, no bare point. */ +export const DECIMAL_INPUT_RE = /^(-?)(0|[1-9][0-9]*)(?:\.([0-9]+))?(?:[eE]([+-]?[0-9]+))?$/; + +/** W1: the canonical spelling a decimal takes on the wire. */ +export const DECIMAL_CANONICAL_RE = /^-?(0|[1-9][0-9]*)(\.[0-9]*[1-9])?$/; + +/** An expanded value longer than this many digits is out of range (W3): `1e999999999` must not allocate. */ +const MAX_EXPANDED_DIGITS = 1000; + +/** + * W1 + W2: parse `text` (the JSON-number grammar) and answer its canonical string, or `undefined` + * when it is not a number in that grammar or is too large to expand. Exact: it works on the digits + * and the exponent, never on a double. + */ +export function canonicalDecimal(text: string): string | undefined { + const m = DECIMAL_INPUT_RE.exec(text); + if (m === null) return undefined; + const negative = m[1] === "-"; + const intPart = m[2] ?? ""; + const fracPart = m[3] ?? ""; + const exponent = m[4] === undefined ? 0 : Number(m[4]); + if (!Number.isSafeInteger(exponent) || Math.abs(exponent) > MAX_EXPANDED_DIGITS) return undefined; + let digits = intPart + fracPart; + let point = intPart.length + exponent; // the decimal point sits after `point` digits + const lead = digits.length - digits.replace(/^0+/, "").length; + digits = digits.slice(lead); + point -= lead; + digits = digits.replace(/0+$/, ""); + if (digits.length === 0) return "0"; + let plain: string; + if (point <= 0) plain = `0.${"0".repeat(-point)}${digits}`; + else if (point >= digits.length) plain = digits + "0".repeat(point - digits.length); + else plain = `${digits.slice(0, point)}.${digits.slice(point)}`; + if (plain.length > MAX_EXPANDED_DIGITS) return undefined; + return negative ? `-${plain}` : plain; +} + +/** + * W4: the canonical string of an IEEE double at 15 significant digits (SQLite's own text + * precision). `undefined` for NaN and the infinities. + */ +export function decimalFromDouble15(value: number): string | undefined { + if (!Number.isFinite(value)) return undefined; + return canonicalDecimal(value.toPrecision(15)); +} + +/** + * W2 for a value read off a parsed JSON request body: a string in the JSON-number grammar, or a + * JSON number (exact to 15 significant digits, W4). `undefined` for anything else. + */ +export function parseDecimalInput(value: unknown): string | undefined { + if (typeof value === "string") return canonicalDecimal(value); + if (typeof value === "number") return decimalFromDouble15(value); + return undefined; +} + +/** W2 for a query-string filter operand: the same grammar as a body string, compared exactly. */ +export function parseDecimalOperand(text: string): string | undefined { + return canonicalDecimal(text); +} + +/** + * W3: whether the canonical decimal `canonical` fits a column declared `@precision` / `@scale` + * (at most `precision - scale` integer digits and `scale` fraction digits). A column rounds an + * excess fraction silently and refuses an excess integer part, so neither may reach it. + */ +export function decimalFits(canonical: string, precision: number | undefined, scale: number | undefined): boolean { + if (precision === undefined) return true; + const s = scale ?? 0; + const unsigned = canonical.startsWith("-") ? canonical.slice(1) : canonical; + const dot = unsigned.indexOf("."); + const intDigits = dot === -1 ? unsigned : unsigned.slice(0, dot); + const fracDigits = dot === -1 ? 0 : unsigned.length - dot - 1; + const intLen = intDigits === "0" ? 0 : intDigits.length; + return fracDigits <= s && intLen <= precision - s; +} + +/** + * W1/W4 on a value a driver hands back for a decimal column: a string (Postgres `numeric`, MySQL + * `DECIMAL`, a SQLite text value) is canonicalized, a number (a SQLite or D1 REAL or INTEGER) is + * rendered at 15 significant digits and canonicalized, a bigint is its digits. Null passes + * through. A value in none of those shapes is left for the caller to see. + */ +export function decimalToWire(value: unknown): unknown { + if (typeof value === "string") return canonicalDecimal(value) ?? value; + if (typeof value === "number") return decimalFromDouble15(value) ?? value; + if (typeof value === "bigint") return value.toString(); + return value; +} + +/** + * Rewrite the values under `columns` to their canonical string. Returns `undefined` when there is + * nothing to do. Which keys are decimals comes from the caller (the generated route passes the + * names of the entity's `field.decimal` fields), never from the shape of a value: an integer + * column is not a decimal because it is a number. + */ export function decimalWire( columns: readonly string[] | undefined, ): ((row: unknown) => unknown) | undefined { @@ -24,9 +123,17 @@ export function decimalWire( if (row === null || typeof row !== "object") return row; const out = { ...(row as Record) }; for (const c of columns) { - const v = out[c]; - if (typeof v === "number") out[c] = String(v); + if (c in out) out[c] = decimalToWire(out[c]); } return out; }; } + +/** A wire mapper with the decimal rewrite of `columns` applied after `base` (which may be a timestamp mapper). */ +export function withDecimals( + base: (row: unknown) => unknown, + columns: readonly string[] | undefined, +): (row: unknown) => unknown { + const decimals = decimalWire(columns); + return decimals === undefined ? base : (row) => decimals(base(row)); +} diff --git a/examples/advanced-modeling/codegen/runtime/drizzle-fastify/filter-allowlist.ts b/examples/advanced-modeling/codegen/runtime/drizzle-fastify/filter-allowlist.ts index e5f043aa2..386b10e1e 100644 --- a/examples/advanced-modeling/codegen/runtime/drizzle-fastify/filter-allowlist.ts +++ b/examples/advanced-modeling/codegen/runtime/drizzle-fastify/filter-allowlist.ts @@ -11,6 +11,7 @@ import { OPS_BY_SUBTYPE, opsForSubType, FIELD_SUBTYPE_DATE, + FIELD_SUBTYPE_DECIMAL, FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_UUID, @@ -28,6 +29,7 @@ export type FilterSubType = "string" | "number" | "boolean" | "datetime"; */ export const FILTER_VALUE_FORMATS = [ FIELD_SUBTYPE_DATE, + FIELD_SUBTYPE_DECIMAL, FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_UUID, @@ -52,7 +54,9 @@ export interface FilterFieldRule { readonly dateValues?: boolean; /** * The field's exact wire format, set by codegen for `field.date` / `field.time` / - * `field.timestamp` / `field.uuid`. The filter parser rejects a value that does not + * `field.timestamp` / `field.uuid` / `field.decimal`. A `decimal` operand is parsed with the + * JSON-number grammar and bound as an exact string, never through a double (ADR-0019 + * Amendment 1, W2). The filter parser rejects a value that does not * match it with `invalid_filter_value` instead of binding it — on SQLite a malformed * date compared as text and silently matched nothing; on Postgres it was a driver * error. Absent (an allowlist generated before this existed), a `datetime` value is diff --git a/examples/advanced-modeling/codegen/runtime/drizzle-fastify/filter-parser.ts b/examples/advanced-modeling/codegen/runtime/drizzle-fastify/filter-parser.ts index 11c20e585..76abece6c 100644 --- a/examples/advanced-modeling/codegen/runtime/drizzle-fastify/filter-parser.ts +++ b/examples/advanced-modeling/codegen/runtime/drizzle-fastify/filter-parser.ts @@ -15,7 +15,8 @@ import { ANY_TEMPORAL_EXPECTED, FORMAT_EXPECTED, PLUS_OFFSET_HINT, looksLikeUnencodedPlusOffset, matchesAnyTemporal, matchesFormat, utcIsoIfZoned, utcIsoInstant, } from "./filter-value-format.js"; -import { FIELD_SUBTYPE_TIMESTAMP } from "@metaobjectsdev/metadata"; +import { FIELD_SUBTYPE_DECIMAL, FIELD_SUBTYPE_TIMESTAMP } from "@metaobjectsdev/metadata"; +import { canonicalDecimal } from "../decimal-wire.js"; // biome-ignore lint/suspicious/noExplicitAny: dynamic dispatch over user's Drizzle table type AnyTable = any; @@ -215,14 +216,14 @@ function compileOp( throw new FilterParseError("filter.unsupported_op", `Op "${op}" not supported for field "${field}".`, { field, op, allowed: rule.ops }); } switch (op as FilterOp) { - case "eq": return eq(col as any, coerce(value, rule, col, field, op)); - case "ne": return ne(col as any, coerce(value, rule, col, field, op)); - case "gt": return gt(col as any, coerce(value, rule, col, field, op)); - case "gte": return gte(col as any, coerce(value, rule, col, field, op)); - case "lt": return lt(col as any, coerce(value, rule, col, field, op)); - case "lte": return lte(col as any, coerce(value, rule, col, field, op)); + case "eq": return eq(col as any, coerce(value, rule, col, field, op, dialect)); + case "ne": return ne(col as any, coerce(value, rule, col, field, op, dialect)); + case "gt": return gt(col as any, coerce(value, rule, col, field, op, dialect)); + case "gte": return gte(col as any, coerce(value, rule, col, field, op, dialect)); + case "lt": return lt(col as any, coerce(value, rule, col, field, op, dialect)); + case "lte": return lte(col as any, coerce(value, rule, col, field, op, dialect)); case "in": { - const list = String(value).split(",").map((v) => coerce(v.trim(), rule, col, field, op)); + const list = String(value).split(",").map((v) => coerce(v.trim(), rule, col, field, op, dialect)); if (list.length > maxInList) { throw new FilterParseError("filter.in_too_large", `In-list size ${list.length} exceeds limit ${maxInList}.`, { field, limit: maxInList }); } @@ -327,10 +328,22 @@ function assertWellFormed(s: string, rule: FilterFieldRule, col: unknown, field: } /** Check a comparison value against the field's rule, then coerce it for binding. */ -function coerce(value: unknown, rule: FilterFieldRule, col: unknown, field: string, op: string): unknown { +function coerce(value: unknown, rule: FilterFieldRule, col: unknown, field: string, op: string, dialect: SqlDialect): unknown { if (value === null || value === undefined) return null; const s = typeof value === "string" ? value : String(value); assertWellFormed(s, rule, col, field, op); + // A decimal operand is exact text, never a number (ADR-0019 Amendment 1, W2): parsed with + // the JSON-number grammar and bound as its canonical string. MySQL compares a DECIMAL column + // with a string operand as DOUBLES, so there the operand is cast to a wide DECIMAL first. + // SQLite and D1 hold a decimal as a double (W4) and order every number before any text, so a + // text operand against a computed column (a view's sum or average has no NUMERIC affinity) + // would compare false in the wrong direction: the operand is bound as the number it is, exact + // to the 15 digits W4 promises. + if (rule.format === FIELD_SUBTYPE_DECIMAL) { + const canonical = canonicalDecimal(s) as string; + if (dialect === "mysql") return sql`CAST(${canonical} AS DECIMAL(65,30))`; + return dialect === "sqlite" ? Number(canonical) : canonical; + } // A timestamp bound is compared in the same UTC spelling the generated schemas store: // on SQLite/D1 the column is TEXT and compares as text. An instant field (`instant`, i.e. // not @localTime) reads a zoneless or date-only bound as UTC, as its write schema does — diff --git a/examples/advanced-modeling/codegen/runtime/drizzle-fastify/filter-value-format.ts b/examples/advanced-modeling/codegen/runtime/drizzle-fastify/filter-value-format.ts index fec7b54a7..5def0974f 100644 --- a/examples/advanced-modeling/codegen/runtime/drizzle-fastify/filter-value-format.ts +++ b/examples/advanced-modeling/codegen/runtime/drizzle-fastify/filter-value-format.ts @@ -13,11 +13,13 @@ import { FIELD_SUBTYPE_DATE, + FIELD_SUBTYPE_DECIMAL, FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_UUID, } from "@metaobjectsdev/metadata"; import type { FilterValueFormat } from "./filter-allowlist.js"; +import { canonicalDecimal } from "../decimal-wire.js"; const DATE_RE = /^(\d{4})-(\d{2})-(\d{2})$/; const TIME_RE = /^(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d{1,9})?)?$/; @@ -45,6 +47,7 @@ const CHECKS: Readonly boolean>> = { [FIELD_SUBTYPE_TIME]: (s) => TIME_RE.test(s), [FIELD_SUBTYPE_TIMESTAMP]: isTimestamp, [FIELD_SUBTYPE_UUID]: (s) => UUID_RE.test(s), + [FIELD_SUBTYPE_DECIMAL]: (s) => canonicalDecimal(s) !== undefined, }; /** The `expected` member of the `invalid_filter_value` envelope, per format. */ @@ -53,6 +56,7 @@ export const FORMAT_EXPECTED: Readonly> = { [FIELD_SUBTYPE_TIME]: "time (HH:MM[:SS[.fff]])", [FIELD_SUBTYPE_TIMESTAMP]: "timestamp (YYYY-MM-DD[THH:MM[:SS[.fff]]][Z|±HH:MM])", [FIELD_SUBTYPE_UUID]: "uuid (8-4-4-4-12 hex)", + [FIELD_SUBTYPE_DECIMAL]: "decimal number (digits, optional fraction, optional exponent)", }; /** `expected` for a temporal rule that does not say WHICH temporal format it is. */ diff --git a/examples/advanced-modeling/codegen/runtime/drizzle-fastify/index.ts b/examples/advanced-modeling/codegen/runtime/drizzle-fastify/index.ts index 734594217..d159a176a 100644 --- a/examples/advanced-modeling/codegen/runtime/drizzle-fastify/index.ts +++ b/examples/advanced-modeling/codegen/runtime/drizzle-fastify/index.ts @@ -29,12 +29,13 @@ export type { FilterAllowlist, SortAllowlist } from "./filter-allowlist.js"; import { parseFilterParams, parsePageBound, FilterParseError } from "./filter-parser.js"; import { isTruthyFlag, contractErrorCode, coerceIdForColumn, firstRow, extractRowCount } from "./util.js"; import { timestampWire } from "../timestamp-wire.js"; +import { withDecimals } from "../decimal-wire.js"; import { supportsReturning, type SqlDialect } from "../sql-dialect.js"; import { withContractErrorHandler } from "./route-error-handler.js"; import { validationErrorBody } from "../route-errors.js"; export { isTruthyFlag, contractErrorCode, parseId, coerceIdForColumn } from "./util.js"; export { timestampWire, canonicalTimestamp } from "../timestamp-wire.js"; -export { decimalWire } from "../decimal-wire.js"; +export { decimalWire, withDecimals, canonicalDecimal, parseDecimalInput, parseDecimalOperand } from "../decimal-wire.js"; // --------------------------------------------------------------------------- // Loose types — we don't bind to a specific Drizzle backend so the helper @@ -85,6 +86,13 @@ export interface CrudRoutesOptions { updateMethod?: "patch" | "put"; filterAllowlist?: FilterAllowlist; sortAllowlist?: SortAllowlist; + /** + * The names of the entity's `field.decimal` fields. A decimal is a canonical string on the + * wire on every dialect (ADR-0019 Amendment 1): Postgres and MySQL return the column's scale + * (`"100.5000"`) and SQLite hands back a number, so each row the routes send is rewritten + * under these keys. The generated route passes them; absent, rows are sent as read. + */ + decimalColumns?: readonly string[]; /** Dialect — required if filterAllowlist or sortAllowlist is set (for dialect-specific `like` lowering: SQLite lowers to GLOB to stay case-sensitive; ADR-0049). */ dialect?: SqlDialect; /** @@ -123,7 +131,7 @@ function readSource(opts: VerbOptions): AnyTable { * non-timestamp of the same name. */ function readWire(opts: VerbOptions): (row: unknown) => unknown { - return timestampWire(readSource(opts)); + return withDecimals(timestampWire(readSource(opts)), opts.decimalColumns); } /** @@ -132,7 +140,7 @@ function readWire(opts: VerbOptions): (row: unknown) => unknown { * against the union of both sources. */ function echoWire(opts: VerbOptions): (row: unknown) => unknown { - return timestampWire(opts.table, opts.readView); + return withDecimals(timestampWire(opts.table, opts.readView), opts.decimalColumns); } /** #214 — after a write to the base table, re-read the row THROUGH the replica diff --git a/examples/advanced-modeling/codegen/runtime/drizzle-fastify/mount-m2m.ts b/examples/advanced-modeling/codegen/runtime/drizzle-fastify/mount-m2m.ts index cd6ea8320..31e1e4e8c 100644 --- a/examples/advanced-modeling/codegen/runtime/drizzle-fastify/mount-m2m.ts +++ b/examples/advanced-modeling/codegen/runtime/drizzle-fastify/mount-m2m.ts @@ -34,6 +34,7 @@ import type { FastifyInstance, RouteShorthandOptions } from "fastify"; import { and, eq, or, inArray } from "drizzle-orm"; import { coerceIdForColumn } from "./util.js"; import { timestampWire } from "../timestamp-wire.js"; +import { withDecimals } from "../decimal-wire.js"; import { withContractErrorHandler } from "./route-error-handler.js"; // Loose Drizzle types — the helper works across libsql / better-sqlite3 / pg. @@ -60,6 +61,8 @@ export interface M2mRouteOptions { targetColumn: string; /** Target entity PK column (physical column name, e.g. "id"). Defaults to "id". */ targetPkColumn?: string; + /** The names of the TARGET entity's `field.decimal` fields, sent as canonical strings (ADR-0019 Amendment 1). */ + decimalColumns?: readonly string[]; /** Undirected self-join: union both junction FK columns on read. */ symmetric: boolean; /** @@ -87,7 +90,7 @@ export function mountM2mRoute(opts: M2mRouteOptions): void { const route = `${opts.path}/:id/${opts.relationName}`; // Route-scoped contract error handler — see route-error-handler.ts. const ro = withContractErrorHandler(opts.routeOptions); - const toWire = timestampWire(opts.targetTable); + const toWire = withDecimals(timestampWire(opts.targetTable), opts.decimalColumns); opts.fastify.get(route, ro, async (req, reply) => { const { id } = req.params as { id: string }; diff --git a/examples/advanced-modeling/codegen/runtime/drizzle-fastify/mount-read-only.ts b/examples/advanced-modeling/codegen/runtime/drizzle-fastify/mount-read-only.ts index b4f926228..471334410 100644 --- a/examples/advanced-modeling/codegen/runtime/drizzle-fastify/mount-read-only.ts +++ b/examples/advanced-modeling/codegen/runtime/drizzle-fastify/mount-read-only.ts @@ -6,7 +6,7 @@ import { parseFilterParams, parsePageBound, RAW_VIEW_MAX_LIMIT, FilterParseError import type { FilterAllowlist, SortAllowlist } from "./filter-allowlist.js"; import { isTruthyFlag, contractErrorCode, coerceIdForColumn, rawIdLiteral, viewBaseConfig } from "./util.js"; import { timestampWire } from "../timestamp-wire.js"; -import { decimalWire } from "../decimal-wire.js"; +import { withDecimals } from "../decimal-wire.js"; import { withContractErrorHandler } from "./route-error-handler.js"; // biome-ignore lint/suspicious/noExplicitAny: dynamic dispatch over user-supplied views @@ -51,11 +51,11 @@ export interface MountReadOnlyOptions { /** The noun in the 405 message, which is free prose. Default "projection". */ readonly resource?: "projection" | "report"; /** - * The names of the view's decimal fields, for a dialect that has no decimal. SQLite hands - * a computed decimal (a report's ratio, average or sum) back as a REAL, a JS number, where - * Postgres and MySQL return the string the read schema types it as. A number under one of - * these keys is sent as its string, so the route answers the same on every engine. The - * generated route passes it for a report on SQLite and omits it otherwise. Default none. + * The names of the view's `field.decimal` fields. A decimal is a canonical string on the wire + * on every dialect (ADR-0019 Amendment 1): Postgres and MySQL return the column's scale + * (`"100.5000"`), and SQLite hands back a number, a REAL for a computed one (a report's ratio, + * average or sum). Each row is rewritten under these keys. The generated route passes them; + * absent, rows are sent as read. Default none. */ readonly decimalColumns?: readonly string[]; } @@ -160,8 +160,7 @@ export function mountReadOnlyCrudRoutes(opts: MountReadOnlyOptions): void { const useRawSql = isEmptyColumnView(view) && !!viewName; // The raw-SQL branch has no declared columns, so nothing names a timestamp there. const timestamps = timestampWire(view); - const decimals = decimalWire(opts.decimalColumns); - const toWire = decimals === undefined ? timestamps : (row: unknown) => decimals(timestamps(row)); + const toWire = withDecimals(timestamps, opts.decimalColumns); // ── List ────────────────────────────────────────────────────────────────── fastify.get(path, ro, async (req, reply) => { diff --git a/examples/advanced-modeling/codegen/templates/decimal-wire-helper.ts b/examples/advanced-modeling/codegen/templates/decimal-wire-helper.ts new file mode 100644 index 000000000..c2b3a8422 --- /dev/null +++ b/examples/advanced-modeling/codegen/templates/decimal-wire-helper.ts @@ -0,0 +1,47 @@ +// The wire rule for a field.decimal WRITE (ADR-0019 Amendment 1, W2 and W3), as text a generated +// model module carries as a local function. Generated model code imports nothing from +// MetaObjects, so the rule is emitted, not imported; runtime-ts/src/decimal-wire.ts holds the +// same grammar and both are held to fixtures/decimal-wire/vectors.json. + +export const DECIMAL_WRITE_HELPER = "decimalWrite"; + +/** The Zod expression of a decimal WRITE field: a string in the JSON-number grammar or a JSON + * number, rewritten to the canonical string, refused when the column cannot hold it (W3). */ +export function zodDecimalWriteExpr(precision: number | undefined, scale: number | undefined): string { + const args = precision === undefined ? "" : `, ${precision}, ${scale ?? 0}`; + return `z.union([z.string(), z.number()]).transform((v, ctx) => ${DECIMAL_WRITE_HELPER}(v, ctx${args}))`; +} + +export const DECIMAL_WRITE_HELPER_DECL = [ + "/** A field.decimal on the wire: a string in the JSON-number grammar (an exponent is allowed) or a JSON", + " * number, rewritten to its canonical string (plain notation, no trailing fraction zeros, zero as \"0\").", + " * Garbage, or a value the column cannot hold without rounding, is a validation issue. Generated. */", + `function ${DECIMAL_WRITE_HELPER}(v: string | number, ctx: z.RefinementCtx, precision?: number, scale = 0): string {`, + ' const text = typeof v === "number" ? (Number.isFinite(v) ? v.toPrecision(15) : "") : v;', + " const m = /^(-?)(0|[1-9][0-9]*)(?:\\.([0-9]+))?(?:[eE]([+-]?[0-9]+))?$/.exec(text);", + " const e = m === null || m[4] === undefined ? 0 : Number(m[4]);", + " if (m === null || !Number.isSafeInteger(e) || Math.abs(e) > 1000) {", + ' ctx.addIssue({ code: "custom", message: "Not a decimal number" });', + " return z.NEVER;", + " }", + ' const whole = m[2] ?? "";', + ' let digits = whole + (m[3] ?? "");', + " let point = whole.length + e;", + ' const lead = digits.length - digits.replace(/^0+/, "").length;', + " digits = digits.slice(lead).replace(/0+$/, \"\");", + " point -= lead;", + ' if (digits.length === 0) return "0";', + " const plain = point <= 0", + ' ? `0.${"0".repeat(-point)}${digits}`', + " : point >= digits.length", + ' ? digits + "0".repeat(point - digits.length)', + " : `${digits.slice(0, point)}.${digits.slice(point)}`;", + " const fraction = point >= digits.length ? 0 : digits.length - Math.max(point, 0) + Math.max(-point, 0);", + " const integer = Math.max(point, 0);", + " if (plain.length > 1000 || (precision !== undefined && (fraction > scale || integer > precision - scale))) {", + ' ctx.addIssue({ code: "custom", message: "Decimal out of range for the column" });', + " return z.NEVER;", + " }", + ' return m[1] === "-" ? `-${plain}` : plain;', + "}", +].join("\n"); diff --git a/examples/advanced-modeling/codegen/templates/entity-constants.ts b/examples/advanced-modeling/codegen/templates/entity-constants.ts index 294191ce4..b6d0ecccb 100644 --- a/examples/advanced-modeling/codegen/templates/entity-constants.ts +++ b/examples/advanced-modeling/codegen/templates/entity-constants.ts @@ -74,6 +74,8 @@ function renderFieldEntry(f: UiFieldDescriptor): string { `view: ${JSON.stringify(f.view)}`, ]; if (f.htmlType !== undefined) entries.push(`htmlType: ${JSON.stringify(f.htmlType)}`); + if (f.step !== undefined) entries.push(`step: ${JSON.stringify(f.step)}`); + if (f.inputMode !== undefined) entries.push(`inputMode: ${JSON.stringify(f.inputMode)}`); if (f.rules.length > 0) entries.push(`rules: { ${f.rules.map(renderRule).join(", ")} }`); // A `field.enum`'s member symbols. Emitted whenever the field has them, because the diff --git a/examples/advanced-modeling/codegen/templates/field-meta.ts b/examples/advanced-modeling/codegen/templates/field-meta.ts index a6b30a7e1..6182a3130 100644 --- a/examples/advanced-modeling/codegen/templates/field-meta.ts +++ b/examples/advanced-modeling/codegen/templates/field-meta.ts @@ -283,3 +283,39 @@ export function humanize(s: string): string { .replace(/\b\w/g, (c) => c.toUpperCase()) .trim(); } + +/** The `` attributes a numeric field subtype needs. */ +export interface NumericInputControl { + readonly htmlType: "number" | "text"; + /** `"any"` for a binary float: the browser's implicit step of 1 refuses `12.5`. */ + readonly step?: "any"; + /** `"decimal"` for an exact decimal rendered as a text input. */ + readonly inputMode?: "decimal"; +} + +/** + * The one answer both form tiers give for a numeric field subtype (the `` meta that + * `useEntityForm` spreads, and the nested sub-form in codegen-ts-react), or undefined for a + * non-numeric subtype. + * + * `` has an implicit step of 1, so native validation blocks `12.5`. + * double/float therefore get `step="any"`. A decimal is an exact value that travels as a + * string: a number input would parse it to a double for validation and min/max (wrong past + * 15 digits), so it is `type="text" inputmode="decimal"` and RHF/Zod own the validation. + * int, long and currency keep step 1. + */ +export function numericInputControl(subType: string): NumericInputControl | undefined { + switch (subType) { + case FIELD_SUBTYPE_INT: + case FIELD_SUBTYPE_LONG: + case FIELD_SUBTYPE_CURRENCY: + return { htmlType: "number" }; + case FIELD_SUBTYPE_DOUBLE: + case FIELD_SUBTYPE_FLOAT: + return { htmlType: "number", step: "any" }; + case FIELD_SUBTYPE_DECIMAL: + return { htmlType: "text", inputMode: "decimal" }; + default: + return undefined; + } +} diff --git a/examples/advanced-modeling/codegen/templates/filter-allowlist.ts b/examples/advanced-modeling/codegen/templates/filter-allowlist.ts index 0812a0bb4..4a0cd087f 100644 --- a/examples/advanced-modeling/codegen/templates/filter-allowlist.ts +++ b/examples/advanced-modeling/codegen/templates/filter-allowlist.ts @@ -69,6 +69,7 @@ const VALUE_FORMAT_SUBTYPES = new Set([ FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_UUID, + FIELD_SUBTYPE_DECIMAL, ]); /** Subtypes whose values are whole numbers — `field.currency` is integer minor units. */ diff --git a/examples/advanced-modeling/codegen/templates/zod-validators.ts b/examples/advanced-modeling/codegen/templates/zod-validators.ts index 113186b89..643429fac 100644 --- a/examples/advanced-modeling/codegen/templates/zod-validators.ts +++ b/examples/advanced-modeling/codegen/templates/zod-validators.ts @@ -13,7 +13,7 @@ import { code, joinCode, imp, type Code } from "@metaobjectsdev/codegen-ts/engin import { MetaObject, MetaField, type MetaRoot, stripPackage } from "@metaobjectsdev/metadata"; import { FIELD_SUBTYPE_STRING, FIELD_SUBTYPE_INT, FIELD_SUBTYPE_LONG, FIELD_SUBTYPE_CURRENCY, - FIELD_SUBTYPE_BOOLEAN, FIELD_SUBTYPE_DOUBLE, FIELD_SUBTYPE_FLOAT, + FIELD_SUBTYPE_BOOLEAN, FIELD_SUBTYPE_DECIMAL, FIELD_ATTR_PRECISION, FIELD_ATTR_SCALE, FIELD_SUBTYPE_DOUBLE, FIELD_SUBTYPE_FLOAT, FIELD_SUBTYPE_DATE, FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_ENUM, FIELD_SUBTYPE_OBJECT, FIELD_SUBTYPE_MAP, FIELD_SUBTYPE_UUID, FIELD_SUBTYPE_URI, FIELD_SUBTYPE_INET, @@ -43,6 +43,7 @@ import { ZOD_TIMESTAMP_UTC_EXPR, } from "./date-time-regex.js"; import { renderDocsFor } from "./jsdoc.js"; +import { DECIMAL_WRITE_HELPER_DECL, zodDecimalWriteExpr } from "./decimal-wire-helper.js"; import { sharedEnumForField } from "@metaobjectsdev/codegen-ts/engine"; import { sharedEnumImportSpecifier } from "@metaobjectsdev/codegen-ts/engine"; import { sharedEnumZodConstName } from "./enums-file.js"; @@ -74,6 +75,13 @@ function utcTimestampHelperSince(since: number): string { return utcTimestampUses > since ? `${UTC_TIMESTAMP_HELPER_DECL}\n\n` : ""; } +// The same device for the decimal write rule (decimal-wire-helper.ts). +let decimalWriteUses = 0; + +function decimalHelperSince(since: number): string { + return decimalWriteUses > since ? `${DECIMAL_WRITE_HELPER_DECL}\n\n` : ""; +} + /** True when this object declares at least one @autoSet timestamp field * (onCreate or onUpdate). Drives whether codegen emits the #203 @@ -259,6 +267,7 @@ function isInsertRequiredPk(field: MetaField, assignedPk: Set): boolean export function renderInsertSchemaOnly(obj: MetaObject, ctx?: RenderContext): Code { const z = imp("z@zod"); const utcUsesBefore = utcTimestampUses; + const decimalUsesBefore = decimalWriteUses; const autoGenPkFields = autoGenPkFieldNames(obj); const assignedPkFields = assignedPkFieldNames(obj); const tphPin = tphDiscriminatorPin(obj); @@ -310,7 +319,7 @@ export function renderInsertSchemaOnly(obj: MetaObject, ctx?: RenderContext): Co const docsPrefix = docs ? `${docs}\n` : ""; return code` -${utcTimestampHelperSince(utcUsesBefore)}${docsPrefix}export const ${insertSchemaName} = ${z}.object({ +${utcTimestampHelperSince(utcUsesBefore)}${decimalHelperSince(decimalUsesBefore)}${docsPrefix}export const ${insertSchemaName} = ${z}.object({ ${joinCode(insertFieldLines, { on: ",\n" })} }); `; @@ -418,6 +427,7 @@ export function updateSchemaFields(obj: MetaObject): SchemaFieldShape[] { export function renderZodValidators(obj: MetaObject, ctx?: RenderContext): Code { const z = imp("z@zod"); const utcUsesBefore = utcTimestampUses; + const decimalUsesBefore = decimalWriteUses; const autoGenPkFields = autoGenPkFieldNames(obj); const pkFields = primaryKeyFieldNames(obj); const assignedPkFields = assignedPkFieldNames(obj); @@ -540,7 +550,7 @@ export type ${objName}CreatePreserving = ${z}.input (idField !== undefined && idField !== DEFAULT_ID_FIELD ? `\n${indent}idColumn: ${JSON.stringify(idField)},` @@ -297,7 +298,7 @@ export async function ${handlerName}(fastify: ${FastifyInstanceSym}) { updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${exposeLinePrefixed} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ")}${exposeLinePrefixed} }); ${m2mMountsPrefixed} }, { prefix: ${JSON.stringify(ctx.apiPrefix)} }); } @@ -322,7 +323,7 @@ export async function ${handlerName}(fastify: ${FastifyInstanceSym}) { updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${exposeLineFlat} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ")}${exposeLineFlat} }); ${m2mMountsFlat}} `; @@ -478,7 +479,7 @@ function renderM2mMount( sourceColumn: ${sourceColumn}, targetColumn: ${targetColumn}, targetPkColumn: ${targetPkColumn}, - symmetric: ${entry.symmetric ? "true" : "false"},${discriminatorLine}${sourceDiscriminatorLine} + symmetric: ${entry.symmetric ? "true" : "false"},${decimalColumnsLine(entry.target, " ")}${discriminatorLine}${sourceDiscriminatorLine} });`; } @@ -579,7 +580,7 @@ function renderTphRoutes( updateSchema: ${baseUpdateSym}, filterAllowlist: ${baseFilterSym}, sortAllowlist: ${baseSortSym}, - dialect: ${dialectLit},${polymorphicExposeLine} + dialect: ${dialectLit},${decimalColumnsLine([base, ...plan.subtypes.map((st) => st.entity)], " ")}${polymorphicExposeLine} });`; // FR-018 x FR-017 — M:N traversal inside a TPH hierarchy. This file never consulted @@ -622,7 +623,7 @@ function renderTphRoutes( updateSchema: ${subUpdateSym}, filterAllowlist: ${subFilterSym}, sortAllowlist: ${subSortSym}, - dialect: ${dialectLit}, + dialect: ${dialectLit},${decimalColumnsLine(sub, " ")} discriminator: { column: ${JSON.stringify(discField)}, value: ${JSON.stringify(value)} },${exposeLine(expose, " ")} });`; // This subtype's own M:N navigations, mounted beneath its segment and gated on the diff --git a/examples/showcase/codegen/runtime/decimal-wire.ts b/examples/showcase/codegen/runtime/decimal-wire.ts index c71e7034c..fab6c35d5 100644 --- a/examples/showcase/codegen/runtime/decimal-wire.ts +++ b/examples/showcase/codegen/runtime/decimal-wire.ts @@ -1,21 +1,120 @@ -// The wire spelling of a field.decimal that a read-only mount SENDS. +// The wire form of a field.decimal — ADR-0019 Amendment 1 (W1-W4). The ONE decimal boundary of +// the TypeScript runtime: every decimal column of an entity, a projection or a report, on every +// dialect, crosses to a client through here, and every decimal a client sends comes in through +// here (filter operands) or through the same grammar (the generated Zod write schemas, which +// carry a local copy of `canonicalDecimal` because generated model code imports nothing from +// MetaObjects; test/decimal-wire.test.ts holds both to fixtures/decimal-wire/vectors.json). // -// A decimal is a string on the wire: it is the string Postgres `numeric` and MySQL `DECIMAL` -// really return, and the string the TypeScript read schema types it as. SQLite has no -// decimal. A decimal field on a table is a `text` column and stays a string, but the value -// a report's view computes (a ratio, an average, a sum of a decimal field) is a REAL, which -// a SQLite driver hands back as a JS number. Sent as it is, the same route answers -// `"paidShare": "0.4"` on Postgres and `"paidShare": 0.4` on SQLite, and the type says -// `string` for both. +// W1 A decimal is a JSON string in canonical form, `^-?(0|[1-9][0-9]*)(\.[0-9]*[1-9])?$`: +// plain notation, no exponent, no trailing fractional zeros, zero as "0". The server never +// formats a scale; a client that wants `12.50` formats with the field's `@scale`. +// W2 Input is a string in the JSON-number grammar (an exponent is allowed on input only) or a +// JSON number. Anything else is refused: `parseDecimalInput` answers `undefined`. +// W3 A value the declared column cannot hold is an error, never a silent rounding +// (`decimalFits`, applied by the generated write schemas). +// W4 SQLite and D1 have no decimal type: a decimal on them is a double, exact to 15 significant +// digits. The read path renders the double with 15 significant digits, then canonicalizes +// it, so `0.1 + 0.2` reads `"0.3"` (`decimalFromDouble15`). // -// So a mount takes the NAMES of its decimal fields (`decimalColumns`, which the generated -// report route passes on SQLite) and sends a number among them as its string. Which keys are -// decimals comes from the caller, never from the shape of a value: an integer column is not -// a decimal because it is a number. A string, a null and anything else pass through. The -// digits are JavaScript's own shortest round-trip spelling of the double (`1.6666666666666667`); -// like every decimal's spelling on the wire, they are the engine's and not part of the contract. - -/** Rewrite a number under any of `columns` to its string. Returns `undefined` when there is nothing to do. */ +// A string is never turned into a number on the way (no `Number()` on a decimal anywhere): an +// operand or a body value stays text, and is bound to SQL as text. + +/** W2: the JSON-number grammar, exponent allowed. No whitespace, no leading `+`, no bare point. */ +export const DECIMAL_INPUT_RE = /^(-?)(0|[1-9][0-9]*)(?:\.([0-9]+))?(?:[eE]([+-]?[0-9]+))?$/; + +/** W1: the canonical spelling a decimal takes on the wire. */ +export const DECIMAL_CANONICAL_RE = /^-?(0|[1-9][0-9]*)(\.[0-9]*[1-9])?$/; + +/** An expanded value longer than this many digits is out of range (W3): `1e999999999` must not allocate. */ +const MAX_EXPANDED_DIGITS = 1000; + +/** + * W1 + W2: parse `text` (the JSON-number grammar) and answer its canonical string, or `undefined` + * when it is not a number in that grammar or is too large to expand. Exact: it works on the digits + * and the exponent, never on a double. + */ +export function canonicalDecimal(text: string): string | undefined { + const m = DECIMAL_INPUT_RE.exec(text); + if (m === null) return undefined; + const negative = m[1] === "-"; + const intPart = m[2] ?? ""; + const fracPart = m[3] ?? ""; + const exponent = m[4] === undefined ? 0 : Number(m[4]); + if (!Number.isSafeInteger(exponent) || Math.abs(exponent) > MAX_EXPANDED_DIGITS) return undefined; + let digits = intPart + fracPart; + let point = intPart.length + exponent; // the decimal point sits after `point` digits + const lead = digits.length - digits.replace(/^0+/, "").length; + digits = digits.slice(lead); + point -= lead; + digits = digits.replace(/0+$/, ""); + if (digits.length === 0) return "0"; + let plain: string; + if (point <= 0) plain = `0.${"0".repeat(-point)}${digits}`; + else if (point >= digits.length) plain = digits + "0".repeat(point - digits.length); + else plain = `${digits.slice(0, point)}.${digits.slice(point)}`; + if (plain.length > MAX_EXPANDED_DIGITS) return undefined; + return negative ? `-${plain}` : plain; +} + +/** + * W4: the canonical string of an IEEE double at 15 significant digits (SQLite's own text + * precision). `undefined` for NaN and the infinities. + */ +export function decimalFromDouble15(value: number): string | undefined { + if (!Number.isFinite(value)) return undefined; + return canonicalDecimal(value.toPrecision(15)); +} + +/** + * W2 for a value read off a parsed JSON request body: a string in the JSON-number grammar, or a + * JSON number (exact to 15 significant digits, W4). `undefined` for anything else. + */ +export function parseDecimalInput(value: unknown): string | undefined { + if (typeof value === "string") return canonicalDecimal(value); + if (typeof value === "number") return decimalFromDouble15(value); + return undefined; +} + +/** W2 for a query-string filter operand: the same grammar as a body string, compared exactly. */ +export function parseDecimalOperand(text: string): string | undefined { + return canonicalDecimal(text); +} + +/** + * W3: whether the canonical decimal `canonical` fits a column declared `@precision` / `@scale` + * (at most `precision - scale` integer digits and `scale` fraction digits). A column rounds an + * excess fraction silently and refuses an excess integer part, so neither may reach it. + */ +export function decimalFits(canonical: string, precision: number | undefined, scale: number | undefined): boolean { + if (precision === undefined) return true; + const s = scale ?? 0; + const unsigned = canonical.startsWith("-") ? canonical.slice(1) : canonical; + const dot = unsigned.indexOf("."); + const intDigits = dot === -1 ? unsigned : unsigned.slice(0, dot); + const fracDigits = dot === -1 ? 0 : unsigned.length - dot - 1; + const intLen = intDigits === "0" ? 0 : intDigits.length; + return fracDigits <= s && intLen <= precision - s; +} + +/** + * W1/W4 on a value a driver hands back for a decimal column: a string (Postgres `numeric`, MySQL + * `DECIMAL`, a SQLite text value) is canonicalized, a number (a SQLite or D1 REAL or INTEGER) is + * rendered at 15 significant digits and canonicalized, a bigint is its digits. Null passes + * through. A value in none of those shapes is left for the caller to see. + */ +export function decimalToWire(value: unknown): unknown { + if (typeof value === "string") return canonicalDecimal(value) ?? value; + if (typeof value === "number") return decimalFromDouble15(value) ?? value; + if (typeof value === "bigint") return value.toString(); + return value; +} + +/** + * Rewrite the values under `columns` to their canonical string. Returns `undefined` when there is + * nothing to do. Which keys are decimals comes from the caller (the generated route passes the + * names of the entity's `field.decimal` fields), never from the shape of a value: an integer + * column is not a decimal because it is a number. + */ export function decimalWire( columns: readonly string[] | undefined, ): ((row: unknown) => unknown) | undefined { @@ -24,9 +123,17 @@ export function decimalWire( if (row === null || typeof row !== "object") return row; const out = { ...(row as Record) }; for (const c of columns) { - const v = out[c]; - if (typeof v === "number") out[c] = String(v); + if (c in out) out[c] = decimalToWire(out[c]); } return out; }; } + +/** A wire mapper with the decimal rewrite of `columns` applied after `base` (which may be a timestamp mapper). */ +export function withDecimals( + base: (row: unknown) => unknown, + columns: readonly string[] | undefined, +): (row: unknown) => unknown { + const decimals = decimalWire(columns); + return decimals === undefined ? base : (row) => decimals(base(row)); +} diff --git a/examples/showcase/codegen/runtime/drizzle-fastify/filter-allowlist.ts b/examples/showcase/codegen/runtime/drizzle-fastify/filter-allowlist.ts index e5f043aa2..386b10e1e 100644 --- a/examples/showcase/codegen/runtime/drizzle-fastify/filter-allowlist.ts +++ b/examples/showcase/codegen/runtime/drizzle-fastify/filter-allowlist.ts @@ -11,6 +11,7 @@ import { OPS_BY_SUBTYPE, opsForSubType, FIELD_SUBTYPE_DATE, + FIELD_SUBTYPE_DECIMAL, FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_UUID, @@ -28,6 +29,7 @@ export type FilterSubType = "string" | "number" | "boolean" | "datetime"; */ export const FILTER_VALUE_FORMATS = [ FIELD_SUBTYPE_DATE, + FIELD_SUBTYPE_DECIMAL, FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_UUID, @@ -52,7 +54,9 @@ export interface FilterFieldRule { readonly dateValues?: boolean; /** * The field's exact wire format, set by codegen for `field.date` / `field.time` / - * `field.timestamp` / `field.uuid`. The filter parser rejects a value that does not + * `field.timestamp` / `field.uuid` / `field.decimal`. A `decimal` operand is parsed with the + * JSON-number grammar and bound as an exact string, never through a double (ADR-0019 + * Amendment 1, W2). The filter parser rejects a value that does not * match it with `invalid_filter_value` instead of binding it — on SQLite a malformed * date compared as text and silently matched nothing; on Postgres it was a driver * error. Absent (an allowlist generated before this existed), a `datetime` value is diff --git a/examples/showcase/codegen/runtime/drizzle-fastify/filter-parser.ts b/examples/showcase/codegen/runtime/drizzle-fastify/filter-parser.ts index 11c20e585..76abece6c 100644 --- a/examples/showcase/codegen/runtime/drizzle-fastify/filter-parser.ts +++ b/examples/showcase/codegen/runtime/drizzle-fastify/filter-parser.ts @@ -15,7 +15,8 @@ import { ANY_TEMPORAL_EXPECTED, FORMAT_EXPECTED, PLUS_OFFSET_HINT, looksLikeUnencodedPlusOffset, matchesAnyTemporal, matchesFormat, utcIsoIfZoned, utcIsoInstant, } from "./filter-value-format.js"; -import { FIELD_SUBTYPE_TIMESTAMP } from "@metaobjectsdev/metadata"; +import { FIELD_SUBTYPE_DECIMAL, FIELD_SUBTYPE_TIMESTAMP } from "@metaobjectsdev/metadata"; +import { canonicalDecimal } from "../decimal-wire.js"; // biome-ignore lint/suspicious/noExplicitAny: dynamic dispatch over user's Drizzle table type AnyTable = any; @@ -215,14 +216,14 @@ function compileOp( throw new FilterParseError("filter.unsupported_op", `Op "${op}" not supported for field "${field}".`, { field, op, allowed: rule.ops }); } switch (op as FilterOp) { - case "eq": return eq(col as any, coerce(value, rule, col, field, op)); - case "ne": return ne(col as any, coerce(value, rule, col, field, op)); - case "gt": return gt(col as any, coerce(value, rule, col, field, op)); - case "gte": return gte(col as any, coerce(value, rule, col, field, op)); - case "lt": return lt(col as any, coerce(value, rule, col, field, op)); - case "lte": return lte(col as any, coerce(value, rule, col, field, op)); + case "eq": return eq(col as any, coerce(value, rule, col, field, op, dialect)); + case "ne": return ne(col as any, coerce(value, rule, col, field, op, dialect)); + case "gt": return gt(col as any, coerce(value, rule, col, field, op, dialect)); + case "gte": return gte(col as any, coerce(value, rule, col, field, op, dialect)); + case "lt": return lt(col as any, coerce(value, rule, col, field, op, dialect)); + case "lte": return lte(col as any, coerce(value, rule, col, field, op, dialect)); case "in": { - const list = String(value).split(",").map((v) => coerce(v.trim(), rule, col, field, op)); + const list = String(value).split(",").map((v) => coerce(v.trim(), rule, col, field, op, dialect)); if (list.length > maxInList) { throw new FilterParseError("filter.in_too_large", `In-list size ${list.length} exceeds limit ${maxInList}.`, { field, limit: maxInList }); } @@ -327,10 +328,22 @@ function assertWellFormed(s: string, rule: FilterFieldRule, col: unknown, field: } /** Check a comparison value against the field's rule, then coerce it for binding. */ -function coerce(value: unknown, rule: FilterFieldRule, col: unknown, field: string, op: string): unknown { +function coerce(value: unknown, rule: FilterFieldRule, col: unknown, field: string, op: string, dialect: SqlDialect): unknown { if (value === null || value === undefined) return null; const s = typeof value === "string" ? value : String(value); assertWellFormed(s, rule, col, field, op); + // A decimal operand is exact text, never a number (ADR-0019 Amendment 1, W2): parsed with + // the JSON-number grammar and bound as its canonical string. MySQL compares a DECIMAL column + // with a string operand as DOUBLES, so there the operand is cast to a wide DECIMAL first. + // SQLite and D1 hold a decimal as a double (W4) and order every number before any text, so a + // text operand against a computed column (a view's sum or average has no NUMERIC affinity) + // would compare false in the wrong direction: the operand is bound as the number it is, exact + // to the 15 digits W4 promises. + if (rule.format === FIELD_SUBTYPE_DECIMAL) { + const canonical = canonicalDecimal(s) as string; + if (dialect === "mysql") return sql`CAST(${canonical} AS DECIMAL(65,30))`; + return dialect === "sqlite" ? Number(canonical) : canonical; + } // A timestamp bound is compared in the same UTC spelling the generated schemas store: // on SQLite/D1 the column is TEXT and compares as text. An instant field (`instant`, i.e. // not @localTime) reads a zoneless or date-only bound as UTC, as its write schema does — diff --git a/examples/showcase/codegen/runtime/drizzle-fastify/filter-value-format.ts b/examples/showcase/codegen/runtime/drizzle-fastify/filter-value-format.ts index fec7b54a7..5def0974f 100644 --- a/examples/showcase/codegen/runtime/drizzle-fastify/filter-value-format.ts +++ b/examples/showcase/codegen/runtime/drizzle-fastify/filter-value-format.ts @@ -13,11 +13,13 @@ import { FIELD_SUBTYPE_DATE, + FIELD_SUBTYPE_DECIMAL, FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_UUID, } from "@metaobjectsdev/metadata"; import type { FilterValueFormat } from "./filter-allowlist.js"; +import { canonicalDecimal } from "../decimal-wire.js"; const DATE_RE = /^(\d{4})-(\d{2})-(\d{2})$/; const TIME_RE = /^(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d{1,9})?)?$/; @@ -45,6 +47,7 @@ const CHECKS: Readonly boolean>> = { [FIELD_SUBTYPE_TIME]: (s) => TIME_RE.test(s), [FIELD_SUBTYPE_TIMESTAMP]: isTimestamp, [FIELD_SUBTYPE_UUID]: (s) => UUID_RE.test(s), + [FIELD_SUBTYPE_DECIMAL]: (s) => canonicalDecimal(s) !== undefined, }; /** The `expected` member of the `invalid_filter_value` envelope, per format. */ @@ -53,6 +56,7 @@ export const FORMAT_EXPECTED: Readonly> = { [FIELD_SUBTYPE_TIME]: "time (HH:MM[:SS[.fff]])", [FIELD_SUBTYPE_TIMESTAMP]: "timestamp (YYYY-MM-DD[THH:MM[:SS[.fff]]][Z|±HH:MM])", [FIELD_SUBTYPE_UUID]: "uuid (8-4-4-4-12 hex)", + [FIELD_SUBTYPE_DECIMAL]: "decimal number (digits, optional fraction, optional exponent)", }; /** `expected` for a temporal rule that does not say WHICH temporal format it is. */ diff --git a/examples/showcase/codegen/runtime/drizzle-fastify/index.ts b/examples/showcase/codegen/runtime/drizzle-fastify/index.ts index 734594217..d159a176a 100644 --- a/examples/showcase/codegen/runtime/drizzle-fastify/index.ts +++ b/examples/showcase/codegen/runtime/drizzle-fastify/index.ts @@ -29,12 +29,13 @@ export type { FilterAllowlist, SortAllowlist } from "./filter-allowlist.js"; import { parseFilterParams, parsePageBound, FilterParseError } from "./filter-parser.js"; import { isTruthyFlag, contractErrorCode, coerceIdForColumn, firstRow, extractRowCount } from "./util.js"; import { timestampWire } from "../timestamp-wire.js"; +import { withDecimals } from "../decimal-wire.js"; import { supportsReturning, type SqlDialect } from "../sql-dialect.js"; import { withContractErrorHandler } from "./route-error-handler.js"; import { validationErrorBody } from "../route-errors.js"; export { isTruthyFlag, contractErrorCode, parseId, coerceIdForColumn } from "./util.js"; export { timestampWire, canonicalTimestamp } from "../timestamp-wire.js"; -export { decimalWire } from "../decimal-wire.js"; +export { decimalWire, withDecimals, canonicalDecimal, parseDecimalInput, parseDecimalOperand } from "../decimal-wire.js"; // --------------------------------------------------------------------------- // Loose types — we don't bind to a specific Drizzle backend so the helper @@ -85,6 +86,13 @@ export interface CrudRoutesOptions { updateMethod?: "patch" | "put"; filterAllowlist?: FilterAllowlist; sortAllowlist?: SortAllowlist; + /** + * The names of the entity's `field.decimal` fields. A decimal is a canonical string on the + * wire on every dialect (ADR-0019 Amendment 1): Postgres and MySQL return the column's scale + * (`"100.5000"`) and SQLite hands back a number, so each row the routes send is rewritten + * under these keys. The generated route passes them; absent, rows are sent as read. + */ + decimalColumns?: readonly string[]; /** Dialect — required if filterAllowlist or sortAllowlist is set (for dialect-specific `like` lowering: SQLite lowers to GLOB to stay case-sensitive; ADR-0049). */ dialect?: SqlDialect; /** @@ -123,7 +131,7 @@ function readSource(opts: VerbOptions): AnyTable { * non-timestamp of the same name. */ function readWire(opts: VerbOptions): (row: unknown) => unknown { - return timestampWire(readSource(opts)); + return withDecimals(timestampWire(readSource(opts)), opts.decimalColumns); } /** @@ -132,7 +140,7 @@ function readWire(opts: VerbOptions): (row: unknown) => unknown { * against the union of both sources. */ function echoWire(opts: VerbOptions): (row: unknown) => unknown { - return timestampWire(opts.table, opts.readView); + return withDecimals(timestampWire(opts.table, opts.readView), opts.decimalColumns); } /** #214 — after a write to the base table, re-read the row THROUGH the replica diff --git a/examples/showcase/codegen/runtime/drizzle-fastify/mount-m2m.ts b/examples/showcase/codegen/runtime/drizzle-fastify/mount-m2m.ts index cd6ea8320..31e1e4e8c 100644 --- a/examples/showcase/codegen/runtime/drizzle-fastify/mount-m2m.ts +++ b/examples/showcase/codegen/runtime/drizzle-fastify/mount-m2m.ts @@ -34,6 +34,7 @@ import type { FastifyInstance, RouteShorthandOptions } from "fastify"; import { and, eq, or, inArray } from "drizzle-orm"; import { coerceIdForColumn } from "./util.js"; import { timestampWire } from "../timestamp-wire.js"; +import { withDecimals } from "../decimal-wire.js"; import { withContractErrorHandler } from "./route-error-handler.js"; // Loose Drizzle types — the helper works across libsql / better-sqlite3 / pg. @@ -60,6 +61,8 @@ export interface M2mRouteOptions { targetColumn: string; /** Target entity PK column (physical column name, e.g. "id"). Defaults to "id". */ targetPkColumn?: string; + /** The names of the TARGET entity's `field.decimal` fields, sent as canonical strings (ADR-0019 Amendment 1). */ + decimalColumns?: readonly string[]; /** Undirected self-join: union both junction FK columns on read. */ symmetric: boolean; /** @@ -87,7 +90,7 @@ export function mountM2mRoute(opts: M2mRouteOptions): void { const route = `${opts.path}/:id/${opts.relationName}`; // Route-scoped contract error handler — see route-error-handler.ts. const ro = withContractErrorHandler(opts.routeOptions); - const toWire = timestampWire(opts.targetTable); + const toWire = withDecimals(timestampWire(opts.targetTable), opts.decimalColumns); opts.fastify.get(route, ro, async (req, reply) => { const { id } = req.params as { id: string }; diff --git a/examples/showcase/codegen/runtime/drizzle-fastify/mount-read-only.ts b/examples/showcase/codegen/runtime/drizzle-fastify/mount-read-only.ts index b4f926228..471334410 100644 --- a/examples/showcase/codegen/runtime/drizzle-fastify/mount-read-only.ts +++ b/examples/showcase/codegen/runtime/drizzle-fastify/mount-read-only.ts @@ -6,7 +6,7 @@ import { parseFilterParams, parsePageBound, RAW_VIEW_MAX_LIMIT, FilterParseError import type { FilterAllowlist, SortAllowlist } from "./filter-allowlist.js"; import { isTruthyFlag, contractErrorCode, coerceIdForColumn, rawIdLiteral, viewBaseConfig } from "./util.js"; import { timestampWire } from "../timestamp-wire.js"; -import { decimalWire } from "../decimal-wire.js"; +import { withDecimals } from "../decimal-wire.js"; import { withContractErrorHandler } from "./route-error-handler.js"; // biome-ignore lint/suspicious/noExplicitAny: dynamic dispatch over user-supplied views @@ -51,11 +51,11 @@ export interface MountReadOnlyOptions { /** The noun in the 405 message, which is free prose. Default "projection". */ readonly resource?: "projection" | "report"; /** - * The names of the view's decimal fields, for a dialect that has no decimal. SQLite hands - * a computed decimal (a report's ratio, average or sum) back as a REAL, a JS number, where - * Postgres and MySQL return the string the read schema types it as. A number under one of - * these keys is sent as its string, so the route answers the same on every engine. The - * generated route passes it for a report on SQLite and omits it otherwise. Default none. + * The names of the view's `field.decimal` fields. A decimal is a canonical string on the wire + * on every dialect (ADR-0019 Amendment 1): Postgres and MySQL return the column's scale + * (`"100.5000"`), and SQLite hands back a number, a REAL for a computed one (a report's ratio, + * average or sum). Each row is rewritten under these keys. The generated route passes them; + * absent, rows are sent as read. Default none. */ readonly decimalColumns?: readonly string[]; } @@ -160,8 +160,7 @@ export function mountReadOnlyCrudRoutes(opts: MountReadOnlyOptions): void { const useRawSql = isEmptyColumnView(view) && !!viewName; // The raw-SQL branch has no declared columns, so nothing names a timestamp there. const timestamps = timestampWire(view); - const decimals = decimalWire(opts.decimalColumns); - const toWire = decimals === undefined ? timestamps : (row: unknown) => decimals(timestamps(row)); + const toWire = withDecimals(timestamps, opts.decimalColumns); // ── List ────────────────────────────────────────────────────────────────── fastify.get(path, ro, async (req, reply) => { diff --git a/examples/showcase/codegen/templates/decimal-wire-helper.ts b/examples/showcase/codegen/templates/decimal-wire-helper.ts new file mode 100644 index 000000000..c2b3a8422 --- /dev/null +++ b/examples/showcase/codegen/templates/decimal-wire-helper.ts @@ -0,0 +1,47 @@ +// The wire rule for a field.decimal WRITE (ADR-0019 Amendment 1, W2 and W3), as text a generated +// model module carries as a local function. Generated model code imports nothing from +// MetaObjects, so the rule is emitted, not imported; runtime-ts/src/decimal-wire.ts holds the +// same grammar and both are held to fixtures/decimal-wire/vectors.json. + +export const DECIMAL_WRITE_HELPER = "decimalWrite"; + +/** The Zod expression of a decimal WRITE field: a string in the JSON-number grammar or a JSON + * number, rewritten to the canonical string, refused when the column cannot hold it (W3). */ +export function zodDecimalWriteExpr(precision: number | undefined, scale: number | undefined): string { + const args = precision === undefined ? "" : `, ${precision}, ${scale ?? 0}`; + return `z.union([z.string(), z.number()]).transform((v, ctx) => ${DECIMAL_WRITE_HELPER}(v, ctx${args}))`; +} + +export const DECIMAL_WRITE_HELPER_DECL = [ + "/** A field.decimal on the wire: a string in the JSON-number grammar (an exponent is allowed) or a JSON", + " * number, rewritten to its canonical string (plain notation, no trailing fraction zeros, zero as \"0\").", + " * Garbage, or a value the column cannot hold without rounding, is a validation issue. Generated. */", + `function ${DECIMAL_WRITE_HELPER}(v: string | number, ctx: z.RefinementCtx, precision?: number, scale = 0): string {`, + ' const text = typeof v === "number" ? (Number.isFinite(v) ? v.toPrecision(15) : "") : v;', + " const m = /^(-?)(0|[1-9][0-9]*)(?:\\.([0-9]+))?(?:[eE]([+-]?[0-9]+))?$/.exec(text);", + " const e = m === null || m[4] === undefined ? 0 : Number(m[4]);", + " if (m === null || !Number.isSafeInteger(e) || Math.abs(e) > 1000) {", + ' ctx.addIssue({ code: "custom", message: "Not a decimal number" });', + " return z.NEVER;", + " }", + ' const whole = m[2] ?? "";', + ' let digits = whole + (m[3] ?? "");', + " let point = whole.length + e;", + ' const lead = digits.length - digits.replace(/^0+/, "").length;', + " digits = digits.slice(lead).replace(/0+$/, \"\");", + " point -= lead;", + ' if (digits.length === 0) return "0";', + " const plain = point <= 0", + ' ? `0.${"0".repeat(-point)}${digits}`', + " : point >= digits.length", + ' ? digits + "0".repeat(point - digits.length)', + " : `${digits.slice(0, point)}.${digits.slice(point)}`;", + " const fraction = point >= digits.length ? 0 : digits.length - Math.max(point, 0) + Math.max(-point, 0);", + " const integer = Math.max(point, 0);", + " if (plain.length > 1000 || (precision !== undefined && (fraction > scale || integer > precision - scale))) {", + ' ctx.addIssue({ code: "custom", message: "Decimal out of range for the column" });', + " return z.NEVER;", + " }", + ' return m[1] === "-" ? `-${plain}` : plain;', + "}", +].join("\n"); diff --git a/examples/showcase/codegen/templates/entity-constants.ts b/examples/showcase/codegen/templates/entity-constants.ts index 294191ce4..b6d0ecccb 100644 --- a/examples/showcase/codegen/templates/entity-constants.ts +++ b/examples/showcase/codegen/templates/entity-constants.ts @@ -74,6 +74,8 @@ function renderFieldEntry(f: UiFieldDescriptor): string { `view: ${JSON.stringify(f.view)}`, ]; if (f.htmlType !== undefined) entries.push(`htmlType: ${JSON.stringify(f.htmlType)}`); + if (f.step !== undefined) entries.push(`step: ${JSON.stringify(f.step)}`); + if (f.inputMode !== undefined) entries.push(`inputMode: ${JSON.stringify(f.inputMode)}`); if (f.rules.length > 0) entries.push(`rules: { ${f.rules.map(renderRule).join(", ")} }`); // A `field.enum`'s member symbols. Emitted whenever the field has them, because the diff --git a/examples/showcase/codegen/templates/field-meta.ts b/examples/showcase/codegen/templates/field-meta.ts index a6b30a7e1..6182a3130 100644 --- a/examples/showcase/codegen/templates/field-meta.ts +++ b/examples/showcase/codegen/templates/field-meta.ts @@ -283,3 +283,39 @@ export function humanize(s: string): string { .replace(/\b\w/g, (c) => c.toUpperCase()) .trim(); } + +/** The `` attributes a numeric field subtype needs. */ +export interface NumericInputControl { + readonly htmlType: "number" | "text"; + /** `"any"` for a binary float: the browser's implicit step of 1 refuses `12.5`. */ + readonly step?: "any"; + /** `"decimal"` for an exact decimal rendered as a text input. */ + readonly inputMode?: "decimal"; +} + +/** + * The one answer both form tiers give for a numeric field subtype (the `` meta that + * `useEntityForm` spreads, and the nested sub-form in codegen-ts-react), or undefined for a + * non-numeric subtype. + * + * `` has an implicit step of 1, so native validation blocks `12.5`. + * double/float therefore get `step="any"`. A decimal is an exact value that travels as a + * string: a number input would parse it to a double for validation and min/max (wrong past + * 15 digits), so it is `type="text" inputmode="decimal"` and RHF/Zod own the validation. + * int, long and currency keep step 1. + */ +export function numericInputControl(subType: string): NumericInputControl | undefined { + switch (subType) { + case FIELD_SUBTYPE_INT: + case FIELD_SUBTYPE_LONG: + case FIELD_SUBTYPE_CURRENCY: + return { htmlType: "number" }; + case FIELD_SUBTYPE_DOUBLE: + case FIELD_SUBTYPE_FLOAT: + return { htmlType: "number", step: "any" }; + case FIELD_SUBTYPE_DECIMAL: + return { htmlType: "text", inputMode: "decimal" }; + default: + return undefined; + } +} diff --git a/examples/showcase/codegen/templates/filter-allowlist.ts b/examples/showcase/codegen/templates/filter-allowlist.ts index 0812a0bb4..4a0cd087f 100644 --- a/examples/showcase/codegen/templates/filter-allowlist.ts +++ b/examples/showcase/codegen/templates/filter-allowlist.ts @@ -69,6 +69,7 @@ const VALUE_FORMAT_SUBTYPES = new Set([ FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_UUID, + FIELD_SUBTYPE_DECIMAL, ]); /** Subtypes whose values are whole numbers — `field.currency` is integer minor units. */ diff --git a/examples/showcase/codegen/templates/zod-validators.ts b/examples/showcase/codegen/templates/zod-validators.ts index 113186b89..643429fac 100644 --- a/examples/showcase/codegen/templates/zod-validators.ts +++ b/examples/showcase/codegen/templates/zod-validators.ts @@ -13,7 +13,7 @@ import { code, joinCode, imp, type Code } from "@metaobjectsdev/codegen-ts/engin import { MetaObject, MetaField, type MetaRoot, stripPackage } from "@metaobjectsdev/metadata"; import { FIELD_SUBTYPE_STRING, FIELD_SUBTYPE_INT, FIELD_SUBTYPE_LONG, FIELD_SUBTYPE_CURRENCY, - FIELD_SUBTYPE_BOOLEAN, FIELD_SUBTYPE_DOUBLE, FIELD_SUBTYPE_FLOAT, + FIELD_SUBTYPE_BOOLEAN, FIELD_SUBTYPE_DECIMAL, FIELD_ATTR_PRECISION, FIELD_ATTR_SCALE, FIELD_SUBTYPE_DOUBLE, FIELD_SUBTYPE_FLOAT, FIELD_SUBTYPE_DATE, FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_ENUM, FIELD_SUBTYPE_OBJECT, FIELD_SUBTYPE_MAP, FIELD_SUBTYPE_UUID, FIELD_SUBTYPE_URI, FIELD_SUBTYPE_INET, @@ -43,6 +43,7 @@ import { ZOD_TIMESTAMP_UTC_EXPR, } from "./date-time-regex.js"; import { renderDocsFor } from "./jsdoc.js"; +import { DECIMAL_WRITE_HELPER_DECL, zodDecimalWriteExpr } from "./decimal-wire-helper.js"; import { sharedEnumForField } from "@metaobjectsdev/codegen-ts/engine"; import { sharedEnumImportSpecifier } from "@metaobjectsdev/codegen-ts/engine"; import { sharedEnumZodConstName } from "./enums-file.js"; @@ -74,6 +75,13 @@ function utcTimestampHelperSince(since: number): string { return utcTimestampUses > since ? `${UTC_TIMESTAMP_HELPER_DECL}\n\n` : ""; } +// The same device for the decimal write rule (decimal-wire-helper.ts). +let decimalWriteUses = 0; + +function decimalHelperSince(since: number): string { + return decimalWriteUses > since ? `${DECIMAL_WRITE_HELPER_DECL}\n\n` : ""; +} + /** True when this object declares at least one @autoSet timestamp field * (onCreate or onUpdate). Drives whether codegen emits the #203 @@ -259,6 +267,7 @@ function isInsertRequiredPk(field: MetaField, assignedPk: Set): boolean export function renderInsertSchemaOnly(obj: MetaObject, ctx?: RenderContext): Code { const z = imp("z@zod"); const utcUsesBefore = utcTimestampUses; + const decimalUsesBefore = decimalWriteUses; const autoGenPkFields = autoGenPkFieldNames(obj); const assignedPkFields = assignedPkFieldNames(obj); const tphPin = tphDiscriminatorPin(obj); @@ -310,7 +319,7 @@ export function renderInsertSchemaOnly(obj: MetaObject, ctx?: RenderContext): Co const docsPrefix = docs ? `${docs}\n` : ""; return code` -${utcTimestampHelperSince(utcUsesBefore)}${docsPrefix}export const ${insertSchemaName} = ${z}.object({ +${utcTimestampHelperSince(utcUsesBefore)}${decimalHelperSince(decimalUsesBefore)}${docsPrefix}export const ${insertSchemaName} = ${z}.object({ ${joinCode(insertFieldLines, { on: ",\n" })} }); `; @@ -418,6 +427,7 @@ export function updateSchemaFields(obj: MetaObject): SchemaFieldShape[] { export function renderZodValidators(obj: MetaObject, ctx?: RenderContext): Code { const z = imp("z@zod"); const utcUsesBefore = utcTimestampUses; + const decimalUsesBefore = decimalWriteUses; const autoGenPkFields = autoGenPkFieldNames(obj); const pkFields = primaryKeyFieldNames(obj); const assignedPkFields = assignedPkFieldNames(obj); @@ -540,7 +550,7 @@ export type ${objName}CreatePreserving = ${z}.input public static IEndpointRouteBuilder MapSubscriberRoutes(this IEndpointRouteBuilder app, string prefix = "/api") { + IEndpointRouteBuilder mount = app; + app = app.MapGroup(string.Empty).AddEndpointFilter(new DecimalRangeEndpointFilter()); app.MapGet(prefix + "/subscribers", async (HttpContext http, AppDbContext db) => { var qs = http.Request.Query; IQueryable q = db.Subscribers.AsNoTracking(); // Filter: ?filter[][]= against the per-entity allowlist. - var filter = FilterParser.Parse(qs, SubscriberFilterAllowlist.Fields, SubscriberFilterAllowlist.OpsByField); + var filter = EfCoreFilterDispatch.Validate(FilterParser.Parse(qs, SubscriberFilterAllowlist.Fields, SubscriberFilterAllowlist.OpsByField)); if (filter.ErrorEnvelope is not null) return Results.BadRequest(new { error = filter.ErrorEnvelope, field = filter.Field }); q = EfCoreFilterDispatch.ApplyFilter(q, filter.Predicates); @@ -182,7 +184,7 @@ async System.Threading.Tasks.Task UpdateSubscriber(long id, HttpContext return Results.NoContent(); }); - return app; + return mount; } private static IOrderedQueryable ApplySortSubscriber(IQueryable q, string field, bool desc) diff --git a/examples/showcase/generated/python/subscriber_router.py b/examples/showcase/generated/python/subscriber_router.py index 2efc2bb26..fab10e0c5 100644 --- a/examples/showcase/generated/python/subscriber_router.py +++ b/examples/showcase/generated/python/subscriber_router.py @@ -21,6 +21,12 @@ from metaobjects.codegen.runtime.constraint_errors import ( classify_constraint_error, ) +from metaobjects.codegen.runtime.decimal_wire import ( + coerce_decimal_fields, + coerce_decimal_predicates, + parse_json_object, + to_wire, +) from metaobjects.codegen.runtime.filter_parser import ( FilterPredicate, parse_filter, @@ -63,6 +69,20 @@ class _SortClause(BaseModel): ) +_DECIMAL_FIELDS: frozenset[str] = frozenset() + + +_DECIMAL_LIMITS: dict[str, tuple[int, int]] = {} + + +async def _json_body(request: Request) -> dict[str, Any] | None: + """GENERATED — the request body as a JSON object (non-integer numbers are exact `Decimal`s), or None.""" + try: + return parse_json_object(await request.body()) + except ValueError: + return None + + _FROZEN_FIELDS: frozenset[str] = frozenset() @@ -129,12 +149,16 @@ def list_subscribers( ) if filter_result.error_envelope is not None: return JSONResponse(status_code=400, content=filter_result.error_envelope) - predicates = filter_result.predicates + predicates, decimal_error = coerce_decimal_predicates( + filter_result.predicates, _DECIMAL_FIELDS + ) + if decimal_error is not None: + return JSONResponse(status_code=400, content=decimal_error) rows = repo.list(actual_limit, actual_offset, sort_clause, predicates) if with_count == 1: total = repo.count(predicates) - return {"rows": rows, "total": total} - return rows + return {"rows": to_wire(rows), "total": total} + return to_wire(rows) @router.get("/{subscriber_id}") @@ -145,14 +169,22 @@ def get_subscriber( row = repo.find_by_id(subscriber_id) if row is None: return JSONResponse(status_code=404, content={"error": "not_found"}) - return row + return to_wire(row) @router.post("", status_code=status.HTTP_201_CREATED) def create_subscriber( - dto: dict[str, Any], + dto: Annotated[dict[str, Any] | None, Depends(_json_body)], repo: Annotated[SubscriberRepository, Depends(get_repository)], ) -> Any: + if ( + dto is None + ): # the body was not a JSON object: the cross-port 400, not FastAPI's 422 + return JSONResponse(status_code=400, content={"error": "validation"}) + try: + dto = coerce_decimal_fields(dto, _DECIMAL_FIELDS, _DECIMAL_LIMITS) + except ValueError: + return JSONResponse(status_code=400, content={"error": "validation"}) # FR-036: run the generated field constraints (length / pattern / # numeric bounds) before persisting; a violation is the cross-port 400. try: @@ -163,7 +195,7 @@ def create_subscriber( _asnow = _dt.datetime.now(_dt.timezone.utc) dto["createdAt"] = _asnow try: - return repo.create(dto) + return to_wire(repo.create(dto)) except Exception as exc: failure = classify_constraint_error(exc) if failure is None: @@ -175,9 +207,17 @@ def create_subscriber( @router.put("/{subscriber_id}") def update_subscriber( subscriber_id: int, - dto: dict[str, Any], + dto: Annotated[dict[str, Any] | None, Depends(_json_body)], repo: Annotated[SubscriberRepository, Depends(get_repository)], ) -> Any: + if ( + dto is None + ): # the body was not a JSON object: the cross-port 400, not FastAPI's 422 + return JSONResponse(status_code=400, content={"error": "validation"}) + try: + dto = coerce_decimal_fields(dto, _DECIMAL_FIELDS, _DECIMAL_LIMITS) + except ValueError: + return JSONResponse(status_code=400, content={"error": "validation"}) # FR-035 PATCH-2: an explicit null on a @required field is a 400 — # a present null on a NON-required field falls through and clears it, # and an OMITTED required field is untouched (never a 400). @@ -200,7 +240,7 @@ def update_subscriber( return JSONResponse(status_code=failure.status, content=failure.body()) if saved is None: return JSONResponse(status_code=404, content={"error": "not_found"}) - return saved + return to_wire(saved) @router.delete("/{subscriber_id}", status_code=status.HTTP_204_NO_CONTENT) diff --git a/examples/showcase/site-payload.json b/examples/showcase/site-payload.json index d827f32f4..42720501a 100644 --- a/examples/showcase/site-payload.json +++ b/examples/showcase/site-payload.json @@ -7,8 +7,8 @@ "metamodel": "1.1" }, "counts": { - "fixtures": 395, - "corpora": 29, + "fixtures": 410, + "corpora": 30, "baseTypes": 17 }, "snippets": { diff --git a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md index 1012d6fc5..98cb47063 100644 --- a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md +++ b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md @@ -141,8 +141,7 @@ The full hunt for each axis of Phase 1b (Partial / Deep path). Each axis runs in taxonomy impurity (entity over read-only primary source; read model that should be `object.projection`; `value` carrying identity/source, ADR-0028); copy-pasted base-field blocks — or the same-named field re-declared across objects (drift signature 10) — instead of - abstract + `extends`; `@`-prefixed YAML keys (ADR-0006); relative refs - in committed canonical JSON (ADR-0032); DB-type-as-logical-subtype (ADR-0013); per-port + abstract + `extends`; `@`-prefixed YAML keys (ADR-0006); DB-type-as-logical-subtype (ADR-0013); per-port migration engine where schema is Node-`meta`-owned (ADR-0015). - [ ] **H2. Wrong native type — `field.` + `@dbColumnType` that hides the real type diff --git a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-codegen/references/java.md b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-codegen/references/java.md index 3f211a60d..2e3872643 100644 --- a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-codegen/references/java.md +++ b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-codegen/references/java.md @@ -41,9 +41,10 @@ field names; a database whose columns are named exactly like the fields declares There is no per-generator or runtime naming: the `columnNaming` generator arg fails `mvn metaobjects:generate`, and `SimpleMappingHandlerDB.setColumnNaming` throws. A Spring -application that loads its metadata from the classpath points `metaobjects.physical-naming-config` -at the project's `.metaobjects/config.json` packaged as a resource (for example -`classpath:.metaobjects/config.json`). Details and the 1.0 to 1.1 steps: +application reads `classpath:.metaobjects/config.json` when the project packages that file as a +resource; `metaobjects.physical-naming-config` points elsewhere (a `classpath:` location or a file +path). `MetaDataLoader.fromUris` and `fromDirectory` find the file above the metadata. With no file +found, the loader logs one WARN and uses snake_case. Details and the 1.0 to 1.1 steps: . ## Plugin config in `pom.xml` diff --git a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-codegen/references/kotlin.md b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-codegen/references/kotlin.md index 3a047c508..6a80fafdd 100644 --- a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-codegen/references/kotlin.md +++ b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-codegen/references/kotlin.md @@ -59,9 +59,10 @@ field names; a database whose columns are named exactly like the fields declares There is no per-generator or runtime naming: the `columnNaming` generator arg fails `mvn metaobjects:generate`, and `SimpleMappingHandlerDB.setColumnNaming` throws (the Exposed tables and the names generator already read the setting). A Spring -application that loads its metadata from the classpath points `metaobjects.physical-naming-config` -at the project's `.metaobjects/config.json` packaged as a resource (for example -`classpath:.metaobjects/config.json`). Details and the 1.0 to 1.1 steps: +application reads `classpath:.metaobjects/config.json` when the project packages that file as a +resource; `metaobjects.physical-naming-config` points elsewhere (a `classpath:` location or a file +path). `MetaDataLoader.fromUris` and `fromDirectory` find the file above the metadata. With no file +found, the loader logs one WARN and uses snake_case. Details and the 1.0 to 1.1 steps: . ## Plugin config in `pom.xml` diff --git a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md index 1012d6fc5..98cb47063 100644 --- a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md +++ b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md @@ -141,8 +141,7 @@ The full hunt for each axis of Phase 1b (Partial / Deep path). Each axis runs in taxonomy impurity (entity over read-only primary source; read model that should be `object.projection`; `value` carrying identity/source, ADR-0028); copy-pasted base-field blocks — or the same-named field re-declared across objects (drift signature 10) — instead of - abstract + `extends`; `@`-prefixed YAML keys (ADR-0006); relative refs - in committed canonical JSON (ADR-0032); DB-type-as-logical-subtype (ADR-0013); per-port + abstract + `extends`; `@`-prefixed YAML keys (ADR-0006); DB-type-as-logical-subtype (ADR-0013); per-port migration engine where schema is Node-`meta`-owned (ADR-0015). - [ ] **H2. Wrong native type — `field.` + `@dbColumnType` that hides the real type diff --git a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-codegen/references/java.md b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-codegen/references/java.md index 3f211a60d..2e3872643 100644 --- a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-codegen/references/java.md +++ b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-codegen/references/java.md @@ -41,9 +41,10 @@ field names; a database whose columns are named exactly like the fields declares There is no per-generator or runtime naming: the `columnNaming` generator arg fails `mvn metaobjects:generate`, and `SimpleMappingHandlerDB.setColumnNaming` throws. A Spring -application that loads its metadata from the classpath points `metaobjects.physical-naming-config` -at the project's `.metaobjects/config.json` packaged as a resource (for example -`classpath:.metaobjects/config.json`). Details and the 1.0 to 1.1 steps: +application reads `classpath:.metaobjects/config.json` when the project packages that file as a +resource; `metaobjects.physical-naming-config` points elsewhere (a `classpath:` location or a file +path). `MetaDataLoader.fromUris` and `fromDirectory` find the file above the metadata. With no file +found, the loader logs one WARN and uses snake_case. Details and the 1.0 to 1.1 steps: . ## Plugin config in `pom.xml` diff --git a/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md index 1012d6fc5..98cb47063 100644 --- a/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md +++ b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md @@ -141,8 +141,7 @@ The full hunt for each axis of Phase 1b (Partial / Deep path). Each axis runs in taxonomy impurity (entity over read-only primary source; read model that should be `object.projection`; `value` carrying identity/source, ADR-0028); copy-pasted base-field blocks — or the same-named field re-declared across objects (drift signature 10) — instead of - abstract + `extends`; `@`-prefixed YAML keys (ADR-0006); relative refs - in committed canonical JSON (ADR-0032); DB-type-as-logical-subtype (ADR-0013); per-port + abstract + `extends`; `@`-prefixed YAML keys (ADR-0006); DB-type-as-logical-subtype (ADR-0013); per-port migration engine where schema is Node-`meta`-owned (ADR-0015). - [ ] **H2. Wrong native type — `field.` + `@dbColumnType` that hides the real type diff --git a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md index 1012d6fc5..98cb47063 100644 --- a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md +++ b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md @@ -141,8 +141,7 @@ The full hunt for each axis of Phase 1b (Partial / Deep path). Each axis runs in taxonomy impurity (entity over read-only primary source; read model that should be `object.projection`; `value` carrying identity/source, ADR-0028); copy-pasted base-field blocks — or the same-named field re-declared across objects (drift signature 10) — instead of - abstract + `extends`; `@`-prefixed YAML keys (ADR-0006); relative refs - in committed canonical JSON (ADR-0032); DB-type-as-logical-subtype (ADR-0013); per-port + abstract + `extends`; `@`-prefixed YAML keys (ADR-0006); DB-type-as-logical-subtype (ADR-0013); per-port migration engine where schema is Node-`meta`-owned (ADR-0015). - [ ] **H2. Wrong native type — `field.` + `@dbColumnType` that hides the real type diff --git a/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md b/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md index 1012d6fc5..98cb47063 100644 --- a/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md +++ b/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-audit/references/surface-review-axes.md @@ -141,8 +141,7 @@ The full hunt for each axis of Phase 1b (Partial / Deep path). Each axis runs in taxonomy impurity (entity over read-only primary source; read model that should be `object.projection`; `value` carrying identity/source, ADR-0028); copy-pasted base-field blocks — or the same-named field re-declared across objects (drift signature 10) — instead of - abstract + `extends`; `@`-prefixed YAML keys (ADR-0006); relative refs - in committed canonical JSON (ADR-0032); DB-type-as-logical-subtype (ADR-0013); per-port + abstract + `extends`; `@`-prefixed YAML keys (ADR-0006); DB-type-as-logical-subtype (ADR-0013); per-port migration engine where schema is Node-`meta`-owned (ADR-0015). - [ ] **H2. Wrong native type — `field.` + `@dbColumnType` that hides the real type diff --git a/fixtures/api-contract-conformance/README.md b/fixtures/api-contract-conformance/README.md index 9ee39956f..1f068ae3a 100644 --- a/fixtures/api-contract-conformance/README.md +++ b/fixtures/api-contract-conformance/README.md @@ -52,8 +52,9 @@ fixtures/api-contract-conformance/ └── filter-invalid-value.yaml # uncoercible value → 400; the fourth envelope, previously ungated ``` -Beside the 31 core scenarios above sit six **sub-corpora**, each a directory with its -own `meta.json`, `seed.json`, `scenarios/` and `README.md` (82 scenarios in all): +Beside the 31 core scenarios above sit seven **sub-corpora**, each a directory with its +own `meta.json`, `seed.json`, `scenarios/` and `README.md` (70 scenarios in all, 101 with the +core): | Directory | Scenarios | Gates | Lanes | |---|---|---|---| @@ -62,7 +63,8 @@ own `meta.json`, `seed.json`, `scenarios/` and `README.md` (82 scenarios in all) | `jsonb/` | 2 | typed value-object columns | both | | `write-through/` | 2 | an entity that writes a table and reads a view | generated only | | `projection/` | 12 | a read-only `object.projection`: reads served, every write `405`; no declared identity means no `/{id}` route; a key not named `id` addresses the row; decimal and float filters. Its `docs-routes.json` pins the routes each port documents | generated only | -| `report/` | 16 | a view-backed `object.report` (FR-044): list, filter, sort and paging on derived fields, `POST` is `405`, no `/{id}` route; a `@spine` report keeps a row with no facts, and a measure with `@default` filters and sorts as its default. Also carries `schema.postgres.sql`, the TypeScript-produced table and views the C# lane executes | generated only | +| `report/` | 27 | a view-backed `object.report` (FR-044): list, filter, sort and paging on derived fields, `POST` is `405`, no `/{id}` route; a `@spine` report keeps a row with no facts, and a measure with `@default` filters and sorts as its default; a dimension over an array or a map is neither filterable nor sortable. Also carries `schema.postgres.sql`, the TypeScript-produced table and views the C# lane executes | generated only | +| `decimal/` | 8 | `field.decimal` is an exact canonical string on the wire (ADR-0019 Amendment 1): read, filter, sort, write and a report over decimals. A scenario's `engines:` key is exact; the `-15` twins run on SQLite and D1 as well. Also carries `schema.postgres.sql` / `schema.sqlite.sql` / `schema.mysql.sql` | generated only | `meta.json` declares a single canonical `Author` entity in the `acme::blog` package: @@ -88,6 +90,8 @@ description: > Free-text describing what behavior the scenario verifies. setup: # optional truncate: true # opt-in: empty the table before this scenario's requests +engines: [postgres, mysql] # optional, EXACT when present: only these engines run the file + # (postgres | mysql | sqlite | d1); absent = every engine the lane has requests: - id: r1 # stable id per request (referenced in test logs) method: GET # GET | POST | PATCH | PUT | DELETE diff --git a/fixtures/api-contract-conformance/decimal/README.md b/fixtures/api-contract-conformance/decimal/README.md new file mode 100644 index 000000000..5b0cae49c --- /dev/null +++ b/fixtures/api-contract-conformance/decimal/README.md @@ -0,0 +1,57 @@ +# `api-contract-conformance/decimal/` — a decimal is exact on the wire (ADR-0019 Amendment 1) + +Cross-port REST contract for `field.decimal`, over one writable entity (`Entry`, a +`decimal(28,10)` `amount`) and one served report (`EntryTotals`) read from it. + +## The rule (W1 to W4) + +- **W1, emit.** A decimal is a JSON **string** in canonical form, `^-?(0|[1-9][0-9]*)(\.[0-9]*[1-9])?$`: + plain notation, no exponent, no trailing fraction zeros, zero as `"0"`. The server never formats + a scale (`100.50` is stored; `"100.5"` is sent). The same holds for a report's decimal measures + (a sum, an average, a maximum, a ratio) and for a `null`: `null` stays `null`. +- **W2, accept.** A request body carries a decimal as a string in the JSON-number grammar (an + exponent is allowed: `"1.5e1"` is `15`) or as a JSON number. Anything else (`"12abc"`, `"NaN"`, + `true`, an object) is `400 {"error": "validation"}`. A filter operand uses the same grammar and + is compared **exactly**: `12345678901234567.89` and `12345678901234567.8` are different numbers + though they are one double. A bad operand is `400 {"error": "invalid_filter_value", "field": ...}`. +- **W3, range.** A value the declared column cannot hold (more fraction digits than `@scale`, more + integer digits than `@precision - @scale`) is `400 validation` on input. Nothing is rounded. +- **W4, floating engines.** SQLite and D1 have no decimal: they keep a double, which is exact to + 15 significant digits. A port on them renders a read at 15 significant digits, so `0.1 + 0.2` is + `"0.3"`. They do not promise `12345678901234567.89`, and equality against a computed double + (a view's sum) is not promised either, so the `-15` scenarios use ranges there. + +## Scenarios + +`engines:` is exact (see the persistence corpus README): a scenario with it runs only on the +listed engines; a scenario without runs on all four. A `-15` scenario is the subset of its twin +whose values fit 15 significant digits (buckets `A`, `D`, `E`, `F`), which every engine runs. + +| Scenario | Engines | What it pins | +|---|---|---| +| `decimal-read-exact` | postgres, mysql | list and by-id reads are canonical strings; `100.50` reads `"100.5"`, `0.0000000000` reads `"0"`, 19 digits survive | +| `decimal-read-exact-15` | all | the 15-digit subset | +| `decimal-filter-sort-exact` | postgres, mysql | `eq` / `gt` / `lt` / `in` are exact; `100.500` and `1.005e2` equal `100.5`; a numeric, not textual, sort; three bad operands answer `invalid_filter_value` | +| `decimal-filter-sort-exact-15` | all | the 15-digit subset, plus a bad operand | +| `decimal-write-exact` | postgres, mysql | a 27-digit value, a JSON number, an exponent string and a trailing-zero string are stored canonical; garbage, an 11th fraction digit, a 19th integer digit and a boolean are `400 validation` and persist nothing | +| `decimal-write-exact-15` | all | the 15-digit subset | +| `decimal-report-exact` | postgres, mysql | the `EntryTotals` rows are exactly `seed.json` `reports`; a decimal measure filters exactly; `entries` (a count) is a JSON number | +| `decimal-report-exact-15` | all | buckets A, D, E, F of the report | + +## Files + +``` +decimal/ +├── README.md +├── meta.json # Entry + the EntryTotals report (served at /api/entry_totals) +├── seed.json # `entries` (eleven rows) and `reports` (what v_entry_totals returns) +├── schema.postgres.sql # TypeScript-produced: the table and the view +├── schema.sqlite.sql # the same, for SQLite and D1 +├── schema.mysql.sql # MySQL 8.4: the hand-written table plus the view buildReportViews lowers +└── scenarios/ +``` + +`seed.json` stores amounts as **strings** (`"100.50"`, `"0.0000000000"`), never numbers, so a +port can build its own decimal from the text without a float in between. `reports` is what the +seam lanes serve in place of the view; a TypeScript test holds it equal to what the views return +on Postgres and MySQL. Ids are numbers on the wire. diff --git a/fixtures/api-contract-conformance/decimal/meta.json b/fixtures/api-contract-conformance/decimal/meta.json new file mode 100644 index 000000000..9b35e3286 --- /dev/null +++ b/fixtures/api-contract-conformance/decimal/meta.json @@ -0,0 +1,25 @@ +{ "metadata.root": { + "package": "acme::ledger", + "children": [ + { "object.entity": { "name": "Entry", "children": [ + { "source.rdb": { "@table": "entries" } }, + { "field.long": { "name": "id", "@filterable": true, "@sortable": true } }, + { "field.string": { "name": "bucket", "@required": true, "@maxLength": 8, "@filterable": true, "@sortable": true } }, + { "field.decimal": { "name": "amount", "@precision": 28, "@scale": 10, "@filterable": true, "@sortable": true } }, + { "field.boolean": { "name": "flagged", "@required": true } }, + { "identity.primary": { "name": "pk", "@fields": "id", "@generation": "increment" } }, + { "dimension.attribute": { "name": "bucket", "@of": "Entry.bucket" } }, + { "measure.aggregate": { "name": "entries", "@agg": "count", "@of": "Entry.id" } }, + { "measure.aggregate": { "name": "flaggedEntries", "@agg": "count", "@of": "Entry.id", "@filter": { "flagged": true } } }, + { "measure.aggregate": { "name": "total", "@agg": "sum", "@of": "Entry.amount" } }, + { "measure.aggregate": { "name": "totalOrZero", "@agg": "sum", "@of": "Entry.amount", "@default": 0 } }, + { "measure.aggregate": { "name": "mean", "@agg": "avg", "@of": "Entry.amount" } }, + { "measure.aggregate": { "name": "top", "@agg": "max", "@of": "Entry.amount" } }, + { "measure.ratio": { "name": "flaggedShare", "@numerator": "flaggedEntries", "@denominator": "entries" } }, + { "measure.ratio": { "name": "perFlagged", "@numerator": "entries", "@denominator": "flaggedEntries" } } + ] } }, + { "object.report": { "name": "EntryTotals", "@from": "Entry", "@dimensions": ["bucket"], + "@measures": ["entries", "total", "totalOrZero", "mean", "top", "flaggedShare", "perFlagged"], + "children": [ { "source.rdb": { "@kind": "view", "@view": "v_entry_totals" } } ] } } + ] +} } diff --git a/fixtures/api-contract-conformance/decimal/scenarios/decimal-filter-sort-exact-15.yaml b/fixtures/api-contract-conformance/decimal/scenarios/decimal-filter-sort-exact-15.yaml new file mode 100644 index 000000000..640b28de6 --- /dev/null +++ b/fixtures/api-contract-conformance/decimal/scenarios/decimal-filter-sort-exact-15.yaml @@ -0,0 +1,44 @@ +name: decimal-filter-sort-exact-15 +description: > + The decimal-filter-sort-exact cases whose values fit 15 significant digits, run on every + engine. SQLite and D1 compare a double, exactly to 15 digits (W4). +requests: + - id: r1 + method: GET + path: /api/entries?filter[amount][eq]=0.1 + expect: + status: 200 + body: + equals: + - { id: 1, bucket: A, amount: "0.1", flagged: true } + - id: r2 + method: GET + path: /api/entries?filter[amount][gt]=99.5&filter[bucket][in]=A,D,E,F + expect: + status: 200 + body: + equals: + - { id: 7, bucket: D, amount: "100.5", flagged: false } + - id: r3 + method: GET + path: /api/entries?filter[amount][eq]=1.005e2 + expect: + status: 200 + body: + equals: + - { id: 7, bucket: D, amount: "100.5", flagged: false } + - id: r4 + method: GET + path: /api/entries?filter[bucket][in]=A,D&sort=amount:desc + expect: + status: 200 + body: + ids: [7, 8, 2, 1] + - id: r5 + method: GET + path: /api/entries?filter[amount][eq]=abc + expect: + status: 400 + body: + error: invalid_filter_value + field: amount diff --git a/fixtures/api-contract-conformance/decimal/scenarios/decimal-filter-sort-exact.yaml b/fixtures/api-contract-conformance/decimal/scenarios/decimal-filter-sort-exact.yaml new file mode 100644 index 000000000..3cca53779 --- /dev/null +++ b/fixtures/api-contract-conformance/decimal/scenarios/decimal-filter-sort-exact.yaml @@ -0,0 +1,97 @@ +name: decimal-filter-sort-exact +description: > + A decimal filter operand is text in the JSON-number grammar and is compared EXACTLY (W2): + `12345678901234567.89` and `12345678901234567.8` are different numbers though they are the same + double. Trailing zeros and an exponent do not change the number. A bad operand is + 400 invalid_filter_value naming the field. Sorting is numeric, not textual. Postgres and MySQL only (W4). +engines: [postgres, mysql] +requests: + - id: r1 + method: GET + path: /api/entries?filter[amount][eq]=12345678901234567.89 + expect: + status: 200 + body: + equals: + - { id: 3, bucket: B, amount: "12345678901234567.89", flagged: false } + - id: r2 + method: GET + path: /api/entries?filter[amount][gt]=12345678901234567.8 + expect: + status: 200 + body: + equals: + - { id: 3, bucket: B, amount: "12345678901234567.89", flagged: false } + - id: r3 + method: GET + path: /api/entries?filter[amount][gt]=12345678901234567.89 + expect: + status: 200 + body: + length: 0 + - id: r4 + method: GET + path: /api/entries?filter[amount][eq]=100.500 + expect: + status: 200 + body: + equals: + - { id: 7, bucket: D, amount: "100.5", flagged: false } + - id: r5 + method: GET + path: /api/entries?filter[amount][eq]=1.005e2 + expect: + status: 200 + body: + equals: + - { id: 7, bucket: D, amount: "100.5", flagged: false } + - id: r6 + method: GET + path: /api/entries?filter[amount][lt]=0.01&sort=id:asc + expect: + status: 200 + body: + equals: + - { id: 5, bucket: C, amount: "-0.0000000001", flagged: false } + - { id: 6, bucket: C, amount: "-12345678.987654321", flagged: false } + - { id: 9, bucket: E, amount: "0", flagged: false } + - id: r7 + method: GET + path: /api/entries?filter[amount][in]=0.1,0.2&sort=id:asc + expect: + status: 200 + body: + equals: + - { id: 1, bucket: A, amount: "0.1", flagged: true } + - { id: 2, bucket: A, amount: "0.2", flagged: false } + - id: r8 + method: GET + path: /api/entries?filter[bucket][in]=A,B,C,D&sort=amount:desc + expect: + status: 200 + body: + ids: [3, 7, 8, 2, 1, 4, 5, 6] + - id: r9 + method: GET + path: /api/entries?filter[amount][eq]=abc + expect: + status: 400 + body: + error: invalid_filter_value + field: amount + - id: r10 + method: GET + path: /api/entries?filter[amount][gt]=1e + expect: + status: 400 + body: + error: invalid_filter_value + field: amount + - id: r11 + method: GET + path: /api/entries?filter[amount][eq]=0x10 + expect: + status: 400 + body: + error: invalid_filter_value + field: amount diff --git a/fixtures/api-contract-conformance/decimal/scenarios/decimal-read-exact-15.yaml b/fixtures/api-contract-conformance/decimal/scenarios/decimal-read-exact-15.yaml new file mode 100644 index 000000000..f6ac6894e --- /dev/null +++ b/fixtures/api-contract-conformance/decimal/scenarios/decimal-read-exact-15.yaml @@ -0,0 +1,28 @@ +name: decimal-read-exact-15 +description: > + The subset of decimal-read-exact whose values fit 15 significant digits (buckets A, D, E, F), + which every engine runs (W4). SQLite and D1 keep a decimal as a double, so `0.1` is the + canonical "0.1" only because the read renders the double at 15 significant digits. +requests: + - id: r1 + method: GET + path: /api/entries?filter[bucket][in]=A,D,E,F&sort=id:asc + expect: + status: 200 + body: + equals: + - { id: 1, bucket: A, amount: "0.1", flagged: true } + - { id: 2, bucket: A, amount: "0.2", flagged: false } + - { id: 7, bucket: D, amount: "100.5", flagged: false } + - { id: 8, bucket: D, amount: "99.5", flagged: false } + - { id: 9, bucket: E, amount: "0", flagged: false } + - { id: 10, bucket: E, amount: null, flagged: false } + - { id: 11, bucket: F, amount: null, flagged: false } + - id: r2 + method: GET + path: /api/entries/7 + expect: + status: 200 + body: + row: + amount: "100.5" diff --git a/fixtures/api-contract-conformance/decimal/scenarios/decimal-read-exact.yaml b/fixtures/api-contract-conformance/decimal/scenarios/decimal-read-exact.yaml new file mode 100644 index 000000000..904177409 --- /dev/null +++ b/fixtures/api-contract-conformance/decimal/scenarios/decimal-read-exact.yaml @@ -0,0 +1,53 @@ +name: decimal-read-exact +description: > + A field.decimal is a canonical JSON string on the wire (ADR-0019 Amendment 1, W1): plain + notation, no trailing fraction zeros, zero as "0", never a JSON number. The seed stores + `100.50` and `0.0000000000`; the wire says "100.5" and "0". `12345678901234567.89` has 19 + significant digits, more than a double holds, so a route that converts a decimal to a number + anywhere answers the wrong text. Postgres and MySQL only (W4): decimal-read-exact-15 is the + subset every engine runs. +engines: [postgres, mysql] +requests: + - id: r1 + method: GET + path: /api/entries?sort=id:asc + expect: + status: 200 + body: + equals: + - { id: 1, bucket: A, amount: "0.1", flagged: true } + - { id: 2, bucket: A, amount: "0.2", flagged: false } + - { id: 3, bucket: B, amount: "12345678901234567.89", flagged: false } + - { id: 4, bucket: B, amount: "0.01", flagged: true } + - { id: 5, bucket: C, amount: "-0.0000000001", flagged: false } + - { id: 6, bucket: C, amount: "-12345678.987654321", flagged: false } + - { id: 7, bucket: D, amount: "100.5", flagged: false } + - { id: 8, bucket: D, amount: "99.5", flagged: false } + - { id: 9, bucket: E, amount: "0", flagged: false } + - { id: 10, bucket: E, amount: null, flagged: false } + - { id: 11, bucket: F, amount: null, flagged: false } + - id: r2 + method: GET + path: /api/entries/3 + expect: + status: 200 + body: + row: + id: 3 + amount: "12345678901234567.89" + - id: r3 + method: GET + path: /api/entries/7 + expect: + status: 200 + body: + row: + amount: "100.5" + - id: r4 + method: GET + path: /api/entries/9 + expect: + status: 200 + body: + row: + amount: "0" diff --git a/fixtures/api-contract-conformance/decimal/scenarios/decimal-report-exact-15.yaml b/fixtures/api-contract-conformance/decimal/scenarios/decimal-report-exact-15.yaml new file mode 100644 index 000000000..d372df213 --- /dev/null +++ b/fixtures/api-contract-conformance/decimal/scenarios/decimal-report-exact-15.yaml @@ -0,0 +1,34 @@ +name: decimal-report-exact-15 +description: > + The decimal-report-exact buckets whose values fit 15 significant digits (A, D, E, F), run on + every engine. SQLite and D1 compute the sum of 0.1 and 0.2 as the double 0.30000000000000004; the + wire says "0.3" (W4). The filter is a range because equality on a computed double is + not promised: 0.30000000000000004 is not the number 0.3, and W4 makes SQLite exact to 15 digits only for what it stores. +requests: + - id: r1 + method: GET + path: /api/entry_totals?filter[bucket][in]=A,D,E,F&sort=bucket:asc + expect: + status: 200 + body: + equals: + - { bucket: "A", entries: 2, total: "0.3", totalOrZero: "0.3", mean: "0.15", top: "0.2", flaggedShare: "0.5", perFlagged: "2" } + - { bucket: "D", entries: 2, total: "200", totalOrZero: "200", mean: "100", top: "100.5", flaggedShare: "0", perFlagged: null } + - { bucket: "E", entries: 2, total: "0", totalOrZero: "0", mean: "0", top: "0", flaggedShare: "0", perFlagged: null } + - { bucket: "F", entries: 1, total: null, totalOrZero: "0", mean: null, top: null, flaggedShare: "0", perFlagged: null } + - id: r2 + method: GET + path: /api/entry_totals?filter[bucket][in]=A,D,E,F&filter[total][gt]=0.25&filter[total][lt]=1 + expect: + status: 200 + body: + equals: + - { bucket: "A", entries: 2, total: "0.3", totalOrZero: "0.3", mean: "0.15", top: "0.2", flaggedShare: "0.5", perFlagged: "2" } + - id: r3 + method: GET + path: /api/entry_totals?filter[bucket][in]=A,D,E,F&filter[total][isNull]=true + expect: + status: 200 + body: + equals: + - { bucket: "F", entries: 1, total: null, totalOrZero: "0", mean: null, top: null, flaggedShare: "0", perFlagged: null } diff --git a/fixtures/api-contract-conformance/decimal/scenarios/decimal-report-exact.yaml b/fixtures/api-contract-conformance/decimal/scenarios/decimal-report-exact.yaml new file mode 100644 index 000000000..3ad7720fe --- /dev/null +++ b/fixtures/api-contract-conformance/decimal/scenarios/decimal-report-exact.yaml @@ -0,0 +1,53 @@ +name: decimal-report-exact +description: > + A report over a decimal is exact on the wire: a sum of 12345678901234567.89 and 0.01 is + "12345678901234567.9", an average keeps its digits, an empty sum is null (the key is present) unless + the measure declares @default (then "0"), and a ratio with a zero denominator is null. A decimal + measure filters exactly (W2). The count `entries` is a JSON number. Postgres and MySQL only (W4). +engines: [postgres, mysql] +requests: + - id: r1 + method: GET + path: /api/entry_totals?sort=bucket:asc + expect: + status: 200 + body: + equals: + - { bucket: "A", entries: 2, total: "0.3", totalOrZero: "0.3", mean: "0.15", top: "0.2", flaggedShare: "0.5", perFlagged: "2" } + - { bucket: "B", entries: 2, total: "12345678901234567.9", totalOrZero: "12345678901234567.9", mean: "6172839450617283.95", top: "12345678901234567.89", flaggedShare: "0.5", perFlagged: "2" } + - { bucket: "C", entries: 2, total: "-12345678.9876543211", totalOrZero: "-12345678.9876543211", mean: "-6172839.49382716055", top: "-0.0000000001", flaggedShare: "0", perFlagged: null } + - { bucket: "D", entries: 2, total: "200", totalOrZero: "200", mean: "100", top: "100.5", flaggedShare: "0", perFlagged: null } + - { bucket: "E", entries: 2, total: "0", totalOrZero: "0", mean: "0", top: "0", flaggedShare: "0", perFlagged: null } + - { bucket: "F", entries: 1, total: null, totalOrZero: "0", mean: null, top: null, flaggedShare: "0", perFlagged: null } + - id: r2 + method: GET + path: /api/entry_totals?filter[total][eq]=12345678901234567.9 + expect: + status: 200 + body: + equals: + - { bucket: "B", entries: 2, total: "12345678901234567.9", totalOrZero: "12345678901234567.9", mean: "6172839450617283.95", top: "12345678901234567.89", flaggedShare: "0.5", perFlagged: "2" } + - id: r3 + method: GET + path: /api/entry_totals?filter[top][gt]=12345678901234567.8 + expect: + status: 200 + body: + equals: + - { bucket: "B", entries: 2, total: "12345678901234567.9", totalOrZero: "12345678901234567.9", mean: "6172839450617283.95", top: "12345678901234567.89", flaggedShare: "0.5", perFlagged: "2" } + - id: r4 + method: GET + path: /api/entry_totals?filter[mean][gt]=100&sort=bucket:asc + expect: + status: 200 + body: + equals: + - { bucket: "B", entries: 2, total: "12345678901234567.9", totalOrZero: "12345678901234567.9", mean: "6172839450617283.95", top: "12345678901234567.89", flaggedShare: "0.5", perFlagged: "2" } + - id: r5 + method: GET + path: /api/entry_totals?filter[total][isNull]=true + expect: + status: 200 + body: + equals: + - { bucket: "F", entries: 1, total: null, totalOrZero: "0", mean: null, top: null, flaggedShare: "0", perFlagged: null } diff --git a/fixtures/api-contract-conformance/decimal/scenarios/decimal-write-exact-15.yaml b/fixtures/api-contract-conformance/decimal/scenarios/decimal-write-exact-15.yaml new file mode 100644 index 000000000..d1d8750da --- /dev/null +++ b/fixtures/api-contract-conformance/decimal/scenarios/decimal-write-exact-15.yaml @@ -0,0 +1,72 @@ +name: decimal-write-exact-15 +description: > + The decimal-write-exact cases whose values fit 15 significant digits, run on every engine. A + JSON number is read exactly to 15 significant digits (W4), so 0.3 is "0.3". +requests: + - id: r1 + method: POST + path: /api/entries + body: { bucket: W, amount: "0.1", flagged: false } + expect: + status: 201 + body: + row: + amount: "0.1" + hasId: true + - id: r2 + method: POST + path: /api/entries + body: { bucket: W, amount: 0.3, flagged: false } + expect: + status: 201 + body: + row: + amount: "0.3" + - id: r3 + method: POST + path: /api/entries + body: { bucket: W, amount: "1e-2", flagged: false } + expect: + status: 201 + body: + row: + amount: "0.01" + - id: r4 + method: POST + path: /api/entries + body: { bucket: W, amount: "100.2500", flagged: false } + expect: + status: 201 + body: + row: + amount: "100.25" + - id: r5 + method: GET + path: /api/entries?filter[bucket][eq]=W&sort=id:asc + expect: + status: 200 + body: + length: 4 + - id: r6 + method: POST + path: /api/entries + body: { bucket: Y, amount: "12abc", flagged: false } + expect: + status: 400 + body: + error: validation + - id: r7 + method: POST + path: /api/entries + body: { bucket: Y, amount: "NaN", flagged: false } + expect: + status: 400 + body: + error: validation + - id: r8 + method: GET + path: /api/entries?filter[bucket][eq]=Y + expect: + status: 200 + body: + length: 0 diff --git a/fixtures/api-contract-conformance/decimal/scenarios/decimal-write-exact.yaml b/fixtures/api-contract-conformance/decimal/scenarios/decimal-write-exact.yaml new file mode 100644 index 000000000..687a666b3 --- /dev/null +++ b/fixtures/api-contract-conformance/decimal/scenarios/decimal-write-exact.yaml @@ -0,0 +1,91 @@ +name: decimal-write-exact +description: > + A decimal comes in as a string in the JSON-number grammar or as a JSON number (W2), and is + stored and answered as its canonical string. A value the column cannot hold (more fraction digits + than @scale, more integer digits than @precision - @scale) or a string that is not a number is + 400 validation (W3); nothing is rounded. @precision 28 and @scale 10 here. Postgres and MySQL only (W4). +engines: [postgres, mysql] +requests: + - id: r1 + method: POST + path: /api/entries + body: { bucket: W, amount: "12345678901234567.8901234567", flagged: false } + expect: + status: 201 + body: + row: + amount: "12345678901234567.8901234567" + hasId: true + - id: r2 + method: GET + path: /api/entries?filter[amount][eq]=12345678901234567.8901234567 + expect: + status: 200 + body: + length: 1 + - id: r3 + method: POST + path: /api/entries + body: { bucket: X, amount: 0.3, flagged: false } + expect: + status: 201 + body: + row: + amount: "0.3" + - id: r4 + method: POST + path: /api/entries + body: { bucket: X, amount: "1.5e1", flagged: false } + expect: + status: 201 + body: + row: + amount: "15" + - id: r5 + method: POST + path: /api/entries + body: { bucket: X, amount: "100.2500", flagged: false } + expect: + status: 201 + body: + row: + amount: "100.25" + - id: r6 + method: POST + path: /api/entries + body: { bucket: Y, amount: "12abc", flagged: false } + expect: + status: 400 + body: + error: validation + - id: r7 + method: POST + path: /api/entries + body: { bucket: Y, amount: "0.12345678901", flagged: false } + expect: + status: 400 + body: + error: validation + - id: r8 + method: POST + path: /api/entries + body: { bucket: Y, amount: "1234567890123456789", flagged: false } + expect: + status: 400 + body: + error: validation + - id: r9 + method: POST + path: /api/entries + body: { bucket: Y, amount: true, flagged: false } + expect: + status: 400 + body: + error: validation + - id: r10 + method: GET + path: /api/entries?filter[bucket][eq]=Y + expect: + status: 200 + body: + length: 0 diff --git a/fixtures/api-contract-conformance/decimal/schema.mysql.sql b/fixtures/api-contract-conformance/decimal/schema.mysql.sql new file mode 100644 index 000000000..064d6dd7b --- /dev/null +++ b/fixtures/api-contract-conformance/decimal/schema.mysql.sql @@ -0,0 +1,23 @@ +-- @generated by @metaobjectsdev/integration-tests — DO NOT EDIT. +-- api-contract decimal sub-corpus schema (MySQL 8.4, literal column naming). +-- Regenerate: `bun run gen:decimal-api-schema` in server/typescript/packages/integration-tests. + +CREATE TABLE `entries` ( + `id` BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY, + `bucket` VARCHAR(8) NOT NULL, + `amount` DECIMAL(28,10), + `flagged` BOOLEAN NOT NULL +); + +CREATE VIEW `v_entry_totals` AS + SELECT + e.`bucket` AS `bucket`, + COUNT(e.`id`) AS `entries`, + SUM(e.`amount`) AS `total`, + COALESCE(SUM(e.`amount`), 0) AS `totalOrZero`, + AVG(e.`amount`) AS `mean`, + MAX(e.`amount`) AS `top`, + COUNT(CASE WHEN e.`flagged` = TRUE THEN e.`id` END) / NULLIF(COUNT(e.`id`), 0) AS `flaggedShare`, + COUNT(e.`id`) / NULLIF(COUNT(CASE WHEN e.`flagged` = TRUE THEN e.`id` END), 0) AS `perFlagged` + FROM `entries` e + GROUP BY e.`bucket`; diff --git a/fixtures/api-contract-conformance/decimal/schema.postgres.sql b/fixtures/api-contract-conformance/decimal/schema.postgres.sql new file mode 100644 index 000000000..85d48e3fd --- /dev/null +++ b/fixtures/api-contract-conformance/decimal/schema.postgres.sql @@ -0,0 +1,25 @@ +-- @generated by @metaobjectsdev/integration-tests — DO NOT EDIT. +-- api-contract decimal sub-corpus schema (Postgres, literal column naming). +-- Regenerate: `bun run gen:decimal-api-schema` in server/typescript/packages/integration-tests. + +CREATE TABLE "entries" ( + "id" BIGINT GENERATED BY DEFAULT AS IDENTITY NOT NULL, + "bucket" VARCHAR(8) NOT NULL, + "amount" NUMERIC(28,10), + "flagged" BOOLEAN NOT NULL, + CONSTRAINT "entries_pkey" PRIMARY KEY ("id") +); + +CREATE VIEW "v_entry_totals" AS + SELECT + e."bucket" AS "bucket", + COUNT(e."id") AS "entries", + SUM(e."amount") AS "total", + COALESCE(SUM(e."amount"), 0) AS "totalOrZero", + AVG(e."amount") AS "mean", + MAX(e."amount") AS "top", + CAST(COUNT(e."id") FILTER (WHERE e."flagged" = TRUE) AS NUMERIC) / NULLIF(COUNT(e."id"), 0) AS "flaggedShare", + CAST(COUNT(e."id") AS NUMERIC) / NULLIF(COUNT(e."id") FILTER (WHERE e."flagged" = TRUE), 0) AS "perFlagged" + FROM "entries" e + GROUP BY e."bucket"; +COMMENT ON VIEW "v_entry_totals" IS 'metaobjects:v1:sha256:5290cebc2ffad56830125b30824ac07830ca635415bddc97572e015f77d51be9'; diff --git a/fixtures/api-contract-conformance/decimal/schema.sqlite.sql b/fixtures/api-contract-conformance/decimal/schema.sqlite.sql new file mode 100644 index 000000000..7d4c72fb7 --- /dev/null +++ b/fixtures/api-contract-conformance/decimal/schema.sqlite.sql @@ -0,0 +1,23 @@ +-- @generated by @metaobjectsdev/integration-tests — DO NOT EDIT. +-- api-contract decimal sub-corpus schema (SQLite, literal column naming). +-- Regenerate: `bun run gen:decimal-api-schema` in server/typescript/packages/integration-tests. + +CREATE TABLE "entries" ( + "id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, + "bucket" VARCHAR(8) NOT NULL, + "amount" NUMERIC(28,10), + "flagged" INTEGER NOT NULL +); + +CREATE VIEW "v_entry_totals" AS + SELECT + e."bucket" AS "bucket", + COUNT(e."id") AS "entries", + SUM(e."amount") AS "total", + COALESCE(SUM(e."amount"), 0.0) AS "totalOrZero", + AVG(e."amount") AS "mean", + MAX(e."amount") AS "top", + CAST(COUNT(CASE WHEN e."flagged" = 1 THEN e."id" END) AS REAL) / NULLIF(COUNT(e."id"), 0) AS "flaggedShare", + CAST(COUNT(e."id") AS REAL) / NULLIF(COUNT(CASE WHEN e."flagged" = 1 THEN e."id" END), 0) AS "perFlagged" + FROM "entries" e + GROUP BY e."bucket"; diff --git a/fixtures/api-contract-conformance/decimal/seed.json b/fixtures/api-contract-conformance/decimal/seed.json new file mode 100644 index 000000000..3cd36f226 --- /dev/null +++ b/fixtures/api-contract-conformance/decimal/seed.json @@ -0,0 +1,25 @@ +{ + "entries": [ + { "id": 1, "bucket": "A", "amount": "0.1", "flagged": true }, + { "id": 2, "bucket": "A", "amount": "0.2", "flagged": false }, + { "id": 3, "bucket": "B", "amount": "12345678901234567.89", "flagged": false }, + { "id": 4, "bucket": "B", "amount": "0.01", "flagged": true }, + { "id": 5, "bucket": "C", "amount": "-0.0000000001", "flagged": false }, + { "id": 6, "bucket": "C", "amount": "-12345678.987654321", "flagged": false }, + { "id": 7, "bucket": "D", "amount": "100.50", "flagged": false }, + { "id": 8, "bucket": "D", "amount": "99.5", "flagged": false }, + { "id": 9, "bucket": "E", "amount": "0.0000000000", "flagged": false }, + { "id": 10, "bucket": "E", "amount": null, "flagged": false }, + { "id": 11, "bucket": "F", "amount": null, "flagged": false } + ], + "reports": { + "EntryTotals": [ + { "bucket": "A", "entries": 2, "total": "0.3", "totalOrZero": "0.3", "mean": "0.15", "top": "0.2", "flaggedShare": "0.5", "perFlagged": "2" }, + { "bucket": "B", "entries": 2, "total": "12345678901234567.9", "totalOrZero": "12345678901234567.9", "mean": "6172839450617283.95", "top": "12345678901234567.89", "flaggedShare": "0.5", "perFlagged": "2" }, + { "bucket": "C", "entries": 2, "total": "-12345678.9876543211", "totalOrZero": "-12345678.9876543211", "mean": "-6172839.49382716055", "top": "-0.0000000001", "flaggedShare": "0", "perFlagged": null }, + { "bucket": "D", "entries": 2, "total": "200", "totalOrZero": "200", "mean": "100", "top": "100.5", "flaggedShare": "0", "perFlagged": null }, + { "bucket": "E", "entries": 2, "total": "0", "totalOrZero": "0", "mean": "0", "top": "0", "flaggedShare": "0", "perFlagged": null }, + { "bucket": "F", "entries": 1, "total": null, "totalOrZero": "0", "mean": null, "top": null, "flaggedShare": "0", "perFlagged": null } + ] + } +} diff --git a/fixtures/api-contract-conformance/projection/README.md b/fixtures/api-contract-conformance/projection/README.md index 544dcf19a..17a574683 100644 --- a/fixtures/api-contract-conformance/projection/README.md +++ b/fixtures/api-contract-conformance/projection/README.md @@ -43,8 +43,9 @@ the cross-port contract would drop a capability two ports already shipped. inherited `@fields` (`id`) instead of the computed key broke here: TypeScript mounted no item route, Kotlin emitted a table and controller that did not compile, and C# built a model with no key for the projection, so every request answered `500`. -- **Filters apply to `field.decimal` and `field.float`.** `InvoiceLedger` carries one of each; - the scenarios assert only how many rows match, because each port spells a decimal its own way. +- **Filters apply to `field.decimal` and `field.float`.** `InvoiceLedger` carries one of each. + A decimal is asserted as its canonical string (`3.0` is `"3"`, ADR-0019 Amendment 1), and its + operand compares exactly (`eq=12.50` matches `12.5`). - **The api docs list the same routes** (`docs-routes.json`, see below). - **A filter error names the field**, exactly as on a writable route (see `docs/features/api-contract.md` → "Error response"). diff --git a/fixtures/api-contract-conformance/projection/scenarios/filter-decimal.yaml b/fixtures/api-contract-conformance/projection/scenarios/filter-decimal.yaml index 20458ca9e..b3527e364 100644 --- a/fixtures/api-contract-conformance/projection/scenarios/filter-decimal.yaml +++ b/fixtures/api-contract-conformance/projection/scenarios/filter-decimal.yaml @@ -2,9 +2,9 @@ name: projection-filter-decimal description: > The FR-009 filter grammar applies to a `field.decimal` of a projection, in every port. The operand is parsed as a decimal and compared as one (Kotlin answered a - 500 on this before the decimal and float coercers were split). Only how many rows - match is asserted, because each port spells a decimal its own way on the wire. - Seed discounts: 7.25, 12.5, 3.0, 1.0. + 500 on this before the decimal and float coercers were split). A decimal reaches the + wire as the canonical string of ADR-0019 Amendment 1: the seed's 3.0 is "3", 12.5 is + "12.5". Seed discounts: 7.25, 12.5, 3.0, 1.0. requests: - id: gt method: GET @@ -12,9 +12,19 @@ requests: expect: { status: 200, body: { length: 1 } } - id: lte method: GET - path: "/api/invoice_ledgers?filter[discount][lte]=7.25" - expect: { status: 200, body: { length: 3 } } + path: "/api/invoice_ledgers?filter[discount][lte]=7.25&sort=number:asc" + expect: + status: 200 + body: + equals: + - { number: 1, reference: "INV-1001", discount: "7.25", weight: 0.5 } + - { number: 3, reference: "INV-1003", discount: "3", weight: 2.5 } + - { number: 4, reference: "INV-1004", discount: "1", weight: 0.25 } - id: eq method: GET - path: "/api/invoice_ledgers?filter[discount][eq]=12.5" - expect: { status: 200, body: { length: 1 } } + path: "/api/invoice_ledgers?filter[discount][eq]=12.50" + expect: + status: 200 + body: + equals: + - { number: 2, reference: "INV-1002", discount: "12.5", weight: 1.5 } diff --git a/fixtures/api-contract-conformance/report/README.md b/fixtures/api-contract-conformance/report/README.md index e417a142e..c93860a05 100644 --- a/fixtures/api-contract-conformance/report/README.md +++ b/fixtures/api-contract-conformance/report/README.md @@ -51,11 +51,9 @@ object uses (`InvoicesByMonth` is `/api/invoices_by_months`). accepted in every port and orders by the stored value, and `?filter[status][eq]=OPEN` matches it. It reaches the wire as its string symbol; in C# that is host configuration (a `JsonStringEnumConverter`), as for any enum the routes return. -- **A decimal's spelling is not asserted.** `paidShare` is a ratio, so it is a decimal, - and each port spells a decimal its own way. The scenarios that touch it assert only - how many rows match. - TypeScript sends a string, on SQLite as well as Postgres, and holds that in its own SQLite lane - (`integration-tests/test/api-contract-report-sqlite.test.ts`), outside this cross-port corpus. +- **A ratio is a decimal**, so `paidShare` reaches the wire as the canonical string + (`"0.4"` for 2/5) in every port and on every engine (ADR-0019 Amendment 1). Exactness + beyond that is the `decimal/` sub-corpus's job. - The default page size stays per port, as documented: TypeScript and C# return every row when `limit` is omitted, Java, Kotlin and Python the first 50. The scenarios that page pass `limit` explicitly. @@ -129,22 +127,19 @@ report/ Every top-level key but `reports` is a **base table**, named by its table name and listed **parents first** (`products` before `sales`, which references it): a lane inserts the -keys in file order and every foreign key holds. The full-stack lanes (TypeScript, C#) -insert them and let the real views derive the report rows. `reports` is what the -views return for those rows, and it is what the **seam lanes** (Java, Kotlin, Python) -serve: their in-memory repository or H2 table stands in for the view behind the consumer -seam. A TypeScript test holds the two halves together, so the seam lanes cannot drift -from what the SQL returns, and another holds the base tables to the model and to that -order. `paidShare` is a string in the seed so a seam lane can build its own decimal from -it without a float in between. +keys in file order and every foreign key holds. Every lane is full stack: it inserts them +and lets the real views derive the report rows. `reports` is what the views return for +those rows, and lanes check their views against it (TypeScript and Python on every engine +they run), so a view that drifts from the expected rows fails before any scenario does. +Another test holds the base tables to the model and to that order. ### Why the view SQL is a committed artifact **View SQL is produced by TypeScript only** (ADR-0015). No other port emits SQL for a report, in product code or in a test harness. `schema.postgres.sql` is what TypeScript produces from `meta.json` (literal column naming), committed and drift-checked by -`test/api-contract-report-corpus.test.ts`, and the C# lane executes that file. Java, -Kotlin and Python do not run it: they seed `reports` rows behind their seam. +`test/api-contract-report-corpus.test.ts`, and every non-TypeScript lane executes that +file (and `schema.sqlite.sql` / `schema.mysql.sql` on those engines). Regenerate it with `bun run gen:report-api-schema` in `server/typescript/packages/integration-tests`. @@ -158,11 +153,11 @@ hand-rolled reference server would answer every scenario by construction. | Port | Generated lane | Note | |---|---|---| -| TypeScript | wired | `server/typescript/packages/integration-tests/test/api-contract-report.test.ts` (25 scenarios + a seed-vs-view check). Full stack: generated Fastify routes over the real views on Testcontainers Postgres. `api-contract-report-sqlite.test.ts` runs the same 25 on SQLite, `api-contract-report-mysql.test.ts` on MySQL 8.4, and the SQLite test file also runs them through local D1 (Miniflare) | +| TypeScript | wired | `server/typescript/packages/integration-tests/test/api-contract-report.test.ts` (27 scenarios + a seed-vs-view check). Full stack: generated Fastify routes over the real views on Testcontainers Postgres. `api-contract-report-sqlite.test.ts` runs the same 27 on SQLite, `api-contract-report-mysql.test.ts` on MySQL 8.4, and the SQLite test file also runs them through local D1 (Miniflare) | | C# | wired | `server/csharp/MetaObjects.IntegrationTests/Api/ApiContractReportConformanceTest.cs`. Full stack: generated routes and EF Core over `schema.postgres.sql` on Testcontainers Postgres, over `schema.sqlite.sql` on SQLite, and over `schema.mysql.sql` on MySQL 8.4 | -| Java | wired | `server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportGeneratedApiContractConformanceTest.java` (25 scenarios + a scenario-count check). Generated controllers behind an in-memory repository seeded from `reports` | -| Kotlin | wired | `server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/report/ReportGeneratedApiContractConformanceTest.kt` (25 scenarios, the count check, and a check that the sourceless report generated nothing). Generated controllers over the generated Exposed table objects, seeded from `reports` | -| Python | wired | `server/python/tests/integration/test_api_contract_report.py` (25 scenarios, a check that exactly the served reports are generated, and `/api/invoice_days` is `404`). Generated routers behind in-memory repositories seeded from `reports` | +| Java | wired | `server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdb{Postgres,Sqlite,MySql}ApiContractConformanceTest.java` (27 scenarios + a scenario-count check, on each engine). Full stack: generated controllers over OMDB reading the real views | +| Kotlin | wired | `server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/report/ReportGeneratedApiContractConformanceTest.kt` (27 scenarios, the count check, and a check that the sourceless report generated nothing). Full stack: generated controllers over the generated Exposed table objects, reading the real views on Postgres, SQLite and MySQL | +| Python | wired | `server/python/tests/integration/test_api_contract_report.py` (27 scenarios, a check that exactly the served reports are generated, a views-vs-seed check, and `/api/invoice_days` is `404`). Full stack: generated FastAPI routers over `ObjectManager`, reading the real views on Postgres, SQLite and MySQL | The scenarios use only assertion keys every runner already has (`equals`, `length`, `envelope`, `error` with `field`, and a status with no `body`). diff --git a/fixtures/api-contract-conformance/report/meta.json b/fixtures/api-contract-conformance/report/meta.json index 8d69d6b55..51c12e83c 100644 --- a/fixtures/api-contract-conformance/report/meta.json +++ b/fixtures/api-contract-conformance/report/meta.json @@ -138,6 +138,26 @@ "@dimensions": ["productId", "productName"], "@measures": ["sales", "hasRefund", "hasRefundOrFalse"], "children": [ { "source.rdb": { "@kind": "view", "@view": "v_product_refunds" } } ] + }}, + { "object.entity": { + "name": "Ticket", + "children": [ + { "source.rdb": { "@table": "tickets" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "tags", "isArray": true } }, + { "field.map": { "name": "attrs", "@valueType": "string" } }, + { "identity.primary": { "name": "pk", "@fields": "id", "@generation": "increment" } }, + { "dimension.attribute": { "name": "tags", "@of": "Ticket.tags" } }, + { "dimension.attribute": { "name": "attrs", "@of": "Ticket.attrs" } }, + { "measure.aggregate": { "name": "tickets", "@agg": "count", "@of": "Ticket.id" } } + ] + }}, + { "object.report": { + "name": "TicketsByTags", + "@from": "Ticket", + "@dimensions": ["tags", "attrs"], + "@measures": ["tickets"], + "children": [ { "source.rdb": { "@kind": "view", "@view": "v_tickets_by_tags" } } ] }} ] } diff --git a/fixtures/api-contract-conformance/report/scenarios/filter-array-and-map-dimension.yaml b/fixtures/api-contract-conformance/report/scenarios/filter-array-and-map-dimension.yaml new file mode 100644 index 000000000..f3cf6703b --- /dev/null +++ b/fixtures/api-contract-conformance/report/scenarios/filter-array-and-map-dimension.yaml @@ -0,0 +1,29 @@ +name: report-filter-array-and-map-dimension +description: > + Filtering on an array dimension (`tags`) or a map dimension (`attrs`) is rejected with the + `invalid_filter_field` envelope naming the field: neither carries a filter band, so neither + is in the generated allowlist. +requests: + - id: r1 + method: GET + path: /api/tickets_by_tags?filter[tags][eq]=urgent + expect: + status: 400 + body: + error: "invalid_filter_field" + field: "tags" + - id: r2 + method: GET + path: /api/tickets_by_tags?filter[attrs][eq]=x + expect: + status: 400 + body: + error: "invalid_filter_field" + field: "attrs" + - id: r3 + method: GET + path: /api/tickets_by_tags?filter[tickets][gte]=0 + expect: + status: 200 + body: + equals: [] diff --git a/fixtures/api-contract-conformance/report/scenarios/filter-on-measure.yaml b/fixtures/api-contract-conformance/report/scenarios/filter-on-measure.yaml index 5427f7d10..21c1359ca 100644 --- a/fixtures/api-contract-conformance/report/scenarios/filter-on-measure.yaml +++ b/fixtures/api-contract-conformance/report/scenarios/filter-on-measure.yaml @@ -1,9 +1,9 @@ name: report-filter-on-measure description: > The FR-009 filter grammar applies to a report's derived fields, measures included. - r1 filters on a count, r2 on a null sum, r3 and r4 on a ratio. A ratio is a decimal: - its spelling is the port's own and is not asserted, so r3 and r4 assert only how many - rows match (the ratio is 2/5). + r1 filters on a count, r2 on a null sum, r3 and r4 on a ratio. A ratio is a decimal, so + it reaches the wire as the canonical string ("0.4" for 2/5) and its operand is compared + exactly. requests: - id: r1 method: GET @@ -29,7 +29,8 @@ requests: expect: status: 200 body: - length: 1 + equals: + - { invoices: 5, totalCents: 315500, paidShare: "0.4" } - id: r4 method: GET path: /api/invoice_totals?filter[paidShare][gt]=0.5 diff --git a/fixtures/api-contract-conformance/report/scenarios/list-totals.yaml b/fixtures/api-contract-conformance/report/scenarios/list-totals.yaml index a0f00419c..bd676c4d8 100644 --- a/fixtures/api-contract-conformance/report/scenarios/list-totals.yaml +++ b/fixtures/api-contract-conformance/report/scenarios/list-totals.yaml @@ -1,8 +1,8 @@ name: report-list-totals description: > - GET /api/invoice_totals — a report with no dimensions is exactly one row. It carries - a ratio, whose spelling is not asserted, so the row is counted rather than compared. - withCount=1 wraps the same list in the { rows, total } envelope. + GET /api/invoice_totals — a report with no dimensions is exactly one row. Its ratio is a + decimal, so it is the canonical string of ADR-0019 Amendment 1 (2/5 is "0.4" on every + engine). withCount=1 wraps the same list in the { rows, total } envelope. requests: - id: r1 method: GET @@ -10,7 +10,8 @@ requests: expect: status: 200 body: - length: 1 + equals: + - { invoices: 5, totalCents: 315500, paidShare: "0.4" } - id: r2 method: GET path: /api/invoice_totals?withCount=1 diff --git a/fixtures/api-contract-conformance/report/scenarios/sort-array-and-map-dimension.yaml b/fixtures/api-contract-conformance/report/scenarios/sort-array-and-map-dimension.yaml new file mode 100644 index 000000000..c2578ac8a --- /dev/null +++ b/fixtures/api-contract-conformance/report/scenarios/sort-array-and-map-dimension.yaml @@ -0,0 +1,31 @@ +name: report-sort-array-and-map-dimension +description: > + A dimension over an array field (`tags`, field.string isArray) or a map field (`attrs`, + field.map) is a derived field of the report, but it is neither sortable nor filterable: an + array or a map has no ordering and no scalar operator. Sorting on either is rejected with + the `invalid_sort` envelope naming the field, exactly as for a field the report does not + derive. The `tickets` count measure stays sortable. +requests: + - id: r1 + method: GET + path: /api/tickets_by_tags?sort=tags:asc + expect: + status: 400 + body: + error: "invalid_sort" + field: "tags" + - id: r2 + method: GET + path: /api/tickets_by_tags?sort=attrs:desc + expect: + status: 400 + body: + error: "invalid_sort" + field: "attrs" + - id: r3 + method: GET + path: /api/tickets_by_tags?sort=tickets:asc + expect: + status: 200 + body: + equals: [] diff --git a/fixtures/api-contract-conformance/report/schema.mysql.sql b/fixtures/api-contract-conformance/report/schema.mysql.sql index 29c2ca4f9..d4636f871 100644 --- a/fixtures/api-contract-conformance/report/schema.mysql.sql +++ b/fixtures/api-contract-conformance/report/schema.mysql.sql @@ -34,6 +34,12 @@ CREATE TABLE `sales` ( FOREIGN KEY (`productId`) REFERENCES `products` (`id`) ); +CREATE TABLE `tickets` ( + `id` BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY, + `tags` JSON, + `attrs` JSON +); + CREATE VIEW `v_invoice_status_totals` AS SELECT i.`status` AS `status`, @@ -117,3 +123,11 @@ CREATE VIEW `v_product_refunds` AS FROM `products` p LEFT OUTER JOIN `sales` s ON p.`id` = s.`productId` GROUP BY p.`id`, p.`name`; + +CREATE VIEW `v_tickets_by_tags` AS + SELECT + t.`tags` AS `tags`, + t.`attrs` AS `attrs`, + COUNT(t.`id`) AS `tickets` + FROM `tickets` t + GROUP BY t.`tags`, t.`attrs`; diff --git a/fixtures/api-contract-conformance/report/schema.postgres.sql b/fixtures/api-contract-conformance/report/schema.postgres.sql index 11ec91ffa..24e914329 100644 --- a/fixtures/api-contract-conformance/report/schema.postgres.sql +++ b/fixtures/api-contract-conformance/report/schema.postgres.sql @@ -36,6 +36,13 @@ CREATE TABLE "sales" ( CONSTRAINT "sales_pkey" PRIMARY KEY ("id") ); +CREATE TABLE "tickets" ( + "id" BIGINT GENERATED BY DEFAULT AS IDENTITY NOT NULL, + "tags" TEXT[], + "attrs" JSONB, + CONSTRAINT "tickets_pkey" PRIMARY KEY ("id") +); + ALTER TABLE "products" ADD CONSTRAINT "products_categoryId_fk" FOREIGN KEY ("categoryId") REFERENCES "categories" ("id"); ALTER TABLE "sales" ADD CONSTRAINT "sales_productId_fk" FOREIGN KEY ("productId") REFERENCES "products" ("id"); @@ -132,3 +139,12 @@ CREATE VIEW "v_product_refunds" AS LEFT OUTER JOIN "sales" s ON p."id" = s."productId" GROUP BY p."id", p."name"; COMMENT ON VIEW "v_product_refunds" IS 'metaobjects:v1:sha256:afc9cfc384c2db9e29ddc26e0cac58ee5d555f452fb41446ad99eeeab05051af'; + +CREATE VIEW "v_tickets_by_tags" AS + SELECT + t."tags" AS "tags", + t."attrs" AS "attrs", + COUNT(t."id") AS "tickets" + FROM "tickets" t + GROUP BY t."tags", t."attrs"; +COMMENT ON VIEW "v_tickets_by_tags" IS 'metaobjects:v1:sha256:38b4020ab33a2857d8cdf5eb860216342aa43fa8c4a914d330dbdd72646f8793'; diff --git a/fixtures/api-contract-conformance/report/schema.sqlite.sql b/fixtures/api-contract-conformance/report/schema.sqlite.sql index 18fa2265f..f4193b971 100644 --- a/fixtures/api-contract-conformance/report/schema.sqlite.sql +++ b/fixtures/api-contract-conformance/report/schema.sqlite.sql @@ -34,6 +34,12 @@ CREATE TABLE "sales" ( FOREIGN KEY ("productId") REFERENCES "products" ("id") ); +CREATE TABLE "tickets" ( + "id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, + "tags" TEXT, + "attrs" TEXT +); + CREATE VIEW "v_invoice_status_totals" AS SELECT i."status" AS "status", @@ -117,3 +123,11 @@ CREATE VIEW "v_product_refunds" AS FROM "products" p LEFT OUTER JOIN "sales" s ON p."id" = s."productId" GROUP BY p."id", p."name"; + +CREATE VIEW "v_tickets_by_tags" AS + SELECT + t."tags" AS "tags", + t."attrs" AS "attrs", + COUNT(t."id") AS "tickets" + FROM "tickets" t + GROUP BY t."tags", t."attrs"; diff --git a/fixtures/api-contract-conformance/tph/README.md b/fixtures/api-contract-conformance/tph/README.md index 27155947f..5c45a2231 100644 --- a/fixtures/api-contract-conformance/tph/README.md +++ b/fixtures/api-contract-conformance/tph/README.md @@ -71,9 +71,9 @@ Identical to the parent `api-contract-conformance` corpus `ids` / `length` / `row` (subset key-match) / `error` / `empty` / `fieldsNotEqual` cover TPH (the last pins the `@autoSet` leg: `autoCreatedAt` preserved vs `autoUpdatedAt` bumped). -Decimal subtype values are **not** asserted over the API wire (cross-port numeric -formatting differs); the runtime-layer decimal contract is pinned by -`persistence-conformance` instead. The `row` assertions pin the discriminator +A subtype's decimal is asserted over the wire as the canonical string (`copayAmount` `"12.50"` +is `"12.5"`, ADR-0019 Amendment 1), filters exactly on its subtype segment, and refuses a value +with more digits than its `@scale` with `400 validation`. The `row` assertions pin the discriminator value (`type`) and the integer/string subtype-only columns. ## Both lanes (the cross-port gate) diff --git a/fixtures/api-contract-conformance/tph/scenarios/tph-per-subtype-list-and-create.yaml b/fixtures/api-contract-conformance/tph/scenarios/tph-per-subtype-list-and-create.yaml index 33db4d85e..9a18b245c 100644 --- a/fixtures/api-contract-conformance/tph/scenarios/tph-per-subtype-list-and-create.yaml +++ b/fixtures/api-contract-conformance/tph/scenarios/tph-per-subtype-list-and-create.yaml @@ -4,7 +4,10 @@ description: > (`/auths/bridge`) carved out of the single table. GET filters to that subtype; POST creates a row of that subtype — the discriminator is injected by the route handler from the URL, NEVER taken from the body (the body omits - it). The created row joins the polymorphic collection. + it). The created row joins the polymorphic collection. A subtype's decimal + (`CopayAuth.copayAmount`, decimal 10,2) is the canonical string on the wire, filters + exactly on its own segment, and refuses more digits than its declared scale + (ADR-0019 Amendment 1). requests: - id: r1 method: GET @@ -44,3 +47,37 @@ requests: status: 200 body: length: 4 + - id: r6 + method: POST + path: /api/auths/copay + body: + reference: "REF-COPAY" + copayAmount: "8.10" + expect: + status: 201 + body: + row: { type: "Copay", reference: "REF-COPAY", copayAmount: "8.1" } + - id: r7 + method: GET + path: /api/auths/copay?filter[copayAmount][gt]=10&sort=id:asc + expect: + status: 200 + body: + ids: [2] + - id: r8 + method: GET + path: /api/auths/copay?filter[copayAmount][eq]=8.1 + expect: + status: 200 + body: + length: 1 + - id: r9 + method: POST + path: /api/auths/copay + body: + reference: "REF-COPAY-2" + copayAmount: "1.234" + expect: + status: 400 + body: + error: validation diff --git a/fixtures/api-contract-conformance/tph/scenarios/tph-polymorphic-list-and-get.yaml b/fixtures/api-contract-conformance/tph/scenarios/tph-polymorphic-list-and-get.yaml index a9d0db611..1e7453b65 100644 --- a/fixtures/api-contract-conformance/tph/scenarios/tph-polymorphic-list-and-get.yaml +++ b/fixtures/api-contract-conformance/tph/scenarios/tph-polymorphic-list-and-get.yaml @@ -25,7 +25,7 @@ requests: expect: status: 200 body: - row: { id: 2, type: "Copay", reference: "REF-2" } + row: { id: 2, type: "Copay", reference: "REF-2", copayAmount: "12.5" } - id: r4 method: GET path: /api/auths/3 diff --git a/fixtures/conformance/attr-int-integral-float/expected.json b/fixtures/conformance/attr-int-integral-float/expected.json new file mode 100644 index 000000000..27878c177 --- /dev/null +++ b/fixtures/conformance/attr-int-integral-float/expected.json @@ -0,0 +1,53 @@ +{ + "metadata.root": { + "package": "acme::shop", + "children": [ + { + "object.entity": { + "name": "Item", + "children": [ + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "sku", + "@maxLength": 10 + } + }, + { + "field.decimal": { + "name": "price", + "@precision": 10, + "@scale": 2 + } + }, + { + "field.int": { + "name": "qty", + "children": [ + { + "validator.numeric": { + "@max": 100, + "@min": 0 + } + } + ] + } + }, + { + "identity.primary": { + "name": "pk", + "@fields": [ + "id" + ] + } + } + ] + } + } + ] + } +} diff --git a/fixtures/conformance/attr-int-integral-float/input/meta.shop.json b/fixtures/conformance/attr-int-integral-float/input/meta.shop.json new file mode 100644 index 000000000..cba9c54b2 --- /dev/null +++ b/fixtures/conformance/attr-int-integral-float/input/meta.shop.json @@ -0,0 +1,54 @@ +{ + "metadata.root": { + "package": "acme::shop", + "children": [ + { + "object.entity": { + "name": "Item", + "children": [ + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "sku", + "@maxLength": 10.0 + } + }, + { + "field.decimal": { + "name": "price", + "@precision": 10.0, + "@scale": 2.0, + "children": [] + } + }, + { + "field.int": { + "name": "qty", + "children": [ + { + "validator.numeric": { + "@min": 0.0, + "@max": 100.0 + } + } + ] + } + }, + { + "identity.primary": { + "name": "pk", + "@fields": [ + "id" + ] + } + } + ] + } + } + ] + } +} diff --git a/fixtures/conformance/attr-int-integral-float/providers.json b/fixtures/conformance/attr-int-integral-float/providers.json new file mode 100644 index 000000000..8ec88fc09 --- /dev/null +++ b/fixtures/conformance/attr-int-integral-float/providers.json @@ -0,0 +1 @@ +["metaobjects-core-types"] diff --git a/fixtures/conformance/error-attr-int-fractional/expected-errors.json b/fixtures/conformance/error-attr-int-fractional/expected-errors.json new file mode 100644 index 000000000..d4f33e812 --- /dev/null +++ b/fixtures/conformance/error-attr-int-fractional/expected-errors.json @@ -0,0 +1,15 @@ +{ + "errors": [ + { + "code": "ERR_BAD_ATTR_VALUE", + "source": { + "format": "json", + "files": [ + "meta.shop.json" + ], + "jsonPath": "$['metadata.root'].children[0]['object.entity'].children[1]['field.string']" + } + } + ], + "warnings": [] +} diff --git a/fixtures/conformance/error-attr-int-fractional/input/meta.shop.json b/fixtures/conformance/error-attr-int-fractional/input/meta.shop.json new file mode 100644 index 000000000..504a679c3 --- /dev/null +++ b/fixtures/conformance/error-attr-int-fractional/input/meta.shop.json @@ -0,0 +1,33 @@ +{ + "metadata.root": { + "package": "acme::shop", + "children": [ + { + "object.entity": { + "name": "Item", + "children": [ + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "sku", + "@maxLength": 10.5 + } + }, + { + "identity.primary": { + "name": "pk", + "@fields": [ + "id" + ] + } + } + ] + } + } + ] + } +} diff --git a/fixtures/conformance/error-attr-int-fractional/providers.json b/fixtures/conformance/error-attr-int-fractional/providers.json new file mode 100644 index 000000000..8ec88fc09 --- /dev/null +++ b/fixtures/conformance/error-attr-int-fractional/providers.json @@ -0,0 +1 @@ +["metaobjects-core-types"] diff --git a/fixtures/conformance/error-dimension-over-object/expected-errors.json b/fixtures/conformance/error-dimension-over-object/expected-errors.json new file mode 100644 index 000000000..b19fddbba --- /dev/null +++ b/fixtures/conformance/error-dimension-over-object/expected-errors.json @@ -0,0 +1,15 @@ +{ + "errors": [ + { + "code": "ERR_INVALID_DIMENSION", + "source": { + "format": "json", + "files": [ + "meta.events.json" + ], + "jsonPath": "$['metadata.root'].children[1]['object.entity'].children[7]['dimension.attribute']" + } + } + ], + "warnings": [] +} diff --git a/fixtures/conformance/error-dimension-over-object/input/meta.events.json b/fixtures/conformance/error-dimension-over-object/input/meta.events.json new file mode 100644 index 000000000..eaa4c934a --- /dev/null +++ b/fixtures/conformance/error-dimension-over-object/input/meta.events.json @@ -0,0 +1,75 @@ +{ + "metadata.root": { + "package": "acme", + "children": [ + { + "object.value": { + "name": "Tags", + "children": [ + { + "field.string": { + "name": "label" + } + } + ] + } + }, + { + "object.entity": { + "name": "Event", + "children": [ + { + "source.rdb": { + "@table": "events" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "kind" + } + }, + { + "field.object": { + "name": "payload", + "@objectRef": "acme::Tags", + "@storage": "jsonb" + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + }, + { + "dimension.attribute": { + "name": "kind", + "@of": "Event.kind" + } + }, + { + "measure.aggregate": { + "name": "events", + "@agg": "count", + "@of": "Event.id" + } + }, + { + "dimension.attribute": { + "name": "payload", + "@of": "Event.payload" + } + } + ] + } + } + ] + } +} diff --git a/fixtures/conformance/error-dimension-over-object/providers.json b/fixtures/conformance/error-dimension-over-object/providers.json new file mode 100644 index 000000000..f9bb435a3 --- /dev/null +++ b/fixtures/conformance/error-dimension-over-object/providers.json @@ -0,0 +1 @@ +["metaobjects-core-types","metaobjects-db"] diff --git a/fixtures/conformance/error-dimension-via-base-lacks-field/expected-errors.json b/fixtures/conformance/error-dimension-via-base-lacks-field/expected-errors.json new file mode 100644 index 000000000..5171eb531 --- /dev/null +++ b/fixtures/conformance/error-dimension-via-base-lacks-field/expected-errors.json @@ -0,0 +1,15 @@ +{ + "errors": [ + { + "code": "ERR_INVALID_DIMENSION", + "source": { + "format": "json", + "files": [ + "meta.shop.json" + ], + "jsonPath": "$['metadata.root'].children[2]['object.entity'].children[5]['dimension.attribute']" + } + } + ], + "warnings": [] +} diff --git a/fixtures/conformance/error-dimension-via-base-lacks-field/input/meta.shop.json b/fixtures/conformance/error-dimension-via-base-lacks-field/input/meta.shop.json new file mode 100644 index 000000000..49c414b0c --- /dev/null +++ b/fixtures/conformance/error-dimension-via-base-lacks-field/input/meta.shop.json @@ -0,0 +1,39 @@ +{ + "metadata.root": { + "package": "acme::shop", + "children": [ + { + "object.entity": { + "name": "Item", + "children": [ + { "source.rdb": { "@table": "items" } }, + { "field.long": { "name": "id" } }, + { "identity.primary": { "name": "id", "@fields": ["id"] } } + ] + } + }, + { + "object.entity": { + "name": "Special", + "extends": "Item", + "children": [ + { "field.string": { "name": "tag" } } + ] + } + }, + { + "object.entity": { + "name": "Order", + "children": [ + { "source.rdb": { "@table": "orders" } }, + { "field.long": { "name": "id" } }, + { "field.long": { "name": "itemId" } }, + { "identity.primary": { "name": "id", "@fields": ["id"] } }, + { "relationship.association": { "name": "item", "@objectRef": "Special", "@cardinality": "one" } }, + { "dimension.attribute": { "name": "itemTag", "@of": "Item.tag", "@via": "Order.item" } } + ] + } + } + ] + } +} diff --git a/fixtures/conformance/error-dimension-via-base-lacks-field/providers.json b/fixtures/conformance/error-dimension-via-base-lacks-field/providers.json new file mode 100644 index 000000000..f9bb435a3 --- /dev/null +++ b/fixtures/conformance/error-dimension-via-base-lacks-field/providers.json @@ -0,0 +1 @@ +["metaobjects-core-types","metaobjects-db"] diff --git a/fixtures/conformance/error-extends-bare-cross-package/expected-errors.json b/fixtures/conformance/error-extends-bare-cross-package/expected-errors.json index e255bdc51..1ec1003b6 100644 --- a/fixtures/conformance/error-extends-bare-cross-package/expected-errors.json +++ b/fixtures/conformance/error-extends-bare-cross-package/expected-errors.json @@ -2,6 +2,10 @@ "errors": [ { "code": "ERR_UNRESOLVED_SUPER", + "messageIncludes": [ + "other::Base", + "extends: \"other::Base\"" + ], "source": { "format": "resolved", "files": [ diff --git a/fixtures/conformance/error-measure-default-on-count-integral-float/expected-errors.json b/fixtures/conformance/error-measure-default-on-count-integral-float/expected-errors.json new file mode 100644 index 000000000..57f749961 --- /dev/null +++ b/fixtures/conformance/error-measure-default-on-count-integral-float/expected-errors.json @@ -0,0 +1,15 @@ +{ + "errors": [ + { + "code": "ERR_INVALID_MEASURE", + "source": { + "format": "json", + "files": [ + "meta.shop.json" + ], + "jsonPath": "$['metadata.root'].children[2]['object.entity'].children[17]['measure.aggregate']" + } + } + ], + "warnings": [] +} diff --git a/fixtures/conformance/error-measure-default-on-count-integral-float/input/meta.shop.json b/fixtures/conformance/error-measure-default-on-count-integral-float/input/meta.shop.json new file mode 100644 index 000000000..c678b41ee --- /dev/null +++ b/fixtures/conformance/error-measure-default-on-count-integral-float/input/meta.shop.json @@ -0,0 +1,321 @@ +{ + "metadata.root": { + "package": "acme::shop", + "children": [ + { + "object.entity": { + "name": "Catalog", + "children": [ + { + "source.rdb": { + "@table": "catalogs" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "name", + "@required": true + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + } + ] + } + }, + { + "object.entity": { + "name": "Program", + "children": [ + { + "source.rdb": { + "@table": "programs" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "title", + "@required": true + } + }, + { + "field.timestamp": { + "name": "publishedAt" + } + }, + { + "field.long": { + "name": "catalogId" + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + }, + { + "identity.reference": { + "name": "fkCatalog", + "@references": "Catalog", + "@fields": [ + "catalogId" + ] + } + }, + { + "relationship.association": { + "name": "catalog", + "@objectRef": "Catalog", + "@cardinality": "one" + } + } + ] + } + }, + { + "object.entity": { + "name": "Purchase", + "children": [ + { + "source.rdb": { + "@table": "purchases" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.long": { + "name": "programId" + } + }, + { + "field.string": { + "name": "customerEmail" + } + }, + { + "field.currency": { + "name": "amountCents" + } + }, + { + "field.string": { + "name": "status" + } + }, + { + "field.timestamp": { + "name": "purchasedAt" + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + }, + { + "identity.reference": { + "name": "fkProgram", + "@references": "Program", + "@fields": [ + "programId" + ] + } + }, + { + "relationship.association": { + "name": "program", + "@objectRef": "Program", + "@cardinality": "one" + } + }, + { + "segment.filter": { + "name": "active", + "@filter": { + "status": "active" + } + } + }, + { + "dimension.attribute": { + "name": "programId", + "@of": "Program.id", + "@via": "Purchase.program" + } + }, + { + "dimension.attribute": { + "name": "programTitle", + "@of": "Program.title", + "@via": "Purchase.program" + } + }, + { + "dimension.time": { + "name": "publishedAt", + "@of": "Program.publishedAt", + "@via": "Purchase.program", + "@grains": [ + "month", + "year" + ] + } + }, + { + "dimension.time": { + "name": "purchasedAt", + "@of": "Purchase.purchasedAt", + "@grains": [ + "day", + "month" + ] + } + }, + { + "dimension.attribute": { + "name": "programByRef", + "@of": "Program.title", + "@via": "Purchase.fkProgram" + } + }, + { + "dimension.attribute": { + "name": "catalogName", + "@of": "Catalog.name", + "@via": "Purchase.program.catalog" + } + }, + { + "measure.aggregate": { + "name": "purchases", + "@agg": "count", + "@of": "Purchase.id", + "@default": 0.0 + } + }, + { + "measure.aggregate": { + "name": "revenue", + "@agg": "sum", + "@of": "Purchase.amountCents", + "@default": 0 + } + }, + { + "measure.aggregate": { + "name": "avgAmount", + "@agg": "avg", + "@of": "Purchase.amountCents", + "@default": 0 + } + }, + { + "measure.aggregate": { + "name": "smallest", + "@agg": "min", + "@of": "Purchase.amountCents", + "@default": -1 + } + }, + { + "measure.aggregate": { + "name": "lastPurchaseAt", + "@agg": "max", + "@of": "Purchase.purchasedAt" + } + }, + { + "measure.ratio": { + "name": "revenuePerPurchase", + "@numerator": "revenue", + "@denominator": "purchases", + "@default": 0 + } + } + ] + } + }, + { + "object.report": { + "name": "ProgramSales", + "@from": "Purchase", + "@spine": "Purchase.program", + "@dimensions": [ + "programId", + "programTitle", + "publishedAt:month", + "catalogName" + ], + "@measures": [ + "purchases", + "revenue", + "avgAmount", + "smallest", + "lastPurchaseAt", + "revenuePerPurchase" + ], + "@segment": "active", + "children": [ + { + "source.rdb": { + "@kind": "view", + "@view": "v_program_sales" + } + } + ] + } + }, + { + "object.report": { + "name": "CatalogSales", + "@from": "Purchase", + "@spine": "Purchase.program.catalog", + "@dimensions": [ + "catalogName" + ], + "@measures": [ + "purchases", + "revenue" + ] + } + }, + { + "object.report": { + "name": "SalesByDay", + "@from": "Purchase", + "@dimensions": [ + "purchasedAt:day" + ], + "@measures": [ + "revenue" + ] + } + } + ] + } +} diff --git a/fixtures/conformance/error-measure-default-on-count-integral-float/providers.json b/fixtures/conformance/error-measure-default-on-count-integral-float/providers.json new file mode 100644 index 000000000..f9bb435a3 --- /dev/null +++ b/fixtures/conformance/error-measure-default-on-count-integral-float/providers.json @@ -0,0 +1 @@ +["metaobjects-core-types","metaobjects-db"] diff --git a/fixtures/conformance/error-measure-default-on-count-non-numeric/expected-errors.json b/fixtures/conformance/error-measure-default-on-count-non-numeric/expected-errors.json new file mode 100644 index 000000000..91e6c9639 --- /dev/null +++ b/fixtures/conformance/error-measure-default-on-count-non-numeric/expected-errors.json @@ -0,0 +1,15 @@ +{ + "errors": [ + { + "code": "ERR_BAD_ATTR_VALUE", + "source": { + "format": "json", + "files": [ + "meta.shop.json" + ], + "jsonPath": "$['metadata.root'].children[2]['object.entity'].children[17]['measure.aggregate']" + } + } + ], + "warnings": [] +} diff --git a/fixtures/conformance/error-measure-default-on-count-non-numeric/input/meta.shop.json b/fixtures/conformance/error-measure-default-on-count-non-numeric/input/meta.shop.json new file mode 100644 index 000000000..1d179b0e3 --- /dev/null +++ b/fixtures/conformance/error-measure-default-on-count-non-numeric/input/meta.shop.json @@ -0,0 +1,321 @@ +{ + "metadata.root": { + "package": "acme::shop", + "children": [ + { + "object.entity": { + "name": "Catalog", + "children": [ + { + "source.rdb": { + "@table": "catalogs" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "name", + "@required": true + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + } + ] + } + }, + { + "object.entity": { + "name": "Program", + "children": [ + { + "source.rdb": { + "@table": "programs" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "title", + "@required": true + } + }, + { + "field.timestamp": { + "name": "publishedAt" + } + }, + { + "field.long": { + "name": "catalogId" + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + }, + { + "identity.reference": { + "name": "fkCatalog", + "@references": "Catalog", + "@fields": [ + "catalogId" + ] + } + }, + { + "relationship.association": { + "name": "catalog", + "@objectRef": "Catalog", + "@cardinality": "one" + } + } + ] + } + }, + { + "object.entity": { + "name": "Purchase", + "children": [ + { + "source.rdb": { + "@table": "purchases" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.long": { + "name": "programId" + } + }, + { + "field.string": { + "name": "customerEmail" + } + }, + { + "field.currency": { + "name": "amountCents" + } + }, + { + "field.string": { + "name": "status" + } + }, + { + "field.timestamp": { + "name": "purchasedAt" + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + }, + { + "identity.reference": { + "name": "fkProgram", + "@references": "Program", + "@fields": [ + "programId" + ] + } + }, + { + "relationship.association": { + "name": "program", + "@objectRef": "Program", + "@cardinality": "one" + } + }, + { + "segment.filter": { + "name": "active", + "@filter": { + "status": "active" + } + } + }, + { + "dimension.attribute": { + "name": "programId", + "@of": "Program.id", + "@via": "Purchase.program" + } + }, + { + "dimension.attribute": { + "name": "programTitle", + "@of": "Program.title", + "@via": "Purchase.program" + } + }, + { + "dimension.time": { + "name": "publishedAt", + "@of": "Program.publishedAt", + "@via": "Purchase.program", + "@grains": [ + "month", + "year" + ] + } + }, + { + "dimension.time": { + "name": "purchasedAt", + "@of": "Purchase.purchasedAt", + "@grains": [ + "day", + "month" + ] + } + }, + { + "dimension.attribute": { + "name": "programByRef", + "@of": "Program.title", + "@via": "Purchase.fkProgram" + } + }, + { + "dimension.attribute": { + "name": "catalogName", + "@of": "Catalog.name", + "@via": "Purchase.program.catalog" + } + }, + { + "measure.aggregate": { + "name": "purchases", + "@agg": "count", + "@of": "Purchase.id", + "@default": "zero" + } + }, + { + "measure.aggregate": { + "name": "revenue", + "@agg": "sum", + "@of": "Purchase.amountCents", + "@default": 0 + } + }, + { + "measure.aggregate": { + "name": "avgAmount", + "@agg": "avg", + "@of": "Purchase.amountCents", + "@default": 0 + } + }, + { + "measure.aggregate": { + "name": "smallest", + "@agg": "min", + "@of": "Purchase.amountCents", + "@default": -1 + } + }, + { + "measure.aggregate": { + "name": "lastPurchaseAt", + "@agg": "max", + "@of": "Purchase.purchasedAt" + } + }, + { + "measure.ratio": { + "name": "revenuePerPurchase", + "@numerator": "revenue", + "@denominator": "purchases", + "@default": 0 + } + } + ] + } + }, + { + "object.report": { + "name": "ProgramSales", + "@from": "Purchase", + "@spine": "Purchase.program", + "@dimensions": [ + "programId", + "programTitle", + "publishedAt:month", + "catalogName" + ], + "@measures": [ + "purchases", + "revenue", + "avgAmount", + "smallest", + "lastPurchaseAt", + "revenuePerPurchase" + ], + "@segment": "active", + "children": [ + { + "source.rdb": { + "@kind": "view", + "@view": "v_program_sales" + } + } + ] + } + }, + { + "object.report": { + "name": "CatalogSales", + "@from": "Purchase", + "@spine": "Purchase.program.catalog", + "@dimensions": [ + "catalogName" + ], + "@measures": [ + "purchases", + "revenue" + ] + } + }, + { + "object.report": { + "name": "SalesByDay", + "@from": "Purchase", + "@dimensions": [ + "purchasedAt:day" + ], + "@measures": [ + "revenue" + ] + } + } + ] + } +} diff --git a/fixtures/conformance/error-measure-default-on-count-non-numeric/providers.json b/fixtures/conformance/error-measure-default-on-count-non-numeric/providers.json new file mode 100644 index 000000000..f9bb435a3 --- /dev/null +++ b/fixtures/conformance/error-measure-default-on-count-non-numeric/providers.json @@ -0,0 +1 @@ +["metaobjects-core-types","metaobjects-db"] diff --git a/fixtures/conformance/error-measure-over-object/expected-errors.json b/fixtures/conformance/error-measure-over-object/expected-errors.json new file mode 100644 index 000000000..ebe5e1db1 --- /dev/null +++ b/fixtures/conformance/error-measure-over-object/expected-errors.json @@ -0,0 +1,15 @@ +{ + "errors": [ + { + "code": "ERR_INVALID_MEASURE", + "source": { + "format": "json", + "files": [ + "meta.events.json" + ], + "jsonPath": "$['metadata.root'].children[1]['object.entity'].children[7]['measure.aggregate']" + } + } + ], + "warnings": [] +} diff --git a/fixtures/conformance/error-measure-over-object/input/meta.events.json b/fixtures/conformance/error-measure-over-object/input/meta.events.json new file mode 100644 index 000000000..24cd84de5 --- /dev/null +++ b/fixtures/conformance/error-measure-over-object/input/meta.events.json @@ -0,0 +1,76 @@ +{ + "metadata.root": { + "package": "acme", + "children": [ + { + "object.value": { + "name": "Tags", + "children": [ + { + "field.string": { + "name": "label" + } + } + ] + } + }, + { + "object.entity": { + "name": "Event", + "children": [ + { + "source.rdb": { + "@table": "events" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "kind" + } + }, + { + "field.object": { + "name": "payload", + "@objectRef": "acme::Tags", + "@storage": "jsonb" + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + }, + { + "dimension.attribute": { + "name": "kind", + "@of": "Event.kind" + } + }, + { + "measure.aggregate": { + "name": "events", + "@agg": "count", + "@of": "Event.id" + } + }, + { + "measure.aggregate": { + "name": "payloads", + "@agg": "count", + "@of": "Event.payload" + } + } + ] + } + } + ] + } +} diff --git a/fixtures/conformance/error-measure-over-object/providers.json b/fixtures/conformance/error-measure-over-object/providers.json new file mode 100644 index 000000000..f9bb435a3 --- /dev/null +++ b/fixtures/conformance/error-measure-over-object/providers.json @@ -0,0 +1 @@ +["metaobjects-core-types","metaobjects-db"] diff --git a/fixtures/conformance/error-report-abstract/expected-errors.json b/fixtures/conformance/error-report-abstract/expected-errors.json new file mode 100644 index 000000000..34c7b0213 --- /dev/null +++ b/fixtures/conformance/error-report-abstract/expected-errors.json @@ -0,0 +1,15 @@ +{ + "errors": [ + { + "code": "ERR_INVALID_REPORT", + "source": { + "format": "json", + "files": [ + "meta.events.json" + ], + "jsonPath": "$['metadata.root'].children[2]['object.report']" + } + } + ], + "warnings": [] +} diff --git a/fixtures/conformance/error-report-abstract/input/meta.events.json b/fixtures/conformance/error-report-abstract/input/meta.events.json new file mode 100644 index 000000000..a9d0bf079 --- /dev/null +++ b/fixtures/conformance/error-report-abstract/input/meta.events.json @@ -0,0 +1,90 @@ +{ + "metadata.root": { + "package": "acme", + "children": [ + { + "object.value": { + "name": "Tags", + "children": [ + { + "field.string": { + "name": "label" + } + } + ] + } + }, + { + "object.entity": { + "name": "Event", + "children": [ + { + "source.rdb": { + "@table": "events" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "kind" + } + }, + { + "field.object": { + "name": "payload", + "@objectRef": "acme::Tags", + "@storage": "jsonb" + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + }, + { + "dimension.attribute": { + "name": "kind", + "@of": "Event.kind" + } + }, + { + "measure.aggregate": { + "name": "events", + "@agg": "count", + "@of": "Event.id" + } + } + ] + } + }, + { + "object.report": { + "name": "EventCounts", + "@from": "Event", + "@dimensions": [ + "kind" + ], + "@measures": [ + "events" + ], + "abstract": true, + "children": [ + { + "source.rdb": { + "@kind": "view", + "@view": "v_event_counts" + } + } + ] + } + } + ] + } +} diff --git a/fixtures/conformance/error-report-abstract/providers.json b/fixtures/conformance/error-report-abstract/providers.json new file mode 100644 index 000000000..f9bb435a3 --- /dev/null +++ b/fixtures/conformance/error-report-abstract/providers.json @@ -0,0 +1 @@ +["metaobjects-core-types","metaobjects-db"] diff --git a/fixtures/conformance/error-report-source-kind-materialized-view/expected-errors.json b/fixtures/conformance/error-report-source-kind-materialized-view/expected-errors.json new file mode 100644 index 000000000..5bbcac130 --- /dev/null +++ b/fixtures/conformance/error-report-source-kind-materialized-view/expected-errors.json @@ -0,0 +1,15 @@ +{ + "errors": [ + { + "code": "ERR_INVALID_REPORT", + "source": { + "format": "json", + "files": [ + "meta.events.json" + ], + "jsonPath": "$['metadata.root'].children[2]['object.report'].children[0]['source.rdb']" + } + } + ], + "warnings": [] +} diff --git a/fixtures/conformance/error-report-source-kind-materialized-view/input/meta.events.json b/fixtures/conformance/error-report-source-kind-materialized-view/input/meta.events.json new file mode 100644 index 000000000..40cebaa34 --- /dev/null +++ b/fixtures/conformance/error-report-source-kind-materialized-view/input/meta.events.json @@ -0,0 +1,89 @@ +{ + "metadata.root": { + "package": "acme", + "children": [ + { + "object.value": { + "name": "Tags", + "children": [ + { + "field.string": { + "name": "label" + } + } + ] + } + }, + { + "object.entity": { + "name": "Event", + "children": [ + { + "source.rdb": { + "@table": "events" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "kind" + } + }, + { + "field.object": { + "name": "payload", + "@objectRef": "acme::Tags", + "@storage": "jsonb" + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + }, + { + "dimension.attribute": { + "name": "kind", + "@of": "Event.kind" + } + }, + { + "measure.aggregate": { + "name": "events", + "@agg": "count", + "@of": "Event.id" + } + } + ] + } + }, + { + "object.report": { + "name": "EventCounts", + "@from": "Event", + "@dimensions": [ + "kind" + ], + "@measures": [ + "events" + ], + "children": [ + { + "source.rdb": { + "@kind": "materializedView", + "@materializedView": "mv_event_counts" + } + } + ] + } + } + ] + } +} diff --git a/fixtures/conformance/error-report-source-kind-materialized-view/providers.json b/fixtures/conformance/error-report-source-kind-materialized-view/providers.json new file mode 100644 index 000000000..f9bb435a3 --- /dev/null +++ b/fixtures/conformance/error-report-source-kind-materialized-view/providers.json @@ -0,0 +1 @@ +["metaobjects-core-types","metaobjects-db"] diff --git a/fixtures/conformance/error-report-source-kind-stored-proc/expected-errors.json b/fixtures/conformance/error-report-source-kind-stored-proc/expected-errors.json new file mode 100644 index 000000000..5bbcac130 --- /dev/null +++ b/fixtures/conformance/error-report-source-kind-stored-proc/expected-errors.json @@ -0,0 +1,15 @@ +{ + "errors": [ + { + "code": "ERR_INVALID_REPORT", + "source": { + "format": "json", + "files": [ + "meta.events.json" + ], + "jsonPath": "$['metadata.root'].children[2]['object.report'].children[0]['source.rdb']" + } + } + ], + "warnings": [] +} diff --git a/fixtures/conformance/error-report-source-kind-stored-proc/input/meta.events.json b/fixtures/conformance/error-report-source-kind-stored-proc/input/meta.events.json new file mode 100644 index 000000000..63fc06561 --- /dev/null +++ b/fixtures/conformance/error-report-source-kind-stored-proc/input/meta.events.json @@ -0,0 +1,89 @@ +{ + "metadata.root": { + "package": "acme", + "children": [ + { + "object.value": { + "name": "Tags", + "children": [ + { + "field.string": { + "name": "label" + } + } + ] + } + }, + { + "object.entity": { + "name": "Event", + "children": [ + { + "source.rdb": { + "@table": "events" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "kind" + } + }, + { + "field.object": { + "name": "payload", + "@objectRef": "acme::Tags", + "@storage": "jsonb" + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + }, + { + "dimension.attribute": { + "name": "kind", + "@of": "Event.kind" + } + }, + { + "measure.aggregate": { + "name": "events", + "@agg": "count", + "@of": "Event.id" + } + } + ] + } + }, + { + "object.report": { + "name": "EventCounts", + "@from": "Event", + "@dimensions": [ + "kind" + ], + "@measures": [ + "events" + ], + "children": [ + { + "source.rdb": { + "@kind": "storedProc", + "@proc": "sp_event_counts" + } + } + ] + } + } + ] + } +} diff --git a/fixtures/conformance/error-report-source-kind-stored-proc/providers.json b/fixtures/conformance/error-report-source-kind-stored-proc/providers.json new file mode 100644 index 000000000..f9bb435a3 --- /dev/null +++ b/fixtures/conformance/error-report-source-kind-stored-proc/providers.json @@ -0,0 +1 @@ +["metaobjects-core-types","metaobjects-db"] diff --git a/fixtures/conformance/error-report-source-kind-table-function/expected-errors.json b/fixtures/conformance/error-report-source-kind-table-function/expected-errors.json new file mode 100644 index 000000000..5bbcac130 --- /dev/null +++ b/fixtures/conformance/error-report-source-kind-table-function/expected-errors.json @@ -0,0 +1,15 @@ +{ + "errors": [ + { + "code": "ERR_INVALID_REPORT", + "source": { + "format": "json", + "files": [ + "meta.events.json" + ], + "jsonPath": "$['metadata.root'].children[2]['object.report'].children[0]['source.rdb']" + } + } + ], + "warnings": [] +} diff --git a/fixtures/conformance/error-report-source-kind-table-function/input/meta.events.json b/fixtures/conformance/error-report-source-kind-table-function/input/meta.events.json new file mode 100644 index 000000000..11903af1f --- /dev/null +++ b/fixtures/conformance/error-report-source-kind-table-function/input/meta.events.json @@ -0,0 +1,89 @@ +{ + "metadata.root": { + "package": "acme", + "children": [ + { + "object.value": { + "name": "Tags", + "children": [ + { + "field.string": { + "name": "label" + } + } + ] + } + }, + { + "object.entity": { + "name": "Event", + "children": [ + { + "source.rdb": { + "@table": "events" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "kind" + } + }, + { + "field.object": { + "name": "payload", + "@objectRef": "acme::Tags", + "@storage": "jsonb" + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + }, + { + "dimension.attribute": { + "name": "kind", + "@of": "Event.kind" + } + }, + { + "measure.aggregate": { + "name": "events", + "@agg": "count", + "@of": "Event.id" + } + } + ] + } + }, + { + "object.report": { + "name": "EventCounts", + "@from": "Event", + "@dimensions": [ + "kind" + ], + "@measures": [ + "events" + ], + "children": [ + { + "source.rdb": { + "@kind": "tableFunction", + "@function": "fn_event_counts" + } + } + ] + } + } + ] + } +} diff --git a/fixtures/conformance/error-report-source-kind-table-function/providers.json b/fixtures/conformance/error-report-source-kind-table-function/providers.json new file mode 100644 index 000000000..f9bb435a3 --- /dev/null +++ b/fixtures/conformance/error-report-source-kind-table-function/providers.json @@ -0,0 +1 @@ +["metaobjects-core-types","metaobjects-db"] diff --git a/fixtures/conformance/error-requirement-level-fractional/expected-errors.json b/fixtures/conformance/error-requirement-level-fractional/expected-errors.json new file mode 100644 index 000000000..3750d2214 --- /dev/null +++ b/fixtures/conformance/error-requirement-level-fractional/expected-errors.json @@ -0,0 +1,15 @@ +{ + "errors": [ + { + "code": "ERR_BAD_ATTR_VALUE", + "source": { + "format": "json", + "files": [ + "meta.requirements.json" + ], + "jsonPath": "$['metadata.root'].children[0]['requirement.functional']" + } + } + ], + "warnings": [] +} diff --git a/fixtures/conformance/error-requirement-level-fractional/input/meta.requirements.json b/fixtures/conformance/error-requirement-level-fractional/input/meta.requirements.json new file mode 100644 index 000000000..4243f649c --- /dev/null +++ b/fixtures/conformance/error-requirement-level-fractional/input/meta.requirements.json @@ -0,0 +1,16 @@ +{ + "metadata.root": { + "package": "acme::caps", + "children": [ + { + "requirement.functional": { + "name": "OrderRecord", + "@level": 4.5, + "@status": "live", + "@statement": "An order records what was bought", + "@counterexample": "An order row that cannot say what it holds" + } + } + ] + } +} diff --git a/fixtures/conformance/error-requirement-level-fractional/providers.json b/fixtures/conformance/error-requirement-level-fractional/providers.json new file mode 100644 index 000000000..8ec88fc09 --- /dev/null +++ b/fixtures/conformance/error-requirement-level-fractional/providers.json @@ -0,0 +1 @@ +["metaobjects-core-types"] diff --git a/fixtures/conformance/measure-default-integral-float/expected.json b/fixtures/conformance/measure-default-integral-float/expected.json new file mode 100644 index 000000000..d50324554 --- /dev/null +++ b/fixtures/conformance/measure-default-integral-float/expected.json @@ -0,0 +1,332 @@ +{ + "metadata.root": { + "package": "acme::shop", + "children": [ + { + "object.entity": { + "name": "Catalog", + "children": [ + { + "source.rdb": { + "@table": "catalogs" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "name", + "@required": true + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + } + ] + } + }, + { + "object.entity": { + "name": "Program", + "children": [ + { + "source.rdb": { + "@table": "programs" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "title", + "@required": true + } + }, + { + "field.timestamp": { + "name": "publishedAt" + } + }, + { + "field.long": { + "name": "catalogId" + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + }, + { + "identity.reference": { + "name": "fkCatalog", + "@fields": [ + "catalogId" + ], + "@references": "acme::shop::Catalog" + } + }, + { + "relationship.association": { + "name": "catalog", + "@cardinality": "one", + "@objectRef": "acme::shop::Catalog" + } + } + ] + } + }, + { + "object.entity": { + "name": "Purchase", + "children": [ + { + "source.rdb": { + "@table": "purchases" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.long": { + "name": "programId" + } + }, + { + "field.string": { + "name": "customerEmail" + } + }, + { + "field.currency": { + "name": "amountCents" + } + }, + { + "field.string": { + "name": "status" + } + }, + { + "field.timestamp": { + "name": "purchasedAt" + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + }, + { + "identity.reference": { + "name": "fkProgram", + "@fields": [ + "programId" + ], + "@references": "acme::shop::Program" + } + }, + { + "relationship.association": { + "name": "program", + "@cardinality": "one", + "@objectRef": "acme::shop::Program" + } + }, + { + "segment.filter": { + "name": "active", + "@filter": { + "status": { + "eq": "active" + } + } + } + }, + { + "dimension.attribute": { + "name": "programId", + "@of": "acme::shop::Program.id", + "@via": "acme::shop::Purchase.program" + } + }, + { + "dimension.attribute": { + "name": "programTitle", + "@of": "acme::shop::Program.title", + "@via": "acme::shop::Purchase.program" + } + }, + { + "dimension.time": { + "name": "publishedAt", + "@grains": [ + "month", + "year" + ], + "@of": "acme::shop::Program.publishedAt", + "@via": "acme::shop::Purchase.program" + } + }, + { + "dimension.time": { + "name": "purchasedAt", + "@grains": [ + "day", + "month" + ], + "@of": "acme::shop::Purchase.purchasedAt" + } + }, + { + "dimension.attribute": { + "name": "programByRef", + "@of": "acme::shop::Program.title", + "@via": "acme::shop::Purchase.fkProgram" + } + }, + { + "dimension.attribute": { + "name": "catalogName", + "@of": "acme::shop::Catalog.name", + "@via": "acme::shop::Purchase.program.catalog" + } + }, + { + "measure.aggregate": { + "name": "purchases", + "@agg": "count", + "@of": [ + "acme::shop::Purchase.id" + ] + } + }, + { + "measure.aggregate": { + "name": "revenue", + "@agg": "sum", + "@default": 0, + "@of": [ + "acme::shop::Purchase.amountCents" + ] + } + }, + { + "measure.aggregate": { + "name": "avgAmount", + "@agg": "avg", + "@default": 0, + "@of": [ + "acme::shop::Purchase.amountCents" + ] + } + }, + { + "measure.aggregate": { + "name": "smallest", + "@agg": "min", + "@default": -1, + "@of": [ + "acme::shop::Purchase.amountCents" + ] + } + }, + { + "measure.aggregate": { + "name": "lastPurchaseAt", + "@agg": "max", + "@of": [ + "acme::shop::Purchase.purchasedAt" + ] + } + }, + { + "measure.ratio": { + "name": "revenuePerPurchase", + "@default": 0, + "@denominator": "purchases", + "@numerator": "revenue" + } + } + ] + } + }, + { + "object.report": { + "name": "ProgramSales", + "@dimensions": [ + "programId", + "programTitle", + "publishedAt:month", + "catalogName" + ], + "@from": "acme::shop::Purchase", + "@measures": [ + "purchases", + "revenue", + "avgAmount", + "smallest", + "lastPurchaseAt", + "revenuePerPurchase" + ], + "@segment": "active", + "@spine": "acme::shop::Purchase.program", + "children": [ + { + "source.rdb": { + "@kind": "view", + "@view": "v_program_sales" + } + } + ] + } + }, + { + "object.report": { + "name": "CatalogSales", + "@dimensions": [ + "catalogName" + ], + "@from": "acme::shop::Purchase", + "@measures": [ + "purchases", + "revenue" + ], + "@spine": "acme::shop::Purchase.program.catalog" + } + }, + { + "object.report": { + "name": "SalesByDay", + "@dimensions": [ + "purchasedAt:day" + ], + "@from": "acme::shop::Purchase", + "@measures": [ + "revenue" + ] + } + } + ] + } +} diff --git a/fixtures/conformance/measure-default-integral-float/input/meta.shop.json b/fixtures/conformance/measure-default-integral-float/input/meta.shop.json new file mode 100644 index 000000000..7cfe39bf4 --- /dev/null +++ b/fixtures/conformance/measure-default-integral-float/input/meta.shop.json @@ -0,0 +1,320 @@ +{ + "metadata.root": { + "package": "acme::shop", + "children": [ + { + "object.entity": { + "name": "Catalog", + "children": [ + { + "source.rdb": { + "@table": "catalogs" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "name", + "@required": true + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + } + ] + } + }, + { + "object.entity": { + "name": "Program", + "children": [ + { + "source.rdb": { + "@table": "programs" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.string": { + "name": "title", + "@required": true + } + }, + { + "field.timestamp": { + "name": "publishedAt" + } + }, + { + "field.long": { + "name": "catalogId" + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + }, + { + "identity.reference": { + "name": "fkCatalog", + "@references": "Catalog", + "@fields": [ + "catalogId" + ] + } + }, + { + "relationship.association": { + "name": "catalog", + "@objectRef": "Catalog", + "@cardinality": "one" + } + } + ] + } + }, + { + "object.entity": { + "name": "Purchase", + "children": [ + { + "source.rdb": { + "@table": "purchases" + } + }, + { + "field.long": { + "name": "id" + } + }, + { + "field.long": { + "name": "programId" + } + }, + { + "field.string": { + "name": "customerEmail" + } + }, + { + "field.currency": { + "name": "amountCents" + } + }, + { + "field.string": { + "name": "status" + } + }, + { + "field.timestamp": { + "name": "purchasedAt" + } + }, + { + "identity.primary": { + "name": "id", + "@fields": [ + "id" + ] + } + }, + { + "identity.reference": { + "name": "fkProgram", + "@references": "Program", + "@fields": [ + "programId" + ] + } + }, + { + "relationship.association": { + "name": "program", + "@objectRef": "Program", + "@cardinality": "one" + } + }, + { + "segment.filter": { + "name": "active", + "@filter": { + "status": "active" + } + } + }, + { + "dimension.attribute": { + "name": "programId", + "@of": "Program.id", + "@via": "Purchase.program" + } + }, + { + "dimension.attribute": { + "name": "programTitle", + "@of": "Program.title", + "@via": "Purchase.program" + } + }, + { + "dimension.time": { + "name": "publishedAt", + "@of": "Program.publishedAt", + "@via": "Purchase.program", + "@grains": [ + "month", + "year" + ] + } + }, + { + "dimension.time": { + "name": "purchasedAt", + "@of": "Purchase.purchasedAt", + "@grains": [ + "day", + "month" + ] + } + }, + { + "dimension.attribute": { + "name": "programByRef", + "@of": "Program.title", + "@via": "Purchase.fkProgram" + } + }, + { + "dimension.attribute": { + "name": "catalogName", + "@of": "Catalog.name", + "@via": "Purchase.program.catalog" + } + }, + { + "measure.aggregate": { + "name": "purchases", + "@agg": "count", + "@of": "Purchase.id" + } + }, + { + "measure.aggregate": { + "name": "revenue", + "@agg": "sum", + "@of": "Purchase.amountCents", + "@default": 0.0 + } + }, + { + "measure.aggregate": { + "name": "avgAmount", + "@agg": "avg", + "@of": "Purchase.amountCents", + "@default": 0 + } + }, + { + "measure.aggregate": { + "name": "smallest", + "@agg": "min", + "@of": "Purchase.amountCents", + "@default": -1.0 + } + }, + { + "measure.aggregate": { + "name": "lastPurchaseAt", + "@agg": "max", + "@of": "Purchase.purchasedAt" + } + }, + { + "measure.ratio": { + "name": "revenuePerPurchase", + "@numerator": "revenue", + "@denominator": "purchases", + "@default": 0 + } + } + ] + } + }, + { + "object.report": { + "name": "ProgramSales", + "@from": "Purchase", + "@spine": "Purchase.program", + "@dimensions": [ + "programId", + "programTitle", + "publishedAt:month", + "catalogName" + ], + "@measures": [ + "purchases", + "revenue", + "avgAmount", + "smallest", + "lastPurchaseAt", + "revenuePerPurchase" + ], + "@segment": "active", + "children": [ + { + "source.rdb": { + "@kind": "view", + "@view": "v_program_sales" + } + } + ] + } + }, + { + "object.report": { + "name": "CatalogSales", + "@from": "Purchase", + "@spine": "Purchase.program.catalog", + "@dimensions": [ + "catalogName" + ], + "@measures": [ + "purchases", + "revenue" + ] + } + }, + { + "object.report": { + "name": "SalesByDay", + "@from": "Purchase", + "@dimensions": [ + "purchasedAt:day" + ], + "@measures": [ + "revenue" + ] + } + } + ] + } +} diff --git a/fixtures/conformance/measure-default-integral-float/providers.json b/fixtures/conformance/measure-default-integral-float/providers.json new file mode 100644 index 000000000..f9bb435a3 --- /dev/null +++ b/fixtures/conformance/measure-default-integral-float/providers.json @@ -0,0 +1 @@ +["metaobjects-core-types","metaobjects-db"] diff --git a/fixtures/conformance/overlay-explicit-package-not-applied-to-target/expected.json b/fixtures/conformance/overlay-explicit-package-not-applied-to-target/expected.json new file mode 100644 index 000000000..f30b863fd --- /dev/null +++ b/fixtures/conformance/overlay-explicit-package-not-applied-to-target/expected.json @@ -0,0 +1,50 @@ +{ + "metadata.root": { + "package": "acme::shop", + "children": [ + { + "object.entity": { + "name": "Order", + "children": [ + { + "field.long": { + "name": "id" + } + }, + { + "identity.primary": { + "name": "pk", + "@fields": [ + "id" + ] + } + } + ] + } + }, + { + "requirement.functional": { + "name": "R", + "@counterexample": "c", + "@level": 3, + "@statement": "s", + "@status": "live", + "children": [ + { + "requirement.functional": { + "name": "C", + "@counterexample": "c", + "@implementedBy": [ + "Order" + ], + "@level": 4, + "@statement": "s", + "@status": "live" + } + } + ] + } + } + ] + } +} diff --git a/fixtures/conformance/overlay-explicit-package-not-applied-to-target/input/meta.a-order.json b/fixtures/conformance/overlay-explicit-package-not-applied-to-target/input/meta.a-order.json new file mode 100644 index 000000000..411e42153 --- /dev/null +++ b/fixtures/conformance/overlay-explicit-package-not-applied-to-target/input/meta.a-order.json @@ -0,0 +1,27 @@ +{ + "metadata.root": { + "children": [ + { + "object.entity": { + "name": "Order", + "children": [ + { + "field.long": { + "name": "id" + } + }, + { + "identity.primary": { + "name": "pk", + "@fields": [ + "id" + ] + } + } + ] + } + } + ], + "package": "acme::shop" + } +} \ No newline at end of file diff --git a/fixtures/conformance/overlay-explicit-package-not-applied-to-target/input/meta.b-requirement.json b/fixtures/conformance/overlay-explicit-package-not-applied-to-target/input/meta.b-requirement.json new file mode 100644 index 000000000..7a8cd6988 --- /dev/null +++ b/fixtures/conformance/overlay-explicit-package-not-applied-to-target/input/meta.b-requirement.json @@ -0,0 +1,16 @@ +{ + "metadata.root": { + "package": "acme::shop", + "children": [ + { + "requirement.functional": { + "name": "R", + "@level": 3, + "@status": "live", + "@statement": "s", + "@counterexample": "c" + } + } + ] + } +} \ No newline at end of file diff --git a/fixtures/conformance/overlay-explicit-package-not-applied-to-target/input/meta.c-overlay.json b/fixtures/conformance/overlay-explicit-package-not-applied-to-target/input/meta.c-overlay.json new file mode 100644 index 000000000..a584cad21 --- /dev/null +++ b/fixtures/conformance/overlay-explicit-package-not-applied-to-target/input/meta.c-overlay.json @@ -0,0 +1,27 @@ +{ + "metadata.root": { + "children": [ + { + "requirement.functional": { + "name": "R", + "package": "acme::shop", + "overlay": true, + "children": [ + { + "requirement.functional": { + "name": "C", + "@level": 4, + "@status": "live", + "@statement": "s", + "@counterexample": "c", + "@implementedBy": [ + "Order" + ] + } + } + ] + } + } + ] + } +} \ No newline at end of file diff --git a/fixtures/conformance/overlay-explicit-package-not-applied-to-target/providers.json b/fixtures/conformance/overlay-explicit-package-not-applied-to-target/providers.json new file mode 100644 index 000000000..8ec88fc09 --- /dev/null +++ b/fixtures/conformance/overlay-explicit-package-not-applied-to-target/providers.json @@ -0,0 +1 @@ +["metaobjects-core-types"] diff --git a/fixtures/conformance/requirement-level-integral-float/expected.json b/fixtures/conformance/requirement-level-integral-float/expected.json new file mode 100644 index 000000000..842196db2 --- /dev/null +++ b/fixtures/conformance/requirement-level-integral-float/expected.json @@ -0,0 +1,16 @@ +{ + "metadata.root": { + "package": "acme::caps", + "children": [ + { + "requirement.functional": { + "name": "OrderRecord", + "@counterexample": "An order row that cannot say what it holds", + "@level": 4, + "@statement": "An order records what was bought", + "@status": "live" + } + } + ] + } +} diff --git a/fixtures/conformance/requirement-level-integral-float/input/meta.requirements.json b/fixtures/conformance/requirement-level-integral-float/input/meta.requirements.json new file mode 100644 index 000000000..42e19b373 --- /dev/null +++ b/fixtures/conformance/requirement-level-integral-float/input/meta.requirements.json @@ -0,0 +1,16 @@ +{ + "metadata.root": { + "package": "acme::caps", + "children": [ + { + "requirement.functional": { + "name": "OrderRecord", + "@level": 4.0, + "@status": "live", + "@statement": "An order records what was bought", + "@counterexample": "An order row that cannot say what it holds" + } + } + ] + } +} diff --git a/fixtures/conformance/requirement-level-integral-float/providers.json b/fixtures/conformance/requirement-level-integral-float/providers.json new file mode 100644 index 000000000..8ec88fc09 --- /dev/null +++ b/fixtures/conformance/requirement-level-integral-float/providers.json @@ -0,0 +1 @@ +["metaobjects-core-types"] diff --git a/fixtures/conformance/yaml-twin-exclusions.json b/fixtures/conformance/yaml-twin-exclusions.json index ed8349629..4ff2e3e18 100644 --- a/fixtures/conformance/yaml-twin-exclusions.json +++ b/fixtures/conformance/yaml-twin-exclusions.json @@ -3,6 +3,7 @@ "notes": "Gate 1 (YAML twin, ADR-0059): fixtures whose input is deliberately format-native, so reading the same text as YAML is expected to report a different error. Each entry names why. Every port also skips the provider-extension-* fixtures whose test-only providers it cannot compose.", "exclusions": { "error-attr-child-not-object": "Asserts a JSON-shaped body error; the YAML desugar reads a scalar body as { name: } (ADR-0006 rule 2).", + "error-measure-default-on-count-non-numeric": "Asserts a JSON value-type error on the quoted \"zero\"; the YAML coercion guard (ADR-0006 D2) reports ERR_YAML_COERCION under its own code.", "error-attr-wrong-type": "Asserts a JSON value-type error that the YAML coercion guard (ADR-0006 D2) reports under its own code.", "error-bare-key-no-concrete-default": "A bare type key is YAML sugar for the default subtype (ADR-0006 rule 1); in canonical JSON it is an error.", "error-bare-key-type-without-base": "A bare type key is YAML sugar for the default subtype (ADR-0006 rule 1); in canonical JSON it is an error.", diff --git a/fixtures/cube-model/README.md b/fixtures/cube-model/README.md index 9868680b7..05f8370e7 100644 --- a/fixtures/cube-model/README.md +++ b/fixtures/cube-model/README.md @@ -48,7 +48,7 @@ never copied from it. A golden blessed from the implementation proves nothing. | `dimension-via` | B: `@via`; E row 3 | `sql: '{Program.title}'` (a member, never a column); `title` added to `Program` as `public: false`; a declared dimension over the field (`status`) is reused, nothing added | | `dimension-via-join-target` | A row 2; E rows 4 and 5 | a multi-hop `@via` onto entities that are no cube makes join-target cubes (`public: false`), each joined on its own hop's holder; the dimension reads the far cube | | `dimension-time` | B: `dimension.time` + `@grains`; C | instant and naive timestamps as the column, a date cast to `TIMESTAMP`; `meta.grains` in declared order | -| `dimension-time-two-grains` | F (list form) | one time dimension at two grains in one served report: `time_dimensions` with one entry per grain | +| `dimension-time-two-grains` | F (one rollup per grain) | one time dimension at two grains in one served report: a Cube query names a time dimension once, so one single-granularity rollup per grain (`ProgramGrains_month`, `ProgramGrains_week`, coarsest first), never one rollup repeating the dimension | | `dimension-time-mysql` | C (MySQL arm) | under `"dialect": "mysql"`: a timestamp is the backtick-quoted column, and a date, as a time dimension or as an attribute, is `CAST(… AS DATETIME)` (MySQL's `CAST` has no `TIMESTAMP` target; Table C's `TIMESTAMP` is the Postgres spelling) | | `measure-count` | B; D | `type: count`, `sql` the `@of` column (the key, and a nullable column) | | `measure-count-distinct` | B; D | `type: count_distinct` over one column | @@ -65,7 +65,7 @@ never copied from it. A golden blessed from the implementation proves nothing. | `relative-date` | B: relative filter (Plan 2 Table E) | `(now() - INTERVAL 'P7D')` for an instant, `((now() AT TIME ZONE 'UTC') - INTERVAL 'P7D')` for a naive timestamp, `CAST(… AS DATE)` for a date, `+` for a future offset, and the same SQL in a measure's condition | | `report-rollup` | B: served report; F | a `rollup` named after the report: dimensions and measures in listed order, `segments` with `@segment` then the scope segment `programsByMonthScope`, the one-time-dimension form; a report with no dimensions is a measures-only rollup; rollups are written coarsest first, so `ProgramTotals` precedes `ProgramsByMonth` | | `report-relative-no-rollup` | B; F (Review Focus 3) | a relative `@filter` gives the scope segment and no rollup; a relative condition on a listed measure gives no rollup and no segment | -| `report-sourceless` | B: sourceless report; A | a sourceless, an abstract and a `materializedView` report add nothing to their `@from` cube | +| `report-sourceless` | B: sourceless report; A | a sourceless report adds nothing to its `@from` cube (an abstract report and a non-`view` source are refused at load, so they have no case here) | | `report-spine` | B: report `@spine`; the zero-rows / measure-defaults plan, Table D | a served `@spine` report is a Cube view `model/views/.yml`, no rollup and no scope segment: its first `join_path` is the spine cube (`Program`, a join-target cube here) with the members the listed dimensions read under the dimensions' names (`id` as `programKey`; `title` as `programTitle`, carrying the dimension's description, and the same `title` again as `programName`, a second dimension over the same field and path, which Cube accepts), its last `Program.Facts` with the listed measures, and the report's description is the view's. `Facts` is a standalone `public: false` cube over the `@from` table with the report's scope in its own `sql` (`ProgramLongWeeks`: its `@segment` then its `@filter`, ANDed; `ProgramRoster`: none), its key, and `@from`'s definitions of the listed measures and what they read (a defaulted measure's `Raw` included). The spine cube gets one `one_to_many` join onto each facts cube, on the spine hop's columns; `Week`, the ordinary fact cube, is what it would be without the reports | | `report-spine-multi-hop` | B: report `@spine`, two hops | `Session.fkWeek.fkProgram`: the spine cube `Program` joins a standalone `public: false` chain cube `ProgramSessions_fkWeek` over `weeks`, which joins `ProgramSessionsFacts`, each `one_to_many` on its hop's columns; the view's last `join_path` is `Program.ProgramSessions_fkWeek.ProgramSessionsFacts`. The ordinary `Week` and `Session` cubes gain nothing | | `entity-tph-subtype` | K; A TPH row | a TPH subtype's cube is `sql: 'SELECT * FROM "auths" WHERE "type" = ''Bridge'''`, not `sql_table`; the base and a subtype with no vocabulary get none | diff --git a/fixtures/cube-model/canonical/expected/model/cubes/Entry.yml b/fixtures/cube-model/canonical/expected/model/cubes/Entry.yml new file mode 100644 index 000000000..b4eba5fbf --- /dev/null +++ b/fixtures/cube-model/canonical/expected/model/cubes/Entry.yml @@ -0,0 +1,48 @@ +# @generated by @metaobjectsdev/codegen-ts — cube-model +cubes: + - name: Entry + sql_table: '"entries"' + dimensions: + - name: id + sql: '{CUBE}."id"' + type: number + primary_key: true + - name: bucket + sql: '{CUBE}."bucket"' + type: string + measures: + - name: entries + sql: '{CUBE}."id"' + type: count + - name: flaggedEntries + sql: '{CUBE}."id"' + type: count + filters: + - sql: '{CUBE}."flagged" = TRUE' + - name: total + sql: '{CUBE}."amount"' + type: sum + - name: totalOrZeroRaw + sql: '{CUBE}."amount"' + type: sum + public: false + - name: totalOrZero + sql: 'COALESCE({totalOrZeroRaw}, 0)' + type: number + - name: mean + sql: '{CUBE}."amount"' + type: avg + - name: top + sql: '{CUBE}."amount"' + type: max + - name: flaggedShare + sql: 'CAST({flaggedEntries} AS NUMERIC) / NULLIF({entries}, 0)' + type: number + - name: perFlagged + sql: 'CAST({entries} AS NUMERIC) / NULLIF({flaggedEntries}, 0)' + type: number + pre_aggregations: + - name: EntryTotals + type: rollup + measures: [CUBE.entries, CUBE.total, CUBE.totalOrZero, CUBE.mean, CUBE.top, CUBE.flaggedShare, CUBE.perFlagged] + dimensions: [CUBE.bucket] diff --git a/fixtures/cube-model/canonical/expected/model/cubes/Gadget.yml b/fixtures/cube-model/canonical/expected/model/cubes/Gadget.yml new file mode 100644 index 000000000..683a1c3e2 --- /dev/null +++ b/fixtures/cube-model/canonical/expected/model/cubes/Gadget.yml @@ -0,0 +1,14 @@ +# @generated by @metaobjectsdev/codegen-ts — cube-model +cubes: + - name: Gadget + sql_table: '"gadgets"' + public: false + dimensions: + - name: id + sql: '{CUBE}."id"' + type: number + primary_key: true + - name: label + sql: '{CUBE}."gadget_code"' + type: number + public: false diff --git a/fixtures/cube-model/canonical/expected/model/cubes/GadgetEvent.yml b/fixtures/cube-model/canonical/expected/model/cubes/GadgetEvent.yml new file mode 100644 index 000000000..28c0ca198 --- /dev/null +++ b/fixtures/cube-model/canonical/expected/model/cubes/GadgetEvent.yml @@ -0,0 +1,25 @@ +# @generated by @metaobjectsdev/codegen-ts — cube-model +cubes: + - name: GadgetEvent + sql_table: '"gadget_events"' + joins: + - name: Gadget + relationship: many_to_one + sql: '{CUBE}."gadgetId" = {Gadget}."id"' + dimensions: + - name: id + sql: '{CUBE}."id"' + type: number + primary_key: true + - name: gadgetLabel + sql: '{Gadget.label}' + type: number + measures: + - name: gadgetEvents + sql: '{CUBE}."id"' + type: count + pre_aggregations: + - name: GadgetEventsByLabel + type: rollup + measures: [CUBE.gadgetEvents] + dimensions: [CUBE.gadgetLabel] diff --git a/fixtures/cube-model/canonical/expected/model/cubes/Program.yml b/fixtures/cube-model/canonical/expected/model/cubes/Program.yml index 59d831073..a427a9c7f 100644 --- a/fixtures/cube-model/canonical/expected/model/cubes/Program.yml +++ b/fixtures/cube-model/canonical/expected/model/cubes/Program.yml @@ -85,25 +85,30 @@ cubes: measures: [CUBE.programs, CUBE.notDrafts, CUBE.listValue] dimensions: [CUBE.status] segments: [CUBE.introOrFree, CUBE.liveIntroProgramsScope] + - name: ProgramsOverTime_year + type: rollup + measures: [CUBE.programs, CUBE.avgPrice, CUBE.cheapest, CUBE.priciest] + time_dimension: CUBE.createdAt + granularity: year + - name: ProgramsOverTime_quarter + type: rollup + measures: [CUBE.programs, CUBE.avgPrice, CUBE.cheapest, CUBE.priciest] + time_dimension: CUBE.createdAt + granularity: quarter - name: ProgramsByWeek type: rollup measures: [CUBE.programs] segments: [CUBE.published] time_dimension: CUBE.createdAt granularity: week + - name: ProgramsOverTime_day + type: rollup + measures: [CUBE.programs, CUBE.avgPrice, CUBE.cheapest, CUBE.priciest] + time_dimension: CUBE.createdAt + granularity: day - name: ProgramsByMonth type: rollup measures: [CUBE.programs, CUBE.listValue] dimensions: [CUBE.status] time_dimension: CUBE.createdAt granularity: month - - name: ProgramsOverTime - type: rollup - measures: [CUBE.programs, CUBE.avgPrice, CUBE.cheapest, CUBE.priciest] - time_dimensions: - - dimension: CUBE.createdAt - granularity: day - - dimension: CUBE.createdAt - granularity: quarter - - dimension: CUBE.createdAt - granularity: year diff --git a/fixtures/cube-model/canonical/seed.mysql.sql b/fixtures/cube-model/canonical/seed.mysql.sql index 42dc922ec..8bf766218 100644 --- a/fixtures/cube-model/canonical/seed.mysql.sql +++ b/fixtures/cube-model/canonical/seed.mysql.sql @@ -67,3 +67,23 @@ INSERT INTO `all_types` ( ('00000000-0000-4000-8000-000000000005','r5',99,1,0.5,0.5,9,true,'2026-05-01','00:00:00', '2026-05-01 00:00:00','2026-05-01 00:00:00',9999,'HIGH',5, 'a1111111-1111-4111-8111-111111111111','urn:x:5','10.0.0.5','fe80::5'); +INSERT INTO `gadgets` (`id`,`gadget_code`) VALUES + (1, 10), + (2, 20), + (3, 30); +INSERT INTO `gadget_events` (`id`,`gadgetId`) VALUES + (1, 1), + (2, 1), + (3, 1), + (4, 2), + (5, 3), + (6, 3); +INSERT INTO `entries` (`id`,`bucket`,`amount`,`flagged`) VALUES + (1, 'A', 0.1, TRUE), + (2, 'A', 0.2, FALSE), + (3, 'B', 1.25, TRUE), + (4, 'B', 2.75, TRUE), + (5, 'C', NULL, FALSE), + (6, 'C', NULL, TRUE), + (7, 'D', 100.50, FALSE), + (8, 'D', 99.5, FALSE); diff --git a/fixtures/cube-model/canonical/seed.sql b/fixtures/cube-model/canonical/seed.sql index 211632e0a..12bd66842 100644 --- a/fixtures/cube-model/canonical/seed.sql +++ b/fixtures/cube-model/canonical/seed.sql @@ -75,3 +75,23 @@ INSERT INTO "all_types" ( ('00000000-0000-4000-8000-000000000005','r5',99,1,0.5,0.5,9,true,'2026-05-01','00:00:00', '2026-05-01T00:00:00','2026-05-01T00:00:00Z',9999,'HIGH',5, 'a1111111-1111-4111-8111-111111111111','urn:x:5','10.0.0.5','fe80::5'); +INSERT INTO "gadgets" ("id","gadget_code") VALUES + (1, 10), + (2, 20), + (3, 30); +INSERT INTO "gadget_events" ("id","gadgetId") VALUES + (1, 1), + (2, 1), + (3, 1), + (4, 2), + (5, 3), + (6, 3); +INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES + (1, 'A', 0.1, true), + (2, 'A', 0.2, false), + (3, 'B', 1.25, true), + (4, 'B', 2.75, true), + (5, 'C', NULL, false), + (6, 'C', NULL, true), + (7, 'D', 100.50, false), + (8, 'D', 99.5, false); diff --git a/fixtures/cube-model/dimension-time-two-grains/expected/model/cubes/Program.yml b/fixtures/cube-model/dimension-time-two-grains/expected/model/cubes/Program.yml index 980a1da6a..cb095ed66 100644 --- a/fixtures/cube-model/dimension-time-two-grains/expected/model/cubes/Program.yml +++ b/fixtures/cube-model/dimension-time-two-grains/expected/model/cubes/Program.yml @@ -17,11 +17,13 @@ cubes: sql: '{CUBE}."id"' type: count pre_aggregations: - - name: ProgramGrains + - name: ProgramGrains_month type: rollup measures: [CUBE.programs] - time_dimensions: - - dimension: CUBE.createdAt - granularity: week - - dimension: CUBE.createdAt - granularity: month + time_dimension: CUBE.createdAt + granularity: month + - name: ProgramGrains_week + type: rollup + measures: [CUBE.programs] + time_dimension: CUBE.createdAt + granularity: week diff --git a/fixtures/cube-model/report-sourceless/meta.json b/fixtures/cube-model/report-sourceless/meta.json index 6d4c5119f..9bd60deba 100644 --- a/fixtures/cube-model/report-sourceless/meta.json +++ b/fixtures/cube-model/report-sourceless/meta.json @@ -49,47 +49,6 @@ }, "children": [] } - }, - { - "object.report": { - "name": "AbstractTotals", - "abstract": true, - "@from": "Program", - "@measures": [ - "programs" - ], - "@filter": { - "title": "x" - }, - "children": [ - { - "source.rdb": { - "@kind": "view", - "@view": "v_abstracttotals" - } - } - ] - } - }, - { - "object.report": { - "name": "Materialized", - "@from": "Program", - "@measures": [ - "programs" - ], - "@filter": { - "title": "x" - }, - "children": [ - { - "source.rdb": { - "@kind": "materializedView", - "@materializedView": "v_materialized" - } - } - ] - } } ] } diff --git a/fixtures/decimal-wire/README.md b/fixtures/decimal-wire/README.md new file mode 100644 index 000000000..f8f4535c1 --- /dev/null +++ b/fixtures/decimal-wire/README.md @@ -0,0 +1,18 @@ +# Decimal wire vectors + +The shared vectors for the decimal wire rule, [ADR-0019 Amendment 1](../../spec/decisions/ADR-0019-runtime-return-type-contract.md#amendment-1-2026-10-21-the-decimal-wire-form). +Every port runs them in a unit test against its **production** wire helper (the code its +generated routes and DTOs call), not a copy written for the test. No database is involved: the +persistence corpus (`decimal-exact*` scenarios) and the api-contract `decimal/` sub-corpus gate +the read path and the HTTP bytes. + +| Section | What the test does | Passes when | +|---|---|---| +| `format` | builds the port's exact value from `in` (`new BigDecimal(in)`, `decimal.Parse(in, NumberStyles.Float, CultureInfo.InvariantCulture)`, `Decimal(in)`, the string in TypeScript) and formats it for the wire (W1) | the output equals `out` byte for byte | +| `parse` | parses the JSON text `json` as a request-body value (W2) | the result equals `value` exactly (`compareTo == 0`, `==`, `Decimal ==`, canonical string equality in TypeScript), `null` for `null`, or the parse is refused when `error` is `true` | +| `filterOperand` | parses the query-string text `text` as a decimal filter operand (W2) | as `parse` | +| `double15` | renders the IEEE double written as `double` at 15 significant digits, then canonicalizes it (W4) | the output equals `out`; run by every port with a SQLite or D1 read path | + +`1234567890123456789012345678` (28 digits) is the largest `format` value: it is inside C#'s +`decimal` range (W3). JSON-number inputs in `parse` have 15 significant digits or fewer, the +limit W2 guarantees for a number. diff --git a/fixtures/decimal-wire/vectors.json b/fixtures/decimal-wire/vectors.json new file mode 100644 index 000000000..3cd0b1e88 --- /dev/null +++ b/fixtures/decimal-wire/vectors.json @@ -0,0 +1,73 @@ +{ + "$comment": "The decimal wire rule (ADR-0019 Amendment 1, W1-W4). Each port's production wire helper must pass every vector; see README.md.", + "format": [ + { "in": "12.50", "out": "12.5" }, + { "in": "200.0", "out": "200" }, + { "in": "200", "out": "200" }, + { "in": "0E-18", "out": "0" }, + { "in": "-0", "out": "0" }, + { "in": "-0.000", "out": "0" }, + { "in": "0.0000000000", "out": "0" }, + { "in": "1E+3", "out": "1000" }, + { "in": "1E-10", "out": "0.0000000001" }, + { "in": "-0.0000000001", "out": "-0.0000000001" }, + { "in": "0.40000000000000000000", "out": "0.4" }, + { "in": "12345678901234567.8900000000", "out": "12345678901234567.89" }, + { "in": "6172839450617283.9500000000", "out": "6172839450617283.95" }, + { "in": "-12345678.9876543210", "out": "-12345678.987654321" }, + { "in": "1234567890123456789012345678", "out": "1234567890123456789012345678" } + ], + "parse": [ + { "json": "\"12.50\"", "value": "12.5" }, + { "json": "12.50", "value": "12.5" }, + { "json": "\"12345678901234567.89\"", "value": "12345678901234567.89" }, + { "json": "\"-12345678.987654321\"", "value": "-12345678.987654321" }, + { "json": "\"1e-7\"", "value": "0.0000001" }, + { "json": "\"1E+3\"", "value": "1000" }, + { "json": "\"0\"", "value": "0" }, + { "json": "\"-0.0\"", "value": "0" }, + { "json": "0.1", "value": "0.1" }, + { "json": "-42", "value": "-42" }, + { "json": "null", "value": null }, + { "json": "\"\"", "error": true }, + { "json": "\" 1.5\"", "error": true }, + { "json": "\"1.5 \"", "error": true }, + { "json": "\"1,5\"", "error": true }, + { "json": "\"abc\"", "error": true }, + { "json": "\"NaN\"", "error": true }, + { "json": "\"Infinity\"", "error": true }, + { "json": "\"-Infinity\"", "error": true }, + { "json": "\"0x10\"", "error": true }, + { "json": "\"+1\"", "error": true }, + { "json": "\"01\"", "error": true }, + { "json": "\"1.\"", "error": true }, + { "json": "\".5\"", "error": true }, + { "json": "true", "error": true }, + { "json": "[1]", "error": true }, + { "json": "{}", "error": true } + ], + "filterOperand": [ + { "text": "12345678901234567.89", "value": "12345678901234567.89" }, + { "text": "-0.0000000001", "value": "-0.0000000001" }, + { "text": "1e-7", "value": "0.0000001" }, + { "text": "0", "value": "0" }, + { "text": "abc", "error": true }, + { "text": "0x10", "error": true }, + { "text": "", "error": true }, + { "text": " 1", "error": true }, + { "text": "NaN", "error": true }, + { "text": "Infinity", "error": true } + ], + "double15": [ + { "double": "0.30000000000000004", "out": "0.3" }, + { "double": "1e-10", "out": "0.0000000001" }, + { "double": "-1e-10", "out": "-0.0000000001" }, + { "double": "0.3333333333333333", "out": "0.333333333333333" }, + { "double": "12.5", "out": "12.5" }, + { "double": "100.25", "out": "100.25" }, + { "double": "200", "out": "200" }, + { "double": "0", "out": "0" }, + { "double": "-0", "out": "0" }, + { "double": "123456789012345680", "out": "123456789012346000" } + ] +} diff --git a/fixtures/persistence-conformance/README.md b/fixtures/persistence-conformance/README.md index 30eeb6581..0da8f0c18 100644 --- a/fixtures/persistence-conformance/README.md +++ b/fixtures/persistence-conformance/README.md @@ -167,6 +167,45 @@ queries: - { programId: "1", weekCount: 3 } ``` +### Which engines run a scenario (`engines:`) + +A scenario file may carry a top-level `engines:` list drawn from `postgres`, `mysql`, `sqlite` +and `d1`. **When the key is present it is exact**: only the listed engines run the file, and a +runner never skips one by name. When it is absent the historical rule holds: Postgres runs every +scenario and the other engines run the `report-*` ones. An unknown name, or an empty list, is a +runner error. The key is how a scenario says what an engine can hold (a decimal of 19 significant +digits on SQLite, which keeps it as a double) once, in the file. + +```yaml +name: decimal-exact +engines: [postgres, mysql] +``` + +### Decimal scenarios (ADR-0019 Amendment 1) + +`field.decimal` is a **canonical JSON string** on every engine and every port: plain notation, no +trailing fraction zeros, zero as `"0"` (`^-?(0|[1-9][0-9]*)(\.[0-9]*[1-9])?$`). The seed table is +`Entry` (`id`, `bucket`, `amount` `decimal(28,10)`, `flagged`) and the served report is +`EntryTotals` (`bucket`, `entries`, `total`, `totalOrZero`, `mean`, `top`, `flaggedShare`, +`perFlagged`), over eleven rows chosen to break a binary float: `0.1` and `0.2`, +`12345678901234567.89` (19 significant digits), `-0.0000000001`, `100.50` stored with a scale, +`0.0000000000`, and rows with a null amount. + +| Scenario | Engines | What it pins | +|---|---|---| +| `decimal-exact` | postgres, mysql | eleven `op: roundtrip` inserts through the write codec; a list of canonical strings; `amount eq` and `gt` against `12345678901234567.89` and `12345678901234567.8` each answer only id 3; a trailing-zero and an exponent operand; `op: create` of `"12abc"` fails | +| `decimal-exact-15` | all four | the same through the write codec for the rows of at most 15 significant digits (buckets A, D, E, F) | +| `report-decimal-exact` | postgres, mysql | the `EntryTotals` view over the seed: a sum is `"12345678901234567.9"`, an empty sum is `null`, a `@default: 0` sum is `"0"`, a zero-denominator ratio is `null` | +| `report-decimal-exact-15` | all four | the A, D, E, F buckets of the same report, run on SQLite and D1 as well | + +A runner holds a decimal to **the type, not the value's shape**: before comparing it, every field +the model declares `decimal` (an entity field, or a report field from `report-shapes.json`) must be +`null` or a string in the canonical spelling, or the scenario fails with "decimal arrived as a +binary float". A runner must NOT canonicalize a decimal on the actual side before that check (the +earlier runner did, which hid a port that returned `100.50` or a double). The MySQL operand +comparison is the usual trap: `DECIMAL = 'string'` compares as doubles, so the operand is cast to a +wide `DECIMAL` first. + ## Query DSL (v2-day-one) The DSL is intentionally small. Each port translates it to the most idiomatic @@ -332,8 +371,9 @@ spelling on the actual side (a SQLite ratio is a REAL, a MySQL `DECIMAL` keeps i MySQL `DATETIME` is the UTC wall clock). A scenario whose seed cannot be shared (an instant relative to the database clock) adds a `seed-data-engine:` map keyed by `sqlite` / `mysql`, which wins over `seed-data` for that engine; the rest keep one Postgres-spelled `seed-data`, -which the MySQL runners translate mechanically (`"id"` to `` `id` ``, a UTC `Z` dropped). Java has -no SQLite driver and Python has only a Postgres one, so they do not run those engines. +which the MySQL runners translate mechanically (`"id"` to `` `id` ``, a UTC `Z` dropped). Every port +runs them on SQLite and MySQL (Java through OMDB's `SQLiteDriver`, Python through `SqliteDriver` +and `MySqlDriver`). A scenario with an `engines:` list runs on exactly those engines. Rows are keyed by the report's **derived field names** (dimensions in `@dimensions` order, then measures in `@measures` order; a time dimension is named ``, for example `createdAtMonth`). diff --git a/fixtures/persistence-conformance/canonical/meta.fitness.json b/fixtures/persistence-conformance/canonical/meta.fitness.json index de21b2cad..38de722c7 100644 --- a/fixtures/persistence-conformance/canonical/meta.fitness.json +++ b/fixtures/persistence-conformance/canonical/meta.fitness.json @@ -470,6 +470,52 @@ "@dimensions": ["kind"], "@measures": ["auths", "distinctRefs"], "@filter": { "type": { "ne": "PriorAuth" } }, "children": [ { "source.rdb": { "@kind": "view", "@view": "v_auths_by_kind" } } ] } }, + { "object.entity": { + "name": "Labeled", "abstract": true, + "children": [ + { "field.long": { "name": "id" } }, + { "field.string": { "name": "label", "@column": "label_text" } }, + { "identity.primary": { "name": "id", "@fields": "id", "@generation": "increment" } } ] } }, + { "object.entity": { + "name": "Gadget", "extends": "Labeled", + "children": [ + { "source.rdb": { "@table": "gadgets" } }, + { "field.int": { "name": "label", "@column": "gadget_code", "@required": true } } ] } }, + { "object.entity": { + "name": "GadgetEvent", + "children": [ + { "source.rdb": { "@table": "gadget_events" } }, + { "field.long": { "name": "id" } }, + { "field.long": { "name": "gadgetId", "@required": true } }, + { "identity.primary": { "name": "id", "@fields": "id", "@generation": "increment" } }, + { "identity.reference": { "name": "fkGadget", "@fields": "gadgetId", "@references": "Gadget" } }, + { "dimension.attribute": { "name": "gadgetLabel", "@of": "Labeled.label", "@via": "GadgetEvent.fkGadget" } }, + { "measure.aggregate": { "name": "gadgetEvents", "@agg": "count", "@of": "GadgetEvent.id" } } ] } }, + { "object.report": { "name": "GadgetEventsByLabel", "@from": "GadgetEvent", + "@dimensions": ["gadgetLabel"], "@measures": ["gadgetEvents"], + "children": [ { "source.rdb": { "@kind": "view", "@view": "v_gadget_events_by_label" } } ] } }, + + { "object.entity": { "name": "Entry", "children": [ + { "source.rdb": { "@table": "entries" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "bucket", "@required": true, "@maxLength": 8 } }, + { "field.decimal": { "name": "amount", "@precision": 28, "@scale": 10, "@filterable": true, "@sortable": true } }, + { "field.boolean": { "name": "flagged", "@required": true } }, + { "identity.primary": { "name": "pk", "@fields": "id", "@generation": "increment" } }, + { "dimension.attribute": { "name": "bucket", "@of": "Entry.bucket" } }, + { "measure.aggregate": { "name": "entries", "@agg": "count", "@of": "Entry.id" } }, + { "measure.aggregate": { "name": "flaggedEntries", "@agg": "count", "@of": "Entry.id", "@filter": { "flagged": true } } }, + { "measure.aggregate": { "name": "total", "@agg": "sum", "@of": "Entry.amount" } }, + { "measure.aggregate": { "name": "totalOrZero", "@agg": "sum", "@of": "Entry.amount", "@default": 0 } }, + { "measure.aggregate": { "name": "mean", "@agg": "avg", "@of": "Entry.amount" } }, + { "measure.aggregate": { "name": "top", "@agg": "max", "@of": "Entry.amount" } }, + { "measure.ratio": { "name": "flaggedShare", "@numerator": "flaggedEntries", "@denominator": "entries" } }, + { "measure.ratio": { "name": "perFlagged", "@numerator": "entries", "@denominator": "flaggedEntries" } } + ] } }, + { "object.report": { "name": "EntryTotals", "@from": "Entry", "@dimensions": ["bucket"], + "@measures": ["entries", "total", "totalOrZero", "mean", "top", "flaggedShare", "perFlagged"], + "children": [ { "source.rdb": { "@kind": "view", "@view": "v_entry_totals" } } ] } }, + { "template.prompt": { "name": "coachNote", "@payloadRef": "ProgramBrief", diff --git a/fixtures/persistence-conformance/canonical/schema.mysql.sql b/fixtures/persistence-conformance/canonical/schema.mysql.sql index 9e079ee34..1099c8e87 100644 --- a/fixtures/persistence-conformance/canonical/schema.mysql.sql +++ b/fixtures/persistence-conformance/canonical/schema.mysql.sql @@ -64,6 +64,24 @@ CREATE TABLE `auths` ( `urgency` VARCHAR(16) ); +CREATE TABLE `gadgets` ( + `id` BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY, + `gadget_code` INT NOT NULL +); + +CREATE TABLE `gadget_events` ( + `id` BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY, + `gadgetId` BIGINT NOT NULL, + FOREIGN KEY (`gadgetId`) REFERENCES `gadgets` (`id`) +); + +CREATE TABLE `entries` ( + `id` BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY, + `bucket` VARCHAR(8) NOT NULL, + `amount` DECIMAL(28,10), + `flagged` BOOLEAN NOT NULL +); + CREATE TABLE `all_types` ( `id` VARCHAR(36) NOT NULL DEFAULT (UUID()) PRIMARY KEY, `sVal` VARCHAR(200) NOT NULL, @@ -323,3 +341,24 @@ CREATE VIEW `v_auths_by_kind` AS FROM `auths` a WHERE a.`type` <> 'PriorAuth' GROUP BY a.`type`; + +CREATE VIEW `v_gadget_events_by_label` AS + SELECT + g0.`gadget_code` AS `gadgetLabel`, + COUNT(g.`id`) AS `gadgetEvents` + FROM `gadget_events` g + INNER JOIN `gadgets` g0 ON g0.`id` = g.`gadgetId` + GROUP BY g0.`gadget_code`; + +CREATE VIEW `v_entry_totals` AS + SELECT + e.`bucket` AS `bucket`, + COUNT(e.`id`) AS `entries`, + SUM(e.`amount`) AS `total`, + COALESCE(SUM(e.`amount`), 0) AS `totalOrZero`, + AVG(e.`amount`) AS `mean`, + MAX(e.`amount`) AS `top`, + COUNT(CASE WHEN e.`flagged` = TRUE THEN e.`id` END) / NULLIF(COUNT(e.`id`), 0) AS `flaggedShare`, + COUNT(e.`id`) / NULLIF(COUNT(CASE WHEN e.`flagged` = TRUE THEN e.`id` END), 0) AS `perFlagged` + FROM `entries` e + GROUP BY e.`bucket`; diff --git a/fixtures/persistence-conformance/canonical/schema.postgres.sql b/fixtures/persistence-conformance/canonical/schema.postgres.sql index 69bc94c63..edb7bcbba 100644 --- a/fixtures/persistence-conformance/canonical/schema.postgres.sql +++ b/fixtures/persistence-conformance/canonical/schema.postgres.sql @@ -139,6 +139,26 @@ CREATE TABLE "all_types" ( CONSTRAINT "all_types_intEnumVal_chk" CHECK ("intEnumVal" IN (0, 5, 9)) ); +CREATE TABLE "gadgets" ( + "id" BIGINT GENERATED BY DEFAULT AS IDENTITY NOT NULL, + "gadget_code" INTEGER NOT NULL, + CONSTRAINT "gadgets_pkey" PRIMARY KEY ("id") +); + +CREATE TABLE "gadget_events" ( + "id" BIGINT GENERATED BY DEFAULT AS IDENTITY NOT NULL, + "gadgetId" BIGINT NOT NULL, + CONSTRAINT "gadget_events_pkey" PRIMARY KEY ("id") +); + +CREATE TABLE "entries" ( + "id" BIGINT GENERATED BY DEFAULT AS IDENTITY NOT NULL, + "bucket" VARCHAR(8) NOT NULL, + "amount" NUMERIC(28,10), + "flagged" BOOLEAN NOT NULL, + CONSTRAINT "entries_pkey" PRIMARY KEY ("id") +); + CREATE UNIQUE INDEX "byTitle" ON "programs" ("title"); CREATE INDEX "idx_programs_title_status" ON "programs" ("title", "status"); @@ -165,6 +185,8 @@ ALTER TABLE "post_referrals" ADD CONSTRAINT "post_referrals_postId_fk" FOREIGN K ALTER TABLE "post_referrals" ADD CONSTRAINT "post_referrals_referralId_fk" FOREIGN KEY ("referralId") REFERENCES "auths" ("id"); +ALTER TABLE "gadget_events" ADD CONSTRAINT "gadget_events_gadgetId_fk" FOREIGN KEY ("gadgetId") REFERENCES "gadgets" ("id"); + CREATE VIEW "v_program" AS SELECT p.id AS id, @@ -444,3 +466,26 @@ CREATE VIEW "v_auths_by_kind" AS WHERE a."type" <> 'PriorAuth' GROUP BY a."type"; COMMENT ON VIEW "v_auths_by_kind" IS 'metaobjects:v1:sha256:600fcf3a54c8a8f51b2771850689073c7715193fe2528cd7af77bc2e97d22168'; + +CREATE VIEW "v_gadget_events_by_label" AS + SELECT + g0."gadget_code" AS "gadgetLabel", + COUNT(g."id") AS "gadgetEvents" + FROM "gadget_events" g + INNER JOIN "gadgets" g0 ON g0."id" = g."gadgetId" + GROUP BY g0."gadget_code"; +COMMENT ON VIEW "v_gadget_events_by_label" IS 'metaobjects:v1:sha256:ce353e347259448fb4055d7b20cb3de6210ce157c6824d3ab2b0b2dba661cc82'; + +CREATE VIEW "v_entry_totals" AS + SELECT + e."bucket" AS "bucket", + COUNT(e."id") AS "entries", + SUM(e."amount") AS "total", + COALESCE(SUM(e."amount"), 0) AS "totalOrZero", + AVG(e."amount") AS "mean", + MAX(e."amount") AS "top", + CAST(COUNT(e."id") FILTER (WHERE e."flagged" = TRUE) AS NUMERIC) / NULLIF(COUNT(e."id"), 0) AS "flaggedShare", + CAST(COUNT(e."id") AS NUMERIC) / NULLIF(COUNT(e."id") FILTER (WHERE e."flagged" = TRUE), 0) AS "perFlagged" + FROM "entries" e + GROUP BY e."bucket"; +COMMENT ON VIEW "v_entry_totals" IS 'metaobjects:v1:sha256:5290cebc2ffad56830125b30824ac07830ca635415bddc97572e015f77d51be9'; diff --git a/fixtures/persistence-conformance/canonical/schema.sqlite.sql b/fixtures/persistence-conformance/canonical/schema.sqlite.sql index b8ff5a985..b2c086a80 100644 --- a/fixtures/persistence-conformance/canonical/schema.sqlite.sql +++ b/fixtures/persistence-conformance/canonical/schema.sqlite.sql @@ -139,6 +139,24 @@ CREATE TABLE "all_types" ( CONSTRAINT "all_types_intEnumVal_chk" CHECK ("intEnumVal" IN (0, 5, 9)) ); +CREATE TABLE "gadgets" ( + "id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, + "gadget_code" INTEGER NOT NULL +); + +CREATE TABLE "gadget_events" ( + "id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, + "gadgetId" INTEGER NOT NULL, + FOREIGN KEY ("gadgetId") REFERENCES "gadgets" ("id") +); + +CREATE TABLE "entries" ( + "id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, + "bucket" VARCHAR(8) NOT NULL, + "amount" NUMERIC(28,10), + "flagged" INTEGER NOT NULL +); + CREATE UNIQUE INDEX "byTitle" ON "programs" ("title"); CREATE INDEX "idx_programs_title_status" ON "programs" ("title", "status"); @@ -397,3 +415,24 @@ CREATE VIEW "v_auths_by_kind" AS FROM "auths" a WHERE a."type" <> 'PriorAuth' GROUP BY a."type"; + +CREATE VIEW "v_gadget_events_by_label" AS + SELECT + g0."gadget_code" AS "gadgetLabel", + COUNT(g."id") AS "gadgetEvents" + FROM "gadget_events" g + INNER JOIN "gadgets" g0 ON g0."id" = g."gadgetId" + GROUP BY g0."gadget_code"; + +CREATE VIEW "v_entry_totals" AS + SELECT + e."bucket" AS "bucket", + COUNT(e."id") AS "entries", + SUM(e."amount") AS "total", + COALESCE(SUM(e."amount"), 0.0) AS "totalOrZero", + AVG(e."amount") AS "mean", + MAX(e."amount") AS "top", + CAST(COUNT(CASE WHEN e."flagged" = 1 THEN e."id" END) AS REAL) / NULLIF(COUNT(e."id"), 0) AS "flaggedShare", + CAST(COUNT(e."id") AS REAL) / NULLIF(COUNT(CASE WHEN e."flagged" = 1 THEN e."id" END), 0) AS "perFlagged" + FROM "entries" e + GROUP BY e."bucket"; diff --git a/fixtures/persistence-conformance/migrations/bootstrap-canonical-from-empty.yaml b/fixtures/persistence-conformance/migrations/bootstrap-canonical-from-empty.yaml index 457c81ffe..0bf7bec00 100644 --- a/fixtures/persistence-conformance/migrations/bootstrap-canonical-from-empty.yaml +++ b/fixtures/persistence-conformance/migrations/bootstrap-canonical-from-empty.yaml @@ -68,8 +68,13 @@ expect: # auths backs the FR-017 table-per-hierarchy (TPH) polymorphic corpus — # the discriminator base + its subtypes collapse to one physical table. - { table_name: "auths" } + # entries backs the decimal wire scenarios and the EntryTotals report (ADR-0019 Amendment 1). + - { table_name: "entries" } - { table_name: "follows" } - { table_name: "friendships" } + # gadgets / gadget_events: the report-via-redeclared-field scenario's tables. + - { table_name: "gadget_events" } + - { table_name: "gadgets" } - { table_name: "measurements" } - { table_name: "nodes" } - { table_name: "people" } diff --git a/fixtures/persistence-conformance/queries/decimal-exact-15.yaml b/fixtures/persistence-conformance/queries/decimal-exact-15.yaml new file mode 100644 index 000000000..45debb488 --- /dev/null +++ b/fixtures/persistence-conformance/queries/decimal-exact-15.yaml @@ -0,0 +1,55 @@ +name: decimal-exact-15 +description: | + The subset of decimal-exact every engine holds (W4): values of at most 15 significant digits. + SQLite and D1 keep a decimal in a NUMERIC column as a double, so 0.1 comes back as the + canonical "0.1" only because the read renders the double at 15 significant digits. A runtime that + hands the raw double to a caller, or "100.50" with its scale, fails the typed decimal gate. +engines: [postgres, mysql, sqlite, d1] +queries: + - name: insert-1 + op: roundtrip + entity: Entry + insert: { id: 1, bucket: A, amount: "0.1", flagged: true } + expect: { bucket: A, amount: "0.1", flagged: true } + - name: insert-2 + op: roundtrip + entity: Entry + insert: { id: 2, bucket: A, amount: "0.2", flagged: false } + expect: { bucket: A, amount: "0.2", flagged: false } + - name: insert-7 + op: roundtrip + entity: Entry + insert: { id: 7, bucket: D, amount: "100.50", flagged: false } + expect: { bucket: D, amount: "100.5", flagged: false } + - name: insert-8 + op: roundtrip + entity: Entry + insert: { id: 8, bucket: D, amount: "99.5", flagged: false } + expect: { bucket: D, amount: "99.5", flagged: false } + - name: insert-9 + op: roundtrip + entity: Entry + insert: { id: 9, bucket: E, amount: "0.0000000000", flagged: false } + expect: { bucket: E, amount: "0", flagged: false } + - name: insert-10 + op: roundtrip + entity: Entry + insert: { id: 10, bucket: E, amount: null, flagged: false } + expect: { bucket: E, amount: null, flagged: false } + - name: insert-11 + op: roundtrip + entity: Entry + insert: { id: 11, bucket: F, amount: null, flagged: false } + expect: { bucket: F, amount: null, flagged: false } + - name: list-reads-canonical-strings + op: list + entity: Entry + sort: [{ field: id, dir: asc }] + expect: + - { id: "1", bucket: A, amount: "0.1", flagged: true } + - { id: "2", bucket: A, amount: "0.2", flagged: false } + - { id: "7", bucket: D, amount: "100.5", flagged: false } + - { id: "8", bucket: D, amount: "99.5", flagged: false } + - { id: "9", bucket: E, amount: "0", flagged: false } + - { id: "10", bucket: E, amount: null, flagged: false } + - { id: "11", bucket: F, amount: null, flagged: false } diff --git a/fixtures/persistence-conformance/queries/decimal-exact.yaml b/fixtures/persistence-conformance/queries/decimal-exact.yaml new file mode 100644 index 000000000..239012dbc --- /dev/null +++ b/fixtures/persistence-conformance/queries/decimal-exact.yaml @@ -0,0 +1,112 @@ +name: decimal-exact +description: | + A field.decimal is exact on the wire (ADR-0019 Amendment 1, W1 and W2): written as a string + through the runtime's write codec, read back as a canonical string (plain notation, no trailing + fraction zeros, zero as "0"), and compared exactly by a filter. `12345678901234567.89` has 19 + significant digits, more than a double holds, so a runtime that converts a decimal to a binary + float anywhere on the path (a filter operand, a bound parameter, a read) returns the wrong row + or the wrong text. Only Postgres and MySQL hold a decimal exactly (W4), so only they run it; + decimal-exact-15 is the 15-digit subset every engine runs. +engines: [postgres, mysql] +queries: + - name: insert-1 + op: roundtrip + entity: Entry + insert: { id: 1, bucket: A, amount: "0.1", flagged: true } + expect: { bucket: A, amount: "0.1", flagged: true } + - name: insert-2 + op: roundtrip + entity: Entry + insert: { id: 2, bucket: A, amount: "0.2", flagged: false } + expect: { bucket: A, amount: "0.2", flagged: false } + - name: insert-3 + op: roundtrip + entity: Entry + insert: { id: 3, bucket: B, amount: "12345678901234567.89", flagged: false } + expect: { bucket: B, amount: "12345678901234567.89", flagged: false } + - name: insert-4 + op: roundtrip + entity: Entry + insert: { id: 4, bucket: B, amount: "0.01", flagged: true } + expect: { bucket: B, amount: "0.01", flagged: true } + - name: insert-5 + op: roundtrip + entity: Entry + insert: { id: 5, bucket: C, amount: "-0.0000000001", flagged: false } + expect: { bucket: C, amount: "-0.0000000001", flagged: false } + - name: insert-6 + op: roundtrip + entity: Entry + insert: { id: 6, bucket: C, amount: "-12345678.987654321", flagged: false } + expect: { bucket: C, amount: "-12345678.987654321", flagged: false } + - name: insert-7 + op: roundtrip + entity: Entry + insert: { id: 7, bucket: D, amount: "100.50", flagged: false } + expect: { bucket: D, amount: "100.5", flagged: false } + - name: insert-8 + op: roundtrip + entity: Entry + insert: { id: 8, bucket: D, amount: "99.5", flagged: false } + expect: { bucket: D, amount: "99.5", flagged: false } + - name: insert-9 + op: roundtrip + entity: Entry + insert: { id: 9, bucket: E, amount: "0.0000000000", flagged: false } + expect: { bucket: E, amount: "0", flagged: false } + - name: insert-10 + op: roundtrip + entity: Entry + insert: { id: 10, bucket: E, amount: null, flagged: false } + expect: { bucket: E, amount: null, flagged: false } + - name: insert-11 + op: roundtrip + entity: Entry + insert: { id: 11, bucket: F, amount: null, flagged: false } + expect: { bucket: F, amount: null, flagged: false } + - name: list-reads-canonical-strings + op: list + entity: Entry + sort: [{ field: id, dir: asc }] + expect: + - { id: "1", bucket: A, amount: "0.1", flagged: true } + - { id: "2", bucket: A, amount: "0.2", flagged: false } + - { id: "3", bucket: B, amount: "12345678901234567.89", flagged: false } + - { id: "4", bucket: B, amount: "0.01", flagged: true } + - { id: "5", bucket: C, amount: "-0.0000000001", flagged: false } + - { id: "6", bucket: C, amount: "-12345678.987654321", flagged: false } + - { id: "7", bucket: D, amount: "100.5", flagged: false } + - { id: "8", bucket: D, amount: "99.5", flagged: false } + - { id: "9", bucket: E, amount: "0", flagged: false } + - { id: "10", bucket: E, amount: null, flagged: false } + - { id: "11", bucket: F, amount: null, flagged: false } + - name: filter-eq-is-exact + op: list + entity: Entry + filter: { amount: { eq: "12345678901234567.89" } } + expect: + - { id: "3", bucket: B, amount: "12345678901234567.89", flagged: false } + - name: filter-gt-is-exact + op: list + entity: Entry + filter: { amount: { gt: "12345678901234567.8" } } + expect: + - { id: "3", bucket: B, amount: "12345678901234567.89", flagged: false } + - name: filter-eq-ignores-trailing-zeros + op: list + entity: Entry + filter: { amount: { eq: "100.5000" } } + expect: + - { id: "7", bucket: D, amount: "100.5", flagged: false } + - name: filter-eq-takes-an-exponent + op: list + entity: Entry + filter: { amount: { eq: "1.005e2" } } + expect: + - { id: "7", bucket: D, amount: "100.5", flagged: false } + - name: write-rejects-garbage + op: create + entity: Entry + expect-error: true + data: { id: 99, bucket: Z, amount: "12abc", flagged: false } + expect: null diff --git a/fixtures/persistence-conformance/queries/report-decimal-exact-15.yaml b/fixtures/persistence-conformance/queries/report-decimal-exact-15.yaml new file mode 100644 index 000000000..8fd340455 --- /dev/null +++ b/fixtures/persistence-conformance/queries/report-decimal-exact-15.yaml @@ -0,0 +1,23 @@ +name: report-decimal-exact-15 +description: | + The report-decimal-exact buckets whose values fit 15 significant digits (A, D, E, F), run on every + engine. SQLite and D1 compute a sum of 0.1 and 0.2 as the double 0.30000000000000004; the read + renders it at 15 significant digits (W4), so the wire says "0.3" and the ratio says "0.5". +seed-data: | + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (1, 'A', 0.1, TRUE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (2, 'A', 0.2, FALSE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (7, 'D', 100.50, FALSE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (8, 'D', 99.5, FALSE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (9, 'E', 0.0000000000, FALSE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (10, 'E', NULL, FALSE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (11, 'F', NULL, FALSE); +queries: + - name: entry-totals + op: list + entity: EntryTotals + sort: [{ field: bucket, dir: asc }] + expect: + - { bucket: A, entries: "2", total: "0.3", totalOrZero: "0.3", mean: "0.15", top: "0.2", flaggedShare: "0.5", perFlagged: "2" } + - { bucket: D, entries: "2", total: "200", totalOrZero: "200", mean: "100", top: "100.5", flaggedShare: "0", perFlagged: null } + - { bucket: E, entries: "2", total: "0", totalOrZero: "0", mean: "0", top: "0", flaggedShare: "0", perFlagged: null } + - { bucket: F, entries: "1", total: null, totalOrZero: "0", mean: null, top: null, flaggedShare: "0", perFlagged: null } diff --git a/fixtures/persistence-conformance/queries/report-decimal-exact.yaml b/fixtures/persistence-conformance/queries/report-decimal-exact.yaml new file mode 100644 index 000000000..77995399c --- /dev/null +++ b/fixtures/persistence-conformance/queries/report-decimal-exact.yaml @@ -0,0 +1,32 @@ +name: report-decimal-exact +description: | + A report over a decimal is exact on the wire (ADR-0019 Amendment 1): a sum of 12345678901234567.89 + and 0.01 is "12345678901234567.9", an average keeps its digits, an empty sum is null unless the + measure declares @default (then "0"), and a ratio with a zero denominator is null. The view is + computed by the engine, so this is the engine's own decimal arithmetic read back through the + runtime. Postgres and MySQL only (W4); report-decimal-exact-15 is the subset every engine runs. +engines: [postgres, mysql] +seed-data: | + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (1, 'A', 0.1, TRUE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (2, 'A', 0.2, FALSE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (3, 'B', 12345678901234567.89, FALSE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (4, 'B', 0.01, TRUE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (5, 'C', -0.0000000001, FALSE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (6, 'C', -12345678.987654321, FALSE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (7, 'D', 100.50, FALSE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (8, 'D', 99.5, FALSE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (9, 'E', 0.0000000000, FALSE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (10, 'E', NULL, FALSE); + INSERT INTO "entries" ("id","bucket","amount","flagged") VALUES (11, 'F', NULL, FALSE); +queries: + - name: entry-totals + op: list + entity: EntryTotals + sort: [{ field: bucket, dir: asc }] + expect: + - { bucket: A, entries: "2", total: "0.3", totalOrZero: "0.3", mean: "0.15", top: "0.2", flaggedShare: "0.5", perFlagged: "2" } + - { bucket: B, entries: "2", total: "12345678901234567.9", totalOrZero: "12345678901234567.9", mean: "6172839450617283.95", top: "12345678901234567.89", flaggedShare: "0.5", perFlagged: "2" } + - { bucket: C, entries: "2", total: "-12345678.9876543211", totalOrZero: "-12345678.9876543211", mean: "-6172839.49382716055", top: "-0.0000000001", flaggedShare: "0", perFlagged: null } + - { bucket: D, entries: "2", total: "200", totalOrZero: "200", mean: "100", top: "100.5", flaggedShare: "0", perFlagged: null } + - { bucket: E, entries: "2", total: "0", totalOrZero: "0", mean: "0", top: "0", flaggedShare: "0", perFlagged: null } + - { bucket: F, entries: "1", total: null, totalOrZero: "0", mean: null, top: null, flaggedShare: "0", perFlagged: null } diff --git a/fixtures/persistence-conformance/queries/report-via-redeclared-field.yaml b/fixtures/persistence-conformance/queries/report-via-redeclared-field.yaml new file mode 100644 index 000000000..de823e6d1 --- /dev/null +++ b/fixtures/persistence-conformance/queries/report-via-redeclared-field.yaml @@ -0,0 +1,26 @@ +name: report-via-redeclared-field +description: | + A @via dimension whose @of names the base entity (`Labeled.label`, a string in column + `label_text`) while the entity the @via reaches (`Gadget`) redeclares the field as an int in + column `gadget_code`. The rule: the field is the effective field of the entity the @via + reaches, so the report groups by `gadgets.gadget_code` and `gadgetLabel` is an int (it is + `typeSource` `fitness::Gadget.label` in report-shapes.json), never the base's string column. + + gadget 1 has code 7 and two events; gadget 2 has code 7 and one; gadget 3 has code 9 and one +seed-data: | + INSERT INTO "gadgets" ("id","gadget_code") VALUES (1, 7), (2, 7), (3, 9); + INSERT INTO "gadget_events" ("id","gadgetId") VALUES (10, 1), (11, 1), (12, 2), (13, 3); +queries: + - name: grouped-by-the-redeclared-column + op: list + entity: GadgetEventsByLabel + sort: [{ field: gadgetLabel, dir: asc }] + expect: + - { gadgetLabel: 7, gadgetEvents: "3" } + - { gadgetLabel: 9, gadgetEvents: "1" } + - name: filter-on-the-redeclared-int + op: list + entity: GadgetEventsByLabel + filter: { gadgetLabel: { gt: 7 } } + expect: + - { gadgetLabel: 9, gadgetEvents: "1" } diff --git a/fixtures/persistence-conformance/report-shapes.json b/fixtures/persistence-conformance/report-shapes.json index 5c28b57dc..3bec6f7f9 100644 --- a/fixtures/persistence-conformance/report-shapes.json +++ b/fixtures/persistence-conformance/report-shapes.json @@ -937,6 +937,90 @@ "typeSource": null } ] + }, + { + "report": "fitness::GadgetEventsByLabel", + "from": "fitness::GadgetEvent", + "view": "v_gadget_events_by_label", + "fields": [ + { + "name": "gadgetLabel", + "role": "dimension", + "subType": "int", + "required": true, + "typeSource": "fitness::Gadget.label" + }, + { + "name": "gadgetEvents", + "role": "measure", + "subType": "long", + "required": true, + "typeSource": null + } + ] + }, + { + "report": "fitness::EntryTotals", + "from": "fitness::Entry", + "view": "v_entry_totals", + "fields": [ + { + "name": "bucket", + "role": "dimension", + "subType": "string", + "required": true, + "typeSource": "fitness::Entry.bucket" + }, + { + "name": "entries", + "role": "measure", + "subType": "long", + "required": true, + "typeSource": null + }, + { + "name": "total", + "role": "measure", + "subType": "decimal", + "required": false, + "typeSource": null + }, + { + "name": "totalOrZero", + "role": "measure", + "subType": "decimal", + "required": true, + "typeSource": null + }, + { + "name": "mean", + "role": "measure", + "subType": "decimal", + "required": false, + "typeSource": null + }, + { + "name": "top", + "role": "measure", + "subType": "decimal", + "required": false, + "typeSource": "fitness::Entry.amount" + }, + { + "name": "flaggedShare", + "role": "measure", + "subType": "decimal", + "required": false, + "typeSource": null + }, + { + "name": "perFlagged", + "role": "measure", + "subType": "decimal", + "required": false, + "typeSource": null + } + ] } ] } diff --git a/fixtures/requirement-check-conformance/README.md b/fixtures/requirement-check-conformance/README.md index ec7837492..5d77cc2dc 100644 --- a/fixtures/requirement-check-conformance/README.md +++ b/fixtures/requirement-check-conformance/README.md @@ -108,6 +108,9 @@ it is wrong, unless the reference is shown to be wrong first. | `l5-names-object` | `ERR_REQUIREMENT_L5_NOT_MEMBER` on a functional L5, for an object that exists and for one that does not. | | `l5-member-resolves-clean` | A functional L5 naming a field, an identity, and a view under a field: no diagnostics. | | `bad-level-out-of-range` | `ERR_REQUIREMENT_BAD_LEVEL` for levels 0 and 6 on functional requirements. The message has no architectural suffix. | +| `package-packageless-file-beside-packaged` | A package-less requirements document beside packaged ones: the requirement's package is "", so a bare reference to an entity that lives only in `acme::shop` is `ERR_REQUIREMENT_DANGLING_REF`. A regression pin; every port agrees. | +| `package-overlay-child-from-packageless-file` | A child contributed through an `overlay: true` requirement that declares its own package, from a package-less file: the child resolves in the nearest declaring ancestor's package (`acme::shop`), so the reference is clean. | +| `package-overlay-child-from-other-package-file` | The same overlay shape from a file of a different package: the child still resolves in the declaring ancestor's package, so the reference is clean. | | `level-nesting` | `ERR_REQUIREMENT_LEVEL_NESTING` for a child at its parent's level and for a child above it. | | `arch-live-no-implementers` | `ERR_REQUIREMENT_ARCH_NO_IMPLEMENTERS` on a flat policy that is `live`, on one that is `partial`, and on a levelled L4 that is `live`: the rule is "may name the model and names nothing", not "has no level". | | `arch-planned-clean` | A `planned` flat policy with no implementer reports nothing. | diff --git a/fixtures/requirement-check-conformance/package-overlay-child-from-other-package-file/expected.json b/fixtures/requirement-check-conformance/package-overlay-child-from-other-package-file/expected.json new file mode 100644 index 000000000..1e8affbf9 --- /dev/null +++ b/fixtures/requirement-check-conformance/package-overlay-child-from-other-package-file/expected.json @@ -0,0 +1,15 @@ +{ + "diagnostics": [], + "summary": { + "total": 2, + "functional": 2, + "architectural": 0, + "byStatus": { + "live": 2 + }, + "undecided": 0, + "deferredUntracked": 0, + "entitiesTotal": 1, + "entitiesClaimed": 1 + } +} diff --git a/fixtures/requirement-check-conformance/package-overlay-child-from-other-package-file/input/meta.a-shop.json b/fixtures/requirement-check-conformance/package-overlay-child-from-other-package-file/input/meta.a-shop.json new file mode 100644 index 000000000..5612afd03 --- /dev/null +++ b/fixtures/requirement-check-conformance/package-overlay-child-from-other-package-file/input/meta.a-shop.json @@ -0,0 +1,36 @@ +{ + "metadata.root": { + "package": "acme::shop", + "children": [ + { + "object.entity": { + "name": "Order", + "children": [ + { + "field.long": { + "name": "id" + } + }, + { + "identity.primary": { + "name": "pk", + "@fields": [ + "id" + ] + } + } + ] + } + }, + { + "requirement.functional": { + "name": "Orders", + "@level": 3, + "@status": "live", + "@statement": "An order is recorded when it is placed.", + "@counterexample": "A placed order has no row." + } + } + ] + } +} diff --git a/fixtures/requirement-check-conformance/package-overlay-child-from-other-package-file/input/meta.b-overlay.json b/fixtures/requirement-check-conformance/package-overlay-child-from-other-package-file/input/meta.b-overlay.json new file mode 100644 index 000000000..ff42917d0 --- /dev/null +++ b/fixtures/requirement-check-conformance/package-overlay-child-from-other-package-file/input/meta.b-overlay.json @@ -0,0 +1,28 @@ +{ + "metadata.root": { + "children": [ + { + "requirement.functional": { + "name": "Orders", + "overlay": true, + "package": "acme::shop", + "children": [ + { + "requirement.functional": { + "name": "Recorded", + "@level": 4, + "@status": "live", + "@statement": "An order is recorded when it is placed.", + "@counterexample": "A placed order has no row.", + "@implementedBy": [ + "Order" + ] + } + } + ] + } + } + ], + "package": "acme::billing" + } +} diff --git a/fixtures/requirement-check-conformance/package-overlay-child-from-packageless-file/expected.json b/fixtures/requirement-check-conformance/package-overlay-child-from-packageless-file/expected.json new file mode 100644 index 000000000..1e8affbf9 --- /dev/null +++ b/fixtures/requirement-check-conformance/package-overlay-child-from-packageless-file/expected.json @@ -0,0 +1,15 @@ +{ + "diagnostics": [], + "summary": { + "total": 2, + "functional": 2, + "architectural": 0, + "byStatus": { + "live": 2 + }, + "undecided": 0, + "deferredUntracked": 0, + "entitiesTotal": 1, + "entitiesClaimed": 1 + } +} diff --git a/fixtures/requirement-check-conformance/package-overlay-child-from-packageless-file/input/meta.a-shop.json b/fixtures/requirement-check-conformance/package-overlay-child-from-packageless-file/input/meta.a-shop.json new file mode 100644 index 000000000..5612afd03 --- /dev/null +++ b/fixtures/requirement-check-conformance/package-overlay-child-from-packageless-file/input/meta.a-shop.json @@ -0,0 +1,36 @@ +{ + "metadata.root": { + "package": "acme::shop", + "children": [ + { + "object.entity": { + "name": "Order", + "children": [ + { + "field.long": { + "name": "id" + } + }, + { + "identity.primary": { + "name": "pk", + "@fields": [ + "id" + ] + } + } + ] + } + }, + { + "requirement.functional": { + "name": "Orders", + "@level": 3, + "@status": "live", + "@statement": "An order is recorded when it is placed.", + "@counterexample": "A placed order has no row." + } + } + ] + } +} diff --git a/fixtures/requirement-check-conformance/package-overlay-child-from-packageless-file/input/meta.b-overlay.json b/fixtures/requirement-check-conformance/package-overlay-child-from-packageless-file/input/meta.b-overlay.json new file mode 100644 index 000000000..128300bf6 --- /dev/null +++ b/fixtures/requirement-check-conformance/package-overlay-child-from-packageless-file/input/meta.b-overlay.json @@ -0,0 +1,27 @@ +{ + "metadata.root": { + "children": [ + { + "requirement.functional": { + "name": "Orders", + "overlay": true, + "package": "acme::shop", + "children": [ + { + "requirement.functional": { + "name": "Recorded", + "@level": 4, + "@status": "live", + "@statement": "An order is recorded when it is placed.", + "@counterexample": "A placed order has no row.", + "@implementedBy": [ + "Order" + ] + } + } + ] + } + } + ] + } +} diff --git a/fixtures/requirement-check-conformance/package-packageless-file-beside-packaged/expected.json b/fixtures/requirement-check-conformance/package-packageless-file-beside-packaged/expected.json new file mode 100644 index 000000000..cd55088ba --- /dev/null +++ b/fixtures/requirement-check-conformance/package-packageless-file-beside-packaged/expected.json @@ -0,0 +1,27 @@ +{ + "diagnostics": [ + { + "severity": "error", + "code": "ERR_REQUIREMENT_DANGLING_REF", + "path": "Orders", + "message": "'Order' does not resolve in the loaded model (status 'live' — the model moved and the requirement is stale). An object named \"Order\" exists in: acme::shop::Order. Qualify it with its package (FQN)." + }, + { + "severity": "warn", + "code": "WARN_REQUIREMENT_OBJECT_UNCLAIMED", + "message": "no requirement claims 'acme::shop::Order'. Add it to an L4 requirement's 'implementedBy'." + } + ], + "summary": { + "total": 1, + "functional": 1, + "architectural": 0, + "byStatus": { + "live": 1 + }, + "undecided": 0, + "deferredUntracked": 0, + "entitiesTotal": 1, + "entitiesClaimed": 0 + } +} diff --git a/fixtures/requirement-check-conformance/package-packageless-file-beside-packaged/input/meta.a-shop.json b/fixtures/requirement-check-conformance/package-packageless-file-beside-packaged/input/meta.a-shop.json new file mode 100644 index 000000000..529a0193f --- /dev/null +++ b/fixtures/requirement-check-conformance/package-packageless-file-beside-packaged/input/meta.a-shop.json @@ -0,0 +1,27 @@ +{ + "metadata.root": { + "children": [ + { + "object.entity": { + "name": "Order", + "children": [ + { + "field.long": { + "name": "id" + } + }, + { + "identity.primary": { + "name": "pk", + "@fields": [ + "id" + ] + } + } + ] + } + } + ], + "package": "acme::shop" + } +} diff --git a/fixtures/requirement-check-conformance/package-packageless-file-beside-packaged/input/meta.b-requirements.json b/fixtures/requirement-check-conformance/package-packageless-file-beside-packaged/input/meta.b-requirements.json new file mode 100644 index 000000000..49acbd561 --- /dev/null +++ b/fixtures/requirement-check-conformance/package-packageless-file-beside-packaged/input/meta.b-requirements.json @@ -0,0 +1,18 @@ +{ + "metadata.root": { + "children": [ + { + "requirement.functional": { + "name": "Orders", + "@level": 4, + "@status": "live", + "@statement": "An order is recorded when it is placed.", + "@counterexample": "A placed order has no row.", + "@implementedBy": [ + "Order" + ] + } + } + ] + } +} diff --git a/fixtures/source-resolution-conformance/README.md b/fixtures/source-resolution-conformance/README.md index b911f36bd..40168fbd1 100644 --- a/fixtures/source-resolution-conformance/README.md +++ b/fixtures/source-resolution-conformance/README.md @@ -154,40 +154,26 @@ README.md affect resolution AND happened to also validate it" — the two are indistinguishable from the outside, and only the first is what every other port's "ignore the rest" behavior demonstrates. - **A genuinely unrecognized key (e.g. `"foo": 1`, unknown to all four ports) - is a real, confirmed, cross-port DIVERGENCE, not covered by this corpus.** - Verified empirically: `resolveCollection` (`collection.ts`) calls - `loadConfig`, which parses the WHOLE file through `ConfigSchema.parse` — - `.strict()` at the top level (`config.ts`) — so a key no version of - TypeScript has ever declared throws a `ZodError` and resolution never - reaches the source-listing step at all, while Java/C#/Python all resolve - successfully, silently ignoring it. - - **RULED: the asymmetry is INTENDED, and neither side moves.** It is not a - defect that leaked; it falls out of who owns the file. `.metaobjects/config.json` - is TypeScript's, and TypeScript is the only port that models its whole - vocabulary — so it is the only port that CAN tell a typo from a key a sibling - owns. `.strict()` is what converts that knowledge into a diagnostic, and the - hazard is specific and proven: a stripped `scopes` (for `scope`) silently means - "everything in scope", and a stripped `migrate.scopee` silently governs the whole - database. Java, C# and Python know only `schema_version` + `sources`; for them - every other key is indistinguishable from a TS-owned one, so strictness there - could only be imitated by embedding TypeScript's key list and keeping it in - lockstep forever — at which point any port a release behind would REJECT a - config a newer `meta` had just written. Tolerance is the only coherent behaviour - for a partial reader, and strictness the only coherent behaviour for the owner. - - So this stays OFF the shared corpus permanently — not deferred. A shared case - asserts ONE outcome, and the correct outcome here differs by port BY DESIGN; - adding one could only be done by making some port wrong. Each half is pinned in - the port that owns it instead: TypeScript's in `sdk/test/config.test.ts` - (unknown top-level key ⇒ throws, alongside the nested `scopes`/`scopee` cases), - and the tolerant half in `SourceResolverTests.cs`, - `tests/config/test_source_resolver.py` and `SourceResolverTest.java` - (unknown top-level key ⇒ resolves normally). The - forward-compatibility cost is real and accepted: a config written by a NEWER - `meta` hard-fails an OLDER one. That is what `schema_version` is for, and a loud - failure beats a `scope` that silently matched everything. + **A genuinely unrecognized top-level key (e.g. `"foo": 1`, unknown to all four + ports) is ignored by every port since 1.1.0, and TypeScript also says so.** + `loadConfig` (`sdk/src/config.ts`) prints one warning per file and key to stderr, + naming the key and saying a newer MetaObjects may have written it, and resolves + normally. Before 1.1.0 TypeScript refused it (`ConfigSchema` is `.strict()` at the + top level), so a config a newer release wrote with a new key (1.1.0's `naming`) + failed every 1.0.x TypeScript tool in the estate. A tool a release behind must keep + working, so an unknown top-level key cannot be fatal. + + What stays fatal in TypeScript is what TypeScript can tell is a mistake: a wrong + type on a key it knows, and an unknown key inside a block it owns (the stripped + `scopes` for `scope`, `migrate.scopee`), because those silently change what a known + setting means. Java, C# and Python read only `schema_version`, `sources` and + `naming` and ignore the rest, as before. + + This stays off the shared corpus: the observable outcome (resolution succeeds) is + the same everywhere, and the warning is TypeScript's own output. Each port pins its + half: `sdk/test/config.test.ts` (unknown top-level key warns and resolves; nested + typo throws), `SourceResolverTests.cs`, `tests/config/test_source_resolver.py` and + `SourceResolverTest.java` (unknown top-level key resolves normally). ## Order is deliberately NOT pinned diff --git a/scripts/check-release-tag.mjs b/scripts/check-release-tag.mjs new file mode 100644 index 000000000..edec8972e --- /dev/null +++ b/scripts/check-release-tag.mjs @@ -0,0 +1,99 @@ +#!/usr/bin/env node +// A publish workflow publishes the version COMMITTED in its manifest, whatever tag started it. +// A `python-v1.1.1` tag pushed on a commit whose pyproject still says 1.1.0 would publish +// 1.1.0 (or fail half way, as a duplicate). This check runs first in each publish workflow and +// refuses the run when the tag that triggered it does not name the manifest's version. +// +// node scripts/check-release-tag.mjs [] +// +// On a tag push (GITHUB_REF_TYPE=tag) the tag must be `-v`, and a +// version override is refused (the tag is the version). On a manual run there is no tag to +// compare, so the check only reports what will be published. + +import { readFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +const REPO = join(dirname(fileURLToPath(import.meta.url)), ".."); + +/** + * @typedef {{ prefix: string, file: string, read: (text: string) => string | undefined }} PortSpec + * @typedef {"npm" | "python" | "csharp" | "java"} PortName + */ + +/** Each publishing port: its tag prefix and how to read its committed version. @type {Record} */ +export const PORTS = { + npm: { + prefix: "npm", + file: "server/typescript/packages/cli/package.json", + read: (text) => /** @type {{ version?: string }} */ (JSON.parse(text)).version, + }, + python: { + prefix: "python", + file: "server/python/pyproject.toml", + read: (text) => /^version\s*=\s*"([^"]+)"/m.exec(text)?.[1], + }, + csharp: { + prefix: "csharp", + file: "server/csharp/Directory.Build.props", + read: (text) => /([^<]+)<\/Version>/.exec(text)?.[1], + }, + java: { + prefix: "java", + file: "server/java/pom.xml", + // The parent's own , the first one after its (not the 's + // or a dependency's): the reactor pom has no , so the first is it. + read: (text) => /([^<]+)<\/version>/.exec(text)?.[1], + }, +}; + +/** + * The verdict for one run. Pure, so the test drives it without a checkout or an environment. + * Returns `{ ok: true, message }` or `{ ok: false, message }`. + * @param {{ port: string | undefined, manifestVersion: string | undefined, refType: string | undefined, refName: string | undefined, override?: string }} run + * @returns {{ ok: boolean, message: string }} + */ +export function checkReleaseTag({ port, manifestVersion, refType, refName, override }) { + const spec = portSpec(port); + if (spec === undefined) return { ok: false, message: `unknown port '${port}' (expected ${Object.keys(PORTS).join(", ")})` }; + if (!manifestVersion) return { ok: false, message: `no version found in ${spec.file}` }; + if (refType !== "tag") { + const v = override || manifestVersion; + return { ok: true, message: `manual run, no tag to compare: publishing ${port} ${v}` }; + } + if (override) { + return { ok: false, message: `a tag run takes its version from the tag; drop the version override '${override}'` }; + } + const want = `${spec.prefix}-v${manifestVersion}`; + if (refName !== want) { + return { + ok: false, + message: + `tag '${refName}' does not match ${spec.file}, which says ${manifestVersion}. ` + + `Push '${want}', or commit the version the tag names before tagging.`, + }; + } + return { ok: true, message: `tag '${refName}' matches ${spec.file} (${manifestVersion})` }; +} + +/** @param {string | undefined} port @returns {PortSpec | undefined} */ +function portSpec(port) { + return port !== undefined && Object.hasOwn(PORTS, port) ? PORTS[/** @type {PortName} */ (port)] : undefined; +} + +function main() { + const [port, override = ""] = process.argv.slice(2); + const spec = portSpec(port); + const manifestVersion = spec === undefined ? undefined : spec.read(readFileSync(join(REPO, spec.file), "utf8")); + const verdict = checkReleaseTag({ + port, + manifestVersion, + refType: process.env.GITHUB_REF_TYPE, + refName: process.env.GITHUB_REF_NAME, + override, + }); + (verdict.ok ? console.log : console.error)(`${verdict.ok ? "✓" : "✗"} ${verdict.message}`); + process.exit(verdict.ok ? 0 : 1); +} + +if (process.argv[1] && fileURLToPath(import.meta.url) === process.argv[1]) main(); diff --git a/scripts/ci-local.sh b/scripts/ci-local.sh index 611442883..dd59f9e4a 100755 --- a/scripts/ci-local.sh +++ b/scripts/ci-local.sh @@ -427,7 +427,9 @@ gate_site_reference() { bun_install && bun scripts/build-site-reference.ts --che # Also runs the Maven-coordinate derivation`s own tests: the same rule serves four release # scripts and two of them used to hardcode major 7, so the 1.0 cases (never released, never # exercised) are the ones worth a test at all. See scripts/maven-coordinate.mjs. -gate_release_tag() { bun test scripts/finish-release.test.ts scripts/eject-release-gate.test.ts && node scripts/test-maven-coordinate.mjs && node scripts/test-prerelease-version.mjs && bash tools/prerelease/test-prerelease-link.sh; } +# And the publish workflows' tag check (scripts/check-release-tag.mjs), whose test also reads +# each real manifest, so a layout change fails here rather than on a release tag. +gate_release_tag() { bun test scripts/finish-release.test.ts scripts/eject-release-gate.test.ts && node scripts/test-maven-coordinate.mjs && node scripts/test-prerelease-version.mjs && node scripts/test-check-release-tag.mjs && node scripts/test-changelog-promote.mjs && bash tools/prerelease/test-prerelease-link.sh; } # ── repo-root scripts/ typechecks ───────────────────────────────────────────── # `bun test` transpiles per file and never typechecks, and `bun run --filter '*' diff --git a/scripts/eject-release-gate.test.ts b/scripts/eject-release-gate.test.ts index 1c72aec0d..98e743f50 100644 --- a/scripts/eject-release-gate.test.ts +++ b/scripts/eject-release-gate.test.ts @@ -24,6 +24,7 @@ function repoWith(body: string | undefined, scripts: string[] = []): string { created.push(root); mkdirSync(join(root, "scripts"), { recursive: true }); for (const script of ["eject-release-gate.mjs", ...scripts]) { + mkdirSync(dirname(join(root, "scripts", script)), { recursive: true }); writeFileSync(join(root, "scripts", script), readFileSync(join(REPO, "scripts", script))); } if (body !== undefined) put(root, EJECT_MANIFEST, body); @@ -103,7 +104,7 @@ describe("eject release gate — the command and its wiring", () => { // check after it is the branch check, so a fixture on another branch tells the two apart: // a refused version stops before "must be on main", and an allowed one reaches it. test("release.mjs asks the gate first in its preflight, and stops on a refusal", () => { - const RELEASE_SCRIPTS = ["release.mjs", "publish-set.mjs", "maven-coordinate.mjs"]; + const RELEASE_SCRIPTS = ["release.mjs", "lib/changelog-promote.mjs", "publish-set.mjs", "maven-coordinate.mjs"]; const onBranch = (pending: (typeof ENTRY)[]) => { const root = repoWith(JSON.stringify(manifest(pending)), RELEASE_SCRIPTS); const git = (...args: string[]) => spawnSync("git", ["-C", root, ...args], { encoding: "utf8" }); diff --git a/scripts/lib/changelog-promote.mjs b/scripts/lib/changelog-promote.mjs new file mode 100644 index 000000000..eaf8fe78d --- /dev/null +++ b/scripts/lib/changelog-promote.mjs @@ -0,0 +1,31 @@ +// How `scripts/release.mjs` dates the CHANGELOG section of the version it releases. +// +// Two layouts reach a release: +// +// 1. Entries under `## [Unreleased]`: they become `## [] — `. +// 2. `[Unreleased]` empty and the section for already written (a MINOR prepared +// over several release candidates, as 1.1.0 was). That section carries the date it was +// STARTED, not the date it ships, so the release restamps it with today. +// +// Anything else (an empty `[Unreleased]` above some other version) is left as it is and +// reported, because there is nothing for this release to promote. + +/** + * @param {string} changelog the CHANGELOG.md text + * @param {string} version the version being released (e.g. "1.1.0") + * @param {string} today YYYY-MM-DD + * @param {string} subtitle the italic line written under a newly promoted heading + * @returns {{ text: string, action: "promoted" | "redated" | "unchanged" | "nothing" }} + */ +export function promoteChangelog(changelog, version, today, subtitle) { + const empty = /## \[Unreleased\]\s*\n\s*\n## \[([^\]]+)\] — (\S+)/.exec(changelog); + if (empty === null) { + const text = changelog.replace(/## \[Unreleased\]\n/, `## [Unreleased]\n\n## [${version}] — ${today}\n\n${subtitle}\n`); + return { text, action: text === changelog ? "nothing" : "promoted" }; + } + const [, next, date] = empty; + if (next !== version) return { text: changelog, action: "nothing" }; + if (date === today) return { text: changelog, action: "unchanged" }; + const heading = `## [${version}] — ${date}`; + return { text: changelog.replace(heading, `## [${version}] — ${today}`), action: "redated" }; +} diff --git a/scripts/regen-showcase.ts b/scripts/regen-showcase.ts index c6469dcf4..cdd1c71e5 100644 --- a/scripts/regen-showcase.ts +++ b/scripts/regen-showcase.ts @@ -45,16 +45,20 @@ * Java and Kotlin arrive together because they have no standalone codegen CLI: on * the JVM, codegen runs in the build tool (docs/features/cli.md), so one * `mvn metaobjects:generate` over `examples/showcase/jvm/pom.xml` drives both. That - * pom names the plugin by VERSION rather than building it, so the JVM port needs the - * artifacts resolvable — from Maven Central at a released version, or from - * `cd server/java && mvn install` while the reactor sits on a -SNAPSHOT. + * pom names the plugin by VERSION rather than building it, so whatever Maven resolves + * at that version is what gets checked. At a release cut that version is not on Maven + * Central yet (npm publishes first), and `~/.m2` may hold a local install of the same + * version made from an older commit: the gate would then check stale plugin code. So + * the JVM leg first installs THIS tree's reactor into a throwaway local repository and + * resolves the plugin from there (`jvmRepository`); third-party artifacts still come + * from your usual local repository, read-only, as its tail. */ import { spawnSync } from "node:child_process"; import { cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, statSync, writeFileSync, } from "node:fs"; -import { tmpdir } from "node:os"; +import { homedir, tmpdir } from "node:os"; import { join, relative, resolve } from "node:path"; const REPO = resolve(import.meta.dirname, ".."); @@ -92,9 +96,10 @@ interface Port { generate(project: string): void; } -function run(argv: string[], opts: { cwd?: string } = {}): void { +function run(argv: string[], opts: { cwd?: string; env?: NodeJS.ProcessEnv } = {}): void { const r = spawnSync(argv[0]!, argv.slice(1), { cwd: opts.cwd ?? REPO, + env: opts.env ?? process.env, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"], }); @@ -211,11 +216,44 @@ const PORTS: Port[] = [ name: "jvm", dirs: ["java", "kotlin"], tool: "mvn", - generate: (project) => - run(["mvn", "-q", "metaobjects:generate"], { cwd: join(project, "jvm") }), + generate: (project) => { + const repo = jvmRepository(); + try { + run(["mvn", "-q", "-B", "metaobjects:generate", ...repo.args], { cwd: join(project, "jvm"), env: repo.env }); + } finally { + repo.cleanup(); + } + }, }, ]; +/** The `-Dmaven.repo.local=` token in a `MAVEN_ARGS` value, if any. */ +const REPO_LOCAL_ARG = /(?:^|\s)-Dmaven\.repo\.local=(\S+)/; + +/** + * A throwaway local Maven repository holding this tree's reactor, freshly installed. + * + * The head repository is empty apart from that install, so the plugin version the showcase + * pom names resolves to the code in this checkout and nothing else. Its tail is the + * repository you normally use (the one `MAVEN_ARGS` names, else `~/.m2/repository`), read + * only, so third-party artifacts are not downloaded again. `MAVEN_ARGS` is passed on without + * its own `-Dmaven.repo.local`, which would otherwise compete with ours. + */ +function jvmRepository(): { args: string[]; env: NodeJS.ProcessEnv; cleanup(): void } { + const mavenArgs = process.env.MAVEN_ARGS ?? ""; + const tail = REPO_LOCAL_ARG.exec(mavenArgs)?.[1] ?? join(homedir(), ".m2", "repository"); + const head = mkdtempSync(join(tmpdir(), "showcase-m2-")); + const args = [`-Dmaven.repo.local=${head}`, `-Dmaven.repo.local.tail=${tail}`]; + const env = { ...process.env, MAVEN_ARGS: mavenArgs.replace(REPO_LOCAL_ARG, " ").trim() }; + try { + run(["mvn", "-q", "-B", "-DskipTests", "install", ...args], { cwd: join(REPO, "server", "java"), env }); + } catch (e) { + rmSync(head, { recursive: true, force: true }); + throw e; + } + return { args, env, cleanup: () => rmSync(head, { recursive: true, force: true }) }; +} + /** * The generators whose output is COMMITTED under `examples/showcase/generated//`. * diff --git a/scripts/release.mjs b/scripts/release.mjs index ae481bdb9..04ef11d35 100644 --- a/scripts/release.mjs +++ b/scripts/release.mjs @@ -28,6 +28,7 @@ import { readFileSync, writeFileSync, mkdtempSync, mkdirSync, existsSync } from import { join, dirname } from "node:path"; import { tmpdir, homedir } from "node:os"; import { fileURLToPath } from "node:url"; +import { promoteChangelog } from "./lib/changelog-promote.mjs"; import { createInterface } from "node:readline/promises"; import { publishSet } from "./publish-set.mjs"; import { mavenVersion } from "./maven-coordinate.mjs"; @@ -205,18 +206,22 @@ const bad = Object.entries(pj.dependencies || {}).filter(([k, v]) => k.startsWit if (bad.length) die(`packed cli pins stale/workspace sibling deps: ${JSON.stringify(bad)}`); ok(`packed deps verified: all @metaobjectsdev/* pinned to ${VERSION}`); -// --- promote CHANGELOG [Unreleased] → [VERSION] --------------------------- +// --- promote CHANGELOG [Unreleased] → [VERSION], or date a prepared [VERSION] ---- +// The rule and its cases live in scripts/lib/changelog-promote.mjs (tested by +// scripts/test-changelog-promote.mjs): a section prepared ahead of the cut carries the date it +// was started, and is restamped with the release date here rather than by hand. const today = out("date +%F"); -let cl = readFileSync("CHANGELOG.md", "utf8"); -if (/## \[Unreleased\]\s*\n\s*\n## /.test(cl)) { - console.warn(`\x1b[33m! CHANGELOG [Unreleased] is empty — add an entry before releasing\x1b[0m`); +const promoted = promoteChangelog( + readFileSync("CHANGELOG.md", "utf8"), + VERSION, + today, + `_npm \`${VERSION}\` (full lockstep across all ${set.length} \`@metaobjectsdev/*\` publish candidates)._`, +); +if (promoted.action === "nothing") { + console.warn(`\x1b[33m! CHANGELOG has no entries and no [${VERSION}] section — add one before releasing\x1b[0m`); } else { - cl = cl.replace( - /## \[Unreleased\]\n/, - `## [Unreleased]\n\n## [${VERSION}] — ${today}\n\n_npm \`${VERSION}\` (full lockstep across all ${set.length} \`@metaobjectsdev/*\` publish candidates)._\n`, - ); - writeFileSync("CHANGELOG.md", cl); - ok(`CHANGELOG promoted → [${VERSION}]`); + writeFileSync("CHANGELOG.md", promoted.text); + ok(promoted.action === "promoted" ? `CHANGELOG promoted → [${VERSION}]` : `CHANGELOG [${VERSION}] dated ${today}`); } // --- refresh the site payload, which EMBEDS the versions Phase 1 just changed ------ diff --git a/scripts/test-changelog-promote.mjs b/scripts/test-changelog-promote.mjs new file mode 100644 index 000000000..be17b0756 --- /dev/null +++ b/scripts/test-changelog-promote.mjs @@ -0,0 +1,34 @@ +#!/usr/bin/env node +// Tests for scripts/lib/changelog-promote.mjs. +// +// node scripts/test-changelog-promote.mjs + +import { promoteChangelog } from "./lib/changelog-promote.mjs"; + +let fails = 0; +const check = (cond, label) => (cond ? console.log(`ok: ${label}`) : (console.error(`FAIL: ${label}`), fails++)); + +const SUB = "_npm `1.1.0`._"; +const withEntries = "# Changelog\n\n## [Unreleased]\n\n- a fix\n\n## [1.0.13] — 2026-10-03\n"; +const prepared = "# Changelog\n\n## [Unreleased]\n\n## [1.1.0] — 2026-10-10\n\n_npm `1.1.0`._\n\n## [1.0.13] — 2026-10-03\n"; + +let r = promoteChangelog(withEntries, "1.1.0", "2026-10-22", SUB); +check(r.action === "promoted", "entries under [Unreleased] are promoted"); +check(r.text.includes("## [Unreleased]\n\n## [1.1.0] — 2026-10-22\n\n_npm `1.1.0`._\n\n- a fix"), "the promoted section is dated today and keeps its entries"); + +r = promoteChangelog(prepared, "1.1.0", "2026-10-22", SUB); +check(r.action === "redated", "a prepared section for this version is redated"); +check(r.text.includes("## [1.1.0] — 2026-10-22") && !r.text.includes("2026-10-10"), "the prepared section carries today's date"); +check(r.text.includes("## [1.0.13] — 2026-10-03"), "older sections keep their dates"); + +r = promoteChangelog(prepared, "1.1.0", "2026-10-10", SUB); +check(r.action === "unchanged" && r.text === prepared, "a section already dated today is left alone"); + +r = promoteChangelog(prepared, "1.1.1", "2026-10-22", SUB); +check(r.action === "nothing" && r.text === prepared, "an empty [Unreleased] above another version promotes nothing"); + +if (fails > 0) { + console.error(`\n${fails} check(s) failed`); + process.exit(1); +} +console.log("\nall checks passed"); diff --git a/scripts/test-check-release-tag.mjs b/scripts/test-check-release-tag.mjs new file mode 100644 index 000000000..d0497ca6a --- /dev/null +++ b/scripts/test-check-release-tag.mjs @@ -0,0 +1,45 @@ +#!/usr/bin/env node +// Tests for scripts/check-release-tag.mjs. +// +// node scripts/test-check-release-tag.mjs + +import { readFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { PORTS, checkReleaseTag } from "./check-release-tag.mjs"; + +const REPO = join(dirname(fileURLToPath(import.meta.url)), ".."); +let fails = 0; +const check = (cond, label) => (cond ? console.log(`ok: ${label}`) : (console.error(`FAIL: ${label}`), fails++)); + +const run = (over) => checkReleaseTag({ port: "python", manifestVersion: "1.1.0", refType: "tag", refName: "python-v1.1.0", override: "", ...over }); + +check(run({}).ok, "a tag naming the manifest version passes"); +check(!run({ refName: "python-v1.1.1" }).ok, "a tag naming another version is refused"); +check(run({ refName: "python-v1.1.1" }).message.includes("1.1.0"), "the refusal names the manifest version"); +check(!run({ refName: "npm-v1.1.0" }).ok, "another port's tag is refused"); +check(!run({ override: "1.1.0-rc.7" }).ok, "a version override on a tag run is refused"); +check(run({ refType: "branch", refName: "main" }).ok, "a manual run has no tag to compare and passes"); +check(run({ refType: "branch", override: "1.1.0-rc.7" }).message.includes("1.1.0-rc.7"), "a manual run reports the override it publishes"); +check(!run({ manifestVersion: undefined }).ok, "a manifest with no version is refused"); +check(!checkReleaseTag({ port: "ruby", manifestVersion: "1", refType: "tag", refName: "ruby-v1" }).ok, "an unknown port is refused"); +check(checkReleaseTag({ port: "java", manifestVersion: "8.1.0", refType: "tag", refName: "java-v8.1.0" }).ok, "the Maven tag carries the Maven version"); + +// Every port's reader finds a version in the real manifest, so a manifest layout change +// fails here rather than at a release. +for (const [port, spec] of Object.entries(PORTS)) { + const v = spec.read(readFileSync(join(REPO, spec.file), "utf8")); + check(typeof v === "string" && /^\d+\.\d+\.\d+/.test(v), `${port}: reads a version from ${spec.file} (${v})`); +} +// The Java reader reads the reactor's own version: the poms' versions are checked equal by +// check-pom-versions.sh, and this must agree with the Maven derivation of the npm version. +const { mavenVersion } = await import("./maven-coordinate.mjs"); +const npm = PORTS.npm.read(readFileSync(join(REPO, PORTS.npm.file), "utf8")); +const java = PORTS.java.read(readFileSync(join(REPO, PORTS.java.file), "utf8")); +check(java === mavenVersion(npm) || java.endsWith("-SNAPSHOT"), `java reads the reactor version (${java}) for npm ${npm}`); + +if (fails > 0) { + console.error(`\n${fails} check(s) failed`); + process.exit(1); +} +console.log("\nall checks passed"); diff --git a/server/csharp/Directory.Build.props b/server/csharp/Directory.Build.props index 5f9fd9ffd..d48c70186 100644 --- a/server/csharp/Directory.Build.props +++ b/server/csharp/Directory.Build.props @@ -18,6 +18,8 @@ https://github.com/metaobjectsdev/metaobjects https://github.com/metaobjectsdev/metaobjects git + + Release notes: https://github.com/metaobjectsdev/metaobjects/blob/v$(Version)/CHANGELOG.md. Upgrading from 1.0.x: https://github.com/metaobjectsdev/metaobjects/blob/v$(Version)/docs/features/migrations/upgrading-1.0-to-1.1.md metaobjects;metadata;codegen;model-driven;cross-language diff --git a/server/csharp/MetaObjects.Codegen.Tests/CodegenCompileConformanceTests.cs b/server/csharp/MetaObjects.Codegen.Tests/CodegenCompileConformanceTests.cs index 25b0f7214..ecd5b1f0b 100644 --- a/server/csharp/MetaObjects.Codegen.Tests/CodegenCompileConformanceTests.cs +++ b/server/csharp/MetaObjects.Codegen.Tests/CodegenCompileConformanceTests.cs @@ -149,7 +149,7 @@ public void Every_generated_file_compiles_with_zero_errors(string selection, boo var files = generators.SelectMany(g => g.Generate(ctx)).ToList(); Assert.True(files.Count > 0, $"selection '{selection}' generated no files at all"); - // FR-044 — every view-backed report of the corpus (twenty-three today) generates a keyless + // FR-044 — every view-backed report of the corpus (twenty-five today) generates a keyless // row class and a filter allowlist (their routes file is in the excluded framework tier), // and no names artifact. Named, because a compile gate passes trivially over a file that // was never emitted. ProgramRoster, ProgramLongWeeks and FitnessTotalsFilled carry diff --git a/server/csharp/MetaObjects.Codegen.Tests/DecimalWireTests.cs b/server/csharp/MetaObjects.Codegen.Tests/DecimalWireTests.cs new file mode 100644 index 000000000..a6cff1b82 --- /dev/null +++ b/server/csharp/MetaObjects.Codegen.Tests/DecimalWireTests.cs @@ -0,0 +1,276 @@ +// ADR-0019 Amendment 1 (W1-W4) — the C# half of the decimal wire rule, gated against the +// shared vectors in fixtures/decimal-wire/vectors.json. No database: the production wire +// helper (Runtime/DecimalWire.cs, the SAME text the entity generator emits into every +// generated project) runs every vector, and the emitted text is proven identical to it. +// The HTTP bytes and the read path are gated by the api-contract and persistence lanes. + +using System.Globalization; +using System.Reflection; +using System.Text.Json; +using System.Text.Json.Serialization; +using Microsoft.CodeAnalysis; +using Microsoft.CodeAnalysis.CSharp; +using MetaObjects.Codegen.Generators; +using MetaObjects.Codegen.Runtime; +using MetaObjects.Loader; +using MetaObjects.Meta; +using Xunit; + +namespace MetaObjects.Codegen.Tests; + +public class DecimalWireTests +{ + private static JsonElement Vectors() => + JsonDocument.Parse(File.ReadAllText(Path.Combine( + CorpusPaths.RepoRoot(), "fixtures", "decimal-wire", "vectors.json"))).RootElement; + + private static decimal Exact(string s) => + decimal.Parse(s, NumberStyles.Float, CultureInfo.InvariantCulture); + + public static TheoryData Section(string name) + { + var data = new TheoryData(); + foreach (var v in Vectors().GetProperty(name).EnumerateArray()) data.Add(v.GetRawText()); + return data; + } + + public static TheoryData FormatVectors => Section("format"); + public static TheoryData ParseVectors => Section("parse"); + public static TheoryData OperandVectors => Section("filterOperand"); + public static TheoryData Double15Vectors => Section("double15"); + + public sealed class Holder + { + [JsonConverter(typeof(DecimalWire.DecimalJsonConverter))] + public decimal Amount { get; set; } + + [JsonConverter(typeof(DecimalWire.NullableDecimalJsonConverter))] + public decimal? Maybe { get; set; } + + [JsonConverter(typeof(DecimalWire.DecimalListJsonConverter))] + public IList Many { get; set; } = new List(); + } + + [Theory, MemberData(nameof(FormatVectors))] + public void W1_format(string vector) + { + var v = JsonDocument.Parse(vector).RootElement; + Assert.Equal(v.GetProperty("out").GetString(), DecimalWire.Format(Exact(v.GetProperty("in").GetString()!))); + } + + [Theory, MemberData(nameof(FormatVectors))] + public void W1_a_property_serializes_the_canonical_string_under_any_host_options(string vector) + { + var v = JsonDocument.Parse(vector).RootElement; + var want = v.GetProperty("out").GetString()!; + // Host options that would otherwise write a number, and one that would try to read strings as numbers. + var host = new JsonSerializerOptions { NumberHandling = JsonNumberHandling.AllowReadingFromString }; + var json = JsonSerializer.Serialize(new Holder { Amount = Exact(v.GetProperty("in").GetString()!), Maybe = null }, host); + Assert.Equal($"{{\"Amount\":\"{want}\",\"Maybe\":null,\"Many\":[]}}", json); + } + + [Theory, MemberData(nameof(ParseVectors))] + public void W2_body_parse(string vector) + { + var v = JsonDocument.Parse(vector).RootElement; + var json = v.GetProperty("json").GetString()!; + var body = $"{{\"Amount\":{(json == "null" ? "0" : json)},\"Maybe\":{json}}}"; + if (v.TryGetProperty("error", out var err) && err.GetBoolean()) + { + // The generated routes turn this exception into 400 {"error":"validation"}. + Assert.ThrowsAny(() => JsonSerializer.Deserialize(body)); + return; + } + var parsed = JsonSerializer.Deserialize(body)!; + var value = v.GetProperty("value"); + if (value.ValueKind == JsonValueKind.Null) Assert.Null(parsed.Maybe); + else + { + Assert.Equal(Exact(value.GetString()!), parsed.Maybe); + Assert.Equal(Exact(value.GetString()!), parsed.Amount); + } + } + + [Theory, MemberData(nameof(OperandVectors))] + public void W2_filter_operand(string vector) + { + var v = JsonDocument.Parse(vector).RootElement; + var ok = DecimalWire.TryParse(v.GetProperty("text").GetString(), out var got); + if (v.TryGetProperty("error", out var err) && err.GetBoolean()) { Assert.False(ok); return; } + Assert.True(ok); + Assert.Equal(Exact(v.GetProperty("value").GetString()!), got); + } + + [Theory, MemberData(nameof(Double15Vectors))] + public void W4_double_rendered_at_15_significant_digits(string vector) + { + var v = JsonDocument.Parse(vector).RootElement; + var d = double.Parse(v.GetProperty("double").GetString()!, NumberStyles.Float, CultureInfo.InvariantCulture); + Assert.Equal(v.GetProperty("out").GetString(), DecimalWire.FormatDouble15(d)); + } + + [Theory] + [InlineData("\"79228162514264337593543950336\"")] // decimal.MaxValue + 1 + [InlineData("\"1e30\"")] + [InlineData("\"1e-30\"")] // would round to 0, never silently + [InlineData("\"0.12345678901234567890123456789012\"")] // 32 fractional digits + [InlineData("1e999999999999")] + public void W3_a_value_decimal_cannot_hold_is_refused_not_rounded(string json) => + Assert.ThrowsAny(() => JsonSerializer.Deserialize($"{{\"Amount\":{json}}}")); + + [Theory] + [InlineData("0.1234567890", 28, 10, true)] + [InlineData("0.12345678901", 28, 10, false)] + [InlineData("12345678901234567.8901234567", 28, 10, true)] + [InlineData("1234567890123456789", 28, 10, false)] + [InlineData("-0.5", 1, 1, true)] + [InlineData("1.5", 1, 1, false)] + [InlineData("100.2500", 5, 2, true)] + [InlineData("1000", 5, 2, false)] + public void W3_a_declared_precision_and_scale_bound_the_input(string text, int precision, int scale, bool fits) + { + Assert.True(DecimalWire.TryParse(text, out var value)); + Assert.Equal(fits, DecimalWire.Fits(value, precision, scale)); + } + + [Fact] + public void W3_the_28_digit_edge_is_accepted() + { + var h = JsonSerializer.Deserialize("{\"Amount\":\"1234567890123456789012345678\"}")!; + Assert.Equal(1234567890123456789012345678m, h.Amount); + Assert.Equal(decimal.MaxValue, JsonSerializer.Deserialize("{\"Amount\":\"79228162514264337593543950335\"}")!.Amount); + } + + [Fact] + public void An_array_of_decimals_crosses_as_canonical_strings() + { + var h = JsonSerializer.Deserialize("{\"Many\":[\"1.50\",2,\"0.10\"]}")!; + Assert.Equal(new[] { 1.5m, 2m, 0.1m }, h.Many); + Assert.Contains("\"Many\":[\"1.5\",\"2\",\"0.1\"]", JsonSerializer.Serialize(h)); + Assert.ThrowsAny(() => JsonSerializer.Deserialize("{\"Many\":[\"abc\"]}")); + } + + public sealed class Row { public decimal Amount { get; set; } public int Count { get; set; } } + + private static FilterParseResult Check(string field, string op, object value) => + EfCoreFilterDispatch.Validate(FilterParseResult.Ok([new FilterPredicate(field, op, value)])); + + [Theory] + [InlineData("amount", "eq", "abc")] + [InlineData("amount", "gt", " 1")] + [InlineData("amount", "lt", "0x10")] + [InlineData("amount", "eq", "")] + [InlineData("amount", "eq", "+1")] + [InlineData("amount", "eq", "1e30")] // outside decimal: refused, not rounded + [InlineData("count", "eq", "abc")] // every typed operand shares the one envelope + public void A_malformed_operand_is_invalid_filter_value_naming_the_field(string field, string op, string operand) + { + var r = Check(field, op, operand); + Assert.Equal("invalid_filter_value", r.ErrorEnvelope); + Assert.Equal(field, r.Field); + } + + [Fact] + public void A_good_operand_passes_and_an_in_list_is_checked_element_wise() + { + Assert.Null(Check("amount", "eq", "12345678901234567.89").ErrorEnvelope); + Assert.Null(Check("amount", "gte", "1e-7").ErrorEnvelope); + Assert.Null(Check("amount", "in", new List { "1", "2.50" }).ErrorEnvelope); + Assert.Equal("invalid_filter_value", Check("amount", "in", new List { "1", "x" }).ErrorEnvelope); + } + + [Fact] + public void A_driver_decimal_overflow_anywhere_in_the_chain_is_recognized() + { + var overflow = new OverflowException("Value was either too large or too small for a Decimal."); + Assert.True(DecimalRangeEndpointFilter.IsDecimalOverflow(new InvalidOperationException("wrapped", overflow))); + Assert.False(DecimalRangeEndpointFilter.IsDecimalOverflow(new OverflowException("Arithmetic operation resulted in an overflow."))); + Assert.False(DecimalRangeEndpointFilter.IsDecimalOverflow(new InvalidOperationException("boom"))); + } + + // ---- the emitted copy IS the production helper ------------------------------------------- + + + private static IReadOnlyList EmitFitness(out MetaRoot root) + { + var result = new MetaDataLoader().Load([new FileSource(CorpusPaths.FitnessMetadata)]); + Assert.Empty(result.Errors); + root = result.Root; + root.PhysicalNaming = CorpusPaths.Naming; + var ctx = new GenContext + { + Entities = root.Objects(), + Root = root, + Config = new GenConfig + { + OutDir = Path.Combine(Path.GetTempPath(), "unused-decimal-wire"), + Namespace = "Decimal.Wire.Generated", + ColumnNamingStrategy = CorpusPaths.Naming.Columns, + EmitAbstractShapes = false, + }, + }; + return new IGenerator[] { new EntityGenerator(), new DbContextGenerator(), new FilterAllowlistGenerator(), new CallableGenerator() } + .SelectMany(g => g.Generate(ctx)).ToList(); + } + + [Fact] + public void The_emitted_helper_is_the_production_helper_with_only_the_namespace_changed() + { + var files = EmitFitness(out _); + var emitted = files.Where(f => f.Path == "DecimalWire.g.cs").ToList(); + Assert.NotEmpty(emitted); + var production = File.ReadAllText(Path.Combine( + CorpusPaths.RepoRoot(), "server", "csharp", "MetaObjects.Codegen", "Runtime", "DecimalWire.cs")).Replace("\r\n", "\n"); + foreach (var f in emitted) + { + var ns = f.Content.Split('\n').First(l => l.StartsWith("namespace ", StringComparison.Ordinal)); + Assert.Equal( + production.Replace("namespace MetaObjects.Codegen.Runtime;", ns), + f.Content.Replace("\r\n", "\n")); + Assert.DoesNotContain("MetaObjects", ns.Replace("namespace ", ""), StringComparison.Ordinal); + } + } + + [Fact] + public void Every_generated_decimal_property_carries_its_converter_and_the_output_serializes_canonically() + { + var files = EmitFitness(out _); + // No generated entity code may import MetaObjects at runtime (CLAUDE.md). + foreach (var f in files.Where(f => f.Path.EndsWith(".g.cs", StringComparison.Ordinal) && !f.Path.Contains("Routes"))) + if (f.Content.Contains("DecimalWire.", StringComparison.Ordinal)) + Assert.DoesNotContain("MetaObjects.Codegen", f.Content); + + var trees = files.Select(f => CSharpSyntaxTree.ParseText(f.Content, new CSharpParseOptions(LanguageVersion.CSharp12), path: f.Path)); + var comp = CSharpCompilation.Create("decimal_wire_" + Guid.NewGuid().ToString("N"), trees, + DbContextCompileTests.BuildReferences(), new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary)); + using var ms = new MemoryStream(); + var emit = comp.Emit(ms); + Assert.True(emit.Success, string.Join("\n", emit.Diagnostics.Where(d => d.Severity == DiagnosticSeverity.Error))); + var asm = Assembly.Load(ms.ToArray()); + + var decimals = asm.GetTypes().SelectMany(t => t.GetProperties(BindingFlags.Public | BindingFlags.Instance | BindingFlags.DeclaredOnly)) + .Where(p => p.PropertyType == typeof(decimal) || p.PropertyType == typeof(decimal?) + || p.PropertyType == typeof(IList)).ToList(); + Assert.NotEmpty(decimals); + foreach (var p in decimals) + Assert.True(p.GetCustomAttribute() is { } a + && (a.ConverterType?.DeclaringType?.Name == "DecimalWire" || a.GetType().DeclaringType?.Name == "DecimalWire"), + $"{p.DeclaringType!.Name}.{p.Name} has no per-property DecimalWire converter"); + + // A real generated entity: AllTypes.decVal is a @required field.decimal(18,6). + var allTypes = asm.GetTypes().Single(t => t.Name == "AllTypes"); + var instance = Activator.CreateInstance(allTypes)!; + allTypes.GetProperty("DecVal")!.SetValue(instance, 12.500000m); + var json = JsonSerializer.Serialize(instance, allTypes, new JsonSerializerOptions { NumberHandling = JsonNumberHandling.AllowReadingFromString }); + Assert.Contains("\"DecVal\":\"12.5\"", json); + Assert.Throws(() => JsonSerializer.Deserialize("{\"DecVal\":\"abc\"}", allTypes)); + var back = JsonSerializer.Deserialize("{\"DecVal\":\"123456789012.89\"}", allTypes)!; + Assert.Equal(123456789012.89m, allTypes.GetProperty("DecVal")!.GetValue(back)); + + // W3: decVal is decimal(18,6). Seven fraction digits, or thirteen integer digits, are refused, never rounded. + Assert.Throws(() => JsonSerializer.Deserialize("{\"DecVal\":\"0.1234567\"}", allTypes)); + Assert.Throws(() => JsonSerializer.Deserialize("{\"DecVal\":\"1234567890123\"}", allTypes)); + var edge = JsonSerializer.Deserialize("{\"DecVal\":\"123456789012.123456\"}", allTypes)!; + Assert.Equal(123456789012.123456m, allTypes.GetProperty("DecVal")!.GetValue(edge)); + } +} diff --git a/server/csharp/MetaObjects.Codegen.Tests/EntityGeneratorTests.cs b/server/csharp/MetaObjects.Codegen.Tests/EntityGeneratorTests.cs index 8f3844862..6fcc8c493 100644 --- a/server/csharp/MetaObjects.Codegen.Tests/EntityGeneratorTests.cs +++ b/server/csharp/MetaObjects.Codegen.Tests/EntityGeneratorTests.cs @@ -109,6 +109,10 @@ public void Routes_generator_emits_crud_endpoints() Assert.Contains("public static class SubscriberRoutes", src); Assert.Contains("public static IEndpointRouteBuilder MapSubscriberRoutes(this IEndpointRouteBuilder app, string prefix = \"/api\")", src); + // ADR-0019 Amendment 1 (W3): the routes mount on a group carrying the decimal-range filter. + Assert.Contains("IEndpointRouteBuilder mount = app;", src); + Assert.Contains("app = app.MapGroup(string.Empty).AddEndpointFilter(new DecimalRangeEndpointFilter());", src); + Assert.Contains("return mount;", src); // GET list now takes HttpContext (for qs parsing: limit/offset/sort/withCount). Assert.Contains("app.MapGet(prefix + \"/subscribers\", async (HttpContext http, AppDbContext db) =>", src); Assert.Contains("app.MapGet(prefix + \"/subscribers/{id}\", async (long id, AppDbContext db) =>", src); diff --git a/server/csharp/MetaObjects.Codegen.Tests/ExtendsAbstractFieldInheritanceCodegenTests.cs b/server/csharp/MetaObjects.Codegen.Tests/ExtendsAbstractFieldInheritanceCodegenTests.cs index 59e109070..e6439c4c7 100644 --- a/server/csharp/MetaObjects.Codegen.Tests/ExtendsAbstractFieldInheritanceCodegenTests.cs +++ b/server/csharp/MetaObjects.Codegen.Tests/ExtendsAbstractFieldInheritanceCodegenTests.cs @@ -64,7 +64,7 @@ public void Generated_entity_honors_every_inherited_field_property() // `tags` extends the abstract array field.string(isArray:true, @maxLength:40): // array-ness (ICollection, NOT a scalar string) must be inherited. - Assert.Contains("ICollection Tags", contact); + Assert.Contains("IList Tags", contact); Assert.DoesNotContain("public string Tags ", contact); // `addresses` extends field.object(isArray:true, @objectRef:Address, @storage:jsonb): diff --git a/server/csharp/MetaObjects.Codegen.Tests/ExtractorCodegenTests.cs b/server/csharp/MetaObjects.Codegen.Tests/ExtractorCodegenTests.cs index 931a97b28..ee569f988 100644 --- a/server/csharp/MetaObjects.Codegen.Tests/ExtractorCodegenTests.cs +++ b/server/csharp/MetaObjects.Codegen.Tests/ExtractorCodegenTests.cs @@ -167,10 +167,10 @@ public void Generated_extract_populates_strict_graph_from_dirty_text() var tags = ((IEnumerable)order.GetType().GetProperty("Tags")!.GetValue(order)!).Cast().ToList(); Assert.Equal(new object[] { "x", "y" }, tags.ToArray()); - // required NON-STRING scalar-array (int[]) populates as ICollection — proves the + // required NON-STRING scalar-array (int[]) populates as IList — proves the // kind-typed mirror element (IReadOnlyList?) narrows to the strict int element. var scoresProp = order.GetType().GetProperty("Scores")!; - Assert.Equal(typeof(ICollection), scoresProp.PropertyType); + Assert.Equal(typeof(IList), scoresProp.PropertyType); var scores = ((IEnumerable)scoresProp.GetValue(order)!).Cast().ToList(); Assert.Equal(new object[] { 3, 7 }, scores.ToArray()); @@ -178,21 +178,21 @@ public void Generated_extract_populates_strict_graph_from_dirty_text() // narrows the kind-typed mirror (IReadOnlyList?) to the strict long element. The // CS8619-as-error compile gate would have failed had the element type mismatched. var quantitiesProp = order.GetType().GetProperty("Quantities")!; - Assert.Equal(typeof(ICollection), quantitiesProp.PropertyType); + Assert.Equal(typeof(IList), quantitiesProp.PropertyType); var quantities = ((IEnumerable)quantitiesProp.GetValue(order)!).Cast().ToList(); Assert.Equal(new object[] { 10L, 9000000000L }, quantities.ToArray()); Assert.IsType(quantities[0]); // required NON-STRING scalar-array (double[]) — proves the double `x!.Value` unwrap. var weightsProp = order.GetType().GetProperty("Weights")!; - Assert.Equal(typeof(ICollection), weightsProp.PropertyType); + Assert.Equal(typeof(IList), weightsProp.PropertyType); var weights = ((IEnumerable)weightsProp.GetValue(order)!).Cast().ToList(); Assert.Equal(new object[] { 1.5d, 2.25d }, weights.ToArray()); Assert.IsType(weights[0]); // required NON-STRING scalar-array (bool[]) — proves the bool `x!.Value` unwrap. var flagsProp = order.GetType().GetProperty("FlagsArr")!; - Assert.Equal(typeof(ICollection), flagsProp.PropertyType); + Assert.Equal(typeof(IList), flagsProp.PropertyType); var flags = ((IEnumerable)flagsProp.GetValue(order)!).Cast().ToList(); Assert.Equal(new object[] { true, false, true }, flags.ToArray()); Assert.IsType(flags[0]); @@ -369,7 +369,7 @@ public void Poco_types_enum_as_nested_enum_lenient_mirror_stays_string() Assert.Contains("public enum OrderPriority { LOW, HIGH }", poco); Assert.Contains("public enum OrderLabels { A, B }", poco); Assert.Contains("public OrderPriority Priority { get; set; }", poco); - Assert.Contains("public ICollection Labels { get; set; } = new List();", poco); + Assert.Contains("public IList Labels { get; set; } = new List();", poco); // The extract mapper coerces the string mirror via System.Enum.Parse. var extractorSrc = Assert.Single(new ExtractorGenerator().Generate(Ctx(root))).Content; diff --git a/server/csharp/MetaObjects.Codegen.Tests/Issue195NativeTypingCodegenTests.cs b/server/csharp/MetaObjects.Codegen.Tests/Issue195NativeTypingCodegenTests.cs index dbc13d259..9eaac8799 100644 --- a/server/csharp/MetaObjects.Codegen.Tests/Issue195NativeTypingCodegenTests.cs +++ b/server/csharp/MetaObjects.Codegen.Tests/Issue195NativeTypingCodegenTests.cs @@ -78,7 +78,7 @@ public void Collect_projection_field_is_a_non_null_collection() { var cls = ProjectionClass(); // collect → COALESCE([]) → non-null List (never a nullable collection). - Assert.Contains("public ICollection Labels { get; set; } = new List();", cls); + Assert.Contains("public IList Labels { get; set; } = new List();", cls); } [Fact] diff --git a/server/csharp/MetaObjects.Codegen.Tests/OrphanSweepTests.cs b/server/csharp/MetaObjects.Codegen.Tests/OrphanSweepTests.cs index 91fdd6b2f..00e60621a 100644 --- a/server/csharp/MetaObjects.Codegen.Tests/OrphanSweepTests.cs +++ b/server/csharp/MetaObjects.Codegen.Tests/OrphanSweepTests.cs @@ -9,6 +9,7 @@ // touched; a run narrowed by --generators sweeps nothing. using System.Text.Json; +using MetaObjects.Codegen.Generators; using MetaObjects.Meta; using Xunit; @@ -105,6 +106,84 @@ public void A_renamed_value_object_is_deleted_and_verify_goes_green() Assert.Equal(0, Verify().Code); } + /// One package, one entity and (optionally) one live L4 requirement over it. + private void RequirementModel(bool withRequirement) + { + var children = new List + { + new Dictionary + { + ["object.entity"] = new Dictionary + { + ["name"] = "Order", + ["children"] = new object[] + { + new Dictionary { ["field.uuid"] = new Dictionary { ["name"] = "id" } }, + new Dictionary + { + ["identity.primary"] = new Dictionary + { + ["name"] = "pk", ["@fields"] = new[] { "id" }, + }, + }, + }, + }, + }, + }; + if (withRequirement) + children.Add(new Dictionary + { + ["requirement.functional"] = new Dictionary + { + ["name"] = "Orders", + ["@level"] = 4, + ["@status"] = "live", + ["@statement"] = "An order is recorded when it is placed.", + ["@counterexample"] = "A placed order has no row.", + ["@implementedBy"] = new[] { "Order" }, + }, + }); + File.WriteAllText(Path.Combine(Meta, "meta.acme.json"), JsonSerializer.Serialize(new Dictionary + { + ["metadata.root"] = new Dictionary { ["package"] = "acme", ["children"] = children }, + })); + } + + [Fact] + public void Requirement_tests_are_deleted_once_their_package_has_no_requirements_left() + { + var testsDir = Path.Combine(_root, "tests"); + _suite = [new RequirementTestsGenerator { OutDir = testsDir, WarnUncovered = false }]; + RequirementModel(withRequirement: true); + Assert.Equal(0, Gen().Code); + var files = Directory.GetFiles(testsDir, "*.g.cs").Select(Path.GetFileName).Order(StringComparer.Ordinal).ToList(); + Assert.Equal(["Requirements_acme_Tests.g.cs", "Requirements_acme_Witnesses.g.cs"], files); + + RequirementModel(withRequirement: false); + var gen = Gen(); + Assert.Equal(0, gen.Code); + Assert.Empty(Directory.GetFiles(testsDir, "*.g.cs")); + Assert.Contains("removed: ", gen.Out); + Assert.DoesNotContain(Manifest().Keys, k => k.Contains("Requirements_acme_", StringComparison.Ordinal)); + } + + [Fact] + public void An_edited_requirement_test_file_is_refused_when_its_package_has_no_requirements_left() + { + var testsDir = Path.Combine(_root, "tests"); + _suite = [new RequirementTestsGenerator { OutDir = testsDir, WarnUncovered = false }]; + RequirementModel(withRequirement: true); + Assert.Equal(0, Gen().Code); + var tests = Path.Combine(testsDir, "Requirements_acme_Tests.g.cs"); + File.AppendAllText(tests, "// mine\n"); + + RequirementModel(withRequirement: false); + var gen = Gen(); + Assert.True(File.Exists(tests)); + Assert.False(File.Exists(Path.Combine(testsDir, "Requirements_acme_Witnesses.g.cs"))); + Assert.Contains("Requirements_acme_Tests.g.cs", gen.Err + gen.Out); + } + [Fact] public void A_hand_edited_orphan_is_refused_and_verify_says_delete_it() { diff --git a/server/csharp/MetaObjects.Codegen.Tests/ReportRowCodegenTests.cs b/server/csharp/MetaObjects.Codegen.Tests/ReportRowCodegenTests.cs index 4ca613601..0818ab855 100644 --- a/server/csharp/MetaObjects.Codegen.Tests/ReportRowCodegenTests.cs +++ b/server/csharp/MetaObjects.Codegen.Tests/ReportRowCodegenTests.cs @@ -86,6 +86,10 @@ private static string Report(string name, string sourceBody, string dims = "", s + "\"@dimensions\": [" + dims + "], \"@measures\": [" + measures + "]" + children + " } }"; } + private static LoadResult TryLoad(params string[] reports) => + new MetaDataLoader().Load( + [new InMemoryStringSource(ModelTemplate.Replace("<>", string.Concat(reports)), id: "meta.shop.json")]); + private static MetaRoot Load(params string[] reports) { var json = ModelTemplate.Replace("<>", string.Concat(reports)); @@ -161,15 +165,11 @@ public void A_view_backed_report_generates_its_row_and_mapping(string what, stri public static TheoryData InertSources => new() { { "no source", "" }, - // The lowering skips these kinds, so no relation with the Table B columns is promised. - { "a materialized view", "\"@kind\": \"materializedView\", \"@materializedView\": \"mv_sales\"" }, - { "a stored procedure", "\"@kind\": \"storedProc\", \"@proc\": \"sp_sales\"" }, - { "a table function", "\"@kind\": \"tableFunction\", \"@function\": \"fn_sales\"" }, }; [Theory] [MemberData(nameof(InertSources))] - public void A_report_that_is_not_a_view_generates_nothing(string what, string source) + public void A_sourceless_report_generates_nothing(string what, string source) { var with = EmitAll(RunnerContext(Load(Report("SalesTotal", source)))); var without = EmitAll(RunnerContext(Load())); @@ -179,6 +179,19 @@ public void A_report_that_is_not_a_view_generates_nothing(string what, string so Assert.True(content == with[path], $"{what}: {path} changed"); } + [Theory] + [InlineData("\"@kind\": \"materializedView\", \"@materializedView\": \"mv_sales\"", "materializedView")] + [InlineData("\"@kind\": \"storedProc\", \"@proc\": \"sp_sales\"", "storedProc")] + [InlineData("\"@kind\": \"tableFunction\", \"@function\": \"fn_sales\"", "tableFunction")] + public void A_report_over_a_source_that_is_not_a_view_is_refused_at_load(string source, string kind) + { + // R10: a report is read through a view, so the loader refuses the other read-only kinds + // before any generator sees them. + var error = Assert.Single(TryLoad(Report("SalesTotal", source)).Errors); + Assert.Equal(ErrorCode.ERR_INVALID_REPORT, error.Code); + Assert.Contains($"has @kind '{kind}'; a report is read through a view", error.Message); + } + [Fact] public void A_context_built_from_the_unfiltered_root_generates_the_same_files() { @@ -246,7 +259,7 @@ public void A_served_report_gets_a_routes_file_with_the_list_route_and_a_405_and Assert.DoesNotContain("projection", routes); // It reads the report's own DbSet and names the report's own allowlist. Assert.Contains("IQueryable q = db.StoreTotals.AsNoTracking();", routes); - Assert.Contains("FilterParser.Parse(qs, StoreTotalsFilterAllowlist.Fields, StoreTotalsFilterAllowlist.OpsByField);", routes); + Assert.Contains("EfCoreFilterDispatch.Validate(FilterParser.Parse(qs, StoreTotalsFilterAllowlist.Fields, StoreTotalsFilterAllowlist.OpsByField));", routes); // Every derived scalar is sortable. foreach (var field in new[] { "Purchases", "Buyers", "Revenue" }) Assert.Contains($" \"{field}\",", routes); @@ -795,54 +808,39 @@ public void A_sourceless_report_with_a_colliding_name_is_not_refused() ] } } """; - private static MetaRoot LoadObjectDimension(bool viewBacked) + private static LoadResult TryLoadObjectDimension(bool viewBacked) { string source = viewBacked ? ", \"children\": [ { \"source.rdb\": { \"@kind\": \"view\", \"@view\": \"v_by_destination\" } } ]" : ""; - var result = new MetaDataLoader().Load( + return new MetaDataLoader().Load( [new InMemoryStringSource(ObjectDimensionModel.Replace("<>", source), id: "meta.shop.json")]); - Assert.True(result.Errors.Count == 0, - "model did not load:\n" + string.Join("\n", result.Errors.Select(e => $" {e.Code}: {e.Message}"))); - return result.Root; } - [Fact] - public void A_dimension_over_a_field_object_is_refused_naming_the_report_and_the_dimension() - { - // The loader accepts it. Left alone, the row class silently has no property for - // the dimension, so the report would read without the column it groups by. - var root = LoadObjectDimension(viewBacked: true); - foreach (var generator in new IGenerator[] { new EntityGenerator(), new DbContextGenerator() }) - { - var ex = Assert.Throws(() => generator.Generate(RunnerContext(root)).ToList()); - Assert.Equal( - "report \"SalesByDestination\": its dimension \"destination\" reads \"acme::shop::Sale.shipTo\", " + - "a field.object. A report over a field.object is not supported; group by a scalar field.", - ex.Message); - } - } - - [Fact] - public void A_sourceless_report_over_a_field_object_generates_nothing_and_is_not_refused() + [Theory] + [InlineData(true)] + [InlineData(false)] + public void A_dimension_over_a_field_object_is_refused_at_load_naming_the_dimension(bool viewBacked) { - var files = EmitAll(RunnerContext(LoadObjectDimension(viewBacked: false))); - Assert.DoesNotContain(files.Keys, k => k.Contains("SalesByDestination", StringComparison.Ordinal)); + // D5: left alone, the row class silently had no property for the dimension, so the + // report would read without the column it groups by. The loader refuses it, with or + // without a source. + var error = Assert.Single(TryLoadObjectDimension(viewBacked).Errors); + Assert.Equal(ErrorCode.ERR_INVALID_DIMENSION, error.Code); + Assert.Contains("dimension 'destination'", error.Message); + Assert.Contains("a report cannot group by", error.Message); } [Fact] - public void An_abstract_view_backed_report_generates_nothing() + public void An_abstract_report_is_refused_at_load() { - // An abstract object gets no class in this port, report or not. The TypeScript, - // Java and Python runtimes still read the view (docs/features/reporting.md, Known limits). + // R11: an abstract object gets no class in this port, and no port reads or serves an + // abstract report, so the loader refuses it. string report = Report("SalesTotal", "\"@kind\": \"view\", \"@view\": \"v_sales\"") .Replace("\"name\": \"SalesTotal\",", "\"name\": \"SalesTotal\", \"abstract\": true,"); Assert.Contains("\"abstract\": true", report); - var with = EmitAll(RunnerContext(Load(report))); - var without = EmitAll(RunnerContext(Load())); - - Assert.Equal(without.Keys.OrderBy(k => k).ToList(), with.Keys.OrderBy(k => k).ToList()); - foreach (var (path, content) in without) - Assert.True(content == with[path], $"{path} changed"); + var error = Assert.Single(TryLoad(report).Errors); + Assert.Equal(ErrorCode.ERR_INVALID_REPORT, error.Code); + Assert.Contains("a report cannot be abstract", error.Message); } } diff --git a/server/csharp/MetaObjects.Codegen.Tests/ScalarArrayCodegenTests.cs b/server/csharp/MetaObjects.Codegen.Tests/ScalarArrayCodegenTests.cs index d078e87c4..c288cde94 100644 --- a/server/csharp/MetaObjects.Codegen.Tests/ScalarArrayCodegenTests.cs +++ b/server/csharp/MetaObjects.Codegen.Tests/ScalarArrayCodegenTests.cs @@ -92,7 +92,7 @@ private static MetaRoot Load(string json) }; // ------------------------------------------------------------------------- - // EntityGenerator — scalar array → ICollection + // EntityGenerator — scalar array → IList // ------------------------------------------------------------------------- [Fact] @@ -101,10 +101,10 @@ public void Scalar_array_field_emits_collection_property_with_initializer() var ctx = Ctx(Load(ScalarArrayModel)); var src = Assert.Single(new EntityGenerator().Generate(ctx)).Content; - // ICollection with an explicit List initializer (uniform with + // IList with an explicit List initializer (uniform with // the M:N-nav and object-isArray emission paths; the property type is the // wider interface so adopters can assign string[]/HashSet/etc.). - Assert.Contains("public ICollection Tags { get; set; } = new List();", src); + Assert.Contains("public IList Tags { get; set; } = new List();", src); // Guard: the scalar form must NOT appear. Assert.DoesNotContain("public string? Tags", src); Assert.DoesNotContain("public string Tags", src); @@ -121,7 +121,7 @@ public void Scalar_array_field_still_carries_column_attribute() } // ------------------------------------------------------------------------- - // EntityGenerator — enum array → ICollection + // EntityGenerator — enum array → IList // ------------------------------------------------------------------------- [Fact] @@ -130,8 +130,8 @@ public void Enum_array_field_emits_collection_of_enum_property() var ctx = Ctx(Load(EnumArrayModel)); var src = Assert.Single(new EntityGenerator().Generate(ctx)).Content; - // ICollection with an explicit List initializer. - Assert.Contains("public ICollection Statuses { get; set; } = new List();", src); + // IList with an explicit List initializer. + Assert.Contains("public IList Statuses { get; set; } = new List();", src); // Guard: the scalar form must NOT appear. Assert.DoesNotContain("public OrderStatuses? Statuses", src); } @@ -216,8 +216,8 @@ public void Uuid_array_field_emits_ICollection_Guid_property() var ctx = Ctx(Load(UuidArrayModel)); var src = Assert.Single(new EntityGenerator().Generate(ctx)).Content; - // field.uuid isArray:true → ICollection with List initializer. - Assert.Contains("public ICollection Refs { get; set; } = new List();", src); + // field.uuid isArray:true → IList with List initializer. + Assert.Contains("public IList Refs { get; set; } = new List();", src); // Guard: the scalar form must NOT appear. Assert.DoesNotContain("public Guid? Refs", src); Assert.DoesNotContain("public Guid Refs", src); diff --git a/server/csharp/MetaObjects.Codegen/ApiDocs/CSharpApiModelBuilder.cs b/server/csharp/MetaObjects.Codegen/ApiDocs/CSharpApiModelBuilder.cs index c9907544d..28ff94044 100644 --- a/server/csharp/MetaObjects.Codegen/ApiDocs/CSharpApiModelBuilder.cs +++ b/server/csharp/MetaObjects.Codegen/ApiDocs/CSharpApiModelBuilder.cs @@ -300,6 +300,7 @@ private static IReadOnlyList PayloadFields(MetaData vo, MetaRoot roo { string? note = null; string type; + bool nestedRef = false; if (CSharpNaming.ScalarForField(f) is { } scalar) type = scalar; else if (f.SubType == MetaObjects.Core.Field.FieldConstants.FIELD_SUBTYPE_ENUM && vo is MetaObject owner) @@ -309,10 +310,14 @@ private static IReadOnlyList PayloadFields(MetaData vo, MetaRoot roo note = "allowed: " + string.Join(" | ", values); } else if (ValueObjectNames.ResolveFieldRef(f, root) is { } nested) + { type = ValueObjectNames.TypeName(nested, root); + nestedRef = true; + } else type = f.SubType; - if (f.ResolvedIsArray()) type = $"ICollection<{type}>"; + // A scalar or enum array is an IList (the shape EF Core 8 on SQLite materializes); an array of value objects stays ICollection. + if (f.ResolvedIsArray()) type = nestedRef ? $"ICollection<{type}>" : $"IList<{type}>"; rows.Add(new FieldShape(f.Name, type, !Fr010FieldMapping.IsRequired(f), note)); } return rows; diff --git a/server/csharp/MetaObjects.Codegen/Generators/EntityGenerator.cs b/server/csharp/MetaObjects.Codegen/Generators/EntityGenerator.cs index 93dd62dd9..87188a612 100644 --- a/server/csharp/MetaObjects.Codegen/Generators/EntityGenerator.cs +++ b/server/csharp/MetaObjects.Codegen/Generators/EntityGenerator.cs @@ -179,6 +179,15 @@ public virtual IEnumerable Generate(GenContext ctx) ctx.Config, PackageBindingResolver.EffectivePackage(e), e.Name, fallbackContext: e.Name)) .Distinct(StringComparer.Ordinal)) files.Add(EmitNetValidationFile(ns)); + + // ADR-0019 Amendment 1 — the decimal wire helper, ONCE per namespace that has a decimal + // property (every emitted shape that can carry one: mapped entities incl. projections, + // report rows, TPH classes and abstract shapes, sourceless shapes and value objects). + foreach (var ns in mapped.Concat(unmappedShapes).Concat(valueObjects).Where(HasDecimalWireField) + .Select(e => PackageBindingResolver.Resolve( + ctx.Config, PackageBindingResolver.EffectivePackage(e), e.Name, fallbackContext: e.Name)) + .Distinct(StringComparer.Ordinal)) + files.Add(EmitDecimalWireFile(ns)); return files; } @@ -592,7 +601,8 @@ protected virtual string TphSubtypeScalarProperty( var arr = new StringBuilder(); arr.AppendLine($" [Column({CSharpNaming.ColumnRef(entity, field, strategy, includeNames)})]"); AppendArrayValidatorAttributes(arr, field); - arr.Append($" public ICollection<{baseType}> {propName} {{ get; set; }} = new List<{baseType}>();"); + AppendDecimalConverterAttribute(arr, field, nullable: false); + arr.Append($" public IList<{baseType}> {propName} {{ get; set; }} = new List<{baseType}>();"); return arr.ToString(); } @@ -603,6 +613,7 @@ protected virtual string TphSubtypeScalarProperty( else AppendNumericValidatorAttributes(sb, field); var required = CSharpNaming.IsRequired(entity, field); + AppendDecimalConverterAttribute(sb, field, nullable: !required); var nullable = required ? "" : "?"; // Mirrors ScalarProperty's own initializer handling (its non-TPH sibling): a // literal-safe @default (bool/int/long/double/float/decimal/string) wins over the @@ -628,7 +639,7 @@ protected virtual string TphSubtypeEnumProperty(MetaObject entity, MetaField fie var propName = PropertyName(field); var colAttr = $" [Column({CSharpNaming.ColumnRef(entity, field, strategy, config.IncludeNames)})]\n"; if (field.ResolvedIsArray()) // ADR-0039: resolving — array-ness inheritable via extends - return $"{colAttr} public ICollection<{typeName}> {propName} {{ get; set; }} = new List<{typeName}>();"; + return $"{colAttr} public IList<{typeName}> {propName} {{ get; set; }} = new List<{typeName}>();"; var nullable = CSharpNaming.IsRequired(entity, field) ? "" : "?"; return $"{colAttr} public {typeName}{nullable} {propName} {{ get; set; }}"; } @@ -997,6 +1008,311 @@ private static void AppendNetConverterAttribute(StringBuilder sb, MetaField fiel sb.AppendLine(" [JsonConverter(typeof(MetaNetValidation.IPAddressJsonConverter))]"); } + // ADR-0019 Amendment 1 — the decimal wire form. This text is the Runtime/DecimalWire.cs + // source with its namespace as __NS__; DecimalWireTests gates the two byte-for-byte after + // the namespace swap. It lives HERE (not read from the HelperRuntime resource) so an + // ejected EntityGenerator stays self-contained, the way the MetaNetValidation text is. + private const string DecimalWireSourceTemplate = """ +// DecimalWire — the wire form of a field.decimal (ADR-0019 Amendment 1, W1-W4). +// +// A decimal crosses JSON as a canonical STRING: `^-?(0|[1-9][0-9]*)(\.[0-9]*[1-9])?$`, +// plain notation, no exponent, no trailing fractional zeros, zero as "0". A JSON number is +// a double to every JavaScript client, so the string is the only spelling a browser reads +// exactly. Input takes a string in the JSON-number grammar or a JSON number; anything else +// is refused. A value `decimal` cannot hold (more than 28-29 significant digits) is an +// error, never a rounded value (W3). +// +// ONE source, two homes. This file is the package's production helper (the filter +// dispatcher parses its operands through it). The entity generator emits the SAME text, +// with only the namespace changed, as `DecimalWire.g.cs` into every namespace that has a +// decimal property, so the `[JsonConverter(typeof(DecimalWire.…))]` on each property +// resolves inside the generated project and the generated entity code imports nothing +// from MetaObjects (CLAUDE.md). DecimalWireTests gates the two texts byte-for-byte. +// Keep this file free of references to anything but the BCL. + +#nullable enable +using System; +using System.Buffers; +using System.Collections.Generic; +using System.Globalization; +using System.Text; +using System.Text.Json; +using System.Text.Json.Serialization; + +namespace __NS__; + +/// The canonical wire form of a decimal: format, parse, and the System.Text.Json converters. +public static class DecimalWire +{ + /// W1. The canonical string of : plain notation, trailing fractional zeros stripped, zero as "0". + public static string Format(decimal value) + { + var s = value.ToString(CultureInfo.InvariantCulture); + if (s.IndexOf('.') >= 0) + { + s = s.TrimEnd('0'); + if (s.EndsWith('.')) s = s.Substring(0, s.Length - 1); + } + return s == "-0" || s.Length == 0 ? "0" : s; + } + + /// + /// W2. Parse text in the JSON-number grammar (-?(0|[1-9][0-9]*)(\.[0-9]+)?([eE][+-]?[0-9]+)?, + /// no whitespace, no sign other than -). False for anything else, and for a value + /// decimal cannot hold exactly (W3). + /// + public static bool TryParse(string? text, out decimal value) + { + value = 0m; + if (text is null) return false; + int i = 0, n = text.Length; + bool negative = false; + if (i < n && text[i] == '-') { negative = true; i++; } + if (i >= n) return false; + int intStart = i; + if (text[i] == '0') i++; + else if (text[i] >= '1' && text[i] <= '9') { while (i < n && IsDigit(text[i])) i++; } + else return false; + int intEnd = i; + int fracStart = i, fracEnd = i; + if (i < n && text[i] == '.') + { + i++; + fracStart = i; + while (i < n && IsDigit(text[i])) i++; + if (i == fracStart) return false; + fracEnd = i; + } + long exponent = 0; + if (i < n && (text[i] == 'e' || text[i] == 'E')) + { + i++; + bool expNegative = false; + if (i < n && (text[i] == '+' || text[i] == '-')) { expNegative = text[i] == '-'; i++; } + int expStart = i; + while (i < n && IsDigit(text[i])) + { + if (exponent < 100000) exponent = exponent * 10 + (text[i] - '0'); + i++; + } + if (i == expStart) return false; + if (expNegative) exponent = -exponent; + } + if (i != n) return false; + var digits = text.Substring(intStart, intEnd - intStart) + text.Substring(fracStart, fracEnd - fracStart); + return TryBuild(negative, digits, (intEnd - intStart) + exponent, out value); + } + + /// W2 for an already-parsed JSON value (a PATCH merge reads one property at a time): a string or a number. + public static bool TryParseElement(JsonElement element, out decimal value) + { + value = 0m; + return element.ValueKind switch + { + JsonValueKind.String => TryParse(element.GetString(), out value), + JsonValueKind.Number => TryParse(element.GetRawText(), out value), + _ => false, + }; + } + + /// + /// W4. The value of an IEEE double at 15 significant digits (SQLite's own text precision), + /// canonicalized. A SQLite or D1 NUMERIC column is a double, so this is the most a read of + /// one can promise. Throws beyond decimal's range. + /// + public static decimal FromDouble15(double value) + { + if (double.IsNaN(value) || double.IsInfinity(value)) + throw new OverflowException("a decimal cannot hold " + value.ToString(CultureInfo.InvariantCulture)); + var e = value.ToString("E14", CultureInfo.InvariantCulture); // d.ddddddddddddddE+ddd + bool negative = e[0] == '-'; + int ePos = e.IndexOf('E'); + var mantissa = e.Substring(negative ? 1 : 0, ePos - (negative ? 1 : 0)).Replace(".", ""); + long exponent = long.Parse(e.Substring(ePos + 1), NumberStyles.AllowLeadingSign, CultureInfo.InvariantCulture); + if (!TryBuild(negative, mantissa, 1 + exponent, out var result)) + throw new OverflowException("a decimal cannot hold " + e); + return result; + } + + /// W4 as text: of . + public static string FormatDouble15(double value) => Format(FromDouble15(value)); + + private static bool IsDigit(char c) => c >= '0' && c <= '9'; + + // digits is the unsigned digit run; the decimal point sits after pointPos of them. + private static bool TryBuild(bool negative, string digits, long pointPos, out decimal value) + { + value = 0m; + int lead = 0; + while (lead < digits.Length && digits[lead] == '0') lead++; + int end = digits.Length; + while (end > lead && digits[end - 1] == '0') end--; + if (lead == end) return true; // zero, in any spelling + var d = digits.Substring(lead, end - lead); + long point = pointPos - lead; + // decimal holds at most 29 integer digits and 28 fractional places. + if (point > 29 || point < -28) return false; + var sb = new StringBuilder(); + if (negative) sb.Append('-'); + if (point <= 0) sb.Append("0.").Append('0', (int)-point).Append(d); + else if (point >= d.Length) sb.Append(d).Append('0', (int)point - d.Length); + else sb.Append(d, 0, (int)point).Append('.').Append(d, (int)point, d.Length - (int)point); + var plain = sb.ToString(); + if (!decimal.TryParse(plain, NumberStyles.AllowLeadingSign | NumberStyles.AllowDecimalPoint, + CultureInfo.InvariantCulture, out value)) return false; + // decimal.Parse ROUNDS an excess digit; a round trip that does not reproduce the + // input means it was rounded (W3: an error, never silent rounding). + return Format(value) == plain; + } + + /// + /// W3 for a declared column: true when fits @precision and + /// @scale without rounding: no more fraction digits than , no + /// more integer digits than minus . + /// Counted on the canonical form, so trailing zeros never count. + /// + public static bool Fits(decimal value, int precision, int scale) + { + var text = Format(value); + if (text.StartsWith('-')) text = text.Substring(1); + var dot = text.IndexOf('.'); + var integerDigits = (dot < 0 ? text : text.Substring(0, dot)).TrimStart('0').Length; + var fractionDigits = dot < 0 ? 0 : text.Length - dot - 1; + return fractionDigits <= scale && integerDigits <= precision - scale; + } + + /// + /// W3 for a declared property: false when of + /// carries and does not fit it. + /// True for a property with no declared precision (the PATCH merge asks this per key). + /// + public static bool FitsDeclared(Type owner, string property, decimal value) + { + var declared = owner.GetProperty(property)?.GetCustomAttributes(typeof(DeclaredAttribute), true); + return declared is not { Length: > 0 } || ((DeclaredAttribute)declared[0]).Accepts(value); + } + + private static decimal ReadOne(ref Utf8JsonReader reader, int? precision = null, int? scale = null) + { + decimal result; + switch (reader.TokenType) + { + case JsonTokenType.String: + if (!TryParse(reader.GetString(), out result)) + throw new JsonException("expected a decimal as a JSON-number string or a JSON number"); + break; + case JsonTokenType.Number: + var raw = Encoding.UTF8.GetString(reader.HasValueSequence ? reader.ValueSequence.ToArray() : reader.ValueSpan.ToArray()); + if (!TryParse(raw, out result)) throw new JsonException("decimal out of range"); + break; + default: + throw new JsonException("expected a decimal as a JSON-number string or a JSON number"); + } + if (precision is int p && scale is int sc && !Fits(result, p, sc)) + throw new JsonException("decimal exceeds the declared precision and scale"); + return result; + } + + /// + /// A decimal property with a declared @precision and @scale: the same wire form + /// as the plain converters, and a value the column cannot hold is refused, never rounded (W3). + /// + [AttributeUsage(AttributeTargets.Property)] + public sealed class DeclaredAttribute(int precision, int scale) : JsonConverterAttribute + { + public int Precision { get; } = precision; + public int Scale { get; } = scale; + + public bool Accepts(decimal value) => Fits(value, Precision, Scale); + + public override JsonConverter? CreateConverter(Type typeToConvert) + { + if (typeToConvert == typeof(decimal)) return new DecimalJsonConverter(Precision, Scale); + if (typeToConvert == typeof(decimal?)) return new NullableDecimalJsonConverter(Precision, Scale); + if (typeToConvert == typeof(IList)) return new DecimalListJsonConverter(Precision, Scale); + return null; + } + } + + /// A decimal property: written as the canonical string, read from a string or a number. + public sealed class DecimalJsonConverter(int? precision, int? scale) : JsonConverter + { + public DecimalJsonConverter() : this(null, null) { } + + public override decimal Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) + => ReadOne(ref reader, precision, scale); + + public override void Write(Utf8JsonWriter writer, decimal value, JsonSerializerOptions options) + => writer.WriteStringValue(Format(value)); + } + + /// A nullable decimal property: null is JSON null, with the key present. + public sealed class NullableDecimalJsonConverter(int? precision, int? scale) : JsonConverter + { + public NullableDecimalJsonConverter() : this(null, null) { } + + public override bool HandleNull => true; + + public override decimal? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) + => reader.TokenType == JsonTokenType.Null ? null : ReadOne(ref reader, precision, scale); + + public override void Write(Utf8JsonWriter writer, decimal? value, JsonSerializerOptions options) + { + if (value is null) writer.WriteNullValue(); + else writer.WriteStringValue(Format(value.Value)); + } + } + + /// An @isArray decimal property, typed IList<decimal> (the one collection shape EF Core 8 on SQLite can materialize): an array of canonical strings. + public sealed class DecimalListJsonConverter(int? precision, int? scale) : JsonConverter> + { + public DecimalListJsonConverter() : this(null, null) { } + + public override IList? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) + { + if (reader.TokenType == JsonTokenType.Null) return null; + if (reader.TokenType != JsonTokenType.StartArray) throw new JsonException("expected an array of decimals"); + var list = new List(); + while (reader.Read() && reader.TokenType != JsonTokenType.EndArray) list.Add(ReadOne(ref reader, precision, scale)); + return list; + } + + public override void Write(Utf8JsonWriter writer, IList value, JsonSerializerOptions options) + { + writer.WriteStartArray(); + foreach (var d in value) writer.WriteStringValue(Format(d)); + writer.WriteEndArray(); + } + } +} +"""; + + private static EmittedFile EmitDecimalWireFile(string ns) => + new("DecimalWire.g.cs", DecimalWireSourceTemplate.Replace("__NS__", ns, StringComparison.Ordinal) + "\n"); + + // true iff FIELD surfaces as a CLR decimal (single or @isArray) and so carries the + // per-property wire converter. A jsonb override keeps the JsonDocument type, not decimal. + private static bool IsDecimalWireField(MetaField field) => + field.SubType == FIELD_SUBTYPE_DECIMAL && CSharpNaming.DbColumnTypeClrOverride(field) is null; + + private static bool HasDecimalWireField(MetaObject entity) => entity.Fields().Any(IsDecimalWireField); + + // The per-property converter: host JSON options cannot drop it. `required` decides which + // closed type the property has (decimal, decimal?, IList). + private static void AppendDecimalConverterAttribute(StringBuilder sb, MetaField field, bool nullable) + { + if (!IsDecimalWireField(field)) return; + // W3: a declared @precision and @scale bound the input; the attribute carries both. + if (field.Precision is long precision) + { + sb.AppendLine($" [DecimalWire.Declared({precision}, {field.Scale ?? 0})]"); + return; + } + var converter = field.ResolvedIsArray() ? "DecimalListJsonConverter" + : nullable ? "NullableDecimalJsonConverter" : "DecimalJsonConverter"; + sb.AppendLine($" [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.{converter}))]"); + } + // #234 — emit the codegen-owned MetaNetValidation.cs into `ns`: two strict System.Text.Json // converters implementing the Contract-A rules (absolute-scheme URI; IPv4/IPv6 literal, no DNS) // verified against the shared validation-conformance corpus. Dependency-free, so the generated @@ -1109,7 +1425,7 @@ protected virtual string EnumProperty( var propName = PropertyName(field); var colAttr = withAttributes ? $" [Column({CSharpNaming.ColumnRef(entity, field, strategy, config.IncludeNames)})]\n" : ""; if (field.ResolvedIsArray()) // ADR-0039: resolving — array-ness inheritable via extends - return $"{colAttr} public ICollection<{typeName}> {propName} {{ get; set; }} = new List<{typeName}>();"; + return $"{colAttr} public IList<{typeName}> {propName} {{ get; set; }} = new List<{typeName}>();"; var required = CSharpNaming.IsRequired(entity, field); var type = required ? typeName : typeName + "?"; return $"{colAttr} public {type} {propName} {{ get; set; }}"; @@ -1160,7 +1476,10 @@ protected virtual string ScalarProperty( // validator.array @min/@max → element-count bounds on the collection. if (emitValidation) AppendArrayValidatorAttributes(arr, field); - arr.Append($" public ICollection<{baseType}> {propName} {{ get; set; }} = new List<{baseType}>();"); + // EF Core 8 on SQLite materializes a primitive collection only through IList (an + // ICollection property throws at query compile time), so a scalar array is an IList. + AppendDecimalConverterAttribute(arr, field, nullable: false); + arr.Append($" public IList<{baseType}> {propName} {{ get; set; }} = new List<{baseType}>();"); return arr.ToString(); } @@ -1212,6 +1531,9 @@ protected virtual string ScalarProperty( // is rejected at the wire tier — aligning the C# entity with the TS/Python strict contract. if (emitValidation) AppendNetConverterAttribute(sb, field); + // ADR-0019 Amendment 1: a decimal crosses JSON as a canonical string. Attached per + // property (not through host JsonSerializerOptions) so no host configuration drops it. + AppendDecimalConverterAttribute(sb, field, nullable: !nonNull); var type = nonNull ? baseType : baseType + "?"; // A literal-safe @default (bool / integer / floating / string) emits a property diff --git a/server/csharp/MetaObjects.Codegen/Generators/RoutesGenerator.cs b/server/csharp/MetaObjects.Codegen/Generators/RoutesGenerator.cs index f3774bcea..cd0510717 100644 --- a/server/csharp/MetaObjects.Codegen/Generators/RoutesGenerator.cs +++ b/server/csharp/MetaObjects.Codegen/Generators/RoutesGenerator.cs @@ -182,6 +182,16 @@ protected virtual EmittedFile GenerateStandardRoutes(MetaObject entity, GenConte .Select(f => CSharpNaming.Pascal(f.Name)) .ToList(); + // ADR-0019 Amendment 1 (W4): SQLite refuses `ORDER BY` over a decimal expression, and its + // NUMERIC column is a double anyway, so on SQLite a decimal sorts by its double. Every + // other engine sorts the exact decimal. + var decimalSortFields = entity.Fields() + .Where(f => f.SubType == FIELD_SUBTYPE_DECIMAL && !f.ResolvedIsArray() + && CSharpNaming.DbColumnTypeClrOverride(f) is null) + .Select(f => CSharpNaming.Pascal(f.Name)) + .ToHashSet(StringComparer.Ordinal); + bool sqliteSort = sortFields.Any(decimalSortFields.Contains); + var sb = new StringBuilder(); sb.AppendLine("// "); sb.AppendLine("// Generated by MetaObjects routes-generator. Do not edit by hand."); @@ -209,6 +219,7 @@ protected virtual EmittedFile GenerateStandardRoutes(MetaObject entity, GenConte AppendUnauthenticatedRoutesDoc(sb, cls, isProjection); sb.AppendLine($" public static IEndpointRouteBuilder Map{cls}Routes(this IEndpointRouteBuilder app, string prefix = \"/api\")"); sb.AppendLine(" {"); + AppendDecimalRangeGroup(sb); // List — honours ?limit / ?offset / ?sort / ?withCount per api-contract.md. sb.AppendLine(" app.MapGet(prefix + \"/" + route + "\", async (HttpContext http, AppDbContext db) =>"); @@ -217,7 +228,7 @@ protected virtual EmittedFile GenerateStandardRoutes(MetaObject entity, GenConte sb.AppendLine($" IQueryable<{readCls}> q = db." + readDbSet + ".AsNoTracking();"); sb.AppendLine(); sb.AppendLine(" // Filter: ?filter[][]= against the per-entity allowlist."); - sb.AppendLine($" var filter = FilterParser.Parse(qs, {cls}FilterAllowlist.Fields, {cls}FilterAllowlist.OpsByField);"); + sb.AppendLine($" var filter = EfCoreFilterDispatch.Validate<{readCls}>(FilterParser.Parse(qs, {cls}FilterAllowlist.Fields, {cls}FilterAllowlist.OpsByField));"); sb.AppendLine(" if (filter.ErrorEnvelope is not null)"); // `field` names the rejected filter field — required cross-port on every // filter envelope, so a caller with several filters knows which one failed. @@ -238,7 +249,9 @@ protected virtual EmittedFile GenerateStandardRoutes(MetaObject entity, GenConte sb.AppendLine(" var desc = parts.Length > 1"); sb.AppendLine(" ? string.Equals(parts[1], \"desc\", System.StringComparison.OrdinalIgnoreCase)"); sb.AppendLine(" : SortDefaultDesc.TryGetValue(resolved, out var dd) && dd;"); - sb.AppendLine($" q = ApplySort{cls}(q, resolved, desc);"); + sb.AppendLine(sqliteSort + ? $" q = ApplySort{cls}(q, resolved, desc, db.Database.ProviderName?.Contains(\"Sqlite\", System.StringComparison.OrdinalIgnoreCase) == true);" + : $" q = ApplySort{cls}(q, resolved, desc);"); sb.AppendLine(" }"); sb.AppendLine(); sb.AppendLine(" // Pagination: ?offset=N first (skip), then ?limit=N (take)."); @@ -311,7 +324,7 @@ protected virtual EmittedFile GenerateStandardRoutes(MetaObject entity, GenConte sb.AppendLine(" .Value.SerializerOptions;"); sb.AppendLine(" var entry = db.Entry(existing);"); AppendUpdateAutoSet(sb, autoSetFields); - AppendPartialMergeLoop(sb, null, voFields, mapVoFields, autoSetNavs, frozenNavs); + AppendPartialMergeLoop(sb, null, voFields, mapVoFields, autoSetNavs, frozenNavs, HasDecimalField(entity)); AppendSaveChanges(sb); AppendArrayNullClears(sb, voFields, tableRef, pkColumnRef); // #214 — writes target the table; for a write-through entity re-read the row @@ -355,19 +368,22 @@ protected virtual EmittedFile GenerateStandardRoutes(MetaObject entity, GenConte AppendM2mRoute(sb, nav, route, pkType!, ctx.Root); sb.AppendLine(); - sb.AppendLine(" return app;"); + sb.AppendLine(" return mount;"); sb.AppendLine(" }"); sb.AppendLine(); // Allowlisted property name → IOrderedQueryable dispatch via EF.Property. // Reflection-free at runtime; EF Core translates EF.Property(x, "") // into the column reference at SQL generation. - sb.AppendLine($" private static IOrderedQueryable<{readCls}> ApplySort{cls}(IQueryable<{readCls}> q, string field, bool desc)"); + sb.AppendLine($" private static IOrderedQueryable<{readCls}> ApplySort{cls}(IQueryable<{readCls}> q, string field, bool desc{(sqliteSort ? ", bool sqlite" : "")})"); sb.AppendLine(" {"); sb.AppendLine(" return field switch"); sb.AppendLine(" {"); foreach (var name in sortFields) { - sb.AppendLine($" \"{name}\" => desc ? q.OrderByDescending(x => EF.Property(x!, \"{name}\")) : q.OrderBy(x => EF.Property(x!, \"{name}\")),"); + var exact = $"desc ? q.OrderByDescending(x => EF.Property(x!, \"{name}\")) : q.OrderBy(x => EF.Property(x!, \"{name}\"))"; + sb.AppendLine(decimalSortFields.Contains(name) + ? $" \"{name}\" => sqlite ? (desc ? q.OrderByDescending(x => (double?)EF.Property(x!, \"{name}\")) : q.OrderBy(x => (double?)EF.Property(x!, \"{name}\"))) : ({exact})," + : $" \"{name}\" => {exact},"); } sb.AppendLine(" _ => (IOrderedQueryable<" + readCls + ">)q.OrderBy(x => 0),"); sb.AppendLine(" };"); @@ -504,13 +520,14 @@ protected virtual EmittedFile GenerateTphRoutes(MetaObject baseEntity, TphPlan t AppendUnauthenticatedRoutesDoc(sb, baseCls, readOnly: false); sb.AppendLine($" public static IEndpointRouteBuilder Map{baseCls}Routes(this IEndpointRouteBuilder app, string prefix = \"/api\")"); sb.AppendLine(" {"); + AppendDecimalRangeGroup(sb); // --- Polymorphic list: GET / (union of all subtypes; each row carries its discriminator) --- sb.AppendLine(" app.MapGet(prefix + \"/" + baseRoute + "\", async (HttpContext http, AppDbContext db) =>"); sb.AppendLine(" {"); sb.AppendLine(" var qs = http.Request.Query;"); sb.AppendLine($" IQueryable<{baseCls}> q = db.{dbSet}.AsNoTracking();"); - sb.AppendLine($" var filter = FilterParser.Parse(qs, {baseCls}FilterAllowlist.Fields, {baseCls}FilterAllowlist.OpsByField);"); + sb.AppendLine($" var filter = EfCoreFilterDispatch.Validate<{baseCls}>(FilterParser.Parse(qs, {baseCls}FilterAllowlist.Fields, {baseCls}FilterAllowlist.OpsByField));"); sb.AppendLine(" if (filter.ErrorEnvelope is not null) return Results.BadRequest(new { error = filter.ErrorEnvelope, field = filter.Field });"); sb.AppendLine(" q = EfCoreFilterDispatch.ApplyFilter(q, filter.Predicates);"); sb.AppendLine(" if (qs.TryGetValue(\"sort\", out var sortRaw) && !string.IsNullOrWhiteSpace(sortRaw))"); @@ -555,7 +572,7 @@ protected virtual EmittedFile GenerateTphRoutes(MetaObject baseEntity, TphPlan t } sb.AppendLine(); - sb.AppendLine(" return app;"); + sb.AppendLine(" return mount;"); sb.AppendLine(" }"); sb.AppendLine(); @@ -898,6 +915,22 @@ private static void AppendCreateVoValidation( // ConfigureHttpJsonOptions serializer options, NOT defaults, so a custom converter (e.g. // the field.timestamp DateTimeOffset converter) is honored; default options 500 on it. // Assumes `HttpContext http` is in scope. + // ADR-0019 Amendment 1 (W3): every route of the entity is mounted on a group carrying the + // decimal-range filter, so a stored value past C# decimal's range is a named error on read + // rather than an unhandled 500. `mount` is what the caller passed; `app` is the filtered group, + // so every `app.MapGet(...)` below is unchanged and the method still returns what it was given + // (`mount`). The parameter keeps its name `app`, so a caller passing `app:` by name still compiles. + // true iff the entity has a field that surfaces as a CLR decimal (and so a DecimalWire copy + // is emitted into its namespace). + private static bool HasDecimalField(MetaObject entity) => + entity.Fields().Any(f => f.SubType == FIELD_SUBTYPE_DECIMAL && CSharpNaming.DbColumnTypeClrOverride(f) is null); + + private static void AppendDecimalRangeGroup(StringBuilder sb) + { + sb.AppendLine(" IEndpointRouteBuilder mount = app;"); + sb.AppendLine(" app = app.MapGroup(string.Empty).AddEndpointFilter(new DecimalRangeEndpointFilter());"); + } + private static void AppendJsonOptsLocal(StringBuilder sb) { sb.AppendLine(" var jsonOpts = ((Microsoft.Extensions.Options.IOptions)"); @@ -929,7 +962,7 @@ private static void AppendCreateValidation(StringBuilder sb) private static void AppendPartialMergeLoop( StringBuilder sb, string? discProp, IReadOnlyList voFields, IReadOnlyList mapVoFields, - IReadOnlyList autoSetNavs, IReadOnlyList frozenNavs) + IReadOnlyList autoSetNavs, IReadOnlyList frozenNavs, bool hasDecimal) { // A nullable array-of-VO column that a request clears to null (present-null) can NOT be // NULLed by an owned-nav assignment — EF OwnsMany(...).ToJson writes `[]` for a null @@ -1025,7 +1058,32 @@ private static void AppendPartialMergeLoop( sb.AppendLine(" continue;"); sb.AppendLine(" }"); sb.AppendLine(" var clr = System.Nullable.GetUnderlyingType(target.ClrType) ?? target.ClrType;"); - sb.AppendLine(" var __val = System.Text.Json.JsonSerializer.Deserialize(prop.Value.GetRawText(), clr, jsonOpts);"); + // ADR-0019 Amendment 1 (W2): a decimal is read from a JSON-number string or a JSON number, + // exactly; anything else, or a value decimal cannot hold, is the 400 validation envelope. + // `DecimalWire` resolves to the copy the entity generator emits into this namespace (the + // routes and the entities share it), so the output differs between a packaged and an + // ejected generator only in the helper-runtime `using`, as before. + if (hasDecimal) + { + sb.AppendLine(" object? __val;"); + sb.AppendLine(" if (clr == typeof(decimal))"); + sb.AppendLine(" {"); + sb.AppendLine(" if (!DecimalWire.TryParseElement(prop.Value, out var __dec) || !DecimalWire.FitsDeclared(existing.GetType(), target.Name, __dec)) return Results.BadRequest(new { error = \"validation\" });"); + sb.AppendLine(" __val = __dec;"); + sb.AppendLine(" }"); + sb.AppendLine(" else if (clr == typeof(System.Collections.Generic.IList))"); + sb.AppendLine(" {"); + sb.AppendLine(" if (prop.Value.ValueKind != System.Text.Json.JsonValueKind.Array) return Results.BadRequest(new { error = \"validation\" });"); + sb.AppendLine(" var __decs = new System.Collections.Generic.List();"); + sb.AppendLine(" foreach (var __el in prop.Value.EnumerateArray())"); + sb.AppendLine(" if (DecimalWire.TryParseElement(__el, out var __d1) && DecimalWire.FitsDeclared(existing.GetType(), target.Name, __d1)) __decs.Add(__d1); else return Results.BadRequest(new { error = \"validation\" });"); + sb.AppendLine(" __val = __decs;"); + sb.AppendLine(" }"); + sb.AppendLine(" else"); + sb.AppendLine(" __val = System.Text.Json.JsonSerializer.Deserialize(prop.Value.GetRawText(), clr, jsonOpts);"); + } + else + sb.AppendLine(" var __val = System.Text.Json.JsonSerializer.Deserialize(prop.Value.GetRawText(), clr, jsonOpts);"); sb.AppendLine(" // FR-036 A2 — validate the PRESENT, non-null value with the same per-field"); sb.AppendLine(" // DataAnnotations as create. This validates ONLY the present value (never the"); sb.AppendLine(" // whole row) so an ABSENT @required field is untouched, not a 400."); @@ -1068,7 +1126,7 @@ private static void AppendTphSubtypeRoutes( sb.AppendLine(" {"); sb.AppendLine(" var qs = http.Request.Query;"); sb.AppendLine($" IQueryable<{subCls}> q = db.{dbSet}.OfType<{subCls}>().AsNoTracking();"); - sb.AppendLine($" var filter = FilterParser.Parse(qs, {subCls}FilterAllowlist.Fields, {subCls}FilterAllowlist.OpsByField);"); + sb.AppendLine($" var filter = EfCoreFilterDispatch.Validate<{subCls}>(FilterParser.Parse(qs, {subCls}FilterAllowlist.Fields, {subCls}FilterAllowlist.OpsByField));"); sb.AppendLine(" if (filter.ErrorEnvelope is not null) return Results.BadRequest(new { error = filter.ErrorEnvelope, field = filter.Field });"); sb.AppendLine(" q = EfCoreFilterDispatch.ApplyFilter(q, filter.Predicates);"); sb.AppendLine(" if (qs.TryGetValue(\"sort\", out var sortRaw) && !string.IsNullOrWhiteSpace(sortRaw))"); @@ -1128,7 +1186,7 @@ private static void AppendTphSubtypeRoutes( AppendUpdateAutoSet(sb, autoSetFields); // ADR-0045 + the 0.19.4 lesson: TPH is a SEPARATE code path per port, so the // per-subtype merge states the same exclusion rather than inheriting it. - AppendPartialMergeLoop(sb, discProp, [], mapVoFields, autoSetNavs, frozenNavs); + AppendPartialMergeLoop(sb, discProp, [], mapVoFields, autoSetNavs, frozenNavs, HasDecimalField(st.Entity)); AppendSaveChanges(sb); sb.AppendLine(" return Results.Ok(existing);"); sb.AppendLine(" }"); diff --git a/server/csharp/MetaObjects.Codegen/HelperRuntime.cs b/server/csharp/MetaObjects.Codegen/HelperRuntime.cs index 0247dace2..79d920212 100644 --- a/server/csharp/MetaObjects.Codegen/HelperRuntime.cs +++ b/server/csharp/MetaObjects.Codegen/HelperRuntime.cs @@ -55,7 +55,8 @@ public static class HelperRuntime /// The helper files, in the order eject reports them. Each is referenced by generated /// routes except Iso8601TimestampConverter.cs, which the host registers to give /// those routes the api contract's timestamp spelling — it serves the same wire, so - /// the adopter owns it with them. + /// the adopter owns it with them. DecimalWire.cs is the decimal wire form + /// (ADR-0019 Amendment 1): the filter dispatcher parses decimal operands through it. /// public static readonly IReadOnlyList Files = [ @@ -66,6 +67,8 @@ public static class HelperRuntime "ValueObjectValidator.cs", "ConstraintErrors.cs", "Iso8601TimestampConverter.cs", + "DecimalWire.cs", + "DecimalRangeEndpointFilter.cs", ]; /// Runtime/ files that are CORE and never copied (see the header). diff --git a/server/csharp/MetaObjects.Codegen/MetaObjects.Codegen.csproj b/server/csharp/MetaObjects.Codegen/MetaObjects.Codegen.csproj index a3aa9a618..b078ee7b8 100644 --- a/server/csharp/MetaObjects.Codegen/MetaObjects.Codegen.csproj +++ b/server/csharp/MetaObjects.Codegen/MetaObjects.Codegen.csproj @@ -85,5 +85,7 @@ + + diff --git a/server/csharp/MetaObjects.Codegen/ReportRows.cs b/server/csharp/MetaObjects.Codegen/ReportRows.cs index 7f2777dc7..d735c997e 100644 --- a/server/csharp/MetaObjects.Codegen/ReportRows.cs +++ b/server/csharp/MetaObjects.Codegen/ReportRows.cs @@ -177,7 +177,11 @@ private static MetaField DerivedField(ReportField f) // last, of the finished field, because an int-backed enum's band depends on a // carried attr. A report author cannot narrow this set: there is no node to put // `@filterable` on. - if (QueryConstants.OpsForField(field).Length > 0) field.SetAttr(FIELD_ATTR_FILTERABLE, true); + // An array or a map has no scalar operator and no ordering, so a dimension over one is + // neither filterable nor (see RoutesGenerator's sort list) sortable. + if (!field.ResolvedIsArray() && field.SubType != FIELD_SUBTYPE_MAP + && QueryConstants.OpsForField(field).Length > 0) + field.SetAttr(FIELD_ATTR_FILTERABLE, true); return field; } diff --git a/server/csharp/MetaObjects.Codegen/Runtime/DecimalRangeEndpointFilter.cs b/server/csharp/MetaObjects.Codegen/Runtime/DecimalRangeEndpointFilter.cs new file mode 100644 index 000000000..6d36bb527 --- /dev/null +++ b/server/csharp/MetaObjects.Codegen/Runtime/DecimalRangeEndpointFilter.cs @@ -0,0 +1,51 @@ +// DecimalRangeEndpointFilter — ADR-0019 Amendment 1 (W3), the READ side. +// +// C#'s `decimal` holds 28 to 29 significant digits; Postgres `numeric` holds 131072. A stored +// value past C#'s range makes the driver throw OverflowException while a row is read, which +// ASP.NET Core's default handler turns into a bare 500 with a stack trace in development. +// The value cannot be returned without rounding it, and a rounded decimal is a wrong answer, +// so the route answers with a NAMED error instead. It is a 500, not a 4xx: the caller did +// nothing wrong, the stored data is outside what this port can represent. +// +// The generated `MapRoutes` mounts its routes on a group carrying this filter, so +// every read and write handler of the entity is covered by one registration. + +using Microsoft.AspNetCore.Http; + +namespace MetaObjects.Codegen.Runtime; + +/// Turns a driver's decimal-overflow-on-read into 500 {"error":"decimal_out_of_range"}. +public sealed class DecimalRangeEndpointFilter : IEndpointFilter +{ + /// The cross-port-style error code of the named error. + public const string ErrorCode = "decimal_out_of_range"; + + public async ValueTask InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next) + { + try + { + return await next(context); + } + catch (Exception ex) when (IsDecimalOverflow(ex)) + { + return Results.Json( + new + { + error = ErrorCode, + message = "a stored decimal is outside the range this server's decimal type can hold " + + "(28 to 29 significant digits); it is not returned rounded", + }, + statusCode: StatusCodes.Status500InternalServerError); + } + } + + /// True iff the exception chain holds the OverflowException a driver throws when a + /// numeric does not fit decimal (Npgsql: "Value was either too large or too small for a Decimal"). + public static bool IsDecimalOverflow(Exception? ex) + { + for (var e = ex; e is not null; e = e.InnerException) + if (e is OverflowException && e.Message.Contains("Decimal", StringComparison.OrdinalIgnoreCase)) + return true; + return false; + } +} diff --git a/server/csharp/MetaObjects.Codegen/Runtime/DecimalWire.cs b/server/csharp/MetaObjects.Codegen/Runtime/DecimalWire.cs new file mode 100644 index 000000000..e7e373316 --- /dev/null +++ b/server/csharp/MetaObjects.Codegen/Runtime/DecimalWire.cs @@ -0,0 +1,272 @@ +// DecimalWire — the wire form of a field.decimal (ADR-0019 Amendment 1, W1-W4). +// +// A decimal crosses JSON as a canonical STRING: `^-?(0|[1-9][0-9]*)(\.[0-9]*[1-9])?$`, +// plain notation, no exponent, no trailing fractional zeros, zero as "0". A JSON number is +// a double to every JavaScript client, so the string is the only spelling a browser reads +// exactly. Input takes a string in the JSON-number grammar or a JSON number; anything else +// is refused. A value `decimal` cannot hold (more than 28-29 significant digits) is an +// error, never a rounded value (W3). +// +// ONE source, two homes. This file is the package's production helper (the filter +// dispatcher parses its operands through it). The entity generator emits the SAME text, +// with only the namespace changed, as `DecimalWire.g.cs` into every namespace that has a +// decimal property, so the `[JsonConverter(typeof(DecimalWire.…))]` on each property +// resolves inside the generated project and the generated entity code imports nothing +// from MetaObjects (CLAUDE.md). DecimalWireTests gates the two texts byte-for-byte. +// Keep this file free of references to anything but the BCL. + +#nullable enable +using System; +using System.Buffers; +using System.Collections.Generic; +using System.Globalization; +using System.Text; +using System.Text.Json; +using System.Text.Json.Serialization; + +namespace MetaObjects.Codegen.Runtime; + +/// The canonical wire form of a decimal: format, parse, and the System.Text.Json converters. +public static class DecimalWire +{ + /// W1. The canonical string of : plain notation, trailing fractional zeros stripped, zero as "0". + public static string Format(decimal value) + { + var s = value.ToString(CultureInfo.InvariantCulture); + if (s.IndexOf('.') >= 0) + { + s = s.TrimEnd('0'); + if (s.EndsWith('.')) s = s.Substring(0, s.Length - 1); + } + return s == "-0" || s.Length == 0 ? "0" : s; + } + + /// + /// W2. Parse text in the JSON-number grammar (-?(0|[1-9][0-9]*)(\.[0-9]+)?([eE][+-]?[0-9]+)?, + /// no whitespace, no sign other than -). False for anything else, and for a value + /// decimal cannot hold exactly (W3). + /// + public static bool TryParse(string? text, out decimal value) + { + value = 0m; + if (text is null) return false; + int i = 0, n = text.Length; + bool negative = false; + if (i < n && text[i] == '-') { negative = true; i++; } + if (i >= n) return false; + int intStart = i; + if (text[i] == '0') i++; + else if (text[i] >= '1' && text[i] <= '9') { while (i < n && IsDigit(text[i])) i++; } + else return false; + int intEnd = i; + int fracStart = i, fracEnd = i; + if (i < n && text[i] == '.') + { + i++; + fracStart = i; + while (i < n && IsDigit(text[i])) i++; + if (i == fracStart) return false; + fracEnd = i; + } + long exponent = 0; + if (i < n && (text[i] == 'e' || text[i] == 'E')) + { + i++; + bool expNegative = false; + if (i < n && (text[i] == '+' || text[i] == '-')) { expNegative = text[i] == '-'; i++; } + int expStart = i; + while (i < n && IsDigit(text[i])) + { + if (exponent < 100000) exponent = exponent * 10 + (text[i] - '0'); + i++; + } + if (i == expStart) return false; + if (expNegative) exponent = -exponent; + } + if (i != n) return false; + var digits = text.Substring(intStart, intEnd - intStart) + text.Substring(fracStart, fracEnd - fracStart); + return TryBuild(negative, digits, (intEnd - intStart) + exponent, out value); + } + + /// W2 for an already-parsed JSON value (a PATCH merge reads one property at a time): a string or a number. + public static bool TryParseElement(JsonElement element, out decimal value) + { + value = 0m; + return element.ValueKind switch + { + JsonValueKind.String => TryParse(element.GetString(), out value), + JsonValueKind.Number => TryParse(element.GetRawText(), out value), + _ => false, + }; + } + + /// + /// W4. The value of an IEEE double at 15 significant digits (SQLite's own text precision), + /// canonicalized. A SQLite or D1 NUMERIC column is a double, so this is the most a read of + /// one can promise. Throws beyond decimal's range. + /// + public static decimal FromDouble15(double value) + { + if (double.IsNaN(value) || double.IsInfinity(value)) + throw new OverflowException("a decimal cannot hold " + value.ToString(CultureInfo.InvariantCulture)); + var e = value.ToString("E14", CultureInfo.InvariantCulture); // d.ddddddddddddddE+ddd + bool negative = e[0] == '-'; + int ePos = e.IndexOf('E'); + var mantissa = e.Substring(negative ? 1 : 0, ePos - (negative ? 1 : 0)).Replace(".", ""); + long exponent = long.Parse(e.Substring(ePos + 1), NumberStyles.AllowLeadingSign, CultureInfo.InvariantCulture); + if (!TryBuild(negative, mantissa, 1 + exponent, out var result)) + throw new OverflowException("a decimal cannot hold " + e); + return result; + } + + /// W4 as text: of . + public static string FormatDouble15(double value) => Format(FromDouble15(value)); + + private static bool IsDigit(char c) => c >= '0' && c <= '9'; + + // digits is the unsigned digit run; the decimal point sits after pointPos of them. + private static bool TryBuild(bool negative, string digits, long pointPos, out decimal value) + { + value = 0m; + int lead = 0; + while (lead < digits.Length && digits[lead] == '0') lead++; + int end = digits.Length; + while (end > lead && digits[end - 1] == '0') end--; + if (lead == end) return true; // zero, in any spelling + var d = digits.Substring(lead, end - lead); + long point = pointPos - lead; + // decimal holds at most 29 integer digits and 28 fractional places. + if (point > 29 || point < -28) return false; + var sb = new StringBuilder(); + if (negative) sb.Append('-'); + if (point <= 0) sb.Append("0.").Append('0', (int)-point).Append(d); + else if (point >= d.Length) sb.Append(d).Append('0', (int)point - d.Length); + else sb.Append(d, 0, (int)point).Append('.').Append(d, (int)point, d.Length - (int)point); + var plain = sb.ToString(); + if (!decimal.TryParse(plain, NumberStyles.AllowLeadingSign | NumberStyles.AllowDecimalPoint, + CultureInfo.InvariantCulture, out value)) return false; + // decimal.Parse ROUNDS an excess digit; a round trip that does not reproduce the + // input means it was rounded (W3: an error, never silent rounding). + return Format(value) == plain; + } + + /// + /// W3 for a declared column: true when fits @precision and + /// @scale without rounding: no more fraction digits than , no + /// more integer digits than minus . + /// Counted on the canonical form, so trailing zeros never count. + /// + public static bool Fits(decimal value, int precision, int scale) + { + var text = Format(value); + if (text.StartsWith('-')) text = text.Substring(1); + var dot = text.IndexOf('.'); + var integerDigits = (dot < 0 ? text : text.Substring(0, dot)).TrimStart('0').Length; + var fractionDigits = dot < 0 ? 0 : text.Length - dot - 1; + return fractionDigits <= scale && integerDigits <= precision - scale; + } + + /// + /// W3 for a declared property: false when of + /// carries and does not fit it. + /// True for a property with no declared precision (the PATCH merge asks this per key). + /// + public static bool FitsDeclared(Type owner, string property, decimal value) + { + var declared = owner.GetProperty(property)?.GetCustomAttributes(typeof(DeclaredAttribute), true); + return declared is not { Length: > 0 } || ((DeclaredAttribute)declared[0]).Accepts(value); + } + + private static decimal ReadOne(ref Utf8JsonReader reader, int? precision = null, int? scale = null) + { + decimal result; + switch (reader.TokenType) + { + case JsonTokenType.String: + if (!TryParse(reader.GetString(), out result)) + throw new JsonException("expected a decimal as a JSON-number string or a JSON number"); + break; + case JsonTokenType.Number: + var raw = Encoding.UTF8.GetString(reader.HasValueSequence ? reader.ValueSequence.ToArray() : reader.ValueSpan.ToArray()); + if (!TryParse(raw, out result)) throw new JsonException("decimal out of range"); + break; + default: + throw new JsonException("expected a decimal as a JSON-number string or a JSON number"); + } + if (precision is int p && scale is int sc && !Fits(result, p, sc)) + throw new JsonException("decimal exceeds the declared precision and scale"); + return result; + } + + /// + /// A decimal property with a declared @precision and @scale: the same wire form + /// as the plain converters, and a value the column cannot hold is refused, never rounded (W3). + /// + [AttributeUsage(AttributeTargets.Property)] + public sealed class DeclaredAttribute(int precision, int scale) : JsonConverterAttribute + { + public int Precision { get; } = precision; + public int Scale { get; } = scale; + + public bool Accepts(decimal value) => Fits(value, Precision, Scale); + + public override JsonConverter? CreateConverter(Type typeToConvert) + { + if (typeToConvert == typeof(decimal)) return new DecimalJsonConverter(Precision, Scale); + if (typeToConvert == typeof(decimal?)) return new NullableDecimalJsonConverter(Precision, Scale); + if (typeToConvert == typeof(IList)) return new DecimalListJsonConverter(Precision, Scale); + return null; + } + } + + /// A decimal property: written as the canonical string, read from a string or a number. + public sealed class DecimalJsonConverter(int? precision, int? scale) : JsonConverter + { + public DecimalJsonConverter() : this(null, null) { } + + public override decimal Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) + => ReadOne(ref reader, precision, scale); + + public override void Write(Utf8JsonWriter writer, decimal value, JsonSerializerOptions options) + => writer.WriteStringValue(Format(value)); + } + + /// A nullable decimal property: null is JSON null, with the key present. + public sealed class NullableDecimalJsonConverter(int? precision, int? scale) : JsonConverter + { + public NullableDecimalJsonConverter() : this(null, null) { } + + public override bool HandleNull => true; + + public override decimal? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) + => reader.TokenType == JsonTokenType.Null ? null : ReadOne(ref reader, precision, scale); + + public override void Write(Utf8JsonWriter writer, decimal? value, JsonSerializerOptions options) + { + if (value is null) writer.WriteNullValue(); + else writer.WriteStringValue(Format(value.Value)); + } + } + + /// An @isArray decimal property, typed IList<decimal> (the one collection shape EF Core 8 on SQLite can materialize): an array of canonical strings. + public sealed class DecimalListJsonConverter(int? precision, int? scale) : JsonConverter> + { + public DecimalListJsonConverter() : this(null, null) { } + + public override IList? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) + { + if (reader.TokenType == JsonTokenType.Null) return null; + if (reader.TokenType != JsonTokenType.StartArray) throw new JsonException("expected an array of decimals"); + var list = new List(); + while (reader.Read() && reader.TokenType != JsonTokenType.EndArray) list.Add(ReadOne(ref reader, precision, scale)); + return list; + } + + public override void Write(Utf8JsonWriter writer, IList value, JsonSerializerOptions options) + { + writer.WriteStartArray(); + foreach (var d in value) writer.WriteStringValue(Format(d)); + writer.WriteEndArray(); + } + } +} diff --git a/server/csharp/MetaObjects.Codegen/Runtime/EfCoreFilterDispatch.cs b/server/csharp/MetaObjects.Codegen/Runtime/EfCoreFilterDispatch.cs index 3209152f1..bbdd0bcd2 100644 --- a/server/csharp/MetaObjects.Codegen/Runtime/EfCoreFilterDispatch.cs +++ b/server/csharp/MetaObjects.Codegen/Runtime/EfCoreFilterDispatch.cs @@ -49,6 +49,40 @@ public static IQueryable ApplyFilter( return source; } + /// + /// Check every operand of a successfully parsed filter against the CLR type of its + /// property, so a malformed operand is the contract's 400 invalid_filter_value + /// (naming the field) instead of an exception out of . + /// is substrate-agnostic and cannot know a column's type; this + /// is the first place that does. A decimal operand is parsed with the wire grammar + /// () and compared exactly, never through a double. + /// A failed parse result is returned unchanged. + /// + public static FilterParseResult Validate(FilterParseResult parsed) + { + if (parsed.ErrorEnvelope is not null) return parsed; + foreach (var p in parsed.Predicates) + { + if (p.Op is "isNull" or "like") continue; + var prop = ResolveProperty(typeof(T), p.Field); + if (prop is null) continue; // the allowlist gate upstream guarantees a match; BuildLambda reports a miss + var memberType = prop.PropertyType; + var underlying = Nullable.GetUnderlyingType(memberType) ?? memberType; + try + { + if (p.Op == "in") + foreach (var item in (IReadOnlyList)p.Value!) CoerceTo(item, memberType, underlying); + else + CoerceTo((string)(p.Value ?? ""), memberType, underlying); + } + catch (Exception ex) when (ex is FormatException or InvalidCastException or OverflowException or ArgumentException) + { + return FilterParseResult.Err("invalid_filter_value", p.Field); + } + } + return parsed; + } + /// /// Build an Expression<Func<T, bool>> for a single predicate /// against the resolved CLR property of . @@ -164,6 +198,13 @@ private static Expression> BuildLambda(FilterPredicate p) return TimeOnly.Parse(raw, CultureInfo.InvariantCulture); if (underlying.IsEnum) return Enum.Parse(underlying, raw, ignoreCase: false); if (underlying == typeof(bool)) return bool.Parse(raw); + // ADR-0019 Amendment 1 (W2): a decimal operand is the JSON-number grammar, parsed + // exactly. Convert.ChangeType accepted surrounding whitespace, thousands separators + // and a leading '+', and rounded past 28-29 digits. + if (underlying == typeof(decimal)) + return DecimalWire.TryParse(raw, out var exact) + ? exact + : throw new FormatException("not a decimal: " + raw); return Convert.ChangeType(raw, underlying, CultureInfo.InvariantCulture); } } diff --git a/server/csharp/MetaObjects.Conformance.Tests/ConformanceAdapter.cs b/server/csharp/MetaObjects.Conformance.Tests/ConformanceAdapter.cs index e4ee22b96..cb278e8da 100644 --- a/server/csharp/MetaObjects.Conformance.Tests/ConformanceAdapter.cs +++ b/server/csharp/MetaObjects.Conformance.Tests/ConformanceAdapter.cs @@ -75,7 +75,8 @@ public sealed record ErrorEnvelopeRecord( IReadOnlyList Files, string? JsonPath, string? Referrer = null, - string? Target = null); + string? Target = null, + string? Message = null); /// /// Result of loading a fixture's input directory. @@ -219,7 +220,7 @@ private static LoadOutcome LoadFixtureCore( } private static ErrorEnvelopeRecord BuildEnvelope(MetaError err, string inputDir) => - BuildEnvelopeFromSource(err.Code.ToString(), err.Envelope, inputDir); + BuildEnvelopeFromSource(err.Code.ToString(), err.Envelope, inputDir) with { Message = err.Message }; private static ErrorEnvelopeRecord BuildWarningEnvelope(LoaderWarning w, string inputDir) => BuildEnvelopeFromSource(w.Code, w.Source, inputDir); diff --git a/server/csharp/MetaObjects.Conformance.Tests/ConformanceTests.cs b/server/csharp/MetaObjects.Conformance.Tests/ConformanceTests.cs index 30da8c2ad..175974e78 100644 --- a/server/csharp/MetaObjects.Conformance.Tests/ConformanceTests.cs +++ b/server/csharp/MetaObjects.Conformance.Tests/ConformanceTests.cs @@ -138,6 +138,14 @@ private static bool RunChecks(Fixture fix, LoadOutcome outcome, out string detai $"envelope[{i}].code: expected '{w.Code}', got '{g.Code}'"); continue; } + foreach (var needle in w.MessageIncludes ?? []) + { + if (g.Message is null) + failures.Add($"envelope[{i}].messageIncludes: the adapter exposes no message"); + else if (!g.Message.Contains(needle, StringComparison.Ordinal)) + failures.Add( + $"envelope[{i}].messageIncludes: '{needle}' is not in message '{g.Message}'"); + } if (w.Source is null) continue; if (w.Source.Format != g.Format) failures.Add( diff --git a/server/csharp/MetaObjects.Conformance.Tests/DataConverterTests.cs b/server/csharp/MetaObjects.Conformance.Tests/DataConverterTests.cs index dc947db1e..18aba28cd 100644 --- a/server/csharp/MetaObjects.Conformance.Tests/DataConverterTests.cs +++ b/server/csharp/MetaObjects.Conformance.Tests/DataConverterTests.cs @@ -6,6 +6,22 @@ namespace MetaObjects.Conformance.Tests; public class DataConverterTests { + [Theory] + [InlineData("4.0", 4L)] + [InlineData("1e2", 100L)] + [InlineData("-3.0", -3L)] + public void An_integral_json_number_written_with_a_point_converts_to_the_integer(string json, long expected) + { + Assert.Equal(expected, DataConverter.ConvertToDataType(DataType.Int, JsonDocument.Parse(json).RootElement)); + Assert.Equal(expected, DataConverter.ConvertToDataType(DataType.Long, JsonDocument.Parse(json).RootElement)); + } + + [Fact] + public void A_fractional_json_number_stays_a_double_for_the_type_check_to_refuse() + { + Assert.IsType(DataConverter.ConvertToDataType(DataType.Int, JsonDocument.Parse("10.5").RootElement)); + } + // --- ToAttrValue ----------------------------------------------------------- [Fact] diff --git a/server/csharp/MetaObjects.Conformance.Tests/OperationScript.cs b/server/csharp/MetaObjects.Conformance.Tests/OperationScript.cs index 94fabe775..3f6c89ab6 100644 --- a/server/csharp/MetaObjects.Conformance.Tests/OperationScript.cs +++ b/server/csharp/MetaObjects.Conformance.Tests/OperationScript.cs @@ -53,7 +53,8 @@ public sealed record ExpectedErrorSource( /// One expected error in the parsed expected-errors.json. Source /// is non-null iff the file used the FR5a envelope shape. /// - public sealed record ExpectedError(string Code, ExpectedErrorSource? Source); + public sealed record ExpectedError( + string Code, ExpectedErrorSource? Source, IReadOnlyList? MessageIncludes = null); /// /// Result of parsing expected-errors.json. @@ -195,7 +196,16 @@ private static IReadOnlyList ParseEnvelopeEntryArray( } source = new ExpectedErrorSource(fmt, files, jsonPath, referrer, target); } - entries.Add(new ExpectedError(codeProp.GetString()!, source)); + IReadOnlyList? messageIncludes = null; + if (item.TryGetProperty("messageIncludes", out var miEl)) + { + if (miEl.ValueKind != JsonValueKind.Array || miEl.GetArrayLength() == 0 || + miEl.EnumerateArray().Any(e => e.ValueKind != JsonValueKind.String || e.GetString()!.Length == 0)) + throw new InvalidOperationException( + $"expected-errors.json {fieldName}[{idx}]: 'messageIncludes' must be a non-empty array of non-empty strings"); + messageIncludes = miEl.EnumerateArray().Select(e => e.GetString()!).ToList(); + } + entries.Add(new ExpectedError(codeProp.GetString()!, source, messageIncludes)); idx++; } return entries; diff --git a/server/csharp/MetaObjects.Conformance.Tests/ReportShapeTests.cs b/server/csharp/MetaObjects.Conformance.Tests/ReportShapeTests.cs index 6465f377f..758faeaff 100644 --- a/server/csharp/MetaObjects.Conformance.Tests/ReportShapeTests.cs +++ b/server/csharp/MetaObjects.Conformance.Tests/ReportShapeTests.cs @@ -69,7 +69,7 @@ public void The_artifact_covers_the_canonical_reports() ["ProgramMinutes", "FitnessTotals", "ProgramsByMonth", "ProgramsByWeek", "RecentPrograms", "AssetActivity", "ProgramRoster", "ProgramLongWeeks", "ProgramFlags", "ProgramFlagRoster", "FitnessTotalsFilled", "MeasurementStats", "MeasurementStatsFilled", "AssetDays", "AllTypesBreakdown", "ProgramsOverTime", "LiveIntroPrograms", - "ProgramsEmptyScope", "WeekShapes", "SessionsByProgram", "ProgramSessions", "NodesByParent", "AuthsByKind"], + "ProgramsEmptyScope", "WeekShapes", "SessionsByProgram", "ProgramSessions", "NodesByParent", "AuthsByKind", "GadgetEventsByLabel", "EntryTotals"], reports); } diff --git a/server/csharp/MetaObjects.Conformance.Tests/ReportingValidationTests.cs b/server/csharp/MetaObjects.Conformance.Tests/ReportingValidationTests.cs index 0f72c398f..2563d30d8 100644 --- a/server/csharp/MetaObjects.Conformance.Tests/ReportingValidationTests.cs +++ b/server/csharp/MetaObjects.Conformance.Tests/ReportingValidationTests.cs @@ -492,17 +492,12 @@ public void M8_default_on_a_max_of_a_timestamp_is_refused() } [Fact] - public void A_non_integer_default_on_a_count_reports_M7_and_the_attribute_type_error() + public void A_non_numeric_default_on_a_count_reports_only_the_attribute_type_error() { var m = Positive(); PatchMember(m, "Purchase", "purchases", ("@default", "zero")); var errors = Load(m).Errors; - Assert.Equal( - [ErrorCode.ERR_BAD_ATTR_VALUE, ErrorCode.ERR_INVALID_MEASURE], - errors.Select(e => e.Code).OrderBy(c => c.ToString(), System.StringComparer.Ordinal)); - Assert.Contains( - "measure 'purchases' on entity 'acme::shop::Purchase': @default cannot apply to @agg: count.", - errors.Single(e => e.Code == ErrorCode.ERR_INVALID_MEASURE).Message); + Assert.Equal([ErrorCode.ERR_BAD_ATTR_VALUE], errors.Select(e => e.Code)); } [Theory] diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractAssertions.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractAssertions.cs index 02ba43eb3..e6bc127b5 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractAssertions.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractAssertions.cs @@ -210,7 +210,10 @@ private static bool StructuralEquals(object? a, object? b) { if (a is null && b is null) return true; if (a is null || b is null) return false; - if (IsNumber(a) && IsNumber(b)) return Convert.ToInt64(a) == Convert.ToInt64(b); + // Exact compare: integers and decimals meet as decimal (every integer type and + // decimal widen losslessly), so 0.4 != 0 and 12345678901234567.89 != 12345678901234568. + // A number is never equal to a string, so a wire number where "0.3" is expected fails. + if (IsNumber(a) && IsNumber(b)) return Convert.ToDecimal(a) == Convert.ToDecimal(b); if (a is List al && b is List bl) { if (al.Count != bl.Count) return false; @@ -230,7 +233,7 @@ private static bool StructuralEquals(object? a, object? b) } private static bool IsNumber(object o) => - o is sbyte or byte or short or ushort or int or uint or long or ulong; + o is sbyte or byte or short or ushort or int or uint or long or ulong or decimal; private static bool IsEmpty(object? body) => body switch { diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractAssertionsExactnessTests.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractAssertionsExactnessTests.cs new file mode 100644 index 000000000..1c15b721c --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractAssertionsExactnessTests.cs @@ -0,0 +1,51 @@ +// The api-contract assertion engine compares non-integer numbers exactly. It used to +// truncate every number to Int64, so a wire `0.4` satisfied an expected `0`, and a wire +// number satisfied an expected string. ADR-0019 Amendment 1 (W1) makes a decimal a JSON +// string, so an engine that cannot tell the two apart cannot gate the rule. No database. + +using Xunit; +using Xunit.Sdk; + +namespace MetaObjects.IntegrationTests.Api; + +public sealed class ApiContractAssertionsExactnessTests +{ + private static void Check(object? actual, object? expected) => + ApiContractAssertions.AssertResponse("exactness", new ApiRequest( + "r", "GET", "/x", null, 200, + new Dictionary { ["equals"] = expected }), 200, actual); + + private static Dictionary Row(object? v) => new() { ["amount"] = v }; + + [Fact] + public void A_fraction_is_not_equal_to_zero() => + Assert.Throws(() => Check(Row(0.4m), Row(0L))); + + [Fact] + public void Two_decimals_that_a_double_cannot_tell_apart_differ() => + Assert.Throws(() => + Check(Row(12345678901234567.89m), Row(12345678901234568m))); + + [Fact] + public void A_wire_number_is_not_the_expected_string() => + Assert.Throws(() => Check(Row(0.3m), Row("0.3"))); + + [Fact] + public void A_wire_string_is_not_the_expected_number() => + Assert.Throws(() => Check(Row("0.3"), Row(0.3m))); + + [Fact] + public void Equal_decimals_and_integers_pass() + { + Check(Row(12345678901234567.89m), Row(12345678901234567.89m)); + Check(Row(5L), Row(5m)); + Check(Row("12.5"), Row("12.5")); + } + + [Fact] + public void The_yaml_reader_keeps_a_fraction_exact() + { + var node = new YamlDotNet.RepresentationModel.YamlScalarNode("12345678901234567.89"); + Assert.Equal(12345678901234567.89m, ApiContractScenarioLoader.YamlToObject(node)); + } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractConformanceTest.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractConformanceTest.cs index 22e3d3d73..aaad07fde 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractConformanceTest.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractConformanceTest.cs @@ -102,7 +102,7 @@ public static IEnumerable Scenarios() => { if (jv.TryGetValue(out var b)) return b; if (jv.TryGetValue(out var lv)) return lv; - if (jv.TryGetValue(out var dv)) return dv; + if (jv.TryGetValue(out var dv)) return dv; // exact: never a double (ADR-0019 Amendment 1) if (jv.TryGetValue(out var sv)) return sv; return jv.ToString(); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractCorpusPaths.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractCorpusPaths.cs index aec821f60..843be3386 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractCorpusPaths.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractCorpusPaths.cs @@ -76,6 +76,16 @@ internal static class ApiContractCorpusPaths public static readonly string ReportSchemaSqliteSql = Path.Combine(ReportDir, "schema.sqlite.sql"); public static readonly string ReportSchemaMysqlSql = Path.Combine(ReportDir, "schema.mysql.sql"); + // ADR-0019 Amendment 1 decimal subcorpus — a writable Entry (a decimal(28,10) amount) plus a + // served EntryTotals report over it. Generated lane only; `engines:` is exact per scenario. + public static readonly string DecimalDir = Path.Combine(Corpus, "decimal"); + public static readonly string DecimalScenariosDir = Path.Combine(DecimalDir, "scenarios"); + public static readonly string DecimalSeedFile = Path.Combine(DecimalDir, "seed.json"); + public static readonly string DecimalMetaJson = Path.Combine(DecimalDir, "meta.json"); + public static readonly string DecimalSchemaSql = Path.Combine(DecimalDir, "schema.postgres.sql"); + public static readonly string DecimalSchemaSqliteSql = Path.Combine(DecimalDir, "schema.sqlite.sql"); + public static readonly string DecimalSchemaMysqlSql = Path.Combine(DecimalDir, "schema.mysql.sql"); + // #214 write-through read-your-writes subcorpus. public static readonly string WriteThroughDir = Path.Combine(Corpus, "write-through"); public static readonly string WriteThroughScenariosDir = Path.Combine(WriteThroughDir, "scenarios"); diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractDecimalConformanceTest.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractDecimalConformanceTest.cs new file mode 100644 index 000000000..6621b8b2d --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractDecimalConformanceTest.cs @@ -0,0 +1,108 @@ +// ApiContractDecimalConformanceTest — the C# lane of fixtures/api-contract-conformance/decimal/ +// (ADR-0019 Amendment 1): a decimal is an exact canonical string on the wire. +// +// Generated lane only, like report/ and projection/: what is under test is whether this port's +// GENERATOR emits routes that read, write, filter and sort a decimal exactly, so a hand-rolled +// reference server would answer every scenario by construction. The routes are the real +// generators' output, Roslyn-compiled and hosted by GeneratedRoutesServer against the +// committed TypeScript-produced schema of each engine (Postgres, SQLite, MySQL). A scenario's +// `engines:` list is exact: it runs on the engines it names and on no others. + +using System.Text.Json; +using System.Text.Json.Nodes; +using MetaObjects.IntegrationTests.Runner; +using Xunit; + +namespace MetaObjects.IntegrationTests.Api; + +[Collection(MySqlCollection.Name)] +public sealed class ApiContractDecimalConformanceTest(MySqlServer mysqlServer) +{ + private const string Namespace = "MetaObjects.ApiContract.DecimalGenerated"; + + [Theory] + [MemberData(nameof(PostgresScenarios))] + public async Task Api_contract_decimal_generated(string scenarioPath) + { + var scenario = ApiContractScenarioLoader.LoadScenario(scenarioPath); + await using var pg = await PostgresContainer.StartAsync(); + await RunAsync(scenario, ScenarioEngine.Postgres, pg.ConnectionString); + } + + [Theory] + [MemberData(nameof(SqliteScenarios))] + public async Task Api_contract_decimal_generated_on_sqlite(string scenarioPath) + { + var scenario = ApiContractScenarioLoader.LoadScenario(scenarioPath); + using var db = new SqliteTempDatabase("mo-decimal-api-"); + await RunAsync(scenario, ScenarioEngine.Sqlite, $"Data Source={db.FilePath}"); + } + + [Theory] + [MemberData(nameof(MySqlScenarios))] + public async Task Api_contract_decimal_generated_on_mysql(string scenarioPath) + { + var scenario = ApiContractScenarioLoader.LoadScenario(scenarioPath); + await using var mysql = await mysqlServer.CreateDatabaseAsync(); + await RunAsync(scenario, ScenarioEngine.MySql, mysql.ConnectionString); + } + + private static async Task RunAsync(ApiScenario scenario, ScenarioEngine engine, string connectionString) + { + await ReportFixture.ProvisionSchemaAsync( + connectionString, engine, + ApiContractCorpusPaths.DecimalSchemaSql, ApiContractCorpusPaths.DecimalSchemaSqliteSql, + ApiContractCorpusPaths.DecimalSchemaMysqlSql); + await ReportFixture.ApplySeedAsync(connectionString, ApiContractCorpusPaths.DecimalSeedFile, engine); + await using var server = await GeneratedRoutesServer.StartAsync( + ApiContractCorpusPaths.DecimalMetaJson, Namespace, engine, connectionString); + + using var client = new HttpClient { BaseAddress = new Uri(server.BaseUrl) }; + foreach (var req in scenario.Requests) + { + var request = new HttpRequestMessage(new HttpMethod(req.Method), ApiContractWire.VerbatimUri(client, req.Path)); + if (req.Body is not null) + request.Content = new StringContent(JsonSerializer.Serialize(req.Body), System.Text.Encoding.UTF8, "application/json"); + var response = await client.SendAsync(request); + string bodyText = await response.Content.ReadAsStringAsync(); + object? parsed = string.IsNullOrEmpty(bodyText) ? null : ToObject(JsonNode.Parse(bodyText)); + ApiContractAssertions.AssertResponse(scenario.Name, req, (int)response.StatusCode, parsed); + } + } + + public static IEnumerable PostgresScenarios() => ScenariosFor(ScenarioEngine.Postgres); + public static IEnumerable SqliteScenarios() => ScenariosFor(ScenarioEngine.Sqlite); + public static IEnumerable MySqlScenarios() => ScenariosFor(ScenarioEngine.MySql); + + private static IEnumerable ScenariosFor(ScenarioEngine engine) => + Directory.EnumerateFiles(ApiContractCorpusPaths.DecimalScenariosDir, "*.yaml", SearchOption.TopDirectoryOnly) + .OrderBy(p => p, StringComparer.Ordinal) + .Where(p => ApiContractScenarioLoader.LoadScenario(p).RunsOn(engine)) + .Select(p => new object[] { p }); + + private static object? ToObject(JsonNode? node) + { + if (node is null) return null; + if (node is JsonObject obj) + { + var d = new Dictionary(StringComparer.Ordinal); + foreach (var kvp in obj) d[kvp.Key] = ToObject(kvp.Value); + return d; + } + if (node is JsonArray arr) + { + var l = new List(arr.Count); + foreach (var item in arr) l.Add(ToObject(item)); + return l; + } + if (node is JsonValue jv) + { + if (jv.TryGetValue(out var b)) return b; + if (jv.TryGetValue(out var lv)) return lv; + if (jv.TryGetValue(out var dv)) return dv; // exact: never a double (ADR-0019 Amendment 1) + if (jv.TryGetValue(out var sv)) return sv; + return jv.ToString(); + } + return null; + } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractGeneratedConformanceTest.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractGeneratedConformanceTest.cs index ac468e7f7..9db742566 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractGeneratedConformanceTest.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractGeneratedConformanceTest.cs @@ -105,7 +105,7 @@ public static IEnumerable Scenarios() => { if (jv.TryGetValue(out var b)) return b; if (jv.TryGetValue(out var lv)) return lv; - if (jv.TryGetValue(out var dv)) return dv; + if (jv.TryGetValue(out var dv)) return dv; // exact: never a double (ADR-0019 Amendment 1) if (jv.TryGetValue(out var sv)) return sv; return jv.ToString(); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractJsonbConformanceTest.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractJsonbConformanceTest.cs index 0155f1673..79de3e47d 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractJsonbConformanceTest.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractJsonbConformanceTest.cs @@ -97,7 +97,7 @@ public static IEnumerable Scenarios() => { if (jv.TryGetValue(out var b)) return b; if (jv.TryGetValue(out var lv)) return lv; - if (jv.TryGetValue(out var dv)) return dv; + if (jv.TryGetValue(out var dv)) return dv; // exact: never a double (ADR-0019 Amendment 1) if (jv.TryGetValue(out var sv)) return sv; return jv.ToString(); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractM2mConformanceTest.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractM2mConformanceTest.cs index 2c82e402b..46dfe4610 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractM2mConformanceTest.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractM2mConformanceTest.cs @@ -77,7 +77,7 @@ public static IEnumerable Scenarios() => { if (jv.TryGetValue(out var b)) return b; if (jv.TryGetValue(out var lv)) return lv; - if (jv.TryGetValue(out var dv)) return dv; + if (jv.TryGetValue(out var dv)) return dv; // exact: never a double (ADR-0019 Amendment 1) if (jv.TryGetValue(out var sv)) return sv; return jv.ToString(); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractProjectionConformanceTest.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractProjectionConformanceTest.cs index 21cce9203..ad787ca6d 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractProjectionConformanceTest.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractProjectionConformanceTest.cs @@ -77,7 +77,7 @@ public static IEnumerable Scenarios() => { if (jv.TryGetValue(out var b)) return b; if (jv.TryGetValue(out var lv)) return lv; - if (jv.TryGetValue(out var dv)) return dv; + if (jv.TryGetValue(out var dv)) return dv; // exact: never a double (ADR-0019 Amendment 1) if (jv.TryGetValue(out var sv)) return sv; return jv.ToString(); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractReportConformanceTest.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractReportConformanceTest.cs index 3f33a8055..358adec68 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractReportConformanceTest.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractReportConformanceTest.cs @@ -107,7 +107,7 @@ public static IEnumerable Scenarios() => { if (jv.TryGetValue(out var b)) return b; if (jv.TryGetValue(out var lv)) return lv; - if (jv.TryGetValue(out var dv)) return dv; + if (jv.TryGetValue(out var dv)) return dv; // exact: never a double (ADR-0019 Amendment 1) if (jv.TryGetValue(out var sv)) return sv; return jv.ToString(); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractScenarios.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractScenarios.cs index 26d995ff0..3b6a93775 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractScenarios.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractScenarios.cs @@ -4,6 +4,7 @@ // is the cross-port contract documented at // fixtures/api-contract-conformance/README.md. +using MetaObjects.IntegrationTests.Runner; using YamlDotNet.RepresentationModel; namespace MetaObjects.IntegrationTests.Api; @@ -23,7 +24,16 @@ public sealed record ApiScenario( string Description, string SourcePath, bool Truncate, // true when setup.truncate is set - IReadOnlyList Requests); + IReadOnlyList Requests, + IReadOnlyList? Engines = null) // `engines:` — EXACT when present; null = every engine the lane has +{ + /// The engine names a scenario file may list. + public static readonly IReadOnlyList KnownEngines = ["postgres", "mysql", "sqlite", "d1"]; + + /// Whether runs this scenario: the engines: list is exact; absent means all. + public bool RunsOn(MetaObjects.IntegrationTests.Runner.ScenarioEngine engine) => + Engines is null || Engines.Contains(engine.Name(), StringComparer.Ordinal); +} /// /// Builds the request URI for a scenario path so the path reaches the server EXACTLY as the @@ -80,12 +90,27 @@ public static ApiScenario LoadScenario(string yamlPath) } } + IReadOnlyList? engines = null; + if (root.TryGetValue("engines", out var enginesObj)) + { + var names = (enginesObj as List)?.Select(e => e as string).ToList() + ?? throw new InvalidOperationException($"{yamlPath}: 'engines' must be a list of engine names"); + if (names.Count == 0) + throw new InvalidOperationException($"{yamlPath}: 'engines' is an empty list"); + foreach (var name in names) + if (name is null || !ApiScenario.KnownEngines.Contains(name, StringComparer.Ordinal)) + throw new InvalidOperationException( + $"{yamlPath}: unknown engine '{name}' in 'engines' (known: {string.Join(", ", ApiScenario.KnownEngines)})"); + engines = names!; + } + return new ApiScenario( Name: AsString(root, "name", Path.GetFileName(yamlPath)), Description: AsString(root, "description", ""), SourcePath: yamlPath, Truncate: truncate, - Requests: requests); + Requests: requests, + Engines: engines); } private static ApiRequest ParseRequest(Dictionary r, string yamlPath) @@ -165,7 +190,9 @@ private static ApiRequest ParseRequest(Dictionary r, string yam if (s == "false" || s == "False" || s == "FALSE") return false; if (long.TryParse(s, System.Globalization.NumberStyles.Integer, System.Globalization.CultureInfo.InvariantCulture, out var lv)) return lv; - if (double.TryParse(s, System.Globalization.NumberStyles.Float, + // Exact, never a double: 0.4 and 0 must differ, and so must 0.4 and "0.4" (ADR-0019 + // Amendment 1). A number outside decimal's range stays a string-typed token. + if (decimal.TryParse(s, System.Globalization.NumberStyles.Float, System.Globalization.CultureInfo.InvariantCulture, out var dv)) return dv; return s; } diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractTphConformanceTest.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractTphConformanceTest.cs index f8c4de77e..89002aa2a 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractTphConformanceTest.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractTphConformanceTest.cs @@ -95,7 +95,7 @@ public static IEnumerable Scenarios() => { if (jv.TryGetValue(out var b)) return b; if (jv.TryGetValue(out var lv)) return lv; - if (jv.TryGetValue(out var dv)) return dv; + if (jv.TryGetValue(out var dv)) return dv; // exact: never a double (ADR-0019 Amendment 1) if (jv.TryGetValue(out var sv)) return sv; return jv.ToString(); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractWriteThroughConformanceTest.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractWriteThroughConformanceTest.cs index 38fb4f388..d1da47760 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractWriteThroughConformanceTest.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ApiContractWriteThroughConformanceTest.cs @@ -75,7 +75,7 @@ public static IEnumerable Scenarios() => { if (jv.TryGetValue(out var b)) return b; if (jv.TryGetValue(out var lv)) return lv; - if (jv.TryGetValue(out var dv)) return dv; + if (jv.TryGetValue(out var dv)) return dv; // exact: never a double (ADR-0019 Amendment 1) if (jv.TryGetValue(out var sv)) return sv; return jv.ToString(); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/AuthorApiServer.cs b/server/csharp/MetaObjects.IntegrationTests/Api/AuthorApiServer.cs index 5b8158464..a2b08d6ff 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/AuthorApiServer.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/AuthorApiServer.cs @@ -736,7 +736,7 @@ private async Task DeleteAuthorAsync(HttpListenerContext ctx, string idStr) { if (jv.TryGetValue(out var b)) return b; if (jv.TryGetValue(out var lv)) return lv; - if (jv.TryGetValue(out var dv)) return dv; + if (jv.TryGetValue(out var dv)) return dv; // exact: never a double (ADR-0019 Amendment 1) if (jv.TryGetValue(out var sv)) return sv; return jv.ToString(); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/DecimalArrayReportRowTests.cs b/server/csharp/MetaObjects.IntegrationTests/Api/DecimalArrayReportRowTests.cs new file mode 100644 index 000000000..8b4d45aa8 --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Api/DecimalArrayReportRowTests.cs @@ -0,0 +1,53 @@ +// DecimalArrayReportRowTests — a report dimension over a `field.decimal isArray` column, read through +// the generated row on SQLite. EF Core 8 on SQLite materializes a primitive collection only through +// IList, so the row's property type and the decimal converter must both work with it, and each +// element must still cross the wire as a canonical decimal string (ADR-0019 Amendment 1, W1). + +using System.Text; +using MetaObjects.IntegrationTests.Runner; +using Xunit; + +namespace MetaObjects.IntegrationTests.Api; + +public sealed class DecimalArrayReportRowTests +{ + private const string Meta = """ + { "metadata.root": { "package": "acme::prices", "children": [ + { "object.entity": { "name": "Item", "children": [ + { "source.rdb": { "@table": "items" } }, + { "field.long": { "name": "id" } }, + { "field.decimal": { "name": "prices", "isArray": true, "@precision": 12, "@scale": 4 } }, + { "identity.primary": { "name": "pk", "@fields": "id", "@generation": "increment" } }, + { "dimension.attribute": { "name": "prices", "@of": "Item.prices" } }, + { "measure.aggregate": { "name": "items", "@agg": "count", "@of": "Item.id" } } + ] } }, + { "object.report": { "name": "ItemsByPrices", "@from": "Item", "@dimensions": ["prices"], "@measures": ["items"], + "children": [ { "source.rdb": { "@kind": "view", "@view": "v_items_by_prices" } } ] } } + ] } } + """; + + private const string Schema = """ + CREATE TABLE "items" ("id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, "prices" TEXT); + CREATE VIEW "v_items_by_prices" AS + SELECT i."prices" AS "prices", COUNT(i."id") AS "items" FROM "items" i GROUP BY i."prices"; + """; + + [Fact] + public async Task An_array_of_decimals_in_a_report_row_reads_on_sqlite_as_canonical_strings() + { + using var db = new SqliteTempDatabase("mo-decimal-array-"); + var conn = $"Data Source={db.FilePath}"; + await ScenarioEngine.Sqlite.ExecuteScriptAsync(conn, Schema + """INSERT INTO "items" ("prices") VALUES ('["1.5","2.2500","100"]');"""); + var path = Path.Combine(Path.GetTempPath(), "decimal-array-" + Guid.NewGuid().ToString("N") + ".json"); + await File.WriteAllTextAsync(path, Meta); + GeneratedRoutesServer server; + try { server = await GeneratedRoutesServer.StartAsync(path, "MetaObjects.DecimalArray.Generated", ScenarioEngine.Sqlite, conn); } + finally { File.Delete(path); } + await using var _ = server; + using var c = new HttpClient { BaseAddress = new Uri(server.BaseUrl) }; + var res = await c.GetAsync("/api/items_by_prices"); + var body = await res.Content.ReadAsStringAsync(); + Assert.True(res.IsSuccessStatusCode, $"{(int)res.StatusCode} {body}"); + Assert.Contains("\"prices\":[\"1.5\",\"2.25\",\"100\"]", body); + } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/DecimalWireGeneratedApiTests.cs b/server/csharp/MetaObjects.IntegrationTests/Api/DecimalWireGeneratedApiTests.cs new file mode 100644 index 000000000..3f43800c4 --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Api/DecimalWireGeneratedApiTests.cs @@ -0,0 +1,151 @@ +// DecimalWireGeneratedApiTests — the C# generated routes against ADR-0019 Amendment 1 +// (W1-W4) over real HTTP bytes, on a model of its own (one entity with decimal columns). +// +// SQLite needs no Docker; the Postgres tests do (exactness of the comparison, and the W3 read +// overflow, which only a numeric wider than C# decimal can produce). + +using System.Text; +using System.Text.Json; +using MetaObjects.IntegrationTests.Runner; +using Xunit; + +namespace MetaObjects.IntegrationTests.Api; + +public sealed class DecimalWireGeneratedApiTests +{ + private const string Meta = """ + { "metadata.root": { "package": "acme::money", "children": [ + { "object.entity": { "name": "Ledger", "children": [ + { "source.rdb": { "@table": "ledgers" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "label" } }, + { "field.decimal": { "name": "amount", "@required": true, "@filterable": true, "@precision": 28, "@scale": 10 } }, + { "field.decimal": { "name": "rate", "@precision": 12, "@scale": 4 } }, + { "identity.primary": { "name": "pk", "@fields": "id", "@generation": "increment" } } + ] } } + ] } } + """; + + private const string SqliteSchema = """CREATE TABLE "ledgers" ("id" INTEGER PRIMARY KEY AUTOINCREMENT, "label" TEXT, "amount" TEXT NOT NULL, "rate" TEXT)"""; + private const string PgSchema = """CREATE TABLE "ledgers" ("id" BIGSERIAL PRIMARY KEY, "label" TEXT, "amount" NUMERIC(60,10) NOT NULL, "rate" NUMERIC(12,4))"""; + + private static async Task Start(ScenarioEngine engine, string conn) + { + await engine.ExecuteScriptAsync(conn, engine == ScenarioEngine.Sqlite ? SqliteSchema : PgSchema); + var path = Path.Combine(Path.GetTempPath(), "decimal-wire-" + Guid.NewGuid().ToString("N") + ".json"); + await File.WriteAllTextAsync(path, Meta); + try { return await GeneratedRoutesServer.StartAsync(path, "MetaObjects.DecimalWire.Generated", engine, conn); } + finally { File.Delete(path); } + } + + private static async Task<(int Status, string Body)> Send(HttpClient c, string method, string path, string? json = null) + { + var req = new HttpRequestMessage(new HttpMethod(method), path); + if (json is not null) req.Content = new StringContent(json, Encoding.UTF8, "application/json"); + var res = await c.SendAsync(req); + return ((int)res.StatusCode, await res.Content.ReadAsStringAsync()); + } + + private static string Field(string body, string name) => + JsonDocument.Parse(body).RootElement.GetProperty(name).GetRawText(); + + [Fact] + public async Task W1_W2_on_sqlite() + { + using var db = new SqliteTempDatabase("mo-decimal-wire-"); + await using var server = await Start(ScenarioEngine.Sqlite, $"Data Source={db.FilePath}"); + using var c = new HttpClient { BaseAddress = new Uri(server.BaseUrl) }; + + // W2: a string is exact, a number is accepted; W1: both come back as the canonical string. + var (s1, b1) = await Send(c, "POST", "/api/ledgers", """{"label":"a","amount":"12345678901234567.89"}"""); + Assert.Equal(201, s1); + Assert.Equal("\"12345678901234567.89\"", Field(b1, "amount")); + Assert.Equal("null", Field(b1, "rate")); // key present, null + var (s2, b2) = await Send(c, "POST", "/api/ledgers", """{"label":"b","amount":12.50,"rate":"0.4000"}"""); + Assert.Equal(201, s2); + Assert.Equal("\"12.5\"", Field(b2, "amount")); + Assert.Equal("\"0.4\"", Field(b2, "rate")); + var (_, list) = await Send(c, "GET", "/api/ledgers"); + Assert.Contains("\"amount\":\"12345678901234567.89\"", list); + Assert.Contains("\"amount\":\"12.5\"", list); + + // W2: garbage and W3: out of range are 400 validation, never a stored value or a 500. + foreach (var bad in new[] { "\"\"", "\" 1.5\"", "\"1,5\"", "\"abc\"", "\"NaN\"", "\"0x10\"", "true", "[1]", "\"1e30\"", "\"1e-30\"" }) + { + var (st, body) = await Send(c, "POST", "/api/ledgers", $"{{\"label\":\"x\",\"amount\":{bad}}}"); + Assert.True(st == 400, $"POST amount={bad}: expected 400, got {st} {body}"); + Assert.Contains("validation", body); + } + + // PATCH goes through the same grammar. + var (sp, bp) = await Send(c, "PATCH", "/api/ledgers/1", """{"amount":"0.10"}"""); + Assert.Equal(200, sp); + Assert.Equal("\"0.1\"", Field(bp, "amount")); + var (sb, bb) = await Send(c, "PATCH", "/api/ledgers/1", """{"amount":"abc"}"""); + Assert.Equal(400, sb); + Assert.Contains("validation", bb); + var (sn, bn) = await Send(c, "PATCH", "/api/ledgers/2", """{"rate":null}"""); + Assert.Equal(200, sn); + Assert.Equal("null", Field(bn, "rate")); + + // W2 filter operand: exact, and a malformed one is 400 invalid_filter_value naming the field. + var (sf, bf) = await Send(c, "GET", "/api/ledgers?filter%5Bamount%5D=12.50"); + Assert.Equal(200, sf); + Assert.Contains("\"label\":\"b\"", bf); + Assert.DoesNotContain("\"label\":\"a\"", bf); + foreach (var bad in new[] { "abc", "0x10", "%201", "NaN", "1e30" }) + { + var (sx, bx) = await Send(c, "GET", $"/api/ledgers?filter%5Bamount%5D%5Beq%5D={bad}"); + Assert.True(sx == 400, $"filter operand {bad}: expected 400, got {sx} {bx}"); + Assert.Contains("invalid_filter_value", bx); + Assert.Contains("\"field\":\"amount\"", bx); + } + } + + [Fact] + public async Task W4_a_sqlite_read_of_a_double_is_exact_to_15_digits() + { + using var db = new SqliteTempDatabase("mo-decimal-w4-"); + var conn = $"Data Source={db.FilePath}"; + await using var c = await ScenarioEngine.Sqlite.OpenConnectionAsync(conn); + await using var cmd = c.CreateCommand(); + cmd.CommandText = "SELECT 0.1 + 0.2, 1.0 / 3, 123456789012345680.0, 12.5"; + await using var r = await cmd.ExecuteReaderAsync(); + Assert.True(await r.ReadAsync()); + Assert.Equal("0.3", MetaObjects.Codegen.Runtime.DecimalWire.Format(r.GetDecimal(0))); + Assert.Equal("0.333333333333333", MetaObjects.Codegen.Runtime.DecimalWire.Format(r.GetDecimal(1))); + Assert.Equal("123456789012346000", MetaObjects.Codegen.Runtime.DecimalWire.Format(r.GetDecimal(2))); + Assert.Equal("12.5", MetaObjects.Codegen.Runtime.DecimalWire.Format(r.GetDecimal(3))); + // ...which is what the helper's own W4 rendering says for the same doubles. + Assert.Equal(r.GetDecimal(0), MetaObjects.Codegen.Runtime.DecimalWire.FromDouble15(0.1 + 0.2)); + } + + [Fact] + public async Task Postgres_compares_exactly_and_names_a_read_beyond_decimal_range() + { + await using var pg = await PostgresContainer.StartAsync(); + await using var server = await Start(ScenarioEngine.Postgres, pg.ConnectionString); + using var c = new HttpClient { BaseAddress = new Uri(server.BaseUrl) }; + + await Send(c, "POST", "/api/ledgers", """{"label":"lo","amount":"12345678901234567.89"}"""); + await Send(c, "POST", "/api/ledgers", """{"label":"hi","amount":"12345678901234568"}"""); + // A double cannot tell these two apart; the comparison must. + var (_, gt) = await Send(c, "GET", "/api/ledgers?filter%5Bamount%5D%5Bgt%5D=12345678901234567.89"); + Assert.Contains("\"label\":\"hi\"", gt); + Assert.DoesNotContain("\"label\":\"lo\"", gt); + var (_, eq) = await Send(c, "GET", "/api/ledgers?filter%5Bamount%5D=12345678901234567.8900000000"); + Assert.Contains("\"label\":\"lo\"", eq); + Assert.DoesNotContain("\"label\":\"hi\"", eq); + var (sbad, bbad) = await Send(c, "GET", "/api/ledgers?filter%5Bamount%5D%5Bgt%5D=abc"); + Assert.Equal(400, sbad); + Assert.Contains("invalid_filter_value", bbad); + + // W3, read: a numeric wider than C# decimal is a NAMED error, not a stack trace. + await ScenarioEngine.Postgres.ExecuteScriptAsync(pg.ConnectionString, + """INSERT INTO "ledgers" ("label","amount") VALUES ('huge', 123456789012345678901234567890123456.5)"""); + var (sh, bh) = await Send(c, "GET", "/api/ledgers"); + Assert.Equal(500, sh); + Assert.Contains("decimal_out_of_range", bh); + Assert.DoesNotContain(" at ", bh); + } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/EntityArraySqliteTests.cs b/server/csharp/MetaObjects.IntegrationTests/Api/EntityArraySqliteTests.cs new file mode 100644 index 000000000..dc07eb7a3 --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Api/EntityArraySqliteTests.cs @@ -0,0 +1,67 @@ +// EntityArraySqliteTests — a writable entity with scalar array fields (`field.string isArray`, +// `field.decimal isArray`) through the generated routes on SQLite: POST, GET and PATCH. EF Core 8 on +// SQLite materializes a primitive collection only through IList, so the entity's property type +// must be one it can read; a decimal element still crosses the wire as a canonical string. + +using System.Text; +using System.Text.Json; +using MetaObjects.IntegrationTests.Runner; +using Xunit; + +namespace MetaObjects.IntegrationTests.Api; + +public sealed class EntityArraySqliteTests +{ + private const string Meta = """ + { "metadata.root": { "package": "acme::bags", "children": [ + { "object.entity": { "name": "Bag", "children": [ + { "source.rdb": { "@table": "bags" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "tags", "isArray": true } }, + { "field.decimal": { "name": "prices", "isArray": true, "@precision": 12, "@scale": 4 } }, + { "identity.primary": { "name": "pk", "@fields": "id", "@generation": "increment" } } + ] } } + ] } } + """; + + private const string Schema = """CREATE TABLE "bags" ("id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, "tags" TEXT, "prices" TEXT);"""; + + private static async Task<(int Status, string Body)> Send(HttpClient c, string method, string path, string? json = null) + { + var req = new HttpRequestMessage(new HttpMethod(method), path); + if (json is not null) req.Content = new StringContent(json, Encoding.UTF8, "application/json"); + var res = await c.SendAsync(req); + return ((int)res.StatusCode, await res.Content.ReadAsStringAsync()); + } + + [Fact] + public async Task Scalar_arrays_round_trip_through_post_get_and_patch_on_sqlite() + { + using var db = new SqliteTempDatabase("mo-entity-array-"); + var conn = $"Data Source={db.FilePath}"; + await ScenarioEngine.Sqlite.ExecuteScriptAsync(conn, Schema); + var path = Path.Combine(Path.GetTempPath(), "entity-array-" + Guid.NewGuid().ToString("N") + ".json"); + await File.WriteAllTextAsync(path, Meta); + GeneratedRoutesServer server; + try { server = await GeneratedRoutesServer.StartAsync(path, "MetaObjects.EntityArray.Generated", ScenarioEngine.Sqlite, conn); } + finally { File.Delete(path); } + await using var _ = server; + using var c = new HttpClient { BaseAddress = new Uri(server.BaseUrl) }; + + var (s1, b1) = await Send(c, "POST", "/api/bags", """{"tags":["a","b"],"prices":["1.5","2.2500",3]}"""); + Assert.True(s1 == 201, $"POST {s1} {b1}"); + var id = JsonDocument.Parse(b1).RootElement.GetProperty("id").GetInt64(); + + var (s2, b2) = await Send(c, "GET", "/api/bags"); + Assert.True(s2 == 200, $"GET {s2} {b2}"); + Assert.Contains("\"tags\":[\"a\",\"b\"]", b2); + Assert.Contains("\"prices\":[\"1.5\",\"2.25\",\"3\"]", b2); + + var (s3, b3) = await Send(c, "PATCH", $"/api/bags/{id}", """{"tags":["z"],"prices":["0.5"]}"""); + Assert.True(s3 == 200, $"PATCH {s3} {b3}"); + var (s4, b4) = await Send(c, "GET", $"/api/bags/{id}"); + Assert.Equal(200, s4); + Assert.Contains("\"tags\":[\"z\"]", b4); + Assert.Contains("\"prices\":[\"0.5\"]", b4); + } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/GeneratedRoutesServer.cs b/server/csharp/MetaObjects.IntegrationTests/Api/GeneratedRoutesServer.cs new file mode 100644 index 000000000..d79f01011 --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Api/GeneratedRoutesServer.cs @@ -0,0 +1,121 @@ +// GeneratedRoutesServer — host the GENERATED routes of any small model over Kestrel. +// +// The decimal wire gate (ADR-0019 Amendment 1) needs a model of its own, and the shared +// api-contract `decimal/` sub-corpus will need the same machinery: this is the part of +// ReportGeneratedServerFactory that is not specific to the report corpus. Entity + +// DbContext + FilterAllowlist + Routes + Names are run, Roslyn-compiled in memory, and every +// MapRoutes is mounted at /api. No JSON option is configured on the host, so what is +// observed is what an adopter who mounts the routes gets. + +using System.Reflection; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.CodeAnalysis; +using Microsoft.CodeAnalysis.CSharp; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using MetaObjects.Codegen; +using MetaObjects.Codegen.Generators; +using MetaObjects.IntegrationTests.Runner; +using MetaObjects.Loader; +using MetaObjects.Meta; + +namespace MetaObjects.IntegrationTests.Api; + +internal sealed class GeneratedRoutesServer : IAsyncDisposable +{ + private readonly WebApplication _app; + + public string BaseUrl { get; } + + private GeneratedRoutesServer(WebApplication app, string baseUrl) { _app = app; BaseUrl = baseUrl; } + + /// The model's metadata file. + /// Namespace of the generated code. + /// Database engine the host uses. + /// Connection string; the caller has already created the schema. + public static async Task StartAsync( + string metaJsonPath, string ns, ScenarioEngine engine, string connectionString) + { + var load = new MetaDataLoader().Load([new FileSource(metaJsonPath)]); + if (load.Errors.Count != 0) + throw new InvalidOperationException("metadata failed to load: " + string.Join("; ", load.Errors.Select(e => e.ToString()))); + var root = load.Root; + root.PhysicalNaming = ApiContractCorpusPaths.Naming; + var ctx = new GenContext + { + Entities = root.Objects(), + Root = root, + Config = new GenConfig + { + OutDir = "/unused", + Namespace = ns, + ColumnNamingStrategy = ApiContractCorpusPaths.Naming.Columns, + EmitAbstractShapes = false, + IncludeNames = true, + }, + }; + var files = new EntityGenerator().Generate(ctx) + .Concat(new DbContextGenerator().Generate(ctx)) + .Concat(new FilterAllowlistGenerator().Generate(ctx)) + .Concat(new RoutesGenerator().Generate(ctx)) + .Concat(new NamesGenerator().Generate(ctx)) + .ToList(); + var routedNames = root.Objects().Where(o => RoutesGenerator.AppliesTo(o, root)) + .Select(o => CSharpNaming.Pascal(o.Name)).ToList(); + + var comp = CSharpCompilation.Create( + "generated_routes_" + Guid.NewGuid().ToString("N"), + files.Select(f => CSharpSyntaxTree.ParseText(f.Content, new CSharpParseOptions(LanguageVersion.CSharp12))), + BuildReferenceSet(), + new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary)); + using var ms = new MemoryStream(); + var emit = comp.Emit(ms); + if (!emit.Success) + throw new InvalidOperationException("generated server failed to compile:\n " + string.Join("\n ", + emit.Diagnostics.Where(d => d.Severity == DiagnosticSeverity.Error).Select(d => $"{d.Id}: {d.GetMessage()}"))); + var assembly = Assembly.Load(ms.ToArray()); + var dbContextType = assembly.GetType($"{ns}.AppDbContext") ?? throw new InvalidOperationException("no AppDbContext"); + + var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0); + l.Start(); + int port = ((System.Net.IPEndPoint)l.LocalEndpoint).Port; + l.Stop(); + var baseUrl = $"http://127.0.0.1:{port}"; + + var builder = WebApplication.CreateBuilder(); + builder.WebHost.UseUrls(baseUrl); + builder.Logging.ClearProviders(); + var addDbContext = typeof(EntityFrameworkServiceCollectionExtensions).GetMethods(BindingFlags.Public | BindingFlags.Static) + .First(m => m.Name == "AddDbContext" && m.IsGenericMethodDefinition && m.GetGenericArguments().Length == 1 && m.GetParameters().Length == 4) + .MakeGenericMethod(dbContextType); + Action configure = opts => opts.UseEngine(engine, connectionString); + addDbContext.Invoke(null, [builder.Services, configure, ServiceLifetime.Scoped, ServiceLifetime.Scoped]); + + var app = builder.Build(); + foreach (var name in routedNames) + assembly.GetType($"{ns}.{name}Routes")!.GetMethod($"Map{name}Routes", BindingFlags.Public | BindingFlags.Static)! + .Invoke(null, [app, "/api"]); + await app.StartAsync(); + return new GeneratedRoutesServer(app, baseUrl); + } + + public async ValueTask DisposeAsync() + { + try { await _app.StopAsync(); } catch { /* ignored */ } + try { await _app.DisposeAsync(); } catch { /* ignored */ } + } + + private static List BuildReferenceSet() + { + var byFileName = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var path in ((string?)AppContext.GetData("TRUSTED_PLATFORM_ASSEMBLIES") ?? "").Split(Path.PathSeparator)) + if (path.EndsWith(".dll", StringComparison.OrdinalIgnoreCase)) byFileName[Path.GetFileName(path)] = path; + var aspNetDir = Path.GetDirectoryName(typeof(WebApplication).Assembly.Location); + if (aspNetDir is not null && Directory.Exists(aspNetDir)) + foreach (var dll in Directory.EnumerateFiles(aspNetDir, "*.dll")) byFileName[Path.GetFileName(dll)] = dll; + return byFileName.Values.Select(loc => (MetadataReference)MetadataReference.CreateFromFile(loc)).ToList(); + } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/JsonbReferenceServer.cs b/server/csharp/MetaObjects.IntegrationTests/Api/JsonbReferenceServer.cs index 9e479dbba..c7a9b3772 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/JsonbReferenceServer.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/JsonbReferenceServer.cs @@ -318,7 +318,7 @@ private static void AddJsonbParam(NpgsqlCommand cmd, string name, JsonNode? node { if (jv.TryGetValue(out var b)) return b; if (jv.TryGetValue(out var lv)) return lv; - if (jv.TryGetValue(out var dv)) return dv; + if (jv.TryGetValue(out var dv)) return dv; // exact: never a double (ADR-0019 Amendment 1) if (jv.TryGetValue(out var sv)) return sv; return jv.ToString(); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ReportFixture.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ReportFixture.cs index be9f6171d..ae5f6c63d 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/ReportFixture.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ReportFixture.cs @@ -10,9 +10,7 @@ // // The seed is `report/seed.json`. Every top-level key but `reports` is a base table, listed // parents first (`products` before `sales`, which references it), and is inserted in file -// order so every foreign key holds: the views derive the report rows from them. `reports` -// is what the seam lanes (Java, Kotlin, Python) serve in place of the views, and is not -// used here. +// order so every foreign key holds: the views derive the report rows from them. using System.Text.Json; // JsonValueKind using System.Text.Json.Nodes; // JsonNode / JsonObject / JsonArray @@ -26,13 +24,19 @@ internal static class ReportFixture private const string SeedReportsKey = "reports"; /// Execute the committed TypeScript-produced schema for this engine's artifact. - public static async Task ProvisionSchemaAsync(string connString, ScenarioEngine engine = ScenarioEngine.Postgres) + public static Task ProvisionSchemaAsync(string connString, ScenarioEngine engine = ScenarioEngine.Postgres) => + ProvisionSchemaAsync(connString, engine, + ApiContractCorpusPaths.ReportSchemaSql, ApiContractCorpusPaths.ReportSchemaSqliteSql, ApiContractCorpusPaths.ReportSchemaMysqlSql); + + /// Execute the committed schema artifact of from the given per-engine paths. + public static async Task ProvisionSchemaAsync( + string connString, ScenarioEngine engine, string postgresSql, string sqliteSql, string mysqlSql) { var path = engine switch { - ScenarioEngine.Sqlite => ApiContractCorpusPaths.ReportSchemaSqliteSql, - ScenarioEngine.MySql => ApiContractCorpusPaths.ReportSchemaMysqlSql, - _ => ApiContractCorpusPaths.ReportSchemaSql, + ScenarioEngine.Sqlite => sqliteSql, + ScenarioEngine.MySql => mysqlSql, + _ => postgresSql, }; await engine.ExecuteScriptAsync(connString, await File.ReadAllTextAsync(path)); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/ReportGeneratedServerFactory.cs b/server/csharp/MetaObjects.IntegrationTests/Api/ReportGeneratedServerFactory.cs index c434b62e8..3676dbedf 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/ReportGeneratedServerFactory.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/ReportGeneratedServerFactory.cs @@ -44,7 +44,7 @@ internal sealed class ReportGeneratedServerFactory : IAsyncDisposable // Ordinal order. InvoiceDays is in the model and declares no source. private static readonly string[] ExpectedRoutedNames = ["Category", "Invoice", "InvoiceCalendar", "InvoiceDisputes", "InvoiceStatusTotals", "InvoiceTotals", - "InvoicesByMonth", "Product", "ProductRefunds", "ProductRevenue", "Sale", "SalesByCategory", "SalesBySku"]; + "InvoicesByMonth", "Product", "ProductRefunds", "ProductRevenue", "Sale", "SalesByCategory", "SalesBySku", "Ticket", "TicketsByTags"]; private const string SourcelessReport = "InvoiceDays"; private readonly ScenarioEngine _engine; diff --git a/server/csharp/MetaObjects.IntegrationTests/Api/TphReferenceServer.cs b/server/csharp/MetaObjects.IntegrationTests/Api/TphReferenceServer.cs index 51ac57d2e..fd076bf0a 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Api/TphReferenceServer.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Api/TphReferenceServer.cs @@ -20,6 +20,7 @@ using System.Text; using System.Text.Json; using System.Text.Json.Nodes; +using MetaObjects.Codegen.Runtime; using MetaObjects.IntegrationTests.Runner; using Npgsql; @@ -120,7 +121,7 @@ private async Task DispatchAsync(HttpListenerContext ctx) // the base allowlist includes the discriminator). if (method == "GET" && segs.Length == 2) { - var (filters, error, field) = ParseBaseFilters(query); + var (filters, error, field) = ParseFilters(query, BaseFilterOps); if (error is not null) { await SendJsonAsync(ctx, 400, new Dictionary { ["error"] = error, ["field"] = field }); @@ -138,12 +139,25 @@ private async Task DispatchAsync(HttpListenerContext ctx) { if (method == "GET") { - await SendJsonAsync(ctx, 200, await ListAsync(disc, query)); + var (subFilters, subError, subField) = ParseFilters(query, SubtypeFilterOps(disc)); + if (subError is not null) + { + await SendJsonAsync(ctx, 400, new Dictionary { ["error"] = subError, ["field"] = subField }); + return; + } + await SendJsonAsync(ctx, 200, await ListAsync(disc, query, subFilters)); return; } if (method == "POST") { var body = await ReadBodyAsync(ctx); + // ADR-0019 Amendment 1 (W3): copayAmount is decimal(10,2); a value it would + // round or overflow is a 400 validation, never stored rounded. + if (!CopayAmountFits(body)) + { + await SendJsonAsync(ctx, 400, new Dictionary { ["error"] = "validation" }); + return; + } // FR-036 — the per-subtype CREATE validates the present-value field // constraints, matching the generated controller's Dto DataAnnotations // (and the vanilla POST + the per-subtype PATCH). `reference` is the base @@ -186,7 +200,7 @@ private async Task DispatchAsync(HttpListenerContext ctx) } // FR-036 — a present value exceeding @maxLength:80 (reference/approver) is a // 400, matching the generated controller's per-present-value PATCH validation. - if (ExceedsMaxLength(body)) + if (ExceedsMaxLength(body) || !CopayAmountFits(body)) { await SendJsonAsync(ctx, 400, new Dictionary { ["error"] = "validation" }); return; @@ -226,10 +240,22 @@ private async Task DispatchAsync(HttpListenerContext ctx) ["reference"] = new[] { "eq", "ne", "in" }, }; + // A per-subtype collection filters by the base's columns plus that subtype's own. Only + // CopayAuth.copayAmount is filterable among the subtype columns the corpus exercises; a + // decimal operand is compared exactly (ADR-0019 Amendment 1, W2). + private static IReadOnlyDictionary SubtypeFilterOps(string discriminator) + { + var ops = new Dictionary(BaseFilterOps, StringComparer.Ordinal); + ops.Remove("type"); + if (discriminator == "Copay") ops["copayAmount"] = new[] { "eq", "ne", "gt", "gte", "lt", "lte", "in" }; + return ops; + } + private sealed record BaseFilter(string Field, string Op, string[] Values); - private static (List Filters, string? Error, string? Field) ParseBaseFilters( - System.Collections.Specialized.NameValueCollection query) + private static (List Filters, string? Error, string? Field) ParseFilters( + System.Collections.Specialized.NameValueCollection query, + IReadOnlyDictionary filterOps) { var filters = new List(); foreach (string? key in query.AllKeys) @@ -239,11 +265,12 @@ private static (List Filters, string? Error, string? Field) ParseBas if (!m.Success) continue; string field = m.Groups[1].Value; string op = m.Groups[2].Success ? m.Groups[2].Value : "eq"; - if (!BaseFilterOps.TryGetValue(field, out var ops)) return (filters, "invalid_filter_field", field); + if (!filterOps.TryGetValue(field, out var ops)) return (filters, "invalid_filter_field", field); if (!ops.Contains(op)) return (filters, "invalid_filter_op", field); string raw = query[key] ?? ""; string[] values = op == "in" ? raw.Split(',') : new[] { raw }; if (field == "id" && values.Any(v => !long.TryParse(v, out _))) return (filters, "invalid_filter_value", field); + if (field == "copayAmount" && values.Any(v => !DecimalWire.TryParse(v, out _))) return (filters, "invalid_filter_value", field); filters.Add(new BaseFilter(field, op, values)); } return (filters, null, null); @@ -260,7 +287,12 @@ private static (List Filters, string? Error, string? Field) ParseBas if (discriminator is not null) { where.Add("\"type\" = @disc"); binds.Add(("@disc", discriminator)); } foreach (var f in filters ?? Array.Empty()) { - object Bind(string v) => f.Field == "id" ? long.Parse(v) : v; + object Bind(string v) => f.Field switch + { + "id" => long.Parse(v), + "copayAmount" => DecimalWire.TryParse(v, out var d) ? d : throw new FormatException(v), + _ => v, + }; var names = new List(); foreach (var v in f.Values) { @@ -269,9 +301,10 @@ private static (List Filters, string? Error, string? Field) ParseBas names.Add(name); } string sqlOp = f.Op switch { "eq" => "=", "ne" => "<>", "gt" => ">", "gte" => ">=", "lt" => "<", "lte" => "<=", _ => "IN" }; + string column = f.Field; where.Add(f.Op == "in" - ? $"\"{f.Field}\" IN ({string.Join(", ", names)})" - : $"\"{f.Field}\" {sqlOp} {names[0]}"); + ? $"\"{column}\" IN ({string.Join(", ", names)})" + : $"\"{column}\" {sqlOp} {names[0]}"); } if (where.Count > 0) sql.Append(" WHERE ").Append(string.Join(" AND ", where)); // ?sort=field:dir — only the corpus's `id` sort is exercised; gate to known columns. @@ -415,22 +448,39 @@ private static bool IsOverLength(JsonObject body, string col) => private static object CoerceColumn(string col, JsonNode v) => col switch { "quantity" => v.GetValue(), - "copayAmount" => v.GetValueKind() == JsonValueKind.String - ? decimal.Parse(v.GetValue(), CultureInfo.InvariantCulture) - : v.GetValue(), + "copayAmount" => ReadDecimal(v) ?? throw new FormatException("copayAmount"), _ => v.GetValue(), }; + // A decimal arrives as a string or a JSON number, read exactly (ADR-0019 Amendment 1, W2). + private static decimal? ReadDecimal(JsonNode v) => + v is JsonValue jv && (jv.TryGetValue(out var text) + ? DecimalWire.TryParse(text, out var d) + : DecimalWire.TryParse(jv.ToJsonString(), out d)) + ? d + : null; + + // W3: copayAmount is decimal(10,2); a value the column would round or overflow is refused + // rather than stored rounded. Absent or null is the caller's business, not this check's. + private static bool CopayAmountFits(JsonObject body) + { + if (!body.TryGetPropertyValue("copayAmount", out var node) || node is null + || node.GetValueKind() == JsonValueKind.Null) return true; + return ReadDecimal(node) is { } d && DecimalWire.Fits(d, 10, 2); + } + // Map a DB row into the wire dictionary. `type` is the discriminator value (text); - // integer/string columns pass through; copayAmount surfaces as a number for the - // wire (the corpus does NOT assert decimals over HTTP). + // integer/string columns pass through; a decimal is its canonical string (W1), as the + // generated controller sends it. private static Dictionary RowFrom(NpgsqlDataReader rdr) { var d = new Dictionary(StringComparer.Ordinal); for (int i = 0; i < rdr.FieldCount; i++) { var name = rdr.GetName(i); - d[name] = rdr.IsDBNull(i) ? null : rdr.GetValue(i); + d[name] = rdr.IsDBNull(i) ? null + : rdr.GetValue(i) is decimal dec ? DecimalWire.Format(dec) + : rdr.GetValue(i); } return d; } diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/AllTypes.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/AllTypes.g.cs index 3a0b4717d..f7d31c026 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Generated/AllTypes.g.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/AllTypes.g.cs @@ -32,6 +32,7 @@ public enum AllTypesIntEnumVal { DRAFT, PUBLISHED, ARCHIVED } [Column(AllTypesNames.FValColumn)] public float FVal { get; set; } [Column(AllTypesNames.DecValColumn)] + [DecimalWire.Declared(18, 6)] public decimal DecVal { get; set; } [Column(AllTypesNames.BValColumn)] public bool BVal { get; set; } diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/AllTypesBreakdown.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/AllTypesBreakdown.g.cs index 3e2b4c241..1d344d1ba 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Generated/AllTypesBreakdown.g.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/AllTypesBreakdown.g.cs @@ -27,6 +27,7 @@ public enum AllTypesBreakdownLegacyStatus { DRAFT, PUBLISHED, ARCHIVED } [Column("totalInts")] public long? TotalInts { get; set; } [Column("avgDec")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.NullableDecimalJsonConverter))] public decimal? AvgDec { get; set; } [Column("maxMoney")] public long? MaxMoney { get; set; } diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/AppDbContext.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/AppDbContext.g.cs index 8a9924462..0e7ed14de 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Generated/AppDbContext.g.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/AppDbContext.g.cs @@ -16,10 +16,15 @@ public AppDbContext(DbContextOptions options) : base(options) { } public DbSet AssetDays { get; set; } = default!; public DbSet Auths { get; set; } = default!; public DbSet AuthsByKinds { get; set; } = default!; + public DbSet Entries { get; set; } = default!; + public DbSet EntryTotals { get; set; } = default!; public DbSet FitnessTotals { get; set; } = default!; public DbSet FitnessTotalsFilleds { get; set; } = default!; public DbSet Follows { get; set; } = default!; public DbSet Friendships { get; set; } = default!; + public DbSet Gadgets { get; set; } = default!; + public DbSet GadgetEvents { get; set; } = default!; + public DbSet GadgetEventsByLabels { get; set; } = default!; public DbSet LiveIntroPrograms { get; set; } = default!; public DbSet Measurements { get; set; } = default!; public DbSet MeasurementStats { get; set; } = default!; @@ -59,8 +64,10 @@ protected override void OnModelCreating(ModelBuilder modelBuilder) modelBuilder.Entity().HasNoKey().ToView("v_asset_days"); modelBuilder.Entity().HasNoKey().ToView("v_auths_by_kind"); modelBuilder.Entity().Property(x => x.Kind).HasConversion(); + modelBuilder.Entity().HasNoKey().ToView("v_entry_totals"); modelBuilder.Entity().HasNoKey().ToView("v_fitness_totals"); modelBuilder.Entity().HasNoKey().ToView("v_fitness_totals_filled"); + modelBuilder.Entity().HasNoKey().ToView("v_gadget_events_by_label"); modelBuilder.Entity().HasNoKey().ToView("v_live_intro_programs"); modelBuilder.Entity().Property(x => x.Status).HasConversion(); modelBuilder.Entity().HasNoKey().ToView("v_measurement_stats"); @@ -99,10 +106,12 @@ protected override void OnModelCreating(ModelBuilder modelBuilder) modelBuilder.Entity().Property(x => x.Payload).HasColumnType("jsonb"); modelBuilder.Entity().Property(x => x.Type).HasConversion(); modelBuilder.Entity().HasDiscriminator(e => e.Type).HasValue(Auth.AuthType.Bridge).HasValue(Auth.AuthType.Copay).HasValue(Auth.AuthType.PriorAuth).HasValue(Auth.AuthType.Referral); + modelBuilder.Entity().Property(x => x.Amount).HasPrecision(28, 10); modelBuilder.Entity().HasOne().WithMany().HasForeignKey(e => e.FollowerId).OnDelete(DeleteBehavior.NoAction); modelBuilder.Entity().HasOne().WithMany().HasForeignKey(e => e.FolloweeId).OnDelete(DeleteBehavior.NoAction); modelBuilder.Entity().HasOne().WithMany().HasForeignKey(e => e.PersonAId).OnDelete(DeleteBehavior.NoAction); modelBuilder.Entity().HasOne().WithMany().HasForeignKey(e => e.PersonBId).OnDelete(DeleteBehavior.NoAction); + modelBuilder.Entity().HasOne().WithMany().HasForeignKey(e => e.GadgetId).OnDelete(DeleteBehavior.NoAction); modelBuilder.Entity().Property(x => x.PreciseKg).HasPrecision(9, 4); modelBuilder.Entity().HasOne().WithMany().HasForeignKey(e => e.ParentId).OnDelete(DeleteBehavior.NoAction); modelBuilder.Entity().HasMany(x => x.Tags).WithMany().UsingEntity(l => l.HasOne().WithMany().HasForeignKey(j => j.TagId), r => r.HasOne().WithMany().HasForeignKey(j => j.PostId)); diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/CopayAuth.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/CopayAuth.g.cs index f85031fa3..48dbdcc91 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Generated/CopayAuth.g.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/CopayAuth.g.cs @@ -11,5 +11,6 @@ namespace MetaObjects.IntegrationTests.Generated; public class CopayAuth : Auth { [Column(CopayAuthNames.CopayAmountColumn)] + [DecimalWire.Declared(10, 2)] public decimal? CopayAmount { get; set; } } diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/DecimalWire.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/DecimalWire.g.cs new file mode 100644 index 000000000..bcc543fb9 --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/DecimalWire.g.cs @@ -0,0 +1,272 @@ +// DecimalWire — the wire form of a field.decimal (ADR-0019 Amendment 1, W1-W4). +// +// A decimal crosses JSON as a canonical STRING: `^-?(0|[1-9][0-9]*)(\.[0-9]*[1-9])?$`, +// plain notation, no exponent, no trailing fractional zeros, zero as "0". A JSON number is +// a double to every JavaScript client, so the string is the only spelling a browser reads +// exactly. Input takes a string in the JSON-number grammar or a JSON number; anything else +// is refused. A value `decimal` cannot hold (more than 28-29 significant digits) is an +// error, never a rounded value (W3). +// +// ONE source, two homes. This file is the package's production helper (the filter +// dispatcher parses its operands through it). The entity generator emits the SAME text, +// with only the namespace changed, as `DecimalWire.g.cs` into every namespace that has a +// decimal property, so the `[JsonConverter(typeof(DecimalWire.…))]` on each property +// resolves inside the generated project and the generated entity code imports nothing +// from MetaObjects (CLAUDE.md). DecimalWireTests gates the two texts byte-for-byte. +// Keep this file free of references to anything but the BCL. + +#nullable enable +using System; +using System.Buffers; +using System.Collections.Generic; +using System.Globalization; +using System.Text; +using System.Text.Json; +using System.Text.Json.Serialization; + +namespace MetaObjects.IntegrationTests.Generated; + +/// The canonical wire form of a decimal: format, parse, and the System.Text.Json converters. +public static class DecimalWire +{ + /// W1. The canonical string of : plain notation, trailing fractional zeros stripped, zero as "0". + public static string Format(decimal value) + { + var s = value.ToString(CultureInfo.InvariantCulture); + if (s.IndexOf('.') >= 0) + { + s = s.TrimEnd('0'); + if (s.EndsWith('.')) s = s.Substring(0, s.Length - 1); + } + return s == "-0" || s.Length == 0 ? "0" : s; + } + + /// + /// W2. Parse text in the JSON-number grammar (-?(0|[1-9][0-9]*)(\.[0-9]+)?([eE][+-]?[0-9]+)?, + /// no whitespace, no sign other than -). False for anything else, and for a value + /// decimal cannot hold exactly (W3). + /// + public static bool TryParse(string? text, out decimal value) + { + value = 0m; + if (text is null) return false; + int i = 0, n = text.Length; + bool negative = false; + if (i < n && text[i] == '-') { negative = true; i++; } + if (i >= n) return false; + int intStart = i; + if (text[i] == '0') i++; + else if (text[i] >= '1' && text[i] <= '9') { while (i < n && IsDigit(text[i])) i++; } + else return false; + int intEnd = i; + int fracStart = i, fracEnd = i; + if (i < n && text[i] == '.') + { + i++; + fracStart = i; + while (i < n && IsDigit(text[i])) i++; + if (i == fracStart) return false; + fracEnd = i; + } + long exponent = 0; + if (i < n && (text[i] == 'e' || text[i] == 'E')) + { + i++; + bool expNegative = false; + if (i < n && (text[i] == '+' || text[i] == '-')) { expNegative = text[i] == '-'; i++; } + int expStart = i; + while (i < n && IsDigit(text[i])) + { + if (exponent < 100000) exponent = exponent * 10 + (text[i] - '0'); + i++; + } + if (i == expStart) return false; + if (expNegative) exponent = -exponent; + } + if (i != n) return false; + var digits = text.Substring(intStart, intEnd - intStart) + text.Substring(fracStart, fracEnd - fracStart); + return TryBuild(negative, digits, (intEnd - intStart) + exponent, out value); + } + + /// W2 for an already-parsed JSON value (a PATCH merge reads one property at a time): a string or a number. + public static bool TryParseElement(JsonElement element, out decimal value) + { + value = 0m; + return element.ValueKind switch + { + JsonValueKind.String => TryParse(element.GetString(), out value), + JsonValueKind.Number => TryParse(element.GetRawText(), out value), + _ => false, + }; + } + + /// + /// W4. The value of an IEEE double at 15 significant digits (SQLite's own text precision), + /// canonicalized. A SQLite or D1 NUMERIC column is a double, so this is the most a read of + /// one can promise. Throws beyond decimal's range. + /// + public static decimal FromDouble15(double value) + { + if (double.IsNaN(value) || double.IsInfinity(value)) + throw new OverflowException("a decimal cannot hold " + value.ToString(CultureInfo.InvariantCulture)); + var e = value.ToString("E14", CultureInfo.InvariantCulture); // d.ddddddddddddddE+ddd + bool negative = e[0] == '-'; + int ePos = e.IndexOf('E'); + var mantissa = e.Substring(negative ? 1 : 0, ePos - (negative ? 1 : 0)).Replace(".", ""); + long exponent = long.Parse(e.Substring(ePos + 1), NumberStyles.AllowLeadingSign, CultureInfo.InvariantCulture); + if (!TryBuild(negative, mantissa, 1 + exponent, out var result)) + throw new OverflowException("a decimal cannot hold " + e); + return result; + } + + /// W4 as text: of . + public static string FormatDouble15(double value) => Format(FromDouble15(value)); + + private static bool IsDigit(char c) => c >= '0' && c <= '9'; + + // digits is the unsigned digit run; the decimal point sits after pointPos of them. + private static bool TryBuild(bool negative, string digits, long pointPos, out decimal value) + { + value = 0m; + int lead = 0; + while (lead < digits.Length && digits[lead] == '0') lead++; + int end = digits.Length; + while (end > lead && digits[end - 1] == '0') end--; + if (lead == end) return true; // zero, in any spelling + var d = digits.Substring(lead, end - lead); + long point = pointPos - lead; + // decimal holds at most 29 integer digits and 28 fractional places. + if (point > 29 || point < -28) return false; + var sb = new StringBuilder(); + if (negative) sb.Append('-'); + if (point <= 0) sb.Append("0.").Append('0', (int)-point).Append(d); + else if (point >= d.Length) sb.Append(d).Append('0', (int)point - d.Length); + else sb.Append(d, 0, (int)point).Append('.').Append(d, (int)point, d.Length - (int)point); + var plain = sb.ToString(); + if (!decimal.TryParse(plain, NumberStyles.AllowLeadingSign | NumberStyles.AllowDecimalPoint, + CultureInfo.InvariantCulture, out value)) return false; + // decimal.Parse ROUNDS an excess digit; a round trip that does not reproduce the + // input means it was rounded (W3: an error, never silent rounding). + return Format(value) == plain; + } + + /// + /// W3 for a declared column: true when fits @precision and + /// @scale without rounding: no more fraction digits than , no + /// more integer digits than minus . + /// Counted on the canonical form, so trailing zeros never count. + /// + public static bool Fits(decimal value, int precision, int scale) + { + var text = Format(value); + if (text.StartsWith('-')) text = text.Substring(1); + var dot = text.IndexOf('.'); + var integerDigits = (dot < 0 ? text : text.Substring(0, dot)).TrimStart('0').Length; + var fractionDigits = dot < 0 ? 0 : text.Length - dot - 1; + return fractionDigits <= scale && integerDigits <= precision - scale; + } + + /// + /// W3 for a declared property: false when of + /// carries and does not fit it. + /// True for a property with no declared precision (the PATCH merge asks this per key). + /// + public static bool FitsDeclared(Type owner, string property, decimal value) + { + var declared = owner.GetProperty(property)?.GetCustomAttributes(typeof(DeclaredAttribute), true); + return declared is not { Length: > 0 } || ((DeclaredAttribute)declared[0]).Accepts(value); + } + + private static decimal ReadOne(ref Utf8JsonReader reader, int? precision = null, int? scale = null) + { + decimal result; + switch (reader.TokenType) + { + case JsonTokenType.String: + if (!TryParse(reader.GetString(), out result)) + throw new JsonException("expected a decimal as a JSON-number string or a JSON number"); + break; + case JsonTokenType.Number: + var raw = Encoding.UTF8.GetString(reader.HasValueSequence ? reader.ValueSequence.ToArray() : reader.ValueSpan.ToArray()); + if (!TryParse(raw, out result)) throw new JsonException("decimal out of range"); + break; + default: + throw new JsonException("expected a decimal as a JSON-number string or a JSON number"); + } + if (precision is int p && scale is int sc && !Fits(result, p, sc)) + throw new JsonException("decimal exceeds the declared precision and scale"); + return result; + } + + /// + /// A decimal property with a declared @precision and @scale: the same wire form + /// as the plain converters, and a value the column cannot hold is refused, never rounded (W3). + /// + [AttributeUsage(AttributeTargets.Property)] + public sealed class DeclaredAttribute(int precision, int scale) : JsonConverterAttribute + { + public int Precision { get; } = precision; + public int Scale { get; } = scale; + + public bool Accepts(decimal value) => Fits(value, Precision, Scale); + + public override JsonConverter? CreateConverter(Type typeToConvert) + { + if (typeToConvert == typeof(decimal)) return new DecimalJsonConverter(Precision, Scale); + if (typeToConvert == typeof(decimal?)) return new NullableDecimalJsonConverter(Precision, Scale); + if (typeToConvert == typeof(IList)) return new DecimalListJsonConverter(Precision, Scale); + return null; + } + } + + /// A decimal property: written as the canonical string, read from a string or a number. + public sealed class DecimalJsonConverter(int? precision, int? scale) : JsonConverter + { + public DecimalJsonConverter() : this(null, null) { } + + public override decimal Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) + => ReadOne(ref reader, precision, scale); + + public override void Write(Utf8JsonWriter writer, decimal value, JsonSerializerOptions options) + => writer.WriteStringValue(Format(value)); + } + + /// A nullable decimal property: null is JSON null, with the key present. + public sealed class NullableDecimalJsonConverter(int? precision, int? scale) : JsonConverter + { + public NullableDecimalJsonConverter() : this(null, null) { } + + public override bool HandleNull => true; + + public override decimal? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) + => reader.TokenType == JsonTokenType.Null ? null : ReadOne(ref reader, precision, scale); + + public override void Write(Utf8JsonWriter writer, decimal? value, JsonSerializerOptions options) + { + if (value is null) writer.WriteNullValue(); + else writer.WriteStringValue(Format(value.Value)); + } + } + + /// An @isArray decimal property, typed IList<decimal> (the one collection shape EF Core 8 on SQLite can materialize): an array of canonical strings. + public sealed class DecimalListJsonConverter(int? precision, int? scale) : JsonConverter> + { + public DecimalListJsonConverter() : this(null, null) { } + + public override IList? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) + { + if (reader.TokenType == JsonTokenType.Null) return null; + if (reader.TokenType != JsonTokenType.StartArray) throw new JsonException("expected an array of decimals"); + var list = new List(); + while (reader.Read() && reader.TokenType != JsonTokenType.EndArray) list.Add(ReadOne(ref reader, precision, scale)); + return list; + } + + public override void Write(Utf8JsonWriter writer, IList value, JsonSerializerOptions options) + { + writer.WriteStartArray(); + foreach (var d in value) writer.WriteStringValue(Format(d)); + writer.WriteEndArray(); + } + } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/Entry.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/Entry.g.cs new file mode 100644 index 000000000..68b3966af --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/Entry.g.cs @@ -0,0 +1,27 @@ +// +// Generated by MetaObjects entity-generator. Do not edit by hand. +#nullable enable +using System; +using System.Collections.Generic; +using System.ComponentModel.DataAnnotations; +using System.ComponentModel.DataAnnotations.Schema; + +namespace MetaObjects.IntegrationTests.Generated; + +[Table(EntryNames.SourcePrimaryTable)] +public class Entry +{ + [Key] + [Column(EntryNames.IdColumn)] + public long Id { get; set; } + [Column(EntryNames.BucketColumn)] + [Required(AllowEmptyStrings = true)] + [MaxLength(8)] + [MinLength(1)] + public string Bucket { get; set; } = default!; + [Column(EntryNames.AmountColumn)] + [DecimalWire.Declared(28, 10)] + public decimal? Amount { get; set; } + [Column(EntryNames.FlaggedColumn)] + public bool Flagged { get; set; } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/EntryNames.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/EntryNames.g.cs new file mode 100644 index 000000000..65edd7ab0 --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/EntryNames.g.cs @@ -0,0 +1,42 @@ +// +// Generated by MetaObjects names-generator. Do not edit by hand. +#nullable enable +using System.Collections.Generic; + +namespace MetaObjects.IntegrationTests.Generated; + +/// +/// GENERATED — per-object physical database names for Entry (spec A1/A2/A6). +/// +public abstract class EntryNames +{ + public const string Type = "object"; + public const string SubType = "entity"; + public const string Name = "Entry"; + + public const string SourcePrimaryType = "source"; + public const string SourcePrimarySubType = "rdb"; + public const string SourcePrimaryKind = "table"; + public const string SourcePrimaryTable = "entries"; + + public const string AmountField = "amount"; + public const string AmountColumn = "amount"; + public const string BucketField = "bucket"; + public const string BucketColumn = "bucket"; + public const string FlaggedField = "flagged"; + public const string FlaggedColumn = "flagged"; + public const string IdField = "id"; + public const string IdColumn = "id"; + + public const string IdentityPkType = "identity"; + public const string IdentityPkSubType = "primary"; + public const string IdentityPkName = "pk"; + + public static readonly Dictionary ColumnsByField = new(System.StringComparer.Ordinal) + { + ["amount"] = AmountColumn, + ["bucket"] = BucketColumn, + ["flagged"] = FlaggedColumn, + ["id"] = IdColumn, + }; +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/EntryTotals.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/EntryTotals.g.cs new file mode 100644 index 000000000..0e40146f5 --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/EntryTotals.g.cs @@ -0,0 +1,38 @@ +// +// Generated by MetaObjects entity-generator. Do not edit by hand. +#nullable enable +using System; +using System.Collections.Generic; +using System.ComponentModel.DataAnnotations; +using System.ComponentModel.DataAnnotations.Schema; + +namespace MetaObjects.IntegrationTests.Generated; + +public class EntryTotals +{ + [Column("bucket")] + [Required(AllowEmptyStrings = true)] + [MaxLength(8)] + [MinLength(1)] + public string Bucket { get; set; } = default!; + [Column("entries")] + public long Entries { get; set; } + [Column("total")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.NullableDecimalJsonConverter))] + public decimal? Total { get; set; } + [Column("totalOrZero")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.DecimalJsonConverter))] + public decimal TotalOrZero { get; set; } + [Column("mean")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.NullableDecimalJsonConverter))] + public decimal? Mean { get; set; } + [Column("top")] + [DecimalWire.Declared(28, 10)] + public decimal? Top { get; set; } + [Column("flaggedShare")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.NullableDecimalJsonConverter))] + public decimal? FlaggedShare { get; set; } + [Column("perFlagged")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.NullableDecimalJsonConverter))] + public decimal? PerFlagged { get; set; } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/FitnessTotals.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/FitnessTotals.g.cs index 63c89ca40..3114346de 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Generated/FitnessTotals.g.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/FitnessTotals.g.cs @@ -15,5 +15,6 @@ public class FitnessTotals [Column("totalMinutes")] public long? TotalMinutes { get; set; } [Column("longShare")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.NullableDecimalJsonConverter))] public decimal? LongShare { get; set; } } diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/FitnessTotalsFilled.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/FitnessTotalsFilled.g.cs index 3f9d9007f..3cf2862eb 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Generated/FitnessTotalsFilled.g.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/FitnessTotalsFilled.g.cs @@ -15,5 +15,6 @@ public class FitnessTotalsFilled [Column("totalMinutesOrZero")] public long TotalMinutesOrZero { get; set; } [Column("longShareOrZero")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.DecimalJsonConverter))] public decimal LongShareOrZero { get; set; } } diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/Gadget.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/Gadget.g.cs new file mode 100644 index 000000000..8e68272c8 --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/Gadget.g.cs @@ -0,0 +1,19 @@ +// +// Generated by MetaObjects entity-generator. Do not edit by hand. +#nullable enable +using System; +using System.Collections.Generic; +using System.ComponentModel.DataAnnotations; +using System.ComponentModel.DataAnnotations.Schema; + +namespace MetaObjects.IntegrationTests.Generated; + +[Table(GadgetNames.SourcePrimaryTable)] +public class Gadget +{ + [Key] + [Column(GadgetNames.IdColumn)] + public long Id { get; set; } + [Column(GadgetNames.LabelColumn)] + public int Label { get; set; } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEvent.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEvent.g.cs new file mode 100644 index 000000000..afd34a3fb --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEvent.g.cs @@ -0,0 +1,19 @@ +// +// Generated by MetaObjects entity-generator. Do not edit by hand. +#nullable enable +using System; +using System.Collections.Generic; +using System.ComponentModel.DataAnnotations; +using System.ComponentModel.DataAnnotations.Schema; + +namespace MetaObjects.IntegrationTests.Generated; + +[Table(GadgetEventNames.SourcePrimaryTable)] +public class GadgetEvent +{ + [Key] + [Column(GadgetEventNames.IdColumn)] + public long Id { get; set; } + [Column(GadgetEventNames.GadgetIdColumn)] + public long GadgetId { get; set; } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEventNames.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEventNames.g.cs new file mode 100644 index 000000000..7b6c3f10c --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEventNames.g.cs @@ -0,0 +1,39 @@ +// +// Generated by MetaObjects names-generator. Do not edit by hand. +#nullable enable +using System.Collections.Generic; + +namespace MetaObjects.IntegrationTests.Generated; + +/// +/// GENERATED — per-object physical database names for GadgetEvent (spec A1/A2/A6). +/// +public abstract class GadgetEventNames +{ + public const string Type = "object"; + public const string SubType = "entity"; + public const string Name = "GadgetEvent"; + + public const string SourcePrimaryType = "source"; + public const string SourcePrimarySubType = "rdb"; + public const string SourcePrimaryKind = "table"; + public const string SourcePrimaryTable = "gadget_events"; + + public const string GadgetIdField = "gadgetId"; + public const string GadgetIdColumn = "gadgetId"; + public const string IdField = "id"; + public const string IdColumn = "id"; + + public const string IdentityFkGadgetType = "identity"; + public const string IdentityFkGadgetSubType = "reference"; + public const string IdentityFkGadgetName = "fkGadget"; + public const string IdentityIdType = "identity"; + public const string IdentityIdSubType = "primary"; + public const string IdentityIdName = "id"; + + public static readonly Dictionary ColumnsByField = new(System.StringComparer.Ordinal) + { + ["gadgetId"] = GadgetIdColumn, + ["id"] = IdColumn, + }; +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEventQueries.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEventQueries.g.cs new file mode 100644 index 000000000..b2341c199 --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEventQueries.g.cs @@ -0,0 +1,24 @@ +// +// Generated by MetaObjects entity-generator (reverse FK finders, ADR-0038). Do not edit by hand. +#nullable enable +using System.Collections.Generic; +using System.Linq; +using Microsoft.EntityFrameworkCore; + +namespace MetaObjects.IntegrationTests.Generated; + +/// Reverse-relationship FK finders for GadgetEvent (ADR-0038): explicit, +/// single-query reads — no lazy navigation, no N+1. +public static class GadgetEventQueries +{ + /// The GadgetEvent rows whose GadgetId equals (one indexed query). + public static async System.Threading.Tasks.Task> FindGadgetEventsByGadget(AppDbContext db, long gadgetId) => + await db.GadgetEvents.AsNoTracking().Where(e => e.GadgetId == gadgetId).ToListAsync(); + + /// The GadgetEvent rows whose GadgetId is in (one batched IN query; anti-N+1). + public static async System.Threading.Tasks.Task> FindGadgetEventsByGadgetIn(AppDbContext db, IReadOnlyCollection gadgetIds) + { + if (gadgetIds.Count == 0) return new List(); + return await db.GadgetEvents.AsNoTracking().Where(e => gadgetIds.Contains(e.GadgetId)).ToListAsync(); + } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEventsByLabel.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEventsByLabel.g.cs new file mode 100644 index 000000000..9328e0316 --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/GadgetEventsByLabel.g.cs @@ -0,0 +1,17 @@ +// +// Generated by MetaObjects entity-generator. Do not edit by hand. +#nullable enable +using System; +using System.Collections.Generic; +using System.ComponentModel.DataAnnotations; +using System.ComponentModel.DataAnnotations.Schema; + +namespace MetaObjects.IntegrationTests.Generated; + +public class GadgetEventsByLabel +{ + [Column("gadgetLabel")] + public int GadgetLabel { get; set; } + [Column("gadgetEvents")] + public long GadgetEvents { get; set; } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/GadgetNames.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/GadgetNames.g.cs new file mode 100644 index 000000000..27581f17e --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/GadgetNames.g.cs @@ -0,0 +1,30 @@ +// +// Generated by MetaObjects names-generator. Do not edit by hand. +#nullable enable +using System.Collections.Generic; + +namespace MetaObjects.IntegrationTests.Generated; + +/// +/// GENERATED — per-object physical database names for Gadget (spec A1/A2/A6). +/// +public abstract class GadgetNames : LabeledNames +{ + public new const string Type = "object"; + public new const string SubType = "entity"; + public new const string Name = "Gadget"; + + public const string SourcePrimaryType = "source"; + public const string SourcePrimarySubType = "rdb"; + public const string SourcePrimaryKind = "table"; + public const string SourcePrimaryTable = "gadgets"; + + public new const string LabelField = "label"; + public new const string LabelColumn = "gadget_code"; + + public static new readonly Dictionary ColumnsByField = new(System.StringComparer.Ordinal) + { + ["id"] = IdColumn, + ["label"] = LabelColumn, + }; +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/LabeledNames.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/LabeledNames.g.cs new file mode 100644 index 000000000..4062b333c --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/LabeledNames.g.cs @@ -0,0 +1,31 @@ +// +// Generated by MetaObjects names-generator. Do not edit by hand. +#nullable enable +using System.Collections.Generic; + +namespace MetaObjects.IntegrationTests.Generated; + +/// +/// GENERATED — per-object physical database names for Labeled (spec A1/A2/A6). +/// +public abstract class LabeledNames +{ + public const string Type = "object"; + public const string SubType = "entity"; + public const string Name = "Labeled"; + + public const string IdField = "id"; + public const string IdColumn = "id"; + public const string LabelField = "label"; + public const string LabelColumn = "label_text"; + + public const string IdentityIdType = "identity"; + public const string IdentityIdSubType = "primary"; + public const string IdentityIdName = "id"; + + public static readonly Dictionary ColumnsByField = new(System.StringComparer.Ordinal) + { + ["id"] = IdColumn, + ["label"] = LabelColumn, + }; +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/Measurement.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/Measurement.g.cs index 486967a74..e573199fc 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Generated/Measurement.g.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/Measurement.g.cs @@ -19,5 +19,6 @@ public class Measurement [Column(MeasurementNames.MassKgColumn)] public double MassKg { get; set; } [Column(MeasurementNames.PreciseKgColumn)] + [DecimalWire.Declared(9, 4)] public decimal PreciseKg { get; set; } } diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/MeasurementStats.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/MeasurementStats.g.cs index fe138355a..7ae92122b 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Generated/MeasurementStats.g.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/MeasurementStats.g.cs @@ -21,10 +21,13 @@ public class MeasurementStats [Column("minTempC")] public float? MinTempC { get; set; } [Column("totalPreciseKg")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.NullableDecimalJsonConverter))] public decimal? TotalPreciseKg { get; set; } [Column("avgPreciseKg")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.NullableDecimalJsonConverter))] public decimal? AvgPreciseKg { get; set; } [Column("maxPreciseKg")] + [DecimalWire.Declared(9, 4)] public decimal? MaxPreciseKg { get; set; } [Column("heavyReadings")] public long HeavyReadings { get; set; } diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/MeasurementStatsFilled.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/MeasurementStatsFilled.g.cs index d94e29b25..95828fb81 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Generated/MeasurementStatsFilled.g.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/MeasurementStatsFilled.g.cs @@ -13,7 +13,9 @@ public class MeasurementStatsFilled [Column("readings")] public long Readings { get; set; } [Column("avgPreciseKgOrMinusOne")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.DecimalJsonConverter))] public decimal AvgPreciseKgOrMinusOne { get; set; } [Column("maxPreciseKgOrZero")] + [DecimalWire.Declared(9, 4)] public decimal MaxPreciseKgOrZero { get; set; } } diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramMinutes.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramMinutes.g.cs index 42f262166..9e97193eb 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramMinutes.g.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramMinutes.g.cs @@ -28,11 +28,13 @@ public class ProgramMinutes [Column("totalMinutes")] public long? TotalMinutes { get; set; } [Column("avgMinutes")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.NullableDecimalJsonConverter))] public decimal? AvgMinutes { get; set; } [Column("minMinutes")] public int? MinMinutes { get; set; } [Column("maxMinutes")] public int? MaxMinutes { get; set; } [Column("longShare")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.NullableDecimalJsonConverter))] public decimal? LongShare { get; set; } } diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramRoster.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramRoster.g.cs index 615a97010..3188cbf52 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramRoster.g.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramRoster.g.cs @@ -24,7 +24,9 @@ public class ProgramRoster [Column("totalMinutesOrZero")] public long TotalMinutesOrZero { get; set; } [Column("longShare")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.NullableDecimalJsonConverter))] public decimal? LongShare { get; set; } [Column("longShareOrZero")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.DecimalJsonConverter))] public decimal LongShareOrZero { get; set; } } diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramsEmptyScope.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramsEmptyScope.g.cs index cf7368c83..afb05b851 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramsEmptyScope.g.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramsEmptyScope.g.cs @@ -13,5 +13,6 @@ public class ProgramsEmptyScope [Column("programs")] public long Programs { get; set; } [Column("avgPriceOrMinusOne")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.DecimalJsonConverter))] public decimal AvgPriceOrMinusOne { get; set; } } diff --git a/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramsOverTime.g.cs b/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramsOverTime.g.cs index d26cdacec..4b8e1c1d5 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramsOverTime.g.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Generated/ProgramsOverTime.g.cs @@ -19,6 +19,7 @@ public class ProgramsOverTime [Column("programs")] public long Programs { get; set; } [Column("avgPrice")] + [System.Text.Json.Serialization.JsonConverter(typeof(DecimalWire.NullableDecimalJsonConverter))] public decimal? AvgPrice { get; set; } [Column("cheapest")] public long? Cheapest { get; set; } diff --git a/server/csharp/MetaObjects.IntegrationTests/QueryScenarioMySqlTests.cs b/server/csharp/MetaObjects.IntegrationTests/QueryScenarioMySqlTests.cs index a095ee322..85f293417 100644 --- a/server/csharp/MetaObjects.IntegrationTests/QueryScenarioMySqlTests.cs +++ b/server/csharp/MetaObjects.IntegrationTests/QueryScenarioMySqlTests.cs @@ -18,12 +18,12 @@ public sealed class QueryScenarioMySqlTests(MySqlServer server) { [Theory] [MemberData(nameof(Scenarios))] - public async Task Report_scenario_on_mysql(string scenarioPath) + public async Task Scenario_on_mysql(string scenarioPath) { var scenario = ScenarioLoader.LoadQuery(scenarioPath); await using var mysql = await server.CreateDatabaseAsync(); await QueryScenarioRunner.RunMySqlAsync(scenario, mysql.ConnectionString); } - public static IEnumerable Scenarios() => CorpusPaths.ReportQueryScenarios(); + public static IEnumerable Scenarios() => CorpusPaths.ScenariosFor(ScenarioEngine.MySql); } diff --git a/server/csharp/MetaObjects.IntegrationTests/QueryScenarioSqliteTests.cs b/server/csharp/MetaObjects.IntegrationTests/QueryScenarioSqliteTests.cs index d5cc5d274..e365e039a 100644 --- a/server/csharp/MetaObjects.IntegrationTests/QueryScenarioSqliteTests.cs +++ b/server/csharp/MetaObjects.IntegrationTests/QueryScenarioSqliteTests.cs @@ -16,12 +16,12 @@ public sealed class QueryScenarioSqliteTests { [Theory] [MemberData(nameof(Scenarios))] - public async Task Report_scenario_on_sqlite(string scenarioPath) + public async Task Scenario_on_sqlite(string scenarioPath) { var scenario = ScenarioLoader.LoadQuery(scenarioPath); using var db = new SqliteTempDatabase("mo-report-"); await QueryScenarioRunner.RunSqliteAsync(scenario, db.FilePath); } - public static IEnumerable Scenarios() => CorpusPaths.ReportQueryScenarios(); + public static IEnumerable Scenarios() => CorpusPaths.ScenariosFor(ScenarioEngine.Sqlite); } diff --git a/server/csharp/MetaObjects.IntegrationTests/QueryScenarioTests.cs b/server/csharp/MetaObjects.IntegrationTests/QueryScenarioTests.cs index 6921adec2..d04530d94 100644 --- a/server/csharp/MetaObjects.IntegrationTests/QueryScenarioTests.cs +++ b/server/csharp/MetaObjects.IntegrationTests/QueryScenarioTests.cs @@ -17,8 +17,5 @@ public async Task Query_scenario(string scenarioPath) await QueryScenarioRunner.RunAsync(scenario, pg); } - public static IEnumerable Scenarios() => - Directory.EnumerateFiles(CorpusPaths.QueriesDir, "*.yaml", SearchOption.TopDirectoryOnly) - .OrderBy(p => p, StringComparer.Ordinal) - .Select(p => new object[] { p }); + public static IEnumerable Scenarios() => CorpusPaths.ScenariosFor(ScenarioEngine.Postgres); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Runner/CorpusPaths.cs b/server/csharp/MetaObjects.IntegrationTests/Runner/CorpusPaths.cs index 444f90b0f..e4b67954b 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Runner/CorpusPaths.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Runner/CorpusPaths.cs @@ -19,13 +19,14 @@ internal static class CorpusPaths public static readonly string QueriesDir = Path.Combine(Corpus, "queries"); /// - /// The report-* query scenarios as xUnit member data — the only corpus subset whose - /// tables exist in the SQLite and MySQL schema artifacts. One home for the name predicate, - /// so a dialect-gated scenario cannot be silently skipped by one lane's enumeration. + /// The query scenarios runs, as xUnit member data: one home for the + /// engines: rule (see ), so a dialect-gated scenario + /// cannot be silently skipped by one lane's enumeration. /// - public static IEnumerable ReportQueryScenarios() => - Directory.EnumerateFiles(QueriesDir, "report-*.yaml", SearchOption.TopDirectoryOnly) + public static IEnumerable ScenariosFor(ScenarioEngine engine) => + Directory.EnumerateFiles(QueriesDir, "*.yaml", SearchOption.TopDirectoryOnly) .OrderBy(p => p, StringComparer.Ordinal) + .Where(p => ScenarioLoader.LoadQuery(p).RunsOn(engine)) .Select(p => new object[] { p }); /// diff --git a/server/csharp/MetaObjects.IntegrationTests/Runner/DbContextAdapter.cs b/server/csharp/MetaObjects.IntegrationTests/Runner/DbContextAdapter.cs index 955b7ce39..507086333 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Runner/DbContextAdapter.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Runner/DbContextAdapter.cs @@ -366,6 +366,12 @@ private static IQueryable ApplySort(IQueryable queryable, IReadOnlyList // date string; DateOnly is not IConvertible either. Same parse the write codec uses. if (underlying == typeof(DateOnly)) return WriteCoercion.ParseDate(raw.ToString()!); + // A decimal operand is exact: the string form (plain or exponent notation) parses through the + // shipped wire rule, never through a double. + if (underlying == typeof(decimal)) + return MetaObjects.Codegen.Runtime.DecimalWire.TryParse(raw.ToString(), out var exact) + ? exact + : throw new FormatException($"not a decimal: {raw}"); return Convert.ChangeType(raw, underlying, System.Globalization.CultureInfo.InvariantCulture); } diff --git a/server/csharp/MetaObjects.IntegrationTests/Runner/DecimalGate.cs b/server/csharp/MetaObjects.IntegrationTests/Runner/DecimalGate.cs new file mode 100644 index 000000000..ba0629114 --- /dev/null +++ b/server/csharp/MetaObjects.IntegrationTests/Runner/DecimalGate.cs @@ -0,0 +1,85 @@ +// DecimalGate — the typed decimal gate of the persistence corpus (ADR-0019 Amendment 1). +// +// A runner holds a decimal to the TYPE, not to the shape of its value: before a row is compared, +// every field the model declares decimal (an entity field, or a report field typed `decimal` in +// report-shapes.json) must be null or a decimal that normalizes to the canonical string +// ^-?(0|[1-9][0-9]*)(\.[0-9]*[1-9])?$. A double or float (a "binary float") fails the scenario +// even when its printed value would happen to match. The check runs on the value the runtime +// produced, BEFORE the comparison canonicalizes anything. + +using System.Text.Json; +using System.Text.RegularExpressions; +using MetaObjects.Loader; +using MetaObjects.Meta; +using Xunit.Sdk; +using static MetaObjects.Core.Field.FieldConstants; + +namespace MetaObjects.IntegrationTests.Runner; + +internal static partial class DecimalGate +{ + [GeneratedRegex(@"^-?(0|[1-9][0-9]*)(\.[0-9]*[1-9])?$")] + private static partial Regex Canonical(); + + /// Entity (or report) short name to the names of its decimal fields, matched case-insensitively. + private static readonly Lazy>> Declared = new(Load); + + private static IReadOnlyDictionary> Load() + { + var map = new Dictionary>(StringComparer.Ordinal); + var loaded = MetaDataLoader.FromUris([new Uri(CorpusPaths.FitnessMetadataFile)]); + if (loaded.Errors.Count > 0) + throw new InvalidOperationException( + "failed to load corpus metadata: " + string.Join("; ", loaded.Errors.Select(e => e.Message))); + foreach (var obj in loaded.Root.Objects().Where(o => !o.IsReport())) + { + var names = obj.Fields().Where(f => f.SubType == FIELD_SUBTYPE_DECIMAL) + .Select(f => f.Name).ToHashSet(StringComparer.OrdinalIgnoreCase); + if (names.Count > 0) map[obj.Name] = names; + } + using var shapes = JsonDocument.Parse(File.ReadAllText(Path.Combine(CorpusPaths.Corpus, "report-shapes.json"))); + foreach (var report in shapes.RootElement.GetProperty("reports").EnumerateArray()) + { + var fq = report.GetProperty("report").GetString()!; + var name = fq[(fq.LastIndexOf("::", StringComparison.Ordinal) + 2)..]; + var names = report.GetProperty("fields").EnumerateArray() + .Where(f => f.GetProperty("subType").GetString() == FIELD_SUBTYPE_DECIMAL) + .Select(f => f.GetProperty("name").GetString()!) + .ToHashSet(StringComparer.OrdinalIgnoreCase); + if (names.Count > 0) map[name] = names; + } + return map; + } + + /// Fail the scenario when a declared-decimal field of is not a canonical decimal. + public static void Check(string scenarioPath, QuerySpec spec, object? actual) + { + if (!Declared.Value.TryGetValue(spec.Entity, out var fields)) return; + switch (actual) + { + case IReadOnlyDictionary row: + CheckRow(scenarioPath, spec, fields, row); + break; + case IReadOnlyList> rows: + foreach (var row in rows) CheckRow(scenarioPath, spec, fields, row); + break; + } + } + + private static void CheckRow(string scenarioPath, QuerySpec spec, HashSet fields, + IReadOnlyDictionary row) + { + foreach (var (name, value) in row) + { + if (!fields.Contains(name) || value is null) continue; + if (value is not decimal d) + throw new XunitException( + $"{scenarioPath} / {spec.Name}: decimal arrived as a binary float: field '{name}' is " + + $"{value.GetType().Name} ({value}), not a decimal"); + var text = Normalization.NormalizeValue(d) as string; + if (text is null || !Canonical().IsMatch(text)) + throw new XunitException( + $"{scenarioPath} / {spec.Name}: field '{name}' is not a canonical decimal string: '{text}'"); + } + } +} diff --git a/server/csharp/MetaObjects.IntegrationTests/Runner/QueryScenarioRunner.cs b/server/csharp/MetaObjects.IntegrationTests/Runner/QueryScenarioRunner.cs index 1a725849f..e52569352 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Runner/QueryScenarioRunner.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Runner/QueryScenarioRunner.cs @@ -88,6 +88,7 @@ private static async Task RunAsync(QueryScenario scenario, ScenarioEngine engine if (engine != ScenarioEngine.Postgres) throw new InvalidOperationException($"{scenario.SourcePath}: op:relate runs on Postgres only"); var actual = await ResolveRelateAsync(connectionString, metaRoot, spec); + DecimalGate.Check(scenario.SourcePath, spec, actual); AssertResult(scenario.SourcePath, spec, actual); continue; } @@ -120,11 +121,13 @@ private static async Task RunAsync(QueryScenario scenario, ScenarioEngine engine // write codec + read path together (the structural complement to the // read-only scenarios). See RoundtripWriter. var written = await RoundtripWriter.ExecuteAsync(db, spec); + DecimalGate.Check(scenario.SourcePath, spec, written); AssertResult(scenario.SourcePath, spec, written); continue; } var efActual = await DbContextAdapter.ExecuteAsync(db, spec); + DecimalGate.Check(scenario.SourcePath, spec, efActual); AssertResult(scenario.SourcePath, spec, efActual); } } diff --git a/server/csharp/MetaObjects.IntegrationTests/Runner/ReportViewsDbContext.cs b/server/csharp/MetaObjects.IntegrationTests/Runner/ReportViewsDbContext.cs index 229cda065..fa9ab2e46 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Runner/ReportViewsDbContext.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Runner/ReportViewsDbContext.cs @@ -26,11 +26,15 @@ public sealed class ReportViewsDbContext : AppDbContext { public ReportViewsDbContext(DbContextOptions options) : base(options) { } + // The one table the decimal scenarios write (`Entry`): plain scalars every engine maps, so it + // stays in the model. The other tables carry Postgres-only columns (jsonb, inet, uuid). + private static readonly HashSet WritableTables = [typeof(Entry)]; + protected override void OnModelCreating(ModelBuilder modelBuilder) { base.OnModelCreating(modelBuilder); var tables = modelBuilder.Model.GetEntityTypes() - .Where(e => e.GetViewName() is null) + .Where(e => e.GetViewName() is null && !WritableTables.Contains(e.ClrType)) .Select(e => e.ClrType) .ToList(); foreach (var table in tables) modelBuilder.Ignore(table); diff --git a/server/csharp/MetaObjects.IntegrationTests/Runner/Scenario.cs b/server/csharp/MetaObjects.IntegrationTests/Runner/Scenario.cs index 7456c0aec..9420af4b6 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Runner/Scenario.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Runner/Scenario.cs @@ -18,9 +18,23 @@ public sealed record QueryScenario( string SourcePath, string? SeedData, IReadOnlyList Queries, - IReadOnlyDictionary? SeedDataEngine = null) // per-engine seed SQL (`seed-data-engine:`), keyed by engine name + IReadOnlyDictionary? SeedDataEngine = null, // per-engine seed SQL (`seed-data-engine:`), keyed by engine name + IReadOnlyList? Engines = null) // `engines:` — EXACT when present; null keeps the historical rule : Scenario(Name, Description, SourcePath) { + /// The engine names a scenario file may list. + public static readonly IReadOnlyList KnownEngines = ["postgres", "mysql", "sqlite", "d1"]; + + /// + /// Whether runs this scenario. With engines: the list is exact + /// and a runner never skips an engine by name. Without it the historical rule holds: Postgres + /// runs every scenario and the other engines run the report-* ones. + /// + public bool RunsOn(ScenarioEngine engine) => + Engines is not null + ? Engines.Contains(engine.Name(), StringComparer.Ordinal) + : engine == ScenarioEngine.Postgres || Name.StartsWith("report-", StringComparison.Ordinal); + /// The seed SQL for : its seed-data-engine entry, else the Postgres seed-data. public string? SeedFor(string engine) => SeedDataEngine is not null && SeedDataEngine.TryGetValue(engine, out var own) ? own : SeedData; diff --git a/server/csharp/MetaObjects.IntegrationTests/Runner/ScenarioLoader.cs b/server/csharp/MetaObjects.IntegrationTests/Runner/ScenarioLoader.cs index bcf555689..0fb9c3ede 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Runner/ScenarioLoader.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Runner/ScenarioLoader.cs @@ -85,6 +85,7 @@ public static QueryScenario LoadQuery(string yamlPath) SourcePath: yamlPath, SeedData: raw.SeedData, SeedDataEngine: raw.SeedDataEngine, + Engines: ValidEngines(yamlPath, raw.Engines), Queries: (raw.Queries ?? []).Select(q => new QuerySpec( Name: q.Name ?? throw new InvalidOperationException($"{yamlPath}: query missing 'name'"), Op: q.Op ?? throw new InvalidOperationException($"{yamlPath}: query missing 'op'"), @@ -101,6 +102,19 @@ public static QueryScenario LoadQuery(string yamlPath) Expect: q.Expect)).ToList()); } + /// An unknown engine name, or an empty list, is a runner error (never a silent skip). + private static IReadOnlyList? ValidEngines(string yamlPath, List? engines) + { + if (engines is null) return null; + if (engines.Count == 0) + throw new InvalidOperationException($"{yamlPath}: 'engines' is an empty list"); + foreach (var name in engines) + if (!QueryScenario.KnownEngines.Contains(name, StringComparer.Ordinal)) + throw new InvalidOperationException( + $"{yamlPath}: unknown engine '{name}' in 'engines' (known: {string.Join(", ", QueryScenario.KnownEngines)})"); + return engines; + } + // ---------- YAML wire shapes (lower-cased + hyphenated via the naming convention) ---------- private sealed class QueryYaml @@ -111,6 +125,8 @@ private sealed class QueryYaml // Per-engine seed SQL, keyed by engine name. Only a seed that cannot be one SQL // text across engines (a clock-relative expression) carries one. public Dictionary? SeedDataEngine { get; set; } + // `engines:` — exact when present (postgres | mysql | sqlite | d1). + public List? Engines { get; set; } public List? Queries { get; set; } } diff --git a/server/csharp/MetaObjects.IntegrationTests/Runner/WriteCoercion.cs b/server/csharp/MetaObjects.IntegrationTests/Runner/WriteCoercion.cs index 9caf1ccd3..99529c980 100644 --- a/server/csharp/MetaObjects.IntegrationTests/Runner/WriteCoercion.cs +++ b/server/csharp/MetaObjects.IntegrationTests/Runner/WriteCoercion.cs @@ -86,7 +86,10 @@ public static PropertyInfo Property(Type entity, string fieldName) => if (underlying == typeof(long)) return long.Parse(raw, NumberStyles.Integer, CultureInfo.InvariantCulture); if (underlying == typeof(double)) return double.Parse(raw, NumberStyles.Float, CultureInfo.InvariantCulture); if (underlying == typeof(float)) return float.Parse(raw, NumberStyles.Float, CultureInfo.InvariantCulture); - if (underlying == typeof(decimal)) return decimal.Parse(raw, NumberStyles.Float, CultureInfo.InvariantCulture); + if (underlying == typeof(decimal)) + return MetaObjects.Codegen.Runtime.DecimalWire.TryParse(raw, out var exact) + ? exact + : throw new FormatException($"not a decimal: {raw}"); if (underlying == typeof(DateOnly)) return ParseDate(raw); if (underlying == typeof(TimeOnly)) return ParseTime(raw); if (underlying == typeof(DateTime)) return ParseDateTime(raw); diff --git a/server/csharp/MetaObjects/Core/Requirement/RequirementCheck.cs b/server/csharp/MetaObjects/Core/Requirement/RequirementCheck.cs index 7eae9bf1d..a4850a376 100644 --- a/server/csharp/MetaObjects/Core/Requirement/RequirementCheck.cs +++ b/server/csharp/MetaObjects/Core/Requirement/RequirementCheck.cs @@ -125,18 +125,18 @@ private static void Walk(MetaData node, string prefix, List"". /// /// - /// Read through , which cuts the node's - /// : its own package, else its file-default package, else the - /// package of the nearest ancestor that declares one. That is Table A's wording, and it is what - /// the Java port does. It agrees with TypeScript on every shape the corpus pins (a nested - /// requirement takes its parent's package, not the file's, because the C# parser hands each - /// child the nearest declaring node's package as its file-default) and on a package-less - /// document loaded after a packaged one (both parsers fall back to the accumulating root's - /// package). It differs in one known shape: a child merged into a package-declaring node from a - /// package-less document. C# takes the declaring ancestor's package; TypeScript takes none. - /// That is a loader-level difference outside the gate, recorded for the owner. + /// Walks the parent chain to the nearest node that declares a package; with none, takes the + /// file-default package of the top-level node the requirement hangs under, else "" + /// (ADR-0059 D3/D10). A child contributed by a different document than its merge target thereby + /// resolves like the node it hangs under, never like the document that contributed it. Applies to + /// requirements only: extends resolution is unchanged. /// - public static string EffectivePackage(MetaData node) => NamingRefs.EffectivePackage(node); + public static string EffectivePackage(MetaData node) + { + MetaData cur = node; + while (string.IsNullOrEmpty(cur.Package) && cur.Parent is { Parent: not null } up) cur = up; + return !string.IsNullOrEmpty(cur.Package) ? cur.Package : cur.FileDefaultPackage ?? ""; + } // -- the scan --------------------------------------------------------------------------- diff --git a/server/csharp/MetaObjects/DataConverter.cs b/server/csharp/MetaObjects/DataConverter.cs index 4b1af26ca..43d33df1c 100644 --- a/server/csharp/MetaObjects/DataConverter.cs +++ b/server/csharp/MetaObjects/DataConverter.cs @@ -264,9 +264,13 @@ private static object ToInteger(JsonElement element, string fnName) { if (element.TryGetInt64(out long lng)) return lng; - // Non-integer JSON number (e.g. 3.7) — pass through as double, - // matching TS `toInteger` which returns `raw` (the number) unchanged. - return element.GetDouble(); + // An integral number written with a decimal point or exponent (4.0, 1e2) is the + // integer it names; a fractional one (3.7) passes through as a double so the attribute + // type check refuses it (ERR_BAD_ATTR_VALUE). + double dbl = element.GetDouble(); + if (double.IsFinite(dbl) && dbl == Math.Floor(dbl) && dbl >= JsSafeIntegerMin && dbl <= JsSafeIntegerMax) + return (long)dbl; + return dbl; } if (element.ValueKind == JsonValueKind.String) { diff --git a/server/csharp/MetaObjects/Loader/MetaDataLoader.cs b/server/csharp/MetaObjects/Loader/MetaDataLoader.cs index cb834014f..c9d9e4c58 100644 --- a/server/csharp/MetaObjects/Loader/MetaDataLoader.cs +++ b/server/csharp/MetaObjects/Loader/MetaDataLoader.cs @@ -440,7 +440,7 @@ public LoadResult Load(IReadOnlyList sources) continue; } errors.Add(new MetaError( - $"the SuperClass '{failure.Ref}' does not exist (referenced by {failure.NodeFqn})", + $"the SuperClass '{failure.Ref}' does not exist (referenced by {failure.NodeFqn}){NamingRefs.DidYouMeanSuperHint(root, failure.Ref)}", ErrorCode.ERR_UNRESOLVED_SUPER, Envelope: envelope)); } diff --git a/server/csharp/MetaObjects/Loader/ValidationPasses.Reporting.cs b/server/csharp/MetaObjects/Loader/ValidationPasses.Reporting.cs index 9a47befa6..6fc0be69a 100644 --- a/server/csharp/MetaObjects/Loader/ValidationPasses.Reporting.cs +++ b/server/csharp/MetaObjects/Loader/ValidationPasses.Reporting.cs @@ -169,6 +169,9 @@ internal static bool IsSelfOrAncestor(MetaData? candidate, MetaData entity) // (type, name): an entity may carry a same-named dimension and relationship. obj.Children().FirstOrDefault(c => c.Type == type && c.Name == name); + /// A field's @objectRef, read with the resolving accessor (ADR-0039). + private static object? ReportingObjectRefOf(MetaData field) => field.Attr(FIELD_ATTR_OBJECT_REF); + private static MetaData? ReportingFieldOf(MetaData obj, string name) => ReportingChildOfType(obj, TYPE_FIELD, name); @@ -350,10 +353,27 @@ void Err(string message) => return; } string fieldName = parts.Value.Path[0]; - var field = ReportingFieldOf(ofEntity, fieldName); + // With @via the field is looked up on the entity @of NAMES (own plus inherited), not on the + // @via terminal: a base the terminal extends must itself carry the field, or the dimension + // names a column the base table lacks. Without @via the host IS the checked entity. + var lookup = via is null ? ofEntity : named!; + var field = ReportingFieldOf(lookup, fieldName); if (field is null) { - Err($"@of '{of}' names no field '{fieldName}' on '{Shown(ctx, ofEntity)}'."); + bool declaredOnTerminal = !ReferenceEquals(lookup, ofEntity) && ReportingFieldOf(ofEntity, fieldName) is not null; + Err(declaredOnTerminal + ? $"@of '{of}' names no field '{fieldName}' on '{Shown(ctx, lookup)}': only '{Shown(ctx, ofEntity)}', " + + $"the entity @via reaches, declares it. Name it '{ofEntity.Name}.{fieldName}'." + : $"@of '{of}' names no field '{fieldName}' on '{Shown(ctx, lookup)}'."); + return; + } + + // D5 — a dimension groups by a scalar. A field.object (or a field carrying @objectRef) is a JSON + // document: no port serves a report over one, and a document is not a value to group rows by. + if (field.SubType == FIELD_SUBTYPE_OBJECT || ReportingObjectRefOf(field) is not null) + { + Err($"@of '{of}' is a field.{field.SubType}{(ReportingObjectRefOf(field) is not null ? " with an @objectRef" : "")}, " + + "which a report cannot group by. Group by a scalar field of it."); return; } @@ -484,10 +504,12 @@ void Err(string message) => // The type rule — a measure's @default is a whole number. ValidateAttrSchema already // refuses a fraction (and a non-number) with ERR_BAD_ATTR_VALUE on this node (see the - // file header), so nothing is reported here. A fraction only skips M7/M8: one mistake, + // file header), so nothing is reported here. A @default that is not an integral number skips M7-M9: one mistake, // one error. ADR-0039: resolving — the raw attribute, inherited or not. var declaredDefault = measure.Attr(REPORTING_ATTR_DEFAULT); - bool fractionalDefault = declaredDefault is double d && double.IsFinite(d) && d != Math.Floor(d); + bool invalidDefault = declaredDefault is not null && + !(declaredDefault is int or long or short || + (declaredDefault is double d && double.IsFinite(d) && d == Math.Floor(d))); if (measure.IsRatio()) { @@ -499,9 +521,8 @@ void Err(string message) => bool clean = CheckAggregateColumns(ctx, measure, Err, out var ofField); // M7 / M8 — where a @default can apply. Only when M0–M4 passed and the @default is - // not a fraction: one mistake, one error. The presence test reads the raw attribute, - // so a non-number value on a count still reports M7 beside the attribute type error. - if (clean && !fractionalDefault && declaredDefault is not null) + // an integral number: one mistake, one error (the attribute type error alone). + if (clean && !invalidDefault && declaredDefault is not null) { string? agg = measure.Agg(); if (agg == AGG_COUNT) @@ -594,6 +615,14 @@ private static bool CheckAggregateColumns(MemberCtx ctx, MetaMeasure measure, Ac err($"@of '{item}' names no field '{parts.Value.Path[0]}' on '{ctx.Declaring.ResolutionKey()}'."); return false; } + // M10 — a measure aggregates scalars. A field.object (or a field carrying @objectRef) is a JSON + // document, which no port aggregates in a report. + if (ofField.SubType == FIELD_SUBTYPE_OBJECT || ReportingObjectRefOf(ofField) is not null) + { + err($"@of '{item}' is a field.{ofField.SubType}{(ReportingObjectRefOf(ofField) is not null ? " with an @objectRef" : "")}, " + + "which a report cannot aggregate. Aggregate a scalar field of it."); + return false; + } fields.Add(ofField); } @@ -794,6 +823,30 @@ void Err(string message, MetaData? node = null, ErrorCode code = ErrorCode.ERR_I } } + // R10 — a report is read through a view. The other read-only kinds are not read by a plain + // SELECT (a stored procedure and a table function are called, not selected from), and no port + // lowers or serves them. A view or materialized view the adopter creates is declared + // `@kind: view` with `@unmanaged: true`; a SELECT reads either one. + // ADR-0039: resolving, as R5 reads it. + foreach (var source in report.Children().OfType().Where(src => src.IsReadOnly())) + { + if (source.EffectiveKind != SOURCE_KIND_VIEW) + { + Err($": {ReportingChildLabel(source)} has @kind '{source.EffectiveKind}'; a report is read through a view, so its " + + "source must be @kind: view. For a view or materialized view you create yourself, declare " + + "@kind: view with @unmanaged: true.", source); + } + } + + // R11 — a report cannot be abstract: its fields are derived from its own @dimensions and + // @measures, no port reads or serves an abstract report, and report inheritance is not part of + // the vocabulary. + if (report.IsAbstract) + { + Err(": is abstract; a report cannot be abstract, because its fields are derived from its own @dimensions " + + "and @measures and no port reads or serves an abstract report. Remove abstract: true."); + } + // R1 — @from resolves to an object.entity. Without it, R2/R3/R6/R7, R8/R9 and // the @filter have nothing to resolve against, so they are skipped. string? fromRef = ReportAccessors.ReportFrom(report); diff --git a/server/csharp/MetaObjects/NamingRefs.cs b/server/csharp/MetaObjects/NamingRefs.cs index 059db0a17..3b54379d2 100644 --- a/server/csharp/MetaObjects/NamingRefs.cs +++ b/server/csharp/MetaObjects/NamingRefs.cs @@ -265,6 +265,27 @@ public static string EffectivePackage(MetaData node) /// Mirrors TS didYouMeanHint. /// public static string DidYouMeanHint(MetaData root, string reference) + { + var found = SameNamedObjects(root, reference); + if (found is null) return ""; + return $" An object named \"{found.Value.ShortName}\" exists in: {string.Join(", ", found.Value.Candidates)}. " + + "Qualify it with its package (FQN)."; + } + + /// + /// The did-you-mean suffix for an UNRESOLVED extends ref: the same candidates as + /// plus the exact spelling to write, because a bare extends + /// resolves in the referrer's own package. Mirrors TS didYouMeanSuperHint. + /// + public static string DidYouMeanSuperHint(MetaData root, string reference) + { + var found = SameNamedObjects(root, reference); + if (found is null) return ""; + return $" An object named \"{found.Value.ShortName}\" exists in: {string.Join(", ", found.Value.Candidates)}. " + + $"Qualify it with its package (FQN): write extends: \"{found.Value.Candidates[0]}\"."; + } + + private static (string ShortName, List Candidates)? SameNamedObjects(MetaData root, string reference) { // Owner = everything before a dotted child tail (FR-024 `Owner.child`). int dot = reference.IndexOf(CHILD_REF_SEPARATOR, StringComparison.Ordinal); @@ -275,8 +296,6 @@ public static string DidYouMeanHint(MetaData root, string reference) .Where(c => c.Type == TYPE_OBJECT && c.Name == shortName) .Select(c => c.ResolutionKey()) .ToList(); - if (candidates.Count == 0) return ""; - return $" An object named \"{shortName}\" exists in: {string.Join(", ", candidates)}. " + - "Qualify it with its package (FQN)."; + return candidates.Count == 0 ? null : (shortName, candidates); } } diff --git a/server/csharp/MetaObjects/Parser.cs b/server/csharp/MetaObjects/Parser.cs index fe71f4d24..356d7cfb5 100644 --- a/server/csharp/MetaObjects/Parser.cs +++ b/server/csharp/MetaObjects/Parser.cs @@ -735,7 +735,7 @@ private static MetaData ParseNodeFresh( // referrer = the declaring node's FQN (already built above); // target = the unresolved supertype ref string. throw new ParseException( - $"the SuperClass '{model.SuperRef}' does not exist in file '{st.Source ?? ""}'", + $"the SuperClass '{model.SuperRef}' does not exist in file '{st.Source ?? ""}'{NamingRefs.DidYouMeanSuperHint(accumRoot, model.SuperRef)}", ErrorCode.ERR_UNRESOLVED_SUPER, st.Source, st.Builder.ToString(), ResolvedSource.From(st.CurrentSource(), model.Fqn(), model.SuperRef)); } diff --git a/server/csharp/README.md b/server/csharp/README.md index a7102f583..70e1dd36b 100644 --- a/server/csharp/README.md +++ b/server/csharp/README.md @@ -4,12 +4,15 @@ The C# port of the cross-language MetaObjects metadata standard. Targets .NET 8 ## What it covers -- **Loader** (`MetaObjects`) — multi-file JSON / YAML loader with overlay merge and cross-file `extends:` resolution. +- **Loader** (`MetaObjects`) — multi-file JSON / YAML loader with overlay merge and cross-file `extends:` resolution. A bare `extends: Base` binds the declaring file's own package, then the root; name a base in another package by its FQN (`extends: acme::common::Base`, ADR-0042). - **Canonical serializer** — byte-identical output to the TypeScript reference (the cross-language wire format). - **YAML authoring front-end** — sigil-free attrs + `[]`-array suffix + fused-key default subtype + the D2 type-coercion guard (ADR-0006), via [YamlDotNet](https://github.com/aaubry/YamlDotNet). The cross-language interchange remains canonical JSON; YAML lowers to it. - **Codegen** (`MetaObjects.Codegen`) *(reference generators you own: `dotnet meta eject` copies one into your repo — see [Own your codegen → C#](../../docs/features/own-your-codegen.md#c-dotnet-meta-eject))* — idiomatic .NET emit: EF Core entities, an `AppDbContext`, ASP.NET minimal-API CRUD routes, filter allowlists, payload records, output parsers/prompts, TPH per-subtype surfaces and M:N navigation. - **Render + verify** (`MetaObjects.Render`) — Mustache render, payload value-object codegen and the `verify` drift gates, byte-identical with the other four ports against the shared render corpus. -- **CLI** (`MetaObjects.Cli`) — packaged as a .NET tool invoked **`dotnet meta`**, with `gen` and `verify` verbs. +- **CLI** (`MetaObjects.Cli`) — packaged as a .NET tool invoked **`dotnet meta`**, with `init`, `gen`, `eject`, `verify` and `fmt` verbs. `init` writes the generator list (`codegen/Generators.cs`); `gen` runs that list, documentation generators included (`dotnet meta docs` was removed in 1.1.0, see [docs-are-generators.md](../../docs/features/migrations/docs-are-generators.md#c)). +- **Physical names** — column and table names the metadata does not give come from `naming` in `.metaobjects/config.json` (default `snake_case`; `"naming": { "columns": "literal" }` for a database whose columns are named like the fields), read by `gen` and `verify`. See [physical-naming.md](../../docs/features/migrations/physical-naming.md). + +Upgrading from 1.0.x: [upgrading-1.0-to-1.1.md](../../docs/features/migrations/upgrading-1.0-to-1.1.md). Release notes for every port: [CHANGELOG.md](../../CHANGELOG.md). - **Conformance runners** — auto-discover the shared corpora under `fixtures/` (metamodel, yaml, render, verify, validation, api-contract, persistence, registry manifest). The corpora are the **oracles** — when a fixture goes red the port is wrong, never the fixture. ## What it does NOT cover (out of scope) @@ -54,6 +57,6 @@ The test suite includes per-fixture `Lint` and `Conformance` theories over the s - Per-pipeline-stage unit tests (`ErrorsTests`, `RegistryTests`, `TreeTests`, `ParserTests`, `SerializerTests`, `LoaderTests`, `SuperResolveTests`, `ValidationTests`, …) - `MetaObjects.Codegen/` — the .NET emit pipeline (`Generators/` holds entity, db-context, routes, payload, filter-allowlist, output-parser/prompt, extractor, template) + `CodegenRunner`, `GeneratorRegistry`, `CodegenDrift` - `MetaObjects.Render/` — Mustache render, payload value objects, `verify` -- `MetaObjects.Cli/` — the `dotnet meta` tool (`gen`, `verify`) +- `MetaObjects.Cli/` — the `dotnet meta` tool (`init`, `gen`, `eject`, `verify`, `fmt`) - `MetaObjects.Codegen.Tests/`, `MetaObjects.Render.Tests/`, `MetaObjects.Cli.Tests/` — per-project unit + golden tests - `MetaObjects.IntegrationTests/` — the generated ASP.NET stack booted over HTTP against Testcontainers Postgres (api-contract + persistence corpora) diff --git a/server/java/MIGRATION.md b/server/java/MIGRATION.md index d598c06fc..f7aa860f4 100644 --- a/server/java/MIGRATION.md +++ b/server/java/MIGRATION.md @@ -1,5 +1,11 @@ # Migration Guide +> **This file covers 7.x and earlier and is kept as history.** Since 8.0.0 the Java line moves +> with the other ports and its upgrade guides live with theirs, in +> [docs/features/migrations/](../../docs/features/migrations/): from 8.0.x (1.0.x) to 8.1.0 +> (1.1.0), read [upgrading-1.0-to-1.1.md](../../docs/features/migrations/upgrading-1.0-to-1.1.md); +> within 8.0.x, [upgrading-within-1.0.md](../../docs/features/migrations/upgrading-within-1.0.md). + ## Migrating from 7.0.0 to 7.1.0 7.1.0 is a cleanup release with three **breaking** packaging changes for consumers already on 7.0.0, plus additive features (full list in [RELEASE_NOTES.md](RELEASE_NOTES.md#version-710-2026-05-28)). diff --git a/server/java/README.md b/server/java/README.md index 55a56ddbd..cc07b39bc 100644 --- a/server/java/README.md +++ b/server/java/README.md @@ -1,6 +1,6 @@ # MetaObjects — Java -The Java port of the cross-language MetaObjects metadata standard. Published to Maven Central at `8.0.0` under `com.metaobjects:*` across 14 modules. Apache 2.0. +The Java port of the cross-language MetaObjects metadata standard. Published to Maven Central under `com.metaobjects:*` across 14 modules, at `8.1.0` for the 1.1.0 release (the Maven major is the npm major + 7). Apache 2.0. For the standard itself (metamodel, conformance corpora, ADRs) see the [repository-level docs](../../spec/) and the parent [README](../../README.md). This document is scoped to the Java implementation. @@ -25,7 +25,7 @@ so they are maintained in one place rather than restated per port. ## Modules -All published to Maven Central under `com.metaobjects:*` at `8.0.0`: +All published to Maven Central under `com.metaobjects:*` (`8.1.0` for the 1.1.0 release): | Module | Purpose | |---|---| @@ -52,7 +52,7 @@ The `archetype` and `examples` directories were removed in 7.1.0 (they had been com.metaobjects metaobjects-metadata - 8.0.0 + 8.1.0 ``` @@ -62,12 +62,12 @@ Spring REST stack (repository interface is consumer-implemented): com.metaobjects metaobjects-codegen-spring - 8.0.0 + 8.1.0 com.metaobjects metaobjects-core-spring - 8.0.0 + 8.1.0 ``` @@ -77,7 +77,7 @@ Maven plugin for `metaobjects:generate` / `metaobjects:verify` / `metaobjects:ed com.metaobjects metaobjects-maven-plugin - 8.0.0 + 8.1.0 generate @@ -86,7 +86,14 @@ Maven plugin for `metaobjects:generate` / `metaobjects:verify` / `metaobjects:ed ``` -The `generate`, `verify`, and `docs` goals are declared `threadSafe` and support parallel multi-module reactor builds (`mvn -T`) (#233). +The `generate`, `verify`, `eject` and `fmt` goals are declared `threadSafe` and support parallel multi-module reactor builds (`mvn -T`) (#233). The `docs` goal was removed in 8.1.0: documentation generators are `` entries that `generate` runs ([docs-are-generators.md](../../docs/features/migrations/docs-are-generators.md#java-and-kotlin)). Upgrading from 8.0.x: [upgrading-1.0-to-1.1.md](../../docs/features/migrations/upgrading-1.0-to-1.1.md). + +Column and table names the metadata does not give come from the `naming` setting in +`.metaobjects/config.json` (default `snake_case`), which the runtime reads too: under Spring from +`classpath:.metaobjects/config.json`, or the location the `metaobjects.physical-naming-config` +property names (a `classpath:` location or a file path); outside Spring, `MetaDataLoader.fromUris` +and `fromDirectory` read the file above the metadata. A database whose columns are named like the +fields sets `"naming": { "columns": "literal" }` ([physical-naming.md](../../docs/features/migrations/physical-naming.md)). Kotlin entry point — adds the Kotlin facade and the KotlinPoet codegen pipeline: @@ -94,12 +101,12 @@ Kotlin entry point — adds the Kotlin facade and the KotlinPoet codegen pipelin com.metaobjects metaobjects-metadata-ktx - 8.0.0 + 8.1.0 com.metaobjects metaobjects-codegen-kotlin - 8.0.0 + 8.1.0 ``` diff --git a/server/java/RELEASE_NOTES.md b/server/java/RELEASE_NOTES.md index 60e16a332..6a5a30c08 100644 --- a/server/java/RELEASE_NOTES.md +++ b/server/java/RELEASE_NOTES.md @@ -1,8 +1,13 @@ # MetaObjects Java — Release Notes -Latest update: 2026-05-28. - -Java release line independent from the TypeScript line. TS is on a separate `0.x` track. Java tags are `vX.Y.Z`. Maven coordinates are `com.metaobjects:*` on Maven Central. Release procedure: [docs/RELEASING-java.md](../../docs/RELEASING-java.md). +> **This file stops at 7.1.0 and is kept as history.** Since 8.0.0 the Java line is released +> with the other ports (the Maven major is the npm major + 7, so npm `1.1.0` is Maven `8.1.0`), +> and every release, Java included, is logged in the root [CHANGELOG.md](../../CHANGELOG.md). +> Upgrading from 8.0.x (1.0.x) to 8.1.0 (1.1.0): +> [upgrading-1.0-to-1.1.md](../../docs/features/migrations/upgrading-1.0-to-1.1.md). Release +> procedure: [docs/RELEASING.md](../../docs/RELEASING.md). + +Maven coordinates are `com.metaobjects:*` on Maven Central. Apache 2.0. See [LICENSE](LICENSE). diff --git a/server/java/codegen-base/src/main/java/com/metaobjects/generator/direct/object/BaseObjectCodeWriter.java b/server/java/codegen-base/src/main/java/com/metaobjects/generator/direct/object/BaseObjectCodeWriter.java index 1d2da67e6..a5fe62f27 100644 --- a/server/java/codegen-base/src/main/java/com/metaobjects/generator/direct/object/BaseObjectCodeWriter.java +++ b/server/java/codegen-base/src/main/java/com/metaobjects/generator/direct/object/BaseObjectCodeWriter.java @@ -46,6 +46,8 @@ public abstract class BaseObjectCodeWriter extends FileDirectWriter objectReferenceMap = new HashMap<>(); protected List importList = new ArrayList<>(); @@ -246,7 +248,7 @@ public String writeObject(MetaObject mo) throws GeneratorIOException { protected void writeObjectMethods(MetaObject mo) { inc(); - for (MetaField mf : mo.getMetaFields(false)) { + for (MetaField mf : emittedFields(mo)) { context.setCurrentField(mf); @@ -282,8 +284,28 @@ protected void writeObjectMethods(MetaObject mo) { ////////////////////////////////////////////////////////////////////// // Utility Methods + /** + * The fields this class declares: the object's own, or, for a {@link #standalone} class, every + * effective field (ADR-0039: the resolving read, because the super is not extended). + */ + protected Collection emittedFields(MetaObject mo) { + return mo.getMetaFields(standalone); + } + + /** True iff {@code mo} redeclares a field of {@code sup} (or its ancestors) with another subtype. */ + private static boolean redeclaresIncompatibly(MetaObject mo, MetaObject sup) { + // own fields on purpose: only the layer declared here can redeclare + for (MetaField own : mo.getMetaFields(false)) { + if (!sup.hasMetaField(own.getName())) continue; + MetaField inherited = sup.getMetaField(own.getName()); + if (!inherited.getSubType().equals(own.getSubType())) return true; + } + return false; + } + protected void initVariables(MetaObject mo) { metaObject = mo; + standalone = false; pkg = getLanguagePackage(mo); name = getClassName(mo); @@ -292,7 +314,13 @@ protected void initVariables(MetaObject mo) { superPkg = getLanguagePackage(superObject); superName = getClassName(superObject); - if (!pkg.equals(superPkg)) { + if (redeclaresIncompatibly(mo, superObject)) { + // A Java subclass cannot narrow a getter to an unrelated return type, so a class + // for an object that redeclares an inherited field with another type stands alone + // and carries every effective field itself. + standalone = true; + fullSuperName = null; + } else if (!pkg.equals(superPkg)) { fullSuperName = superPkg + "." + superName; } else { fullSuperName = superName; @@ -306,7 +334,7 @@ protected void initVariables(MetaObject mo) { } protected void initObjectReferenceMap(MetaObject mo) { - for (MetaField mf : mo.getMetaFields(false)) { + for (MetaField mf : emittedFields(mo)) { if (MetaDataUtil.hasObjectRef(mf)) { objectReferenceMap.put(mf, MetaDataUtil.getObjectRef(mf)); } diff --git a/server/java/codegen-base/src/main/java/com/metaobjects/generator/direct/object/javacode/PojoAwareCodeWriter.java b/server/java/codegen-base/src/main/java/com/metaobjects/generator/direct/object/javacode/PojoAwareCodeWriter.java index a8a1a3002..d871d3388 100644 --- a/server/java/codegen-base/src/main/java/com/metaobjects/generator/direct/object/javacode/PojoAwareCodeWriter.java +++ b/server/java/codegen-base/src/main/java/com/metaobjects/generator/direct/object/javacode/PojoAwareCodeWriter.java @@ -72,7 +72,7 @@ protected void writeObjectMethods(MetaObject mo) { */ protected void writeFields(MetaObject mo) { writeNewLine(); - for (MetaField mf : mo.getMetaFields(false)) { + for (MetaField mf : emittedFields(mo)) { String typeName = getLanguageType(mf); String fieldName = getParameterName(mf); println(true, "private " + typeName + " " + fieldName + ";"); diff --git a/server/java/codegen-base/src/main/java/com/metaobjects/generator/direct/object/javacode/ValueObjectCodeWriter.java b/server/java/codegen-base/src/main/java/com/metaobjects/generator/direct/object/javacode/ValueObjectCodeWriter.java index 0c5f9b47a..b0a4e6bc4 100644 --- a/server/java/codegen-base/src/main/java/com/metaobjects/generator/direct/object/javacode/ValueObjectCodeWriter.java +++ b/server/java/codegen-base/src/main/java/com/metaobjects/generator/direct/object/javacode/ValueObjectCodeWriter.java @@ -91,7 +91,7 @@ protected void writeObjectMethods(MetaObject mo) { */ protected void writeFields(MetaObject mo) { writeNewLine(); - for (MetaField mf : mo.getMetaFields(false)) { + for (MetaField mf : emittedFields(mo)) { String holderName = getHolderFieldName(mf); println(true, "private final " + VALUE_HOLDER_FQN + " " + holderName + " = valueHolder(\"" + mf.getName() + "\");"); diff --git a/server/java/codegen-kotlin-exposed1x-check/src/test/kotlin/com/metaobjects/generator/kotlin/exposed1x/Exposed1xCodegenCompileTest.kt b/server/java/codegen-kotlin-exposed1x-check/src/test/kotlin/com/metaobjects/generator/kotlin/exposed1x/Exposed1xCodegenCompileTest.kt index 18d12dca5..ec4cee084 100644 --- a/server/java/codegen-kotlin-exposed1x-check/src/test/kotlin/com/metaobjects/generator/kotlin/exposed1x/Exposed1xCodegenCompileTest.kt +++ b/server/java/codegen-kotlin-exposed1x-check/src/test/kotlin/com/metaobjects/generator/kotlin/exposed1x/Exposed1xCodegenCompileTest.kt @@ -99,17 +99,19 @@ class Exposed1xCodegenCompileTest { "exposedApi=1 output still referenced the 0.x org.jetbrains.exposed.sql.* " + "package — exposedApi did not fully flip every emission site") // The acceptance-critical trap the issue calls out BY NAME: a java.util.UUID column - // must use javaUUID(...), never bare uuid(...) (which binds kotlin.uuid.Uuid under - // 1.x — "the one silent trap: emitting uuid() still compiles, but it changes the - // Kotlin type"). The Asset entity's `id`/`ownerId` (native field.uuid) route through - // this. `externalId` (field.string @dbColumnType=uuid) does NOT — it now emits the - // package-shared `uuidString(...)` extension (ADR-0037: the escape hatch is - // physical-only, so the property stays String; see - // KotlinExposedTableGenerator.uuidStringSupportBlock) — but id/ownerId alone still - // exercise the javaUUID(...) trap this assertion exists to catch. - assertTrue(Regex("""\bjavaUUID\(""").containsMatchIn(combined), - "expected at least one javaUUID(...) column — the fixture declares field.uuid " + + // must never be a bare uuid(...) (which binds kotlin.uuid.Uuid under 1.x — "the one + // silent trap: emitting uuid() still compiles, but it changes the Kotlin type"). The + // Asset entity's `id`/`ownerId` (native field.uuid) are `portableUuid(...)` columns + // (text off Postgres, MetaPortableColumnTypes.kt), whose type delegates to the + // java.util.UUID-backed `v1.core.java.UUIDColumnType`. `externalId` (field.string + // @dbColumnType=uuid) emits the package-shared `uuidString(...)` extension instead + // (ADR-0037: the escape hatch is physical-only, so the property stays String). + assertTrue(Regex("""\bportableUuid\(""").containsMatchIn(combined), + "expected at least one portableUuid(...) column — the fixture declares field.uuid " + "columns (Asset.id/ownerId); none were found in:\n${emittedPaths.map { it.fileName }}") + assertTrue("import org.jetbrains.exposed.v1.core.java.UUIDColumnType" in combined, + "the portable uuid column must delegate to the java.util.UUID-backed " + + "org.jetbrains.exposed.v1.core.java.UUIDColumnType under exposedApi=1") assertFalse(Regex("""(? KotlinEnumEmitter.kt + KotlinDecimalWireEmitter.kt KotlinExtractSchemaEmitter.kt KotlinOutputFormatSpecEmitter.kt diff --git a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/GeneratorRegistry.kt b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/GeneratorRegistry.kt index 0f679a0c2..2c9612333 100644 --- a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/GeneratorRegistry.kt +++ b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/GeneratorRegistry.kt @@ -125,6 +125,9 @@ private fun ejectPath(simpleClassName: String): String = EJECT_RESOURCE_ROOT + s /** The enum file writer `entity` calls. */ private val ENUM_EMIT: List = listOf(KotlinEnumEmitter::class.java.name) +/** The decimal wire file `entity`, `routes` and `exposed-table` each write beside their own output. */ +private val DECIMAL_WIRE_EMIT: List = listOf(KotlinDecimalWireEmitter::class.java.name) + /** The rest of `api-docs`: the page IR (one file, `KotlinApiModel.kt`, which also holds the unit, symbol, field * and kind types), the builder that fills it, the renderer and the page paths. */ private val API_DOCS_EMIT: List = listOf( @@ -155,7 +158,7 @@ val GENERATOR_REGISTRY: Map = linkedMapOf( layer = GeneratorLayer.MODEL, factory = ::KotlinEntityGenerator, ejectResourcePath = ejectPath("KotlinEntityGenerator"), - ejectWith = ENUM_EMIT, + ejectWith = ENUM_EMIT + DECIMAL_WIRE_EMIT, ), "routes" to GeneratorInfo( name = "routes", @@ -164,6 +167,7 @@ val GENERATOR_REGISTRY: Map = linkedMapOf( layer = GeneratorLayer.API, factory = ::KotlinSpringControllerGenerator, ejectResourcePath = ejectPath("KotlinSpringControllerGenerator"), + ejectWith = DECIMAL_WIRE_EMIT, requires = listOf("entity", "exposed-table", "filter-allowlist", "relations"), ), "repository" to GeneratorInfo( @@ -240,6 +244,7 @@ val GENERATOR_REGISTRY: Map = linkedMapOf( layer = GeneratorLayer.PERSISTENCE, factory = ::KotlinExposedTableGenerator, ejectResourcePath = ejectPath("KotlinExposedTableGenerator"), + ejectWith = DECIMAL_WIRE_EMIT, requires = listOf("entity"), ), "relations" to GeneratorInfo( diff --git a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinDecimalWireEmitter.kt b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinDecimalWireEmitter.kt new file mode 100644 index 000000000..d4b756546 --- /dev/null +++ b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinDecimalWireEmitter.kt @@ -0,0 +1,155 @@ +package com.metaobjects.generator.kotlin + +import com.metaobjects.generator.util.GeneratedFileWriter +import java.nio.file.Path + +/** + * Writes the per-package `MetaDecimalWire.kt` support file (ADR-0019 Amendment 1): the canonical + * decimal wire rule the generated entity, table and controller share, in generated source, so the + * generated code imports nothing from MetaObjects at runtime. Mirrors C#'s per-namespace + * `DecimalWire.g.cs` and the portable column-type file beside it. + * + * The entity, table and controller generators each write it into every package they emit a decimal + * (or, for the controller, a request body handler) into. The content is fixed, so several writers + * of the same path agree byte for byte. + */ +object KotlinDecimalWireEmitter { + + /** File name (sans `.kt`) and the name of the `internal object` it declares. */ + const val FILE_NAME = "MetaDecimalWire" + + /** Write `MetaDecimalWire.kt` into [pkg] under [outRoot]. */ + fun write(pkg: String, outRoot: Path) { + val header = if (pkg.isEmpty()) "" else "package $pkg\n\n" + val outFile = outRoot.resolve(pkg.replace('.', '/')).resolve("$FILE_NAME.kt") + GeneratedFileWriter.write(outFile, header + BODY + "\n") + } + + /** + * The fixed body. `+`-concatenated messages and no Kotlin string templates, so this raw string + * needs no `$` escaping. Depends on Jackson only, which a generated entity already imports. + */ + val BODY: String = """ + |import com.fasterxml.jackson.core.JsonGenerator + |import com.fasterxml.jackson.core.JsonParser + |import com.fasterxml.jackson.core.JsonToken + |import com.fasterxml.jackson.databind.DeserializationContext + |import com.fasterxml.jackson.databind.JsonMappingException + |import com.fasterxml.jackson.databind.JsonNode + |import com.fasterxml.jackson.databind.SerializerProvider + |import com.fasterxml.jackson.databind.deser.std.StdDeserializer + |import com.fasterxml.jackson.databind.ser.std.StdSerializer + |import java.math.BigDecimal + |import java.math.MathContext + |import java.math.RoundingMode + | + |/** + | * GENERATED (ADR-0019 Amendment 1) — do not hand-edit. The decimal wire rule: a decimal is + | * written as a canonical string (plain notation, no trailing fraction zeros, zero as "0"), + | * and read from a string or number in the JSON-number grammar, exactly, never through a + | * binary float. Dependency-free (Jackson only). + | */ + |internal object MetaDecimalWire { + | private val JSON_NUMBER = Regex("-?(0|[1-9][0-9]*)(\\.[0-9]+)?([eE][+-]?[0-9]+)?") + | private const val MAX_INTEGER_DIGITS = 131072 + | private const val MAX_FRACTION_DIGITS = 16383 + | private const val SQLITE_SIGNIFICANT_DIGITS = 15 + | + | /** The canonical string of a decimal: plain notation, no trailing fraction zeros, zero as "0". */ + | fun format(value: BigDecimal?): String? { + | if (value == null) return null + | if (value.signum() == 0) return "0" + | return value.stripTrailingZeros().toPlainString() + | } + | + | /** Strict read of a decimal string (the JSON-number grammar; an exponent is allowed). */ + | fun parse(text: String?): BigDecimal { + | require(text != null && JSON_NUMBER.matches(text)) { "not a decimal: " + text } + | val value = try { + | BigDecimal(text) + | } catch (e: NumberFormatException) { + | throw IllegalArgumentException("decimal out of range: " + text) + | } + | return checkRange(value, text) + | } + | + | /** A filter operand: parsed exactly, the same grammar as a body value. */ + | fun parseFilterOperand(text: String): BigDecimal = parse(text) + | + | /** A JSON number token already parsed as a decimal (never a double). */ + | fun fromJsonNumber(value: BigDecimal): BigDecimal = checkRange(value, value.toString()) + | + | /** A decimal from a JSON node: a string or a number, anything else is rejected. */ + | fun fromNode(node: JsonNode): BigDecimal { + | if (node.isTextual) return parse(node.asText()) + | if (node.isNumber) return fromJsonNumber(node.decimalValue()) + | throw IllegalArgumentException("not a decimal: " + node) + | } + | + | /** SQLite keeps a REAL: a stored double reads at 15 significant digits, so 0.1 + 0.2 reads 0.3. */ + | fun fromDouble15(value: Double): BigDecimal { + | require(!value.isNaN() && !value.isInfinite()) { "not a finite decimal: " + value } + | if (value == 0.0) return BigDecimal.ZERO + | val rounded = BigDecimal(value).round(MathContext(SQLITE_SIGNIFICANT_DIGITS, RoundingMode.HALF_EVEN)) + | return if (rounded.signum() == 0) BigDecimal.ZERO else rounded.stripTrailingZeros() + | } + | + | /** True when [value] fits decimal(precision, scale): at most scale fraction digits and precision - scale integer digits. */ + | fun fits(value: BigDecimal?, precision: Int, scale: Int): Boolean { + | if (value == null || value.signum() == 0) return true + | val stripped = value.stripTrailingZeros() + | val integerDigits = maxOf(0L, stripped.precision().toLong() - stripped.scale()) + | val fractionDigits = maxOf(0, stripped.scale()) + | return fractionDigits <= scale && integerDigits <= precision.toLong() - scale + | } + | + | private fun checkRange(value: BigDecimal, shown: String): BigDecimal { + | if (value.signum() == 0) return BigDecimal.ZERO + | val stripped = value.stripTrailingZeros() + | val integerDigits = stripped.precision().toLong() - stripped.scale() + | require(integerDigits <= MAX_INTEGER_DIGITS && stripped.scale() <= MAX_FRACTION_DIGITS) { + | "decimal out of range: " + shown + | } + | return value + | } + | + | /** The validation envelope for a request body Jackson could not read: names the field when it knows it. */ + | fun unreadableBody(failure: Throwable): Map { + | val out = LinkedHashMap() + | out["error"] = "validation" + | var t: Throwable? = failure + | while (t != null) { + | if (t is JsonMappingException && t.path.isNotEmpty()) { + | val name = t.path[t.path.size - 1].fieldName + | if (name != null) { out["field"] = name; break } + | } + | t = if (t.cause === t) null else t.cause + | } + | return out + | } + | + | /** Writes a decimal as its canonical string. */ + | class Serializer : StdSerializer(BigDecimal::class.java) { + | override fun serialize(value: BigDecimal, gen: JsonGenerator, provider: SerializerProvider) { + | gen.writeString(format(value)) + | } + | } + | + | /** Reads a decimal from a string or a number, exactly; anything else is a 400-shaped mapping error. */ + | class Deserializer : StdDeserializer(BigDecimal::class.java) { + | override fun deserialize(p: JsonParser, ctxt: DeserializationContext): BigDecimal? { + | val token = p.currentToken() + | try { + | if (token == JsonToken.VALUE_STRING) return parse(p.text) + | if (token == JsonToken.VALUE_NUMBER_INT || token == JsonToken.VALUE_NUMBER_FLOAT) { + | return fromJsonNumber(p.decimalValue) + | } + | } catch (e: IllegalArgumentException) { + | return ctxt.handleWeirdStringValue(BigDecimal::class.java, p.text, "%s", e.message) as BigDecimal? + | } + | return ctxt.handleUnexpectedToken(BigDecimal::class.java, p) as BigDecimal? + | } + | } + |} + |""".trimMargin() +} diff --git a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinEntityGenerator.kt b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinEntityGenerator.kt index 67f73c239..50bce3558 100644 --- a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinEntityGenerator.kt +++ b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinEntityGenerator.kt @@ -14,6 +14,7 @@ import com.metaobjects.generator.kotlin.KotlinTypeMapper import com.metaobjects.generator.kotlin.PackageMapping import com.metaobjects.MetaData +import com.metaobjects.field.DecimalField import com.metaobjects.field.EnumField import com.metaobjects.field.InetField import com.metaobjects.field.MetaField @@ -87,6 +88,7 @@ open class KotlinEntityGenerator : MultiFileDirectGeneratorBase() { override fun execute(loader: MetaDataLoader) { parseArgs() val outRoot = Paths.get(outDir.absolutePath) + decimalWirePackages.clear() fr019Config = Fr019SharedEnum.ProvidedEnumConfig.fromArgs(args) // emitAbstractShapes (default OFF): when ON, an abstract entity is emitted as a Kotlin // `interface` shape (read-only properties) instead of being suppressed. It is NEVER @@ -123,8 +125,13 @@ open class KotlinEntityGenerator : MultiFileDirectGeneratorBase() { // #234: emit the strict field.uri / field.inet Jackson deserializers ONCE per package that // has such a field (a same-package MetaNetJson the data-class properties bind through). emitNetJsonSupport(loader, outRoot) + // ADR-0019 Amendment 1: the decimal wire rule, once per package that wrote a decimal property. + for (pkg in decimalWirePackages) KotlinDecimalWireEmitter.write(pkg, outRoot) } + /** Packages whose emitted data classes carry a decimal property: each gets one `MetaDecimalWire.kt`. */ + private val decimalWirePackages = linkedSetOf() + /** * #234: emit the per-package `MetaNetJson.kt` support object (the strict field.uri / field.inet * Jackson deserializers a data-class property binds through via `@field:JsonDeserialize`), ONCE @@ -283,6 +290,7 @@ open class KotlinEntityGenerator : MultiFileDirectGeneratorBase() { propBuilder.addAnnotation(constraint(VALID)) } } + decimalWireAnnotations(field, pkg).forEach { propBuilder.addAnnotation(it) } typeBuilder.addProperty(propBuilder.build()) } @@ -293,7 +301,11 @@ open class KotlinEntityGenerator : MultiFileDirectGeneratorBase() { val propType = resolvePropertyType(field, obj, loader).copy(nullable = true) val propName = field.name ctorBuilder.addParameter(ParameterSpec.builder(propName, propType).defaultValue("null").build()) - typeBuilder.addProperty(PropertySpec.builder(propName, propType).initializer(propName).build()) + typeBuilder.addProperty( + PropertySpec.builder(propName, propType).initializer(propName) + .apply { decimalWireAnnotations(field, pkg).forEach { addAnnotation(it) } } + .build(), + ) } val primaryCtor = ctorBuilder.build() @@ -604,6 +616,27 @@ open class KotlinEntityGenerator : MultiFileDirectGeneratorBase() { * A `@lenient` uri/inet degrades to a plain String ([KotlinTypeMapper.kotlinTypeName]) with NO * deserializer, so it is NOT strict. */ + /** + * ADR-0019 Amendment 1 (W1/W2): a `field.decimal` property is written as a canonical JSON string + * and read from a string or number in the JSON-number grammar, through the generated same-package + * `MetaDecimalWire` (no MetaObjects import). The serializer sits on the getter and the deserializer on the + * constructor parameter, the two places Jackson's Kotlin module looks. `contentUsing` for an + * `@isArray` property (the elements are the decimals). + */ + private fun decimalWireAnnotations(field: MetaField<*>, pkg: String): List { + if (field !is DecimalField) return emptyList() + decimalWirePackages.add(pkg) + val serializer = ClassName(pkg, KotlinDecimalWireEmitter.FILE_NAME, "Serializer") + val deserializer = ClassName(pkg, KotlinDecimalWireEmitter.FILE_NAME, "Deserializer") + val member = if (field.isArrayType) "contentUsing" else "using" + return listOf( + AnnotationSpec.builder(JSON_SERIALIZE).useSiteTarget(AnnotationSpec.UseSiteTarget.GET) + .addMember("$member = %T::class", serializer).build(), + AnnotationSpec.builder(JSON_DESERIALIZE).useSiteTarget(AnnotationSpec.UseSiteTarget.PARAM) + .addMember("$member = %T::class", deserializer).build(), + ) + } + private fun isStrictNetField(field: MetaField<*>): Boolean = (field is UriField || field is InetField) && !KotlinTypeMapper.lenientNetField(field) @@ -690,6 +723,7 @@ open class KotlinEntityGenerator : MultiFileDirectGeneratorBase() { /** #234: `@field:JsonDeserialize(using = …)` binds a STRICT field.uri / field.inet property * to the same-package [NET_JSON_SUPPORT] literal deserializer. */ val JSON_DESERIALIZE = ClassName("com.fasterxml.jackson.databind.annotation", "JsonDeserialize") + val JSON_SERIALIZE = ClassName("com.fasterxml.jackson.databind.annotation", "JsonSerialize") /** The per-package generated support object holding the #234 strict Jackson deserializers. */ const val NET_JSON_SUPPORT = "MetaNetJson" diff --git a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGenerator.kt b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGenerator.kt index 336b413fc..b68d67360 100644 --- a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGenerator.kt +++ b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGenerator.kt @@ -159,6 +159,11 @@ open class KotlinExposedTableGenerator : MultiFileDirectGeneratorBase() + // Packages that emit at least one table carrying a `field.uuid` or `field.decimal` column. + // Each gets ONE shared `MetaPortableColumnTypes.kt` (the engine-aware `Column` and + // `Column` builders), emitted in pass 3e. + val packagesNeedingPortableHelper = sortedSetOf() + // Pass 2: emit one Table per entity using its own metadata + the // inbound FKs accumulated in Pass 1. // FR-024 (ADR-0028): object.projection is emitted too — a projection with @@ -172,7 +177,7 @@ open class KotlinExposedTableGenerator : MultiFileDirectGeneratorBase, packagesNeedingJacksonMapper: MutableSet, packagesNeedingUuidStringHelper: MutableSet, + packagesNeedingPortableHelper: MutableSet, ) { val (pkg, shortName) = PackageMapping.splitFqn(entity.name) val writeSource = KotlinGenUtil.writableRdbSource(entity) @@ -344,6 +359,7 @@ open class KotlinExposedTableGenerator : MultiFileDirectGeneratorBase, packagesNeedingJacksonMapper: MutableSet, packagesNeedingUuidStringHelper: MutableSet, + packagesNeedingPortableHelper: MutableSet, ) { // Table A, from the one JVM predicate every REST-surface generator asks: a report // is served only when it is concrete and its read source is `@kind: view`. @@ -413,6 +430,7 @@ open class KotlinExposedTableGenerator : MultiFileDirectGeneratorBase, colExpr: String, api: ExposedApi): String { if (reportPlans[entity]?.unsizedDecimals?.contains(field.name) == true) { - return "decimal($colExpr, $REPORT_DECIMAL_PRECISION, $REPORT_DECIMAL_SCALE)" + return "${KotlinTypeMapper.EXPOSED_PORTABLE_DECIMAL_FN}($colExpr, $REPORT_DECIMAL_PRECISION, $REPORT_DECIMAL_SCALE)" } return KotlinTypeMapper.exposedColumnSpec(field, colExpr, api) } @@ -575,6 +593,20 @@ open class KotlinExposedTableGenerator : MultiFileDirectGeneratorBase() { + | private val delegate = UUIDColumnType() + | override fun sqlType(): String = delegate.sqlType() + | override fun valueFromDB(value: Any): UUID = requireNotNull(delegate.valueFromDB(value)) { "null uuid column value" } + | override fun notNullValueToDB(value: UUID): Any = + | if (isPostgres()) delegate.notNullValueToDB(value) else value.toString() + | override fun nonNullValueToString(value: UUID): String = + | if (isPostgres()) delegate.nonNullValueToString(value) else "'${'$'}value'" + |} + | + |/** Column builder for `field.uuid`: a `Column`. */ + |internal fun Table.portableUuid(name: String): Column = + | registerColumn(name, MetaPortableUuidColumnType()) + | + |/** GENERATED — do not hand-edit. A decimal column: the exact decimal everywhere but SQLite, which gets a REAL. */ + |internal class MetaPortableDecimalColumnType(precision: Int, scale: Int) : ColumnType() { + | private val delegate = DecimalColumnType(precision, scale) + | override fun sqlType(): String = delegate.sqlType() + | // ADR-0019 Amendment 1 (W4): SQLite stores a REAL, so a value is read at 15 significant + | // digits, which turns `sum(0.1, 0.2)` (0.30000000000000004) back into 0.3. + | override fun valueFromDB(value: Any): BigDecimal = + | if (isSqlite() && value is Number && value !is BigDecimal) { + | MetaDecimalWire.fromDouble15(value.toDouble()) + | } else { + | delegate.valueFromDB(value) + | } + | override fun notNullValueToDB(value: BigDecimal): Any = + | if (isSqlite()) value.toDouble() else delegate.notNullValueToDB(value) + | override fun nonNullValueToString(value: BigDecimal): String = delegate.nonNullValueToString(value) + |} + | + |/** Column builder for `field.decimal`: a `Column`. */ + |internal fun Table.portableDecimal(name: String, precision: Int, scale: Int): Column = + | registerColumn(name, MetaPortableDecimalColumnType(precision, scale)) + |""".trimMargin() + /** * File name (sans `.kt`) of the per-package support file emitted for a `field.string * @dbColumnType=uuid` column (R6 Plan 2b). One per package that has ≥1 such column. diff --git a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinFilterAllowlistGenerator.kt b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinFilterAllowlistGenerator.kt index 7b56ce7d1..98d4d6bfb 100644 --- a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinFilterAllowlistGenerator.kt +++ b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinFilterAllowlistGenerator.kt @@ -95,9 +95,9 @@ open class KotlinFilterAllowlistGenerator : MultiFileDirectGeneratorBase?filter[][in]=A,B` narrows the - // polymorphic list (FR-017 "Filter allowlists"; TS, C# and Python admit it too). It - // EXCLUDES decimal columns — outside the cross-port HTTP filter/value contract. Keeps the - // Java + Kotlin filter surface identical (see the Java SpringFilterAllowlistGenerator). + // polymorphic list (FR-017 "Filter allowlists"; TS, C# and Python admit it too). Decimal + // columns are admitted like any other: their wire form is one rule in every port (ADR-0019 + // Amendment 1), so the old exclusion no longer has a reason. // // The per-subtype routes share this one allowlist, so `//` also accepts the // discriminator (AND'd with the route's own, a no-op or an empty list) and another @@ -106,9 +106,7 @@ open class KotlinFilterAllowlistGenerator : MultiFileDirectGeneratorBase`. val opsByField = if (KotlinTphPlan.isTphBase(entity, loader)) { - val union = (entity.metaFields + KotlinTphPlan.collectSubtypeFields(entity, loader)) - .filter { it !is com.metaobjects.field.DecimalField } - computeFilterableOps(union) + computeFilterableOps(entity.metaFields + KotlinTphPlan.collectSubtypeFields(entity, loader)) } else { computeFilterableOps(entity) } diff --git a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinSpringControllerGenerator.kt b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinSpringControllerGenerator.kt index e745abb3d..b0a6b35e6 100644 --- a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinSpringControllerGenerator.kt +++ b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinSpringControllerGenerator.kt @@ -119,6 +119,9 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase() for (declared in loader.metaObjects) { // FR-044: a served report is emitted from its read model, the same object the @@ -168,6 +171,7 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase = entity.metaFields - .filterNot { it is ObjectField || it is MapField || KotlinTypeMapper.isJsonbOpenBag(it) } + .filterNot { isNonScalarSurface(it) } .map { ScalarFieldSpec(it.name, it.subType, columnElementType(it), intBackedEnumType(it, entity)) } val allowlistName = "${shortName}FilterAllowlist" @@ -435,6 +440,7 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase = transaction {\n") append(" if (validator.validate(dto).isNotEmpty()) return@transaction ResponseEntity.badRequest().body(mapOf(\"error\" to \"validation\") as Any)\n") + for (field in entity.metaFields) appendDecimalRangeCheck(" ", field, "dto.${field.name}") // #203/ADR-0045: capture one now() per temporal type so createdAt == updatedAt exactly. for ((expr, valName) in insertNowVal) append(" val $valName = $expr\n") append(" val newId = ${tableObjectName}.insert {\n") @@ -534,13 +540,14 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase() {}) else null\n") + append(" val v$cap: $ktType? = if (has$cap) ${patchBindExpr(field, fn, ktType)} else null\n") append(" if (has$cap && validator.validateValue(${shortName}::class.java, \"$fn\", v$cap).isNotEmpty()) return@transaction ResponseEntity.badRequest().body(mapOf(\"error\" to \"validation\") as Any)\n") } else { append(" val null$cap = has$cap && body.get(\"$fn\").isNull\n") - append(" val v$cap: $ktType? = if (has$cap && !null$cap) objectMapper.treeToValue(body.get(\"$fn\"), object : TypeReference<$ktType>() {}) else null\n") + append(" val v$cap: $ktType? = if (has$cap && !null$cap) ${patchBindExpr(field, fn, ktType)} else null\n") append(" if (has$cap && !null$cap && validator.validateValue(${shortName}::class.java, \"$fn\", v$cap).isNotEmpty()) return@transaction ResponseEntity.badRequest().body(mapOf(\"error\" to \"validation\") as Any)\n") } + appendDecimalRangeCheck(" ", field, "v$cap") // Program D (spec §0): jakarta validateValue does NOT cascade @Valid into a nested // value-object's constraints, so a present VO is validated EXPLICITLY — // validator.validate(voElement) per element (single VO whole; array VO per element), @@ -681,20 +688,19 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase reuses it to map every column.) // The discriminator stays IN: the base allowlist admits it (FR-017), so it needs a // coercion + WHERE arm like any filterable enum column. val filterSpecs = scalarFields - .filter { it !is com.metaobjects.field.DecimalField && !KotlinTypeMapper.isJsonbOpenBag(it) } + .filter { !KotlinTypeMapper.isJsonbOpenBag(it) } .map { ScalarFieldSpec(it.name, it.subType, columnElementType(it), intBackedEnumType(it, base)) } // #203/ADR-0045: @autoSet columns on the union table are stamped by the controller, never bound // from the per-subtype create body. Same computation as the vanilla emit() (lines ~195-207); the @@ -810,9 +816,11 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase = transaction {\n") for (field in stPatch) { append(" if (validator.validateValue($validationRef::class.java, \"${field.name}\", dto.${field.name}).isNotEmpty()) return@transaction ResponseEntity.badRequest().body(mapOf(\"error\" to \"validation\") as Any)\n") + appendDecimalRangeCheck(" ", field, "dto.${field.name}") } // #203/ADR-0045: capture one now() per temporal type so createdAt == updatedAt exactly. for ((expr, valName) in insertNowVal) append(" val $valName = $expr\n") @@ -1069,13 +1078,14 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase() {}) else null\n") + append(" val v$cap: $ktType? = if (has$cap) ${patchBindExpr(field, fn, ktType)} else null\n") append(" if (has$cap && validator.validateValue($validationRef::class.java, \"$fn\", v$cap).isNotEmpty()) return@transaction ResponseEntity.badRequest().body(mapOf(\"error\" to \"validation\") as Any)\n") } else { append(" val null$cap = has$cap && body.get(\"$fn\").isNull\n") - append(" val v$cap: $ktType? = if (has$cap && !null$cap) objectMapper.treeToValue(body.get(\"$fn\"), object : TypeReference<$ktType>() {}) else null\n") + append(" val v$cap: $ktType? = if (has$cap && !null$cap) ${patchBindExpr(field, fn, ktType)} else null\n") append(" if (has$cap && !null$cap && validator.validateValue($validationRef::class.java, \"$fn\", v$cap).isNotEmpty()) return@transaction ResponseEntity.badRequest().body(mapOf(\"error\" to \"validation\") as Any)\n") } + appendDecimalRangeCheck(" ", field, "v$cap") } // Apply the already-bound + validated values, scoped to (pk eq id) and (disc eq value). // Columns are qualified ($table.col) so a field named like the body/id params can't shadow. @@ -1172,6 +1182,13 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase =\n") + append(" ResponseEntity.badRequest().body(MetaDecimalWire.unreadableBody(e) as Any)\n\n") append(" /**\n") append(" * A database constraint violation is a CLIENT error, not a 500: the\n") append(" * identity.reference / identity.secondary that declare these constraints are the\n") @@ -1204,6 +1221,51 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase, value: String) { + if (field !is DecimalField || field.isArrayType) return + append(indent) + append("if (!${decimalFitsExpr(field, value)}) return@transaction ") + append("ResponseEntity.badRequest().body(mapOf(\"error\" to \"validation\", \"field\" to \"${field.name}\") as Any)\n") + } + + /** + * True for a field that has no scalar column to compare or order by: a value object, a map, the + * `field.string @dbColumnType=jsonb` open bag, and any array (a list column has no comparison + * operator and no order). Such a field is in neither the filter dispatch nor the sort + * allowlist, whatever the entity field it is derived from says. + */ + private fun isNonScalarSurface(field: MetaField<*>): Boolean = + field is ObjectField || field is MapField || field.isArrayType || KotlinTypeMapper.isJsonbOpenBag(field) + + /** + * The expression binding a PRESENT PATCH value of [field] to [ktType]. A scalar `field.decimal` + * binds through the same canonical reader as a create body (ADR-0019 Amendment 1, W2/W3): a + * string in the JSON-number grammar or a JSON number, anything else (or a value past the + * exact range) answers `400 {"error":"validation","field":...}`. Everything else binds through + * the configured ObjectMapper. + */ + private fun patchBindExpr(field: MetaField<*>, fn: String, ktType: String): String = + if (field is DecimalField && !field.isArrayType) { + "runCatching { MetaDecimalWire.fromNode(body.get(\"$fn\")) }" + + ".getOrElse { return@transaction ResponseEntity.badRequest().body(mapOf(\"error\" to \"validation\", \"field\" to \"$fn\") as Any) }" + } else { + "objectMapper.treeToValue(body.get(\"$fn\"), object : TypeReference<$ktType>() {})" + } + /** Capitalize the first char (method-name suffix from a discriminator value). */ private fun capitalizeFirst(s: String): String = if (s.isEmpty()) s else s[0].uppercaseChar() + s.substring(1) @@ -1455,7 +1517,9 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase = entity.metaFields - .filterNot { it is ObjectField || it is MapField || KotlinTypeMapper.isJsonbOpenBag(it) } + .filterNot { isNonScalarSurface(it) } .map { ScalarFieldSpec( it.name, it.subType, columnElementType(it), diff --git a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinTypeMapper.kt b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinTypeMapper.kt index f1ed1d008..51313046d 100644 --- a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinTypeMapper.kt +++ b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinTypeMapper.kt @@ -173,6 +173,22 @@ object KotlinTypeMapper { */ const val EXPOSED_UUID_STRING_FN = "uuidString" + /** + * Builder extension for a native `field.uuid` column: a `Column` that binds + * through Postgres' native `uuid` type and as its 36-character text everywhere else (MySQL and + * SQLite store a uuid as text, where Exposed's own `uuid()` binds 16 raw bytes and a filter + * on the column never matches). Lives in the package-shared `MetaPortableColumnTypes` support file. + */ + const val EXPOSED_PORTABLE_UUID_FN = "portableUuid" + + /** + * Builder extension for a `field.decimal` column: a `Column` that binds a value as + * the exact decimal on every engine but SQLite, which has no decimal and compares a bound + * decimal string with a REAL as text (so `x > 0` is never true). Same support file as + * [EXPOSED_PORTABLE_UUID_FN]. + */ + const val EXPOSED_PORTABLE_DECIMAL_FN = "portableDecimal" + /** * Compute the generated Kotlin enum-class name for an [EnumField] hung off [entity]. * @@ -421,9 +437,7 @@ object KotlinTypeMapper { // (`array("col", columnType)`) under both — only the explicit element // `UUIDColumnType()` instance it's handed changes, inline-qualified in // [exposedColumnSpec] rather than imported, so it needs no import here. - is UuidField -> - if (exposedApi == ExposedApi.V1 && !isArrayResolved(field)) ExposedImports.javaUuid("javaUUID") - else null + is UuidField -> null // IntegerField, LongField, DoubleField, FloatField, BooleanField, CurrencyField, and // EnumField all map to member functions on Table. No additional import required. else -> null @@ -500,7 +514,7 @@ object KotlinTypeMapper { // field.decimal → Exposed `decimal(name, precision, scale)` (Postgres NUMERIC(p,s)). // Exposed requires both precision and scale; read the declared @precision/@scale, // falling back to 19,4 (matching the TS column-mapper default) when absent. - is DecimalField -> "decimal($colExpr, ${decimalPrecision(field)}, ${decimalScale(field)})" + is DecimalField -> "$EXPOSED_PORTABLE_DECIMAL_FN($colExpr, ${decimalPrecision(field)}, ${decimalScale(field)})" is BooleanField -> "bool($colExpr)" is DateField -> "date($colExpr)" // field.time → Exposed `time(name)` (Postgres TIME, java.time.LocalTime). @@ -536,8 +550,7 @@ object KotlinTypeMapper { if (exposedApi == ExposedApi.V1) ExposedImports.javaUuid("UUIDColumnType") else ExposedImports.core(exposedApi, "UUIDColumnType") "array($colExpr, $uuidColumnTypeFqn())" - } else if (exposedApi == ExposedApi.V1) "javaUUID($colExpr)" - else "uuid($colExpr)" + } else "$EXPOSED_PORTABLE_UUID_FN($colExpr)" // field.uri → file-local `uriColumn(...)` extension: a `Column` whose // DDL is plain `text` (Postgres has no uri type). The helper lives in the package-shared // MetaInetUriColumnType.kt support file (emitted once per package). @@ -649,6 +662,14 @@ object KotlinTypeMapper { * whether a package needs the support file (one `uuidString(...)` extension declaration * shared across its tables). */ + /** + * True iff [field] needs the package-shared `MetaPortableColumnTypes.kt` support file: a + * non-array `field.uuid` ([EXPOSED_PORTABLE_UUID_FN]) or `field.decimal` + * ([EXPOSED_PORTABLE_DECIMAL_FN]). Mirrors [exposedColumnSpec]'s two branches. + */ + fun usesPortableHelper(field: MetaField<*>): Boolean = + (field is UuidField || field is DecimalField) && !isArrayResolved(field) + fun usesUuidStringHelper(field: MetaField<*>): Boolean = field is StringField && !isArrayResolved(field) && dbColumnType(field) == DB_COLUMN_TYPE_UUID @@ -690,12 +711,16 @@ object KotlinTypeMapper { private const val DECIMAL_DEFAULT_SCALE = 4 /** Resolve @precision on a DecimalField; default [DECIMAL_DEFAULT_PRECISION]. */ - private fun decimalPrecision(field: DecimalField): Int = + fun decimalPrecision(field: DecimalField): Int = intAttr(field, DecimalField.ATTR_PRECISION) ?: DECIMAL_DEFAULT_PRECISION - /** Resolve @scale on a DecimalField; default [DECIMAL_DEFAULT_SCALE]. */ - private fun decimalScale(field: DecimalField): Int = - intAttr(field, DecimalField.ATTR_SCALE) ?: DECIMAL_DEFAULT_SCALE + /** + * Resolve @scale on a DecimalField. A declared @precision with no @scale means scale 0 (the + * column is `decimal(p)`), as the Java port reads it; with neither, [DECIMAL_DEFAULT_SCALE]. + */ + fun decimalScale(field: DecimalField): Int = + intAttr(field, DecimalField.ATTR_SCALE) + ?: if (intAttr(field, DecimalField.ATTR_PRECISION) != null) 0 else DECIMAL_DEFAULT_SCALE /** * Best-effort read of a named int-valued attribute on [field], resolved THROUGH the diff --git a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinAbstractFieldExtendsInheritanceTest.kt b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinAbstractFieldExtendsInheritanceTest.kt index 16e48c61c..7ef614fdf 100644 --- a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinAbstractFieldExtendsInheritanceTest.kt +++ b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinAbstractFieldExtendsInheritanceTest.kt @@ -72,7 +72,7 @@ class KotlinAbstractFieldExtendsInheritanceTest { "tags must NOT fall back to a scalar column when array-ness is inherited; saw:\n$table") // (2b) `balance` extends the abstract decimal field → decimal(col, 12, 2) (inherited p/s). - assertTrue("val balance = decimal(\"balance\", 12, 2)" in table, + assertTrue("val balance = portableDecimal(\"balance\", 12, 2)" in table, "balance must inherit @precision=12/@scale=2 from the abstract Money field; saw:\n$table") assertFalse("decimal(\"balance\", 19, 4)" in table, "balance must NOT fall back to the default NUMERIC(19,4) — precision/scale are inherited; saw:\n$table") diff --git a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinDecimalWireTest.kt b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinDecimalWireTest.kt new file mode 100644 index 000000000..934a30c1a --- /dev/null +++ b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinDecimalWireTest.kt @@ -0,0 +1,258 @@ +package com.metaobjects.generator.kotlin + +import com.metaobjects.generator.Generator +import com.metaobjects.metadata.ktx.loadString +import java.nio.file.Files +import java.util.TreeMap +import kotlin.io.path.isRegularFile +import kotlin.io.path.readText +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertFailsWith +import kotlin.test.assertFalse +import kotlin.test.assertTrue + +/** + * ADR-0019 Amendment 1, the decimal wire rule, as the Kotlin generators write it. The rule lives in + * a generated, per-package `MetaDecimalWire.kt` that the entity, table and controller call, so the + * generated code imports nothing from MetaObjects. What is pinned here is that file and WIRING it in: the property + * annotations (W1/W2), the filter operand parser (W2), the unreadable-body handler (W2), the + * PATCH bind (W2/W3), the SQLite read at 15 digits (W4) and the TPH allowlist admitting decimals. + */ +@OptIn(org.jetbrains.kotlin.compiler.plugin.ExperimentalCompilerApi::class) +class KotlinDecimalWireTest { + + private val model = """{ + "metadata.root": { "package": "acme::shop", "children": [ + { "object.entity": { "name": "Item", "children": [ + { "field.long": { "name": "id" } }, + { "field.string": { "name": "name", "@maxLength": 40 } }, + { "field.decimal": { "name": "price", "@precision": 12, "@scale": 2, "@filterable": true } }, + { "source.rdb": { "@table": "items" } }, + { "identity.primary": { "@fields": ["id"], "@generation": "increment" } } + ] } } + ] } + }""".trimIndent() + + private val tphModel = """{ + "metadata.root": { "package": "acme::shop", "children": [ + { "object.entity": { "name": "Charge", "@discriminator": "kind", "children": [ + { "source.rdb": { "@table": "charges" } }, + { "field.long": { "name": "id" } }, + { "field.enum": { "name": "kind", "@values": ["Card", "Cash"] } }, + { "field.decimal": { "name": "amount", "@precision": 12, "@scale": 2, "@filterable": true } }, + { "identity.primary": { "name": "pk", "@fields": ["id"], "@generation": "increment" } } + ] } }, + { "object.entity": { "name": "CardCharge", "extends": "Charge", "@discriminatorValue": "Card", "children": [ + { "field.decimal": { "name": "fee", "@precision": 12, "@scale": 2, "@filterable": true } } + ] } } + ] } + }""".trimIndent() + + private fun emit(json: String, generators: List): Map { + val outDir = Files.createTempDirectory("kt-decimal-wire-") + try { + val loader = loadString("decimal-wire", json) + for (gen in generators) { + gen.setArgs(mapOf("outputDir" to outDir.toString())) + gen.execute(loader) + } + val files = TreeMap() + Files.walk(outDir).use { s -> + s.filter { it.isRegularFile() }.forEach { files[outDir.relativize(it).toString()] = it.readText() } + } + return files + } finally { + outDir.toFile().deleteRecursively() + } + } + + private fun all() = listOf( + KotlinEntityGenerator(), KotlinExposedTableGenerator(), + KotlinFilterAllowlistGenerator(), KotlinSpringControllerGenerator(), + ) + + @Test + fun `a decimal property writes a canonical string and reads a string or a number exactly`() { + val src = emit(model, all()).getValue("acme/shop/Item.kt") + assertTrue("@get:JsonSerialize(using = MetaDecimalWire.Serializer::class)" in src, src) + assertTrue("@param:JsonDeserialize(using = MetaDecimalWire.Deserializer::class)" in src, src) + // Only the decimal carries them. + assertEquals(1, Regex("MetaDecimalWire.Serializer::class").findAll(src).count(), src) + } + + @Test + fun `a filter operand on a decimal goes through the canonical grammar, never a double`() { + val src = emit(model, all()).getValue("acme/shop/ItemController.kt") + assertTrue("MetaDecimalWire.parseFilterOperand" in src, src) + assertFalse("runCatching { java.math.BigDecimal(s) }" in src, src) + } + + @Test + fun `a body that cannot be read is a 400 in the validation envelope naming the field`() { + val src = emit(model, all()).getValue("acme/shop/ItemController.kt") + assertTrue("@ExceptionHandler(org.springframework.http.converter.HttpMessageNotReadableException::class)" in src, src) + assertTrue("MetaDecimalWire.unreadableBody(e)" in src, src) + } + + @Test + fun `a PATCH decimal binds through the same reader as a create body and a bad one is a 400 naming the field`() { + val src = emit(model, all()).getValue("acme/shop/ItemController.kt") + assertTrue("MetaDecimalWire.fromNode(body.get(\"price\"))" in src, src) + assertTrue("mapOf(\"error\" to \"validation\", \"field\" to \"price\")" in src, src) + } + + @Test + fun `SQLite reads a decimal at 15 significant digits`() { + val src = emit(model, all()).getValue("acme/shop/MetaPortableColumnTypes.kt") + assertTrue("MetaDecimalWire.fromDouble15(value.toDouble())" in src, src) + assertTrue("isSqlite() && value is Number && value !is BigDecimal" in src, src) + } + + @Test + fun `a TPH base admits its decimal columns to the allowlist and the filter dispatch`() { + val files = emit(tphModel, all()) + val allowlist = files.getValue("acme/shop/ChargeFilterAllowlist.kt") + assertTrue("\"amount\"" in allowlist, allowlist) + assertTrue("\"fee\"" in allowlist, allowlist) + val controller = files.getValue("acme/shop/ChargeController.kt") + assertTrue("import java.math.BigDecimal" in controller, controller) + assertTrue("\"amount\" -> coerceChargeDecimal(op, raw)" in controller, controller) + } + + @Test + fun `a decimal beyond the declared precision and scale is a 400 on create and on PATCH, never rounded (W3)`() { + val src = emit(model, all()).getValue("acme/shop/ItemController.kt") + // @precision 12, @scale 2: at most 2 fraction digits and 10 integer digits. + val fits = "MetaDecimalWire.fits(" ; val bounds = ", 12, 2)" + assertEquals(2, Regex(Regex.escape(fits) + "[^\\n]*?" + Regex.escape(bounds)).findAll(src).count(), src) + assertTrue("if (!MetaDecimalWire.fits(dto.price, 12, 2))" in src, src) + assertTrue("if (!MetaDecimalWire.fits(vPrice, 12, 2))" in src, src) + assertTrue("\"field\" to \"price\"" in src, src) + } + + @Test + fun `a TPH base and its subtype enforce the declared range on their per-subtype writes`() { + val src = emit(tphModel, all()).getValue("acme/shop/ChargeController.kt") + assertTrue("if (!MetaDecimalWire.fits(dto.fee, 12, 2))" in src, src) + assertTrue("if (!MetaDecimalWire.fits(vFee, 12, 2))" in src, src) + } + + @Test + fun `no generated Kotlin source imports or names a MetaObjects package`() { + for ((name, json) in listOf("item" to model, "tph" to tphModel)) { + for ((path, src) in emit(json, all())) { + if (!path.endsWith(".kt")) continue + // Comments may name a MetaObjects document; code, imports and annotations may not. + val code = src.lineSequence().filterNot { it.trimStart().let { l -> l.startsWith("*") || l.startsWith("/*") || l.startsWith("//") } } + .joinToString("\n") + assertFalse("com.metaobjects." in code, "$name/$path references a MetaObjects package:\n$src") + } + } + } + + @Test + fun `the generated MetaDecimalWire compiles with no MetaObjects runtime and follows the wire rule`() { + val files = emit(model, listOf(KotlinEntityGenerator(), KotlinExposedTableGenerator())) + val sources = files.filterKeys { it.endsWith(".kt") }.map { (path, src) -> + com.tschuchort.compiletesting.SourceFile.kotlin(path.substringAfterLast('/'), src) + } + val result = com.tschuchort.compiletesting.KotlinCompilation().apply { + this.sources = sources + inheritClassPath = true + messageOutputStream = System.out + }.compile() + assertEquals(com.tschuchort.compiletesting.KotlinCompilation.ExitCode.OK, result.exitCode, result.messages) + val wire = result.classLoader.loadClass("acme.shop.MetaDecimalWire") + val instance = wire.getField("INSTANCE").get(null) + fun call(name: String, vararg args: Any?): Any? { + val m = wire.methods.first { it.name == name && it.parameterCount == args.size } + return try { m.invoke(instance, *args) } catch (e: java.lang.reflect.InvocationTargetException) { throw e.targetException } + } + assertEquals("100.5", call("format", java.math.BigDecimal("100.50"))) + assertEquals("0", call("format", java.math.BigDecimal("0.0000"))) + assertEquals(java.math.BigDecimal("12345678901234567.89"), call("parse", "12345678901234567.89")) + assertEquals(java.math.BigDecimal("100.5"), (call("parse", "1.005e2") as java.math.BigDecimal).stripTrailingZeros()) + assertFailsWith { call("parse", "12abc") } + assertEquals(java.math.BigDecimal("0.3"), call("fromDouble15", 0.1 + 0.2)) + assertEquals(true, call("fits", java.math.BigDecimal("99.99"), 4, 2)) + assertEquals(false, call("fits", java.math.BigDecimal("0.001"), 4, 2)) + assertEquals(false, call("fits", java.math.BigDecimal("100.1"), 4, 2)) + } + + @Test + fun `a precision with no scale declares scale 0, as the Java port reads it`() { + val json = model.replace("\"@precision\": 12, \"@scale\": 2,", "\"@precision\": 12,") + val files = emit(json, all()) + assertTrue("MetaDecimalWire.fits(dto.price, 12, 0)" in files.getValue("acme/shop/ItemController.kt")) + assertTrue("portableDecimal(\"price\", 12, 0)" in files.getValue("acme/shop/ItemTable.kt")) + } + + /** The shared vectors (`fixtures/decimal-wire/vectors.json`), found by walking up from the working directory. */ + private fun vectors(): com.fasterxml.jackson.databind.JsonNode { + var dir: java.nio.file.Path? = java.nio.file.Paths.get("").toAbsolutePath() + while (dir != null) { + val f = dir.resolve("fixtures/decimal-wire/vectors.json") + if (Files.isRegularFile(f)) return com.fasterxml.jackson.databind.ObjectMapper().readTree(f.toFile()) + dir = dir.parent + } + throw IllegalStateException("fixtures/decimal-wire/vectors.json not found") + } + + @Test + fun `the generated MetaDecimalWire passes every shared decimal-wire vector`() { + val files = emit(model, listOf(KotlinEntityGenerator(), KotlinExposedTableGenerator())) + val result = com.tschuchort.compiletesting.KotlinCompilation().apply { + sources = files.filterKeys { it.endsWith(".kt") }.map { (path, src) -> + com.tschuchort.compiletesting.SourceFile.kotlin(path.substringAfterLast('/'), src) + } + inheritClassPath = true + messageOutputStream = System.out + }.compile() + assertEquals(com.tschuchort.compiletesting.KotlinCompilation.ExitCode.OK, result.exitCode, result.messages) + val wire = result.classLoader.loadClass("acme.shop.MetaDecimalWire") + val instance = wire.getField("INSTANCE").get(null) + fun call(name: String, arg: Any?): Any? { + val m = wire.methods.first { it.name == name && it.parameterCount == 1 } + return try { m.invoke(instance, arg) } catch (e: java.lang.reflect.InvocationTargetException) { throw e.targetException } + } + // W2: a body value goes through the generated Jackson deserializer, as a request body does. + @Suppress("UNCHECKED_CAST") + val deserializer = result.classLoader.loadClass("acme.shop.MetaDecimalWire\$Deserializer") + .getDeclaredConstructor().newInstance() as com.fasterxml.jackson.databind.JsonDeserializer + val mapper = com.fasterxml.jackson.databind.ObjectMapper().registerModule( + com.fasterxml.jackson.databind.module.SimpleModule().addDeserializer(java.math.BigDecimal::class.java, deserializer)) + + fun expectOrRefuse(v: com.fasterxml.jackson.databind.JsonNode, run: () -> Any?, label: String) { + if (v.has("error") && v.get("error").asBoolean()) { + var refused = false + try { run() } catch (e: Exception) { refused = true } + assertTrue(refused, "$label: expected a refusal") + } else if (v.get("value").isNull) { + assertEquals(null, run(), label) + } else { + val got = run() as java.math.BigDecimal + assertEquals(0, got.compareTo(java.math.BigDecimal(v.get("value").asText())), "$label: got $got") + } + } + + val vs = vectors() + for (v in vs.get("format")) { + assertEquals(v.get("out").asText(), call("format", java.math.BigDecimal(v.get("in").asText())), "format ${v.get("in")}") + } + for (v in vs.get("parse")) { + val json = v.get("json").asText() + expectOrRefuse(v, { mapper.readValue(json, java.math.BigDecimal::class.java) }, "parse $json") + } + for (v in vs.get("filterOperand")) { + val text = v.get("text").asText() + expectOrRefuse(v, { call("parseFilterOperand", text) }, "filterOperand $text") + } + for (v in vs.get("double15")) { + val d = v.get("double").asText().toDouble() + assertEquals(v.get("out").asText(), call("format", call("fromDouble15", d)), "double15 ${v.get("double")}") + } + println("decimal-wire vectors: format=${vs.get("format").size()} parse=${vs.get("parse").size()} " + + "filterOperand=${vs.get("filterOperand").size()} double15=${vs.get("double15").size()}") + } +} diff --git a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGeneratorTest.kt b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGeneratorTest.kt index a59a4e540..da516f79e 100644 --- a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGeneratorTest.kt +++ b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinExposedTableGeneratorTest.kt @@ -1523,14 +1523,14 @@ class KotlinExposedTableGeneratorTest { val src = table.readText() // PK is a native uuid column with the gen_random_uuid() server default. assertTrue( - "val id = uuid(\"id\").defaultExpression(CustomFunction(\"gen_random_uuid\", UUIDColumnType()))" in src, + "val id = portableUuid(\"id\").defaultExpression(CustomFunction(\"gen_random_uuid\", UUIDColumnType()))" in src, "expected uuid PK with gen_random_uuid() default; saw:\n$src", ) // The default-expression machinery is imported. assertTrue("import org.jetbrains.exposed.sql.CustomFunction" in src, src) assertTrue("import org.jetbrains.exposed.sql.UUIDColumnType" in src, src) // Non-key uuid stays a plain uuid column (no default), and required → not nullable. - assertTrue("val ownerId = uuid(\"owner_id\")\n" in src, + assertTrue("val ownerId = portableUuid(\"owner_id\")\n" in src, "expected plain non-null uuid ownerId column; saw:\n$src") assertTrue("override val primaryKey = PrimaryKey(id)" in src, src) // gen_random_uuid() is NOT .autoIncrement() — they are distinct generation arms. @@ -1540,6 +1540,60 @@ class KotlinExposedTableGeneratorTest { } } + // === engine-portable uuid and decimal columns ============================ + + /** + * A `field.uuid` and a `field.decimal` column go through the package-shared + * `MetaPortableColumnTypes.kt` builders, which bind the uuid as text everywhere but Postgres and + * the decimal as a REAL on SQLite. A package with neither gets no such file. Exposed's own + * `uuid()` binds 16 bytes on MySQL and SQLite, so a filter on a text uuid column never matched there. + */ + @Test fun `uuid and decimal columns use the engine portable column types and their support file`() { + fun emit(fields: String, api: String): java.io.File { + val fx = """{ + "metadata.root": { "package": "acme::demo", "children": [ + { "object.entity": { "name": "Asset", "children": [ + { "field.long": { "name": "id" } }, + $fields + { "source.rdb": { "@table": "assets" } }, + { "identity.primary": { "name": "pk", "@fields": ["id"], "@generation": "increment" } } + ] } } + ] } + }""".trimIndent() + val outDir = Files.createTempDirectory("ktbl-portable-") + val gen = KotlinExposedTableGenerator() + gen.setArgs(mapOf("outputDir" to outDir.toString(), "exposedApi" to api)) + gen.execute(loadString("portable", fx)) + return outDir.resolve("acme/demo").toFile() + } + for (api in listOf("0", "1")) { + val dir = emit( + """{ "field.uuid": { "name": "sku" } }, { "field.decimal": { "name": "price", "@precision": 10, "@scale": 2 } },""", + api, + ) + try { + val table = dir.resolve("AssetTable.kt").readText() + assertTrue("val sku = portableUuid(\"sku\").nullable()" in table, table) + assertTrue("val price = portableDecimal(\"price\", 10, 2).nullable()" in table, table) + val support = dir.resolve("MetaPortableColumnTypes.kt").readText() + assertTrue("internal fun Table.portableUuid(name: String)" in support, support) + assertTrue("internal fun Table.portableDecimal(name: String, precision: Int, scale: Int)" in support, support) + assertTrue("if (isPostgres()) delegate.notNullValueToDB(value) else value.toString()" in support, support) + assertTrue("if (isSqlite()) value.toDouble() else delegate.notNullValueToDB(value)" in support, support) + val uuidImport = if (api == "1") "org.jetbrains.exposed.v1.core.java.UUIDColumnType" else "org.jetbrains.exposed.sql.UUIDColumnType" + assertTrue("import $uuidImport" in support, support) + } finally { + dir.parentFile.parentFile.deleteRecursively() + } + } + val none = emit("""{ "field.string": { "name": "label", "@maxLength": 20 } },""", "0") + try { + assertTrue(!none.resolve("MetaPortableColumnTypes.kt").exists(), "no uuid or decimal column, no portable support file") + } finally { + none.parentFile.parentFile.deleteRecursively() + } + } + // === index.lookup coverage =============================================== /** diff --git a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinReportRestSurfaceTest.kt b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinReportRestSurfaceTest.kt index 19a033b77..cf830479b 100644 --- a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinReportRestSurfaceTest.kt +++ b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinReportRestSurfaceTest.kt @@ -149,6 +149,10 @@ class KotlinReportRestSurfaceTest { }""" } + /** The loader's refusal text: the exception's own message and every cause's. */ + private fun loadMessage(e: Throwable): String = + generateSequence(e) { it.cause }.mapNotNull { it.message }.joinToString(" | ") + private fun saleTotalsFiles(json: String): Map = emit(loadString("report-rest-model", json)).filterKeys { "SaleTotals" in it } @@ -156,13 +160,59 @@ class KotlinReportRestSurfaceTest { fun `only a concrete report over a view is served`() { assertEquals(4, saleTotalsFiles(saleTotals()).size) assertEquals(emptyMap(), saleTotalsFiles(saleTotals(source = null)), "sourceless") - assertEquals(emptyMap(), saleTotalsFiles(saleTotals(reportAttrs = """"abstract": true,""")), "abstract") + } + + // R10 and R11 are refused when the model loads, so no generator is ever handed an abstract + // report or one over a kind the lowering skips. The loader's own tests pin the refusals; these + // pin that the Kotlin port sees them as load errors rather than as "generates nothing". + @Test + fun `an abstract report is refused at load, not generated as nothing`() { + val e = assertFailsWith { + saleTotalsFiles(saleTotals(reportAttrs = """"abstract": true,""")) + } + assertTrue("a report cannot be abstract" in loadMessage(e), loadMessage(e)) + } + + @Test + fun `a report over a source kind that is not a view is refused at load`() { for (kind in listOf( """"@kind": "materializedView", "@materializedView": "mv_sale_totals"""", """"@kind": "storedProc", "@procedure": "sale_totals"""", """"@kind": "tableFunction", "@function": "sale_totals"""", )) { - assertEquals(emptyMap(), saleTotalsFiles(saleTotals(source = """{ "source.rdb": { $kind } }""")), kind) + val e = assertFailsWith(kind) { + saleTotalsFiles(saleTotals(source = """{ "source.rdb": { $kind } }""")) + } + assertTrue("a report is read through a view" in loadMessage(e), "$kind: ${loadMessage(e)}") + } + } + + // A derived field whose value is a list or a map has no scalar column to compare or order by, + // so it is neither filterable nor sortable whatever the entity field it reads says. + @Test + fun `a derived field over an array or a map is neither filterable nor sortable`() { + for ((shape, field) in listOf( + "array" to """{ "field.string": { "name": "tags", "isArray": true, "@storage": "jsonb" } }""", + "map" to """{ "field.map": { "name": "attrs", "@valueType": "string", "@storage": "jsonb" } }""", + )) { + val name = if (shape == "array") "tags" else "attrs" + val json = saleTotals().replace( + """{ "field.float": { "name": "score" } },""", + """{ "field.float": { "name": "score" } }, $field,""", + ).replace( + """{ "dimension.attribute": { "name": "source", "@of": "Sale.channel" } },""", + """{ "dimension.attribute": { "name": "source", "@of": "Sale.channel" } }, + { "dimension.attribute": { "name": "$name", "@of": "Sale.$name" } },""", + ).replace(""""@dimensions": ["source"]""", """"@dimensions": ["source", "$name"]""") + val files = saleTotalsFiles(json) + val allowlist = files.getValue("acme/shop/SaleTotalsFilterAllowlist.kt") + val controller = files.getValue("acme/shop/SaleTotalsController.kt") + assertFalse("\"$name\"" in allowlist, "$shape in allowlist:\n$allowlist") + // The row is still read from the table; only the filter and sort surface must not name it. + assertFalse("\"$name\" ->" in controller, "$shape has a filter arm:\n$controller") + assertFalse("SaleTotalsTable.$name greater" in controller || "SaleTotalsTable.$name eq" in controller, controller) + assertFalse("SaleTotalsTable.$name to dir" in controller, "$shape is orderable:\n$controller") + assertFalse("\"$name\"" in controller.substringAfter("SortAllowlist").substringBefore(")"), "$shape sortable") } } @@ -217,12 +267,9 @@ class KotlinReportRestSurfaceTest { "@measures": ["sales"], "children": [ $VIEW ] } } ] } }""" - for (gen in listOf(KotlinEntityGenerator(), KotlinFilterAllowlistGenerator(), KotlinSpringControllerGenerator())) { - val e = assertFailsWith(gen.javaClass.simpleName) { - emit(loadString("report-rest-object", json), listOf(gen)) - } - assertTrue("report 'SaleTotals'" in e.message.orEmpty() && "field.object" in e.message.orEmpty(), e.message) - } + // D5: the loader refuses a dimension over a field.object before any generator runs. + val e = assertFailsWith { emit(loadString("report-rest-object", json)) } + assertTrue("@of 'Sale.shipTo' is a field.object" in loadMessage(e), loadMessage(e)) } // --- The allowlist (Table C) -------------------------------------------------------- @@ -329,7 +376,7 @@ class KotlinReportRestSurfaceTest { assertTrue(" \"heaviest\" -> coerceSaleTotalsDecimal(op, raw)\n" in src, src) assertTrue(" \"avgWeight\" -> coerceSaleTotalsDecimal(op, raw)\n" in src, src) assertTrue(" \"topScore\" -> coerceSaleTotalsFloat(op, raw)\n" in src, src) - assertTrue("runCatching { java.math.BigDecimal(s) }" in src, src) + assertTrue("runCatching { MetaDecimalWire.parseFilterOperand(s) }" in src, src) assertTrue("runCatching { java.lang.Float.parseFloat(s) }" in src, src) assertTrue("\"gt\" -> SaleTotalsTable.avgWeight greater (p.value as BigDecimal)" in src, src) } diff --git a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinReportTableGeneratorTest.kt b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinReportTableGeneratorTest.kt index f3f9cf44a..12234e2c2 100644 --- a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinReportTableGeneratorTest.kt +++ b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinReportTableGeneratorTest.kt @@ -1,6 +1,7 @@ package com.metaobjects.generator.kotlin import com.metaobjects.generator.Generator +import com.metaobjects.MetaDataException import com.metaobjects.generator.GeneratorException import com.metaobjects.loader.MetaDataLoader import com.metaobjects.metadata.ktx.loadDirectory @@ -117,10 +118,10 @@ class KotlinReportTableGeneratorTest { | val labels = long("labels") | val slots = long("slots") | val totalMinutes = long("totalMinutes").nullable() - | val avgMinutes = decimal("avgMinutes", 38, 18).nullable() + | val avgMinutes = portableDecimal("avgMinutes", 38, 18).nullable() | val minMinutes = integer("minMinutes").nullable() | val maxMinutes = integer("maxMinutes").nullable() - | val longShare = decimal("longShare", 38, 18).nullable() + | val longShare = portableDecimal("longShare", 38, 18).nullable() |} |""".trimMargin(), canonical().getValue("fitness/ProgramMinutesTable.kt"), @@ -175,12 +176,13 @@ class KotlinReportTableGeneratorTest { @Test fun `the naming strategy applies to the derived field name`() { - // The project DEFAULT (snake_case, ADR-0058) -- not this corpus's committed - // naming.columns: literal, which canonical()/canonicalLoader() apply; a plain - // loadDirectory() here keeps the root at PhysicalNaming.DEFAULT. - val src = emit(loadDirectory("report-table-default-naming", canonicalDir())) - .getValue("fitness/ProgramMinutesTable.kt") - assertTrue(" val avgMinutes = decimal(\"avg_minutes\", 38, 18).nullable()\n" in src, src) + // The project DEFAULT (snake_case, ADR-0058), set explicitly: a directory loader now finds + // this corpus's committed naming.columns: literal config by itself (B1), so the root is + // put back to PhysicalNaming.DEFAULT to pin the strategy rather than the discovery. + val loader = loadDirectory("report-table-default-naming", canonicalDir()) + loader.root.physicalNaming = com.metaobjects.database.PhysicalNaming.DEFAULT + val src = emit(loader).getValue("fitness/ProgramMinutesTable.kt") + assertTrue(" val avgMinutes = portableDecimal(\"avg_minutes\", 38, 18).nullable()\n" in src, src) assertTrue(" val programTitle = varchar(\"program_title\", 200)\n" in src, src) } @@ -193,7 +195,7 @@ class KotlinReportTableGeneratorTest { assertFalse("org.jetbrains.exposed.sql" in byMonth, byMonth) assertTrue(" val createdAtMonth = date(\"createdAtMonth\")\n" in byMonth, byMonth) val minutes = files.getValue("fitness/ProgramMinutesTable.kt") - assertTrue(" val avgMinutes = decimal(\"avgMinutes\", 38, 18).nullable()\n" in minutes, minutes) + assertTrue(" val avgMinutes = portableDecimal(\"avgMinutes\", 38, 18).nullable()\n" in minutes, minutes) } @Test @@ -237,8 +239,8 @@ class KotlinReportTableGeneratorTest { | val weeks = long("weeks") | val totalMinutes = long("totalMinutes").nullable() | val totalMinutesOrZero = long("totalMinutesOrZero") - | val longShare = decimal("longShare", 38, 18).nullable() - | val longShareOrZero = decimal("longShareOrZero", 38, 18) + | val longShare = portableDecimal("longShare", 38, 18).nullable() + | val longShareOrZero = portableDecimal("longShareOrZero", 38, 18) |} |""".trimMargin(), canonical().getValue("fitness/ProgramRosterTable.kt"), @@ -253,7 +255,7 @@ class KotlinReportTableGeneratorTest { // With no dimensions and no @spine, a defaulted sum and ratio are still non-null. val filled = canonical().getValue("fitness/FitnessTotalsFilledTable.kt") assertTrue(" val totalMinutesOrZero = long(\"totalMinutesOrZero\")\n" in filled, filled) - assertTrue(" val longShareOrZero = decimal(\"longShareOrZero\", 38, 18)\n" in filled, filled) + assertTrue(" val longShareOrZero = portableDecimal(\"longShareOrZero\", 38, 18)\n" in filled, filled) assertFalse(".nullable()" in filled, filled) } @@ -324,9 +326,9 @@ class KotlinReportTableGeneratorTest { // The Exposed columns. assertTrue(" val revenueOrZero = long(\"revenueOrZero\")\n" in table, table) assertTrue(" val smallestOrZero = long(\"smallestOrZero\")\n" in table, table) - assertTrue(" val perSaleOrZero = decimal(\"perSaleOrZero\", 38, 18)\n" in table, table) + assertTrue(" val perSaleOrZero = portableDecimal(\"perSaleOrZero\", 38, 18)\n" in table, table) assertTrue(" val revenue = long(\"revenue\").nullable()\n" in table, table) - assertTrue(" val perSale = decimal(\"perSale\", 38, 18).nullable()\n" in table, table) + assertTrue(" val perSale = portableDecimal(\"perSale\", 38, 18).nullable()\n" in table, table) // The row data class: a non-null property has no `?` and no `= null` default. assertTrue(" public val revenueOrZero: Long,\n" in row, row) assertTrue(" public val smallestOrZero: Long,\n" in row, row) @@ -425,13 +427,17 @@ class KotlinReportTableGeneratorTest { } @Test - fun `a report over a kind the lowering skips generates nothing`() { + fun `a report over a kind the lowering skips is refused at load`() { + // R10: the loader refuses it, so no generator is handed a report it would skip. for (kind in listOf( """"@kind": "materializedView", "@materializedView": "mv_sale_totals"""", """"@kind": "storedProc", "@procedure": "sale_totals"""", """"@kind": "tableFunction", "@function": "sale_totals"""", )) { - assertEquals(emptyMap(), reportFiles(model(reportSource = """{ "source.rdb": { $kind } }""")), kind) + val e = assertFailsWith(kind) { + reportFiles(model(reportSource = """{ "source.rdb": { $kind } }""")) + } + assertTrue("a report is read through a view" in loadMessage(e), "$kind: ${loadMessage(e)}") } } @@ -651,16 +657,16 @@ class KotlinReportTableGeneratorTest { // --- What generates nothing, and what is refused --------------------------------------- @Test - fun `an abstract view-backed report generates nothing`() { - // An abstract object gets no table object in this port, report or not. The - // TypeScript, Java and Python runtimes still read the view (docs: Known limits). + fun `an abstract report is refused at load`() { + // R11: a report cannot be abstract, so the generators never see one. val json = model().replace(""""name": "SaleTotals",""", """"name": "SaleTotals", "abstract": true,""") assertTrue("\"abstract\": true" in json) - assertEquals(emptyMap(), reportFiles(json)) + val e = assertFailsWith { reportFiles(json) } + assertTrue("a report cannot be abstract" in loadMessage(e), loadMessage(e)) } @Test - fun `a dimension over a field object is refused, naming the report and the dimension`() { + fun `a dimension over a field object is refused at load, naming the dimension's field`() { val json = """{ "metadata.root": { "package": "acme::shop", "children": [ { "object.value": { "name": "Address", "children": [ { "field.string": { "name": "city" } } ] } }, @@ -677,17 +683,15 @@ class KotlinReportTableGeneratorTest { "children": [ { "source.rdb": { "@kind": "view", "@view": "v_sale_totals" } } ] } } ] } }""" - val e = assertFailsWith { reportFiles(json) } - assertEquals( - "report \"SaleTotals\": its dimension \"destination\" reads \"acme::shop::Sale.shipTo\", a " + - "field.object. A report over a field.object is not supported; group by a scalar field.", - e.message, - ) - // The same report with no view generates nothing and is not refused. - assertEquals(emptyMap(), reportFiles(json.replace( - """"children": [ { "source.rdb": { "@kind": "view", "@view": "v_sale_totals" } } ]""", """"children": []"""))) + // D5: refused by the loader, before the table generator's own backstop could run. + val e = assertFailsWith { reportFiles(json) } + assertTrue("@of 'Sale.shipTo' is a field.object" in loadMessage(e), loadMessage(e)) } + /** The loader's refusal text: the exception's own message and every cause's. */ + private fun loadMessage(e: Throwable): String = + generateSequence(e) { it.cause }.mapNotNull { it.message }.joinToString(" | ") + // --- The emitted reports build ------------------------------------------------------ @Test diff --git a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinTypeMapperTest.kt b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinTypeMapperTest.kt index c12399824..994f357d0 100644 --- a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinTypeMapperTest.kt +++ b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinTypeMapperTest.kt @@ -419,7 +419,7 @@ class KotlinTypeMapperTest { // @precision/@scale flow through (the corpus declares 9,4). Column name // snake_case-d for Postgres convention (mirrors the other arms). val f = DecimalField.create("preciseKg", 9, 4) - assertEquals("decimal(\"precise_kg\", 9, 4)", KotlinTypeMapper.exposedColumnSpec(f)) + assertEquals("portableDecimal(\"precise_kg\", 9, 4)", KotlinTypeMapper.exposedColumnSpec(f)) // `decimal(...)` is a member of Table — no extra import line. assertNull(KotlinTypeMapper.exposedColumnImport(f)) } @@ -432,7 +432,7 @@ class KotlinTypeMapperTest { assertEquals("java.util", tn.packageName) assertEquals("UUID", tn.simpleName) // Column name snake_case-d for Postgres convention. - assertEquals("uuid(\"external_id\")", KotlinTypeMapper.exposedColumnSpec(f)) + assertEquals("portableUuid(\"external_id\")", KotlinTypeMapper.exposedColumnSpec(f)) // `uuid(...)` is a Table member — no extra import line needed. assertNull(KotlinTypeMapper.exposedColumnImport(f)) } diff --git a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinUuidPrimaryKeyCodegenTest.kt b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinUuidPrimaryKeyCodegenTest.kt index 860fa094f..d348c1316 100644 --- a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinUuidPrimaryKeyCodegenTest.kt +++ b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinUuidPrimaryKeyCodegenTest.kt @@ -141,8 +141,8 @@ class KotlinUuidPrimaryKeyCodegenTest { "acme/demo/PostTable.kt") // Exposed's typed references() requires the FK column's SQL type to match the // referenced PK's — long("author_id").references(Column) does not compile. - assertTrue("val authorId = uuid(\"author_id\").references(AuthorTable.id)" in src, - "expected uuid(\"author_id\").references(AuthorTable.id); saw:\n$src") + assertTrue("val authorId = portableUuid(\"author_id\").references(AuthorTable.id)" in src, + "expected portableUuid(\"author_id\").references(AuthorTable.id); saw:\n$src") assertTrue("long(\"author_id\")" !in src, "FK to a uuid PK must not be a long column; saw:\n$src") } @@ -150,8 +150,8 @@ class KotlinUuidPrimaryKeyCodegenTest { @Test fun `inferred inverse FK column from a uuid-PK owner is a uuid column`() { val src = generate(KotlinExposedTableGenerator(), "kuuid-fk-inv", uuidFkFixture, "acme/demo/CommentTable.kt") - assertTrue("val authorId = uuid(\"author_id\").references(AuthorTable.id)" in src, - "expected inferred uuid(\"author_id\") FK on CommentTable; saw:\n$src") + assertTrue("val authorId = portableUuid(\"author_id\").references(AuthorTable.id)" in src, + "expected inferred portableUuid(\"author_id\") FK on CommentTable; saw:\n$src") assertTrue("long(\"author_id\")" !in src, "inverse FK to a uuid-PK owner must not be a long column; saw:\n$src") } diff --git a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-controller/acme/blog/AuthorController.kt b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-controller/acme/blog/AuthorController.kt index 01685f2a8..7a1f14364 100644 --- a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-controller/acme/blog/AuthorController.kt +++ b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-controller/acme/blog/AuthorController.kt @@ -363,6 +363,11 @@ class AuthorController(private val objectMapper: ObjectMapper, private val valid else ResponseEntity.noContent().build() } + /** A request body that cannot be read is a client error naming the field, not a 500. */ + @ExceptionHandler(org.springframework.http.converter.HttpMessageNotReadableException::class) + fun handleUnreadableBody(e: org.springframework.http.converter.HttpMessageNotReadableException): ResponseEntity = + ResponseEntity.badRequest().body(MetaDecimalWire.unreadableBody(e) as Any) + /** * A database constraint violation is a CLIENT error, not a 500: the * identity.reference / identity.secondary that declare these constraints are the diff --git a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-controller/acme/blog/MetaDecimalWire.kt b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-controller/acme/blog/MetaDecimalWire.kt new file mode 100644 index 000000000..4a21e7ec8 --- /dev/null +++ b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-controller/acme/blog/MetaDecimalWire.kt @@ -0,0 +1,124 @@ +package acme.blog + +import com.fasterxml.jackson.core.JsonGenerator +import com.fasterxml.jackson.core.JsonParser +import com.fasterxml.jackson.core.JsonToken +import com.fasterxml.jackson.databind.DeserializationContext +import com.fasterxml.jackson.databind.JsonMappingException +import com.fasterxml.jackson.databind.JsonNode +import com.fasterxml.jackson.databind.SerializerProvider +import com.fasterxml.jackson.databind.deser.std.StdDeserializer +import com.fasterxml.jackson.databind.ser.std.StdSerializer +import java.math.BigDecimal +import java.math.MathContext +import java.math.RoundingMode + +/** + * GENERATED (ADR-0019 Amendment 1) — do not hand-edit. The decimal wire rule: a decimal is + * written as a canonical string (plain notation, no trailing fraction zeros, zero as "0"), + * and read from a string or number in the JSON-number grammar, exactly, never through a + * binary float. Dependency-free (Jackson only). + */ +internal object MetaDecimalWire { + private val JSON_NUMBER = Regex("-?(0|[1-9][0-9]*)(\\.[0-9]+)?([eE][+-]?[0-9]+)?") + private const val MAX_INTEGER_DIGITS = 131072 + private const val MAX_FRACTION_DIGITS = 16383 + private const val SQLITE_SIGNIFICANT_DIGITS = 15 + + /** The canonical string of a decimal: plain notation, no trailing fraction zeros, zero as "0". */ + fun format(value: BigDecimal?): String? { + if (value == null) return null + if (value.signum() == 0) return "0" + return value.stripTrailingZeros().toPlainString() + } + + /** Strict read of a decimal string (the JSON-number grammar; an exponent is allowed). */ + fun parse(text: String?): BigDecimal { + require(text != null && JSON_NUMBER.matches(text)) { "not a decimal: " + text } + val value = try { + BigDecimal(text) + } catch (e: NumberFormatException) { + throw IllegalArgumentException("decimal out of range: " + text) + } + return checkRange(value, text) + } + + /** A filter operand: parsed exactly, the same grammar as a body value. */ + fun parseFilterOperand(text: String): BigDecimal = parse(text) + + /** A JSON number token already parsed as a decimal (never a double). */ + fun fromJsonNumber(value: BigDecimal): BigDecimal = checkRange(value, value.toString()) + + /** A decimal from a JSON node: a string or a number, anything else is rejected. */ + fun fromNode(node: JsonNode): BigDecimal { + if (node.isTextual) return parse(node.asText()) + if (node.isNumber) return fromJsonNumber(node.decimalValue()) + throw IllegalArgumentException("not a decimal: " + node) + } + + /** SQLite keeps a REAL: a stored double reads at 15 significant digits, so 0.1 + 0.2 reads 0.3. */ + fun fromDouble15(value: Double): BigDecimal { + require(!value.isNaN() && !value.isInfinite()) { "not a finite decimal: " + value } + if (value == 0.0) return BigDecimal.ZERO + val rounded = BigDecimal(value).round(MathContext(SQLITE_SIGNIFICANT_DIGITS, RoundingMode.HALF_EVEN)) + return if (rounded.signum() == 0) BigDecimal.ZERO else rounded.stripTrailingZeros() + } + + /** True when [value] fits decimal(precision, scale): at most scale fraction digits and precision - scale integer digits. */ + fun fits(value: BigDecimal?, precision: Int, scale: Int): Boolean { + if (value == null || value.signum() == 0) return true + val stripped = value.stripTrailingZeros() + val integerDigits = maxOf(0L, stripped.precision().toLong() - stripped.scale()) + val fractionDigits = maxOf(0, stripped.scale()) + return fractionDigits <= scale && integerDigits <= precision.toLong() - scale + } + + private fun checkRange(value: BigDecimal, shown: String): BigDecimal { + if (value.signum() == 0) return BigDecimal.ZERO + val stripped = value.stripTrailingZeros() + val integerDigits = stripped.precision().toLong() - stripped.scale() + require(integerDigits <= MAX_INTEGER_DIGITS && stripped.scale() <= MAX_FRACTION_DIGITS) { + "decimal out of range: " + shown + } + return value + } + + /** The validation envelope for a request body Jackson could not read: names the field when it knows it. */ + fun unreadableBody(failure: Throwable): Map { + val out = LinkedHashMap() + out["error"] = "validation" + var t: Throwable? = failure + while (t != null) { + if (t is JsonMappingException && t.path.isNotEmpty()) { + val name = t.path[t.path.size - 1].fieldName + if (name != null) { out["field"] = name; break } + } + t = if (t.cause === t) null else t.cause + } + return out + } + + /** Writes a decimal as its canonical string. */ + class Serializer : StdSerializer(BigDecimal::class.java) { + override fun serialize(value: BigDecimal, gen: JsonGenerator, provider: SerializerProvider) { + gen.writeString(format(value)) + } + } + + /** Reads a decimal from a string or a number, exactly; anything else is a 400-shaped mapping error. */ + class Deserializer : StdDeserializer(BigDecimal::class.java) { + override fun deserialize(p: JsonParser, ctxt: DeserializationContext): BigDecimal? { + val token = p.currentToken() + try { + if (token == JsonToken.VALUE_STRING) return parse(p.text) + if (token == JsonToken.VALUE_NUMBER_INT || token == JsonToken.VALUE_NUMBER_FLOAT) { + return fromJsonNumber(p.decimalValue) + } + } catch (e: IllegalArgumentException) { + return ctxt.handleWeirdStringValue(BigDecimal::class.java, p.text, "%s", e.message) as BigDecimal? + } + return ctxt.handleUnexpectedToken(BigDecimal::class.java, p) as BigDecimal? + } + } +} + diff --git a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/MetaDecimalWire.kt b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/MetaDecimalWire.kt new file mode 100644 index 000000000..203e5cdd5 --- /dev/null +++ b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/MetaDecimalWire.kt @@ -0,0 +1,124 @@ +package acme.catalog + +import com.fasterxml.jackson.core.JsonGenerator +import com.fasterxml.jackson.core.JsonParser +import com.fasterxml.jackson.core.JsonToken +import com.fasterxml.jackson.databind.DeserializationContext +import com.fasterxml.jackson.databind.JsonMappingException +import com.fasterxml.jackson.databind.JsonNode +import com.fasterxml.jackson.databind.SerializerProvider +import com.fasterxml.jackson.databind.deser.std.StdDeserializer +import com.fasterxml.jackson.databind.ser.std.StdSerializer +import java.math.BigDecimal +import java.math.MathContext +import java.math.RoundingMode + +/** + * GENERATED (ADR-0019 Amendment 1) — do not hand-edit. The decimal wire rule: a decimal is + * written as a canonical string (plain notation, no trailing fraction zeros, zero as "0"), + * and read from a string or number in the JSON-number grammar, exactly, never through a + * binary float. Dependency-free (Jackson only). + */ +internal object MetaDecimalWire { + private val JSON_NUMBER = Regex("-?(0|[1-9][0-9]*)(\\.[0-9]+)?([eE][+-]?[0-9]+)?") + private const val MAX_INTEGER_DIGITS = 131072 + private const val MAX_FRACTION_DIGITS = 16383 + private const val SQLITE_SIGNIFICANT_DIGITS = 15 + + /** The canonical string of a decimal: plain notation, no trailing fraction zeros, zero as "0". */ + fun format(value: BigDecimal?): String? { + if (value == null) return null + if (value.signum() == 0) return "0" + return value.stripTrailingZeros().toPlainString() + } + + /** Strict read of a decimal string (the JSON-number grammar; an exponent is allowed). */ + fun parse(text: String?): BigDecimal { + require(text != null && JSON_NUMBER.matches(text)) { "not a decimal: " + text } + val value = try { + BigDecimal(text) + } catch (e: NumberFormatException) { + throw IllegalArgumentException("decimal out of range: " + text) + } + return checkRange(value, text) + } + + /** A filter operand: parsed exactly, the same grammar as a body value. */ + fun parseFilterOperand(text: String): BigDecimal = parse(text) + + /** A JSON number token already parsed as a decimal (never a double). */ + fun fromJsonNumber(value: BigDecimal): BigDecimal = checkRange(value, value.toString()) + + /** A decimal from a JSON node: a string or a number, anything else is rejected. */ + fun fromNode(node: JsonNode): BigDecimal { + if (node.isTextual) return parse(node.asText()) + if (node.isNumber) return fromJsonNumber(node.decimalValue()) + throw IllegalArgumentException("not a decimal: " + node) + } + + /** SQLite keeps a REAL: a stored double reads at 15 significant digits, so 0.1 + 0.2 reads 0.3. */ + fun fromDouble15(value: Double): BigDecimal { + require(!value.isNaN() && !value.isInfinite()) { "not a finite decimal: " + value } + if (value == 0.0) return BigDecimal.ZERO + val rounded = BigDecimal(value).round(MathContext(SQLITE_SIGNIFICANT_DIGITS, RoundingMode.HALF_EVEN)) + return if (rounded.signum() == 0) BigDecimal.ZERO else rounded.stripTrailingZeros() + } + + /** True when [value] fits decimal(precision, scale): at most scale fraction digits and precision - scale integer digits. */ + fun fits(value: BigDecimal?, precision: Int, scale: Int): Boolean { + if (value == null || value.signum() == 0) return true + val stripped = value.stripTrailingZeros() + val integerDigits = maxOf(0L, stripped.precision().toLong() - stripped.scale()) + val fractionDigits = maxOf(0, stripped.scale()) + return fractionDigits <= scale && integerDigits <= precision.toLong() - scale + } + + private fun checkRange(value: BigDecimal, shown: String): BigDecimal { + if (value.signum() == 0) return BigDecimal.ZERO + val stripped = value.stripTrailingZeros() + val integerDigits = stripped.precision().toLong() - stripped.scale() + require(integerDigits <= MAX_INTEGER_DIGITS && stripped.scale() <= MAX_FRACTION_DIGITS) { + "decimal out of range: " + shown + } + return value + } + + /** The validation envelope for a request body Jackson could not read: names the field when it knows it. */ + fun unreadableBody(failure: Throwable): Map { + val out = LinkedHashMap() + out["error"] = "validation" + var t: Throwable? = failure + while (t != null) { + if (t is JsonMappingException && t.path.isNotEmpty()) { + val name = t.path[t.path.size - 1].fieldName + if (name != null) { out["field"] = name; break } + } + t = if (t.cause === t) null else t.cause + } + return out + } + + /** Writes a decimal as its canonical string. */ + class Serializer : StdSerializer(BigDecimal::class.java) { + override fun serialize(value: BigDecimal, gen: JsonGenerator, provider: SerializerProvider) { + gen.writeString(format(value)) + } + } + + /** Reads a decimal from a string or a number, exactly; anything else is a 400-shaped mapping error. */ + class Deserializer : StdDeserializer(BigDecimal::class.java) { + override fun deserialize(p: JsonParser, ctxt: DeserializationContext): BigDecimal? { + val token = p.currentToken() + try { + if (token == JsonToken.VALUE_STRING) return parse(p.text) + if (token == JsonToken.VALUE_NUMBER_INT || token == JsonToken.VALUE_NUMBER_FLOAT) { + return fromJsonNumber(p.decimalValue) + } + } catch (e: IllegalArgumentException) { + return ctxt.handleWeirdStringValue(BigDecimal::class.java, p.text, "%s", e.message) as BigDecimal? + } + return ctxt.handleUnexpectedToken(BigDecimal::class.java, p) as BigDecimal? + } + } +} + diff --git a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/MetaPortableColumnTypes.kt b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/MetaPortableColumnTypes.kt new file mode 100644 index 000000000..2be4edf2a --- /dev/null +++ b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/MetaPortableColumnTypes.kt @@ -0,0 +1,51 @@ +package acme.catalog + +import java.math.BigDecimal +import java.util.UUID +import org.jetbrains.exposed.sql.Column +import org.jetbrains.exposed.sql.ColumnType +import org.jetbrains.exposed.sql.DecimalColumnType +import org.jetbrains.exposed.sql.Table +import org.jetbrains.exposed.sql.UUIDColumnType +import org.jetbrains.exposed.sql.vendors.currentDialect + +private fun isPostgres(): Boolean = currentDialect.name.equals("postgresql", ignoreCase = true) + +private fun isSqlite(): Boolean = currentDialect.name.equals("sqlite", ignoreCase = true) + +/** GENERATED — do not hand-edit. A `java.util.UUID` column: native `uuid` on Postgres, text elsewhere. */ +internal class MetaPortableUuidColumnType : ColumnType() { + private val delegate = UUIDColumnType() + override fun sqlType(): String = delegate.sqlType() + override fun valueFromDB(value: Any): UUID = requireNotNull(delegate.valueFromDB(value)) { "null uuid column value" } + override fun notNullValueToDB(value: UUID): Any = + if (isPostgres()) delegate.notNullValueToDB(value) else value.toString() + override fun nonNullValueToString(value: UUID): String = + if (isPostgres()) delegate.nonNullValueToString(value) else "'$value'" +} + +/** Column builder for `field.uuid`: a `Column`. */ +internal fun Table.portableUuid(name: String): Column = + registerColumn(name, MetaPortableUuidColumnType()) + +/** GENERATED — do not hand-edit. A decimal column: the exact decimal everywhere but SQLite, which gets a REAL. */ +internal class MetaPortableDecimalColumnType(precision: Int, scale: Int) : ColumnType() { + private val delegate = DecimalColumnType(precision, scale) + override fun sqlType(): String = delegate.sqlType() + // ADR-0019 Amendment 1 (W4): SQLite stores a REAL, so a value is read at 15 significant + // digits, which turns `sum(0.1, 0.2)` (0.30000000000000004) back into 0.3. + override fun valueFromDB(value: Any): BigDecimal = + if (isSqlite() && value is Number && value !is BigDecimal) { + MetaDecimalWire.fromDouble15(value.toDouble()) + } else { + delegate.valueFromDB(value) + } + override fun notNullValueToDB(value: BigDecimal): Any = + if (isSqlite()) value.toDouble() else delegate.notNullValueToDB(value) + override fun nonNullValueToString(value: BigDecimal): String = delegate.nonNullValueToString(value) +} + +/** Column builder for `field.decimal`: a `Column`. */ +internal fun Table.portableDecimal(name: String, precision: Int, scale: Int): Column = + registerColumn(name, MetaPortableDecimalColumnType(precision, scale)) + diff --git a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/Product.kt b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/Product.kt index 641d79c89..39b36699b 100644 --- a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/Product.kt +++ b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/Product.kt @@ -1,5 +1,7 @@ package acme.catalog +import com.fasterxml.jackson.databind.`annotation`.JsonDeserialize +import com.fasterxml.jackson.databind.`annotation`.JsonSerialize import jakarta.validation.constraints.NotNull import jakarta.validation.constraints.Size import java.math.BigDecimal @@ -16,6 +18,8 @@ public data class Product( @field:NotNull @field:Size(min = 1, max = 40) public val name: String, + @get:JsonSerialize(using = MetaDecimalWire.Serializer::class) + @param:JsonDeserialize(using = MetaDecimalWire.Deserializer::class) public val price: BigDecimal? = null, public val rating: Float? = null, ) { diff --git a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/ProductController.kt b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/ProductController.kt index 187b49444..f7e34f726 100644 --- a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/ProductController.kt +++ b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/ProductController.kt @@ -200,7 +200,7 @@ private fun coerceProductFloat(op: String, raw: String): ProductCoercedValue? { } private fun coerceProductDecimal(op: String, raw: String): ProductCoercedValue? { - val parse: (String) -> Any? = { s -> runCatching { java.math.BigDecimal(s) }.getOrNull() } + val parse: (String) -> Any? = { s -> runCatching { MetaDecimalWire.parseFilterOperand(s) }.getOrNull() } if (op == "in") { val parts = raw.split(",").map { it.trim() } val list = parts.map { parse(it) ?: return null } @@ -378,6 +378,7 @@ class ProductController(private val objectMapper: ObjectMapper, private val vali @PostMapping fun create(@RequestBody dto: Product): ResponseEntity = transaction { if (validator.validate(dto).isNotEmpty()) return@transaction ResponseEntity.badRequest().body(mapOf("error" to "validation") as Any) + if (!MetaDecimalWire.fits(dto.price, 10, 2)) return@transaction ResponseEntity.badRequest().body(mapOf("error" to "validation", "field" to "price") as Any) val newId = ProductTable.insert { it[name] = dto.name it[price] = dto.price @@ -398,8 +399,9 @@ class ProductController(private val objectMapper: ObjectMapper, private val vali if (hasName && validator.validateValue(Product::class.java, "name", vName).isNotEmpty()) return@transaction ResponseEntity.badRequest().body(mapOf("error" to "validation") as Any) val hasPrice = body.has("price") val nullPrice = hasPrice && body.get("price").isNull - val vPrice: java.math.BigDecimal? = if (hasPrice && !nullPrice) objectMapper.treeToValue(body.get("price"), object : TypeReference() {}) else null + val vPrice: java.math.BigDecimal? = if (hasPrice && !nullPrice) runCatching { MetaDecimalWire.fromNode(body.get("price")) }.getOrElse { return@transaction ResponseEntity.badRequest().body(mapOf("error" to "validation", "field" to "price") as Any) } else null if (hasPrice && !nullPrice && validator.validateValue(Product::class.java, "price", vPrice).isNotEmpty()) return@transaction ResponseEntity.badRequest().body(mapOf("error" to "validation") as Any) + if (!MetaDecimalWire.fits(vPrice, 10, 2)) return@transaction ResponseEntity.badRequest().body(mapOf("error" to "validation", "field" to "price") as Any) val hasRating = body.has("rating") val nullRating = hasRating && body.get("rating").isNull val vRating: kotlin.Float? = if (hasRating && !nullRating) objectMapper.treeToValue(body.get("rating"), object : TypeReference() {}) else null @@ -425,6 +427,11 @@ class ProductController(private val objectMapper: ObjectMapper, private val vali else ResponseEntity.noContent().build() } + /** A request body that cannot be read is a client error naming the field, not a 500. */ + @ExceptionHandler(org.springframework.http.converter.HttpMessageNotReadableException::class) + fun handleUnreadableBody(e: org.springframework.http.converter.HttpMessageNotReadableException): ResponseEntity = + ResponseEntity.badRequest().body(MetaDecimalWire.unreadableBody(e) as Any) + /** * A database constraint violation is a CLIENT error, not a 500: the * identity.reference / identity.secondary that declare these constraints are the diff --git a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/ProductTable.kt b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/ProductTable.kt index 5890fa767..df3a86200 100644 --- a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/ProductTable.kt +++ b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-decimal-float-filter/acme/catalog/ProductTable.kt @@ -6,7 +6,7 @@ import org.jetbrains.exposed.sql.Table object ProductTable : Table("products") { val id = long("id").autoIncrement() val name = varchar("name", 40) - val price = decimal("price", 10, 2).nullable() + val price = portableDecimal("price", 10, 2).nullable() val rating = float("rating").nullable() override val primaryKey = PrimaryKey(id) diff --git a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/Auth.kt b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/Auth.kt index 6b5dc3fbd..4b07f675b 100644 --- a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/Auth.kt +++ b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/Auth.kt @@ -1,5 +1,7 @@ package acme.auth +import com.fasterxml.jackson.databind.`annotation`.JsonDeserialize +import com.fasterxml.jackson.databind.`annotation`.JsonSerialize import java.math.BigDecimal import kotlin.Int import kotlin.Long @@ -14,6 +16,8 @@ public data class Auth( public val type: AuthType? = null, public val reference: String? = null, public val quantity: Int? = null, + @get:JsonSerialize(using = MetaDecimalWire.Serializer::class) + @param:JsonDeserialize(using = MetaDecimalWire.Deserializer::class) public val copayAmount: BigDecimal? = null, public val priorAuthNumber: String? = null, ) { diff --git a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthController.kt b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthController.kt index daf28ffd0..3c8e4920f 100644 --- a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthController.kt +++ b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthController.kt @@ -30,6 +30,7 @@ import jakarta.servlet.http.HttpServletRequest import java.io.ByteArrayOutputStream import java.nio.charset.StandardCharsets import java.sql.Timestamp +import java.math.BigDecimal import org.jetbrains.exposed.sql.TextColumnType import org.jetbrains.exposed.sql.castTo import java.time.LocalDate @@ -155,6 +156,7 @@ private fun coerceAuthValue(field: String, op: String, raw: String): AuthCoerced "type" -> if (op == "in") AuthCoercedValue(raw.split(",").map { it.trim() }) else AuthCoercedValue(raw) "reference" -> if (op == "in") AuthCoercedValue(raw.split(",").map { it.trim() }) else AuthCoercedValue(raw) "quantity" -> coerceAuthInt(op, raw) + "copayAmount" -> coerceAuthDecimal(op, raw) "priorAuthNumber" -> if (op == "in") AuthCoercedValue(raw.split(",").map { it.trim() }) else AuthCoercedValue(raw) else -> null } @@ -190,6 +192,16 @@ private fun coerceAuthDouble(op: String, raw: String): AuthCoercedValue? { return AuthCoercedValue(parse(raw) ?: return null) } +private fun coerceAuthDecimal(op: String, raw: String): AuthCoercedValue? { + val parse: (String) -> Any? = { s -> runCatching { MetaDecimalWire.parseFilterOperand(s) }.getOrNull() } + if (op == "in") { + val parts = raw.split(",").map { it.trim() } + val list = parts.map { parse(it) ?: return null } + return AuthCoercedValue(list) + } + return AuthCoercedValue(parse(raw) ?: return null) +} + private fun coerceAuthDate(op: String, raw: String): AuthCoercedValue? { val parse: (String) -> Any? = { s -> runCatching { LocalDate.parse(s) }.getOrNull() } if (op == "in") { @@ -283,6 +295,17 @@ private fun AuthWhereOp(predicates: List): Op? { "isNull" -> if (p.value as Boolean) AuthTable.quantity.isNull() else AuthTable.quantity.isNotNull() else -> throw IllegalStateException("unsupported op for quantity: " + p.op) } + "copayAmount" -> when (p.op) { + "eq" -> AuthTable.copayAmount eq (p.value as BigDecimal) + "ne" -> AuthTable.copayAmount neq (p.value as BigDecimal) + "gt" -> AuthTable.copayAmount greater (p.value as BigDecimal) + "gte" -> AuthTable.copayAmount greaterEq (p.value as BigDecimal) + "lt" -> AuthTable.copayAmount less (p.value as BigDecimal) + "lte" -> AuthTable.copayAmount lessEq (p.value as BigDecimal) + "in" -> AuthTable.copayAmount inList (p.value as List) + "isNull" -> if (p.value as Boolean) AuthTable.copayAmount.isNull() else AuthTable.copayAmount.isNotNull() + else -> throw IllegalStateException("unsupported op for copayAmount: " + p.op) + } "priorAuthNumber" -> when (p.op) { "eq" -> AuthTable.priorAuthNumber eq (p.value as String) "ne" -> AuthTable.priorAuthNumber neq (p.value as String) @@ -482,6 +505,7 @@ class AuthController(private val objectMapper: ObjectMapper, private val validat @PostMapping("/copay") fun createCopay(@RequestBody dto: Auth): ResponseEntity = transaction { if (validator.validateValue(CopayAuthValidation::class.java, "copayAmount", dto.copayAmount).isNotEmpty()) return@transaction ResponseEntity.badRequest().body(mapOf("error" to "validation") as Any) + if (!MetaDecimalWire.fits(dto.copayAmount, 10, 2)) return@transaction ResponseEntity.badRequest().body(mapOf("error" to "validation", "field" to "copayAmount") as Any) if (validator.validateValue(CopayAuthValidation::class.java, "reference", dto.reference).isNotEmpty()) return@transaction ResponseEntity.badRequest().body(mapOf("error" to "validation") as Any) val newId = AuthTable.insert { it[type] = AuthType.Copay @@ -502,8 +526,9 @@ class AuthController(private val objectMapper: ObjectMapper, private val validat try { val hasCopayAmount = body.has("copayAmount") val nullCopayAmount = hasCopayAmount && body.get("copayAmount").isNull - val vCopayAmount: java.math.BigDecimal? = if (hasCopayAmount && !nullCopayAmount) objectMapper.treeToValue(body.get("copayAmount"), object : TypeReference() {}) else null + val vCopayAmount: java.math.BigDecimal? = if (hasCopayAmount && !nullCopayAmount) runCatching { MetaDecimalWire.fromNode(body.get("copayAmount")) }.getOrElse { return@transaction ResponseEntity.badRequest().body(mapOf("error" to "validation", "field" to "copayAmount") as Any) } else null if (hasCopayAmount && !nullCopayAmount && validator.validateValue(CopayAuthValidation::class.java, "copayAmount", vCopayAmount).isNotEmpty()) return@transaction ResponseEntity.badRequest().body(mapOf("error" to "validation") as Any) + if (!MetaDecimalWire.fits(vCopayAmount, 10, 2)) return@transaction ResponseEntity.badRequest().body(mapOf("error" to "validation", "field" to "copayAmount") as Any) val hasReference = body.has("reference") val vReference: kotlin.String? = if (hasReference) objectMapper.treeToValue(body.get("reference"), object : TypeReference() {}) else null if (hasReference && validator.validateValue(CopayAuthValidation::class.java, "reference", vReference).isNotEmpty()) return@transaction ResponseEntity.badRequest().body(mapOf("error" to "validation") as Any) @@ -613,6 +638,11 @@ class AuthController(private val objectMapper: ObjectMapper, private val validat } + /** A request body that cannot be read is a client error naming the field, not a 500. */ + @ExceptionHandler(org.springframework.http.converter.HttpMessageNotReadableException::class) + fun handleUnreadableBody(e: org.springframework.http.converter.HttpMessageNotReadableException): ResponseEntity = + ResponseEntity.badRequest().body(MetaDecimalWire.unreadableBody(e) as Any) + /** * A database constraint violation is a CLIENT error, not a 500: the * identity.reference / identity.secondary that declare these constraints are the diff --git a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthFilterAllowlist.kt b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthFilterAllowlist.kt index 8a6a933a5..98d49ef76 100644 --- a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthFilterAllowlist.kt +++ b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthFilterAllowlist.kt @@ -11,6 +11,7 @@ object AuthFilterAllowlist { "type", "reference", "quantity", + "copayAmount", "priorAuthNumber", ) @@ -19,6 +20,7 @@ object AuthFilterAllowlist { "type" to setOf("eq", "ne", "in", "like", "isNull"), "reference" to setOf("eq", "ne", "in", "like", "isNull"), "quantity" to setOf("eq", "ne", "gt", "gte", "lt", "lte", "in", "isNull"), + "copayAmount" to setOf("eq", "ne", "gt", "gte", "lt", "lte", "in", "isNull"), "priorAuthNumber" to setOf("eq", "ne", "in", "like", "isNull") ) } diff --git a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthLineController.kt b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthLineController.kt index 79fb2dfb0..0d08c9449 100644 --- a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthLineController.kt +++ b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthLineController.kt @@ -362,6 +362,11 @@ class AuthLineController(private val objectMapper: ObjectMapper, private val val else ResponseEntity.noContent().build() } + /** A request body that cannot be read is a client error naming the field, not a 500. */ + @ExceptionHandler(org.springframework.http.converter.HttpMessageNotReadableException::class) + fun handleUnreadableBody(e: org.springframework.http.converter.HttpMessageNotReadableException): ResponseEntity = + ResponseEntity.badRequest().body(MetaDecimalWire.unreadableBody(e) as Any) + /** * A database constraint violation is a CLIENT error, not a 500: the * identity.reference / identity.secondary that declare these constraints are the diff --git a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthTable.kt b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthTable.kt index 04dc6f204..fef003a3b 100644 --- a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthTable.kt +++ b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/AuthTable.kt @@ -8,7 +8,7 @@ object AuthTable : Table("auths") { val type = enumerationByName("type", 64, AuthType::class).nullable() val reference = varchar("reference", 80) val quantity = integer("quantity").nullable() - val copayAmount = decimal("copay_amount", 10, 2).nullable() + val copayAmount = portableDecimal("copay_amount", 10, 2).nullable() val priorAuthNumber = varchar("prior_auth_number", 80).nullable() override val primaryKey = PrimaryKey(id) diff --git a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/MetaDecimalWire.kt b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/MetaDecimalWire.kt new file mode 100644 index 000000000..b2d598725 --- /dev/null +++ b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/MetaDecimalWire.kt @@ -0,0 +1,124 @@ +package acme.auth + +import com.fasterxml.jackson.core.JsonGenerator +import com.fasterxml.jackson.core.JsonParser +import com.fasterxml.jackson.core.JsonToken +import com.fasterxml.jackson.databind.DeserializationContext +import com.fasterxml.jackson.databind.JsonMappingException +import com.fasterxml.jackson.databind.JsonNode +import com.fasterxml.jackson.databind.SerializerProvider +import com.fasterxml.jackson.databind.deser.std.StdDeserializer +import com.fasterxml.jackson.databind.ser.std.StdSerializer +import java.math.BigDecimal +import java.math.MathContext +import java.math.RoundingMode + +/** + * GENERATED (ADR-0019 Amendment 1) — do not hand-edit. The decimal wire rule: a decimal is + * written as a canonical string (plain notation, no trailing fraction zeros, zero as "0"), + * and read from a string or number in the JSON-number grammar, exactly, never through a + * binary float. Dependency-free (Jackson only). + */ +internal object MetaDecimalWire { + private val JSON_NUMBER = Regex("-?(0|[1-9][0-9]*)(\\.[0-9]+)?([eE][+-]?[0-9]+)?") + private const val MAX_INTEGER_DIGITS = 131072 + private const val MAX_FRACTION_DIGITS = 16383 + private const val SQLITE_SIGNIFICANT_DIGITS = 15 + + /** The canonical string of a decimal: plain notation, no trailing fraction zeros, zero as "0". */ + fun format(value: BigDecimal?): String? { + if (value == null) return null + if (value.signum() == 0) return "0" + return value.stripTrailingZeros().toPlainString() + } + + /** Strict read of a decimal string (the JSON-number grammar; an exponent is allowed). */ + fun parse(text: String?): BigDecimal { + require(text != null && JSON_NUMBER.matches(text)) { "not a decimal: " + text } + val value = try { + BigDecimal(text) + } catch (e: NumberFormatException) { + throw IllegalArgumentException("decimal out of range: " + text) + } + return checkRange(value, text) + } + + /** A filter operand: parsed exactly, the same grammar as a body value. */ + fun parseFilterOperand(text: String): BigDecimal = parse(text) + + /** A JSON number token already parsed as a decimal (never a double). */ + fun fromJsonNumber(value: BigDecimal): BigDecimal = checkRange(value, value.toString()) + + /** A decimal from a JSON node: a string or a number, anything else is rejected. */ + fun fromNode(node: JsonNode): BigDecimal { + if (node.isTextual) return parse(node.asText()) + if (node.isNumber) return fromJsonNumber(node.decimalValue()) + throw IllegalArgumentException("not a decimal: " + node) + } + + /** SQLite keeps a REAL: a stored double reads at 15 significant digits, so 0.1 + 0.2 reads 0.3. */ + fun fromDouble15(value: Double): BigDecimal { + require(!value.isNaN() && !value.isInfinite()) { "not a finite decimal: " + value } + if (value == 0.0) return BigDecimal.ZERO + val rounded = BigDecimal(value).round(MathContext(SQLITE_SIGNIFICANT_DIGITS, RoundingMode.HALF_EVEN)) + return if (rounded.signum() == 0) BigDecimal.ZERO else rounded.stripTrailingZeros() + } + + /** True when [value] fits decimal(precision, scale): at most scale fraction digits and precision - scale integer digits. */ + fun fits(value: BigDecimal?, precision: Int, scale: Int): Boolean { + if (value == null || value.signum() == 0) return true + val stripped = value.stripTrailingZeros() + val integerDigits = maxOf(0L, stripped.precision().toLong() - stripped.scale()) + val fractionDigits = maxOf(0, stripped.scale()) + return fractionDigits <= scale && integerDigits <= precision.toLong() - scale + } + + private fun checkRange(value: BigDecimal, shown: String): BigDecimal { + if (value.signum() == 0) return BigDecimal.ZERO + val stripped = value.stripTrailingZeros() + val integerDigits = stripped.precision().toLong() - stripped.scale() + require(integerDigits <= MAX_INTEGER_DIGITS && stripped.scale() <= MAX_FRACTION_DIGITS) { + "decimal out of range: " + shown + } + return value + } + + /** The validation envelope for a request body Jackson could not read: names the field when it knows it. */ + fun unreadableBody(failure: Throwable): Map { + val out = LinkedHashMap() + out["error"] = "validation" + var t: Throwable? = failure + while (t != null) { + if (t is JsonMappingException && t.path.isNotEmpty()) { + val name = t.path[t.path.size - 1].fieldName + if (name != null) { out["field"] = name; break } + } + t = if (t.cause === t) null else t.cause + } + return out + } + + /** Writes a decimal as its canonical string. */ + class Serializer : StdSerializer(BigDecimal::class.java) { + override fun serialize(value: BigDecimal, gen: JsonGenerator, provider: SerializerProvider) { + gen.writeString(format(value)) + } + } + + /** Reads a decimal from a string or a number, exactly; anything else is a 400-shaped mapping error. */ + class Deserializer : StdDeserializer(BigDecimal::class.java) { + override fun deserialize(p: JsonParser, ctxt: DeserializationContext): BigDecimal? { + val token = p.currentToken() + try { + if (token == JsonToken.VALUE_STRING) return parse(p.text) + if (token == JsonToken.VALUE_NUMBER_INT || token == JsonToken.VALUE_NUMBER_FLOAT) { + return fromJsonNumber(p.decimalValue) + } + } catch (e: IllegalArgumentException) { + return ctxt.handleWeirdStringValue(BigDecimal::class.java, p.text, "%s", e.message) as BigDecimal? + } + return ctxt.handleUnexpectedToken(BigDecimal::class.java, p) as BigDecimal? + } + } +} + diff --git a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/MetaPortableColumnTypes.kt b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/MetaPortableColumnTypes.kt new file mode 100644 index 000000000..8e118e02a --- /dev/null +++ b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-tph/acme/auth/MetaPortableColumnTypes.kt @@ -0,0 +1,51 @@ +package acme.auth + +import java.math.BigDecimal +import java.util.UUID +import org.jetbrains.exposed.sql.Column +import org.jetbrains.exposed.sql.ColumnType +import org.jetbrains.exposed.sql.DecimalColumnType +import org.jetbrains.exposed.sql.Table +import org.jetbrains.exposed.sql.UUIDColumnType +import org.jetbrains.exposed.sql.vendors.currentDialect + +private fun isPostgres(): Boolean = currentDialect.name.equals("postgresql", ignoreCase = true) + +private fun isSqlite(): Boolean = currentDialect.name.equals("sqlite", ignoreCase = true) + +/** GENERATED — do not hand-edit. A `java.util.UUID` column: native `uuid` on Postgres, text elsewhere. */ +internal class MetaPortableUuidColumnType : ColumnType() { + private val delegate = UUIDColumnType() + override fun sqlType(): String = delegate.sqlType() + override fun valueFromDB(value: Any): UUID = requireNotNull(delegate.valueFromDB(value)) { "null uuid column value" } + override fun notNullValueToDB(value: UUID): Any = + if (isPostgres()) delegate.notNullValueToDB(value) else value.toString() + override fun nonNullValueToString(value: UUID): String = + if (isPostgres()) delegate.nonNullValueToString(value) else "'$value'" +} + +/** Column builder for `field.uuid`: a `Column`. */ +internal fun Table.portableUuid(name: String): Column = + registerColumn(name, MetaPortableUuidColumnType()) + +/** GENERATED — do not hand-edit. A decimal column: the exact decimal everywhere but SQLite, which gets a REAL. */ +internal class MetaPortableDecimalColumnType(precision: Int, scale: Int) : ColumnType() { + private val delegate = DecimalColumnType(precision, scale) + override fun sqlType(): String = delegate.sqlType() + // ADR-0019 Amendment 1 (W4): SQLite stores a REAL, so a value is read at 15 significant + // digits, which turns `sum(0.1, 0.2)` (0.30000000000000004) back into 0.3. + override fun valueFromDB(value: Any): BigDecimal = + if (isSqlite() && value is Number && value !is BigDecimal) { + MetaDecimalWire.fromDouble15(value.toDouble()) + } else { + delegate.valueFromDB(value) + } + override fun notNullValueToDB(value: BigDecimal): Any = + if (isSqlite()) value.toDouble() else delegate.notNullValueToDB(value) + override fun nonNullValueToString(value: BigDecimal): String = delegate.nonNullValueToString(value) +} + +/** Column builder for `field.decimal`: a `Column`. */ +internal fun Table.portableDecimal(name: String, precision: Int, scale: Int): Column = + registerColumn(name, MetaPortableDecimalColumnType(precision, scale)) + diff --git a/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-view/acme/report/MetaDecimalWire.kt b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-view/acme/report/MetaDecimalWire.kt new file mode 100644 index 000000000..b25b9c55a --- /dev/null +++ b/server/java/codegen-kotlin/src/test/resources/snapshots/entity-with-view/acme/report/MetaDecimalWire.kt @@ -0,0 +1,124 @@ +package acme.report + +import com.fasterxml.jackson.core.JsonGenerator +import com.fasterxml.jackson.core.JsonParser +import com.fasterxml.jackson.core.JsonToken +import com.fasterxml.jackson.databind.DeserializationContext +import com.fasterxml.jackson.databind.JsonMappingException +import com.fasterxml.jackson.databind.JsonNode +import com.fasterxml.jackson.databind.SerializerProvider +import com.fasterxml.jackson.databind.deser.std.StdDeserializer +import com.fasterxml.jackson.databind.ser.std.StdSerializer +import java.math.BigDecimal +import java.math.MathContext +import java.math.RoundingMode + +/** + * GENERATED (ADR-0019 Amendment 1) — do not hand-edit. The decimal wire rule: a decimal is + * written as a canonical string (plain notation, no trailing fraction zeros, zero as "0"), + * and read from a string or number in the JSON-number grammar, exactly, never through a + * binary float. Dependency-free (Jackson only). + */ +internal object MetaDecimalWire { + private val JSON_NUMBER = Regex("-?(0|[1-9][0-9]*)(\\.[0-9]+)?([eE][+-]?[0-9]+)?") + private const val MAX_INTEGER_DIGITS = 131072 + private const val MAX_FRACTION_DIGITS = 16383 + private const val SQLITE_SIGNIFICANT_DIGITS = 15 + + /** The canonical string of a decimal: plain notation, no trailing fraction zeros, zero as "0". */ + fun format(value: BigDecimal?): String? { + if (value == null) return null + if (value.signum() == 0) return "0" + return value.stripTrailingZeros().toPlainString() + } + + /** Strict read of a decimal string (the JSON-number grammar; an exponent is allowed). */ + fun parse(text: String?): BigDecimal { + require(text != null && JSON_NUMBER.matches(text)) { "not a decimal: " + text } + val value = try { + BigDecimal(text) + } catch (e: NumberFormatException) { + throw IllegalArgumentException("decimal out of range: " + text) + } + return checkRange(value, text) + } + + /** A filter operand: parsed exactly, the same grammar as a body value. */ + fun parseFilterOperand(text: String): BigDecimal = parse(text) + + /** A JSON number token already parsed as a decimal (never a double). */ + fun fromJsonNumber(value: BigDecimal): BigDecimal = checkRange(value, value.toString()) + + /** A decimal from a JSON node: a string or a number, anything else is rejected. */ + fun fromNode(node: JsonNode): BigDecimal { + if (node.isTextual) return parse(node.asText()) + if (node.isNumber) return fromJsonNumber(node.decimalValue()) + throw IllegalArgumentException("not a decimal: " + node) + } + + /** SQLite keeps a REAL: a stored double reads at 15 significant digits, so 0.1 + 0.2 reads 0.3. */ + fun fromDouble15(value: Double): BigDecimal { + require(!value.isNaN() && !value.isInfinite()) { "not a finite decimal: " + value } + if (value == 0.0) return BigDecimal.ZERO + val rounded = BigDecimal(value).round(MathContext(SQLITE_SIGNIFICANT_DIGITS, RoundingMode.HALF_EVEN)) + return if (rounded.signum() == 0) BigDecimal.ZERO else rounded.stripTrailingZeros() + } + + /** True when [value] fits decimal(precision, scale): at most scale fraction digits and precision - scale integer digits. */ + fun fits(value: BigDecimal?, precision: Int, scale: Int): Boolean { + if (value == null || value.signum() == 0) return true + val stripped = value.stripTrailingZeros() + val integerDigits = maxOf(0L, stripped.precision().toLong() - stripped.scale()) + val fractionDigits = maxOf(0, stripped.scale()) + return fractionDigits <= scale && integerDigits <= precision.toLong() - scale + } + + private fun checkRange(value: BigDecimal, shown: String): BigDecimal { + if (value.signum() == 0) return BigDecimal.ZERO + val stripped = value.stripTrailingZeros() + val integerDigits = stripped.precision().toLong() - stripped.scale() + require(integerDigits <= MAX_INTEGER_DIGITS && stripped.scale() <= MAX_FRACTION_DIGITS) { + "decimal out of range: " + shown + } + return value + } + + /** The validation envelope for a request body Jackson could not read: names the field when it knows it. */ + fun unreadableBody(failure: Throwable): Map { + val out = LinkedHashMap() + out["error"] = "validation" + var t: Throwable? = failure + while (t != null) { + if (t is JsonMappingException && t.path.isNotEmpty()) { + val name = t.path[t.path.size - 1].fieldName + if (name != null) { out["field"] = name; break } + } + t = if (t.cause === t) null else t.cause + } + return out + } + + /** Writes a decimal as its canonical string. */ + class Serializer : StdSerializer(BigDecimal::class.java) { + override fun serialize(value: BigDecimal, gen: JsonGenerator, provider: SerializerProvider) { + gen.writeString(format(value)) + } + } + + /** Reads a decimal from a string or a number, exactly; anything else is a 400-shaped mapping error. */ + class Deserializer : StdDeserializer(BigDecimal::class.java) { + override fun deserialize(p: JsonParser, ctxt: DeserializationContext): BigDecimal? { + val token = p.currentToken() + try { + if (token == JsonToken.VALUE_STRING) return parse(p.text) + if (token == JsonToken.VALUE_NUMBER_INT || token == JsonToken.VALUE_NUMBER_FLOAT) { + return fromJsonNumber(p.decimalValue) + } + } catch (e: IllegalArgumentException) { + return ctxt.handleWeirdStringValue(BigDecimal::class.java, p.text, "%s", e.message) as BigDecimal? + } + return ctxt.handleUnexpectedToken(BigDecimal::class.java, p) as BigDecimal? + } + } +} + diff --git a/server/java/codegen-spring/pom.xml b/server/java/codegen-spring/pom.xml index b8303d80d..2e29514db 100644 --- a/server/java/codegen-spring/pom.xml +++ b/server/java/codegen-spring/pom.xml @@ -98,11 +98,13 @@ test - + com.fasterxml.jackson.core jackson-databind - test + provided + + + com.metaobjects + metaobjects-core-spring + ${project.version} + test + com.metaobjects metaobjects-om diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/KotlinCodegenMatchesReferenceTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/KotlinCodegenMatchesReferenceTest.kt index f7f388bed..1cbc4943b 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/KotlinCodegenMatchesReferenceTest.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/KotlinCodegenMatchesReferenceTest.kt @@ -62,7 +62,7 @@ internal class KotlinCodegenMatchesReferenceTest { ExpectedColumn("id", families = setOf("long")), ExpectedColumn("tempC", families = setOf("float")), ExpectedColumn("massKg", families = setOf("double")), - ExpectedColumn("preciseKg", families = setOf("decimal")), + ExpectedColumn("preciseKg", families = setOf("decimal", "portableDecimal")), ), ), "ProgramView" to EntityExpectation( @@ -99,8 +99,8 @@ internal class KotlinCodegenMatchesReferenceTest { // through a generated String-typed entity, so there is no mismatch for it to hold). "Asset" to EntityExpectation( columns = listOf( - ExpectedColumn("id", families = setOf("uuid")), - ExpectedColumn("ownerId", families = setOf("uuid")), + ExpectedColumn("id", families = setOf("uuid", "portableUuid")), + ExpectedColumn("ownerId", families = setOf("uuid", "portableUuid")), ExpectedColumn("externalId", families = setOf("uuidString")), ExpectedColumn("payload", families = setOf("jsonb")), ExpectedColumn("recordedAt", families = setOf("instantWithTimeZone")), @@ -119,10 +119,10 @@ internal class KotlinCodegenMatchesReferenceTest { ExpectedColumn("labels", families = setOf("long"), nullable = false), ExpectedColumn("slots", families = setOf("long"), nullable = false), ExpectedColumn("totalMinutes", families = setOf("long"), nullable = true), - ExpectedColumn("avgMinutes", families = setOf("decimal"), nullable = true), + ExpectedColumn("avgMinutes", families = setOf("decimal", "portableDecimal"), nullable = true), ExpectedColumn("minMinutes", families = setOf("integer"), nullable = true), ExpectedColumn("maxMinutes", families = setOf("integer"), nullable = true), - ExpectedColumn("longShare", families = setOf("decimal"), nullable = true), + ExpectedColumn("longShare", families = setOf("decimal", "portableDecimal"), nullable = true), ), report = "v_program_minutes", ), @@ -130,7 +130,7 @@ internal class KotlinCodegenMatchesReferenceTest { columns = listOf( ExpectedColumn("weeks", families = setOf("long"), nullable = false), ExpectedColumn("totalMinutes", families = setOf("long"), nullable = true), - ExpectedColumn("longShare", families = setOf("decimal"), nullable = true), + ExpectedColumn("longShare", families = setOf("decimal", "portableDecimal"), nullable = true), ), report = "v_fitness_totals", ), @@ -172,8 +172,8 @@ internal class KotlinCodegenMatchesReferenceTest { ExpectedColumn("weeks", families = setOf("long"), nullable = false), ExpectedColumn("totalMinutes", families = setOf("long"), nullable = true), ExpectedColumn("totalMinutesOrZero", families = setOf("long"), nullable = false), - ExpectedColumn("longShare", families = setOf("decimal"), nullable = true), - ExpectedColumn("longShareOrZero", families = setOf("decimal"), nullable = false), + ExpectedColumn("longShare", families = setOf("decimal", "portableDecimal"), nullable = true), + ExpectedColumn("longShareOrZero", families = setOf("decimal", "portableDecimal"), nullable = false), ), report = "v_program_roster", ), @@ -210,7 +210,7 @@ internal class KotlinCodegenMatchesReferenceTest { columns = listOf( ExpectedColumn("weeks", families = setOf("long"), nullable = false), ExpectedColumn("totalMinutesOrZero", families = setOf("long"), nullable = false), - ExpectedColumn("longShareOrZero", families = setOf("decimal"), nullable = false), + ExpectedColumn("longShareOrZero", families = setOf("decimal", "portableDecimal"), nullable = false), ), report = "v_fitness_totals_filled", ), @@ -223,9 +223,9 @@ internal class KotlinCodegenMatchesReferenceTest { ExpectedColumn("avgMassKg", families = setOf("double"), nullable = true), ExpectedColumn("maxTempC", families = setOf("float"), nullable = true), ExpectedColumn("minTempC", families = setOf("float"), nullable = true), - ExpectedColumn("totalPreciseKg", families = setOf("decimal"), nullable = true), - ExpectedColumn("avgPreciseKg", families = setOf("decimal"), nullable = true), - ExpectedColumn("maxPreciseKg", families = setOf("decimal"), nullable = true), + ExpectedColumn("totalPreciseKg", families = setOf("decimal", "portableDecimal"), nullable = true), + ExpectedColumn("avgPreciseKg", families = setOf("decimal", "portableDecimal"), nullable = true), + ExpectedColumn("maxPreciseKg", families = setOf("decimal", "portableDecimal"), nullable = true), ExpectedColumn("heavyReadings", families = setOf("long"), nullable = false), ), report = "v_measurement_stats", @@ -233,8 +233,8 @@ internal class KotlinCodegenMatchesReferenceTest { "MeasurementStatsFilled" to EntityExpectation( columns = listOf( ExpectedColumn("readings", families = setOf("long"), nullable = false), - ExpectedColumn("avgPreciseKgOrMinusOne", families = setOf("decimal"), nullable = false), - ExpectedColumn("maxPreciseKgOrZero", families = setOf("decimal"), nullable = false), + ExpectedColumn("avgPreciseKgOrMinusOne", families = setOf("decimal", "portableDecimal"), nullable = false), + ExpectedColumn("maxPreciseKgOrZero", families = setOf("decimal", "portableDecimal"), nullable = false), ), report = "v_measurement_stats_filled", ), @@ -256,10 +256,10 @@ internal class KotlinCodegenMatchesReferenceTest { ExpectedColumn("level", families = setOf("varchar", "enumerationByName"), nullable = false), ExpectedColumn("legacyStatus", families = setOf("customEnumeration"), nullable = true), ExpectedColumn("dateKey", families = setOf("date"), nullable = false), - ExpectedColumn("uuidKey", families = setOf("uuid"), nullable = false), + ExpectedColumn("uuidKey", families = setOf("uuid", "portableUuid"), nullable = false), ExpectedColumn("allRows", families = setOf("long"), nullable = false), ExpectedColumn("totalInts", families = setOf("long"), nullable = true), - ExpectedColumn("avgDec", families = setOf("decimal"), nullable = true), + ExpectedColumn("avgDec", families = setOf("decimal", "portableDecimal"), nullable = true), ExpectedColumn("maxMoney", families = setOf("long"), nullable = true), ExpectedColumn("minLong", families = setOf("long"), nullable = true), ExpectedColumn("highRows", families = setOf("long"), nullable = false), @@ -272,7 +272,7 @@ internal class KotlinCodegenMatchesReferenceTest { ExpectedColumn("createdAtQuarter", families = setOf("date"), nullable = false), ExpectedColumn("createdAtYear", families = setOf("date"), nullable = false), ExpectedColumn("programs", families = setOf("long"), nullable = false), - ExpectedColumn("avgPrice", families = setOf("decimal"), nullable = true), + ExpectedColumn("avgPrice", families = setOf("decimal", "portableDecimal"), nullable = true), ExpectedColumn("cheapest", families = setOf("long"), nullable = true), ExpectedColumn("priciest", families = setOf("long"), nullable = true), ), @@ -290,7 +290,7 @@ internal class KotlinCodegenMatchesReferenceTest { "ProgramsEmptyScope" to EntityExpectation( columns = listOf( ExpectedColumn("programs", families = setOf("long"), nullable = false), - ExpectedColumn("avgPriceOrMinusOne", families = setOf("decimal"), nullable = false), + ExpectedColumn("avgPriceOrMinusOne", families = setOf("decimal", "portableDecimal"), nullable = false), ), report = "v_programs_empty_scope", ), @@ -340,6 +340,26 @@ internal class KotlinCodegenMatchesReferenceTest { ), report = "v_auths_by_kind", ), + "EntryTotals" to EntityExpectation( + columns = listOf( + ExpectedColumn("bucket", families = setOf("varchar"), nullable = false), + ExpectedColumn("entries", families = setOf("long"), nullable = false), + ExpectedColumn("total", families = setOf("decimal", "portableDecimal"), nullable = true), + ExpectedColumn("totalOrZero", families = setOf("decimal", "portableDecimal"), nullable = false), + ExpectedColumn("mean", families = setOf("decimal", "portableDecimal"), nullable = true), + ExpectedColumn("top", families = setOf("decimal", "portableDecimal"), nullable = true), + ExpectedColumn("flaggedShare", families = setOf("decimal", "portableDecimal"), nullable = true), + ExpectedColumn("perFlagged", families = setOf("decimal", "portableDecimal"), nullable = true), + ), + report = "v_entry_totals", + ), + "GadgetEventsByLabel" to EntityExpectation( + columns = listOf( + ExpectedColumn("gadgetLabel", families = setOf("integer"), nullable = false), + ExpectedColumn("gadgetEvents", families = setOf("long"), nullable = false), + ), + report = "v_gadget_events_by_label", + ), ) @Test diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioConformanceTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioConformanceTest.kt index 793b5c0ee..15bf40949 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioConformanceTest.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioConformanceTest.kt @@ -59,7 +59,7 @@ internal class QueryScenarioConformanceTest { "No query scenarios found under fixtures/persistence-conformance/queries — corpus missing?" ) return all - .filter { it.name !in DEFERRED_SCENARIOS.keys } + .filter { it.name !in DEFERRED_SCENARIOS.keys && it.runsOn(ScenarioEngine.POSTGRES) } .stream() .map { Arguments.of(it.name, it) } } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioMySqlTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioMySqlTest.kt index 332dae68d..e39c77ddd 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioMySqlTest.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioMySqlTest.kt @@ -47,7 +47,7 @@ internal class QueryScenarioMySqlTest { @JvmStatic fun scenarios(): Stream { val corpus = ScenarioLoader.findCorpusRoot() - val reports = ScenarioLoader.loadQueries(corpus.resolve("queries")).filter { it.name.startsWith("report-") } + val reports = ScenarioLoader.loadQueries(corpus.resolve("queries")).filter { it.runsOn(ScenarioEngine.MYSQL) } Assumptions.assumeTrue(reports.isNotEmpty(), "No report-* scenarios under fixtures/persistence-conformance/queries") return reports.stream().map { Arguments.of(it.name, it) } } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioRunner.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioRunner.kt index 85d3da6e0..19be047c1 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioRunner.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioRunner.kt @@ -9,6 +9,8 @@ import com.metaobjects.relationship.MetaRelationship import com.metaobjects.integration.kotlin.Scenarios.QueryScenario import com.metaobjects.integration.kotlin.Scenarios.QuerySpec import com.metaobjects.integration.kotlin.tables.AssetTable +import com.metaobjects.integration.kotlin.tables.EntryTable +import com.metaobjects.integration.kotlin.tables.ReferenceDecimal import com.metaobjects.integration.kotlin.tables.AuthTable import com.metaobjects.integration.kotlin.tables.MeasurementTable import com.metaobjects.integration.kotlin.tables.NodeTable @@ -485,7 +487,7 @@ object QueryScenarioRunner { return when { type == "uuid" -> if (raw is UUID) raw else UUID.fromString(raw.toString().lowercase()) // NUMERIC/DECIMAL: exact BigDecimal from the quoted-string authoring form. - type.contains("numeric") || type.contains("decimal") -> BigDecimal(raw.toString()) + type.contains("numeric") || type.contains("decimal") -> ReferenceDecimal.parse(raw.toString()) type.contains("bigint") || type.contains("int8") -> (raw as? Number)?.toLong() ?: raw.toString().toLong() // REAL / float4 → Float; DOUBLE PRECISION / float8 → Double. Check the float types // before the generic `int` substring guard (neither contains "int", but order-safe). @@ -537,6 +539,8 @@ object QueryScenarioRunner { "Session" -> SessionTable "Asset" -> AssetTable "AllTypes" -> AllTypesTable + // ADR-0019 Amendment 1: the decimal subject table. + "Entry" -> EntryTable // FR-017 TPH: the discriminator base + all its subtypes share the single `auths` table. "Auth", "BridgeAuth", "CopayAuth", "PriorAuthAuth" -> AuthTable else -> error("No Exposed Table registered for entity '$entity' — extend QueryScenarioRunner.tableFor or tables/ReportViews") @@ -685,6 +689,9 @@ object QueryScenarioRunner { // literal (e.g. an `ownerId eq` filter), parse it. Lower-cased first so an // upper-case literal still matches the lowercase-canonical stored value. type == "uuid" -> if (raw is UUID) raw else UUID.fromString(raw.toString().lowercase()) + // A decimal operand is parsed EXACTLY (plain or exponent form), never through a double + // (W2); the column type binds it as an exact decimal, or on SQLite as a number (W4). + type.contains("numeric") || type.contains("decimal") -> ReferenceDecimal.parse(raw.toString()) type.contains("bigint") || type.contains("int8") -> (raw as? Number)?.toLong() ?: raw.toString().toLong() type.contains("int") -> (raw as? Number)?.toInt() ?: raw.toString().toInt() // A filter on a DATE column (a report's date dimension) supplies the day as text. @@ -745,6 +752,10 @@ object QueryScenarioRunner { // LocalDateTime so it lands on the no-`Z` branch. Instant (the TZ-aware shape) is left // as-is for Normalization's Instant branch. if (v is Timestamp) v = v.toLocalDateTime() + // A decimal reads as a canonical STRING (W1): plain notation, no trailing fraction zeros, + // zero as "0". The typed gate then requires exactly that, so a binary float anywhere on + // the read path (a SQLite double that skipped the 15-digit read) fails here, not later. + if (isDecimalColumn(col)) v = decimalWire(table, col, v) // `@dbColumnType:jsonb` open-JSON column (#98) reads back as a parsed kotlinx // JsonElement (or, for an object/map jsonb column, a raw JSON String). Convert // whatever it is to a plain JSON value (via its JSON text — JsonElement.toString() is @@ -760,6 +771,26 @@ object QueryScenarioRunner { return out } + private fun isDecimalColumn(col: Column<*>): Boolean = + col.columnType is org.jetbrains.exposed.sql.DecimalColumnType || + col.columnType is com.metaobjects.integration.kotlin.tables.PortableDecimalColumnType + + /** A decimal wire value: null, or plain notation with no trailing fraction zeros (zero is "0"). */ + private val DECIMAL_WIRE = Regex("^-?(0|[1-9][0-9]*)(\\.[0-9]*[1-9])?$") + + private fun decimalWire(table: Table, col: Column<*>, v: Any?): Any? { + val text = when (v) { + null -> null + is BigDecimal -> ReferenceDecimal.format(v) + else -> throw AssertionError( + "${table.tableName}.${col.name}: decimal arrived as a binary float or other non-decimal " + + "(${v::class.simpleName} $v); the read must yield an exact decimal") + } + if (text != null && !DECIMAL_WIRE.matches(text)) throw AssertionError( + "${table.tableName}.${col.name}: decimal '$text' is not canonical plain notation") + return text + } + /** Jackson mapper for parsing the `@dbColumnType:jsonb` open-JSON String back to a Map. */ private val JSON = ObjectMapper() diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioSqliteTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioSqliteTest.kt index a673c020f..2f2ad9ccf 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioSqliteTest.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/QueryScenarioSqliteTest.kt @@ -29,7 +29,7 @@ internal class QueryScenarioSqliteTest { @JvmStatic fun scenarios(): Stream { val corpus = ScenarioLoader.findCorpusRoot() - val reports = ScenarioLoader.loadQueries(corpus.resolve("queries")).filter { it.name.startsWith("report-") } + val reports = ScenarioLoader.loadQueries(corpus.resolve("queries")).filter { it.runsOn(ScenarioEngine.SQLITE) } Assumptions.assumeTrue(reports.isNotEmpty(), "No report-* scenarios under fixtures/persistence-conformance/queries") return reports.stream().map { Arguments.of(it.name, it) } } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/ScenarioEngine.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/ScenarioEngine.kt index 7538c6735..8d5f3a7c8 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/ScenarioEngine.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/ScenarioEngine.kt @@ -10,7 +10,7 @@ import com.metaobjects.integration.kotlin.Scenarios.QueryScenario * wire form. Spelling only: a wrong VALUE still * fails the comparison, and no `expect` is loosened. */ -enum class ScenarioEngine(private val id: String) { +enum class ScenarioEngine(internal val id: String) { POSTGRES("postgres") { override fun seed(scenario: QueryScenario): String? = scenario.seedData }, diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/ScenarioLoader.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/ScenarioLoader.kt index d85968f31..c952b078e 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/ScenarioLoader.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/ScenarioLoader.kt @@ -80,9 +80,23 @@ object ScenarioLoader { seedDataEngine = (root["seed-data-engine"] as? Map<*, *>) ?.entries?.associate { it.key.toString() to it.value.toString() }, queries = queries, + engines = parseEngines(file, root["engines"]), ) } + /** The engine names a scenario may list; `d1` is a TS-only engine, valid to name, never run here. */ + private val KNOWN_ENGINES = setOf("postgres", "mysql", "sqlite", "d1") + + /** `engines:` is exact when present; an unknown name or an empty list is a runner error. */ + private fun parseEngines(file: Path, raw: Any?): Set? { + if (raw == null) return null + require(raw is List<*> && raw.isNotEmpty()) { "$file: 'engines' must be a non-empty list" } + val names = raw.map { it.toString() }.toSet() + val unknown = names - KNOWN_ENGINES + require(unknown.isEmpty()) { "$file: unknown engine(s) $unknown; expected one of $KNOWN_ENGINES" } + return names + } + @Suppress("UNCHECKED_CAST") private fun parseQuerySpec(q: Map): QuerySpec { val sortsRaw = q["sort"] as? List> diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/Scenarios.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/Scenarios.kt index 3cdc2c0d2..513164bc5 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/Scenarios.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/Scenarios.kt @@ -32,5 +32,14 @@ object Scenarios { val seedData: String? = null, val seedDataEngine: Map? = null, // `seed-data-engine`: seed SQL for an engine whose spelling differs val queries: List, - ) + /** + * `engines:` (ADR-0019 Amendment 1) — EXACT when present: only the named engines run the file. + * Absent keeps the old rule: Postgres runs every scenario, the other engines run `report-*`. + */ + val engines: Set? = null, + ) { + fun runsOn(engine: ScenarioEngine): Boolean = + engines?.let { engine.id in it } + ?: (engine == ScenarioEngine.POSTGRES || name.startsWith("report-")) + } } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractAssertions.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractAssertions.kt index d2581f5a0..dfc64ab6f 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractAssertions.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractAssertions.kt @@ -153,7 +153,10 @@ object ApiContractAssertions { private fun structuralEquals(a: Any?, b: Any?): Boolean { if (a === b) return true if (a == null || b == null) return a == null && b == null - if (a is Number && b is Number) return a.toLong() == b.toLong() + // Exact: BigDecimal.compareTo, never a long or a double, so 0.1 does not equal 0 and a + // 20-digit decimal is compared digit for digit. An integer and a decimal of the same + // value are equal (1 and 1.0), which is what the JSON number grammar says. + if (a is Number && b is Number) return exactDecimal(a).compareTo(exactDecimal(b)) == 0 if (a is List<*> && b is List<*>) { if (a.size != b.size) return false return a.indices.all { structuralEquals(a[it], b[it]) } @@ -165,6 +168,13 @@ object ApiContractAssertions { return a == b } + private fun exactDecimal(n: Number): java.math.BigDecimal = when (n) { + is java.math.BigDecimal -> n + is java.math.BigInteger -> java.math.BigDecimal(n) + is Double, is Float -> java.math.BigDecimal(n.toString()) + else -> java.math.BigDecimal(n.toLong()) + } + private fun isEmpty(body: Any?): Boolean = when (body) { null -> true "" -> true diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractConformanceTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractConformanceTest.kt index 41ca01149..75fe6ea38 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractConformanceTest.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractConformanceTest.kt @@ -68,7 +68,7 @@ internal class ApiContractConformanceTest { } companion object { - private val MAPPER = ObjectMapper() + private val MAPPER = ObjectMapper().enable(com.fasterxml.jackson.databind.DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS) // Loaded once per test class. Each scenario re-applies via applySeed/truncate // so the per-scenario isolation contract holds. diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractScenarioLoader.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractScenarioLoader.kt index 2eb438918..b5ee279fd 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractScenarioLoader.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractScenarioLoader.kt @@ -58,9 +58,22 @@ object ApiContractScenarioLoader { sourcePath = file.toString(), truncate = truncate, requests = rawRequests.map { parseRequest(it) }, + engines = parseEngines(file, root["engines"]), ) } + private val KNOWN_ENGINES = setOf("postgres", "mysql", "sqlite", "d1") + + /** `engines:` is exact when present; a name the corpus does not know is a runner error. */ + private fun parseEngines(file: Path, raw: Any?): Set? { + if (raw == null) return null + require(raw is List<*>) { "$file: 'engines' must be a list" } + val names = raw.map { it.toString() }.toSet() + val unknown = names - KNOWN_ENGINES + require(unknown.isEmpty()) { "$file: unknown engine(s) $unknown; expected one of $KNOWN_ENGINES" } + return names + } + @Suppress("UNCHECKED_CAST") private fun parseRequest(r: Map): ApiRequest { val expect = r["expect"] as? Map diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractScenarios.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractScenarios.kt index 65a79d1c1..ec799731d 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractScenarios.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/ApiContractScenarios.kt @@ -26,5 +26,10 @@ object ApiContractScenarios { val sourcePath: String, val truncate: Boolean, // true when setup.truncate is set val requests: List, - ) + /** The top-level `engines:` key, matched EXACTLY; null = every engine the lane has. */ + val engines: Set? = null, + ) { + /** True when the scenario runs on [engine] (`postgres`, `mysql`, `sqlite`, `d1`). */ + fun runsOn(engine: String): Boolean = engines == null || engine.lowercase() in engines + } } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/TomcatHost.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/TomcatHost.kt index 4e2bf2748..0fc235144 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/TomcatHost.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/TomcatHost.kt @@ -112,7 +112,11 @@ class TomcatHost private constructor( fun parseBody(mapper: ObjectMapper, body: String?): Any? { if (body.isNullOrEmpty()) return null return try { - mapper.readValue(body, Any::class.java) + // Numbers are read exactly (BigDecimal), never through a double, so the corpus can + // compare a decimal wire value digit for digit (ADR-0019 Amendment 1, W1). + mapper.readerFor(Any::class.java) + .with(com.fasterxml.jackson.databind.DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS) + .readValue(body) } catch (notJson: com.fasterxml.jackson.core.JsonProcessingException) { body } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/decimal/DecimalGeneratedApiContractConformanceTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/decimal/DecimalGeneratedApiContractConformanceTest.kt new file mode 100644 index 000000000..7c712619f --- /dev/null +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/decimal/DecimalGeneratedApiContractConformanceTest.kt @@ -0,0 +1,114 @@ +package com.metaobjects.integration.kotlin.api.decimal + +import com.metaobjects.integration.kotlin.MySqlContainer +import com.metaobjects.integration.kotlin.PostgresContainer +import com.metaobjects.integration.kotlin.ScenarioEngine +import com.metaobjects.integration.kotlin.api.ApiContractAssertions +import com.metaobjects.integration.kotlin.api.ApiContractScenarioLoader +import com.metaobjects.integration.kotlin.api.ApiContractScenarios.ApiScenario +import com.metaobjects.integration.kotlin.api.report.generated.GeneratedReportControllerHarness +import com.metaobjects.integration.kotlin.api.report.generated.GeneratedReportControllerHarness.ReportDb +import org.junit.jupiter.api.AfterAll +import org.junit.jupiter.api.Assertions.assertEquals +import org.junit.jupiter.api.Assumptions +import org.junit.jupiter.api.Assertions.assertTrue +import org.junit.jupiter.api.BeforeAll +import org.junit.jupiter.api.DisplayName +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.assertDoesNotThrow +import org.junit.jupiter.api.io.TempDir +import org.junit.jupiter.params.ParameterizedTest +import org.junit.jupiter.params.provider.Arguments +import org.junit.jupiter.params.provider.MethodSource +import java.nio.file.Files +import java.nio.file.Path +import java.util.stream.Stream + +/** + * ADR-0019 Amendment 1 (W1 to W4): the Kotlin GENERATED-controller lane for the `decimal/` api-contract + * sub-corpus, on real Postgres, SQLite and MySQL. A decimal is a canonical string on the wire, a request + * takes a string or a number in the JSON-number grammar, a filter operand is compared exactly, and a value + * the column cannot hold is `400 validation`. Reuses the report lane's harness (same mechanics: the + * committed schema executed verbatim, the base tables seeded, the emitted controllers hosted unmodified). + * Generated lane only, as every port runs it. + */ +@DisplayName("API contract decimal (ADR-0019 Amendment 1) - GENERATED Kotlin Spring controllers over embedded Tomcat, real databases") +internal class DecimalGeneratedApiContractConformanceTest { + + @ParameterizedTest(name = "postgres {0}") + @MethodSource("scenarios") + fun scenarioOnPostgres(name: String, scenario: ApiScenario) = + run(scenario, ReportDb(ScenarioEngine.POSTGRES, PG.jdbcUrl, PG.username, PG.password)) + + @ParameterizedTest(name = "sqlite {0}") + @MethodSource("scenarios") + fun scenarioOnSqlite(name: String, scenario: ApiScenario, @TempDir dir: Path) = + run(scenario, ReportDb(ScenarioEngine.SQLITE, "jdbc:sqlite:${dir.resolve("decimal.db")}", "", "")) + + @ParameterizedTest(name = "mysql {0}") + @MethodSource("scenarios") + fun scenarioOnMySql(name: String, scenario: ApiScenario) = + run(scenario, ReportDb(ScenarioEngine.MYSQL, MYSQL.jdbcUrl, MYSQL.username, MYSQL.password)) + + /** A fresh schema (real tables and views) and base-table seed per scenario, then the requests over HTTP. */ + private fun run(scenario: ApiScenario, db: ReportDb) { + // `engines:` is exact: a scenario that names other engines is not this engine's. + Assumptions.assumeTrue(scenario.runsOn(db.engine.name), "${scenario.name} does not run on ${db.engine}") + assertDoesNotThrow { + HARNESS.reset(db) + for (req in scenario.requests) { + val res = HARNESS.exchange(req.method, req.path, req.body) + val parsed = HARNESS.parseBody(res.body) + ApiContractAssertions.assertResponse(scenario.name, req, res.status, parsed) + } + } + } + + @Test + fun `the corpus has its eight scenarios`() { + // A lane that silently ran fewer would still be green. + assertEquals(8, scenarios().count()) + } + + @Test + fun `the entity and the report both get a controller`() { + for (name in GeneratedReportControllerHarness.DECIMAL_SERVED) { + assertTrue("acme/ledger/${name}Controller.kt" in HARNESS.emittedFiles, "$name was not generated") + } + } + + companion object { + // One server per engine for the class; each scenario rebuilds its schema on it. Lazy, so a + // run that selects one engine's tests does not boot the others' containers. + private val PG by lazy { PostgresContainer() } + private val MYSQL by lazy { MySqlContainer() } + + private lateinit var HARNESS: GeneratedReportControllerHarness + + @BeforeAll + @JvmStatic + fun setUp() { + val corpus = ApiContractScenarioLoader.findCorpusRoot() + val genDir = Files.createTempDirectory("decimal-generated-kotlin-controller") + HARNESS = GeneratedReportControllerHarness( + corpus, genDir, lane = "decimal", entityPkg = "acme.ledger", + served = GeneratedReportControllerHarness.DECIMAL_SERVED, + ) + } + + @AfterAll + @JvmStatic + fun tearDown() { + if (::HARNESS.isInitialized) HARNESS.close() + runCatching { PG.close() } + runCatching { MYSQL.close() } + } + + @JvmStatic + fun scenarios(): Stream { + val corpus = ApiContractScenarioLoader.findCorpusRoot() + val scenarios = ApiContractScenarioLoader.loadScenarios(corpus.resolve("decimal/scenarios")) + return scenarios.stream().map { Arguments.of(it.name, it) } + } + } +} diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/generated/ApiContractGeneratedConformanceTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/generated/ApiContractGeneratedConformanceTest.kt index 427f0cad7..8a05a75f5 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/generated/ApiContractGeneratedConformanceTest.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/generated/ApiContractGeneratedConformanceTest.kt @@ -53,7 +53,7 @@ internal class ApiContractGeneratedConformanceTest { } companion object { - private val MAPPER = ObjectMapper() + private val MAPPER = ObjectMapper().enable(com.fasterxml.jackson.databind.DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS) @Suppress("UNCHECKED_CAST") private val SEED_ROWS: List> by lazy { diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/jsonb/JsonbApiContractConformanceTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/jsonb/JsonbApiContractConformanceTest.kt index bb6ba4975..2f1e260dc 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/jsonb/JsonbApiContractConformanceTest.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/jsonb/JsonbApiContractConformanceTest.kt @@ -70,7 +70,7 @@ internal class JsonbApiContractConformanceTest { } companion object { - private val MAPPER = ObjectMapper() + private val MAPPER = ObjectMapper().enable(com.fasterxml.jackson.databind.DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS) /** The `jsonb/` sub-corpus root (a sibling of the base single-entity corpus). */ private val CORPUS = ApiContractScenarioLoader.findCorpusRoot().resolve("jsonb") diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/jsonb/generated/JsonbGeneratedApiContractConformanceTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/jsonb/generated/JsonbGeneratedApiContractConformanceTest.kt index 82c0024d9..315c15446 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/jsonb/generated/JsonbGeneratedApiContractConformanceTest.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/jsonb/generated/JsonbGeneratedApiContractConformanceTest.kt @@ -47,7 +47,7 @@ internal class JsonbGeneratedApiContractConformanceTest { } companion object { - private val MAPPER = ObjectMapper() + private val MAPPER = ObjectMapper().enable(com.fasterxml.jackson.databind.DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS) /** The `jsonb/` sub-corpus root (sibling of the base single-entity corpus). */ private val CORPUS = ApiContractScenarioLoader.findCorpusRoot().resolve("jsonb") diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/m2m/M2mApiContractConformanceTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/m2m/M2mApiContractConformanceTest.kt index 8708d1870..885b3aa60 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/m2m/M2mApiContractConformanceTest.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/m2m/M2mApiContractConformanceTest.kt @@ -50,7 +50,7 @@ internal class M2mApiContractConformanceTest { } companion object { - private val MAPPER = ObjectMapper() + private val MAPPER = ObjectMapper().enable(com.fasterxml.jackson.databind.DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS) private val CORPUS = M2mScenarios.findM2mCorpus() private val SEED = M2mSeed.load(CORPUS) diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/projection/ProjectionGeneratedApiContractConformanceTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/projection/ProjectionGeneratedApiContractConformanceTest.kt index 377446003..10fa5b104 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/projection/ProjectionGeneratedApiContractConformanceTest.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/projection/ProjectionGeneratedApiContractConformanceTest.kt @@ -45,7 +45,7 @@ internal class ProjectionGeneratedApiContractConformanceTest { } companion object { - private val MAPPER = ObjectMapper() + private val MAPPER = ObjectMapper().enable(com.fasterxml.jackson.databind.DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS) /** * The seed is keyed by the base TABLE (`invoices`) and never by the view: the view diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/report/ReportGeneratedApiContractConformanceTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/report/ReportGeneratedApiContractConformanceTest.kt index 17f4331c7..d3dbddc52 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/report/ReportGeneratedApiContractConformanceTest.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/report/ReportGeneratedApiContractConformanceTest.kt @@ -1,22 +1,27 @@ package com.metaobjects.integration.kotlin.api.report -import com.fasterxml.jackson.databind.ObjectMapper +import com.metaobjects.integration.kotlin.MySqlContainer +import com.metaobjects.integration.kotlin.PostgresContainer +import com.metaobjects.integration.kotlin.ScenarioEngine import com.metaobjects.integration.kotlin.api.ApiContractAssertions import com.metaobjects.integration.kotlin.api.ApiContractScenarioLoader import com.metaobjects.integration.kotlin.api.ApiContractScenarios.ApiScenario import com.metaobjects.integration.kotlin.api.report.generated.GeneratedReportControllerHarness +import com.metaobjects.integration.kotlin.api.report.generated.GeneratedReportControllerHarness.ReportDb import org.junit.jupiter.api.AfterAll import org.junit.jupiter.api.Assertions.assertEquals +import org.junit.jupiter.api.Assumptions import org.junit.jupiter.api.Assertions.assertTrue import org.junit.jupiter.api.BeforeAll import org.junit.jupiter.api.DisplayName import org.junit.jupiter.api.Test import org.junit.jupiter.api.assertDoesNotThrow +import org.junit.jupiter.api.io.TempDir import org.junit.jupiter.params.ParameterizedTest import org.junit.jupiter.params.provider.Arguments import org.junit.jupiter.params.provider.MethodSource -import java.nio.charset.StandardCharsets import java.nio.file.Files +import java.nio.file.Path import java.util.stream.Stream /** @@ -29,14 +34,30 @@ import java.util.stream.Stream * under test is whether the port's GENERATOR emits a route for a served report. A * hand-rolled reference controller would answer every scenario by construction. */ -@DisplayName("API contract report (FR-044) — GENERATED Kotlin Spring controllers over embedded Tomcat") +@DisplayName("API contract report (FR-044) — GENERATED Kotlin Spring controllers over embedded Tomcat, real views") internal class ReportGeneratedApiContractConformanceTest { - @ParameterizedTest(name = "{0}") + @ParameterizedTest(name = "postgres {0}") @MethodSource("scenarios") - fun scenario(name: String, scenario: ApiScenario) { + fun scenarioOnPostgres(name: String, scenario: ApiScenario) = + run(scenario, ReportDb(ScenarioEngine.POSTGRES, PG.jdbcUrl, PG.username, PG.password)) + + @ParameterizedTest(name = "sqlite {0}") + @MethodSource("scenarios") + fun scenarioOnSqlite(name: String, scenario: ApiScenario, @TempDir dir: Path) = + run(scenario, ReportDb(ScenarioEngine.SQLITE, "jdbc:sqlite:${dir.resolve("report.db")}", "", "")) + + @ParameterizedTest(name = "mysql {0}") + @MethodSource("scenarios") + fun scenarioOnMySql(name: String, scenario: ApiScenario) = + run(scenario, ReportDb(ScenarioEngine.MYSQL, MYSQL.jdbcUrl, MYSQL.username, MYSQL.password)) + + /** A fresh schema (real tables and views) and base-table seed per scenario, then the requests over HTTP. */ + private fun run(scenario: ApiScenario, db: ReportDb) { + // `engines:` is exact: a scenario that names other engines is not this engine's. + Assumptions.assumeTrue(scenario.runsOn(db.engine.name), "${scenario.name} does not run on ${db.engine}") assertDoesNotThrow { - HARNESS.reset() // fresh H2 + stand-in tables + seed per scenario + HARNESS.reset(db) for (req in scenario.requests) { val res = HARNESS.exchange(req.method, req.path, req.body) val parsed = HARNESS.parseBody(res.body) @@ -46,9 +67,9 @@ internal class ReportGeneratedApiContractConformanceTest { } @Test - fun `the corpus has its twenty-five scenarios`() { + fun `the corpus has its twenty-seven scenarios`() { // A lane that silently ran fewer would still be green. - assertEquals(25, scenarios().count()) + assertEquals(27, scenarios().count()) } @Test @@ -65,22 +86,10 @@ internal class ReportGeneratedApiContractConformanceTest { } companion object { - private val MAPPER = ObjectMapper() - - /** - * The `reports` half of the seed: what the four views return for the seeded base - * tables. This lane's H2 tables stand in for the views (no view SQL is written in - * Kotlin), so they are seeded with the views' rows; every other top-level key - * (`invoices`, `products`, `sales`) is a base table this lane never creates. - */ - @Suppress("UNCHECKED_CAST") - private val SEED: Map>> by lazy { - val corpus = ApiContractScenarioLoader.findCorpusRoot() - val text = Files.readString(corpus.resolve("report/seed.json"), StandardCharsets.UTF_8) - val parsed = MAPPER.readValue(text, Map::class.java) as Map - (parsed["reports"] as? Map>>) - ?: error("report/seed.json: missing 'reports'") - } + // One server per engine for the class; each scenario rebuilds its schema on it. Lazy, so a + // run that selects one engine's tests does not boot the others' containers. + private val PG by lazy { PostgresContainer() } + private val MYSQL by lazy { MySqlContainer() } private lateinit var HARNESS: GeneratedReportControllerHarness @@ -89,13 +98,15 @@ internal class ReportGeneratedApiContractConformanceTest { fun setUp() { val corpus = ApiContractScenarioLoader.findCorpusRoot() val genDir = Files.createTempDirectory("fr044-generated-kotlin-report-controller") - HARNESS = GeneratedReportControllerHarness(corpus, genDir, SEED) + HARNESS = GeneratedReportControllerHarness(corpus, genDir) } @AfterAll @JvmStatic fun tearDown() { if (::HARNESS.isInitialized) HARNESS.close() + runCatching { PG.close() } + runCatching { MYSQL.close() } } @JvmStatic diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/report/generated/GeneratedReportControllerHarness.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/report/generated/GeneratedReportControllerHarness.kt index cb30702c3..da174ca39 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/report/generated/GeneratedReportControllerHarness.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/report/generated/GeneratedReportControllerHarness.kt @@ -9,24 +9,23 @@ import com.metaobjects.generator.kotlin.KotlinExposedTableGenerator import com.metaobjects.generator.kotlin.KotlinFilterAllowlistGenerator import com.metaobjects.generator.kotlin.KotlinNamesGenerator import com.metaobjects.generator.kotlin.KotlinSpringControllerGenerator +import com.metaobjects.integration.kotlin.ScenarioEngine +import com.metaobjects.integration.kotlin.ScenarioLoader import com.metaobjects.integration.kotlin.api.TomcatHost import com.metaobjects.loader.uri.URIHelper import com.metaobjects.metadata.ktx.loadUris import com.tschuchort.compiletesting.KotlinCompilation import com.tschuchort.compiletesting.SourceFile import org.jetbrains.exposed.sql.Database -import org.jetbrains.exposed.sql.SchemaUtils -import org.jetbrains.exposed.sql.Table -import org.jetbrains.exposed.sql.transactions.transaction import java.net.URI import java.nio.file.Files import java.nio.file.Path -import java.util.concurrent.atomic.AtomicInteger +import java.sql.DriverManager import kotlin.io.path.isRegularFile import kotlin.io.path.readText /** - * FR-044 Plan 3: host the GENERATED Kotlin Spring controllers of the nine served reports + * FR-044 Plan 3: host the GENERATED Kotlin Spring controllers of the ten served reports * of the `report/` corpus over real HTTP (an embedded Tomcat). Mirrors * [com.metaobjects.integration.kotlin.api.projection.generated.GeneratedProjectionControllerHarness]. * @@ -37,22 +36,24 @@ import kotlin.io.path.readText * 3. Compile every emitted `.kt` together. This is the only thing that proves the emitted * report controller COMPILES against its row, table and allowlist: the codegen-compile * gate excludes the framework-bound route tier in every port by design. - * 4. Per scenario: a fresh in-memory H2 (PostgreSQL mode), `SchemaUtils.create(...)` on the - * four generated report table objects, then the `reports` half of `seed.json`. - * 5. Serve the four controllers from one embedded Tomcat ([TomcatHost]). + * 4. Per scenario ([reset]): execute the committed TypeScript-produced `report/schema..sql` + * verbatim (the base tables AND the real views) on a real database, then seed the base + * tables of `seed.json` (every top-level key but `reports`), parents first. + * 5. Serve the controllers from one embedded Tomcat ([TomcatHost]). * - * No view SQL is written here (ADR-0015: TypeScript owns it). H2 creates each generated - * `Table` as a plain table that STANDS IN for the view, and the seed's `reports` half is - * what the four views return for the seeded base tables. The base-table keys (`invoices`, - * `products`, `sales`) are the full-stack lanes' input; this seam lane does not read them. - * What this lane proves is the generated route over the generated binding, not the lowering. + * No view SQL is written here (ADR-0015: TypeScript owns it). The generated `Table` + * objects read the REAL views, so this is a full-stack lane on Postgres, SQLite and MySQL, + * like the TypeScript and C# lanes. The seed's `reports` half (what the views return) is + * not used: the views derive it from the base tables. */ @OptIn(org.jetbrains.kotlin.compiler.plugin.ExperimentalCompilerApi::class) class GeneratedReportControllerHarness( - corpusRoot: Path, + private val corpusRoot: Path, genDir: Path, - /** `seed.json`'s `reports`: report name to the rows its view returns. */ - private val reportRows: Map>>, + /** The sub-corpus directory under the api-contract corpus: `report`, or `decimal` (same mechanics). */ + private val lane: String = "report", + private val entityPkg: String = ENTITY_PKG, + private val served: List = SERVED, ) : AutoCloseable { private val mapper: ObjectMapper = ObjectMapper() @@ -62,17 +63,13 @@ class GeneratedReportControllerHarness( .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS) private val controllerClasses: List> - private val tables: Map - private val dbSeq = AtomicInteger(0) private var host: TomcatHost? = null - /** ADR-0058 — the project's naming.columns rule, read from this corpus's config. */ - private val columnsRule: String /** Every generated file, by path relative to the output root. */ val emittedFiles: Set init { - val metaJson = corpusRoot.resolve("report/meta.json") + val metaJson = corpusRoot.resolve("$lane/meta.json") val uri: URI = URIHelper.toURI("model:file:" + metaJson.toAbsolutePath().toString().replace('\\', '/')) val loader = loadUris("api-contract-report-generated", listOf(uri)) // ADR-0058 — this corpus's committed .metaobjects/config.json (at corpusRoot, @@ -81,7 +78,6 @@ class GeneratedReportControllerHarness( // depend on this. Read with the SAME NeutralConfig reader every port uses. loader.root.physicalNaming = com.metaobjects.config.SourceResolver.resolvePhysicalNaming(corpusRoot) - columnsRule = loader.root.physicalNaming.columns() val srcDir = genDir.resolve("src") Files.createDirectories(srcDir) @@ -103,8 +99,8 @@ class GeneratedReportControllerHarness( // A served report's controller must have been emitted at all; otherwise this fails // downstream as a ClassNotFoundException with no hint that the emit gate was the cause. - for (report in SERVED) { - check("acme/sales/${report}Controller.kt" in emittedFiles) { + for (report in served) { + check("${entityPkg.replace('.', '/')}/${report}Controller.kt" in emittedFiles) { "no controller was generated for the $report report; emitted=$emittedFiles" } } @@ -118,42 +114,82 @@ class GeneratedReportControllerHarness( "generated Kotlin failed to compile:\n${result.messages}" } - controllerClasses = SERVED.map { result.classLoader.loadClass("$ENTITY_PKG.${it}Controller") } - tables = SERVED.associateWith { - result.classLoader.loadClass("$ENTITY_PKG.${it}Table").getDeclaredField("INSTANCE").get(null) as Table - } + controllerClasses = served.map { result.classLoader.loadClass("$entityPkg.${it}Controller") } } - /** Rebuild a fresh in-memory H2, the nine stand-in tables, the seed, and Tomcat. */ - fun reset() { - val dbName = "report_invoice_${dbSeq.incrementAndGet()}" - val db = Database.connect("jdbc:h2:mem:$dbName;DB_CLOSE_DELAY=-1;MODE=PostgreSQL", driver = "org.h2.Driver") - transaction(db) { - for ((report, table) in tables) { - SchemaUtils.create(table) - val rows = reportRows[report] ?: error("report/seed.json: no 'reports.$report' rows") - for (row in rows) { - val fields = row.keys.toList() - val colList = fields.joinToString(", ") { identity(column(table, it)) } - val values = fields.joinToString(", ") { literal(row[it]) } - exec("INSERT INTO ${identity(table)} ($colList) VALUES ($values)") + /** + * Rebuild the schema (real tables and views), the base-table seed, and Tomcat on [target]. + * The target must be empty or hold only a previous run's schema (Postgres and MySQL are + * cleared here; a SQLite target is a fresh file per scenario). + */ + fun reset(target: ReportDb) { + val engine = target.engine + val url = engine.jdbcUrl(target.baseUrl) + val schema = ScenarioLoader.readCanonicalSchema(corpusRoot, "$lane/schema.${engine.name.lowercase()}.sql") + DriverManager.getConnection(url, target.username, target.password).use { c -> + c.createStatement().use { st -> + when (engine) { + ScenarioEngine.POSTGRES -> { + st.execute("DROP SCHEMA public CASCADE") + st.execute("CREATE SCHEMA public") + st.execute(schema) + } + ScenarioEngine.MYSQL -> { + dropMysql(st, schema) + ScenarioEngine.splitStatements(schema).forEach { st.execute(it) } + } + ScenarioEngine.SQLITE -> ScenarioEngine.splitStatements(schema).forEach { st.execute(it) } } + seedBaseTables(st, engine) } } + // The generated controllers open their transactions on Exposed's default database. + Database.connect(url, user = target.username, password = target.password) host?.close() - host = TomcatHost.start(mapper, *controllerClasses.map { it.getDeclaredConstructor().newInstance() }.toTypedArray()) + host = TomcatHost.start(mapper, *controllerClasses.map(::instantiate).toTypedArray()) } /** - * The generated Exposed column for a derived field name. Looked up by the PHYSICAL name - * the generator chose — this corpus's naming.columns rule (ADR-0058), read from the - * SAME loaded root the generators resolved it from — so the seed follows whatever the - * generator emitted rather than a spelling hardcoded here that could drift from it. + * A read-only report controller takes no arguments; a writable entity's takes the ObjectMapper + * (the per-field PATCH bind) and a jakarta Validator (create and present-value validation). */ - private fun column(table: Table, field: String) = - table.columns.firstOrNull { it.name == com.metaobjects.database.ColumnNaming.apply(field, columnsRule) } - ?: error("no column for derived field '$field' on ${table.tableName}; columns=${table.columns.map { it.name }}") + private fun instantiate(c: Class<*>): Any { + val ctor = c.declaredConstructors.minByOrNull { it.parameterCount }!! + return if (ctor.parameterCount == 0) ctor.newInstance() + else ctor.newInstance(mapper, jakarta.validation.Validation.buildDefaultValidatorFactory().validator) + } + + /** Every top-level key of `seed.json` but `reports` is a base table, parents first, in file order. */ + @Suppress("UNCHECKED_CAST") + private fun seedBaseTables(st: java.sql.Statement, engine: ScenarioEngine) { + val seed = ObjectMapper().readValue( + Files.readString(corpusRoot.resolve("$lane/seed.json")), Map::class.java, + ) as Map + val q = if (engine == ScenarioEngine.MYSQL) "`" else "\"" + val tables = seed.filterKeys { it != SEED_REPORTS_KEY } + check(tables.isNotEmpty()) { "$lane/seed.json: no base-table rows to seed" } + for ((table, rows) in tables) { + for (row in rows as List>) { + val cols = row.keys.joinToString(", ") { "$q$it$q" } + val vals = row.values.joinToString(", ") { literal(it) } + st.execute("INSERT INTO $q$table$q ($cols) VALUES ($vals)") + } + // The seed writes explicit ids; move the identity past them (MySQL and SQLite continue from MAX). + if (engine == ScenarioEngine.POSTGRES && rows.any { it.containsKey("id") }) { + st.execute("SELECT setval(pg_get_serial_sequence('$table', 'id'), COALESCE((SELECT MAX(id) FROM \"$table\"), 1))") + } + } + } + + /** MySQL keeps one database across scenarios: drop the artifact's views, then its tables. */ + private fun dropMysql(st: java.sql.Statement, schema: String) { + val created = ScenarioEngine.splitStatements(schema).mapNotNull { MYSQL_CREATE.find(it)?.groupValues } + created.filter { it[1] == "VIEW" }.forEach { st.execute("DROP VIEW IF EXISTS `${it[2]}`") } + st.execute("SET FOREIGN_KEY_CHECKS = 0") + created.filter { it[1] == "TABLE" }.forEach { st.execute("DROP TABLE IF EXISTS `${it[2]}`") } + st.execute("SET FOREIGN_KEY_CHECKS = 1") + } fun exchange(method: String, path: String, jsonBody: Any?): Response { val server = host ?: error("reset() must be called before exchange(...)") @@ -164,28 +200,38 @@ class GeneratedReportControllerHarness( fun parseBody(body: String?): Any? = TomcatHost.parseBody(mapper, body) override fun close() { - host?.close() // H2 in-mem is reclaimed at JVM exit (DB_CLOSE_DELAY=-1). + host?.close() } data class Response(val status: Int, val body: String) + /** A database the lane runs on: [baseUrl] is the engine's JDBC URL before [ScenarioEngine.jdbcUrl] adds its session policy. */ + data class ReportDb(val engine: ScenarioEngine, val baseUrl: String, val username: String, val password: String) + companion object { const val ENTITY_PKG = "acme.sales" + /** The one top-level key of seed.json that is not a base table. */ + private const val SEED_REPORTS_KEY = "reports" + + private val MYSQL_CREATE = Regex("^CREATE (VIEW|TABLE) `([^`]+)`") + /** * The reports of the corpus that declare a view. `InvoiceDays` declares none. * `ProductRevenue` declares `@spine` and a measure with `@default`; `InvoiceCalendar` lists * one time dimension at two grains; `SalesByCategory` reaches a dimension through two references. */ + /** The `decimal/` sub-corpus: one writable entity and the report read from it. */ + val DECIMAL_SERVED = listOf("Entry", "EntryTotals") + val SERVED = listOf( "InvoiceStatusTotals", "InvoicesByMonth", "InvoiceTotals", "InvoiceCalendar", "InvoiceDisputes", - "ProductRevenue", "SalesByCategory", "SalesBySku", "ProductRefunds", + "ProductRevenue", "SalesByCategory", "SalesBySku", "ProductRefunds", "TicketsByTags", ) /** - * A seed value as a SQL literal. A date (`"2026-04-01"`) and the ratio (`"0.4"`, a - * string in the seed so no float sits between it and the column's decimal) are quoted - * strings that H2 converts to the column's own type on insert. + * A seed value as a SQL literal. A date (`"2026-04-30"`) is a quoted string the engine + * converts to the column's own type on insert. */ private fun literal(v: Any?): String = when (v) { null -> "NULL" diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/tph/TphFullSuiteCompilesTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/tph/TphFullSuiteCompilesTest.kt index efde53140..12c45efb0 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/tph/TphFullSuiteCompilesTest.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/tph/TphFullSuiteCompilesTest.kt @@ -140,7 +140,7 @@ class TphFullSuiteCompilesTest { val emitted = Files.walk(outDir).filter { it.isRegularFile() }.sorted().toList() val authTable = emitted.first { it.fileName.toString() == "AuthTable.kt" }.readText() assertTrue("integer(BridgeAuthNames.QUANTITY_COLUMN).nullable()" in authTable, authTable) - assertTrue("decimal(CopayAuthNames.COPAY_AMOUNT_COLUMN, 10, 2).nullable()" in authTable, authTable) + assertTrue("portableDecimal(CopayAuthNames.COPAY_AMOUNT_COLUMN, 10, 2).nullable()" in authTable, authTable) assertTrue("varchar(PriorAuthAuthNames.PRIOR_AUTH_NUMBER_COLUMN, 80).nullable()" in authTable, authTable) assertTrue("enumerationByName(CopayAuthNames.TIER_COLUMN, 64, AuthTier::class).nullable()" in authTable, authTable) diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/tph/TphGeneratedApiContractConformanceTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/tph/TphGeneratedApiContractConformanceTest.kt index b4e50c5ff..f0a516c33 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/tph/TphGeneratedApiContractConformanceTest.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/tph/TphGeneratedApiContractConformanceTest.kt @@ -83,7 +83,7 @@ internal class TphGeneratedApiContractConformanceTest { } companion object { - private val MAPPER = ObjectMapper() + private val MAPPER = ObjectMapper().enable(com.fasterxml.jackson.databind.DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS) /** The single TPH table's seed rows (one per subtype), keyed by table name `auths`. */ @Suppress("UNCHECKED_CAST") diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/writethrough/WriteThroughGeneratedApiContractConformanceTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/writethrough/WriteThroughGeneratedApiContractConformanceTest.kt index 7f45d9c43..fd169eb03 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/writethrough/WriteThroughGeneratedApiContractConformanceTest.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/api/writethrough/WriteThroughGeneratedApiContractConformanceTest.kt @@ -43,7 +43,7 @@ internal class WriteThroughGeneratedApiContractConformanceTest { } companion object { - private val MAPPER = ObjectMapper() + private val MAPPER = ObjectMapper().enable(com.fasterxml.jackson.databind.DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS) @Suppress("UNCHECKED_CAST") private val SEED: Pair>, List>> by lazy { diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/spring/ClasspathNamingDiscoveryTest.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/spring/ClasspathNamingDiscoveryTest.kt new file mode 100644 index 000000000..c98cf5413 --- /dev/null +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/spring/ClasspathNamingDiscoveryTest.kt @@ -0,0 +1,84 @@ +package com.metaobjects.integration.kotlin.spring + +import com.metaobjects.generator.kotlin.KotlinExposedTableGenerator +import com.metaobjects.loader.MetaDataLoader +import com.metaobjects.spring.MetaDataLoaderConfiguration +import java.net.URLClassLoader +import java.nio.file.Files +import java.nio.file.Path +import kotlin.io.path.readText +import kotlin.test.AfterTest +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertTrue + +/** + * B1 / ADR-0058 from Kotlin. A Kotlin Spring application starts from the same + * `MetaDataLoaderConfiguration` bean as a Java one, and with no property set that bean finds the + * project's `naming` at `classpath:.metaobjects/config.json`. The Exposed table generated from + * the loader it returns must then spell the LITERAL column name the project asked for, not the + * snake_case default a loader with no config would derive. + */ +class ClasspathNamingDiscoveryTest { + + private val model = """{ + "metadata.root": { "package": "acme::naming", "children": [ + { "object.entity": { "name": "Booking", "children": [ + { "field.long": { "name": "id" } }, + { "field.string": { "name": "createdBy", "@maxLength": 40 } }, + { "source.rdb": { "@table": "bookings" } }, + { "identity.primary": { "@fields": ["id"], "@generation": "increment" } } + ] } } + ] } + }""".trimIndent() + + private val originalLoader: ClassLoader = Thread.currentThread().contextClassLoader + + @AfterTest + fun restoreClassLoader() { + Thread.currentThread().contextClassLoader = originalLoader + } + + private fun projectRoot(config: String?): Path { + val root = Files.createTempDirectory("kt-naming-") + if (config != null) { + Files.createDirectories(root.resolve(".metaobjects")) + Files.writeString(root.resolve(".metaobjects/config.json"), config) + } + return root + } + + /** The loader the Spring configuration bean builds with [root] on the thread's classpath. */ + private fun springLoader(root: Path): MetaDataLoader { + Thread.currentThread().contextClassLoader = URLClassLoader(arrayOf(root.toUri().toURL()), originalLoader) + val metadata = Files.createTempDirectory("kt-naming-meta-").resolve("meta.booking.json") + Files.writeString(metadata, model) + val config = MetaDataLoaderConfiguration() + config.metadataSources = listOf(metadata.toUri().toString()) + return config.applicationMetaDataLoader() + } + + private fun tableSource(loader: MetaDataLoader): String { + val out = Files.createTempDirectory("kt-naming-out-") + KotlinExposedTableGenerator() + .apply { setArgs(mapOf("outputDir" to out.toString())) } + .execute(loader) + return out.resolve("acme/naming/BookingTable.kt").readText() + } + + @Test + fun `an unset property reads naming from the classpath config and the table spells the literal column`() { + val loader = springLoader(projectRoot("""{"schema_version":1,"naming":{"columns":"literal"}}""")) + assertEquals("literal", loader.root.physicalNaming.columns()) + val src = tableSource(loader) + assertTrue("\"createdBy\"" in src, "the literal column name createdBy is expected in:\n$src") + assertTrue("created_by" !in src, "no snake_case column is expected in:\n$src") + } + + @Test + fun `with no classpath config the same model gets the snake_case default`() { + val loader = springLoader(projectRoot(null)) + assertEquals("snake_case", loader.root.physicalNaming.columns()) + assertTrue("\"created_by\"" in tableSource(loader)) + } +} diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/AllTypesBreakdownView.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/AllTypesBreakdownView.kt index 065f74781..663f27666 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/AllTypesBreakdownView.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/AllTypesBreakdownView.kt @@ -25,7 +25,7 @@ object AllTypesBreakdownView : Table("v_all_types_breakdown") { val uuidKey = uuid("uuidKey") val allRows = long("allRows") val totalInts = long("totalInts").nullable() - val avgDec = decimal("avgDec", 38, 18).nullable() + val avgDec = portableDecimal("avgDec", 38, 18).nullable() val maxMoney = long("maxMoney").nullable() val minLong = long("minLong").nullable() val highRows = long("highRows") diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/AllTypesTable.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/AllTypesTable.kt index 763935979..6533b79c4 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/AllTypesTable.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/AllTypesTable.kt @@ -54,7 +54,7 @@ object AllTypesTable : Table("all_types") { val lVal = long("lVal") val dVal = double("dVal") val fVal = float("fVal") - val decVal = decimal("decVal", 18, 6) + val decVal = portableDecimal("decVal", 18, 6) val bVal = bool("bVal") val dateVal = date("dateVal") // Millisecond-preserving TIME (see PreciseLocalTimeColumnType) — stock `time` truncates sub-seconds. diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/AuthTable.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/AuthTable.kt index b56bf59a2..6c500413e 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/AuthTable.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/AuthTable.kt @@ -26,7 +26,7 @@ object AuthTable : Table("auths") { val type = varchar("type", 64) val reference = varchar("reference", 80) val quantity = integer("quantity").nullable() - val copayAmount = decimal("copayAmount", 10, 2).nullable() + val copayAmount = portableDecimal("copayAmount", 10, 2).nullable() val approver = varchar("approver", 80).nullable() override val primaryKey = PrimaryKey(id) diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/EntryTable.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/EntryTable.kt new file mode 100644 index 000000000..979759c7e --- /dev/null +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/EntryTable.kt @@ -0,0 +1,19 @@ +package com.metaobjects.integration.kotlin.tables + +import org.jetbrains.exposed.sql.Table + +/** + * Hand-written reference Exposed Table mirroring `Entry` from + * `fixtures/persistence-conformance/canonical/meta.fitness.json`: the decimal subject of the + * `decimal-exact` scenarios (ADR-0019 Amendment 1). `amount` is `decimal(28,10)` and nullable; + * its column type is the reference twin of the generated portable decimal, so SQLite reads a + * double at 15 significant digits. + */ +internal object EntryTable : Table("entries") { + val id = long("id").autoIncrement() + val bucket = varchar("bucket", 8) + val amount = portableDecimal("amount", 28, 10).nullable() + val flagged = bool("flagged") + + override val primaryKey = PrimaryKey(id) +} diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/EntryTotalsView.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/EntryTotalsView.kt new file mode 100644 index 000000000..3f0849fdd --- /dev/null +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/EntryTotalsView.kt @@ -0,0 +1,21 @@ +package com.metaobjects.integration.kotlin.tables + +import org.jetbrains.exposed.sql.Table + +/** + * Hand-written reference Exposed Table mapping the `EntryTotals` report (ADR-0019 Amendment 1) + * from `fixtures/persistence-conformance/canonical/meta.fitness.json`, backed by the view + * `v_entry_totals`. Every decimal measure (a sum, an average, a maximum, a ratio) is an exact + * decimal on Postgres and MySQL and a double on SQLite, read at 15 significant digits. A count is + * a `long`. Nullability follows `report-shapes.json`. + */ +internal object EntryTotalsView : Table("v_entry_totals") { + val bucket = varchar("bucket", 8) + val entries = long("entries") + val total = portableDecimal("total", 38, 18).nullable() + val totalOrZero = portableDecimal("totalOrZero", 38, 18) + val mean = portableDecimal("mean", 38, 18).nullable() + val top = portableDecimal("top", 38, 18).nullable() + val flaggedShare = portableDecimal("flaggedShare", 38, 18).nullable() + val perFlagged = portableDecimal("perFlagged", 38, 18).nullable() +} diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/FitnessTotalsFilledView.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/FitnessTotalsFilledView.kt index c1d24c833..5a7c1d2d3 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/FitnessTotalsFilledView.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/FitnessTotalsFilledView.kt @@ -24,5 +24,5 @@ import org.jetbrains.exposed.sql.Table object FitnessTotalsFilledView : Table("v_fitness_totals_filled") { val weeks = long("weeks") val totalMinutesOrZero = long("totalMinutesOrZero") - val longShareOrZero = decimal("longShareOrZero", 38, 18) + val longShareOrZero = portableDecimal("longShareOrZero", 38, 18) } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/FitnessTotalsView.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/FitnessTotalsView.kt index 4fb81819f..dc3bb3474 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/FitnessTotalsView.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/FitnessTotalsView.kt @@ -20,5 +20,5 @@ import org.jetbrains.exposed.sql.Table object FitnessTotalsView : Table("v_fitness_totals") { val weeks = long("weeks") val totalMinutes = long("totalMinutes").nullable() - val longShare = decimal("longShare", 38, 18).nullable() + val longShare = portableDecimal("longShare", 38, 18).nullable() } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/GadgetEventsByLabelView.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/GadgetEventsByLabelView.kt new file mode 100644 index 000000000..f0ef89db6 --- /dev/null +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/GadgetEventsByLabelView.kt @@ -0,0 +1,15 @@ +package com.metaobjects.integration.kotlin.tables + +import org.jetbrains.exposed.sql.Table + +/** + * Hand-written reference Exposed Table mapping the `GadgetEventsByLabel` report from + * `fixtures/persistence-conformance/canonical/meta.fitness.json`, backed by `v_gadget_events_by_label`. + * `gadgetLabel` is an INTEGER: the dimension reaches `Labeled.label` through `@via`, and the entity + * that `@via` reaches (`Gadget`) redeclares it as an int in column `gadget_code`, so the report groups + * by that column and not by the base's string column. + */ +object GadgetEventsByLabelView : Table("v_gadget_events_by_label") { + val gadgetLabel = integer("gadgetLabel") + val gadgetEvents = long("gadgetEvents") +} diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/MeasurementStatsFilledView.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/MeasurementStatsFilledView.kt index c54ae0e16..017a2e1cb 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/MeasurementStatsFilledView.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/MeasurementStatsFilledView.kt @@ -17,6 +17,6 @@ import org.jetbrains.exposed.sql.Table */ object MeasurementStatsFilledView : Table("v_measurement_stats_filled") { val readings = long("readings") - val avgPreciseKgOrMinusOne = decimal("avgPreciseKgOrMinusOne", 38, 18) - val maxPreciseKgOrZero = decimal("maxPreciseKgOrZero", 38, 18) + val avgPreciseKgOrMinusOne = portableDecimal("avgPreciseKgOrMinusOne", 38, 18) + val maxPreciseKgOrZero = portableDecimal("maxPreciseKgOrZero", 38, 18) } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/MeasurementStatsView.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/MeasurementStatsView.kt index ac0cecab9..f23394da3 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/MeasurementStatsView.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/MeasurementStatsView.kt @@ -21,8 +21,8 @@ object MeasurementStatsView : Table("v_measurement_stats") { val avgMassKg = double("avgMassKg").nullable() val maxTempC = float("maxTempC").nullable() val minTempC = float("minTempC").nullable() - val totalPreciseKg = decimal("totalPreciseKg", 38, 18).nullable() - val avgPreciseKg = decimal("avgPreciseKg", 38, 18).nullable() - val maxPreciseKg = decimal("maxPreciseKg", 38, 18).nullable() + val totalPreciseKg = portableDecimal("totalPreciseKg", 38, 18).nullable() + val avgPreciseKg = portableDecimal("avgPreciseKg", 38, 18).nullable() + val maxPreciseKg = portableDecimal("maxPreciseKg", 38, 18).nullable() val heavyReadings = long("heavyReadings") } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/MeasurementTable.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/MeasurementTable.kt index dd44ede69..15775f4ba 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/MeasurementTable.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/MeasurementTable.kt @@ -19,7 +19,7 @@ object MeasurementTable : Table("measurements") { val id = long("id").autoIncrement() val tempC = float("tempC") val massKg = double("massKg") - val preciseKg = decimal("preciseKg", 9, 4) + val preciseKg = portableDecimal("preciseKg", 9, 4) override val primaryKey = PrimaryKey(id) } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/PortableDecimalColumnType.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/PortableDecimalColumnType.kt new file mode 100644 index 000000000..c97e71ad2 --- /dev/null +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/PortableDecimalColumnType.kt @@ -0,0 +1,31 @@ +package com.metaobjects.integration.kotlin.tables + +import org.jetbrains.exposed.sql.ColumnType +import org.jetbrains.exposed.sql.DecimalColumnType +import org.jetbrains.exposed.sql.Column +import org.jetbrains.exposed.sql.Table +import org.jetbrains.exposed.sql.vendors.currentDialect +import java.math.BigDecimal + +/** + * The reference twin of the `MetaPortableDecimalColumnType` that KotlinExposedTableGenerator emits + * (ADR-0019 Amendment 1): the exact decimal on Postgres and MySQL, and on SQLite, which has no + * decimal and keeps a REAL, a number bound as a double (W4) and read at 15 significant digits, + * so `0.1 + 0.2` reads `0.3`. The reference tables use it so the persistence lane exercises the + * same read and write path a generated table has. + */ +class PortableDecimalColumnType(precision: Int, scale: Int) : ColumnType() { + private val delegate = DecimalColumnType(precision, scale) + override fun sqlType(): String = delegate.sqlType() + override fun valueFromDB(value: Any): BigDecimal = + if (isSqlite() && value is Number && value !is BigDecimal) ReferenceDecimal.fromDouble15(value.toDouble()) + else delegate.valueFromDB(value) + override fun notNullValueToDB(value: BigDecimal): Any = + if (isSqlite()) value.toDouble() else delegate.notNullValueToDB(value) + override fun nonNullValueToString(value: BigDecimal): String = delegate.nonNullValueToString(value) +} + +private fun isSqlite(): Boolean = currentDialect.name.equals("sqlite", ignoreCase = true) + +fun Table.portableDecimal(name: String, precision: Int, scale: Int): Column = + registerColumn(name, PortableDecimalColumnType(precision, scale)) diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramMinutesView.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramMinutesView.kt index 012e43c5c..e198086c7 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramMinutesView.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramMinutesView.kt @@ -33,8 +33,8 @@ object ProgramMinutesView : Table("v_program_minutes") { val labels = long("labels") val slots = long("slots") val totalMinutes = long("totalMinutes").nullable() - val avgMinutes = decimal("avgMinutes", 38, 18).nullable() + val avgMinutes = portableDecimal("avgMinutes", 38, 18).nullable() val minMinutes = integer("minMinutes").nullable() val maxMinutes = integer("maxMinutes").nullable() - val longShare = decimal("longShare", 38, 18).nullable() + val longShare = portableDecimal("longShare", 38, 18).nullable() } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramRosterView.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramRosterView.kt index 47f75669a..d1509f577 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramRosterView.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramRosterView.kt @@ -31,6 +31,6 @@ object ProgramRosterView : Table("v_program_roster") { val weeks = long("weeks") val totalMinutes = long("totalMinutes").nullable() val totalMinutesOrZero = long("totalMinutesOrZero") - val longShare = decimal("longShare", 38, 18).nullable() - val longShareOrZero = decimal("longShareOrZero", 38, 18) + val longShare = portableDecimal("longShare", 38, 18).nullable() + val longShareOrZero = portableDecimal("longShareOrZero", 38, 18) } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramStatView.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramStatView.kt index 046b91c6b..c45a43745 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramStatView.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramStatView.kt @@ -36,7 +36,7 @@ object ProgramStatView : Table("v_program_stat") { // never call SchemaUtils.create), so they don't emit DDL — Exposed just reads the column // value back as a BigDecimal, which Normalization renders with the no-trailing-zeros rule. val totalMinutes = long("totalMinutes").nullable() - val avgMinutes = decimal("avgMinutes", 38, 18).nullable() + val avgMinutes = portableDecimal("avgMinutes", 38, 18).nullable() val minMinutes = integer("minMinutes").nullable() val maxMinutes = integer("maxMinutes").nullable() diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramsEmptyScopeView.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramsEmptyScopeView.kt index 1ff8b1012..a59af2dba 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramsEmptyScopeView.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramsEmptyScopeView.kt @@ -17,5 +17,5 @@ import org.jetbrains.exposed.sql.Table */ object ProgramsEmptyScopeView : Table("v_programs_empty_scope") { val programs = long("programs") - val avgPriceOrMinusOne = decimal("avgPriceOrMinusOne", 38, 18) + val avgPriceOrMinusOne = portableDecimal("avgPriceOrMinusOne", 38, 18) } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramsOverTimeView.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramsOverTimeView.kt index 476d2d09c..35e13ff8a 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramsOverTimeView.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ProgramsOverTimeView.kt @@ -21,7 +21,7 @@ object ProgramsOverTimeView : Table("v_programs_over_time") { val createdAtQuarter = date("createdAtQuarter") val createdAtYear = date("createdAtYear") val programs = long("programs") - val avgPrice = decimal("avgPrice", 38, 18).nullable() + val avgPrice = portableDecimal("avgPrice", 38, 18).nullable() val cheapest = long("cheapest").nullable() val priciest = long("priciest").nullable() } diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ReferenceDecimal.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ReferenceDecimal.kt new file mode 100644 index 000000000..864ec7472 --- /dev/null +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ReferenceDecimal.kt @@ -0,0 +1,28 @@ +package com.metaobjects.integration.kotlin.tables + +import java.math.BigDecimal +import java.math.MathContext +import java.math.RoundingMode + +/** + * The decimal wire rule (ADR-0019 Amendment 1) as the hand-written reference tables and the runner + * need it: the canonical string, the strict reader, the 15-digit SQLite read. The generated code has + * its own copy (`MetaDecimalWire`, written beside it); this is the test harness's, so the harness + * depends on no MetaObjects runtime either. + */ +object ReferenceDecimal { + private val JSON_NUMBER = Regex("-?(0|[1-9][0-9]*)(\\.[0-9]+)?([eE][+-]?[0-9]+)?") + + fun format(value: BigDecimal): String = if (value.signum() == 0) "0" else value.stripTrailingZeros().toPlainString() + + fun parse(text: String): BigDecimal { + require(JSON_NUMBER.matches(text)) { "not a decimal: $text" } + return BigDecimal(text) + } + + fun fromDouble15(value: Double): BigDecimal { + if (value == 0.0) return BigDecimal.ZERO + val rounded = BigDecimal(value).round(MathContext(15, RoundingMode.HALF_EVEN)) + return if (rounded.signum() == 0) BigDecimal.ZERO else rounded.stripTrailingZeros() + } +} diff --git a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ReportViews.kt b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ReportViews.kt index c88c220c3..862735881 100644 --- a/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ReportViews.kt +++ b/server/java/integration-tests-kotlin/src/test/kotlin/com/metaobjects/integration/kotlin/tables/ReportViews.kt @@ -42,5 +42,8 @@ internal object ReportViews { "ProgramSessions" to ProgramSessionsView, "NodesByParent" to NodesByParentView, "AuthsByKind" to AuthsByKindView, + // ADR-0019 Amendment 1: decimal measures; and a @via dimension whose target redeclares the field. + "EntryTotals" to EntryTotalsView, + "GadgetEventsByLabel" to GadgetEventsByLabelView, ) } diff --git a/server/java/integration-tests/pom.xml b/server/java/integration-tests/pom.xml index 50533a328..ca26cb09a 100644 --- a/server/java/integration-tests/pom.xml +++ b/server/java/integration-tests/pom.xml @@ -140,6 +140,15 @@ test + + + org.xerial + sqlite-jdbc + 3.46.1.3 + test + + org.junit.jupiter diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/LiteralNamingDiscoveryTest.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/LiteralNamingDiscoveryTest.java new file mode 100644 index 000000000..b20dc8192 --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/LiteralNamingDiscoveryTest.java @@ -0,0 +1,126 @@ +package com.metaobjects.integration; + +import com.metaobjects.loader.MetaDataLoader; +import com.metaobjects.manager.ObjectConnection; +import com.metaobjects.manager.QueryOptions; +import com.metaobjects.manager.db.ObjectManagerDB; +import com.metaobjects.manager.db.driver.SQLiteDriver; +import com.metaobjects.object.MetaObject; +import com.metaobjects.object.value.ValueObject; +import com.metaobjects.registry.ObjectClassRegistry; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import javax.sql.DataSource; +import java.io.PrintWriter; +import java.nio.file.Files; +import java.nio.file.Path; +import java.sql.Connection; +import java.sql.SQLException; +import java.sql.Statement; +import java.util.Collection; +import java.util.List; +import java.util.Map; +import java.util.logging.Logger; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; + +/** + * B1 — a plain {@code MetaDataLoader.fromUris(...)} (no Spring, no Maven plugin) reads the + * {@code naming} of the project's {@code .metaobjects/config.json} found by walking up from + * the metadata file, so an OMDB read against SQLite columns named exactly like the fields + * ({@code firstName}) works. With no config the same read looks for {@code first_name}. + */ +final class LiteralNamingDiscoveryTest { + + private SqliteDatabase sqlite; + + @BeforeEach + void setUp() throws Exception { + ObjectClassRegistry reg = new ObjectClassRegistry(); + reg.register(() -> Map.of("lit::Person", ValueObject.class)); + ObjectClassRegistry.setGlobal(reg); + sqlite = new SqliteDatabase(); + try (Connection c = open(); Statement s = c.createStatement()) { + s.execute("CREATE TABLE people (id INTEGER PRIMARY KEY, \"firstName\" TEXT)"); + s.execute("INSERT INTO people VALUES (1, 'Ada')"); + } + } + + @AfterEach + void tearDown() { + sqlite.close(); + ObjectClassRegistry.resetGlobal(); + } + + private static Path project(String config) throws Exception { + Path root = Files.createTempDirectory("literal-proj-"); + if (config != null) { + Files.createDirectories(root.resolve(".metaobjects")); + Files.writeString(root.resolve(".metaobjects/config.json"), config); + } + Files.createDirectories(root.resolve("metaobjects")); + Files.writeString(root.resolve("metaobjects/meta.person.json"), """ + {"metadata.root":{"package":"lit","children":[ + {"object.entity":{"name":"Person","children":[ + {"source.rdb":{"@table":"people"}}, + {"field.long":{"name":"id"}}, + {"field.string":{"name":"firstName","@maxLength":50}}, + {"identity.primary":{"name":"id","@fields":"id","@generation":"increment"}} + ]}} + ]}}"""); + return root; + } + + private Collection readAll(MetaDataLoader loader) throws Exception { + ObjectManagerDB omdb = new ObjectManagerDB(); + omdb.setDatabaseDriver(new SQLiteDriver()); + omdb.setDataSource(dataSource()); + omdb.init(); + MetaObject person = loader.getMetaObjectByName("lit::Person"); + ObjectConnection oc = omdb.getConnection(); + try { + return omdb.getObjects(oc, person, new QueryOptions()); + } finally { + omdb.releaseConnection(oc); + } + } + + @Test + void fromUrisReadsLiteralColumnsFromTheProjectConfigFoundByWalkingUp() throws Exception { + Path root = project("{\"schema_version\":1,\"naming\":{\"columns\":\"literal\"}}"); + MetaDataLoader loader = MetaDataLoader.fromUris("literal", + List.of(root.resolve("metaobjects/meta.person.json").toUri())); + Collection rows = readAll(loader); + assertEquals(1, rows.size()); + assertEquals("Ada", ((ValueObject) rows.iterator().next()).getString("firstName")); + } + + @Test + void withoutAConfigTheRuntimeLooksForSnakeCaseColumns() throws Exception { + Path root = project(null); + MetaDataLoader loader = MetaDataLoader.fromUris("snake", + List.of(root.resolve("metaobjects/meta.person.json").toUri())); + assertThrows(Exception.class, () -> readAll(loader)); + } + + private DataSource dataSource() { + return new DataSource() { + @Override public Connection getConnection() throws SQLException { return open(); } + @Override public Connection getConnection(String u, String p) throws SQLException { return open(); } + @Override public PrintWriter getLogWriter() { return null; } + @Override public void setLogWriter(PrintWriter out) {} + @Override public void setLoginTimeout(int seconds) {} + @Override public int getLoginTimeout() { return 0; } + @Override public Logger getParentLogger() { return Logger.getGlobal(); } + @Override public T unwrap(Class iface) throws SQLException { throw new SQLException("not a wrapper"); } + @Override public boolean isWrapperFor(Class iface) { return false; } + }; + } + + private Connection open() throws SQLException { + return java.sql.DriverManager.getConnection(sqlite.jdbcUrl()); + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/ObjectManagerDbAdapter.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/ObjectManagerDbAdapter.java index 01e814098..263eba758 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/ObjectManagerDbAdapter.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/ObjectManagerDbAdapter.java @@ -514,8 +514,24 @@ private static Map toRowMap(MetaObject mc, Object instance, * {@link java.math.BigDecimal} natively and {@link Normalization} canonicalizes * it directly (the prior {@code BigDecimal.valueOf(double)} workaround is gone).

*/ + private static final java.util.regex.Pattern CANONICAL_DECIMAL = + java.util.regex.Pattern.compile("^-?(0|[1-9][0-9]*)(\\.[0-9]*[1-9])?$"); + private static Object coerceWireType(MetaField mf, Object value, Map columnSqlTypes) { if (value == null) return null; + // Typed decimal gate (ADR-0019 Amendment 1): the runtime hands a decimal over as an exact + // BigDecimal, and it renders as the canonical string. Checked BEFORE the value is + // normalized, so a double (which would canonicalize to a plausible string) fails here. + if (mf instanceof com.metaobjects.field.DecimalField) { + if (!(value instanceof java.math.BigDecimal bd)) { + throw new AssertionError("decimal field '" + mf.getName() + "' must be a BigDecimal from OMDB, got " + + value.getClass().getName() + " (" + value + ")"); + } + Object wire = Normalization.normalizeValue(bd); + if (!(wire instanceof String str) || !CANONICAL_DECIMAL.matcher(str).matches()) { + throw new AssertionError("decimal field '" + mf.getName() + "' did not render canonically: " + wire); + } + } if (isTimestampTzField(mf) && value instanceof java.util.Date d) { return d.toInstant().atOffset(ZoneOffset.UTC); } diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioMySqlTests.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioMySqlTests.java index 550d1e657..65c936f6f 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioMySqlTests.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioMySqlTests.java @@ -33,7 +33,7 @@ final class QueryScenarioMySqlTests { private static final Path CORPUS = ScenarioLoader.findCorpusRoot(); private static final List SCENARIOS = ScenarioLoader.loadQueries(CORPUS.resolve("queries")).stream() - .filter(s -> s.name().startsWith("report-")).toList(); + .filter(ScenarioEngine.MYSQL::runs).toList(); private static MySqlContainer mysql; @@ -42,7 +42,7 @@ static void beforeAll() { ObjectClassRegistry reg = new ObjectClassRegistry(); Map> bindings = new HashMap<>(); // The entities the report views read; a report itself is served by its read model. - for (String entity : List.of("Program", "Week", "Asset")) { + for (String entity : List.of("Program", "Week", "Asset", "Entry", "Gadget", "GadgetEvent")) { bindings.put("fitness::" + entity, ValueObject.class); } reg.register(() -> bindings); diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioRunner.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioRunner.java index 7fe525e24..d6ada5ff5 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioRunner.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioRunner.java @@ -161,6 +161,11 @@ private static Map probeColumnSqlTypes(JdbcTarget pg, MetaObjec : mc.getPrimaryRdbViewName(); if (relation == null) relation = mc.getPrimaryRdbTableName(); if (relation == null) return Map.of(); + // SQLite has one integer storage class: its catalog calls every integer column INTEGER, + // so the probe cannot tell an INT from a BIGINT and would narrow every key to a number. + // The declared field subtype (int reads as Integer, long as Long) is the only width + // information there, so on SQLite the probe contributes none. + if (engine == ScenarioEngine.SQLITE) return Map.of(); Map types = new LinkedHashMap<>(); String q = engine.quote(); String sql = "SELECT * FROM " + q + relation + q + " WHERE 1=0"; @@ -297,7 +302,7 @@ private static void executeSql(Connection c, String sql, ScenarioEngine engine) } /** Drop every view and table the split schema artifact creates (views first, tables in reverse order). */ - private static void dropSchema(Connection c, List statements) throws SQLException { + public static void dropSchema(Connection c, List statements) throws SQLException { List views = new java.util.ArrayList<>(); List tables = new java.util.ArrayList<>(); java.util.regex.Pattern create = java.util.regex.Pattern.compile("^CREATE (VIEW|TABLE) `([^`]+)`"); diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioSqliteTests.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioSqliteTests.java new file mode 100644 index 000000000..9491f3e8a --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioSqliteTests.java @@ -0,0 +1,59 @@ +package com.metaobjects.integration; + +import com.metaobjects.integration.Scenarios.QueryScenario; +import com.metaobjects.object.value.ValueObject; +import com.metaobjects.registry.ObjectClassRegistry; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; + +import java.nio.file.Path; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Stream; + +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; + +/** + * The persistence corpus' report scenarios ({@code report-*.yaml}) on SQLite, through the OMDB + * {@code SQLiteDriver}. The schema is the TS-produced {@code canonical/schema.sqlite.sql} (the + * {@code meta migrate} tables plus the report views), executed verbatim; the same Postgres + * {@code expect} blocks are asserted, with SQLite's value spelling mapped onto the wire form by + * {@link QueryScenarioRunner}. SQLite is embedded, so each scenario gets its own database file + * and the lane needs no docker. + */ +final class QueryScenarioSqliteTests { + + private static final Path CORPUS = ScenarioLoader.findCorpusRoot(); + private static final List SCENARIOS = + ScenarioLoader.loadQueries(CORPUS.resolve("queries")).stream() + .filter(ScenarioEngine.SQLITE::runs).toList(); + + @BeforeAll + static void beforeAll() { + ObjectClassRegistry reg = new ObjectClassRegistry(); + Map> bindings = new HashMap<>(); + // The entities the report views read; a report itself is served by its read model. + for (String entity : List.of("Program", "Week", "Asset", "Entry", "Gadget", "GadgetEvent")) { + bindings.put("fitness::" + entity, ValueObject.class); + } + reg.register(() -> bindings); + ObjectClassRegistry.setGlobal(reg); + } + + static Stream scenarios() { + return SCENARIOS.stream().map(s -> Arguments.of(s.name(), s)); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("scenarios") + void reportScenario(String name, QueryScenario scenario) { + assertDoesNotThrow(() -> { + try (SqliteDatabase db = new SqliteDatabase()) { + QueryScenarioRunner.run(scenario, db, CORPUS.resolve("canonical"), ScenarioEngine.SQLITE); + } + }); + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioTests.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioTests.java index d592ba899..dd4c58969 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioTests.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/QueryScenarioTests.java @@ -39,7 +39,8 @@ final class QueryScenarioTests { private static final Path CORPUS = ScenarioLoader.findCorpusRoot(); private static final List SCENARIOS = - ScenarioLoader.loadQueries(CORPUS.resolve("queries")); + ScenarioLoader.loadQueries(CORPUS.resolve("queries")).stream() + .filter(ScenarioEngine.POSTGRES::runs).toList(); /** * Scenarios deferred until the underlying Java port gap is closed. @@ -99,6 +100,10 @@ static void beforeAll() { bindings.put("fitness::BridgeAuth", ValueObject.class); bindings.put("fitness::CopayAuth", ValueObject.class); bindings.put("fitness::PriorAuthAuth", ValueObject.class); + // ADR-0019 Amendment 1 decimal corpus + the redeclared-field report's tables. + bindings.put("fitness::Entry", ValueObject.class); + bindings.put("fitness::Gadget", ValueObject.class); + bindings.put("fitness::GadgetEvent", ValueObject.class); reg.register(() -> bindings); ObjectClassRegistry.setGlobal(reg); } diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/RoundtripWriter.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/RoundtripWriter.java index 4b1a33892..550843b58 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/RoundtripWriter.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/RoundtripWriter.java @@ -74,6 +74,10 @@ static Map execute(ObjectManagerDB omdb, ObjectConnection conn, // 2. INSERT via the OMDB runtime write path (the write codecs run here). The PK is left // unset: identity.primary @generation:uuid → OMDB mints the uuid app-side on insert. + // A row that NAMES its generated key (a scenario asserting ids 7, 8, 9...) is written under + // that key: OMDB leaves an identity column to the engine, so the harness positions the + // engine's counter instead of changing the runtime's insert rule. + positionIdentityCounter(conn, mc, spec); omdb.createObject(conn, vo); // 3. Capture the PK OMDB populated, then read the row back BY PK (a fresh DB SELECT, so @@ -92,6 +96,27 @@ static Map execute(ObjectManagerDB omdb, ObjectConnection conn, return row; } + /** Make the engine's next generated key the one the {@code insert:} block names, when it names one. */ + private static void positionIdentityCounter(ObjectConnection conn, MetaObject mc, QuerySpec spec) throws Exception { + MetaField pk = primaryKeyField(mc); + Object named = spec.insert().get(pk.getName()); + if (!(named instanceof Number n) || !(conn.getDatastoreConnection() instanceof java.sql.Connection jdbc)) return; + if (!"increment".equals(mc.getPrimaryIdentity().getGeneration())) return; + String table = mc.getPrimaryRdbTableName(); + long next = n.longValue(); + String product = jdbc.getMetaData().getDatabaseProductName().toLowerCase(java.util.Locale.ROOT); + try (java.sql.Statement st = jdbc.createStatement()) { + if (product.contains("postgres")) { + st.execute("ALTER TABLE \"" + table + "\" ALTER COLUMN \"" + pk.getName() + "\" RESTART WITH " + next); + } else if (product.contains("mysql")) { + st.execute("ALTER TABLE `" + table + "` AUTO_INCREMENT = " + next); + } else if (product.contains("sqlite")) { + st.execute("DELETE FROM sqlite_sequence WHERE name = '" + table + "'"); + st.execute("INSERT INTO sqlite_sequence (name, seq) VALUES ('" + table + "', " + (next - 1) + ")"); + } + } + } + // ----------------------------------------------------------------------- // Field / key resolution // ----------------------------------------------------------------------- diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/ScenarioEngine.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/ScenarioEngine.java index 5291e7b98..1f7dcad57 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/ScenarioEngine.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/ScenarioEngine.java @@ -4,6 +4,7 @@ import com.metaobjects.manager.db.DatabaseDriver; import com.metaobjects.manager.db.driver.MySQLDriver; import com.metaobjects.manager.db.driver.PostgresDriver; +import com.metaobjects.manager.db.driver.SQLiteDriver; import java.util.Map; import java.util.regex.Pattern; @@ -27,6 +28,15 @@ public enum ScenarioEngine { if (own != null && own.get("mysql") != null) return own.get("mysql"); return s.seedData() == null ? null : toMysqlSeed(s.seedData()); } + }, + SQLITE("sqlite", "\"") { + @Override public DatabaseDriver newDriver() { return new SQLiteDriver(); } + // SQLite takes the Postgres seed as written (double-quoted identifiers, ISO instants as text). + @Override public String seed(QueryScenario s) { + Map own = s.seedDataEngine(); + if (own != null && own.get("sqlite") != null) return own.get("sqlite"); + return s.seedData(); + } }; private final String id; @@ -34,6 +44,18 @@ public enum ScenarioEngine { ScenarioEngine(String id, String quote) { this.id = id; this.quote = quote; } + /** The name a scenario's {@code engines:} list uses for this engine. */ + public String id() { return id; } + + /** + * Whether this engine runs {@code scenario}. An {@code engines:} list is EXACT; without one, + * Postgres runs everything and the other engines run the {@code report-*} scenarios. + */ + public boolean runs(QueryScenario scenario) { + if (scenario.engines() != null) return scenario.engines().contains(id); + return this == POSTGRES || scenario.name().startsWith("report-"); + } + /** The schema artifact name under the corpus' {@code canonical/}. */ public String schemaArtifact() { return "canonical/schema." + id + ".sql"; } /** The identifier quote the engine uses. */ @@ -78,7 +100,7 @@ static String toMysqlSeed(String seed) { } /** The statements of a script: split on a {@code ;} at the end of a line (no body carries one mid-line). */ - static java.util.List splitStatements(String script) { + public static java.util.List splitStatements(String script) { StringBuilder sb = new StringBuilder(); for (String line : script.split("\n", -1)) { if (!line.trim().startsWith("--")) sb.append(line).append('\n'); diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/ScenarioLoader.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/ScenarioLoader.java index ac22055c4..e1f0996ca 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/ScenarioLoader.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/ScenarioLoader.java @@ -77,9 +77,30 @@ private static QueryScenario parseQuery(Path file, Map root) { file.toString(), (String) root.get("seed-data"), (Map) root.get("seed-data-engine"), + parseEngines(file, root), Collections.unmodifiableList(queries)); } + /** Engine names a scenario may list (README, "Which engines run a scenario"). */ + private static final List KNOWN_ENGINES = List.of("postgres", "mysql", "sqlite", "d1"); + + /** The scenario's {@code engines:} list, exact when present; {@code null} when absent. */ + private static List parseEngines(Path file, Map root) { + if (!root.containsKey("engines")) return null; + Object raw = root.get("engines"); + if (!(raw instanceof List list) || list.isEmpty()) { + throw new IllegalStateException(file + ": `engines` must be a non-empty list"); + } + List out = new ArrayList<>(); + for (Object o : list) { + if (!(o instanceof String name) || !KNOWN_ENGINES.contains(name)) { + throw new IllegalStateException(file + ": unknown engine " + o + " in `engines` (one of " + KNOWN_ENGINES + ")"); + } + out.add(name); + } + return List.copyOf(out); + } + @SuppressWarnings("unchecked") private static QuerySpec parseQuerySpec(Map q) { List> sortsRaw = (List>) q.get("sort"); diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/Scenarios.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/Scenarios.java index a19e88f93..f0056e0da 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/Scenarios.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/Scenarios.java @@ -34,6 +34,7 @@ public record QueryScenario( String sourcePath, String seedData, Map seedDataEngine, // engine -> seed SQL, when an engine needs its own spelling + List engines, // exact engine list when present; null = the historical rule List queries ) {} } diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/SqliteDatabase.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/SqliteDatabase.java new file mode 100644 index 000000000..a92bab63a --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/SqliteDatabase.java @@ -0,0 +1,36 @@ +package com.metaobjects.integration; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.Files; +import java.nio.file.Path; + +/** + * A throwaway SQLite database file for one test. SQLite is embedded, so there is no container: + * the file lives in a temp directory and {@link #close()} removes it. + */ +public final class SqliteDatabase implements JdbcTarget, AutoCloseable { + private final Path dir; + private final String jdbcUrl; + + public SqliteDatabase() { + try { + dir = Files.createTempDirectory("mo-sqlite-"); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + jdbcUrl = "jdbc:sqlite:" + dir.resolve("test.db"); + } + + @Override public String jdbcUrl() { return jdbcUrl; } + @Override public String username() { return ""; } + @Override public String password() { return ""; } + + @Override public void close() { + try (var files = Files.walk(dir)) { + files.sorted(java.util.Comparator.reverseOrder()).forEach(p -> p.toFile().delete()); + } catch (IOException ignored) { + // a leftover temp file is harmless + } + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/SqliteDecimalReadTest.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/SqliteDecimalReadTest.java new file mode 100644 index 000000000..5d04f8a4b --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/SqliteDecimalReadTest.java @@ -0,0 +1,118 @@ +package com.metaobjects.integration; + +import com.metaobjects.loader.MetaDataLoader; +import com.metaobjects.manager.ObjectConnection; +import com.metaobjects.manager.QueryOptions; +import com.metaobjects.manager.db.ObjectManagerDB; +import com.metaobjects.manager.db.driver.SQLiteDriver; +import com.metaobjects.object.MetaObject; +import com.metaobjects.object.value.ValueObject; +import com.metaobjects.registry.ObjectClassRegistry; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import javax.sql.DataSource; +import java.io.PrintWriter; +import java.math.BigDecimal; +import java.nio.file.Files; +import java.nio.file.Path; +import java.sql.Connection; +import java.sql.SQLException; +import java.sql.Statement; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.logging.Logger; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +/** + * ADR-0019 Amendment 1, W4: a decimal read from SQLite is exact to 15 significant digits. + * A NUMERIC column holds an INTEGER or a REAL; the OMDB read renders a REAL at 15 digits, so + * {@code 0.1 + 0.2} reads {@code 0.3}, never {@code 0.30000000000000004}. + */ +final class SqliteDecimalReadTest { + + private SqliteDatabase sqlite; + private MetaDataLoader loader; + + @BeforeEach + void setUp() throws Exception { + ObjectClassRegistry reg = new ObjectClassRegistry(); + reg.register(() -> Map.of("dec::Entry", ValueObject.class)); + ObjectClassRegistry.setGlobal(reg); + Path root = Files.createTempDirectory("dec-proj-"); + Files.createDirectories(root.resolve(".metaobjects")); + Files.writeString(root.resolve(".metaobjects/config.json"), + "{\"schema_version\":1,\"naming\":{\"columns\":\"literal\"}}"); + Path file = root.resolve("meta.entry.json"); + Files.writeString(file, """ + {"metadata.root":{"package":"dec","children":[ + {"object.entity":{"name":"Entry","children":[ + {"source.rdb":{"@table":"entries"}}, + {"field.long":{"name":"id"}}, + {"field.decimal":{"name":"amount","@precision":12,"@scale":2}}, + {"identity.primary":{"name":"id","@fields":"id","@generation":"increment"}} + ]}} + ]}}"""); + loader = MetaDataLoader.fromUris("dec", List.of(file.toUri())); + sqlite = new SqliteDatabase(); + try (Connection c = open(); Statement s = c.createStatement()) { + s.execute("CREATE TABLE entries (id INTEGER PRIMARY KEY, amount NUMERIC(12,2))"); + s.execute("INSERT INTO entries VALUES (1, 0.1 + 0.2)"); + s.execute("INSERT INTO entries VALUES (2, 12.50)"); + s.execute("INSERT INTO entries VALUES (3, 200)"); + s.execute("INSERT INTO entries VALUES (4, 1234567890123456789)"); + s.execute("INSERT INTO entries VALUES (5, NULL)"); + } + } + + @AfterEach + void tearDown() { + sqlite.close(); + ObjectClassRegistry.resetGlobal(); + } + + @Test + void aStoredRealReadsToFifteenSignificantDigits() throws Exception { + ObjectManagerDB omdb = new ObjectManagerDB(); + omdb.setDatabaseDriver(new SQLiteDriver()); + omdb.setDataSource(dataSource()); + omdb.init(); + MetaObject entry = loader.getMetaObjectByName("dec::Entry"); + ObjectConnection oc = omdb.getConnection(); + List read = new ArrayList<>(); + try { + for (Object o : omdb.getObjects(oc, entry, new QueryOptions())) { + read.add((BigDecimal) ((ValueObject) o).getObject("amount")); + } + } finally { + omdb.releaseConnection(oc); + } + assertEquals(5, read.size()); + assertEquals("0.3", read.get(0).toPlainString(), "0.1 + 0.2"); + assertEquals("12.5", read.get(1).toPlainString()); + assertEquals("200", read.get(2).toPlainString()); + assertEquals("1234567890123456789", read.get(3).toPlainString(), "an INTEGER is exact"); + assertEquals(null, read.get(4)); + } + + private Connection open() throws SQLException { + return java.sql.DriverManager.getConnection(sqlite.jdbcUrl()); + } + + private DataSource dataSource() { + return new DataSource() { + @Override public Connection getConnection() throws SQLException { return open(); } + @Override public Connection getConnection(String u, String p) throws SQLException { return open(); } + @Override public PrintWriter getLogWriter() { return null; } + @Override public void setLogWriter(PrintWriter out) {} + @Override public void setLoginTimeout(int seconds) {} + @Override public int getLoginTimeout() { return 0; } + @Override public Logger getParentLogger() { return Logger.getGlobal(); } + @Override public T unwrap(Class iface) throws SQLException { throw new SQLException("not a wrapper"); } + @Override public boolean isWrapperFor(Class iface) { return false; } + }; + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/SqliteObjectManagerTest.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/SqliteObjectManagerTest.java new file mode 100644 index 000000000..21203528f --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/SqliteObjectManagerTest.java @@ -0,0 +1,195 @@ +package com.metaobjects.integration; + +import com.metaobjects.loader.MetaDataLoader; +import com.metaobjects.manager.ObjectConnection; +import com.metaobjects.manager.QueryOptions; +import com.metaobjects.manager.db.ObjectManagerDB; +import com.metaobjects.manager.db.driver.SQLiteDriver; +import com.metaobjects.manager.exp.Expression; +import com.metaobjects.manager.exp.Range; +import com.metaobjects.manager.exp.SortOrder; +import com.metaobjects.object.MetaObject; +import com.metaobjects.object.value.ValueObject; +import com.metaobjects.registry.ObjectClassRegistry; +import com.metaobjects.loader.uri.URIHelper; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; + +import javax.sql.DataSource; +import java.io.PrintWriter; +import java.math.BigDecimal; +import java.sql.Connection; +import java.sql.SQLException; +import java.sql.Statement; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Date; +import java.util.List; +import java.util.Map; +import java.util.logging.Logger; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * The OMDB runtime against a real SQLite database: insert, read, query, page, update, delete. + * SQLite is embedded, so there is no container and the test needs no docker. + * + *

The table is written here the way {@code meta migrate --dialect sqlite} writes one (a + * {@code TEXT} column for a timestamp and a uuid, {@code INTEGER} for a boolean). What it pins: + *

    + *
  • {@code INTEGER PRIMARY KEY AUTOINCREMENT} is read back with {@code last_insert_rowid()}.
    • + *
  • A column named after a reserved word ({@code rank}) works in INSERT, SELECT, WHERE, + * ORDER BY and UPDATE (identifiers are double-quoted).
    • + *
  • {@code LIMIT .. OFFSET ..} paging.
    • + *
  • A timestamp is ISO-8601 UTC text and round-trips to the same instant whatever the JVM + * zone is; a uuid is lower-case text; a decimal keeps its value.
    • + *
+ */ +final class SqliteObjectManagerTest { + + private static final long CREATED_MS = 1_780_000_000_000L; + + private static SqliteDatabase sqlite; + private static MetaDataLoader loader; + private static java.util.TimeZone previousZone; + + @BeforeAll + static void beforeAll() throws Exception { + // A NON-UTC JVM zone on purpose: the DATETIME round-trip must not depend on it. + previousZone = java.util.TimeZone.getDefault(); + java.util.TimeZone.setDefault(java.util.TimeZone.getTimeZone("America/New_York")); + ObjectClassRegistry reg = new ObjectClassRegistry(); + reg.register(() -> Map.of("shop::Product", ValueObject.class)); + ObjectClassRegistry.setGlobal(reg); + + loader = MetaDataLoader.fromUris("sqlite-shop", + List.of(URIHelper.toURI("model:resource:meta.mysql-shop.json"))); + // This fixture's hand-rolled DDL below declares every column literally + // (createdAt) — it needs the `literal` naming.columns rule (ADR-0058's default + // is now snake_case in every port). + loader.getRoot().setPhysicalNaming( + com.metaobjects.database.PhysicalNaming.of("literal", "snake_case")); + sqlite = new SqliteDatabase(); + try (Connection c = open(); Statement s = c.createStatement()) { + s.execute(""" + CREATE TABLE products ( + id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, + title VARCHAR(200) NOT NULL, + "rank" INTEGER NOT NULL, + active INTEGER NOT NULL, + price NUMERIC(10,2), + createdAt TEXT NOT NULL, + localAt TEXT, + externalId TEXT + )"""); + } + } + + @AfterAll + static void afterAll() { + if (sqlite != null) sqlite.close(); + if (previousZone != null) java.util.TimeZone.setDefault(previousZone); + ObjectClassRegistry.resetGlobal(); + } + + @Test + void crudRoundTripsThroughTheSqliteDriver() throws Exception { + ObjectManagerDB omdb = new ObjectManagerDB(); + omdb.setDatabaseDriver(new SQLiteDriver()); + omdb.setDataSource(dataSource()); + omdb.init(); + + MetaObject product = loader.getMetaObjectByName("shop::Product"); + ObjectConnection oc = omdb.getConnection(); + try { + // --- create: the auto-increment key is read back onto the object ------------- + List created = new ArrayList<>(); + for (int i = 1; i <= 3; i++) { + ValueObject vo = (ValueObject) product.newInstance(); + vo.setString("title", "Product " + i); + vo.setInt("rank", 10 * i); + vo.setBoolean("active", i != 2); + vo.setObject("price", new BigDecimal(i + ".50")); + vo.setDate("createdAt", new Date(CREATED_MS + i)); + vo.setDate("localAt", new Date(CREATED_MS + i)); + vo.setString("externalId", "AAAAAAAA-AAAA-4AAA-8AAA-00000000000" + i); + omdb.createObject(oc, vo); + created.add(vo); + } + assertEquals(1L, created.get(0).getLong("id"), "first insert gets id 1 from last_insert_rowid()"); + assertEquals(3L, created.get(2).getLong("id"), "third insert gets id 3"); + + // --- read by key, including the reserved-word column -------------------------- + ValueObject two = only(omdb.getObjects(oc, product, + new QueryOptions(new Expression("id", 2L, Expression.EQUAL)))); + assertEquals("Product 2", two.getString("title")); + assertEquals(20, two.getInt("rank")); + assertEquals(Boolean.FALSE, two.getBoolean("active")); + assertEquals(0, new BigDecimal("2.50").compareTo((BigDecimal) two.getObject("price"))); + assertEquals(CREATED_MS + 2, two.getDate("createdAt").getTime(), + "a timestamp round-trips to the same instant"); + assertEquals(CREATED_MS + 2, two.getDate("localAt").getTime(), + "an @localTime timestamp round-trips to the same UTC wall clock"); + assertEquals("aaaaaaaa-aaaa-4aaa-8aaa-000000000002", two.getString("externalId"), + "uuid is stored as text and read back lowercase-canonical"); + try (Connection c = open(); Statement s = c.createStatement(); + java.sql.ResultSet rs = s.executeQuery( + "SELECT createdAt FROM products WHERE id = 2")) { + assertTrue(rs.next()); + assertEquals(java.time.Instant.ofEpochMilli(CREATED_MS + 2).toString(), + rs.getString(1), "the column holds the ISO-8601 UTC text of the instant"); + } + + // --- WHERE and ORDER BY on the reserved-word column, then LIMIT paging -------- + QueryOptions page = new QueryOptions( + new Expression("rank", 10, Expression.GREATER), + new SortOrder("rank", SortOrder.DESC), + new Range(2, 2)); + ValueObject second = only(omdb.getObjects(oc, product, page)); + assertEquals("Product 2", second.getString("title"), + "rank > 10 ordered DESC is [30, 20]; the second row of the page is rank 20"); + + // --- update --------------------------------------------------------------------- + two.setString("title", "Renamed"); + two.setInt("rank", 99); + omdb.updateObject(oc, two); + ValueObject reread = only(omdb.getObjects(oc, product, + new QueryOptions(new Expression("id", 2L, Expression.EQUAL)))); + assertEquals("Renamed", reread.getString("title")); + assertEquals(99, reread.getInt("rank")); + + // --- delete --------------------------------------------------------------------- + omdb.deleteObject(oc, reread); + assertEquals(2, omdb.getObjects(oc, product, new QueryOptions()).size()); + } finally { + omdb.releaseConnection(oc); + } + } + + private static ValueObject only(Collection rows) { + assertEquals(1, rows.size(), "expected exactly one row, got " + rows.size()); + Object first = rows.iterator().next(); + assertTrue(first instanceof ValueObject, "row should be a ValueObject"); + return (ValueObject) first; + } + + private static Connection open() throws SQLException { + return java.sql.DriverManager.getConnection(sqlite.jdbcUrl()); + } + + private static DataSource dataSource() { + return new DataSource() { + @Override public Connection getConnection() throws SQLException { return open(); } + @Override public Connection getConnection(String u, String p) throws SQLException { return open(); } + @Override public PrintWriter getLogWriter() { return null; } + @Override public void setLogWriter(PrintWriter out) {} + @Override public void setLoginTimeout(int seconds) {} + @Override public int getLoginTimeout() { return 0; } + @Override public Logger getParentLogger() { return Logger.getGlobal(); } + @Override public T unwrap(Class iface) throws SQLException { throw new SQLException("not a wrapper"); } + @Override public boolean isWrapperFor(Class iface) { return false; } + }; + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractAssertions.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractAssertions.java index 87414442c..d54260d37 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractAssertions.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractAssertions.java @@ -178,14 +178,14 @@ static void assertResponse(String scenarioName, ApiRequest request, int status, } /** - * Structural equality that normalizes Number subtypes - * (Integer/Long/etc are equal if their long value matches) and - * recurses into Maps/Lists. + * Structural equality that compares Numbers exactly (ADR-0019 Amendment 1: never through a + * double or a truncated long — {@code 0.3} must not equal {@code 0}) and recurses into + * Maps/Lists. Response bodies are parsed with {@code USE_BIG_DECIMAL_FOR_FLOATS}. */ private static boolean structuralEquals(Object a, Object b) { if (a == b) return true; if (a == null || b == null) return false; - if (a instanceof Number an && b instanceof Number bn) return an.longValue() == bn.longValue(); + if (a instanceof Number an && b instanceof Number bn) return toBigDecimal(an).compareTo(toBigDecimal(bn)) == 0; if (a instanceof List al && b instanceof List bl) { if (al.size() != bl.size()) return false; for (int i = 0; i < al.size(); i++) @@ -201,6 +201,12 @@ private static boolean structuralEquals(Object a, Object b) { return a.equals(b); } + private static java.math.BigDecimal toBigDecimal(Number n) { + if (n instanceof java.math.BigDecimal d) return d; + if (n instanceof java.math.BigInteger i) return new java.math.BigDecimal(i); + return new java.math.BigDecimal(n.toString()); + } + private static boolean isEmpty(Object body) { if (body == null) return true; if (body instanceof String s) return s.isEmpty(); diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractAssertionsExactNumbersTest.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractAssertionsExactNumbersTest.java new file mode 100644 index 000000000..582b966d4 --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractAssertionsExactNumbersTest.java @@ -0,0 +1,46 @@ +package com.metaobjects.integration.api; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.metaobjects.integration.api.ApiContractScenarios.ApiRequest; +import org.junit.jupiter.api.Test; + +import java.math.BigDecimal; +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; +import static org.junit.jupiter.api.Assertions.assertThrows; + +/** + * The assertion engine compares numbers exactly (ADR-0019 Amendment 1): a non-integer is never + * truncated to a long, and a body number is read as a BigDecimal, not a double. + */ +final class ApiContractAssertionsExactNumbersTest { + + private static void check(Object body, Object expected) { + ApiRequest request = new ApiRequest("r", "GET", "/x", null, 200, Map.of("equals", expected)); + ApiContractAssertions.assertResponse("exact", request, 200, body); + } + + @Test + void aFractionIsNotTruncatedToZero() { + assertThrows(AssertionError.class, () -> check(new BigDecimal("0.3"), 0)); + assertThrows(AssertionError.class, () -> check(Map.of("v", new BigDecimal("12.5")), Map.of("v", 12))); + } + + @Test + void scaleDoesNotMatterButValueDoes() { + assertDoesNotThrow(() -> check(new BigDecimal("0.30"), new BigDecimal("0.3"))); + assertDoesNotThrow(() -> check(List.of(1, new BigDecimal("2.0")), List.of(1L, 2))); + assertThrows(AssertionError.class, + () -> check(new BigDecimal("12345678901234567.89"), new BigDecimal("12345678901234568"))); + } + + @Test + void aBodyNumberBeyondADoubleSurvivesParsing() throws Exception { + Object parsed = TomcatHost.parseBody(new ObjectMapper(), "{\"v\":12345678901234567.89}"); + Object v = ((Map) parsed).get("v"); + assertThrows(AssertionError.class, () -> check(v, new BigDecimal("12345678901234568"))); + assertDoesNotThrow(() -> check(v, new BigDecimal("12345678901234567.89"))); + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractConformanceTest.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractConformanceTest.java index d5781dbba..f4f396d51 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractConformanceTest.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractConformanceTest.java @@ -105,7 +105,7 @@ private void runScenario(AuthorApiServer server, ApiScenario scenario) throws Ex HttpResponse res = client.send(builder.build(), HttpResponse.BodyHandlers.ofString()); Object parsed = (res.body() == null || res.body().isEmpty()) ? null - : MAPPER.readValue(res.body(), Object.class); + : MAPPER.copy().enable(com.fasterxml.jackson.databind.DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS).readValue(res.body(), Object.class); ApiContractAssertions.assertResponse(scenario.name(), req, res.statusCode(), parsed); } } diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractScenarioLoader.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractScenarioLoader.java index ace6424df..4365319d5 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractScenarioLoader.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractScenarioLoader.java @@ -79,9 +79,33 @@ private static ApiScenario parseScenario(Path file, Map root) { asString(root.get("description"), ""), file.toString(), truncate, + parseEngines(file, root), requests); } + private static final List KNOWN_ENGINES = List.of("postgres", "mysql", "sqlite", "d1"); + + /** The scenario's {@code engines:} list, exact when present; {@code null} when absent. */ + private static List parseEngines(Path file, Map root) { + if (!root.containsKey("engines")) return null; + if (!(root.get("engines") instanceof List list) || list.isEmpty()) { + throw new IllegalStateException(file + ": `engines` must be a non-empty list"); + } + List out = new ArrayList<>(); + for (Object o : list) { + if (!(o instanceof String name) || !KNOWN_ENGINES.contains(name)) { + throw new IllegalStateException(file + ": unknown engine " + o + " in `engines` (one of " + KNOWN_ENGINES + ")"); + } + out.add(name); + } + return List.copyOf(out); + } + + /** Whether a scenario runs on {@code engineId}: an {@code engines:} list is exact, absent means every engine. */ + static boolean runsOn(ApiScenario scenario, String engineId) { + return scenario.engines() == null || scenario.engines().contains(engineId); + } + @SuppressWarnings("unchecked") private static ApiRequest parseRequest(Map r) { Map expect = (Map) r.get("expect"); diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractScenarios.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractScenarios.java index 5610d56d9..5f279e5da 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractScenarios.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ApiContractScenarios.java @@ -32,6 +32,7 @@ record ApiScenario( String description, String sourcePath, boolean truncate, // true when setup.truncate is set + List engines, // exact engine list when present; null = every engine the lane has List requests ) {} } diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalCorpus.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalCorpus.java new file mode 100644 index 000000000..4bdba650b --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalCorpus.java @@ -0,0 +1,53 @@ +package com.metaobjects.integration.api; + +import com.fasterxml.jackson.databind.ObjectMapper; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.Map; + +/** + * Locates the decimal api-contract sub-corpus ({@code fixtures/api-contract-conformance/decimal/}, + * ADR-0019 Amendment 1), its committed per-engine schema and its {@code entries} seed. The + * {@code reports} half of {@code seed.json} is what the SEAM lanes serve; this lane lets the + * real view derive the rows, so it never reads it. + */ +final class DecimalCorpus { + private DecimalCorpus() {} + + private static final ObjectMapper MAPPER = new ObjectMapper(); + + static Path root() { + return ApiContractScenarioLoader.findCorpusRoot().resolve("decimal"); + } + + static Path scenariosDir() { + return root().resolve("scenarios"); + } + + /** The committed TypeScript-produced schema of {@code engineId}, read verbatim. */ + static String schema(String engineId) { + try { + return Files.readString(root().resolve("schema." + engineId + ".sql"), StandardCharsets.UTF_8); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + + /** The seed {@code entries} rows, each {@code amount} still the TEXT the file holds (never a double). */ + @SuppressWarnings("unchecked") + static List> entries() { + try { + Map parsed = MAPPER.readValue( + Files.readString(root().resolve("seed.json"), StandardCharsets.UTF_8), + new com.fasterxml.jackson.core.type.TypeReference>() {}); + return (List>) parsed.get("entries"); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbApiContractLane.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbApiContractLane.java new file mode 100644 index 000000000..0c21882ea --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbApiContractLane.java @@ -0,0 +1,185 @@ +package com.metaobjects.integration.api; + +import com.metaobjects.integration.JdbcTarget; +import com.metaobjects.integration.QueryScenarioRunner; +import com.metaobjects.integration.ScenarioEngine; +import com.metaobjects.integration.api.ApiContractScenarios.ApiRequest; +import com.metaobjects.integration.api.ApiContractScenarios.ApiScenario; +import com.metaobjects.integration.api.generated.GeneratedDecimalControllerHarness; +import com.metaobjects.manager.db.ObjectManagerDB; +import com.metaobjects.object.value.ValueObject; +import com.metaobjects.registry.ObjectClassRegistry; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.TestInstance; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; + +import javax.sql.DataSource; +import java.io.PrintWriter; +import java.nio.file.Files; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.SQLException; +import java.sql.Statement; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.logging.Logger; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; +import static org.junit.jupiter.api.Assertions.assertFalse; + +/** + * ADR-0019 Amendment 1 - the Java full-stack lane of the {@code decimal/} api-contract sub-corpus on a + * real database; one subclass per engine. The artifacts under test are the GENERATED + * {@code EntryController} / {@code EntryTotalsController}, DTOs, allowlists and repository interfaces, + * mounted unmodified on embedded Tomcat over OMDB-backed repositories. The database executes the + * committed TypeScript-produced {@code decimal/schema..sql} verbatim; {@code entries} is + * truncated and re-seeded before EVERY scenario (the scenarios write), and the real view derives the + * report. A scenario's {@code engines:} list is exact. + */ +@TestInstance(TestInstance.Lifecycle.PER_CLASS) +abstract class DecimalOmdbApiContractLane { + + private final List scenarios = ApiContractScenarioLoader.loadScenarios(DecimalCorpus.scenariosDir()); + + private GeneratedDecimalControllerHarness harness; + private JdbcTarget target; + + abstract ScenarioEngine engine(); + + abstract JdbcTarget openDatabase(); + + abstract void closeDatabase(); + + @BeforeAll + void setUp() throws Exception { + target = openDatabase(); + provisionSchema(); + + harness = new GeneratedDecimalControllerHarness(DecimalCorpus.root(), Files.createTempDirectory("decimal-omdb-controllers")); + + ObjectClassRegistry reg = new ObjectClassRegistry(); + Map> bindings = new HashMap<>(); + bindings.put("acme::ledger::Entry", ValueObject.class); + reg.register(() -> bindings); + ObjectClassRegistry.setGlobal(reg); + + ObjectManagerDB omdb = new ObjectManagerDB(); + omdb.setDatabaseDriver(engine().newDriver()); + omdb.setDataSource(dataSource()); + omdb.init(); + harness.start(omdb); + } + + @AfterAll + void tearDown() throws Exception { + try { + if (harness != null) harness.close(); + } finally { + closeDatabase(); + } + } + + Stream scenarios() { + return scenarios.stream() + .filter(s -> ApiContractScenarioLoader.runsOn(s, engine().id())) + .map(s -> Arguments.of(s.name(), s)); + } + + @Test + void theEngineRunsAtLeastTheFifteenDigitSubset() { + // A scenarios directory that resolved to nothing, or an engines: filter that dropped + // everything, would leave this lane green and empty. + assertFalse(scenarios().findAny().isEmpty()); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("scenarios") + void scenario(String name, ApiScenario scenario) { + assertDoesNotThrow(() -> { + reseed(); + for (ApiRequest req : scenario.requests()) { + GeneratedDecimalControllerHarness.Response res = + harness.exchange(req.method(), req.path(), req.body()); + Object parsed = harness.parseBody(res.body()); + ApiContractAssertions.assertResponse(scenario.name(), req, res.status(), parsed); + } + }); + } + + // ----------------------------------------------------------------------- + // database provisioning + // ----------------------------------------------------------------------- + + private void provisionSchema() throws SQLException { + ScenarioEngine engine = engine(); + String schema = DecimalCorpus.schema(engine.id()); + try (Connection c = connection()) { + if (engine == ScenarioEngine.POSTGRES) { + try (Statement s = c.createStatement()) { s.execute(schema); } + } else { + List statements = ScenarioEngine.splitStatements(schema); + if (engine == ScenarioEngine.MYSQL) QueryScenarioRunner.dropSchema(c, statements); + try (Statement s = c.createStatement()) { + for (String stmt : statements) s.execute(stmt); + } + } + } + } + + /** Empty {@code entries}, insert the seed rows under their own ids, and park the counter after them. */ + private void reseed() throws SQLException { + ScenarioEngine engine = engine(); + String q = engine.quote(); + List> rows = DecimalCorpus.entries(); + long next = rows.stream().mapToLong(r -> ((Number) r.get("id")).longValue()).max().orElse(0) + 1; + try (Connection c = connection(); Statement s = c.createStatement()) { + s.execute((engine == ScenarioEngine.SQLITE ? "DELETE FROM " : "TRUNCATE TABLE ") + q + "entries" + q); + for (Map row : rows) { + String cols = row.keySet().stream().map(k -> q + k + q).collect(Collectors.joining(", ")); + String vals = row.values().stream().map(DecimalOmdbApiContractLane::literal).collect(Collectors.joining(", ")); + s.execute("INSERT INTO " + q + "entries" + q + " (" + cols + ") VALUES (" + vals + ")"); + } + switch (engine) { + case POSTGRES -> s.execute("ALTER TABLE \"entries\" ALTER COLUMN \"id\" RESTART WITH " + next); + case MYSQL -> s.execute("ALTER TABLE `entries` AUTO_INCREMENT = " + next); + case SQLITE -> { + s.execute("DELETE FROM sqlite_sequence WHERE name = 'entries'"); + s.execute("INSERT INTO sqlite_sequence (name, seq) VALUES ('entries', " + (next - 1) + ")"); + } + } + } + } + + /** A seed value as a SQL literal; a decimal stays the text the file holds, never a double. */ + private static String literal(Object v) { + if (v == null) return "NULL"; + if (v instanceof Boolean) return v.toString(); // TRUE/FALSE: valid on all three engines + if (v instanceof Number) return v.toString(); + return "'" + v.toString().replace("'", "''") + "'"; + } + + private Connection connection() throws SQLException { + return DriverManager.getConnection(target.jdbcUrl(), target.username(), target.password()); + } + + private DataSource dataSource() { + return new DataSource() { + @Override public Connection getConnection() throws SQLException { return connection(); } + @Override public Connection getConnection(String u, String p) throws SQLException { return connection(); } + @Override public PrintWriter getLogWriter() { return null; } + @Override public void setLogWriter(PrintWriter out) {} + @Override public void setLoginTimeout(int seconds) {} + @Override public int getLoginTimeout() { return 0; } + @Override public Logger getParentLogger() { return Logger.getLogger("integration-tests"); } + @Override public T unwrap(Class iface) { return null; } + @Override public boolean isWrapperFor(Class iface) { return false; } + }; + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbMySqlApiContractConformanceTest.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbMySqlApiContractConformanceTest.java new file mode 100644 index 000000000..4b6755789 --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbMySqlApiContractConformanceTest.java @@ -0,0 +1,24 @@ +package com.metaobjects.integration.api; + +import com.metaobjects.integration.MySqlContainer; +import com.metaobjects.integration.JdbcTarget; +import com.metaobjects.integration.ScenarioEngine; +import org.junit.jupiter.api.DisplayName; + +/** The decimal api-contract corpus, GENERATED controllers over an OMDB repository on MySql (throwaway docker container; needs docker). */ +@DisplayName("API contract decimal - GENERATED Spring controllers over OMDB on MySql") +final class DecimalOmdbMySqlApiContractConformanceTest extends DecimalOmdbApiContractLane { + + private MySqlContainer database; + + @Override ScenarioEngine engine() { return ScenarioEngine.MYSQL; } + + @Override JdbcTarget openDatabase() { + database = new MySqlContainer(); + return database; + } + + @Override void closeDatabase() { + if (database != null) database.close(); + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbPostgresApiContractConformanceTest.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbPostgresApiContractConformanceTest.java new file mode 100644 index 000000000..e66b1efe3 --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbPostgresApiContractConformanceTest.java @@ -0,0 +1,24 @@ +package com.metaobjects.integration.api; + +import com.metaobjects.integration.PostgresContainer; +import com.metaobjects.integration.JdbcTarget; +import com.metaobjects.integration.ScenarioEngine; +import org.junit.jupiter.api.DisplayName; + +/** The decimal api-contract corpus, GENERATED controllers over an OMDB repository on Postgres (throwaway docker container; needs docker). */ +@DisplayName("API contract decimal - GENERATED Spring controllers over OMDB on Postgres") +final class DecimalOmdbPostgresApiContractConformanceTest extends DecimalOmdbApiContractLane { + + private PostgresContainer database; + + @Override ScenarioEngine engine() { return ScenarioEngine.POSTGRES; } + + @Override JdbcTarget openDatabase() { + database = new PostgresContainer(); + return database; + } + + @Override void closeDatabase() { + if (database != null) database.close(); + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbSqliteApiContractConformanceTest.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbSqliteApiContractConformanceTest.java new file mode 100644 index 000000000..8d2848b10 --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/DecimalOmdbSqliteApiContractConformanceTest.java @@ -0,0 +1,24 @@ +package com.metaobjects.integration.api; + +import com.metaobjects.integration.SqliteDatabase; +import com.metaobjects.integration.JdbcTarget; +import com.metaobjects.integration.ScenarioEngine; +import org.junit.jupiter.api.DisplayName; + +/** The decimal api-contract corpus, GENERATED controllers over an OMDB repository on SQLite (embedded, no docker). */ +@DisplayName("API contract decimal - GENERATED Spring controllers over OMDB on Sqlite") +final class DecimalOmdbSqliteApiContractConformanceTest extends DecimalOmdbApiContractLane { + + private SqliteDatabase database; + + @Override ScenarioEngine engine() { return ScenarioEngine.SQLITE; } + + @Override JdbcTarget openDatabase() { + database = new SqliteDatabase(); + return database; + } + + @Override void closeDatabase() { + if (database != null) database.close(); + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/JsonbApiContractConformanceTest.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/JsonbApiContractConformanceTest.java index 8b9ba1e42..43796d6ee 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/JsonbApiContractConformanceTest.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/JsonbApiContractConformanceTest.java @@ -67,7 +67,7 @@ void scenario(String name, ApiScenario scenario) { } HttpResponse res = client.send(builder.build(), HttpResponse.BodyHandlers.ofString()); Object parsed = (res.body() == null || res.body().isEmpty()) - ? null : MAPPER.readValue(res.body(), Object.class); + ? null : MAPPER.copy().enable(com.fasterxml.jackson.databind.DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS).readValue(res.body(), Object.class); ApiContractAssertions.assertResponse(scenario.name(), req, res.statusCode(), parsed); } } diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportCorpus.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportCorpus.java index 5e65f976d..d13aa53b7 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportCorpus.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportCorpus.java @@ -12,14 +12,12 @@ /** * Locates the FR-044 view-backed-report sub-corpus - * ({@code fixtures/api-contract-conformance/report/}) and loads its seed rows. + * ({@code fixtures/api-contract-conformance/report/}), its committed schema and its base-table seed. * - *

{@code seed.json} has two halves. Every top-level key but {@code reports} is a base - * table ({@code invoices}, {@code products}, {@code sales}), used by the full-stack lanes. - * This lane is a SEAM lane: its in-memory repository stands in for the view, so it is - * seeded from {@code reports} alone, which is what the served views return for those rows. - * No SQL for a report is produced or run here (ADR-0015: view SQL is TypeScript's). A - * TypeScript test holds the two halves together.

+ *

{@code seed.json} has two halves. Every top-level key but {@code reports} is a base table + * ({@code invoices}, {@code products}, {@code sales}); the OMDB lane inserts them into a real + * database that has executed {@code schema..sql}, and the views derive the report rows. + * No SQL for a report is produced here (ADR-0015: view SQL is TypeScript's).

*/ final class ReportCorpus { private ReportCorpus() {} @@ -32,7 +30,7 @@ private ReportCorpus() {} * references; {@code ProductRefunds} adds boolean {@code @agg any} measures. */ static final List SERVED = List.of("InvoiceStatusTotals", "InvoicesByMonth", "InvoiceTotals", "InvoiceCalendar", "InvoiceDisputes", - "ProductRevenue", "SalesByCategory", "SalesBySku", "ProductRefunds"); + "ProductRevenue", "SalesByCategory", "SalesBySku", "ProductRefunds", "TicketsByTags"); /** The report that declares no view and must generate nothing. */ static final String SOURCELESS = "InvoiceDays"; @@ -46,36 +44,36 @@ static Path scenariosDir() { } /** - * The seed's {@code reports} half, parsed ONCE: a per-report reader would re-read and - * re-parse the whole file once per served report. Sibling corpus classes in this package - * parse once into a static; this keeps the whole {@code reports} map as the index. + * The seed's base tables: every top-level key of {@code seed.json} but {@code reports}, in + * file order. The file lists them parents first, so inserting them in this order keeps every + * foreign key valid. {@code reports} is what the views return for those rows; this lane lets + * the real views derive them, so it never reads that half. */ - private static final Map>> REPORT_ROWS = readReportRows(); - @SuppressWarnings("unchecked") - private static Map>> readReportRows() { + static Map>> baseTables() { try { String text = Files.readString(root().resolve("seed.json"), StandardCharsets.UTF_8); - Map parsed = MAPPER.readValue(text, Map.class); - if (!(parsed.get("reports") instanceof Map reports)) - throw new IllegalStateException("report/seed.json: missing 'reports' object"); + Map parsed = MAPPER.readValue(text, new com.fasterxml.jackson.core.type.TypeReference>() {}); Map>> out = new java.util.LinkedHashMap<>(); - reports.forEach((name, rows) -> { + parsed.forEach((table, rows) -> { + if ("reports".equals(table)) return; if (!(rows instanceof List list)) - throw new IllegalStateException("report/seed.json: 'reports." + name + "' is not an array"); - out.put(String.valueOf(name), (List>) list); + throw new IllegalStateException("report/seed.json: '" + table + "' is not an array"); + out.put(table, (List>) list); }); + if (out.isEmpty()) throw new IllegalStateException("report/seed.json: no base tables"); return out; } catch (IOException e) { throw new UncheckedIOException(e); } } - /** The rows {@code report}'s view returns: {@code reports.} from {@code report/seed.json}. */ - static List> seedRows(String report) { - List> rows = REPORT_ROWS.get(report); - if (rows == null) - throw new IllegalStateException("report/seed.json: missing 'reports." + report + "' array"); - return rows; + /** The committed TypeScript-produced schema of {@code engine}, read verbatim. */ + static String schema(String engineId) { + try { + return Files.readString(root().resolve("schema." + engineId + ".sql"), StandardCharsets.UTF_8); + } catch (IOException e) { + throw new UncheckedIOException(e); + } } } diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportGeneratedApiContractConformanceTest.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportGeneratedApiContractConformanceTest.java deleted file mode 100644 index 140a1c809..000000000 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportGeneratedApiContractConformanceTest.java +++ /dev/null @@ -1,84 +0,0 @@ -package com.metaobjects.integration.api; - -import com.metaobjects.integration.api.ApiContractScenarios.ApiRequest; -import com.metaobjects.integration.api.ApiContractScenarios.ApiScenario; -import com.metaobjects.integration.api.generated.GeneratedReportControllerHarness; -import org.junit.jupiter.api.AfterAll; -import org.junit.jupiter.api.BeforeAll; -import org.junit.jupiter.api.DisplayName; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.params.ParameterizedTest; -import org.junit.jupiter.params.provider.Arguments; -import org.junit.jupiter.params.provider.MethodSource; - -import java.nio.file.Files; -import java.nio.file.Path; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Map; -import java.util.stream.Stream; - -import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; -import static org.junit.jupiter.api.Assertions.assertEquals; - -/** - * FR-044 Plan 3 — the Java GENERATED-controller lane for the view-backed-report - * api-contract sub-corpus. Boots the corpus's served reports' GENERATED Spring controllers - * on one embedded Tomcat, each behind an in-memory repository seam seeded with what its - * view returns, and drives every scenario of the corpus. (Counts move with the fixture; - * the pinned numbers live in {@link #theCorpusCarriesItsTwentyFiveScenarios()} and - * {@link ReportCorpus#SERVED}, not in prose.) - * - *

Generated lane ONLY, on purpose and on every port (see the sub-corpus README). What - * is under test is whether the port's GENERATOR emits a read route for a served report, - * and nothing for a sourceless one. A hand-rolled reference server would answer every - * scenario by construction.

- * - *

Run on-demand: - * {@code mvn -f server/java/integration-tests/pom.xml test -Dtest=ReportGeneratedApiContractConformanceTest}

- */ -@DisplayName("API contract report — GENERATED Spring controllers (codegen-spring) over embedded Tomcat") -final class ReportGeneratedApiContractConformanceTest { - - private static final List SCENARIOS = - ApiContractScenarioLoader.loadScenarios(ReportCorpus.scenariosDir()); - - private static GeneratedReportControllerHarness HARNESS; - - @BeforeAll - static void setUp() throws Exception { - Map>> seed = new LinkedHashMap<>(); - for (String report : ReportCorpus.SERVED) seed.put(report, ReportCorpus.seedRows(report)); - Path genDir = Files.createTempDirectory("report-generated-controllers"); - HARNESS = new GeneratedReportControllerHarness(ReportCorpus.root(), genDir, seed, ReportCorpus.SOURCELESS); - } - - @AfterAll - static void tearDown() throws Exception { - if (HARNESS != null) HARNESS.close(); - } - - static Stream scenarios() { - return SCENARIOS.stream().map(s -> Arguments.of(s.name(), s)); - } - - @Test - void theCorpusCarriesItsTwentyFiveScenarios() { - // A scenarios directory that resolved to nothing would leave this lane green and empty. - assertEquals(25, SCENARIOS.size()); - } - - @ParameterizedTest(name = "{0}") - @MethodSource("scenarios") - void scenario(String name, ApiScenario scenario) { - assertDoesNotThrow(() -> { - HARNESS.reset(); - for (ApiRequest req : scenario.requests()) { - GeneratedReportControllerHarness.Response res = - HARNESS.exchange(req.method(), req.path(), req.body()); - Object parsed = HARNESS.parseBody(res.body()); - ApiContractAssertions.assertResponse(scenario.name(), req, res.status(), parsed); - } - }); - } -} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbApiContractLane.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbApiContractLane.java new file mode 100644 index 000000000..a05790587 --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbApiContractLane.java @@ -0,0 +1,185 @@ +package com.metaobjects.integration.api; + +import com.metaobjects.integration.JdbcTarget; +import com.metaobjects.integration.QueryScenarioRunner; +import com.metaobjects.integration.ScenarioEngine; +import com.metaobjects.integration.api.ApiContractScenarios.ApiRequest; +import com.metaobjects.integration.api.ApiContractScenarios.ApiScenario; +import com.metaobjects.integration.api.generated.GeneratedReportControllerHarness; +import com.metaobjects.manager.db.ObjectManagerDB; +import com.metaobjects.object.value.ValueObject; +import com.metaobjects.registry.ObjectClassRegistry; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.TestInstance; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; + +import javax.sql.DataSource; +import java.io.PrintWriter; +import java.nio.file.Files; +import java.nio.file.Path; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.SQLException; +import java.sql.Statement; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.logging.Logger; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; +import static org.junit.jupiter.api.Assertions.assertEquals; + +/** + * FR-044 — the Java full-stack lane of the view-backed-report api-contract sub-corpus, on a real + * database. One subclass per engine. The artifacts under test are the GENERATED + * {@code Controller} / DTO / allowlist / {@code Repository} interface, mounted + * unmodified on embedded Tomcat; the adopter-owned repository seam is an OMDB-backed + * implementation ({@code OmdbReportRepositorySource}) that reads each report's view through + * {@code ObjectManagerDB} and the report read model. + * + *

The database executes the committed TypeScript-produced {@code report/schema..sql} + * verbatim (no port writes view SQL, ADR-0015), the base tables are seeded from the base-table + * half of {@code report/seed.json}, and the real views derive the rows, exactly as the + * TypeScript and C# lanes do. Every scenario is a read (or a rejected write), so one database + * and one server serve the whole corpus.

+ * + *

Generated lane ONLY, on purpose and on every port (see the sub-corpus README): what is + * under test is whether the port's GENERATOR emits a read route for a served report and nothing + * for a sourceless one. The default page size stays 50 in Java; the scenarios that page pass + * {@code limit} explicitly.

+ */ +@TestInstance(TestInstance.Lifecycle.PER_CLASS) +abstract class ReportOmdbApiContractLane { + + private static final List SCENARIOS = + ApiContractScenarioLoader.loadScenarios(ReportCorpus.scenariosDir()); + + private GeneratedReportControllerHarness harness; + private JdbcTarget target; + + /** The engine this subclass runs on. */ + abstract ScenarioEngine engine(); + + /** Start (or open) the engine's throwaway database. */ + abstract JdbcTarget openDatabase(); + + /** Release what {@link #openDatabase()} started. */ + abstract void closeDatabase(); + + @BeforeAll + void setUp() throws Exception { + target = openDatabase(); + provision(engine(), target); + + Path genDir = Files.createTempDirectory("report-omdb-controllers"); + harness = new GeneratedReportControllerHarness( + ReportCorpus.root(), genDir, ReportCorpus.SERVED, ReportCorpus.SOURCELESS); + + // The report read model's rows are generic value objects; bind every entity of the model. + ObjectClassRegistry reg = new ObjectClassRegistry(); + Map> bindings = new HashMap<>(); + for (String entity : List.of("Invoice", "Category", "Product", "Sale", "Ticket")) { + bindings.put("acme::sales::" + entity, ValueObject.class); + } + reg.register(() -> bindings); + ObjectClassRegistry.setGlobal(reg); + + ObjectManagerDB omdb = new ObjectManagerDB(); + omdb.setDatabaseDriver(engine().newDriver()); + omdb.setDataSource(dataSource(target)); + omdb.init(); + harness.start(omdb); + } + + @AfterAll + void tearDown() throws Exception { + try { + if (harness != null) harness.close(); + } finally { + closeDatabase(); + } + } + + static Stream scenarios() { + return SCENARIOS.stream().map(s -> Arguments.of(s.name(), s)); + } + + @Test + void theCorpusCarriesItsTwentySevenScenarios() { + // A scenarios directory that resolved to nothing would leave this lane green and empty. + assertEquals(27, SCENARIOS.size()); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("scenarios") + void scenario(String name, ApiScenario scenario) { + assertDoesNotThrow(() -> { + for (ApiRequest req : scenario.requests()) { + GeneratedReportControllerHarness.Response res = + harness.exchange(req.method(), req.path(), req.body()); + Object parsed = harness.parseBody(res.body()); + ApiContractAssertions.assertResponse(scenario.name(), req, res.status(), parsed); + } + }); + } + + // ----------------------------------------------------------------------- + // database provisioning + // ----------------------------------------------------------------------- + + /** Execute the engine's schema artifact verbatim, then insert the base-table seed, parents first. */ + private static void provision(ScenarioEngine engine, JdbcTarget target) throws SQLException { + String schema = ReportCorpus.schema(engine.schemaArtifact().replace("canonical/schema.", "").replace(".sql", "")); + try (Connection c = DriverManager.getConnection(target.jdbcUrl(), target.username(), target.password())) { + if (engine == ScenarioEngine.POSTGRES) { + try (Statement s = c.createStatement()) { s.execute(schema); } + } else { + List statements = ScenarioEngine.splitStatements(schema); + // A MySQL server may be an existing shared database: start from an empty schema. + if (engine == ScenarioEngine.MYSQL) QueryScenarioRunner.dropSchema(c, statements); + try (Statement s = c.createStatement()) { + for (String stmt : statements) s.execute(stmt); + } + } + String q = engine.quote(); + try (Statement s = c.createStatement()) { + for (Map.Entry>> table : ReportCorpus.baseTables().entrySet()) { + for (Map row : table.getValue()) { + String cols = row.keySet().stream().map(k -> q + k + q).collect(Collectors.joining(", ")); + String vals = row.values().stream().map(ReportOmdbApiContractLane::literal).collect(Collectors.joining(", ")); + s.execute("INSERT INTO " + q + table.getKey() + q + " (" + cols + ") VALUES (" + vals + ")"); + } + } + } + } + } + + /** A seed value as a SQL literal; the fixture is trusted, and an untyped literal binds to any column type. */ + private static String literal(Object v) { + if (v == null) return "NULL"; + if (v instanceof Boolean || v instanceof Number) return v.toString(); + return "'" + v.toString().replace("'", "''") + "'"; + } + + private static DataSource dataSource(JdbcTarget target) { + return new DataSource() { + @Override public Connection getConnection() throws SQLException { + return DriverManager.getConnection(target.jdbcUrl(), target.username(), target.password()); + } + @Override public Connection getConnection(String u, String p) throws SQLException { return getConnection(); } + @Override public PrintWriter getLogWriter() { return null; } + @Override public void setLogWriter(PrintWriter out) {} + @Override public void setLoginTimeout(int seconds) {} + @Override public int getLoginTimeout() { return 0; } + @Override public Logger getParentLogger() { return Logger.getLogger("integration-tests"); } + @Override public T unwrap(Class iface) { return null; } + @Override public boolean isWrapperFor(Class iface) { return false; } + }; + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbMySqlApiContractConformanceTest.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbMySqlApiContractConformanceTest.java new file mode 100644 index 000000000..3de7fbcd5 --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbMySqlApiContractConformanceTest.java @@ -0,0 +1,24 @@ +package com.metaobjects.integration.api; + +import com.metaobjects.integration.MySqlContainer; +import com.metaobjects.integration.JdbcTarget; +import com.metaobjects.integration.ScenarioEngine; +import org.junit.jupiter.api.DisplayName; + +/** The report api-contract corpus, GENERATED controllers over an OMDB repository on MySql (throwaway docker container; needs docker). */ +@DisplayName("API contract report - GENERATED Spring controllers over OMDB on MySql") +final class ReportOmdbMySqlApiContractConformanceTest extends ReportOmdbApiContractLane { + + private MySqlContainer database; + + @Override ScenarioEngine engine() { return ScenarioEngine.MYSQL; } + + @Override JdbcTarget openDatabase() { + database = new MySqlContainer(); + return database; + } + + @Override void closeDatabase() { + if (database != null) database.close(); + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbPostgresApiContractConformanceTest.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbPostgresApiContractConformanceTest.java new file mode 100644 index 000000000..4ccd3f990 --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbPostgresApiContractConformanceTest.java @@ -0,0 +1,24 @@ +package com.metaobjects.integration.api; + +import com.metaobjects.integration.PostgresContainer; +import com.metaobjects.integration.JdbcTarget; +import com.metaobjects.integration.ScenarioEngine; +import org.junit.jupiter.api.DisplayName; + +/** The report api-contract corpus, GENERATED controllers over an OMDB repository on Postgres (throwaway docker container; needs docker). */ +@DisplayName("API contract report - GENERATED Spring controllers over OMDB on Postgres") +final class ReportOmdbPostgresApiContractConformanceTest extends ReportOmdbApiContractLane { + + private PostgresContainer database; + + @Override ScenarioEngine engine() { return ScenarioEngine.POSTGRES; } + + @Override JdbcTarget openDatabase() { + database = new PostgresContainer(); + return database; + } + + @Override void closeDatabase() { + if (database != null) database.close(); + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbSqliteApiContractConformanceTest.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbSqliteApiContractConformanceTest.java new file mode 100644 index 000000000..6942154fb --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/ReportOmdbSqliteApiContractConformanceTest.java @@ -0,0 +1,24 @@ +package com.metaobjects.integration.api; + +import com.metaobjects.integration.SqliteDatabase; +import com.metaobjects.integration.JdbcTarget; +import com.metaobjects.integration.ScenarioEngine; +import org.junit.jupiter.api.DisplayName; + +/** The report api-contract corpus, GENERATED controllers over an OMDB repository on SQLite (embedded, no docker). */ +@DisplayName("API contract report - GENERATED Spring controllers over OMDB on Sqlite") +final class ReportOmdbSqliteApiContractConformanceTest extends ReportOmdbApiContractLane { + + private SqliteDatabase database; + + @Override ScenarioEngine engine() { return ScenarioEngine.SQLITE; } + + @Override JdbcTarget openDatabase() { + database = new SqliteDatabase(); + return database; + } + + @Override void closeDatabase() { + if (database != null) database.close(); + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/TomcatHost.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/TomcatHost.java index b1fbcb3a6..3a6ab15c2 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/TomcatHost.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/TomcatHost.java @@ -109,7 +109,11 @@ public Response exchange(String method, String path, String jsonBody) throws IOE public static Object parseBody(ObjectMapper mapper, String body) { if (body == null || body.isEmpty()) return null; try { - return mapper.readValue(body, Object.class); + // A response body is read with exact decimals: a JSON number such as 0.3 must not + // become a double before the assertion engine sees it (ADR-0019 Amendment 1). + return mapper.copy() + .enable(com.fasterxml.jackson.databind.DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS) + .readValue(body, Object.class); } catch (com.fasterxml.jackson.core.JsonProcessingException notJson) { return body; } diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/GeneratedDecimalControllerHarness.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/GeneratedDecimalControllerHarness.java new file mode 100644 index 000000000..dfa474a14 --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/GeneratedDecimalControllerHarness.java @@ -0,0 +1,194 @@ +package com.metaobjects.integration.api.generated; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.SerializationFeature; +import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule; +import com.metaobjects.generator.spring.SpringControllerGenerator; +import com.metaobjects.generator.spring.SpringDtoGenerator; +import com.metaobjects.generator.spring.SpringFilterAllowlistGenerator; +import com.metaobjects.generator.spring.SpringRepositoryGenerator; +import com.metaobjects.generator.spring.SpringValueObjectGenerator; +import com.metaobjects.integration.api.TomcatHost; +import com.metaobjects.loader.LoaderOptions; +import com.metaobjects.loader.MetaDataLoader; +import com.metaobjects.loader.uri.URIHelper; +import com.metaobjects.manager.db.ObjectManagerDB; +import com.metaobjects.object.MetaObject; +import jakarta.validation.Validation; +import jakarta.validation.Validator; +import java.io.File; +import java.lang.reflect.Constructor; +import java.net.URI; +import java.net.URL; +import java.net.URLClassLoader; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; +import java.util.stream.Stream; +import javax.tools.DiagnosticCollector; +import javax.tools.JavaCompiler; +import javax.tools.JavaFileObject; +import javax.tools.ToolProvider; + +/** + * ADR-0019 Amendment 1: hosts the GENERATED Spring controllers of the decimal api-contract + * sub-corpus over real HTTP (embedded Tomcat) and over a real database through OMDB: the writable + * {@code Entry} entity ({@code /api/entries}) and the served {@code EntryTotals} report + * ({@code /api/entry_totals}). The controllers, DTOs, repository interfaces and filter allowlists are + * generated UNMODIFIED; the adopter-owned seams are {@link OmdbEntryRepositorySource} and + * {@link OmdbReportRepositorySource}. Same shape as {@link GeneratedReportControllerHarness}. + */ +public final class GeneratedDecimalControllerHarness implements AutoCloseable { + + private static final String PKG = OmdbEntryRepositorySource.PKG; + + private final ObjectMapper mapper = new ObjectMapper() + .registerModule(new JavaTimeModule()) + .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS); + private final Validator validator = Validation.buildDefaultValidatorFactory().getValidator(); + private final MetaDataLoader loader; + private final URLClassLoader classLoader; + private final Constructor entryRepoCtor; + private final Constructor entryControllerCtor; + private final Constructor totalsRepoCtor; + private final Constructor totalsControllerCtor; + + private TomcatHost host; + + public GeneratedDecimalControllerHarness(Path corpusRoot, Path genDir) throws Exception { + Path srcDir = genDir.resolve("src"); + Path classesDir = genDir.resolve("classes"); + Files.createDirectories(srcDir); + Files.createDirectories(classesDir); + + this.loader = loadCorpus(corpusRoot.resolve("meta.json")); + // ADR-0058: the committed config at the api-contract corpus root declares naming.columns: literal. + loader.getRoot().setPhysicalNaming( + com.metaobjects.config.SourceResolver.resolvePhysicalNaming(corpusRoot.getParent())); + + runGenerator(new SpringControllerGenerator(), loader, srcDir); + runGenerator(new SpringDtoGenerator(), loader, srcDir); + runGenerator(new SpringRepositoryGenerator(), loader, srcDir); + runGenerator(new SpringFilterAllowlistGenerator(), loader, srcDir); + + Path pkgDir = srcDir.resolve(PKG.replace('.', '/')); + for (String generated : List.of("EntryController", "EntryTotalsController")) { + if (!Files.exists(pkgDir.resolve(generated + ".java"))) { + throw new IllegalStateException("codegen did not emit " + generated + " under " + pkgDir); + } + } + Files.writeString(pkgDir.resolve("OmdbEntryRepository.java"), OmdbEntryRepositorySource.SOURCE); + Files.writeString(pkgDir.resolve(OmdbReportRepositorySource.simpleName("EntryTotals") + ".java"), + OmdbReportRepositorySource.source(PKG, "EntryTotals")); + + compile(srcDir, classesDir); + + this.classLoader = new URLClassLoader(new URL[]{ classesDir.toUri().toURL() }, getClass().getClassLoader()); + Class entryRepo = classLoader.loadClass(PKG + ".EntryRepository"); + this.entryControllerCtor = classLoader.loadClass(PKG + ".EntryController") + .getDeclaredConstructor(entryRepo, ObjectMapper.class, Validator.class); + this.entryRepoCtor = classLoader.loadClass(OmdbEntryRepositorySource.FQCN) + .getDeclaredConstructor(ObjectManagerDB.class, MetaObject.class); + Class totalsRepo = classLoader.loadClass(PKG + ".EntryTotalsRepository"); + this.totalsControllerCtor = classLoader.loadClass(PKG + ".EntryTotalsController") + .getDeclaredConstructor(totalsRepo); + this.totalsRepoCtor = classLoader.loadClass(OmdbReportRepositorySource.fqcn(PKG, "EntryTotals")) + .getDeclaredConstructor(ObjectManagerDB.class, MetaObject.class); + } + + /** Mount both generated controllers, each over an OMDB-backed repository on {@code omdb}. */ + public void start(ObjectManagerDB omdb) throws Exception { + MetaObject entry = find("Entry"); + MetaObject totals = find("EntryTotals"); + Object entryController = entryControllerCtor.newInstance(entryRepoCtor.newInstance(omdb, entry), mapper, validator); + Object totalsController = totalsControllerCtor.newInstance(totalsRepoCtor.newInstance(omdb, totals)); + if (host != null) host.close(); + this.host = TomcatHost.start(mapper, entryController, totalsController); + } + + private MetaObject find(String name) { + return loader.getMetaObjects().stream().filter(o -> o.getShortName().equals(name)).findFirst() + .orElseThrow(() -> new IllegalStateException("no " + name + " in the model")); + } + + public MetaDataLoader loader() { + return loader; + } + + public Response exchange(String method, String path, Object jsonBody) throws Exception { + TomcatHost.Response res = host.exchange(method, path, jsonBody == null ? null : mapper.writeValueAsString(jsonBody)); + return new Response(res.status(), res.body()); + } + + public Object parseBody(String body) { + return TomcatHost.parseBody(mapper, body); + } + + @Override + public void close() throws Exception { + if (host != null) host.close(); + classLoader.close(); + } + + public record Response(int status, String body) {} + + private static MetaDataLoader loadCorpus(Path metaJson) { + URI uri = URIHelper.toURI( + "model:file:" + metaJson.toAbsolutePath().toString().replace('\\', '/')); + MetaDataLoader loader = new MetaDataLoader( + LoaderOptions.create(false, false, true), + MetaDataLoader.SUBTYPE_MANUAL, + "api-contract-decimal-generated"); + loader.setSourceURIs(List.of(uri)); + loader.init(); + return loader; + } + + private static void runGenerator(Object generator, MetaDataLoader loader, Path outDir) { + Map args = new HashMap<>(); + args.put("outputDir", outDir.toString()); + ((com.metaobjects.generator.direct.MultiFileDirectGeneratorBase) generator).setArgs(args); + ((com.metaobjects.generator.direct.MultiFileDirectGeneratorBase) generator).execute(loader); + } + + private static void compile(Path srcDir, Path classesDir) throws Exception { + List sources; + try (Stream s = Files.walk(srcDir)) { + sources = s.filter(p -> p.toString().endsWith(".java")) + .map(Path::toFile) + .collect(Collectors.toList()); + } + if (sources.isEmpty()) { + throw new IllegalStateException("no generated .java sources under " + srcDir); + } + + JavaCompiler javac = ToolProvider.getSystemJavaCompiler(); + if (javac == null) { + throw new IllegalStateException( + "JDK (not JRE) required — ToolProvider.getSystemJavaCompiler() returned null"); + } + String cp = System.getProperty("java.class.path"); + DiagnosticCollector diags = new DiagnosticCollector<>(); + var fm = javac.getStandardFileManager(diags, null, StandardCharsets.UTF_8); + List opts = List.of("-classpath", cp, "-d", classesDir.toString(), "-parameters"); + + boolean ok = javac.getTask(null, fm, diags, opts, null, + fm.getJavaFileObjectsFromFiles(sources)).call(); + if (!ok) { + StringBuilder sb = new StringBuilder("generated sources failed to compile:\n"); + for (var d : diags.getDiagnostics()) { + sb.append(" ").append(d.getKind()).append(": ").append(d.getMessage(null)).append('\n'); + if (d.getSource() != null) { + sb.append(" at ").append(d.getSource().getName()) + .append(':').append(d.getLineNumber()).append('\n'); + } + } + throw new IllegalStateException(sb.toString()); + } + } +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/GeneratedReportControllerHarness.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/GeneratedReportControllerHarness.java index b8d27b5b7..658bcad59 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/GeneratedReportControllerHarness.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/GeneratedReportControllerHarness.java @@ -11,6 +11,8 @@ import com.metaobjects.loader.LoaderOptions; import com.metaobjects.loader.MetaDataLoader; import com.metaobjects.loader.uri.URIHelper; +import com.metaobjects.manager.db.ObjectManagerDB; +import com.metaobjects.object.MetaObject; import javax.tools.DiagnosticCollector; import javax.tools.JavaCompiler; @@ -42,9 +44,9 @@ *

The artifacts under test are the GENERATED {@code Controller} (the list route * and a 405 on the collection {@code POST}, no item route), the {@code Dto} row, * the {@code FilterAllowlist} and the read-only {@code Repository} - * interface. The only hand-written piece is {@link InMemoryReportRepositorySource}, the - * consumer seam, which stands in for the report's view and is seeded with what that view - * returns. No SQL for a report is produced or run here (ADR-0015).

+ * interface. The only hand-written piece is {@link OmdbReportRepositorySource}, the consumer + * seam: an OMDB-backed repository over the real database the caller provisioned from the + * TypeScript-produced schema artifact. No SQL for a report is produced here (ADR-0015).

* *

The WHOLE model is generated and compiled, the writable {@code Invoice} entity * included, because the interesting failure is a gate that admits one shape and breaks @@ -55,8 +57,7 @@ public final class GeneratedReportControllerHarness implements AutoCloseable { private static final String ENTITY_PKG = "acme.sales"; /** A served report's generated row type and the constructors a scenario rebuilds it with. */ - private record Mount(Class dtoClass, Constructor repoCtor, Constructor controllerCtor, - List> seedRows) {} + private record Mount(String report, Constructor repoCtor, Constructor controllerCtor) {} /** * {@code JavaTimeModule}, with dates as ISO strings: a time dimension at a grain is a @@ -68,36 +69,32 @@ private record Mount(Class dtoClass, Constructor repoCtor, Constructor .registerModule(new JavaTimeModule()) .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS); + private final MetaDataLoader loader; private final URLClassLoader classLoader; private final Map mounts = new LinkedHashMap<>(); private TomcatHost host; /** - * @param seedByReport the rows each served report's view returns, keyed by report name - * @param sourceless a report in the model that declares no view and must generate nothing + * @param served the served reports (each must have been generated), by report name + * @param sourceless a report in the model that declares no view and must generate nothing */ - public GeneratedReportControllerHarness(Path corpusRoot, Path genDir, - Map>> seedByReport, + public GeneratedReportControllerHarness(Path corpusRoot, Path genDir, List served, String sourceless) throws Exception { Path srcDir = genDir.resolve("src"); Path classesDir = genDir.resolve("classes"); Files.createDirectories(srcDir); Files.createDirectories(classesDir); - MetaDataLoader loader = loadCorpus(corpusRoot.resolve("meta.json")); - - // ADR-0058 — this corpus's committed .metaobjects/config.json (at corpusRoot, - - // fixtures/api-contract-conformance/) declares naming.columns: literal; every field - - // here declares no @column, so the generated columns and the runtime's reads of them - - // depend on this. Read with the SAME NeutralConfig reader every port uses. + this.loader = loadCorpus(corpusRoot.resolve("meta.json")); + // ADR-0058 - this corpus's committed .metaobjects/config.json sits at the api-contract + // corpus root (corpusRoot's parent, not under report/) and declares naming.columns: + // literal; the committed schema was created under it, and every field here declares no + // @column, so the generated columns and OMDB's reads of them depend on this. Read with + // the SAME NeutralConfig reader every port uses. loader.getRoot().setPhysicalNaming( - - com.metaobjects.config.SourceResolver.resolvePhysicalNaming(corpusRoot)); + com.metaobjects.config.SourceResolver.resolvePhysicalNaming(corpusRoot.getParent())); runGenerator(new SpringControllerGenerator(), loader, srcDir); runGenerator(new SpringDtoGenerator(), loader, srcDir); @@ -109,7 +106,7 @@ public GeneratedReportControllerHarness(Path corpusRoot, Path genDir, // Every served report's controller must have been emitted at all: a silently // skipped generator would otherwise surface as a ClassNotFoundException with no // hint that codegen, not the harness, was the cause. - for (String report : seedByReport.keySet()) { + for (String report : served) { Path emitted = pkgDir.resolve(report + "Controller.java"); if (!Files.exists(emitted)) { throw new IllegalStateException( @@ -130,37 +127,42 @@ public GeneratedReportControllerHarness(Path corpusRoot, Path genDir, } } - for (String report : seedByReport.keySet()) { - Files.writeString(pkgDir.resolve(InMemoryReportRepositorySource.simpleName(report) + ".java"), - InMemoryReportRepositorySource.source(report)); + for (String report : served) { + Files.writeString(pkgDir.resolve(OmdbReportRepositorySource.simpleName(report) + ".java"), + OmdbReportRepositorySource.source(report)); } compile(srcDir, classesDir); this.classLoader = new URLClassLoader( new URL[]{ classesDir.toUri().toURL() }, getClass().getClassLoader()); - for (Map.Entry>> e : seedByReport.entrySet()) { - String report = e.getKey(); - Class dtoClass = classLoader.loadClass(ENTITY_PKG + "." + report + "Dto"); + for (String report : served) { Class repoInterface = classLoader.loadClass(ENTITY_PKG + "." + report + "Repository"); // (Repository) — no ObjectMapper, no Validator: nothing here binds a body. Constructor controllerCtor = classLoader.loadClass(ENTITY_PKG + "." + report + "Controller").getDeclaredConstructor(repoInterface); - Constructor repoCtor = - classLoader.loadClass(InMemoryReportRepositorySource.fqcn(report)).getDeclaredConstructor(List.class); - mounts.put(report, new Mount(dtoClass, repoCtor, controllerCtor, e.getValue())); + Constructor repoCtor = classLoader.loadClass(OmdbReportRepositorySource.fqcn(report)) + .getDeclaredConstructor(ObjectManagerDB.class, MetaObject.class); + mounts.put(report, new Mount(report, repoCtor, controllerCtor)); } } - /** Re-seed for a scenario: fresh repositories and controllers, all on one Tomcat. */ - public void reset() throws Exception { + /** The loaded model, so the caller can read the same physical naming the generated code was built under. */ + public MetaDataLoader loader() { + return loader; + } + + /** + * Mount every served report's generated controller, each over an OMDB-backed repository on + * {@code omdb}, on one Tomcat. Replaces a previous mount. + */ + public void start(ObjectManagerDB omdb) throws Exception { List controllers = new ArrayList<>(); for (Mount m : mounts.values()) { - List dtos = new ArrayList<>(); - // The seed row becomes the generated row type: "2026-04-01" a LocalDate, "0.4" - // a BigDecimal (from the string, no double in between), null a null component. - for (Map row : m.seedRows()) dtos.add(mapper.convertValue(row, m.dtoClass())); - controllers.add(m.controllerCtor().newInstance(m.repoCtor().newInstance(dtos))); + MetaObject report = loader.getMetaObjects().stream() + .filter(o -> o.getShortName().equals(m.report())).findFirst() + .orElseThrow(() -> new IllegalStateException("no report " + m.report() + " in the model")); + controllers.add(m.controllerCtor().newInstance(m.repoCtor().newInstance(omdb, report))); } if (host != null) host.close(); this.host = TomcatHost.start(mapper, controllers.toArray()); diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/InMemoryProjectionRepositorySource.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/InMemoryProjectionRepositorySource.java index a366a03d0..7c2e5f5a1 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/InMemoryProjectionRepositorySource.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/InMemoryProjectionRepositorySource.java @@ -120,7 +120,10 @@ private static Object column(__NAME__Dto r, String field) { } /** A raw query operand as the column's own type, so the comparison is typed. */ - private static Comparable operand(String field, String raw) { + private static Comparable operand(String field, Object given) { + // ADR-0019 Amendment 1: FilterParser hands a decimal field's operand over as a BigDecimal. + if (given instanceof BigDecimal exact) return comparable(exact); + String raw = (String) given; Class type = component(field).getType(); Object value; if (type == Long.class) value = Long.valueOf(raw); @@ -158,7 +161,7 @@ private static boolean matches(__NAME__Dto r, FilterPredicate p) { } case "in": { if (col == null) return false; - for (String item : (List) p.value()) { + for (Object item : (List) p.value()) { if (comparable(col).compareTo(operand(p.field(), item)) == 0) return true; } return false; @@ -170,7 +173,7 @@ private static boolean matches(__NAME__Dto r, FilterPredicate p) { default: { if (col == null) return false; // compareTo, never equals: BigDecimal.equals is scale-sensitive. - int cmp = comparable(col).compareTo(operand(p.field(), (String) p.value())); + int cmp = comparable(col).compareTo(operand(p.field(), p.value())); return switch (p.op()) { case "eq" -> cmp == 0; case "ne" -> cmp != 0; diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/InMemoryReportRepositorySource.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/InMemoryReportRepositorySource.java deleted file mode 100644 index 6b28c99a5..000000000 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/InMemoryReportRepositorySource.java +++ /dev/null @@ -1,200 +0,0 @@ -package com.metaobjects.integration.api.generated; - -/** - * The Java SOURCE for an in-memory {@code acme.sales.Repository} impl: ONE generic - * template, instantiated once per served report by substituting the report's name. Emitted - * alongside the GENERATED controller / DTO / interface so it compiles against them, then - * loaded and instantiated reflectively by {@link GeneratedReportControllerHarness}. - * - *

The consumer seam MetaObjects leaves unimplemented, and for a report it is - * {@code list} and {@code count} only. If the generator ever emitted {@code findById} or a - * write method on a report's repository interface, this class would stop compiling.

- * - *

It stands in for the report's SQL view: the seeded rows are what that view returns. - * Test scaffolding, not a conformance subject. Its job is to apply the controller-supplied - * {@code List}, {@code SortClause} and {@code limit/offset} faithfully, so - * the GENERATED controller's qs to predicate to repository translation is exercised end to - * end. Envelopes and status codes are the generated controller's job.

- * - *

Generic over the row: a column is read through the DTO record's own component - * accessor and an operand is coerced by that component's declared type, so nothing here - * names a report's fields. A decimal compares as {@link java.math.BigDecimal} (exact, and - * scale-blind: {@code 0.4} equals {@code 0.40}), never through a double.

- */ -final class InMemoryReportRepositorySource { - - private InMemoryReportRepositorySource() {} - - private static final String PKG = "acme.sales"; - private static final String NAME = "__REPORT__"; - - /** Simple name of the emitted impl for {@code report}. */ - static String simpleName(String report) { - return "InMemory" + report + "Repository"; - } - - /** Fully-qualified name of the emitted impl for {@code report}. */ - static String fqcn(String report) { - return PKG + "." + simpleName(report); - } - - /** The impl source for {@code report}. */ - static String source(String report) { - return TEMPLATE.replace(NAME, report); - } - - private static final String TEMPLATE = """ - package acme.sales; - - import com.metaobjects.generator.spring.runtime.FilterPredicate; - - import java.lang.reflect.RecordComponent; - import java.math.BigDecimal; - import java.time.LocalDate; - import java.util.ArrayList; - import java.util.Comparator; - import java.util.List; - - /** - * Hand-written in-memory {@link __REPORT__Repository} (the read-only consumer seam). - * Stands in for the report's SQL view. NOT a conformance subject: test scaffolding. - */ - public final class InMemory__REPORT__Repository implements __REPORT__Repository { - - private final List<__REPORT__Dto> rows = new ArrayList<>(); - - public InMemory__REPORT__Repository(List<__REPORT__Dto> seed) { - rows.addAll(seed); - } - - @Override - public List<__REPORT__Dto> list(int limit, int offset, SortClause sort, List filters) { - List<__REPORT__Dto> out = new ArrayList<>(); - for (__REPORT__Dto r : rows) if (matchesAll(r, filters)) out.add(r); - // No sort: the view's own order, which is the seed's. - if (sort != null) out.sort(comparatorFor(sort)); - int from = Math.min(offset, out.size()); - int to = Math.min(from + limit, out.size()); - return new ArrayList<>(out.subList(from, to)); - } - - @Override - public long count(List filters) { - long n = 0; - for (__REPORT__Dto r : rows) if (matchesAll(r, filters)) n++; - return n; - } - - // --- the row, read generically ------------------------------------------------ - - private static RecordComponent component(String field) { - for (RecordComponent c : __REPORT__Dto.class.getRecordComponents()) { - if (c.getName().equals(field)) return c; - } - throw new IllegalStateException("unknown column: " + field); - } - - private static Object column(__REPORT__Dto r, String field) { - try { - return component(field).getAccessor().invoke(r); - } catch (ReflectiveOperationException e) { - throw new IllegalStateException("cannot read column " + field, e); - } - } - - /** A raw query operand as the column's own type, so the comparison is typed. */ - private static Comparable operand(String field, String raw) { - Class type = component(field).getType(); - Object value; - if (type == Long.class) value = Long.valueOf(raw); - else if (type == Integer.class) value = Integer.valueOf(raw); - else if (type == Double.class) value = Double.valueOf(raw); - else if (type == Float.class) value = Float.valueOf(raw); - else if (type == BigDecimal.class) value = new BigDecimal(raw); - else if (type == LocalDate.class) value = LocalDate.parse(raw); - else if (type == Boolean.class) value = Boolean.valueOf(raw); - else if (type == java.util.UUID.class) value = java.util.UUID.fromString(raw); - else if (type == String.class) value = raw; - else if (type.isEnum()) value = enumConstant(type, raw); - else throw new IllegalStateException("no operand coercion for " + type.getName() + " (" + field + ")"); - return comparable(value); - } - - /** A derived enum column's operand: the declared member of that name. */ - @SuppressWarnings({"unchecked", "rawtypes"}) - private static Object enumConstant(Class type, String raw) { - return Enum.valueOf((Class) type, raw); - } - - @SuppressWarnings("unchecked") - private static Comparable comparable(Object value) { - return (Comparable) value; - } - - // --- predicate application --------------------------------------------------- - - private static boolean matchesAll(__REPORT__Dto r, List filters) { - if (filters == null) return true; - for (FilterPredicate p : filters) if (!matches(r, p)) return false; // implicit AND - return true; - } - - @SuppressWarnings("unchecked") - private static boolean matches(__REPORT__Dto r, FilterPredicate p) { - Object col = column(r, p.field()); - switch (p.op()) { - case "isNull": { - boolean wantNull = Boolean.TRUE.equals(p.value()); - return wantNull == (col == null); - } - case "in": { - if (col == null) return false; - for (String item : (List) p.value()) { - if (comparable(col).compareTo(operand(p.field(), item)) == 0) return true; - } - return false; - } - case "like": { - if (col == null) return false; - return sqlLike(String.valueOf(col), (String) p.value()); - } - default: { - if (col == null) return false; - // compareTo, never equals: BigDecimal.equals is scale-sensitive. - int cmp = comparable(col).compareTo(operand(p.field(), (String) p.value())); - return switch (p.op()) { - case "eq" -> cmp == 0; - case "ne" -> cmp != 0; - case "gt" -> cmp > 0; - case "gte" -> cmp >= 0; - case "lt" -> cmp < 0; - case "lte" -> cmp <= 0; - default -> throw new IllegalStateException("unknown op: " + p.op()); - }; - } - } - } - - /** SQL LIKE with `%` (any run) and `_` (any single char), anchored full-string match. */ - private static boolean sqlLike(String value, String pattern) { - StringBuilder re = new StringBuilder("^"); - for (int i = 0; i < pattern.length(); i++) { - char c = pattern.charAt(i); - if (c == '%') re.append(".*"); - else if (c == '_') re.append('.'); - else re.append(java.util.regex.Pattern.quote(String.valueOf(c))); - } - re.append("$"); - return value.matches(re.toString()); - } - - private static Comparator<__REPORT__Dto> comparatorFor(SortClause sort) { - String field = sort.field(); - component(field); // an unknown sort column is a harness bug, not an empty sort - Comparator<__REPORT__Dto> c = Comparator.comparing( - r -> comparable(column(r, field)), Comparator.nullsLast(Comparator.naturalOrder())); - return "desc".equalsIgnoreCase(sort.direction()) ? c.reversed() : c; - } - } - """; -} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/OmdbEntryRepositorySource.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/OmdbEntryRepositorySource.java new file mode 100644 index 000000000..b6c812483 --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/OmdbEntryRepositorySource.java @@ -0,0 +1,215 @@ +package com.metaobjects.integration.api.generated; + +/** + * The Java SOURCE for an OMDB-backed {@code acme.ledger.EntryRepository}: the adopter-owned seam of + * the decimal api-contract sub-corpus' writable entity, emitted beside the GENERATED controller, DTO + * and interface so it compiles against them (see {@link GeneratedDecimalControllerHarness}). + * + *

It is written the way an adopter writes it: the controller's {@code FilterPredicate} list, + * {@code SortClause} and paging become an OMDB {@code Expression}, {@code SortOrder} and + * {@code Range}; a write authors a {@code ValueObject} and goes through {@code createObject} / + * {@code updateObject} / {@code deleteObject}; a decimal is a {@link java.math.BigDecimal} in both + * directions, so nothing here converts through a double. No SQL is written here.

+ */ +final class OmdbEntryRepositorySource { + + private OmdbEntryRepositorySource() {} + + static final String PKG = "acme.ledger"; + static final String FQCN = PKG + ".OmdbEntryRepository"; + + static final String SOURCE = """ + package acme.ledger; + + import com.fasterxml.jackson.databind.ObjectMapper; + import com.fasterxml.jackson.databind.SerializationFeature; + import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule; + import com.metaobjects.field.MetaField; + import com.metaobjects.generator.spring.runtime.FilterPredicate; + import com.metaobjects.manager.ObjectConnection; + import com.metaobjects.manager.QueryOptions; + import com.metaobjects.manager.db.ObjectManagerDB; + import com.metaobjects.manager.exp.Expression; + import com.metaobjects.manager.exp.Range; + import com.metaobjects.manager.exp.SortOrder; + import com.metaobjects.object.MetaObject; + import com.metaobjects.object.value.ValueObject; + + import java.math.BigDecimal; + import java.util.ArrayList; + import java.util.LinkedHashMap; + import java.util.List; + import java.util.Map; + import java.util.Optional; + + /** Hand-written OMDB-backed {@link EntryRepository}. NOT a conformance subject. */ + public final class OmdbEntryRepository implements EntryRepository { + + private static final ObjectMapper MAPPER = new ObjectMapper() + .registerModule(new JavaTimeModule()) + .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS); + + private final ObjectManagerDB omdb; + private final MetaObject entry; + + public OmdbEntryRepository(ObjectManagerDB omdb, MetaObject entry) { + this.omdb = omdb; + this.entry = entry; + } + + @Override + public List list(int limit, int offset, SortClause sort, List filters) { + ObjectConnection c = omdb.getConnection(); + try { + QueryOptions options = new QueryOptions(); + Expression where = where(filters); + if (where != null) options.setExpression(where); + String field = sort != null ? sort.field() : "id"; + boolean desc = sort != null && "desc".equalsIgnoreCase(sort.direction()); + options.setSortOrder(new SortOrder(field, desc ? SortOrder.DESC : SortOrder.ASC)); + options.setRange(new Range(offset + 1, offset + limit)); // 1-based, inclusive + List out = new ArrayList<>(); + for (Object o : omdb.getObjects(c, entry, options)) out.add(toDto((ValueObject) o)); + return out; + } finally { + omdb.releaseConnection(c); + } + } + + @Override + public long count(List filters) { + ObjectConnection c = omdb.getConnection(); + try { + return omdb.getObjectsCount(c, entry, where(filters)); + } finally { + omdb.releaseConnection(c); + } + } + + @Override + public Optional findById(Long id) { + ObjectConnection c = omdb.getConnection(); + try { + return load(c, id).map(this::toDto); + } finally { + omdb.releaseConnection(c); + } + } + + @Override + public EntryDto create(EntryDto dto) { + ObjectConnection c = omdb.getConnection(); + try { + ValueObject vo = (ValueObject) entry.newInstance(); + vo.setString("bucket", dto.bucket()); + vo.setObject("amount", dto.amount()); + vo.setBoolean("flagged", Boolean.TRUE.equals(dto.flagged())); + omdb.createObject(c, vo); + return load(c, (Long) vo.get("id")).map(this::toDto).orElseThrow(); + } finally { + omdb.releaseConnection(c); + } + } + + @Override + public Optional update(Long id, EntryDto dto) { + ObjectConnection c = omdb.getConnection(); + try { + Optional cur = load(c, id); + if (cur.isEmpty()) return Optional.empty(); + ValueObject vo = cur.get(); + if (dto.bucket() != null) vo.setString("bucket", dto.bucket()); + vo.setObject("amount", dto.amount()); + if (dto.flagged() != null) vo.setBoolean("flagged", dto.flagged()); + omdb.updateObject(c, vo); + return load(c, id).map(this::toDto); + } finally { + omdb.releaseConnection(c); + } + } + + @Override + public Optional patch(Long id, EntryPatch patch) { + ObjectConnection c = omdb.getConnection(); + try { + Optional cur = load(c, id); + if (cur.isEmpty()) return Optional.empty(); + ValueObject vo = cur.get(); + if (patch.hasBucket()) vo.setString("bucket", patch.bucket()); + if (patch.hasAmount()) vo.setObject("amount", patch.amount()); + if (patch.hasFlagged()) vo.setBoolean("flagged", Boolean.TRUE.equals(patch.flagged())); + omdb.updateObject(c, vo); + return load(c, id).map(this::toDto); + } finally { + omdb.releaseConnection(c); + } + } + + @Override + public boolean delete(Long id) { + ObjectConnection c = omdb.getConnection(); + try { + return omdb.deleteObjects(c, entry, new Expression("id", id, Expression.EQUAL)) > 0; + } finally { + omdb.releaseConnection(c); + } + } + + private Optional load(ObjectConnection c, Long id) { + for (Object o : omdb.getObjects(c, entry, + new QueryOptions(new Expression("id", id, Expression.EQUAL)))) { + return Optional.of((ValueObject) o); + } + return Optional.empty(); + } + + // --- predicates -> Expression ------------------------------------------------ + + @SuppressWarnings("unchecked") + private Expression where(List filters) { + if (filters == null) return null; + Expression acc = null; + for (FilterPredicate p : filters) { // implicit AND + Expression e = switch (p.op()) { + case "isNull" -> new Expression(p.field(), null, + Boolean.TRUE.equals(p.value()) ? Expression.EQUAL : Expression.NOT_EQUAL); + case "in" -> { + List items = new ArrayList<>(); + for (Object item : (List) p.value()) items.add(operand(p.field(), item)); + yield new Expression(p.field(), items, Expression.EQUAL); + } + case "like" -> new Expression(p.field(), p.value(), Expression.LIKE); + case "eq" -> new Expression(p.field(), operand(p.field(), p.value()), Expression.EQUAL); + case "ne" -> new Expression(p.field(), operand(p.field(), p.value()), Expression.NOT_EQUAL); + case "gt" -> new Expression(p.field(), operand(p.field(), p.value()), Expression.GREATER); + case "gte" -> new Expression(p.field(), operand(p.field(), p.value()), Expression.EQUAL_GREATER); + case "lt" -> new Expression(p.field(), operand(p.field(), p.value()), Expression.LESSER); + case "lte" -> new Expression(p.field(), operand(p.field(), p.value()), Expression.EQUAL_LESSER); + default -> throw new IllegalStateException("unknown op: " + p.op()); + }; + acc = acc == null ? e : acc.and(e); + } + return acc; + } + + /** A raw query operand as the field's own type, so the bind is typed. */ + private Object operand(String field, Object value) { + // ADR-0019 Amendment 1: a decimal field's operand arrives parsed, as a BigDecimal. + if (value instanceof BigDecimal exact) return exact; + String raw = (String) value; + MetaField f = entry.getMetaField(field); + return switch (f.getSubType()) { + case "long" -> Long.valueOf(raw); + case "boolean" -> Boolean.valueOf(raw); + default -> raw; + }; + } + + private EntryDto toDto(ValueObject vo) { + Map values = new LinkedHashMap<>(); + for (MetaField f : entry.getMetaFields()) values.put(f.getName(), vo.get(f.getName())); + return MAPPER.convertValue(values, EntryDto.class); + } + } + """; +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/OmdbReportRepositorySource.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/OmdbReportRepositorySource.java new file mode 100644 index 000000000..7df48305c --- /dev/null +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/generated/OmdbReportRepositorySource.java @@ -0,0 +1,194 @@ +package com.metaobjects.integration.api.generated; + +/** + * The Java SOURCE for an OMDB-backed {@code acme.sales.Repository} impl: ONE generic + * template, instantiated once per served report by substituting the report's name. Emitted + * alongside the GENERATED controller / DTO / interface so it compiles against them, then + * loaded and instantiated reflectively by {@link GeneratedReportControllerHarness}. + * + *

This is the adopter-owned seam of a served report ({@code list} and {@code count} + * only), written the way an adopter writes it: it translates the controller's + * {@code FilterPredicate} list, {@code SortClause} and {@code limit/offset} into an OMDB + * {@code Expression}, {@code SortOrder} and {@code Range}, and reads the report's view + * through {@code ObjectManagerDB.getObjects} / {@code getObjectsCount}, which route a report + * through its read model. No SQL is written here: the view is the TypeScript-produced one the + * test database executed (ADR-0015), and the SQL the filters become is the driver's.

+ * + *

Generic over the row: an operand is coerced by the read-model field's declared subtype, + * and a row becomes the generated record through Jackson, so nothing here names a report's + * fields.

+ */ +final class OmdbReportRepositorySource { + + private OmdbReportRepositorySource() {} + + private static final String PKG = "acme.sales"; + private static final String NAME = "__REPORT__"; + + /** Simple name of the emitted impl for {@code report}. */ + static String simpleName(String report) { + return "Omdb" + report + "Repository"; + } + + /** Fully-qualified name of the emitted impl for {@code report}. */ + static String fqcn(String report) { + return fqcn(PKG, report); + } + + /** Fully-qualified name of the emitted impl for {@code report} in package {@code pkg}. */ + static String fqcn(String pkg, String report) { + return pkg + "." + simpleName(report); + } + + /** The impl source for {@code report}. */ + static String source(String report) { + return source(PKG, report); + } + + /** The impl source for {@code report}, in the package {@code pkg} its generated types are in. */ + static String source(String pkg, String report) { + return TEMPLATE.replace("__PKG__", pkg).replace(NAME, report); + } + + private static final String TEMPLATE = """ + package __PKG__; + + import com.fasterxml.jackson.databind.ObjectMapper; + import com.fasterxml.jackson.databind.SerializationFeature; + import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule; + import com.metaobjects.field.MetaField; + import com.metaobjects.generator.spring.runtime.FilterPredicate; + import com.metaobjects.manager.ObjectConnection; + import com.metaobjects.manager.QueryOptions; + import com.metaobjects.manager.db.ObjectManagerDB; + import com.metaobjects.manager.exp.Expression; + import com.metaobjects.manager.exp.Range; + import com.metaobjects.manager.exp.SortOrder; + import com.metaobjects.object.MetaObject; + import com.metaobjects.object.value.ValueObject; + + import java.math.BigDecimal; + import java.time.LocalDate; + import java.time.ZoneOffset; + import java.util.ArrayList; + import java.util.LinkedHashMap; + import java.util.List; + import java.util.Map; + + /** + * Hand-written OMDB-backed {@link __REPORT__Repository} (the read-only consumer seam): + * reads the report's view through ObjectManagerDB. NOT a conformance subject: it is the + * adopter's code, and the GENERATED controller above it is what the corpus gates. + */ + public final class Omdb__REPORT__Repository implements __REPORT__Repository { + + private static final ObjectMapper MAPPER = new ObjectMapper() + .registerModule(new JavaTimeModule()) + .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS); + + private final ObjectManagerDB omdb; + private final MetaObject report; + private final MetaObject row; + + public Omdb__REPORT__Repository(ObjectManagerDB omdb, MetaObject report) { + this.omdb = omdb; + this.report = report; + this.row = omdb.readObjectFor(report); + } + + @Override + public List<__REPORT__Dto> list(int limit, int offset, SortClause sort, List filters) { + ObjectConnection c = omdb.getConnection(); + try { + QueryOptions options = new QueryOptions(); + Expression where = where(filters); + if (where != null) options.setExpression(where); + if (sort != null) { + row.getMetaField(sort.field()); // an unknown sort column is a harness bug + options.setSortOrder(new SortOrder(sort.field(), + "desc".equalsIgnoreCase(sort.direction()) ? SortOrder.DESC : SortOrder.ASC)); + } + // Range is 1-based and inclusive. + options.setRange(new Range(offset + 1, offset + limit)); + List<__REPORT__Dto> out = new ArrayList<>(); + for (Object o : omdb.getObjects(c, report, options)) out.add(toDto((ValueObject) o)); + return out; + } finally { + omdb.releaseConnection(c); + } + } + + @Override + public long count(List filters) { + ObjectConnection c = omdb.getConnection(); + try { + return omdb.getObjectsCount(c, report, where(filters)); + } finally { + omdb.releaseConnection(c); + } + } + + // --- predicates -> Expression ------------------------------------------------ + + @SuppressWarnings("unchecked") + private Expression where(List filters) { + Expression acc = null; + if (filters == null) return null; + for (FilterPredicate p : filters) { // implicit AND + Expression e = switch (p.op()) { + case "isNull" -> new Expression(p.field(), null, + Boolean.TRUE.equals(p.value()) ? Expression.EQUAL : Expression.NOT_EQUAL); + case "in" -> { + List items = new ArrayList<>(); + for (Object item : (List) p.value()) items.add(operand(p.field(), item)); + yield new Expression(p.field(), items, Expression.EQUAL); + } + case "like" -> new Expression(p.field(), p.value(), Expression.LIKE); + case "eq" -> new Expression(p.field(), operand(p.field(), p.value()), Expression.EQUAL); + case "ne" -> new Expression(p.field(), operand(p.field(), p.value()), Expression.NOT_EQUAL); + case "gt" -> new Expression(p.field(), operand(p.field(), p.value()), Expression.GREATER); + case "gte" -> new Expression(p.field(), operand(p.field(), p.value()), Expression.EQUAL_GREATER); + case "lt" -> new Expression(p.field(), operand(p.field(), p.value()), Expression.LESSER); + case "lte" -> new Expression(p.field(), operand(p.field(), p.value()), Expression.EQUAL_LESSER); + default -> throw new IllegalStateException("unknown op: " + p.op()); + }; + acc = acc == null ? e : acc.and(e); + } + return acc; + } + + /** A raw query operand as the read-model field's own type, so the bind is typed. */ + private Object operand(String field, Object value) { + // ADR-0019 Amendment 1: FilterParser hands a decimal field's operand over as a BigDecimal. + if (value instanceof BigDecimal exact) return exact; + String raw = (String) value; + MetaField f = row.getMetaField(field); + return switch (f.getSubType()) { + case "long", "currency" -> Long.valueOf(raw); + case "int" -> Integer.valueOf(raw); + case "double" -> Double.valueOf(raw); + case "float" -> Float.valueOf(raw); + case "decimal" -> new BigDecimal(raw); + case "boolean" -> Boolean.valueOf(raw); + case "date" -> LocalDate.parse(raw); + case "uuid" -> raw.toLowerCase(java.util.Locale.ROOT); + default -> raw; // string, enum symbol + }; + } + + // --- row -> generated record ------------------------------------------------- + + private __REPORT__Dto toDto(ValueObject vo) { + Map values = new LinkedHashMap<>(); + for (MetaField f : row.getMetaFields()) { + Object v = vo.get(f.getName()); + if (v instanceof java.util.Date d) { + v = d.toInstant().atZone(ZoneOffset.UTC).toLocalDate().toString(); + } + values.put(f.getName(), v); + } + return MAPPER.convertValue(values, __REPORT__Dto.class); + } + } + """; +} diff --git a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/m2m/M2mApiContractConformanceTest.java b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/m2m/M2mApiContractConformanceTest.java index e1b24792b..a2e07ce7d 100644 --- a/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/m2m/M2mApiContractConformanceTest.java +++ b/server/java/integration-tests/src/test/java/com/metaobjects/integration/api/m2m/M2mApiContractConformanceTest.java @@ -65,7 +65,7 @@ void scenario(String name, M2mScenario scenario) { HttpResponse.BodyHandlers.ofString()); Object parsed = (res.body() == null || res.body().isEmpty()) ? null - : MAPPER.readValue(res.body(), Object.class); + : MAPPER.copy().enable(com.fasterxml.jackson.databind.DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS).readValue(res.body(), Object.class); M2mScenarios.assertResponse(scenario.name(), req, res.statusCode(), parsed); } } diff --git a/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/MetaDataEjectMojo.java b/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/MetaDataEjectMojo.java index 5f251ea37..1e3b5c2a6 100644 --- a/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/MetaDataEjectMojo.java +++ b/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/MetaDataEjectMojo.java @@ -798,8 +798,13 @@ private String ownedSupportStaleness(Path codegenRoot, String simple) { } } - /** The staleness of the owned copy of {@code e}, over every class copied with it; - * {@code null} when there is no such copy. */ + /** + * The staleness of the owned copy of {@code e}: the generator is {@code identical} only when + * its file AND every companion it owns are. The companions are the classes copied with it, + * the page templates it reads (under the owned resources), the shared support rules it + * imports and the helper runtime its output imports. Each companion that differs is named, + * so the verdict says WHICH file to look at; {@code null} when there is no owned copy. + */ private String ownedStaleness(Path codegenRoot, EjectSupport.Entry e) { String langDir = e.port == EjectSupport.Port.KOTLIN ? "kotlin" : "java"; try { @@ -811,26 +816,87 @@ private String ownedStaleness(Path codegenRoot, EjectSupport.Entry e) { Path srcRoot = codegenRoot.resolve("src/main/" + langDir); String ownedPackage = srcRoot.relativize(found.getParent()).toString() .replace(found.getFileSystem().getSeparator(), "."); - int behind = 0; - int ownedOnly = 0; + int[] totals = new int[2]; + List differing = new ArrayList<>(); for (String source : e.sources()) { Path file = found.resolveSibling( EjectSupport.simpleNameOf(source) + "." + e.port.extension); String owned = Files.exists(file) ? Files.readString(file, StandardCharsets.UTF_8) : ""; String reference = EjectSupport.rewriteCopiedImports( readResource(e.resourceOf(source)), e.sources(), ownedPackage); - EjectSupport.Comparison cmp = EjectSupport.compare(owned, reference); - behind += cmp.behind; - ownedOnly += cmp.ownedOnly; + fold(totals, differing, EjectSupport.simpleNameOf(source) + "." + e.port.extension, + EjectSupport.compare(owned, reference)); } - return behind == 0 && ownedOnly == 0 + if (e.port == EjectSupport.Port.JAVA) { + // The page templates the owned generator reads: a template missing from the owned + // resources counts as wholly behind. + Path resourceRoot = codegenRoot.resolve("src/main/resources") + .resolve(EjectSupport.ownedTemplatePrefix(ownedPackage)); + for (String ref : e.resources) { + Path file = resourceRoot.resolve(ref + EjectSupport.TEMPLATE_EXTENSION); + String owned = Files.exists(file) ? Files.readString(file, StandardCharsets.UTF_8) : ""; + String reference = readResource(GeneratorRegistry.EJECT_TEMPLATE_RESOURCE_ROOT + + ref + EjectSupport.TEMPLATE_EXTENSION); + fold(totals, differing, ref + EjectSupport.TEMPLATE_EXTENSION, + EjectSupport.compare(owned, reference)); + } + // The shared support rules the generator imports, found wherever the adopter put + // them (absent: not ejected, so nothing of the adopter's to be behind). + Set support = new java.util.LinkedHashSet<>(); + for (String source : e.sources()) { + support.addAll(EjectSupport.supportFor(readResource(e.resourceOf(source)))); + } + for (String simple : support) { + Path owned = findOwned(codegenRoot, "java", simple + ".java"); + if (owned == null) continue; + fold(totals, differing, simple + ".java", EjectSupport.compare( + Files.readString(owned, StandardCharsets.UTF_8), readSupport(simple))); + } + // The helper runtime its output imports, where this module (or a child) owns it. + for (String simple : e.runtime) { + Path owned = findOwnedRuntime(simple); + if (owned == null) continue; + fold(totals, differing, simple + ".java", EjectSupport.compare( + Files.readString(owned, StandardCharsets.UTF_8), + readResource(GeneratorRegistry.EJECT_RUNTIME_RESOURCE_ROOT + simple + ".java"))); + } + } + return differing.isEmpty() ? "[owned — identical]" - : "[owned — DIFFERS: " + behind + " behind, " + ownedOnly + " of your own]"; - } catch (IOException ex) { + : "[owned — DIFFERS: " + totals[0] + " behind, " + totals[1] + " of your own; differing: " + + String.join(", ", differing) + "]"; + } catch (IOException | MojoExecutionException ex) { return "[owned — could not compare: " + ex.getMessage() + "]"; } } + /** Add one file's comparison to the running totals, naming it when it differs. */ + private static void fold(int[] totals, List differing, String name, EjectSupport.Comparison cmp) { + totals[0] += cmp.behind; + totals[1] += cmp.ownedOnly; + if (cmp.verdict != EjectSupport.Verdict.IDENTICAL) differing.add(name); + } + + /** The owned runtime copy of {@code simple} in this module or one of its children, or {@code null}. */ + private Path findOwnedRuntime(String simple) throws IOException { + if (project == null || project.getBasedir() == null) return null; + Path base = project.getBasedir().toPath(); + List roots = new ArrayList<>(); + roots.add(base.resolve("src/main/java")); + if (project.getModules() != null) { + for (String module : project.getModules()) roots.add(base.resolve(module).resolve("src/main/java")); + } + for (Path root : roots) { + if (!Files.isDirectory(root)) continue; + try (var walk = Files.walk(root)) { + for (Path p : (Iterable) walk.filter(f -> f.getFileName().toString().equals(simple + ".java"))::iterator) { + if (EjectSupport.isOwnedRuntime(Files.readString(p, StandardCharsets.UTF_8))) return p; + } + } + } + return null; + } + // ------------------------------------------------------------------------------------ // helpers // ------------------------------------------------------------------------------------ diff --git a/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/EjectRuntimeRoundTripTest.java b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/EjectRuntimeRoundTripTest.java index b77e111c0..b1d648c1e 100644 --- a/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/EjectRuntimeRoundTripTest.java +++ b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/EjectRuntimeRoundTripTest.java @@ -60,7 +60,7 @@ *
  • The declared runtime sets are exact. Every ejectable generator in both ports is * run against the shared fitness corpus; the helper classes its output imports must be * precisely what its registry entry declares (so eject copies neither too little nor - * too much), each set must compile on the JDK alone (closed, no MetaObjects on the + * too much), each set must compile with Jackson alone (closed, no MetaObjects on the * classpath), and no Kotlin output imports helper runtime at all.
    • *
  • The round trip. Eject the whole Spring web tier into a temp project, compile * and run the owned generators, and compile their output together with the owned @@ -145,7 +145,10 @@ public void eachJavaGeneratorsOutputImportsExactlyTheRuntimeItsEntryDeclares() t if (!(gen instanceof com.metaobjects.generator.direct.MultiFileDirectGeneratorBase)) { a.put("outputFilename", e.stableName + ".out"); } - gen.setArgs(a).execute(loader); + // The shared fitness corpus declares no decimal value-object member, and a value object + // names the decimal wire classes only for one: that generator runs over a small + // dedicated model that has one, so its declared set can be held exact too. + gen.setArgs(a).execute("value-object".equals(e.stableName) ? decimalValueObjectLoader() : loader); Set all = imports(out); for (String imp : all) { @@ -163,7 +166,24 @@ public void eachJavaGeneratorsOutputImportsExactlyTheRuntimeItsEntryDeclares() t declared, closure(used)); if (!declared.isEmpty()) withRuntime++; } - assertEquals("routes, dto and repository carry runtime", 3, withRuntime); + assertEquals("routes, dto, repository and value-object carry runtime", 4, withRuntime); + } + + private static final String DECIMAL_VALUE_OBJECT_MODEL = """ + {"metadata.root":{"package":"acme","children":[ + {"object.value":{"name":"Price","children":[ + {"field.decimal":{"name":"net","@precision":12,"@scale":2}} + ]}} + ]}}"""; + + private MetaDataLoader decimalValueObjectLoader() throws IOException { + Path file = tempFolder.newFolder("decimal-vo").toPath().resolve("meta.price.json"); + Files.writeString(file, DECIMAL_VALUE_OBJECT_MODEL, StandardCharsets.UTF_8); + MetaDataLoader l = new MetaDataLoader(LoaderOptions.create(false, false, true), + MetaDataLoader.SUBTYPE_MANUAL, "eject-decimal-vo"); + l.setSourceURIs(List.of(file.toUri())); + l.init(); + return l; } /** {@code names} plus every runtime class their sources mention by simple name. */ @@ -199,7 +219,7 @@ private static String readRuntimeResource(String simpleName) throws IOException } @Test - public void eachDeclaredRuntimeSetCompilesOnTheJdkAlone() throws Exception { + public void eachDeclaredRuntimeSetCompilesWithJacksonAlone() throws Exception { for (EjectSupport.Entry e : EjectSupport.allEntries()) { if (e.runtime.isEmpty()) continue; Path dir = tempFolder.newFolder("closed-" + e.stableName).toPath().resolve("acme/rt"); @@ -209,7 +229,7 @@ public void eachDeclaredRuntimeSetCompilesOnTheJdkAlone() throws Exception { EjectSupport.ownedRuntimeSource(readRuntimeResource(n), n, "acme.rt"), StandardCharsets.UTF_8); } - compile(List.of(dir), "", tempFolder.newFolder("closed-classes-" + e.stableName).toPath()); + compile(List.of(dir), jacksonOnly(), tempFolder.newFolder("closed-classes-" + e.stableName).toPath()); } } @@ -260,7 +280,10 @@ public void ejectHandsOverTheRuntimeAndTheOwnedOutputUsesIt() throws Exception { // The runtime landed where the generated code compiles, package-renamed and marked. Path runtimeDir = projectDir.resolve("src/main/java/com/acme/runtime"); - Set expected = new TreeSet<>(GeneratorRegistry.get("routes").ejectRuntime()); + // The union of what the owned web tier declares: routes, dto and value-object each + // name the runtime classes their output imports (the decimal wire ones ride on dto). + Set expected = new TreeSet<>(); + for (String tier : WEB_TIER) expected.addAll(GeneratorRegistry.get(tier).ejectRuntime()); Set written; try (Stream s = Files.list(runtimeDir)) { written = s.map(p -> p.getFileName().toString().replace(".java", "")) @@ -392,6 +415,17 @@ private Path runOwnedGenerators(Path projectDir) throws Exception { /** The test classpath with every MetaObjects artifact removed (reactor module output or * a {@code com/metaobjects} jar). */ + /** + * The Jackson jars alone: the owned runtime closes over no MetaObjects artifact, but the + * decimal wire classes and {@code UnreadableBody} are Jackson types, as is every generated + * DTO's own imports. + */ + private static String jacksonOnly() { + return Stream.of(System.getProperty("java.class.path").split(File.pathSeparator)) + .filter(e -> e.replace('\\', '/').contains("/com/fasterxml/jackson/")) + .collect(Collectors.joining(File.pathSeparator)); + } + private static String withoutMetaObjects() { return Stream.of(System.getProperty("java.class.path").split(File.pathSeparator)) .filter(e -> { diff --git a/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/EjectSupportTest.java b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/EjectSupportTest.java index 6e52b741f..cb231304b 100644 --- a/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/EjectSupportTest.java +++ b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/EjectSupportTest.java @@ -295,7 +295,7 @@ public void supportImportsMoveToTheOwnedPackageAndAreNotDrift() { public void onlyTheJavaWebTierCarriesRuntime() { for (EjectSupport.Entry e : EjectSupport.allEntries()) { boolean expected = e.port == EjectSupport.Port.JAVA - && List.of("routes", "dto", "repository").contains(e.stableName); + && List.of("routes", "dto", "repository", "value-object").contains(e.stableName); assertEquals(e.port.id + " " + e.stableName, expected, !e.runtime.isEmpty()); } } @@ -325,7 +325,10 @@ public void everyKotlinEntryResolvesItsCopiedClassesToKotlinSources() { EjectSupport.Entry entity = EjectSupport.resolve( "entity", EjectSupport.ejectableByStableName(), EjectSupport.Port.KOTLIN, null).entry; assertEquals(List.of("com.metaobjects.generator.kotlin.KotlinEntityGenerator", - "com.metaobjects.generator.kotlin.KotlinEnumEmitter"), entity.sources()); + "com.metaobjects.generator.kotlin.KotlinEnumEmitter", + "com.metaobjects.generator.kotlin.KotlinDecimalWireEmitter"), entity.sources()); + assertTrue(entity.resourceOf("com.metaobjects.generator.kotlin.KotlinDecimalWireEmitter") + .endsWith("/kotlin/KotlinDecimalWireEmitter.kt")); assertTrue(entity.resourceOf("com.metaobjects.generator.kotlin.KotlinEnumEmitter"), entity.resourceOf("com.metaobjects.generator.kotlin.KotlinEnumEmitter") .endsWith("/kotlin/KotlinEnumEmitter.kt")); diff --git a/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/EntityExtractorEjectRoundTripTest.java b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/EntityExtractorEjectRoundTripTest.java index d54408942..195e6baaa 100644 --- a/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/EntityExtractorEjectRoundTripTest.java +++ b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/EntityExtractorEjectRoundTripTest.java @@ -372,7 +372,7 @@ public void theListingReportsBothAsEjectableAndMarksTheirOwnedCopies() throws Ex " * GENERATED — extractor for {@code ", " * GENERATED (mine) — extractor for {@code "); String edited = list(projectDir); assertTrue(row(edited, "extractor"), - row(edited, "extractor").endsWith("[owned — DIFFERS: 1 behind, 1 of your own]")); + row(edited, "extractor").contains("[owned — DIFFERS: 1 behind, 1 of your own; differing: ")); // entity copies no extractor, so its row does not move. assertTrue(row(edited, "entity"), row(edited, "entity").endsWith("[owned — identical]")); @@ -382,9 +382,9 @@ public void theListingReportsBothAsEjectableAndMarksTheirOwnedCopies() throws Ex "println(true, \"super(mo);\");", "println(true, \"super(mo); // mine\");"); String writerEdited = list(projectDir); assertTrue(row(writerEdited, "entity"), - row(writerEdited, "entity").endsWith("[owned — DIFFERS: 1 behind, 1 of your own]")); + row(writerEdited, "entity").contains("[owned — DIFFERS: 1 behind, 1 of your own; differing: ")); assertTrue(row(writerEdited, "extractor"), - row(writerEdited, "extractor").endsWith("[owned — DIFFERS: 1 behind, 1 of your own]")); + row(writerEdited, "extractor").contains("[owned — DIFFERS: 1 behind, 1 of your own; differing: ")); } @Test diff --git a/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/MetaDataEjectMojoTest.java b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/MetaDataEjectMojoTest.java index 8baf5a704..682602a2c 100644 --- a/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/MetaDataEjectMojoTest.java +++ b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/MetaDataEjectMojoTest.java @@ -295,6 +295,11 @@ public void listReportsTheOwnedSupportCopyAndWhetherItWasEdited() throws Excepti Files.writeString(support, Files.readString(support, StandardCharsets.UTF_8) + "// mine\n", StandardCharsets.UTF_8); assertTrue(listOutput().contains(" SpringGeneratorSupport [owned — DIFFERS: 0 behind, 1 of your own]")); + // E1: the generator is identical only when every companion it owns is, so the edited + // support rule makes the dto DIFFER, and the verdict names the file. + String out = listOutput(); + assertTrue(out, out.contains("Per-entity Spring DTO record. [owned — DIFFERS: 0 behind, 1 of your own; " + + "differing: SpringGeneratorSupport.java]")); } @Test @@ -355,4 +360,16 @@ public void ejectingTheJavaApiDocsWritesTheTemplatesAndPointsTheCopiedProviderAt String generator = Files.readString(src.resolve("JavaApiDocsGenerator.java"), StandardCharsets.UTF_8); assertTrue(generator, generator.contains("package com.acme.codegen.apidocs.java;")); } + + @Test + public void anEditedTemplateMakesItsGeneratorDifferAndIsNamed() throws Exception { + mojo("api-docs", false, "java", null).execute(); + assertTrue(listOutput(), listOutput().contains("[owned — identical]")); + Path template = codegenRoot() + .resolve("src/main/resources/com/acme/codegen/apidocs/java/templates/api/index.md.mustache"); + Files.writeString(template, Files.readString(template, StandardCharsets.UTF_8) + "\nmy line\n", + StandardCharsets.UTF_8); + String out = listOutput(); + assertTrue(out, out.contains("differing: api/index.md.mustache]")); + } } diff --git a/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/RequirementTestsGeneratorMojoTest.java b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/RequirementTestsGeneratorMojoTest.java index d800ff9f7..96235ad9c 100644 --- a/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/RequirementTestsGeneratorMojoTest.java +++ b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/RequirementTestsGeneratorMojoTest.java @@ -125,6 +125,98 @@ public void aFilterClassOnlyTheProjectClasspathHoldsIsFoundThroughTheGenerateGoa assertTrue(e.toString(), e.toString().contains("com.acme.ProjectOnlyFilter")); } + /** + * The same removal in a real build, where the module has a base directory and so a + * manifest: the full run deletes the files the manifest recorded and the hash still + * matches, and verify is clean. This is the Python 1.1 behaviour. + */ + @Test + public void aPackageThatLostItsLastRequirementHasItsRecordedUntouchedFilesDeletedByTheFullRun() throws Exception { + Path out = tmp.newFolder().toPath(); + MetaDataGeneratorMojo gen = genMojoWithBaseDir(); + configure(gen, model(ORDER_ENTITY, L3_PARENT, L4_ORDER), out); + gen.execute(); + Path testFile = out.resolve("com/acme/requirements/Requirements_acme_shop_Test.java"); + Path witnessFile = out.resolve("com/acme/requirements/Requirements_acme_shop_Witnesses.java"); + assertTrue(Files.exists(testFile)); + assertTrue(Files.exists(witnessFile)); + + Path dropped = model(ORDER_ENTITY); + MetaDataGeneratorMojo regen = genMojoWithBaseDir(); + configure(regen, dropped, out); + regen.execute(); + assertTrue("the recorded, untouched test file is deleted", !Files.exists(testFile)); + assertTrue("the recorded, untouched witness file is deleted", !Files.exists(witnessFile)); + + MetaDataVerifyMojo verify = new MetaDataVerifyMojo(); + configure(verify, dropped, out); + verify.setMode("codegen"); + verify.execute(); + } + + /** + * The same deletion for the Kotlin generator (named by its class, as a POM would): its two + * {@code .kt} files are recorded in the manifest and swept by the full run once the package + * has no requirement left. + */ + @Test + public void aKotlinPackageThatLostItsLastRequirementHasItsRecordedUntouchedFilesDeleted() throws Exception { + String kotlinGenerator = "com.metaobjects.generator.kotlin.KotlinRequirementTestsGenerator"; + Path out = tmp.newFolder().toPath(); + MetaDataGeneratorMojo gen = genMojoWithBaseDir(); + configure(gen, model(ORDER_ENTITY, L3_PARENT, L4_ORDER), out); + gen.setGenerators(new ArrayList<>(List.of(GeneratorParam.builder(kotlinGenerator) + .withArg("outputDir", out.toString()) + .withArg("testPackage", "com.acme.requirements") + .withArg("witnessClass", "com.acme.Witnesses") + .withArg("warnUncovered", "false").build()))); + gen.execute(); + Path testFile = out.resolve("com/acme/requirements/Requirements_acme_shop_Test.kt"); + Path witnessFile = out.resolve("com/acme/requirements/Requirements_acme_shop_Witnesses.kt"); + assertTrue(Files.exists(testFile)); + assertTrue(Files.exists(witnessFile)); + + MetaDataGeneratorMojo regen = genMojoWithBaseDir(); + configure(regen, model(ORDER_ENTITY), out); + regen.setGenerators(new ArrayList<>(List.of(GeneratorParam.builder(kotlinGenerator) + .withArg("outputDir", out.toString()) + .withArg("testPackage", "com.acme.requirements") + .withArg("witnessClass", "com.acme.Witnesses") + .withArg("warnUncovered", "false").build()))); + regen.execute(); + assertTrue("the recorded, untouched Kotlin test file is deleted", !Files.exists(testFile)); + assertTrue("the recorded, untouched Kotlin witness file is deleted", !Files.exists(witnessFile)); + } + + @Test + public void anEditedFileOfAPackageThatLostItsLastRequirementIsKeptAndRefused() throws Exception { + Path out = tmp.newFolder().toPath(); + MetaDataGeneratorMojo gen = genMojoWithBaseDir(); + configure(gen, model(ORDER_ENTITY, L3_PARENT, L4_ORDER), out); + gen.execute(); + Path testFile = out.resolve("com/acme/requirements/Requirements_acme_shop_Test.java"); + Files.writeString(testFile, Files.readString(testFile, StandardCharsets.UTF_8) + "\n// my edit\n", StandardCharsets.UTF_8); + + MetaDataGeneratorMojo regen = genMojoWithBaseDir(); + configure(regen, model(ORDER_ENTITY), out); + regen.execute(); + assertTrue("an edited candidate stays on disk", Files.exists(testFile)); + } + + private MetaDataGeneratorMojo genMojoWithBaseDir() throws Exception { + MavenProject project = Mockito.mock(MavenProject.class); + Mockito.when(project.getBasedir()).thenReturn(tmp.getRoot()); + Mockito.when(project.getRuntimeClasspathElements()).thenReturn(List.of()); + Mockito.when(project.getCompileClasspathElements()).thenReturn(List.of()); + Mockito.when(project.getTestClasspathElements()).thenReturn(List.of()); + MojoExecution execution = Mockito.mock(MojoExecution.class); + Mockito.when(execution.getLifecyclePhase()).thenReturn(AbstractMetaDataMojo.PHASE_GENERATE_SOURCES); + MetaDataGeneratorMojo mojo = new MetaDataGeneratorMojo(); + mojo.project = project; + mojo.execution = execution; + return mojo; + } + @Test public void verifyReportsTheFilesOfAPackageThatLostItsLastRequirementAsStaleInRepo() throws Exception { Path out = tmp.newFolder().toPath(); @@ -137,7 +229,10 @@ public void verifyReportsTheFilesOfAPackageThatLostItsLastRequirementAsStaleInRe inSync.setMode("codegen"); inSync.execute(); - // The requirement is removed. gen never deletes a file, so the two are still there... + // The requirement is removed. This mojo has no project base directory, so there is no + // `.metaobjects/.gen-state` manifest and gen deletes nothing: the files are not recorded + // as generated, so they are not candidates for the orphan sweep (own-your-codegen.md, + // "Files that are no longer generated"). The two are still there... Path dropped = model(ORDER_ENTITY); MetaDataGeneratorMojo regen = new MetaDataGeneratorMojo(); configure(regen, dropped, out); diff --git a/server/java/metadata/src/main/java/com/metaobjects/attr/IntAttribute.java b/server/java/metadata/src/main/java/com/metaobjects/attr/IntAttribute.java index 238026c7a..744f68818 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/attr/IntAttribute.java +++ b/server/java/metadata/src/main/java/com/metaobjects/attr/IntAttribute.java @@ -67,7 +67,7 @@ public void setValueAsString(String value) { String trimmed = item.trim(); if (!trimmed.isEmpty()) { try { - list.add(Integer.parseInt(trimmed)); + list.add(parseIntegral(trimmed)); } catch (NumberFormatException e) { throw new IllegalArgumentException("Invalid integer value in array: " + trimmed, e); } @@ -77,7 +77,7 @@ public void setValueAsString(String value) { } else { // Single value try { - Integer intValue = Integer.parseInt(value.trim()); + Integer intValue = parseIntegral(value.trim()); setValueAsObject(Arrays.asList(intValue)); } catch (NumberFormatException e) { throw new IllegalArgumentException("Invalid integer value: " + value, e); @@ -89,7 +89,7 @@ public void setValueAsString(String value) { setValueAsObject(null); } else { try { - setValueAsObject(Integer.parseInt(value.trim())); + setValueAsObject(parseIntegral(value.trim())); } catch (NumberFormatException e) { throw new IllegalArgumentException("Invalid integer value: " + value, e); } @@ -97,6 +97,19 @@ public void setValueAsString(String value) { } } + /** + * Rule A (cross-port): an int attribute accepts any number whose value is integral, so + * {@code 4}, {@code 4.0} and {@code 1e2} load as an integer, and a fraction is refused. + * The refusal is a {@link NumberFormatException} the callers turn into the load error. + */ + static int parseIntegral(String text) { + try { + return new java.math.BigDecimal(text).intValueExact(); + } catch (ArithmeticException notIntegral) { + throw new NumberFormatException("not an integral int: " + text); + } + } + /** * Register this type with the MetaDataRegistry (called by provider) */ diff --git a/server/java/metadata/src/main/java/com/metaobjects/attr/LongAttribute.java b/server/java/metadata/src/main/java/com/metaobjects/attr/LongAttribute.java index 0c85deb70..9f8eb3366 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/attr/LongAttribute.java +++ b/server/java/metadata/src/main/java/com/metaobjects/attr/LongAttribute.java @@ -47,6 +47,23 @@ public LongAttribute(String name ) { super( SUBTYPE_LONG, name, DataTypes.LONG); } + /** + * Rule A (cross-port): any number whose value is integral loads ({@code 4}, {@code 4.0}, + * {@code 1e2}); a fraction is refused with the generic invalid-value error. + */ + @Override + public void setValueAsString(String value) { + if (value == null || isArrayType()) { + super.setValueAsString(value); + return; + } + try { + setValueAsObject(new java.math.BigDecimal(value.trim()).longValueExact()); + } catch (ArithmeticException | NumberFormatException e) { + throw new IllegalArgumentException("Invalid long value: " + value, e); + } + } + /** * Manually create an Integer MetaAttribute with a value */ diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java index a3e357ad1..ac7275817 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java +++ b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java @@ -170,4 +170,29 @@ public static PhysicalNaming resolvePhysicalNaming(Path root) { .map(NeutralConfig::getPhysicalNaming) .orElse(PhysicalNaming.DEFAULT); } + + /** + * ADR-0058 — the physical-naming rules of the {@code .metaobjects/config.json} found by + * walking up from the first {@code file:} URI (stopping after a directory holding + * {@code .git}), or empty when there is no {@code file:} URI or no config on the walk. + * The same walk the Maven plugin does for its source location. + */ + public static java.util.Optional discoverPhysicalNaming(List uris) { + for (java.net.URI uri : uris) { + if (!"file".equalsIgnoreCase(uri.getScheme())) continue; + Path location; + try { + location = Path.of(uri).toAbsolutePath().normalize(); + } catch (RuntimeException e) { + return java.util.Optional.empty(); + } + for (Path dir = location; dir != null; dir = dir.getParent()) { + java.util.Optional cfg = NeutralConfig.read(dir); + if (cfg.isPresent()) return java.util.Optional.of(cfg.get().getPhysicalNaming()); + if (Files.exists(dir.resolve(".git"))) break; + } + return java.util.Optional.empty(); + } + return java.util.Optional.empty(); + } } diff --git a/server/java/metadata/src/main/java/com/metaobjects/loader/MetaDataLoader.java b/server/java/metadata/src/main/java/com/metaobjects/loader/MetaDataLoader.java index bf67aeebb..2fb938ffb 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/loader/MetaDataLoader.java +++ b/server/java/metadata/src/main/java/com/metaobjects/loader/MetaDataLoader.java @@ -462,7 +462,8 @@ private void throwUnresolvedSuper(PendingExtends p) { throw new com.metaobjects.MetaDataException( "Invalid MetaData [" + p.typeName + "][" + p.child.getShortName() + "], the SuperClass [" + p.superName + "] does not exist (deferred resolution)" - + " in file [" + p.filename + "]", + + " in file [" + p.filename + "]" + + ValidationPhase.didYouMeanExtendsHint(root, p.superName), com.metaobjects.ErrorCode.ERR_UNRESOLVED_SUPER, envelope); } @@ -720,6 +721,9 @@ public static MetaDataLoader fromDirectory(String name, Path directory, sources.addAll(new DirectorySource(directory, opts).expandToList()); loader.load(sources); loader.register(); + // ADR-0058: the same naming discovery fromUris does, walking up from the directory. + loader.getRoot().setPhysicalNaming( + discoverNamingOrWarn(name, List.of(directory.toAbsolutePath().toUri()))); } catch (MetaDataLoadingException e) { throw e; } catch (Exception e) { @@ -760,6 +764,38 @@ public static MetaDataLoader fromUris(String name, List uris) { * @return a fully-initialized loader with all URIs loaded */ public static MetaDataLoader fromUris(String name, List uris, LoaderOptions opts) { + return fromUris(name, uris, opts, null); + } + + /** + * The naming of the {@code .metaobjects/config.json} found above {@code locations}, else ONE + * WARN and the default (ADR-0058). Shared by {@code fromUris} and {@code fromDirectory}. + */ + private static com.metaobjects.database.PhysicalNaming discoverNamingOrWarn(String name, List locations) { + return com.metaobjects.config.SourceResolver.discoverPhysicalNaming(locations).orElseGet(() -> { + log.warn("No .metaobjects/config.json found above the metadata sources of loader '{}': " + + "using the default physical naming (columns snake_case, tables snake_case). " + + "If your database columns are named like the fields, set " + + "\"naming\": { \"columns\": \"literal\" } in the project's .metaobjects/config.json " + + "and, for a classpath runtime, point the Spring property " + + "metaobjects.physical-naming-config at it.", name); + return com.metaobjects.database.PhysicalNaming.DEFAULT; + }); + } + + /** + * Like {@link #fromUris(String, List, LoaderOptions)}, with the project's physical-naming + * rules supplied by the caller (ADR-0058). + * + *

    When {@code physicalNaming} is non-null it is set on the loaded root and nothing is + * discovered. When it is {@code null} the loader reads the {@code naming} block of the + * {@code .metaobjects/config.json} it finds by walking up from the first {@code file:} URI + * (the walk the Maven plugin does), and logs ONE WARN when it finds none and falls back to + * the {@code snake_case} default. An explicit {@code root.setPhysicalNaming(...)} made by + * the caller after this returns always wins.

    + */ + public static MetaDataLoader fromUris(String name, List uris, LoaderOptions opts, + com.metaobjects.database.PhysicalNaming physicalNaming) { MetaDataLoader loader = (opts == null) ? createManual(false, name) : new MetaDataLoader(opts, SUBTYPE_MANUAL, name); @@ -769,6 +805,9 @@ public static MetaDataLoader fromUris(String name, List uris, LoaderOptions for (URI uri : uris) sources.add(new UriSource(uri)); loader.load(sources); loader.register(); + com.metaobjects.database.PhysicalNaming naming = physicalNaming != null + ? physicalNaming : discoverNamingOrWarn(name, uris); + loader.getRoot().setPhysicalNaming(naming); } catch (MetaDataLoadingException e) { throw e; } catch (Exception e) { diff --git a/server/java/metadata/src/main/java/com/metaobjects/loader/ReportingValidation.java b/server/java/metadata/src/main/java/com/metaobjects/loader/ReportingValidation.java index 2866b0e85..b111e12e9 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/loader/ReportingValidation.java +++ b/server/java/metadata/src/main/java/com/metaobjects/loader/ReportingValidation.java @@ -454,9 +454,29 @@ private static void checkDimension(MemberCtx ctx, MetaDimension dim) { return; } String fieldName = parts.path().get(0); - MetaData field = fieldOf(ofEntity, fieldName); + // With @via the field is looked up on the entity @of NAMES (resolving: its own plus + // inherited), not on the @via terminal: a base the terminal extends must itself carry the + // field, or the dimension names a column the base table lacks. Without @via the host IS + // the checked entity. + MetaData lookup = via == null ? ofEntity : named; + MetaData field = fieldOf(lookup, fieldName); if (field == null) { - err.accept("@of '" + of + "' names no field '" + fieldName + "' on '" + shown(ctx, ofEntity) + "'."); + boolean declaredOnTerminal = lookup != ofEntity && fieldOf(ofEntity, fieldName) != null; + err.accept(declaredOnTerminal + ? "@of '" + of + "' names no field '" + fieldName + "' on '" + shown(ctx, lookup) + "': only '" + + shown(ctx, ofEntity) + "', the entity @via reaches, declares it. Name it '" + + ofEntity.getShortName() + "." + fieldName + "'." + : "@of '" + of + "' names no field '" + fieldName + "' on '" + shown(ctx, lookup) + "'."); + return; + } + + // D5 — a dimension groups by a scalar. A field.object (or a field carrying @objectRef) is + // a JSON document: no port serves a report over one, and a document is not a value to + // group rows by. + if (isDocumentField(field)) { + err.accept("@of '" + of + "' is a field." + field.getSubType() + + (field.hasMetaAttr(MetaObject.ATTR_OBJECT_REF) ? " with an @objectRef" : "") + + ", which a report cannot group by. Group by a scalar field of it."); return; } @@ -590,6 +610,21 @@ private static String stringAttr(MetaData node, String name) { return v instanceof String ? (String) v : null; } + /** A field.object, or any field carrying an {@code @objectRef}: a JSON document, not a scalar. */ + private static boolean isDocumentField(MetaData field) { + return com.metaobjects.field.ObjectField.SUBTYPE_OBJECT.equals(field.getSubType()) + || field.hasMetaAttr(MetaObject.ATTR_OBJECT_REF); + } + + private static boolean isIntegralNumber(Object v) { + if (v instanceof Integer || v instanceof Long || v instanceof Short || v instanceof Byte) return true; + if (v instanceof Number n) { + double d = n.doubleValue(); + return !Double.isNaN(d) && !Double.isInfinite(d) && d == Math.rint(d); + } + return false; + } + private static void checkMeasure(MemberCtx ctx, MetaMeasure measure) { Consumer err = message -> ctx.sink().push(measure, ErrorCode.ERR_INVALID_MEASURE, ctx.label(), ": " + message, ctx.suffix()); @@ -603,8 +638,9 @@ private static void checkMeasure(MemberCtx ctx, MetaMeasure measure) { boolean declaresDefault = measure.hasMetaAttr(ReportingConstants.ATTR_DEFAULT); Object declaredDefault = declaresDefault ? measure.getMetaAttr(ReportingConstants.ATTR_DEFAULT).getValue() : null; - boolean fractionalDefault = declaredDefault instanceof Number - && measure.getDefaultValue() == null; + // Rule B (cross-port): M7-M9 run only for a valid integral number; anything else is + // already (or will be) the attribute error alone. + boolean fractionalDefault = declaresDefault && !isIntegralNumber(declaredDefault); if (measure.isRatio()) { checkRatioOperands(ctx, measure, err); @@ -700,6 +736,14 @@ private static AggregateColumns checkAggregateColumns(MemberCtx ctx, MetaMeasure + ctx.declaring().getName() + "'."); return null; } + // M10 — a measure aggregates scalars. A field.object (or a field carrying @objectRef) + // is a JSON document, which no port aggregates in a report. + if (isDocumentField(field)) { + err.accept("@of '" + item + "' is a field." + field.getSubType() + + (field.hasMetaAttr(MetaObject.ATTR_OBJECT_REF) ? " with an @objectRef" : "") + + ", which a report cannot aggregate. Aggregate a scalar field of it."); + return null; + } fields.add(field); } @@ -872,6 +916,31 @@ private static void checkReport(MetaRoot root, MetaObject report, ErrorSink sink } } + // R10 — a report is read through a view. The other read-only kinds are not read by a + // plain SELECT, and no port lowers or serves them. A view or materialized view the adopter + // creates is declared `@kind: view` with `@unmanaged: true`. + // ADR-0039: resolving, as R5 reads it. + for (MetaData child : report.getChildren(MetaData.class, true)) { + if (child instanceof MetaSource src && src.isReadOnly() + && !MetaSource.KIND_VIEW.equals(src.getEffectiveKind())) { + sink.push(child, ErrorCode.ERR_INVALID_REPORT, label, + ": " + childLabel(child) + " has @kind '" + src.getEffectiveKind() + + "'; a report is read through a view, so its source must be @kind: view. " + + "For a view or materialized view you create yourself, declare @kind: view " + + "with @unmanaged: true."); + } + } + + // R11 — a report cannot be abstract: its fields are derived from its own @dimensions and + // @measures, no port reads or serves an abstract report, and report inheritance is not + // part of the vocabulary. + if (com.metaobjects.io.util.IOUtil.isAbstract(report)) { + sink.push(report, ErrorCode.ERR_INVALID_REPORT, label, + ": is abstract; a report cannot be abstract, because its fields are derived from its " + + "own @dimensions and @measures and no port reads or serves an abstract report. " + + "Remove abstract: true."); + } + // R1 — @from resolves to an object.entity. Without it, R2/R3/R6/R7, R8/R9 and // the @filter have nothing to resolve against, so they are skipped. String fromRef = ReportAccessors.reportFrom(report); diff --git a/server/java/metadata/src/main/java/com/metaobjects/loader/ValidationPhase.java b/server/java/metadata/src/main/java/com/metaobjects/loader/ValidationPhase.java index 018f7a11a..d252e72e3 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/loader/ValidationPhase.java +++ b/server/java/metadata/src/main/java/com/metaobjects/loader/ValidationPhase.java @@ -4223,21 +4223,37 @@ public static MetaObject resolveRootObject(MetaRoot root, String ref, String ref * cannot word it differently. It is not the only one: {@code RegisteredValidation} holds * a second, private builder of the same sentence, which does not strip a dotted tail.

    */ public static String didYouMeanHint(MetaRoot root, String ref) { - if (ref == null) return ""; + List found = sameNameObjects(root, ref); + if (found.isEmpty()) return ""; + return " An object named \"" + shortNameOf(ref) + "\" exists in: " + String.join(", ", found) + + ". Qualify it with its package (FQN)."; + } + + /** The did-you-mean hint for an unresolved {@code extends}: names the qualified spelling to write. */ + public static String didYouMeanExtendsHint(MetaRoot root, String ref) { + List found = sameNameObjects(root, ref); + if (found.isEmpty()) return ""; + String spelled = new java.util.ArrayList<>(found).stream() + .map(f -> "extends: \"" + f + "\"").collect(java.util.stream.Collectors.joining(" or ")); + return " An object named \"" + shortNameOf(ref) + "\" exists in: " + String.join(", ", found) + + ". Qualify it with its package (FQN): write " + spelled + "."; + } + + private static String shortNameOf(String ref) { int sep = ref.lastIndexOf(MetaData.PKG_SEPARATOR); String shortName = (sep >= 0) ? ref.substring(sep + MetaData.PKG_SEPARATOR.length()) : ref; int dot = shortName.indexOf('.'); - if (dot >= 0) shortName = shortName.substring(0, dot); - StringBuilder candidates = new StringBuilder(); + return dot >= 0 ? shortName.substring(0, dot) : shortName; + } + + private static List sameNameObjects(MetaRoot root, String ref) { + List out = new java.util.ArrayList<>(); + if (ref == null) return out; + String shortName = shortNameOf(ref); for (MetaData child : root.getChildren(MetaData.class, false)) { - if (!(child instanceof MetaObject)) continue; - if (!shortName.equals(child.getShortName())) continue; - if (candidates.length() > 0) candidates.append(", "); - candidates.append(child.getName()); + if (child instanceof MetaObject && shortName.equals(child.getShortName())) out.add(child.getName()); } - if (candidates.length() == 0) return ""; - return " An object named \"" + shortName + "\" exists in: " + candidates - + ". Qualify it with its package (FQN)."; + return out; } /** diff --git a/server/java/metadata/src/main/java/com/metaobjects/loader/parser/json/CanonicalJsonParser.java b/server/java/metadata/src/main/java/com/metaobjects/loader/parser/json/CanonicalJsonParser.java index 272b39c46..e4fc2442c 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/loader/parser/json/CanonicalJsonParser.java +++ b/server/java/metadata/src/main/java/com/metaobjects/loader/parser/json/CanonicalJsonParser.java @@ -879,7 +879,12 @@ private void processNode(MetaData parent, String type, String subType, // redundantly re-authored package (matches the TS / Python oracles). // (isRoot here just means "direct child of metadata.root"; the doc // root's own package is tracked separately via setDefaultPackageName.) - if (pkg != null) { + // A merge into a node an earlier document already declared leaves that node's own + // authored-package flag alone: the overlay's package places the overlay, it does not + // re-author the target (TS, C# and Python never write it onto the merged target). + boolean mergesIntoDeclared = md != null && (md.getSource() instanceof JsonSource + || md.getSource() instanceof YamlSource || md.getSource() instanceof MergedSource); + if (pkg != null && !mergesIntoDeclared) { md.setPackageAuthored(true); } diff --git a/server/java/metadata/src/main/java/com/metaobjects/reporting/ReportReadModel.java b/server/java/metadata/src/main/java/com/metaobjects/reporting/ReportReadModel.java index 33dc1d0c2..25216933f 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/reporting/ReportReadModel.java +++ b/server/java/metadata/src/main/java/com/metaobjects/reporting/ReportReadModel.java @@ -87,6 +87,7 @@ public final class ReportReadModel extends ReportMetaObject MetaField.ATTR_SCALE, CoreDBMetaDataProvider.LOCAL_TIME, MetaField.ATTR_OBJECT_REF, + MetaField.ATTR_VALUE_TYPE, // a field.map dimension's scalar value type MetaField.ATTR_STORAGE); /** The declared report this model was built from; {@code null} only on a bare instance. */ @@ -225,7 +226,9 @@ private static MetaField derivedField(ReportShape.Field f) { // a dimension). Set on this detached model only: no vocabulary is added and the // declared tree is not touched. Read after the type shape is carried, because the // band is field-level (an int-backed enum drops `like`), and resolving (ADR-0039). - if (!FilterOps.opsForField(field).isEmpty()) { + // A derived field over an array (isArray) or a map is neither filterable nor sortable, + // in every port: an array column has no scalar to compare, a map no order. + if (!field.isArrayType() && !(field instanceof com.metaobjects.field.MapField) && !FilterOps.opsForField(field).isEmpty()) { field.addMetaAttr(BooleanAttribute.create(MetaField.ATTR_FILTERABLE, true)); } return field; diff --git a/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java b/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java index 82d5d86ee..d14478ce4 100644 --- a/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java +++ b/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java @@ -101,4 +101,11 @@ public void schemaVersionNonIntegralRaisesRatherThanTruncating() throws IOExcept MetaDataException ex = assertThrows(MetaDataException.class, () -> NeutralConfig.read(dir)); assertEquals(ErrorCode.ERR_BAD_ATTR_VALUE, ex.getCode().orElseThrow()); } + + @Test + public void anUnknownTopLevelKeyIsIgnoredNotRefused() { + NeutralConfig cfg = NeutralConfig.parse( + "{\"schema_version\":1,\"someFutureKey\":{\"x\":1},\"naming\":{\"columns\":\"literal\"}}", "test"); + assertEquals("literal", cfg.getPhysicalNaming().columns()); + } } diff --git a/server/java/metadata/src/test/java/com/metaobjects/conformance/ConformanceTest.java b/server/java/metadata/src/test/java/com/metaobjects/conformance/ConformanceTest.java index 4eac912cd..f06708063 100644 --- a/server/java/metadata/src/test/java/com/metaobjects/conformance/ConformanceTest.java +++ b/server/java/metadata/src/test/java/com/metaobjects/conformance/ConformanceTest.java @@ -665,6 +665,14 @@ private static void assertEnvelopeAlignment(String label, + "', got '" + g.code + "'"); continue; } + for (String needle : w.messageIncludes) { + if (g.message == null) { + failures.add(label + "[" + i + "].messageIncludes: this adapter exposes no message"); + } else if (!g.message.contains(needle)) { + failures.add(label + "[" + i + "].messageIncludes: message lacks '" + needle + + "': " + g.message); + } + } if (w.source == null) continue; if (!w.source.format.equals(g.format)) { failures.add(label + "[" + i + "].source.format: expected '" @@ -800,6 +808,8 @@ private static final class EnvelopeRecord { // FR5d — populated for format=resolved envelopes; null otherwise. final String referrer; final String target; + /** The actual error message; null when the record is not built from an exception. */ + String message; EnvelopeRecord(String code, String format, List files, String jsonPath, String referrer, String target) { this.code = code; @@ -820,7 +830,9 @@ private static final class EnvelopeRecord { * cross-port harness has a portable file token. */ private static EnvelopeRecord buildEnvelope(MetaDataException ex, Path inputDir) { - return buildEnvelope(extractErrorCode(ex), ex.getEnvelope().orElse(null), inputDir); + EnvelopeRecord rec = buildEnvelope(extractErrorCode(ex), ex.getEnvelope().orElse(null), inputDir); + rec.message = ex.getMessage(); + return rec; } /** diff --git a/server/java/metadata/src/test/java/com/metaobjects/conformance/FixtureLint.java b/server/java/metadata/src/test/java/com/metaobjects/conformance/FixtureLint.java index 428a8a1cc..83bd3f396 100644 --- a/server/java/metadata/src/test/java/com/metaobjects/conformance/FixtureLint.java +++ b/server/java/metadata/src/test/java/com/metaobjects/conformance/FixtureLint.java @@ -130,9 +130,15 @@ public ExpectedErrorSource(String format, List files, String jsonPath, public static final class ExpectedError { public final String code; public final ExpectedErrorSource source; // nullable in legacy shape + /** Case-sensitive substrings the actual message must contain (spec/conformance-tests.md); empty = none. */ + public final List messageIncludes; public ExpectedError(String code, ExpectedErrorSource source) { + this(code, source, List.of()); + } + public ExpectedError(String code, ExpectedErrorSource source, List messageIncludes) { this.code = code; this.source = source; + this.messageIncludes = messageIncludes; } } @@ -287,7 +293,22 @@ private static List parseEnvelopeEntries(JsonArray arr, String la } source = new ExpectedErrorSource(fmt, files, jsonPath, referrer, target); } - out.add(new ExpectedError(code.getAsString(), source)); + List includes = new ArrayList<>(); + JsonElement incEl = obj.get("messageIncludes"); + if (incEl != null) { + if (!incEl.isJsonArray() || incEl.getAsJsonArray().isEmpty()) { + throw new IllegalArgumentException( + "expected-errors.json " + label + " entry " + idx + " messageIncludes must be a non-empty string[]"); + } + for (JsonElement ie : incEl.getAsJsonArray()) { + if (!ie.isJsonPrimitive() || !ie.getAsJsonPrimitive().isString() || ie.getAsString().isEmpty()) { + throw new IllegalArgumentException( + "expected-errors.json " + label + " entry " + idx + " messageIncludes must hold non-empty strings"); + } + includes.add(ie.getAsString()); + } + } + out.add(new ExpectedError(code.getAsString(), source, includes)); idx++; } return out; diff --git a/server/java/metadata/src/test/java/com/metaobjects/loader/FromUrisPhysicalNamingTest.java b/server/java/metadata/src/test/java/com/metaobjects/loader/FromUrisPhysicalNamingTest.java new file mode 100644 index 000000000..f5ec6d184 --- /dev/null +++ b/server/java/metadata/src/test/java/com/metaobjects/loader/FromUrisPhysicalNamingTest.java @@ -0,0 +1,116 @@ +package com.metaobjects.loader; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import com.metaobjects.database.ColumnNaming; +import com.metaobjects.database.PhysicalNaming; +import com.metaobjects.registry.SharedRegistryTestBase; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.net.URI; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.stream.Collectors; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +/** + * ADR-0058 — {@link MetaDataLoader#fromUris} applies the {@code naming} of the + * {@code .metaobjects/config.json} it finds walking up from the first {@code file:} URI, + * warns once per loader when it finds none, and never overrides an explicit rule set after. + */ +public class FromUrisPhysicalNamingTest extends SharedRegistryTestBase { + + private static final String TINY = "{\"metadata.root\":{\"package\":\"x\",\"children\":[]}}"; + private final ListAppender appender = new ListAppender<>(); + private final Logger logger = (Logger) LoggerFactory.getLogger(MetaDataLoader.class); + + @Before public void attach() { + appender.start(); + logger.addAppender(appender); + logger.setLevel(Level.WARN); + } + + @After public void detach() { + logger.detachAppender(appender); + } + + private List warnings() { + return appender.list.stream() + .filter(e -> e.getLevel() == Level.WARN && e.getFormattedMessage().contains("physical naming")) + .map(ILoggingEvent::getFormattedMessage).collect(Collectors.toList()); + } + + private static Path project(String config, String... subdirs) throws IOException { + Path root = Files.createTempDirectory("naming-proj-"); + if (config != null) { + Files.createDirectories(root.resolve(".metaobjects")); + Files.writeString(root.resolve(".metaobjects/config.json"), config); + } + Path dir = root; + for (String s : subdirs) dir = dir.resolve(s); + Files.createDirectories(dir); + Files.writeString(dir.resolve("meta.tiny.json"), TINY); + return root; + } + + private static URI first(Path root, String... subdirs) { + Path dir = root; + for (String s : subdirs) dir = dir.resolve(s); + return dir.resolve("meta.tiny.json").toUri(); + } + + @Test public void findsConfigByWalkingUpFromTheFirstFileUri() throws IOException { + Path root = project("{\"schema_version\":1,\"naming\":{\"columns\":\"literal\"}}", "metaobjects", "deep"); + MetaDataLoader loader = MetaDataLoader.fromUris("walkUp", List.of(first(root, "metaobjects", "deep"))); + assertEquals(ColumnNaming.LITERAL, loader.getRoot().getPhysicalNaming().columns()); + assertTrue(warnings().isEmpty()); + } + + @Test public void warnsOnceAndUsesSnakeCaseWhenNoConfigIsFound() throws IOException { + Path root = project(null, "metaobjects"); + MetaDataLoader loader = MetaDataLoader.fromUris("noConfig", List.of(first(root, "metaobjects"))); + assertEquals(PhysicalNaming.DEFAULT.columns(), loader.getRoot().getPhysicalNaming().columns()); + List w = warnings(); + assertEquals(1, w.size()); + assertTrue(w.get(0), w.get(0).contains("metaobjects.physical-naming-config")); + assertTrue(w.get(0), w.get(0).contains("naming")); + } + + @Test public void anExplicitRuleSetAfterTheLoadWins() throws IOException { + Path root = project("{\"schema_version\":1,\"naming\":{\"columns\":\"literal\"}}", "m"); + MetaDataLoader loader = MetaDataLoader.fromUris("explicit", List.of(first(root, "m"))); + loader.getRoot().setPhysicalNaming(PhysicalNaming.of("kebab-case", "snake_case")); + assertEquals(ColumnNaming.KEBAB_CASE, loader.getRoot().getPhysicalNaming().columns()); + } + + @Test public void aCallerSuppliedRuleSkipsDiscoveryAndTheWarning() throws IOException { + Path root = project(null, "m"); + MetaDataLoader loader = MetaDataLoader.fromUris("supplied", List.of(first(root, "m")), null, + PhysicalNaming.of("literal", "snake_case")); + assertEquals(ColumnNaming.LITERAL, loader.getRoot().getPhysicalNaming().columns()); + assertTrue(warnings().isEmpty()); + } + + @Test public void fromDirectoryDiscoversNamingTheSameWay() throws IOException { + Path root = project("{\"schema_version\":1,\"naming\":{\"columns\":\"literal\"}}", "metaobjects"); + MetaDataLoader loader = MetaDataLoader.fromDirectory("dirWalk", root.resolve("metaobjects")); + assertEquals(ColumnNaming.LITERAL, loader.getRoot().getPhysicalNaming().columns()); + assertTrue(warnings().isEmpty()); + } + + @Test public void fromDirectoryWarnsOnceWhenNoConfigIsFound() throws IOException { + Path root = project(null, "metaobjects"); + MetaDataLoader loader = MetaDataLoader.fromDirectory("dirNone", root.resolve("metaobjects")); + assertEquals(PhysicalNaming.DEFAULT.columns(), loader.getRoot().getPhysicalNaming().columns()); + assertEquals(1, warnings().size()); + } +} diff --git a/server/java/metadata/src/test/java/com/metaobjects/reporting/ReportReadModelTest.java b/server/java/metadata/src/test/java/com/metaobjects/reporting/ReportReadModelTest.java index d489c7155..2010e26c0 100644 --- a/server/java/metadata/src/test/java/com/metaobjects/reporting/ReportReadModelTest.java +++ b/server/java/metadata/src/test/java/com/metaobjects/reporting/ReportReadModelTest.java @@ -348,7 +348,7 @@ public void isDetachedAndLeavesTheLoadedModelUntouched() { for (MetaObject o : canonical.getChildren(MetaObject.class, false)) { if (ReportReadModel.isReport(o)) models.add(ReportReadModel.of(o, canonical)); } - assertEquals(23, models.size()); + assertEquals(25, models.size()); for (ReportReadModel model : models) { assertNull("never attached", model.getParent()); @@ -396,18 +396,15 @@ public void isCachedPerReportNodeAndIdempotent() { """; @Test - public void aDimensionOverAFieldObjectIsRefusedByName() { - MetaRoot root = loadJson(OBJECT_DIMENSION_MODEL); - MetaObject report = object(root, "SalesByDestination"); - // The shape still derives (the loader accepts the model); the read model refuses. - assertEquals("object", ReportShape.of(report, root).fields().get(0).subType()); + public void aDimensionOverAFieldObjectIsRefusedAtLoadByName() { + // D5: the loader refuses it before any read model is built (the read model's own + // refusal stays as the backstop for a tree built in code). try { - ReportReadModel.of(report, root); + loadJson(OBJECT_DIMENSION_MODEL); fail("a report over a field.object must be refused"); } catch (MetaDataException e) { - assertEquals("report 'SalesByDestination': dimension 'destination' reads 'shop::Sale.shipTo'," - + " a field.object. A report over a field.object is not supported; group by a scalar field.", - e.getMessage()); + assertEquals(com.metaobjects.ErrorCode.ERR_INVALID_DIMENSION, e.getCode().orElseThrow()); + assertTrue(e.getMessage(), e.getMessage().contains("@of 'Sale.shipTo' is a field.object")); } } } diff --git a/server/java/omdb/pom.xml b/server/java/omdb/pom.xml index 1a1b063c4..4b4b4d1ae 100644 --- a/server/java/omdb/pom.xml +++ b/server/java/omdb/pom.xml @@ -47,6 +47,13 @@ ${derby.version} test + + + org.xerial + sqlite-jdbc + 3.46.1.3 + test + diff --git a/server/java/omdb/src/main/java/com/metaobjects/manager/db/driver/GenericSQLDriver.java b/server/java/omdb/src/main/java/com/metaobjects/manager/db/driver/GenericSQLDriver.java index aecd89f0b..e15def1cd 100644 --- a/server/java/omdb/src/main/java/com/metaobjects/manager/db/driver/GenericSQLDriver.java +++ b/server/java/omdb/src/main/java/com/metaobjects/manager/db/driver/GenericSQLDriver.java @@ -934,6 +934,15 @@ protected String getOrderString(MetaObject mc, ObjectMappingDB omdb, SortOrder o /** * Gets the SQL WHERE clause for the fields of a class */ + /** + * The SQL for a filter operand compared to {@code f}'s column: a bare {@code ?} by default. + * A dialect whose comparison semantics depend on the operand's bind type overrides it + * (MySQL casts a decimal operand, see {@code MySQLDriver}). + */ + protected String comparisonPlaceholder(MetaField f) { + return "?"; + } + protected String getExpressionString(MetaObject mc, ObjectMappingDB omdb, Expression exp, ArrayList args, String prefix) throws MetaDataException { StringBuilder set = new StringBuilder(); @@ -1032,7 +1041,7 @@ protected String getExpressionString(MetaObject mc, ObjectMappingDB omdb, Expres } else { set.append(","); } - set.append("?"); + set.append(comparisonPlaceholder(f)); args.add(new SQLArg(f, o)); } @@ -1041,22 +1050,22 @@ protected String getExpressionString(MetaObject mc, ObjectMappingDB omdb, Expres // Add these on as arguments switch (c) { case Expression.EQUAL: - set.append("= ?"); + set.append("= " + comparisonPlaceholder(f)); break; case Expression.NOT_EQUAL: - set.append("<> ?"); + set.append("<> " + comparisonPlaceholder(f)); break; case Expression.GREATER: - set.append("> ?"); + set.append("> " + comparisonPlaceholder(f)); break; case Expression.LESSER: - set.append("< ?"); + set.append("< " + comparisonPlaceholder(f)); break; case Expression.EQUAL_GREATER: - set.append(">= ?"); + set.append(">= " + comparisonPlaceholder(f)); break; case Expression.EQUAL_LESSER: - set.append("<= ?"); + set.append("<= " + comparisonPlaceholder(f)); break; // Case-sensitive, verbatim pattern — the column is NOT UPPER-wrapped diff --git a/server/java/omdb/src/main/java/com/metaobjects/manager/db/driver/MySQLDriver.java b/server/java/omdb/src/main/java/com/metaobjects/manager/db/driver/MySQLDriver.java index 785efe2ac..3f56ea6b4 100644 --- a/server/java/omdb/src/main/java/com/metaobjects/manager/db/driver/MySQLDriver.java +++ b/server/java/omdb/src/main/java/com/metaobjects/manager/db/driver/MySQLDriver.java @@ -140,6 +140,16 @@ protected String quoteIdent(String name) { return "`" + name + "`"; } + /** + * A decimal operand is cast: MySQL compares {@code DECIMAL = 'text'} (and a bound parameter + * travels as text) as DOUBLES, so {@code 12345678901234567.89} would equal a neighbouring + * value. {@code CAST(? AS DECIMAL(65,30))} keeps the comparison exact (ADR-0019 Amendment 1). + */ + @Override + protected String comparisonPlaceholder(MetaField f) { + return f instanceof com.metaobjects.field.DecimalField ? "CAST(? AS DECIMAL(65,30))" : "?"; + } + /** * Timestamp and uuid binds that Connector/J accepts. The shared codecs bind a timestamp as * {@code TIMESTAMP_WITH_TIMEZONE} and a uuid as {@code Types.OTHER}; Connector/J rejects diff --git a/server/java/omdb/src/main/java/com/metaobjects/manager/db/driver/SQLiteDriver.java b/server/java/omdb/src/main/java/com/metaobjects/manager/db/driver/SQLiteDriver.java new file mode 100644 index 000000000..c04c1da0e --- /dev/null +++ b/server/java/omdb/src/main/java/com/metaobjects/manager/db/driver/SQLiteDriver.java @@ -0,0 +1,280 @@ +/* + * Copyright 2026 Doug Mealing LLC dba Meta Objects + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.metaobjects.manager.db.driver; + +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.sql.Statement; +import java.sql.Timestamp; +import java.sql.Types; +import java.time.Instant; +import java.time.LocalDate; +import java.time.LocalDateTime; +import java.time.LocalTime; +import java.time.ZoneOffset; +import java.time.format.DateTimeFormatter; +import java.time.format.DateTimeParseException; +import java.util.Locale; + +import com.metaobjects.MetaDataException; +import com.metaobjects.database.CoreDBMetaDataProvider; +import com.metaobjects.field.DateField; +import com.metaobjects.field.DecimalField; +import com.metaobjects.field.MetaField; +import com.metaobjects.field.TimeField; +import com.metaobjects.field.TimestampField; +import com.metaobjects.manager.db.defs.ColumnDef; +import com.metaobjects.manager.exp.Range; + +/** + * OMDB driver for SQLite (and, since D1 is SQLite, the same dialect). + * + *

    MetaObjects owns the SQLite schema ({@code meta migrate --dialect sqlite}); this driver is the + * runtime half: CRUD, query and paging against tables the migration created. It needs only + * {@code java.sql}, so it works with any SQLite JDBC driver; the adopter supplies one. SQLite has + * no date, time or timestamp storage class, and {@code meta migrate} declares those columns + * {@code TEXT}, so this driver pins the spelling rather than leaving it to the JDBC driver's + * session settings: + *

      + *
    • {@code field.timestamp} is ISO-8601 text, {@code yyyy-MM-dd'T'HH:mm:ss.SSS'Z'} for an + * instant and the same without the {@code Z} for {@code @localTime}. Read accepts that, + * a space instead of the {@code T}, no fraction, and an integer of epoch milliseconds.
      • + *
    • {@code field.date} is {@code yyyy-MM-dd} text and {@code field.time} is ISO local-time + * text, so neither depends on the JVM time zone.
      • + *
    • {@code field.uuid} is lower-case text.
      • + *
    • {@code field.decimal} is read from its text: an INTEGER exactly, a REAL to 15 significant digits + * (ADR-0019 Amendment 1, W4: {@code 0.1 + 0.2} reads {@code 0.3}), and bound as a number (an integer or a double), because a text operand never compares + * equal or greater to a stored number in SQLite.
      • + *
    • A boolean is the integer 0/1 SQLite stores it as.
      • + *
    + * Other differences from the generic driver: an {@code AUTOINCREMENT} key comes back through + * {@code last_insert_rowid()}, identifiers are double-quoted, paging is {@code LIMIT .. OFFSET ..}, + * and there is no row-lock clause (SQLite locks the database file). + * + * @since 8.1.0 + */ +public class SQLiteDriver extends GenericSQLDriver { + + private static final DateTimeFormatter INSTANT_TEXT = + DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", Locale.ROOT); + private static final DateTimeFormatter WALL_CLOCK_TEXT = + DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSS", Locale.ROOT); + + public SQLiteDriver() { + super(); + } + + /** The key an {@code INTEGER PRIMARY KEY AUTOINCREMENT} column just generated on this connection. */ + @Override + protected String getLastAutoId(Connection conn, ColumnDef col) throws SQLException { + try (Statement s = conn.createStatement(); + ResultSet rs = s.executeQuery("SELECT last_insert_rowid()")) { + if (!rs.next()) { + throw new SQLException("Unable to get last id for column [" + col + "], no result in result set"); + } + String id = rs.getString(1); + if (id == null || "0".equals(id)) { + throw new SQLException("last_insert_rowid() returned no key for column [" + col + "]"); + } + return id; + } + } + + @Override + protected String quoteIdent(String name) { + if (name.indexOf('"') >= 0) throw new IllegalArgumentException("unsafe identifier: " + name); + return "\"" + name + "\""; + } + + @Override + protected boolean supportsRangeInQuery() { + return true; + } + + /** {@code LIMIT n [OFFSET m]}; SQLite requires LIMIT whenever OFFSET is present. */ + @Override + public String getRangeString(Range range) { + StringBuilder b = new StringBuilder("LIMIT ").append((range.getEnd() - range.getStart()) + 1); + if (range.getStart() > 1) b.append(" OFFSET ").append(range.getStart() - 1); + return b.toString(); + } + + /** SQLite has no row locks: a write locks the database, so the clause is empty. */ + @Override + public String getLockString() throws MetaDataException { + return ""; + } + + @Override + public String getDateFormat() { + return "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"; + } + + @Override + protected void setStatementValue(PreparedStatement s, MetaField f, int index, Object value) throws SQLException { + if (f instanceof TimestampField) { + if (value == null) s.setNull(index, Types.VARCHAR); + else s.setString(index, timestampText(f, value)); + return; + } + if (f instanceof DateField) { + if (value == null) s.setNull(index, Types.VARCHAR); + else s.setString(index, dateText(value)); + return; + } + if (f instanceof DecimalField) { + // sqlite-jdbc binds a BigDecimal as text, and SQLite orders every number before any + // text, so `realColumn > '0'` is false for every row. Bind a number: an integral + // value as an integer, anything else as the double SQLite stores a REAL as anyway. + if (value == null) { + s.setNull(index, Types.NUMERIC); + } else { + java.math.BigDecimal d = value instanceof java.math.BigDecimal bd ? bd : new java.math.BigDecimal(value.toString()); + if (d.signum() == 0 || d.stripTrailingZeros().scale() <= 0 && d.precision() - d.scale() <= 18) { + s.setLong(index, d.longValueExact()); + } else { + s.setDouble(index, d.doubleValue()); + } + } + return; + } + if (f instanceof TimeField) { + if (value == null) s.setNull(index, Types.VARCHAR); + else s.setString(index, value instanceof LocalTime lt ? lt.toString() : LocalTime.parse(value.toString()).toString()); + return; + } + if (isUuidColumn(f) && !isArrayField(f)) { + if (value == null) s.setNull(index, Types.VARCHAR); + else s.setString(index, value.toString().toLowerCase(Locale.ROOT)); + return; + } + super.setStatementValue(s, f, index, value); + } + + @Override + protected void parseField(Object o, MetaField f, ResultSet rs, int j) throws SQLException { + if (f instanceof TimestampField) { + String text = rs.getString(j); + f.setDate(o, text == null ? null : Timestamp.from(parseInstant(text))); + return; + } + if (f instanceof DateField) { + String text = rs.getString(j); + f.setDate(o, text == null ? null + : new java.util.Date(parseDate(text).atStartOfDay(ZoneOffset.UTC).toInstant().toEpochMilli())); + return; + } + if (f instanceof TimeField) { + String text = rs.getString(j); + f.setObject(o, text == null ? null : LocalTime.parse(text)); + return; + } + if (f instanceof DecimalField) { + // sqlite-jdbc's getBigDecimal skips the null bookkeeping wasNull() relies on, and a + // NUMERIC column may hold an integer, a real or text: read the text, which is exact. + String text = rs.getString(j); + f.setObject(o, text == null ? null : decimalFromStoredText(text.trim())); + return; + } + super.parseField(o, f, rs, j); + } + + /** The significant digits SQLite keeps of a stored REAL (ADR-0019 Amendment 1, W4). */ + private static final int SQLITE_SIGNIFICANT_DIGITS = 15; + + /** + * ADR-0019 Amendment 1, W4: SQLite has no decimal type, and a NUMERIC column holds an INTEGER + * (exact) or a REAL (a double). A REAL is read as its value to 15 significant digits, SQLite's + * own text precision, so {@code 0.1 + 0.2} reads {@code 0.3} and not + * {@code 0.30000000000000004}. An integer-looking value is exact. + */ + static java.math.BigDecimal decimalFromStoredText(String text) { + boolean integral = text.indexOf('.') < 0 && text.indexOf('e') < 0 && text.indexOf('E') < 0; + if (integral) return new java.math.BigDecimal(text); + double d; + try { + d = Double.parseDouble(text); + } catch (NumberFormatException notANumber) { + return new java.math.BigDecimal(text); // text SQLite could not convert: say so + } + if (Double.isNaN(d) || Double.isInfinite(d)) return new java.math.BigDecimal(text); + if (d == 0d) return java.math.BigDecimal.ZERO; + java.math.BigDecimal rounded = new java.math.BigDecimal(d) + .round(new java.math.MathContext(SQLITE_SIGNIFICANT_DIGITS, java.math.RoundingMode.HALF_EVEN)); + return rounded.signum() == 0 ? java.math.BigDecimal.ZERO : rounded.stripTrailingZeros(); + } + + private static boolean isArrayField(MetaField f) { + return f.isArrayType(); + } + + private static boolean isLocalTime(MetaField f) { + try { + return f.hasMetaAttr(CoreDBMetaDataProvider.LOCAL_TIME) + && Boolean.parseBoolean(String.valueOf(f.getMetaAttr(CoreDBMetaDataProvider.LOCAL_TIME).getValue()).trim()); + } catch (Exception e) { + return false; + } + } + + private static String timestampText(MetaField f, Object value) { + Instant at; + if (value instanceof LocalDateTime ldt) at = ldt.toInstant(ZoneOffset.UTC); + else if (value instanceof java.util.Date d) at = Instant.ofEpochMilli(d.getTime()); + else if (value instanceof java.time.OffsetDateTime odt) at = odt.toInstant(); + else if (value instanceof Instant i) at = i; + else at = Instant.ofEpochMilli(Long.parseLong(value.toString())); + return isLocalTime(f) + ? WALL_CLOCK_TEXT.format(at.atOffset(ZoneOffset.UTC)) + : INSTANT_TEXT.format(at.atOffset(ZoneOffset.UTC)); + } + + private static String dateText(Object value) { + if (value instanceof LocalDate ld) return ld.toString(); + if (value instanceof java.util.Date d) { + return Instant.ofEpochMilli(d.getTime()).atZone(ZoneOffset.UTC).toLocalDate().toString(); + } + String text = value.toString(); + try { + return parseDate(text).toString(); + } catch (DateTimeParseException e) { + return Instant.ofEpochMilli(Long.parseLong(text)).atZone(ZoneOffset.UTC).toLocalDate().toString(); + } + } + + private static LocalDate parseDate(String text) { + String t = text.trim(); + return LocalDate.parse(t.length() > 10 ? t.substring(0, 10) : t); + } + + /** Reads the spellings a SQLite TEXT or INTEGER timestamp column holds; no zone means UTC. */ + static Instant parseInstant(String text) { + String t = text.trim(); + if (t.matches("-?\\d+")) return Instant.ofEpochMilli(Long.parseLong(t)); + String iso = t.indexOf('T') < 0 && t.length() > 10 ? t.substring(0, 10) + "T" + t.substring(11) : t; + if (iso.endsWith("Z") || iso.matches(".*[+-]\\d{2}:\\d{2}$")) { + return java.time.OffsetDateTime.parse(iso).toInstant(); + } + return LocalDateTime.parse(iso).toInstant(ZoneOffset.UTC); + } + + @Override + public String toString() { + return "SQLite Database Driver"; + } +} diff --git a/server/java/omdb/src/test/java/com/metaobjects/manager/db/driver/SQLiteDriverAutoIdTest.java b/server/java/omdb/src/test/java/com/metaobjects/manager/db/driver/SQLiteDriverAutoIdTest.java new file mode 100644 index 000000000..0a7091363 --- /dev/null +++ b/server/java/omdb/src/test/java/com/metaobjects/manager/db/driver/SQLiteDriverAutoIdTest.java @@ -0,0 +1,47 @@ +package com.metaobjects.manager.db.driver; + +import com.metaobjects.manager.db.defs.ColumnDef; +import com.metaobjects.manager.db.defs.NameDef; +import com.metaobjects.manager.db.defs.TableDef; +import org.junit.Test; + +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.SQLException; +import java.sql.Statement; +import java.sql.Types; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertThrows; + +/** SQLiteDriver reads the key an AUTOINCREMENT column just generated, on the inserting connection. */ +public class SQLiteDriverAutoIdTest { + + private static final class TestDriver extends SQLiteDriver { + String lastAutoId(Connection c, ColumnDef col) throws SQLException { return getLastAutoId(c, col); } + } + + private static ColumnDef idColumn() { + ColumnDef col = new ColumnDef("id", Types.BIGINT); + col.setBaseTable(new TableDef(new NameDef(null, "things"))); + return col; + } + + @Test + public void lastAutoIdIsTheKeyJustInserted() throws Exception { + try (Connection c = DriverManager.getConnection("jdbc:sqlite::memory:"); + Statement s = c.createStatement()) { + s.execute("CREATE TABLE things (id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, name TEXT)"); + s.execute("INSERT INTO things (name) VALUES ('a')"); + s.execute("INSERT INTO things (name) VALUES ('b')"); + assertEquals("2", new TestDriver().lastAutoId(c, idColumn())); + } + } + + @Test + public void lastAutoIdWithNoInsertIsRefused() throws Exception { + try (Connection c = DriverManager.getConnection("jdbc:sqlite::memory:")) { + assertThrows(SQLException.class, () -> new TestDriver().lastAutoId(c, idColumn())); + } + } +} diff --git a/server/java/omdb/src/test/java/com/metaobjects/manager/db/driver/SQLiteDriverPagingTest.java b/server/java/omdb/src/test/java/com/metaobjects/manager/db/driver/SQLiteDriverPagingTest.java new file mode 100644 index 000000000..6fffae415 --- /dev/null +++ b/server/java/omdb/src/test/java/com/metaobjects/manager/db/driver/SQLiteDriverPagingTest.java @@ -0,0 +1,50 @@ +package com.metaobjects.manager.db.driver; + +import com.metaobjects.manager.exp.Range; +import org.junit.Test; + +import java.time.Instant; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertThrows; + +/** SQLiteDriver's paging clause, identifier quoting and timestamp text, with no database. */ +public class SQLiteDriverPagingTest { + + /** Exposes the protected identifier hook. */ + private static final class TestDriver extends SQLiteDriver { + String quote(String n) { return quoteIdent(n); } + } + + private final TestDriver driver = new TestDriver(); + + @Test + public void firstPageIsLimitOnly() { + assertEquals("LIMIT 25", driver.getRangeString(new Range(1, 25))); + } + + @Test + public void laterPageHasLimitBeforeOffset() { + assertEquals("LIMIT 10 OFFSET 10", driver.getRangeString(new Range(11, 20))); + } + + @Test + public void rangeNeedsNoOrderByAndLockIsEmpty() throws Exception { + assertEquals("", driver.getLockString()); + } + + @Test + public void identifiersAreDoubleQuoted() { + assertEquals("\"rank\"", driver.quote("rank")); + assertThrows(IllegalArgumentException.class, () -> driver.quote("a\"b")); + } + + @Test + public void timestampTextSpellingsParseToTheSameInstant() { + Instant expected = Instant.parse("2026-03-04T05:06:07.250Z"); + assertEquals(expected, SQLiteDriver.parseInstant("2026-03-04T05:06:07.250Z")); + assertEquals(expected, SQLiteDriver.parseInstant("2026-03-04 05:06:07.250")); + assertEquals(expected, SQLiteDriver.parseInstant("2026-03-04T05:06:07.250+00:00")); + assertEquals(expected, SQLiteDriver.parseInstant(Long.toString(expected.toEpochMilli()))); + } +} diff --git a/server/java/omdb/src/test/java/com/metaobjects/manager/db/driver/SQLiteDriverValueBindingTest.java b/server/java/omdb/src/test/java/com/metaobjects/manager/db/driver/SQLiteDriverValueBindingTest.java new file mode 100644 index 000000000..9bc957ed7 --- /dev/null +++ b/server/java/omdb/src/test/java/com/metaobjects/manager/db/driver/SQLiteDriverValueBindingTest.java @@ -0,0 +1,69 @@ +package com.metaobjects.manager.db.driver; + +import com.metaobjects.field.DateField; +import com.metaobjects.field.DecimalField; +import com.metaobjects.field.MetaField; +import com.metaobjects.field.TimestampField; +import org.junit.Test; + +import java.math.BigDecimal; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.sql.Statement; +import java.util.Date; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +/** + * How SQLiteDriver binds values against a real SQLite. SQLite orders every number before any + * text, so a decimal operand bound as text would make {@code realColumn > '0'} false for every + * row; these pin the bind to a number, and dates and timestamps to the ISO text the schema holds. + */ +public class SQLiteDriverValueBindingTest { + + /** Exposes the protected bind hook. */ + private static final class TestDriver extends SQLiteDriver { + void bind(PreparedStatement s, MetaField f, int i, Object v) throws SQLException { + setStatementValue(s, f, i, v); + } + } + + private final TestDriver driver = new TestDriver(); + + @Test + public void aDecimalOperandComparesAsANumberAgainstARealColumn() throws Exception { + try (Connection c = DriverManager.getConnection("jdbc:sqlite::memory:"); Statement st = c.createStatement()) { + st.execute("CREATE TABLE t (x REAL)"); + st.execute("INSERT INTO t VALUES (0.4)"); + for (String[] probe : new String[][] {{"0", "1"}, {"0.5", "0"}, {"0.4", "1"}, {"0.40", "1"}}) { + try (PreparedStatement ps = c.prepareStatement("SELECT COUNT(*) FROM (SELECT x * 1 AS y FROM t) WHERE y >= ?")) { + driver.bind(ps, new DecimalField("d"), 1, new BigDecimal(probe[0])); + try (ResultSet rs = ps.executeQuery()) { + assertTrue(rs.next()); + assertEquals("operand " + probe[0], Integer.parseInt(probe[1]), rs.getInt(1)); + } + } + } + } + } + + @Test + public void aDateAndATimestampAreBoundAsIsoText() throws Exception { + try (Connection c = DriverManager.getConnection("jdbc:sqlite::memory:")) { + Date instant = Date.from(java.time.Instant.parse("2026-03-04T05:06:07.250Z")); + try (PreparedStatement ps = c.prepareStatement("SELECT ?, ?")) { + driver.bind(ps, new TimestampField("ts"), 1, instant); + driver.bind(ps, new DateField("d"), 2, java.time.LocalDate.parse("2026-03-04")); + try (ResultSet rs = ps.executeQuery()) { + assertTrue(rs.next()); + assertEquals("2026-03-04T05:06:07.250Z", rs.getString(1)); + assertEquals("2026-03-04", rs.getString(2)); + } + } + } + } +} diff --git a/server/java/pom.xml b/server/java/pom.xml index 620e04d47..40c14c905 100644 --- a/server/java/pom.xml +++ b/server/java/pom.xml @@ -23,9 +23,9 @@ - scm:git:git://github.com/metaobjectsdev/metaobjects-core.git - scm:git:git@github.com:metaobjectsdev/metaobjects-core.git - https://github.com/metaobjectsdev/metaobjects-core + scm:git:git://github.com/metaobjectsdev/metaobjects.git + scm:git:git@github.com:metaobjectsdev/metaobjects.git + https://github.com/metaobjectsdev/metaobjects HEAD diff --git a/server/python/README.md b/server/python/README.md index 94313494a..c9b3482e5 100644 --- a/server/python/README.md +++ b/server/python/README.md @@ -45,10 +45,22 @@ The primary public API is the **loader** (`load_directory` / `load_uris` / distribution also ships the Python implementations of the other pillars used by the CLI and tooling: `codegen` (Pydantic + FastAPI emit — reference generators you own: `metaobjects eject ` copies one into your repo), `render` (Mustache + payload-VO + -verify), and `runtime` (a DB-API 2 `ObjectManager` — pg8000 / psycopg — no SQLAlchemy). +verify), and `runtime` (a DB-API 2 `ObjectManager` — pg8000 / psycopg on Postgres, the standard library's `sqlite3` on SQLite, PyMySQL on MySQL 8 (`pip install metaobjects[mysql]`) — no SQLAlchemy). Schema migrations are owned by the TypeScript `meta` CLI (ADR-0015); there is **no `migrate` module** in this distribution. +### Physical names + +Table and column names are one project setting, `naming` in `.metaobjects/config.json` +(`columns` and `tables`, each `snake_case` by default or `literal`). `metaobjects gen` and +`verify` read it, and so do `load_directory`, `load_uris` and `load_string(..., project_dir=...)`: +they search upward from the metadata directory for that file, so the `ObjectManager` queries the +names the code was generated for. If you keep a database whose columns are named exactly like the +fields, add `"naming": { "columns": "literal" }`. With no config file found the loader warns once +and uses `snake_case`; `metaobjects.naming.attach_naming(root, project_dir)` sets it explicitly and +wins. See [the upgrade guide](../../docs/features/migrations/upgrading-1.0-to-1.1.md) and +[physical naming](../../docs/features/migrations/physical-naming.md). + ## Authoring formats - **Canonical JSON** (`*.json`) — the cross-language interchange shape. diff --git a/server/python/pyproject.toml b/server/python/pyproject.toml index b49f24aa3..dca650bb4 100644 --- a/server/python/pyproject.toml +++ b/server/python/pyproject.toml @@ -47,6 +47,7 @@ Homepage = "https://metaobjects.dev" Repository = "https://github.com/metaobjectsdev/metaobjects" Documentation = "https://github.com/metaobjectsdev/metaobjects/tree/main/docs" Issues = "https://github.com/metaobjectsdev/metaobjects/issues" +Changelog = "https://github.com/metaobjectsdev/metaobjects/blob/main/CHANGELOG.md" [project.optional-dependencies] dev = ["pytest>=8", "mypy>=1.10", "ruff>=0.6", "pydantic>=2"] @@ -56,7 +57,16 @@ dev = ["pytest>=8", "mypy>=1.10", "ruff>=0.6", "pydantic>=2"] # TS/Java pattern of bypassing testcontainers wrappers entirely); the # api-contract harness drives a FastAPI Author API in-process via the # starlette TestClient (httpx). -integration = ["pg8000>=1.31", "fastapi>=0.110", "httpx>=0.27"] +integration = ["pg8000>=1.31", "PyMySQL>=1.1", "fastapi>=0.110", "httpx>=0.27"] +# MySQL for the ObjectManager (`metaobjects.runtime.MySqlDriver`). PyMySQL is pure Python, +# so like pg8000 it installs without native build dependencies. SQLite needs nothing: the +# driver wraps the standard library's `sqlite3`. +mysql = ["PyMySQL>=1.1"] + +[[tool.mypy.overrides]] +# PyMySQL ships no type information. +module = ["pymysql", "pymysql.*"] +ignore_missing_imports = true [build-system] requires = ["hatchling"] diff --git a/server/python/src/metaobjects/cli.py b/server/python/src/metaobjects/cli.py index dc1c80a19..46f73fde7 100644 --- a/server/python/src/metaobjects/cli.py +++ b/server/python/src/metaobjects/cli.py @@ -466,9 +466,10 @@ def load(strict_load: bool) -> LoadResult: providers=[*core_providers, *providers], strict=strict_load, libraries=all_libraries, + discover_naming=False, ) return MetaDataLoader.from_directory( - metadata_dir, strict=strict_load, libraries=all_libraries + metadata_dir, strict=strict_load, libraries=all_libraries, discover_naming=False ) try: @@ -1106,9 +1107,12 @@ def _cmd_list(_args: argparse.Namespace) -> int: root = Path.cwd() for entry in list_generators(): requires = f" (requires: {', '.join(entry.requires)})" if entry.requires else "" - status = owned.owned_status(root, entry) - mark = f" [owned — {status}]" if status is not None else "" + verdict = owned.owned_verdict(root, entry) + mark = f" [owned — {verdict[0]}]" if verdict is not None else "" print(f"{entry.name} — {entry.description}{requires}{mark}") + if verdict is not None: + for path, status in verdict[1]: + print(f" runtime {path.as_posix()} [{status}]") for path, status in owned.runtime_status(root): print(f"runtime {path.as_posix()} [owned — {status}]") return 0 @@ -2723,11 +2727,9 @@ def _build_parser() -> argparse.ArgumentParser: dest="column_naming", choices=COLUMN_NAMING_STRATEGIES, default=None, - help=( - "removed in 1.1.0 (ADR-0058): naming is one project setting now, `naming` in " - ".metaobjects/config.json, read by every port from the loaded root. Passing " - "this flag at all is refused." - ), + # Still PARSED so passing it gets the refusal message, but hidden from --help: a + # removed flag advertised in the usage text reads as a supported one. + help=argparse.SUPPRESS, ) gen.add_argument( "--generators", @@ -2841,11 +2843,9 @@ def _build_parser() -> argparse.ArgumentParser: dest="column_naming", choices=COLUMN_NAMING_STRATEGIES, default=None, - help=( - "removed in 1.1.0 (ADR-0058): naming is one project setting now, `naming` in " - ".metaobjects/config.json, read by every port from the loaded root. Passing " - "this flag at all is refused." - ), + # Still PARSED so passing it gets the refusal message, but hidden from --help: a + # removed flag advertised in the usage text reads as a supported one. + help=argparse.SUPPRESS, ) verify.add_argument( "--templates", diff --git a/server/python/src/metaobjects/codegen/eject.py b/server/python/src/metaobjects/codegen/eject.py index b0cbc3fac..af4b7bb20 100644 --- a/server/python/src/metaobjects/codegen/eject.py +++ b/server/python/src/metaobjects/codegen/eject.py @@ -191,6 +191,35 @@ def owned_status(root: Path, entry: GeneratorEntry) -> str | None: return _compare(path.read_text(encoding="utf-8"), ejected_source(entry)) +def owned_companions(root: Path, entry: GeneratorEntry) -> list[tuple[Path, str]]: + """``(relative path, status)`` of each owned runtime module *entry* ejects, against the + installed copy. A generator and the runtime its output imports are one owned unit: the + generator is ``identical`` only when every companion is too.""" + rows: list[tuple[Path, str]] = [] + for module in entry.runtime: + path = runtime_path(root, module) + if path.exists(): + rows.append((path.relative_to(root), _compare( + path.read_text(encoding="utf-8"), + packaged_runtime_path(module).read_text(encoding="utf-8"), + ))) + return rows + + +def owned_verdict(root: Path, entry: GeneratorEntry) -> tuple[str, list[tuple[Path, str]]] | None: + """The verdict ``--list`` prints for an owned generator plus its differing companions, or + ``None`` if not owned. The generator file's own counts win when it differs; when it is + identical but a companion differs, the verdict says so (TS: ``generator identical, N + renderer(s) differ``) and the caller names each differing file.""" + own = owned_status(root, entry) + if own is None: + return None + differing = [(p, st) for p, st in owned_companions(root, entry) if st != "identical"] + if own == "identical" and differing: + own = f"DIFFERS: generator identical, {len(differing)} runtime module(s) differ" + return own, differing + + def _line_diff(mine_text: str, ref_text: str) -> tuple[int, int]: """``(behind, of your own)``: reference lines the copy lacks, copy lines the reference lacks.""" mine = _normalised(mine_text) @@ -228,7 +257,7 @@ def runtime_status(root: Path) -> list[tuple[Path, str]]: def build_owned(spec: str, root: Path, ctx: GeneratorBuildContext) -> tuple[Generator | None, str | None]: """Resolve a ``module:symbol`` generator entry against the project at *root*. - The symbol may be a Generator, or a factory: one taking ``template_root`` (the + The symbol may be a Generator instance, a Generator class (instantiated), or a factory: one taking ``template_root`` (the render-helper shape) gets the build context's template root; one taking a single positional argument gets the build context; otherwise it is called with none. """ @@ -239,7 +268,9 @@ def build_owned(spec: str, root: Path, ctx: GeneratorBuildContext) -> tuple[Gene obj, err = import_project_symbol(module_name, symbol, root) if err is not None: return None, f"generator {spec!r}: {err}" - if not hasattr(obj, "generate") and callable(obj): + # A generator CLASS has `generate` too (unbound), so test for the class first: it must be + # instantiated, not used as the generator. + if inspect.isclass(obj) or (not hasattr(obj, "generate") and callable(obj)): params = inspect.signature(obj).parameters required = [p for p in params.values() if p.default is inspect.Parameter.empty diff --git a/server/python/src/metaobjects/codegen/generators/filter_allowlist_generator.py b/server/python/src/metaobjects/codegen/generators/filter_allowlist_generator.py index ca20de679..000b1cf10 100644 --- a/server/python/src/metaobjects/codegen/generators/filter_allowlist_generator.py +++ b/server/python/src/metaobjects/codegen/generators/filter_allowlist_generator.py @@ -40,7 +40,9 @@ from metaobjects.codegen.instance_artifacts import is_sourceless_entity from metaobjects.meta.core.field import field_constants as fc from metaobjects.meta.core.field.meta_field import MetaField +from metaobjects.codegen.type_map import field_is_array from metaobjects.meta.core.object.meta_object import MetaObject +from metaobjects.meta.core.object.object_constants import OBJECT_SUBTYPE_REPORT from ..runtime.m2m_codegen import build_object_index from ..runtime.shared_predicates import emits_router @@ -211,6 +213,11 @@ def _compute_filterable_ops( def add(f: MetaField) -> None: if f.sub_type == fc.FIELD_SUBTYPE_OBJECT or f.name in out or not _is_filterable(f): return + # A derived report field over an array or a map is not filterable. + if entity.sub_type == OBJECT_SUBTYPE_REPORT and ( + f.sub_type == fc.FIELD_SUBTYPE_MAP or field_is_array(f) + ): + return ops = self._ops_for_field(f) if ops: out[f.name] = ops diff --git a/server/python/src/metaobjects/codegen/generators/router_generator.py b/server/python/src/metaobjects/codegen/generators/router_generator.py index 241987730..7e73e29ad 100644 --- a/server/python/src/metaobjects/codegen/generators/router_generator.py +++ b/server/python/src/metaobjects/codegen/generators/router_generator.py @@ -50,7 +50,7 @@ from metaobjects.codegen.generators.tph_plan import TphPlan, is_tph_subtype, tph_plan_for from metaobjects.codegen.instance_artifacts import has_item_route from metaobjects.source_resolution import primary_rdb_source -from metaobjects.codegen.type_map import PyType, py_type_for +from metaobjects.codegen.type_map import PyType, field_is_array, py_type_for from metaobjects.meta.core.field import field_constants as fc from metaobjects.loader.validate_field_mutability import ( is_read_only_mutability, @@ -93,7 +93,7 @@ #: The helper runtime modules the emitted router imports (ADR-0034 Amendment 3: helper #: tier, not core). Listed on the registry entry too, which is what ``eject`` copies. -RUNTIME_MODULES: tuple[str, ...] = ("constraint_errors", "filter_parser") +RUNTIME_MODULES: tuple[str, ...] = ("constraint_errors", "decimal_wire", "filter_parser") #: The generated subpackage an owned router emits the runtime into. RUNTIME_SUBPACKAGE = "_runtime" @@ -148,7 +148,71 @@ def _scalar_fields(entity: MetaObject) -> list[MetaField]: """All effective fields minus ObjectField — same gate the Java/Kotlin controllers use for the sort allowlist (object fields have no plain column to sort on).""" - return [f for f in entity.fields() if f.sub_type != fc.FIELD_SUBTYPE_OBJECT] + fields = [f for f in entity.fields() if f.sub_type != fc.FIELD_SUBTYPE_OBJECT] + if entity.sub_type == OBJECT_SUBTYPE_REPORT: + # A derived report field over an array or a map is one jsonb value: ordering by it is + # meaningless, so it is not sortable (nor filterable, see the filter allowlist). + fields = [f for f in fields if f.sub_type != fc.FIELD_SUBTYPE_MAP and not field_is_array(f)] + return fields + + +#: ADR-0019 Amendment 1 (W2): a request body's decimal fields are parsed exactly (no float +#: detour) before validation; a malformed one is the cross-port 400. +_DECIMAL_COERCE: list[str] = [ + " if dto is None: # the body was not a JSON object: the cross-port 400, not FastAPI's 422", + ' return JSONResponse(status_code=400, content={"error": "validation"})', + " try:", + " dto = coerce_decimal_fields(dto, _DECIMAL_FIELDS, _DECIMAL_LIMITS)", + " except ValueError:", + ' return JSONResponse(status_code=400, content={"error": "validation"})', +] + +#: The request-body dependency of every write route: the body is read with +#: ``parse_float=Decimal`` so a JSON number never passes through a ``float`` (W2). A body that +#: is not a JSON object yields ``None`` and the route answers the cross-port +#: ``400 {"error": "validation"}`` (Java and Kotlin do; FastAPI's own 422 is not that contract). +_JSON_BODY_LINES: list[str] = [ + "async def _json_body(request: Request) -> dict[str, Any] | None:", + ' """GENERATED — the request body as a JSON object (non-integer numbers are exact `Decimal`s), or None."""', + " try:", + " return parse_json_object(await request.body())", + " except ValueError:", + " return None", +] + + +def _decimal_field_names(entities: list[MetaObject]) -> list[str]: + """Names of the ``field.decimal`` fields across *entities* (ADR-0039 resolving), in + declaration order, de-duplicated.""" + seen: dict[str, None] = {} + for entity in entities: + for f in entity.fields(): + if f.sub_type == fc.FIELD_SUBTYPE_DECIMAL: + seen.setdefault(f.name, None) + return list(seen) + + +def _decimal_limits(entities: list[MetaObject]) -> dict[str, tuple[int, int]]: + """``{name: (precision, scale)}`` for each decimal field declaring ``@precision`` + (ADR-0039 resolving; the first declaration of a name wins). The column's range (W3): + a value with more fraction digits than the scale, or more integer digits than + ``precision - scale``, is refused on input.""" + out: dict[str, tuple[int, int]] = {} + for entity in entities: + for f in entity.fields(): + if f.sub_type != fc.FIELD_SUBTYPE_DECIMAL or f.name in out: + continue + precision = f.attrs().get(fc.FIELD_ATTR_PRECISION) + scale = f.attrs().get(fc.FIELD_ATTR_SCALE, 0) + if isinstance(precision, int) and not isinstance(precision, bool) and isinstance(scale, int): + out[f.name] = (precision, scale) + return out + + +def _decimal_limits_lines(entities: list[MetaObject]) -> list[str]: + limits = _decimal_limits(entities) + body = "{" + ", ".join(f'"{n}": ({p}, {sc})' for n, (p, sc) in limits.items()) + "}" + return ["", "", f"_DECIMAL_LIMITS: dict[str, tuple[int, int]] = {body}"] def _required_field_names(entity: MetaObject) -> list[str]: @@ -185,7 +249,7 @@ def _frozen_names(entity: MetaObject) -> list[str]: ADR-0045: the OUTERMOST generated write artifact enforces the mode, and for FastAPI that is the ROUTER, not the ``Patch`` model — the handler binds - ``dto: dict[str, Any]`` (the raw body) and passes THAT to the repository, so a + ``dto`` (the raw body as a dict) and passes THAT to the repository, so a field merely absent from the patch model still reaches the row. @autoSet gets away with it because the router OVERWRITES its key server-side; a writeOnce field has no server value to overwrite, so it must be popped explicitly.""" @@ -603,12 +667,16 @@ def _emit_route_handler( f" filter_result = parse_filter(request.query_params, {fields_const}, {ops_const})", " if filter_result.error_envelope is not None:", " return JSONResponse(status_code=400, content=filter_result.error_envelope)", - " predicates = filter_result.predicates", + " predicates, decimal_error = coerce_decimal_predicates(", + " filter_result.predicates, _DECIMAL_FIELDS", + " )", + " if decimal_error is not None:", + " return JSONResponse(status_code=400, content=decimal_error)", " rows = repo.list(actual_limit, actual_offset, sort_clause, predicates)", " if with_count == 1:", " total = repo.count(predicates)", - ' return {"rows": rows, "total": total}', - " return rows", + ' return {"rows": to_wire(rows), "total": total}', + " return to_wire(rows)", ] if name == "get": return [ @@ -620,15 +688,16 @@ def _emit_route_handler( f" row = repo.find_by_id({pk_param})", " if row is None:", ' return JSONResponse(status_code=404, content={"error": "not_found"})', - " return row", + " return to_wire(row)", ] if name == "create": return [ '@router.post("", status_code=status.HTTP_201_CREATED)', f"def create_{snake}(", - " dto: dict[str, Any],", + " dto: Annotated[dict[str, Any] | None, Depends(_json_body)],", f" repo: Annotated[{repo_class}, Depends(get_repository)],", ") -> Any:", + *_DECIMAL_COERCE, " # FR-036: run the generated field constraints (length / pattern /", " # numeric bounds) before persisting; a violation is the cross-port 400.", " try:", @@ -640,7 +709,7 @@ def _emit_route_handler( # (a fresh row's createdAt == updatedAt), ignoring any caller-supplied value. *create_autoset, " try:", - " return repo.create(dto)", + " return to_wire(repo.create(dto))", *_CONSTRAINT_EXCEPT, ] if name == "update": @@ -649,9 +718,10 @@ def _emit_route_handler( f'@router.put("/{{{pk_param}}}")', f"def update_{snake}(", f" {pk_param}: {pk_type},", - " dto: dict[str, Any],", + " dto: Annotated[dict[str, Any] | None, Depends(_json_body)],", f" repo: Annotated[{repo_class}, Depends(get_repository)],", ") -> Any:", + *_DECIMAL_COERCE, " # FR-035 PATCH-2: an explicit null on a @required field is a 400 —", " # a present null on a NON-required field falls through and clears it,", " # and an OMITTED required field is untouched (never a 400).", @@ -674,7 +744,7 @@ def _emit_route_handler( *_CONSTRAINT_EXCEPT, " if saved is None:", ' return JSONResponse(status_code=404, content={"error": "not_found"})', - " return saved", + " return to_wire(saved)", ] if name == "delete": return [ @@ -707,8 +777,8 @@ def _emit_m2m_route( f" {pk_param}: {pk_type},", f" repo: Annotated[{repo_class}, Depends(get_repository)],", ") -> list[Any]:", - f" return repo.find_related_{d.relation_name}" - f"({pk_param}{_target_subtype_arg(d, {d.target_discriminator})})", + f" return to_wire(repo.find_related_{d.relation_name}" + f"({pk_param}{_target_subtype_arg(d, {d.target_discriminator})}))", ] def _emit_tph_m2m_route( @@ -764,8 +834,8 @@ def _emit_tph_m2m_route( " return []", ] lines.append( - f" return repo.find_related_{d.relation_name}" - f"({subtype_expr}, {pk_param}{_target_subtype_arg(d, target_discs)})" + f" return to_wire(repo.find_related_{d.relation_name}" + f"({subtype_expr}, {pk_param}{_target_subtype_arg(d, target_discs)}))" ) return lines @@ -786,12 +856,16 @@ def _emit_tph_list_body( f" filter_result = parse_filter(request.query_params, {fields_const}, {ops_const})", " if filter_result.error_envelope is not None:", " return JSONResponse(status_code=400, content=filter_result.error_envelope)", - " predicates = filter_result.predicates", + " predicates, decimal_error = coerce_decimal_predicates(", + " filter_result.predicates, _DECIMAL_FIELDS", + " )", + " if decimal_error is not None:", + " return JSONResponse(status_code=400, content=decimal_error)", f" rows = {repo_var}.list({subtype_expr}, actual_limit, actual_offset, sort_clause, predicates)", " if with_count == 1:", f" total = {repo_var}.count({subtype_expr}, predicates)", - ' return {"rows": rows, "total": total}', - " return rows", + ' return {"rows": to_wire(rows), "total": total}', + " return to_wire(rows)", ] def _render_tph_router( @@ -866,6 +940,9 @@ def _render_tph_router( req_seen.add(name) required_names.append(name) required_set_body = _py_set_literal(required_names, frozen=True) + decimal_set_body = _py_set_literal( + _decimal_field_names([entity, *(st.entity for st in plan.subtypes)]), frozen=True + ) # FR-037 R1 — non-readWrite @mutability across the base AND every subtype. # Union, stable order: ADR-0045 says the OUTERMOST artifact enforces the mode, # and a TPH per-subtype handler is a SEPARATE code path (the 0.19.4 lesson). @@ -950,6 +1027,12 @@ def _render_tph_router( parts.append(f"from {_runtime_module('constraint_errors')} import (") parts.append(" classify_constraint_error,") parts.append(")") + parts.append(f"from {_runtime_module('decimal_wire')} import (") + parts.append(" coerce_decimal_fields,") + parts.append(" coerce_decimal_predicates,") + parts.append(" parse_json_object,") + parts.append(" to_wire,") + parts.append(")") parts.append(f"from {_runtime_module('filter_parser')} import (") parts.append(" FilterPredicate,") parts.append(" parse_filter,") @@ -972,6 +1055,13 @@ def _render_tph_router( parts.append(f"_REQUIRED_FIELDS: frozenset[str] = {required_set_body}") parts.append("") parts.append("") + parts.append(f"_DECIMAL_FIELDS: frozenset[str] = {decimal_set_body}") + parts.extend(_decimal_limits_lines([entity, *(st.entity for st in plan.subtypes)])) + parts.append("") + parts.append("") + parts.extend(_JSON_BODY_LINES) + parts.append("") + parts.append("") # FR-037 R1 — the names a PATCH must never write. Emitted even when EMPTY so # the generated module always carries the seam (an entity that later gains a # writeOnce field changes one literal, not the module's shape). @@ -1045,9 +1135,10 @@ def list_sig(fn: str, route: str) -> list[str]: parts.append("") parts.append(f'@router.post("/{seg}", status_code=status.HTTP_201_CREATED)') parts.append(f"def create_{plural}_{sfx}(") - parts.append(" dto: dict[str, Any],") + parts.append(" dto: Annotated[dict[str, Any] | None, Depends(_json_body)],") parts.append(f" repo: Annotated[{repo_class}, Depends(get_repository)],") parts.append(") -> Any:") + parts.extend(_DECIMAL_COERCE) parts.append(" # FR-036: run the subtype's field constraints before persisting.") parts.append(" try:") parts.append(f" {sub}Create(**dto)") @@ -1057,7 +1148,7 @@ def list_sig(fn: str, route: str) -> list[str]: # non-@autoSet hierarchy — byte-identical output). parts.extend(create_autoset) parts.append(" try:") - parts.append(f' return repo.create("{val}", dto)') + parts.append(f' return to_wire(repo.create("{val}", dto))') parts.extend(_CONSTRAINT_EXCEPT) parts.append("") parts.append("") @@ -1069,16 +1160,17 @@ def list_sig(fn: str, route: str) -> list[str]: parts.append(f' row = repo.find_by_id("{val}", {pk_param})') parts.append(" if row is None:") parts.append(' return JSONResponse(status_code=404, content={"error": "not_found"})') - parts.append(" return row") + parts.append(" return to_wire(row)") parts.append("") parts.append("") parts.append(f'@router.patch("/{seg}/{{{pk_param}}}")') parts.append(f'@router.put("/{seg}/{{{pk_param}}}")') parts.append(f"def update_{plural}_{sfx}(") parts.append(f" {pk_param}: {pk_type},") - parts.append(" dto: dict[str, Any],") + parts.append(" dto: Annotated[dict[str, Any] | None, Depends(_json_body)],") parts.append(f" repo: Annotated[{repo_class}, Depends(get_repository)],") parts.append(") -> Any:") + parts.extend(_DECIMAL_COERCE) parts.append(" # FR-035 PATCH-2: an explicit null on a @required field is a 400.") parts.append(" for _k in _REQUIRED_FIELDS:") parts.append(" if _k in dto and dto[_k] is None:") @@ -1097,7 +1189,7 @@ def list_sig(fn: str, route: str) -> list[str]: parts.extend(_CONSTRAINT_EXCEPT) parts.append(" if saved is None:") parts.append(' return JSONResponse(status_code=404, content={"error": "not_found"})') - parts.append(" return saved") + parts.append(" return to_wire(saved)") parts.append("") parts.append("") parts.append(f'@router.delete("/{seg}/{{{pk_param}}}", status_code=status.HTTP_204_NO_CONTENT)') @@ -1147,7 +1239,7 @@ def list_sig(fn: str, route: str) -> list[str]: parts.append(f" row = repo.find_by_id(None, {pk_param})") parts.append(" if row is None:") parts.append(' return JSONResponse(status_code=404, content={"error": "not_found"})') - parts.append(" return row") + parts.append(" return to_wire(row)") parts.append("") # FW-8 rule (a): every relationship the BASE resolves — every row of the @@ -1252,6 +1344,7 @@ def render_router( # FR-035 PATCH-2: an explicit null on a @required field (scalar or jsonb) # is a 400 — the update handler guards these before the repo call. required_set_body = _py_set_literal(_required_field_names(entity), frozen=True) + decimal_set_body = _py_set_literal(_decimal_field_names([entity]), frozen=True) # FR-037 R1 — the names a PATCH must never write (readOnly / writeOnce). frozen_set_body = _py_set_literal(_frozen_names(entity), frozen=True) @@ -1302,6 +1395,12 @@ def render_router( parts.append(f"from {_runtime_module('constraint_errors')} import (") parts.append(" classify_constraint_error,") parts.append(")") + parts.append(f"from {_runtime_module('decimal_wire')} import (") + parts.append(" coerce_decimal_fields,") + parts.append(" coerce_decimal_predicates,") + parts.append(" parse_json_object,") + parts.append(" to_wire,") + parts.append(")") parts.append(f"from {_runtime_module('filter_parser')} import (") parts.append(" FilterPredicate,") parts.append(" parse_filter,") @@ -1325,6 +1424,13 @@ def render_router( parts.append(f"_REQUIRED_FIELDS: frozenset[str] = {required_set_body}") parts.append("") parts.append("") + parts.append(f"_DECIMAL_FIELDS: frozenset[str] = {decimal_set_body}") + parts.extend(_decimal_limits_lines([entity])) + parts.append("") + parts.append("") + parts.extend(_JSON_BODY_LINES) + parts.append("") + parts.append("") # FR-037 R1 — the names a PATCH must never write. Emitted even when EMPTY so # the generated module always carries the seam (an entity that later gains a # writeOnce field changes one literal, not the module's shape). @@ -1496,6 +1602,10 @@ def _render_readonly_router( parts.append("from fastapi.responses import JSONResponse") parts.append("from pydantic import BaseModel") parts.append("") + parts.append(f"from {_runtime_module('decimal_wire')} import (") + parts.append(" coerce_decimal_predicates,") + parts.append(" to_wire,") + parts.append(")") parts.append(f"from {_runtime_module('filter_parser')} import (") parts.append(" FilterPredicate,") parts.append(" parse_filter,") @@ -1509,6 +1619,12 @@ def _render_readonly_router( parts.extend(_sort_type_and_tables_lines(sort_field_nodes)) parts.append("") parts.append("") + parts.append( + "_DECIMAL_FIELDS: frozenset[str] = " + + _py_set_literal(_decimal_field_names([entity]), frozen=True) + ) + parts.append("") + parts.append("") parts.extend(_sort_parse_fn_lines()) parts.append("") parts.append("") diff --git a/server/python/src/metaobjects/codegen/runtime/__init__.py b/server/python/src/metaobjects/codegen/runtime/__init__.py index 995a56f52..103c98af6 100644 --- a/server/python/src/metaobjects/codegen/runtime/__init__.py +++ b/server/python/src/metaobjects/codegen/runtime/__init__.py @@ -3,7 +3,7 @@ Two kinds of module live here, and eject copies both into the adopter's ``codegen/runtime/`` (helper tier, not core: ADR-0034 Amendment 3). -``filter_parser`` and ``constraint_errors`` are what generated routers import. Generated +``filter_parser``, ``constraint_errors`` and ``decimal_wire`` are what generated routers import. Generated code from the PACKAGED ``routes`` generator imports them from this package directly; an owned ``routes`` emits the adopter's copy into the generated package instead. So each must stay self-contained — standard library only, no ``metaobjects`` import and no FastAPI / diff --git a/server/python/src/metaobjects/codegen/runtime/decimal_wire.py b/server/python/src/metaobjects/codegen/runtime/decimal_wire.py new file mode 100644 index 000000000..c686945ec --- /dev/null +++ b/server/python/src/metaobjects/codegen/runtime/decimal_wire.py @@ -0,0 +1,184 @@ +"""The decimal wire rule (ADR-0019 Amendment 1, W1-W4) for the generated REST surface. + +One production helper, used by the generated routers and tested against the shared vectors +(``fixtures/decimal-wire/vectors.json``): + +* **W1, emit** -- :func:`format_decimal` / :func:`to_wire`: every ``Decimal`` leaves as a JSON + string in canonical form (plain notation, no exponent, no trailing fractional zeros, zero is + ``"0"``). +* **W2, accept** -- :func:`parse_decimal` (a request-body value) and :func:`parse_decimal_operand` + (a filter operand): a string in the JSON-number grammar, or a JSON number; anything else is a + ``ValueError``. Nothing goes through a ``float``: :func:`parse_json_object` reads a body with + ``parse_float=Decimal``. +* **W3, range** -- a value too large to render is refused, never rounded. +* **W4, SQLite** -- :func:`decimal_from_double15`: a double read from an engine with no decimal + type is exact to 15 significant digits. +""" +from __future__ import annotations + +import dataclasses +import json +import re +from decimal import Decimal, InvalidOperation +from typing import Any, Iterable + +#: A string body value and a filter operand: the JSON-number grammar. An exponent is allowed on +#: input only. No sign other than ``-``, no leading zeros, no bare point, no whitespace. +_JSON_NUMBER = re.compile(r"-?(0|[1-9][0-9]*)(\.[0-9]+)?([eE][+-]?[0-9]+)?") + +#: W3: the widest plain-notation rendering a value may have. A string like ``1e999999999`` is +#: syntactically a number but would render as a gigabyte; it is out of range. +MAX_DIGITS = 1000 + + +def format_decimal(value: Decimal) -> str: + """W1: the canonical wire string of *value*.""" + if not value.is_finite(): + raise ValueError(f"{value} is not a finite decimal") + if value == 0: + return "0" + normalized = value.normalize() + if abs(normalized.adjusted()) > MAX_DIGITS: + raise ValueError("decimal out of range") + return format(normalized, "f") + + +def _checked(value: Decimal) -> Decimal: + if abs(value.adjusted()) > MAX_DIGITS and value != 0: + raise ValueError("decimal out of range") + return value + + +def parse_decimal(value: Any) -> Decimal: + """W2: a request-body decimal (a JSON string or number) as an exact ``Decimal``. + + ``bool`` is not a number here, and neither is anything that is not a string or a number. + A ``float`` (a body not read with ``parse_float=Decimal``) is taken as the decimal it was + written as, which is exact to 15 significant digits. + """ + if isinstance(value, bool): + raise ValueError("a boolean is not a decimal") + if isinstance(value, Decimal): + return _checked(value) if value.is_finite() else _reject() + if isinstance(value, int): + return _checked(Decimal(value)) + if isinstance(value, float): + return _checked(Decimal(repr(value))) if value == value and abs(value) != float("inf") else _reject() + if isinstance(value, str): + return _parse_text(value) + raise ValueError(f"{type(value).__name__} is not a decimal") + + +def parse_decimal_operand(text: str) -> Decimal: + """W2: a filter operand (query-string text) as an exact ``Decimal``.""" + return _parse_text(text) + + +def _reject() -> Decimal: + raise ValueError("not a finite decimal") + + +def _parse_text(text: str) -> Decimal: + if _JSON_NUMBER.fullmatch(text) is None: + raise ValueError(f"{text!r} is not a decimal") + try: + return _checked(Decimal(text)) + except InvalidOperation: + raise ValueError(f"{text!r} is not a decimal") from None + + +def decimal_from_double15(value: float | str) -> Decimal: + """W4: the stored double *value* rendered at 15 significant digits (SQLite's own text + precision), as a ``Decimal``. ``sum(0.1, 0.2)`` reads ``0.3``.""" + as_float = float(value) + if as_float != as_float or as_float in (float("inf"), float("-inf")): + raise ValueError(f"{value!r} is not a finite decimal") + return Decimal(f"{as_float:.15g}") + + +def parse_json_object(body: bytes | str) -> dict[str, Any]: + """A request body as a JSON object, every non-integer number as an exact ``Decimal``. + + Raises ``ValueError`` for invalid JSON or a body that is not an object.""" + try: + parsed = json.loads(body, parse_float=Decimal) + except (json.JSONDecodeError, UnicodeDecodeError, RecursionError) as exc: + raise ValueError("invalid JSON") from exc + if not isinstance(parsed, dict): + raise ValueError("a JSON object is required") + return parsed + + +def check_decimal_range(value: Decimal, precision: int, scale: int) -> None: + """W3: raise ``ValueError`` when *value* cannot be held by ``DECIMAL(precision, scale)`` + -- more fraction digits than *scale* or more integer digits than ``precision - scale``. + Trailing fraction zeros do not count (``100.2500`` is ``100.25``); nothing is rounded.""" + d = value.normalize() + exponent = d.as_tuple().exponent + if not isinstance(exponent, int): + raise ValueError("not a finite number") + if -exponent > scale: + raise ValueError(f"more than {scale} fraction digit(s)") + integer_digits = max(d.adjusted() + 1, 0) if d != 0 else 0 + if integer_digits > precision - scale: + raise ValueError(f"more than {precision - scale} integer digit(s)") + + +def coerce_decimal_fields( + dto: dict[str, Any], + decimal_fields: frozenset[str], + limits: dict[str, tuple[int, int]] | None = None, +) -> dict[str, Any]: + """A copy of *dto* with each present decimal field parsed under W2 (``null`` stays + ``None``) and held to its column's range under W3 (*limits*: name to (precision, scale)). + Raises ``ValueError`` naming the field when one is malformed or out of range.""" + out = dict(dto) + for name in decimal_fields: + if name in out and out[name] is not None: + try: + out[name] = parse_decimal(out[name]) + if limits and name in limits: + check_decimal_range(out[name], *limits[name]) + except ValueError as exc: + raise ValueError(f"{name}: {exc}") from None + return out + + +def coerce_decimal_predicates( + predicates: Iterable[Any], decimal_fields: frozenset[str] +) -> tuple[list[Any], dict[str, str] | None]: + """W2 for a filter: each predicate (a dataclass with ``field`` / ``op`` / ``value``, the + shape ``filter_parser.parse_filter`` returns) on a decimal field has its operand parsed + exactly, so a repository compares a ``Decimal``, never a float or text. Returns + ``(predicates, None)``, or ``([], envelope)`` where *envelope* is the corpus's existing + ``invalid_filter_value`` body naming the first malformed field.""" + out: list[Any] = [] + for p in predicates: + if p.field in decimal_fields and p.op != "isNull": + try: + value = ( + [parse_decimal_operand(v) for v in p.value] + if isinstance(p.value, list) + else parse_decimal_operand(p.value) + ) + except ValueError: + return [], {"error": "invalid_filter_value", "field": p.field} + p = dataclasses.replace(p, value=value) + out.append(p) + return out, None + + +def to_wire(value: Any) -> Any: + """W1 over a whole response: every ``Decimal`` in *value* (a row, a list of rows, the + ``{"rows", "total"}`` envelope, a Pydantic model) as its canonical string. Everything else + is left for the framework's encoder.""" + if isinstance(value, Decimal): + return format_decimal(value) + if isinstance(value, dict): + return {k: to_wire(v) for k, v in value.items()} + if isinstance(value, (list, tuple)): + return [to_wire(v) for v in value] + dump = getattr(value, "model_dump", None) + if callable(dump): + return to_wire(dump()) + return value diff --git a/server/python/src/metaobjects/config/neutral_config.py b/server/python/src/metaobjects/config/neutral_config.py index 59363369c..2de4c5241 100644 --- a/server/python/src/metaobjects/config/neutral_config.py +++ b/server/python/src/metaobjects/config/neutral_config.py @@ -76,6 +76,18 @@ class NeutralConfig: naming: NamingRules +def find_project_dir(start: Path) -> Path | None: + """The nearest directory at or above *start* that holds ``.metaobjects/config.json``, + or ``None``. THE project-config finder for a load that is handed a metadata + location rather than a project (``load_directory`` / ``load_uris``): the project + is wherever the setting is, however deep the metadata directory sits.""" + here = start.resolve() + for candidate in (here, *here.parents): + if (candidate / _METAOBJECTS_DIR / _CONFIG_FILE).is_file(): + return candidate + return None + + def read_neutral_config(config_dir: Path) -> NeutralConfig | None: """Read the neutral subset from ``config_dir/.metaobjects/config.json``. diff --git a/server/python/src/metaobjects/loader/meta_data_loader.py b/server/python/src/metaobjects/loader/meta_data_loader.py index 44f0eb4f0..0c4ba37c4 100644 --- a/server/python/src/metaobjects/loader/meta_data_loader.py +++ b/server/python/src/metaobjects/loader/meta_data_loader.py @@ -15,6 +15,7 @@ from __future__ import annotations import json +import warnings from dataclasses import dataclass, field from pathlib import Path @@ -41,6 +42,39 @@ from .validation_passes import run_validations +def _first_file_location(uris: list[str]) -> Path | None: + from urllib.parse import unquote, urlparse + + for uri in uris: + parsed = urlparse(uri) + if parsed.scheme == "file": + return Path(unquote(parsed.path)) + if parsed.scheme in ("", ) or (len(parsed.scheme) == 1): # plain / drive-letter path + return Path(uri) + return None + + +def _apply_discovered_naming(result: "LoadResult", start: Path) -> None: + """Attach the project's ``naming`` (nearest ``.metaobjects/config.json`` at or above + *start*) to the loaded root; warn once when there is none and the default applies.""" + from ..config.neutral_config import find_project_dir + from ..naming import attach_naming + + project = find_project_dir(start if start.is_dir() else start.parent) + if project is not None: + attach_naming(result.root, project) + return + warnings.warn( + f"no .metaobjects/config.json found at or above {start}: physical names use the " + "snake_case default. A database whose columns are named exactly like the fields " + 'needs "naming": {"columns": "literal"} in that file, or ' + "metaobjects.naming.attach_naming(root, project_dir) after the load " + "(see docs/features/migrations/physical-naming.md).", + UserWarning, + stacklevel=4, + ) + + @dataclass class LoadResult: """The Tier-1 load result. Same field shape across all four ports.""" @@ -159,9 +193,17 @@ def from_directory( recurse: bool = True, strict: bool = False, libraries: list[str] | None = None, + discover_naming: bool = True, ) -> LoadResult: """Load every JSON/YAML file under ``directory`` (recursive by default). + ``discover_naming`` (ADR-0058) — the loaded root's ``naming`` is read from the + nearest ``.metaobjects/config.json`` at or above ``directory``, the setting + ``gen`` and ``verify`` read, so the ObjectManager queries the columns the code + was generated for. With no such file the ``snake_case`` default applies and a + ``UserWarning`` says so. :func:`metaobjects.naming.attach_naming` afterwards + still wins. The CLI passes ``False``: it attaches its own. + ``strict`` (ADR-0023) — when True, an undeclared own ``@-attr`` → ``ERR_UNKNOWN_ATTR``. Defaults False (downstream-friendly open policy). @@ -186,7 +228,10 @@ def from_directory( sources.extend( DirectorySource(directory, exclude=exclude, recurse=recurse).expand() ) - return loader.load(sources) + result = loader.load(sources) + if discover_naming: + _apply_discovered_naming(result, Path(directory)) + return result @classmethod def from_uris( @@ -195,9 +240,14 @@ def from_uris( providers: list[Provider] | None = None, strict: bool = False, libraries: list[str] | None = None, + discover_naming: bool = True, ) -> LoadResult: """Load metadata from a list of URIs (file:// or http(s)://). + ``discover_naming`` — see :meth:`from_directory`; the search starts at the + first ``file://`` URI (or plain path). A load with no file URI has no project + to find and keeps the default silently. + ``strict`` (ADR-0023) — see :meth:`from_directory`. ``libraries`` — see :meth:`from_directory`; prepended the same way, so a @@ -212,7 +262,12 @@ def from_uris( sources.extend(library_sources(libraries)) sources.extend(UriSource(u) for u in uris) - return loader.load(sources) + result = loader.load(sources) + if discover_naming: + start = _first_file_location(uris) + if start is not None: + _apply_discovered_naming(result, start) + return result @classmethod def from_string( @@ -221,13 +276,21 @@ def from_string( format: MetaDataFormat = MetaDataFormat.JSON, providers: list[Provider] | None = None, strict: bool = False, + project_dir: Path | str | None = None, ) -> LoadResult: """Load metadata from an in-memory string (defaults to JSON). ``strict`` (ADR-0023) — see :meth:`from_directory`. + + ``project_dir`` — the project whose ``.metaobjects/config.json`` supplies + ``naming`` (searched upward from it). Omitted, a string has no project and + keeps the ``snake_case`` default without a warning. """ loader = cls(providers=providers, strict=strict) - return loader.load([InMemoryStringSource(content, format=format)]) + result = loader.load([InMemoryStringSource(content, format=format)]) + if project_dir is not None: + _apply_discovered_naming(result, Path(project_dir)) + return result # --- internals ----------------------------------------------------- diff --git a/server/python/src/metaobjects/loader/validate_reporting.py b/server/python/src/metaobjects/loader/validate_reporting.py index b6d33a4cc..d8e0f5270 100644 --- a/server/python/src/metaobjects/loader/validate_reporting.py +++ b/server/python/src/metaobjects/loader/validate_reporting.py @@ -46,6 +46,7 @@ from ..errors import ErrorCode, MetaError from ..meta.core.attr.attr_constants import ATTR_SUBTYPE_FILTER from ..meta.core.field.field_constants import ( + FIELD_ATTR_OBJECT_REF, FIELD_SUBTYPE_BOOLEAN, FIELD_SUBTYPE_CURRENCY, FIELD_SUBTYPE_DATE, @@ -105,6 +106,7 @@ REPORTING_ATTR_VIA, ) from ..meta.meta_data import MetaData +from ..meta.persistence.source.source_constants import SOURCE_KIND_VIEW from ..meta.meta_root import MetaRoot from ..meta.persistence.source.meta_source import MetaSource from ..naming_refs import CHILD_REF_SEP, ref_target, resolve_object_ref @@ -293,6 +295,19 @@ def _field_of(obj: MetaData, name: str) -> MetaData | None: return _child_of_type(obj, TYPE_FIELD, name) +def _object_ref_of(field: MetaData) -> str | None: + """A field's resolving ``@objectRef`` (ADR-0039), or ``None``.""" + v = field.get_meta_attr(FIELD_ATTR_OBJECT_REF) + return v if isinstance(v, str) and v else None + + +def _is_document_field(field: MetaData) -> bool: + """D5 / M10 -- a ``field.object``, or any field carrying an ``@objectRef``, is a JSON + document: no port groups or aggregates a report over one. (A ``field.map`` with a + ``@valueType`` is legal.)""" + return field.sub_type == FIELD_SUBTYPE_OBJECT or _object_ref_of(field) is not None + + def _is_relative_value(v: object) -> TypeGuard[dict[str, object]]: """An object operand carrying a ``now`` key — a relative-date value, well-formed (exactly ``{ now }``) or not. A malformed one is refused, never read as data.""" @@ -465,9 +480,27 @@ def err(message: str) -> None: err(f"@of '{of}' must name a field of '{_shown(ctx, of_entity)}', the entity @via '{via}' reaches.") return field_name = parts.path[0] - field = _field_of(of_entity, field_name) + # With @via the field is looked up on the entity @of NAMES (own plus inherited), not on the + # @via terminal: a base the terminal extends must itself carry the field, or the dimension + # names a column the base table lacks. Without @via the host IS the checked entity. + lookup = of_entity if via is None else named + field = _field_of(lookup, field_name) if lookup is not None else None if field is None: - err(f"@of '{of}' names no field '{field_name}' on '{_shown(ctx, of_entity)}'.") + assert lookup is not None + if lookup is not of_entity and _field_of(of_entity, field_name) is not None: + err( + f"@of '{of}' names no field '{field_name}' on '{_shown(ctx, lookup)}': only " + f"'{_shown(ctx, of_entity)}', the entity @via reaches, declares it. " + f"Name it '{of_entity.name}.{field_name}'." + ) + else: + err(f"@of '{of}' names no field '{field_name}' on '{_shown(ctx, lookup)}'.") + return + + # D5 -- a dimension groups by a scalar. + if _is_document_field(field): + with_ref = " with an @objectRef" if _object_ref_of(field) is not None else "" + err(f"@of '{of}' is a field.{field.sub_type}{with_ref}, which a report cannot group by. Group by a scalar field of it.") return if not dim.is_time(): @@ -608,7 +641,11 @@ def err(message: str) -> None: # type error, and M7/M8 still read the attribute as present). # ADR-0039: resolving (get_meta_attr) — the raw attribute, inherited or not. declared_default = measure.get_meta_attr(REPORTING_ATTR_DEFAULT) - fractional_default = isinstance(declared_default, float) and not declared_default.is_integer() + # MINOR 4 Rule B: M7-M9 run only for a valid (integral, non-bool) number; anything else is + # reported once, as ERR_BAD_ATTR_VALUE, by the attr-schema pass. + fractional_default = declared_default is not None and not ( + isinstance(declared_default, int) and not isinstance(declared_default, bool) + ) if measure.is_ratio(): _check_ratio_operands(ctx, measure, err) @@ -707,6 +744,11 @@ def _check_aggregate_columns( if field is None: err(f"@of '{item}' names no field '{parts.path[0]}' on '{ctx.declaring.resolution_key()}'.") return None + # M10 -- a measure aggregates scalars. + if _is_document_field(field): + with_ref = " with an @objectRef" if _object_ref_of(field) is not None else "" + err(f"@of '{item}' is a field.{field.sub_type}{with_ref}, which a report cannot aggregate. Aggregate a scalar field of it.") + return None fields.append(field) # M2 — a tuple is a distinct count only. @@ -876,6 +918,23 @@ def err(message: str, node: MetaData = report, code: ErrorCode = _ERR_INVALID_RE source, ) + # R10 -- a report is read through a view; no port lowers or serves the other read-only kinds. + for source in (c for c in report.children() if c.type == TYPE_SOURCE): + if isinstance(source, MetaSource) and source.is_read_only() and source.effective_kind() != SOURCE_KIND_VIEW: + err( + f": {_child_label(source)} has @kind '{source.effective_kind()}'; a report is read through a view, " + f"so its source must be @kind: view. For a view or materialized view you create yourself, declare " + f"@kind: view with @unmanaged: true.", + source, + ) + + # R11 -- a report cannot be abstract. + if report.is_abstract: + err( + ": is abstract; a report cannot be abstract, because its fields are derived from its own @dimensions " + "and @measures and no port reads or serves an abstract report. Remove abstract: true." + ) + # R1 — @from resolves to an object.entity. Without it, R2/R3/R6/R7, R8/R9 and # the @filter have nothing to resolve against, so they are skipped. from_ref = report_from(report) diff --git a/server/python/src/metaobjects/meta/core/attr/meta_attr.py b/server/python/src/metaobjects/meta/core/attr/meta_attr.py index a88cab3ce..0b7c8bedd 100644 --- a/server/python/src/metaobjects/meta/core/attr/meta_attr.py +++ b/server/python/src/metaobjects/meta/core/attr/meta_attr.py @@ -70,6 +70,14 @@ def coerce(self, raw: object) -> object: if isinstance(raw, (int, float)): return str(raw) return raw + if ( + self.sub_type in (ATTR_SUBTYPE_INT, ATTR_SUBTYPE_LONG) + and isinstance(raw, float) + and raw.is_integer() + ): + # MINOR 4 Rule A: an int/long attr accepts any integral JSON number (4, 4.0, 1e2) + # and stores the integer; a fractional one stays a float and is ERR_BAD_ATTR_VALUE. + return int(raw) return raw def desugar(self, value: object) -> object: diff --git a/server/python/src/metaobjects/meta/meta_data.py b/server/python/src/metaobjects/meta/meta_data.py index bbc0e907b..8ea1e076c 100644 --- a/server/python/src/metaobjects/meta/meta_data.py +++ b/server/python/src/metaobjects/meta/meta_data.py @@ -26,6 +26,11 @@ def __init__(self, type_: str, sub_type: str, name: str) -> None: # per-file root packages are no longer reachable via the parent chain. # Mirrors TS ``MetaData.fileDefaultPackage`` / ``resolutionKey()``. self.file_default_package: Optional[str] = None + # The package declared by this node or its nearest enclosing node in the SAME + # document, captured at parse time (None for a node built in code). Unlike the + # parse context, an ``overlay: true`` node's explicit package counts, so the + # children it contributes keep it after the merge (requirement package, MINOR 4). + self.scope_package: Optional[str] = None # ADR-0059 R5 -- ``super_ref`` is canonical after the loader's # ``resolve_references`` pass; the ``extends`` value as authored is kept here # (set only when the pass rewrote it) for ``meta fmt``. diff --git a/server/python/src/metaobjects/naming_refs.py b/server/python/src/metaobjects/naming_refs.py index 17d4b6d10..03500c370 100644 --- a/server/python/src/metaobjects/naming_refs.py +++ b/server/python/src/metaobjects/naming_refs.py @@ -318,12 +318,7 @@ def referrer_package_for(node: "MetaData", attr_name: str) -> str: return referrer_package_of(declarer if declarer is not None else node) -def did_you_mean_hint(root: "MetaData", ref: str) -> str: - """ADR-0042 section 5 -- a did-you-mean suffix for an UNRESOLVED object reference: the - FQNs of same-short-name objects that DO exist (typically in other packages), so - the author can qualify a bare ref they meant to point across a package boundary. - Returns ``""`` when no same-short-name object exists. Mirrors the TS ``didYouMeanHint``. - """ +def _same_named_objects(root: "MetaData", ref: str) -> tuple[str, list[str]] | None: owner, _tail = split_child_tail(ref) sep = owner.rfind(PACKAGE_SEP) short_name = owner if sep == -1 else owner[sep + len(PACKAGE_SEP):] @@ -332,14 +327,39 @@ def did_you_mean_hint(root: "MetaData", ref: str) -> str: for c in root.children() if c.type == TYPE_OBJECT and c.name == short_name ] - if not candidates: + return (short_name, candidates) if candidates else None + + +def did_you_mean_hint(root: "MetaData", ref: str) -> str: + """ADR-0042 section 5 -- a did-you-mean suffix for an UNRESOLVED object reference: the + FQNs of same-short-name objects that DO exist (typically in other packages), so + the author can qualify a bare ref they meant to point across a package boundary. + Returns ``""`` when no same-short-name object exists. Mirrors the TS ``didYouMeanHint``. + """ + found = _same_named_objects(root, ref) + if found is None: return "" + short_name, candidates = found return ( f' An object named "{short_name}" exists in: {", ".join(candidates)}. ' "Qualify it with its package (FQN)." ) +def did_you_mean_super_hint(root: "MetaData", ref: str) -> str: + """The did-you-mean suffix for an UNRESOLVED ``extends`` ref: the same candidates as + :func:`did_you_mean_hint`, plus the exact spelling to write (a bare ``extends`` resolves + in the referrer's own package). Mirrors the TS ``didYouMeanSuperHint``.""" + found = _same_named_objects(root, ref) + if found is None: + return "" + short_name, candidates = found + return ( + f' An object named "{short_name}" exists in: {", ".join(candidates)}. ' + f'Qualify it with its package (FQN): write extends: "{candidates[0]}".' + ) + + def ref_target(node: "MetaData", attr_name: str) -> Optional["MetaData"]: """ADR-0059 R7 -- the object a reference attribute on *node* names: the owner the loader bound once (``MetaAttr.resolved_target``), read RESOLVING so an inherited diff --git a/server/python/src/metaobjects/parser.py b/server/python/src/metaobjects/parser.py index e6a9110fa..6e5e7aecf 100644 --- a/server/python/src/metaobjects/parser.py +++ b/server/python/src/metaobjects/parser.py @@ -271,6 +271,7 @@ def _build( result: ParseResult, builder: JsonPathBuilder, ctx_pkg: str = "", + scope_pkg: str = "", parent_type: str = "", yaml_position: YamlPosition | None = None, is_root: bool = False, @@ -387,6 +388,7 @@ def _build( # Recorded even when empty: "" says the file declares no package (ADR-0059 R10), # which is not the same as a node built in code with no file at all. node.file_default_package = (str(pkg) if pkg else None) or ctx_pkg + node.scope_package = (str(pkg) if pkg else None) or scope_pkg or None if pkg: node.package = str(pkg) @@ -487,7 +489,7 @@ def _build( else: child = _build( cw, cbody, registry, source, result, builder, - ctx_pkg=child_ctx_pkg, parent_type=type_, + ctx_pkg=child_ctx_pkg, scope_pkg=node.scope_package or "", parent_type=type_, yaml_position=child_yaml_pos, ) if child is not None: diff --git a/server/python/src/metaobjects/requirement_check.py b/server/python/src/metaobjects/requirement_check.py index a228a9796..cdd4a60f7 100644 --- a/server/python/src/metaobjects/requirement_check.py +++ b/server/python/src/metaobjects/requirement_check.py @@ -129,7 +129,11 @@ def effective_package(req: MetaRequirement) -> str: file's default, else ``""``. Public because the requirement-test identity function must use this ONE answer, not a second copy of it: it is also the referrer package every ``@implementedBy`` reference binds in.""" - return req.package or req.file_default_package or "" + # MINOR 4 item 4: own package, else the nearest enclosing node WITHIN THE SAME DOCUMENT that + # declares one (recorded at parse time as ``scope_package`` -- an overlay's explicit package + # reaches the children it contributes, though the merge leaves the target's alone), else the + # file's default. + return req.package or req.scope_package or req.file_default_package or "" def collect_addressed_requirements(root: MetaData) -> list[AddressedRequirement]: diff --git a/server/python/src/metaobjects/runtime/__init__.py b/server/python/src/metaobjects/runtime/__init__.py index f4f37097e..70df1e18b 100644 --- a/server/python/src/metaobjects/runtime/__init__.py +++ b/server/python/src/metaobjects/runtime/__init__.py @@ -6,7 +6,9 @@ exercise eq/ne/gt/gte/lt/lte/in/like/isNull operators + the top-level `and:` combinator + asc/desc sort + limit/offset. """ -from .object_manager import Filter, ObjectManager, PostgresDriver +from .dialect import MYSQL, POSTGRES, SQLITE, Dialect +from .drivers import MySqlDriver, SqliteDriver +from .object_manager import DbApiDriver, Filter, ObjectManager, PostgresDriver from .validator_runner import ValidationFailure, ValidationResult, run_validators from .llm_recorder import ( STATUS_ERROR, @@ -23,7 +25,14 @@ ) __all__ = [ + "DbApiDriver", + "Dialect", "Filter", + "MYSQL", + "MySqlDriver", + "POSTGRES", + "SQLITE", + "SqliteDriver", "ObjectManager", "PostgresDriver", "ValidationFailure", diff --git a/server/python/src/metaobjects/runtime/dialect.py b/server/python/src/metaobjects/runtime/dialect.py new file mode 100644 index 000000000..b11eaff6e --- /dev/null +++ b/server/python/src/metaobjects/runtime/dialect.py @@ -0,0 +1,298 @@ +"""SQL dialects the ObjectManager can emit for: Postgres, SQLite and MySQL. + +A :class:`Dialect` owns every engine-specific fact the read and write paths need, so the +ObjectManager itself stays one implementation: + +* how an identifier is quoted (Postgres and SQLite use ``"``; MySQL uses a backtick and the + runtime never relies on ``ANSI_QUOTES``) and what a bound parameter looks like (``%s`` for + pg8000 / PyMySQL, ``?`` for the stdlib ``sqlite3``); +* whether ``INSERT`` / ``UPDATE`` can ``RETURNING`` the row (Postgres and SQLite >= 3.35 can; + MySQL cannot, so the ObjectManager re-reads the row by its key); +* how ``LIKE`` stays case-SENSITIVE (ADR-0049) where the engine's default is not; +* how a native Python value is BOUND (``encode_param``) and how what the driver hands back is + turned into the native Python type the metadata declares (``decode_value``). + +The Postgres dialect is the identity on both value hooks: pg8000 already speaks native types +(ADR-0019). SQLite and MySQL decode by the FIELD's declared subtype, because neither engine +has the column types Postgres has (a SQLite boolean is ``0``/``1``, its date and uuid are +``TEXT``; a MySQL ``TIME`` arrives as a ``timedelta``, its ``SUM`` of an integer as a +``Decimal``), so the declared type is the only authority on what a value means. +""" +from __future__ import annotations + +import datetime as _dt +import decimal as _decimal +import json as _json +import sqlite3 as _sqlite3 +import uuid as _uuid +from typing import Any + +from ..codegen.runtime.decimal_wire import decimal_from_double15 +from ..meta.core.field import field_constants as fc +from ..meta.core.field.meta_field import MetaField +from ..meta.persistence.db import db_constants as dbc + +#: Postgres type OIDs the serialization boundary keys the int4-vs-int8 wire rule on +#: (``tests/integration/normalization.py``). A dialect with no OIDs of its own reports the +#: equivalent for the FIELD's declared subtype, so one boundary serves every engine. +PG_OID_INT8 = 20 +PG_OID_INT4 = 23 + + +def sqlite_supports_returning() -> bool: + """``INSERT/UPDATE ... RETURNING`` arrived in SQLite 3.35.""" + return _sqlite3.sqlite_version_info >= (3, 35, 0) + + +class Dialect: + """Postgres behaviour; SQLite and MySQL override only what differs.""" + + name = "postgres" + placeholder = "%s" + #: True when ``INSERT`` / ``UPDATE`` can ``RETURNING`` the written row. + supports_returning = True + #: True when the driver has no column OIDs, so the runtime derives type codes from fields. + synthesizes_type_codes = False + #: The row-of-defaults INSERT tail (``INSERT INTO t ``). + default_values = "DEFAULT VALUES" + #: What ``LIMIT`` must be to mean "no limit" when only an ``OFFSET`` is given (``""``: the + #: engine takes an ``OFFSET`` alone). + unbounded_limit = "" + + def quote(self, ident: str) -> str: + if '"' in ident: + raise ValueError(f"unsafe identifier: {ident}") + return f'"{ident}"' + + def like_clause(self, quoted_column: str, pattern: str) -> tuple[str, list[Any]]: + """``col LIKE pattern`` with the pattern verbatim and the match case-SENSITIVE.""" + return f"{quoted_column} LIKE {self.placeholder}", [pattern] + + def false_predicate(self) -> str: + return "FALSE" + + def encode_param(self, value: Any) -> Any: + """A native Python value as the driver binds it.""" + return value + + def decode_value(self, field: MetaField, value: Any) -> Any: + """What the driver returned for *field*, as the native Python type it declares.""" + return value + + def type_code(self, field: MetaField) -> int | None: + """The Postgres-OID-equivalent wire discriminator of *field*, or ``None``.""" + sub = field.sub_type + if sub in (fc.FIELD_SUBTYPE_LONG, fc.FIELD_SUBTYPE_CURRENCY): + return PG_OID_INT8 + if sub == fc.FIELD_SUBTYPE_INT: + return PG_OID_INT4 + return None + + +class PostgresDialect(Dialect): + pass + + +# ---------------------------------------------------------------------------- +# Shared decoding for the engines whose driver returns storage-class values +# ---------------------------------------------------------------------------- + + +def _is_instant(field: MetaField) -> bool: + """ADR-0036 Wave 2: a ``field.timestamp`` is an instant unless ``@localTime`` opts out.""" + return field.get_meta_attr(dbc.FIELD_ATTR_LOCAL_TIME) is not True # ADR-0039 resolving + + +def _parse_timestamp(text: str) -> _dt.datetime: + iso = text.strip().replace(" ", "T") + if iso.endswith(("Z", "z")): + iso = iso[:-1] + "+00:00" + return _dt.datetime.fromisoformat(iso) + + +def _as_timestamp(field: MetaField, value: Any) -> Any: + """An instant is tz-aware UTC, a ``@localTime`` timestamp a naive wall clock. The engines + here hold both as a UTC wall clock with no zone, so a naive value is read AS UTC.""" + if isinstance(value, str): + value = _parse_timestamp(value) + if not isinstance(value, _dt.datetime): + return value + if _is_instant(field): + if value.tzinfo is None: + return value.replace(tzinfo=_dt.timezone.utc) + return value.astimezone(_dt.timezone.utc) + return value.replace(tzinfo=None) + + +def _as_int(value: Any) -> Any: + if isinstance(value, bool): + return int(value) + if isinstance(value, _decimal.Decimal) and value == value.to_integral_value(): + return int(value) + if isinstance(value, float) and value.is_integer(): + return int(value) + if isinstance(value, str): + try: + return int(value) + except ValueError: + return value + return value + + +def _decode_storage_value(field: MetaField, value: Any) -> Any: + """Decode a SQLite / MySQL value by the field's declared subtype.""" + if value is None: + return None + sub = field.sub_type + if sub == fc.FIELD_SUBTYPE_BOOLEAN: + if isinstance(value, bool): + return value + if isinstance(value, (int, float, _decimal.Decimal)): + return value != 0 + if isinstance(value, str): + return value.strip().lower() in ("1", "true", "t") + return value + if sub in (fc.FIELD_SUBTYPE_INT, fc.FIELD_SUBTYPE_LONG, fc.FIELD_SUBTYPE_CURRENCY): + return _as_int(value) + if sub in (fc.FIELD_SUBTYPE_FLOAT, fc.FIELD_SUBTYPE_DOUBLE): + if isinstance(value, (int, _decimal.Decimal)) and not isinstance(value, bool): + return float(value) + return value + if sub == fc.FIELD_SUBTYPE_DECIMAL: + if isinstance(value, _decimal.Decimal): + return value + if isinstance(value, float): + # SQLite (ADR-0019 Amendment 1, W4): NUMERIC affinity keeps a decimal as an INTEGER or + # a REAL, so the value is exact to 15 significant digits: read it as such. + return decimal_from_double15(value) + if isinstance(value, (int, str)) and not isinstance(value, bool): + return _decimal.Decimal(str(value)) + return value + if sub == fc.FIELD_SUBTYPE_DATE: + if isinstance(value, _dt.datetime): + return value.date() + if isinstance(value, str): + return _dt.date.fromisoformat(value[:10]) + return value + if sub == fc.FIELD_SUBTYPE_TIME: + if isinstance(value, _dt.timedelta): # PyMySQL spells a TIME as a timedelta + seconds = int(value.total_seconds()) + micro = value.microseconds + return _dt.time(seconds // 3600 % 24, seconds // 60 % 60, seconds % 60, micro) + if isinstance(value, str): + return _dt.time.fromisoformat(value) + return value + if sub == fc.FIELD_SUBTYPE_TIMESTAMP: + return _as_timestamp(field, value) + if sub == fc.FIELD_SUBTYPE_UUID: + if isinstance(value, str): + return _uuid.UUID(value) + if isinstance(value, bytes) and len(value) == 16: + return _uuid.UUID(bytes=value) + return value + if sub in (fc.FIELD_SUBTYPE_OBJECT, fc.FIELD_SUBTYPE_MAP): + if isinstance(value, (str, bytes, bytearray)): + storage = field.get_meta_attr(fc.FIELD_ATTR_STORAGE) # ADR-0039 resolving + if storage != "flattened": + return _json.loads(value) + return value + return value + + +def _timestamp_text(value: _dt.datetime) -> str: + """The text an instant is stored as: UTC, millisecond resolution, ``Z`` iff it was aware.""" + aware = value.tzinfo is not None + if aware: + value = value.astimezone(_dt.timezone.utc).replace(tzinfo=None) + text = value.strftime("%Y-%m-%dT%H:%M:%S") + f".{value.microsecond // 1000:03d}" + return text + "Z" if aware else text + + +class SqliteDialect(Dialect): + """SQLite through the stdlib ``sqlite3`` (qmark parameters).""" + + name = "sqlite" + placeholder = "?" + synthesizes_type_codes = True + unbounded_limit = "-1" + + @property + def supports_returning(self) -> bool: # type: ignore[override] + return sqlite_supports_returning() + + def like_clause(self, quoted_column: str, pattern: str) -> tuple[str, list[Any]]: + # SQLite's LIKE ignores ASCII case and has no switch short of a connection pragma; + # GLOB is case-sensitive. A verbatim SQL pattern has only `%` and `_` as wildcards, + # so translate those and neutralise GLOB's own specials. + out: list[str] = [] + for ch in pattern: + if ch == "%": + out.append("*") + elif ch == "_": + out.append("?") + elif ch in "*?[": + out.append(f"[{ch}]") + else: + out.append(ch) + return f"{quoted_column} GLOB {self.placeholder}", ["".join(out)] + + def false_predicate(self) -> str: + return "0 = 1" + + def encode_param(self, value: Any) -> Any: + if isinstance(value, bool): + return int(value) + if isinstance(value, _uuid.UUID): + return str(value) + if isinstance(value, _decimal.Decimal): + # SQLite has no decimal type: a NUMERIC column stores an INTEGER or a REAL, and a + # view column (an AVG, a ratio) has no affinity at all, so a bound TEXT would + # compare as text and sort after every number. Bind the number. + return int(value) if value == value.to_integral_value() else float(value) + if isinstance(value, _dt.datetime): + return _timestamp_text(value) + if isinstance(value, _dt.date): + return value.isoformat() + if isinstance(value, _dt.time): + return value.isoformat() + return value + + def decode_value(self, field: MetaField, value: Any) -> Any: + return _decode_storage_value(field, value) + + +class MySqlDialect(Dialect): + """MySQL 8 through PyMySQL (``%s`` parameters, backtick identifiers).""" + + name = "mysql" + supports_returning = False + synthesizes_type_codes = True + default_values = "() VALUES ()" + unbounded_limit = "18446744073709551615" + + def quote(self, ident: str) -> str: + if "`" in ident: + raise ValueError(f"unsafe identifier: {ident}") + return f"`{ident}`" + + def like_clause(self, quoted_column: str, pattern: str) -> tuple[str, list[Any]]: + # The default collation is case-INSENSITIVE; BINARY makes the match case-sensitive. + return f"{quoted_column} LIKE BINARY {self.placeholder}", [pattern] + + def encode_param(self, value: Any) -> Any: + if isinstance(value, _uuid.UUID): + return str(value) + if isinstance(value, _dt.datetime) and value.tzinfo is not None: + # A DATETIME has no zone and the runtime stores the UTC wall clock. + return value.astimezone(_dt.timezone.utc).replace(tzinfo=None) + if isinstance(value, _dt.time): + return value.isoformat() + return value + + def decode_value(self, field: MetaField, value: Any) -> Any: + return _decode_storage_value(field, value) + + +POSTGRES = PostgresDialect() +SQLITE = SqliteDialect() +MYSQL = MySqlDialect() diff --git a/server/python/src/metaobjects/runtime/drivers.py b/server/python/src/metaobjects/runtime/drivers.py new file mode 100644 index 000000000..91ad569a9 --- /dev/null +++ b/server/python/src/metaobjects/runtime/drivers.py @@ -0,0 +1,44 @@ +"""Drivers for the engines beyond Postgres: SQLite (stdlib ``sqlite3``) and MySQL (PyMySQL). + +Each is a :class:`~metaobjects.runtime.object_manager.DbApiDriver` that names its +:class:`~metaobjects.runtime.dialect.Dialect`; the ObjectManager reads the dialect off the +driver, so ``ObjectManager(root, SqliteDriver(conn))`` is the whole wiring. + +* **SQLite** has no per-column type codes (a column's declared type is not on the cursor), + so the ObjectManager derives the wire discriminator from each field's declared subtype. + ``RETURNING`` needs SQLite >= 3.35; on an older library the write path inserts and then + re-reads the row by key, as it does for MySQL. +* **MySQL** has no ``RETURNING`` at all; the write path re-reads the row by key (the + AUTO_INCREMENT key from ``lastrowid``, or the one the caller supplied). Open the PyMySQL + connection with ``charset="utf8mb4"``; a connection left on a UTC session + (``init_command="SET time_zone = '+00:00'"``) reads a ``TIMESTAMP`` as UTC. + +``sqlite3`` and PyMySQL are imported by the caller, who owns the connection; this module +imports neither. +""" +from __future__ import annotations + +from typing import Any + +from .dialect import MYSQL, SQLITE, Dialect +from .object_manager import DbApiDriver + + +class SqliteDriver(DbApiDriver): + """A ``sqlite3`` connection (qmark parameters, double-quoted identifiers).""" + + dialect: Dialect = SQLITE + + @staticmethod + def _type_codes(description: Any) -> list[Any]: + return [None for _ in description] + + +class MySqlDriver(DbApiDriver): + """A PyMySQL connection (``%s`` parameters, backtick identifiers).""" + + dialect: Dialect = MYSQL + + @staticmethod + def _type_codes(description: Any) -> list[Any]: + return [None for _ in description] diff --git a/server/python/src/metaobjects/runtime/object_manager.py b/server/python/src/metaobjects/runtime/object_manager.py index 503eff224..ba61da963 100644 --- a/server/python/src/metaobjects/runtime/object_manager.py +++ b/server/python/src/metaobjects/runtime/object_manager.py @@ -1,9 +1,10 @@ -"""ObjectManager + PostgresDriver — minimal query runtime. +"""ObjectManager + the DB-API drivers — minimal query runtime. -Compiles a Filter dict to parameterized SQL and runs it via a pg-style -driver. Identifiers are double-quoted throughout (mixed-case columns like -`programId` round-trip through PG); placeholders are pg8000 / psycopg-style -``%s``. +Compiles a Filter dict to parameterized SQL and runs it via a DB-API 2 driver. The SQL is +spelled by the driver's :class:`~metaobjects.runtime.dialect.Dialect`: Postgres (here: +``PostgresDriver``, pg8000 / psycopg) double-quotes identifiers and uses ``%s`` +placeholders (mixed-case columns like `programId` round-trip through PG); SQLite and MySQL +drivers live in :mod:`metaobjects.runtime.drivers`. Per ADR-0019 (runtime return-type contract) the query path returns **native, in-process Python types** — pg8000's own `int` / `Decimal` / `datetime` / @@ -28,6 +29,7 @@ import datetime as _dt import decimal as _decimal import json as _json +import re as _re import uuid as _uuid from collections.abc import Iterable, Mapping from typing import Any, Protocol @@ -54,6 +56,7 @@ collect_symmetric_target_ids, resolve_n2m_descriptor, ) +from .dialect import MYSQL, POSTGRES, Dialect from .tph import TphSubtype, tph_subtype_of from .validator_runner import ValidationResult, run_validators @@ -70,6 +73,8 @@ def execute(self, sql: str, params: tuple[Any, ...] = ...) -> Any: ... def fetchall(self) -> list[Any]: ... @property def description(self) -> Any: ... + @property + def rowcount(self) -> int: ... def close(self) -> None: ... @@ -96,24 +101,40 @@ def __init__(self, rows: list[dict[str, Any]], column_oids: dict[str, int]) -> N self.column_oids = column_oids -class PostgresDriver: - """Wrap a DB-API 2 connection (pg8000 / psycopg). Owns no state itself.""" +class DbApiDriver: + """Wrap a DB-API 2 connection. Owns no state beyond the connection and its + :class:`~metaobjects.runtime.dialect.Dialect`. + + Subclasses pick the dialect and the per-column type metadata; everything else (the + cursor handling, commit-per-write, parameter encoding) is shared. Statements arrive + already spelled for the dialect (quoting, placeholders), and every bound parameter is + run through :meth:`Dialect.encode_param` here, so no caller binds an engine-specific + spelling by hand.""" + + dialect: Dialect = POSTGRES def __init__(self, conn: Connection) -> None: self._conn = conn + def _bind(self, params: tuple[Any, ...]) -> tuple[Any, ...]: + return tuple(self.dialect.encode_param(p) for p in params) + + @staticmethod + def _type_codes(description: Any) -> list[Any]: + """Per-column SQL type codes beside the names (Postgres: the type OID).""" + return [d[1] for d in description] + + def _result(self, description: Any, rows: list[Any]) -> SelectResult: + cols = [d[0] for d in description] + codes = self._type_codes(description) + column_oids = {c: codes[i] for i, c in enumerate(cols) if codes[i] is not None} + return SelectResult([{c: row[i] for i, c in enumerate(cols)} for row in rows], column_oids) + def select(self, sql: str, params: tuple[Any, ...] = ()) -> SelectResult: cur = self._conn.cursor() try: - cur.execute(sql, params) - cols = [d[0] for d in cur.description] - oids = [d[1] for d in cur.description] - column_oids = {c: oids[i] for i, c in enumerate(cols)} - rows = [ - {c: v for c, v in zip(cols, row)} - for row in cur.fetchall() - ] - return SelectResult(rows, column_oids) + cur.execute(sql, self._bind(params)) + return self._result(cur.description, list(cur.fetchall())) finally: cur.close() @@ -136,13 +157,11 @@ def insert_returning(self, sql: str, params: tuple[Any, ...] = ()) -> SelectResu """ cur = self._conn.cursor() try: - cur.execute(sql, params) - cols = [d[0] for d in cur.description] - oids = [d[1] for d in cur.description] - row = cur.fetchone() + cur.execute(sql, self._bind(params)) + description = cur.description + rows = list(cur.fetchall())[:1] self._conn.commit() - column_oids = {c: oids[i] for i, c in enumerate(cols)} - return SelectResult([{c: row[i] for i, c in enumerate(cols)}], column_oids) + return self._result(description, rows) finally: cur.close() @@ -157,15 +176,26 @@ def update_returning( path — so a BIGINT/currency column updates back as a numeric string.""" cur = self._conn.cursor() try: - cur.execute(sql, params) - cols = [d[0] for d in cur.description] - oids = [d[1] for d in cur.description] - row = cur.fetchone() + cur.execute(sql, self._bind(params)) + description = cur.description + rows = list(cur.fetchall())[:1] self._conn.commit() - if row is None: + if not rows: return None - column_oids = {c: oids[i] for i, c in enumerate(cols)} - return SelectResult([{c: row[i] for i, c in enumerate(cols)}], column_oids) + return self._result(description, rows) + finally: + cur.close() + + def insert_last_id(self, sql: str, params: tuple[Any, ...] = ()) -> Any: + """Run an INSERT on an engine with no ``RETURNING`` and answer the key the + engine generated (``lastrowid``; ``0`` / ``None`` when the caller supplied the key). + Commits on success.""" + cur = self._conn.cursor() + try: + cur.execute(sql, self._bind(params)) + last = getattr(cur, "lastrowid", None) + self._conn.commit() + return last finally: cur.close() @@ -174,7 +204,7 @@ def execute_rowcount(self, sql: str, params: tuple[Any, ...] = ()) -> int: count. Commits on success.""" cur = self._conn.cursor() try: - cur.execute(sql, params) + cur.execute(sql, self._bind(params)) count = cur.rowcount self._conn.commit() return int(count) if count is not None else 0 @@ -182,13 +212,19 @@ def execute_rowcount(self, sql: str, params: tuple[Any, ...] = ()) -> int: cur.close() +class PostgresDriver(DbApiDriver): + """Wrap a pg8000 / psycopg connection (the Postgres dialect).""" + + dialect = POSTGRES + + class ObjectManager: """Method-based read API. Translates Filter dicts → parameterized SQL.""" def __init__( self, root: MetaRoot, - driver: PostgresDriver, + driver: DbApiDriver, column_naming: str | None = None, ) -> None: if column_naming is not None: @@ -201,6 +237,9 @@ def __init__( ) self._root = root self._driver = driver + #: The SQL dialect of the driver's engine; a driver that declares none (a test + #: double) is Postgres, the corpus' own engine. + self._d: Dialect = getattr(driver, "dialect", POSTGRES) # How a field with no explicit `@column` becomes a physical column name — # ADR-0058: THE one project setting, read from the loaded root # (`naming_rules_of`), so codegen's `names` generator and this runtime cannot @@ -328,28 +367,42 @@ def _insert_row( col_to_field = {_column_of(f, self._column_naming): f.name for f in returning_fields} if insert_cols: - col_list = ", ".join(_q(c) for c in insert_cols) - placeholders = ", ".join("%s" for _ in insert_cols) - sql = ( - f"INSERT INTO {_q(table)} ({col_list}) VALUES ({placeholders}) " - f"RETURNING {', '.join(_q(c) for c in all_cols)}" - ) + col_list = ", ".join(self._d.quote(c) for c in insert_cols) + placeholders = ", ".join(self._d.placeholder for _ in insert_cols) + sql = f"INSERT INTO {self._d.quote(table)} ({col_list}) VALUES ({placeholders})" else: - sql = ( - f"INSERT INTO {_q(table)} DEFAULT VALUES " - f"RETURNING {', '.join(_q(c) for c in all_cols)}" - ) - result = self._driver.insert_returning(sql, tuple(params)) + sql = f"INSERT INTO {self._d.quote(table)} {self._d.default_values}" + if self._d.supports_returning: + sql += f" RETURNING {', '.join(self._d.quote(c) for c in all_cols)}" + result = self._driver.insert_returning(sql, tuple(params)) + else: + # No RETURNING (MySQL): insert, then re-read the row by its key -- the key the + # caller supplied, else the one the engine generated (AUTO_INCREMENT). + generated = self._driver.insert_last_id(sql, tuple(params)) + pk_field = self._primary_pk_field(entity) + supplied = data.get(pk_field) + pk_value = supplied if supplied is not None else generated + if pk_value is None or (supplied is None and pk_value == 0): + raise ValueError( + f"create('{entity_name}'): the {self._d.name} engine cannot return a " + f"generated key other than an AUTO_INCREMENT one, so supply a value for " + f"'{pk_field}'" + ) + written = self._select_by_key(entity, returning_fields, pk_field, pk_value, tph) + if written is None: + raise RuntimeError( + f"create('{entity_name}'): the inserted row {pk_field}={pk_value!r} " + f"could not be read back" + ) + result = written # Surface the RETURNING column OIDs (mapped to metadata field names) so the # serialization boundary applies the same SQL-type wire rules as a read — a # BIGINT / currency column comes back as a numeric string. Mirrors update(). - self.last_column_oids = { - col_to_field.get(c, c): oid for c, oid in result.column_oids.items() - } + self.last_column_oids = self._type_codes(result, col_to_field, returning_fields) row = result.rows[0] mapped = {col_to_field.get(k, k): v for k, v in row.items()} # int-backed enums come back as integers; hand the caller the symbol. - mapped = _decode_read_row({f.name: f for f in returning_fields}, mapped) + mapped = _decode_read_row({f.name: f for f in returning_fields}, mapped, self._d) # #214: a write-through entity's INSERT RETURNING covers only the table (non-derived) # columns; re-read the persisted row through the replica VIEW by PK so the returned row # carries the derived origin.* fields (read-your-writes). find_by_id routes to the view. @@ -440,17 +493,22 @@ def update( returning_fields = [f for f in entity.fields() if not f.is_derived()] all_cols = [_column_of(f, self._column_naming) for f in returning_fields] col_to_field = {_column_of(f, self._column_naming): f.name for f in returning_fields} - assignments = ", ".join(f"{_q(c)} = %s" for c in set_cols) + assignments = ", ".join(f"{self._d.quote(c)} = {self._d.placeholder}" for c in set_cols) params.append(_coerce_write_value(entity.find_field(pk_field), id_value)) - where = f"{_q(pk_col)} = %s" + where = f"{self._d.quote(pk_col)} = {self._d.placeholder}" if tph is not None: - where += f" AND {_q(_column_of(entity.find_field(tph.field), self._column_naming))} = %s" + where += f" AND {self._d.quote(_column_of(entity.find_field(tph.field), self._column_naming))} = {self._d.placeholder}" params.append(tph.value) - sql = ( - f"UPDATE {_q(table)} SET {assignments} WHERE {where} " - f"RETURNING {', '.join(_q(c) for c in all_cols)}" - ) - result = self._driver.update_returning(sql, tuple(params)) + sql = f"UPDATE {self._d.quote(table)} SET {assignments} WHERE {where}" + if self._d.supports_returning: + sql += f" RETURNING {', '.join(self._d.quote(c) for c in all_cols)}" + result = self._driver.update_returning(sql, tuple(params)) + else: + # No RETURNING (MySQL): update, then re-read the (TPH-scoped) row by its key. A + # row that matched but changed nothing still reads back, which a rowcount would + # report as no match. + self._driver.execute_rowcount(sql, tuple(params)) + result = self._select_by_key(entity, returning_fields, pk_field, id_value, tph) if result is None: self.last_column_oids = {} return self._on_missing_update(entity_name, pk_field, id_value, if_missing) @@ -458,13 +516,11 @@ def update( # serialization boundary applies the same SQL-type wire rules as a read — # a BIGINT / currency column comes back as a numeric string. Mirrors the # find_many bookkeeping. - self.last_column_oids = { - col_to_field.get(c, c): oid for c, oid in result.column_oids.items() - } + self.last_column_oids = self._type_codes(result, col_to_field, returning_fields) row = result.rows[0] mapped = {col_to_field.get(k, k): v for k, v in row.items()} # int-backed enums come back as integers; hand the caller the symbol. - mapped = _decode_read_row({f.name: f for f in returning_fields}, mapped) + mapped = _decode_read_row({f.name: f for f in returning_fields}, mapped, self._d) # #214: re-read the updated row through the replica VIEW by PK so the returned row # carries the derived origin.* fields (write targeted the table, which lacks them). if entity.is_write_through(): @@ -473,6 +529,47 @@ def update( return reread return mapped + def _type_codes( + self, result: SelectResult, col_to_field: dict[str, str], fields: Iterable[MetaField] + ) -> dict[str, int]: + """The per-field SQL type metadata for the serialization boundary. A driver with + column OIDs (Postgres) supplies them; an engine with none (SQLite, MySQL) reports + the equivalent for each field's declared subtype.""" + if self._d.synthesizes_type_codes: + codes = ((f.name, self._d.type_code(f)) for f in fields) + return {name: code for name, code in codes if code is not None} + return {col_to_field.get(c, c): oid for c, oid in result.column_oids.items()} + + def _select_by_key( + self, + entity: MetaObject, + fields: list[MetaField], + pk_field: str, + key: Any, + tph: TphSubtype | None, + ) -> SelectResult | None: + """Read one row of the WRITE table by its key (TPH-scoped), or ``None``: how an + engine with no ``RETURNING`` hands back what it just wrote.""" + ph = self._d.placeholder + cols = [_column_of(f, self._column_naming) for f in fields] + pk = entity.find_field(pk_field) + if pk is None: + raise ValueError(f"Entity '{entity.name}' has no primary key field '{pk_field}'") + pk_col = _column_of(pk, self._column_naming) + params: list[Any] = [_coerce_write_value(pk, key)] + where = f"{self._d.quote(pk_col)} = {ph}" + if tph is not None: + where += f" AND {self._d.quote(_column_of(entity.find_field(tph.field), self._column_naming))} = {ph}" + params.append(tph.value) + sql = ( + f"SELECT {', '.join(self._d.quote(c) for c in cols)} " + f"FROM {self._d.quote(self._table_name(entity))} WHERE {where}" + ) + result = self._driver.select(sql, tuple(params)) + if not result.rows: + return None + return SelectResult(result.rows[:1], result.column_oids) + @staticmethod def _on_missing_update( entity_name: str, pk_field: str, id_value: Any, if_missing: str @@ -497,14 +594,14 @@ def delete(self, entity_name: str, id_value: Any) -> bool: pk_field = self._primary_pk_field(entity) pk_col = _column_of(entity.find_field(pk_field), self._column_naming) params: list[Any] = [_coerce_write_value(entity.find_field(pk_field), id_value)] - where = f"{_q(pk_col)} = %s" + where = f"{self._d.quote(pk_col)} = {self._d.placeholder}" # FR-017 TPH: a subtype delete is scoped to its discriminator (cross-subtype # delete matches no row → False, mirroring the per-subtype route's 404). tph = tph_subtype_of(entity) if tph is not None: - where += f" AND {_q(_column_of(entity.find_field(tph.field), self._column_naming))} = %s" + where += f" AND {self._d.quote(_column_of(entity.find_field(tph.field), self._column_naming))} = {self._d.placeholder}" params.append(tph.value) - sql = f"DELETE FROM {_q(table)} WHERE {where}" + sql = f"DELETE FROM {self._d.quote(table)} WHERE {where}" return self._driver.execute_rowcount(sql, tuple(params)) > 0 def find_many( @@ -522,13 +619,13 @@ def find_many( # fields (incl. derived) are selected — the view exposes them. table = self._read_source_name(entity) cols = [_column_of(f, self._column_naming) for f in entity.fields()] - sql = f'SELECT {", ".join(_q(c) for c in cols)} FROM {_q(table)}' + sql = f'SELECT {", ".join(self._d.quote(c) for c in cols)} FROM {self._d.quote(table)}' params: list[Any] = [] # FR-017 TPH: a subtype read is scoped to its discriminator value (a row of # a different subtype is invisible); the base entity (no @discriminatorValue) # reads polymorphically across the single table. filter = self._scope_filter(filter, tph_subtype_of(entity)) - where = _compile_filter(filter, entity, self._column_naming) if filter else None + where = _compile_filter(filter, entity, self._column_naming, self._d) if filter else None if where is not None: sql += " WHERE " + where[0] params.extend(where[1]) @@ -538,24 +635,24 @@ def find_many( f = entity.find_field(field_name) col = _column_of(f, self._column_naming) if f is not None else field_name d = "DESC" if direction.lower() == "desc" else "ASC" - order_parts.append(f"{_q(col)} {d}") + order_parts.append(f"{self._d.quote(col)} {d}") if order_parts: sql += " ORDER BY " + ", ".join(order_parts) if limit is not None: sql += f" LIMIT {int(limit)}" + elif offset is not None and self._d.unbounded_limit: + sql += f" LIMIT {self._d.unbounded_limit}" # OFFSET needs a LIMIT on this engine if offset is not None: sql += f" OFFSET {int(offset)}" result = self._driver.select(sql, tuple(params)) # Map raw column → metadata field name for cross-port row-shape parity. # Values stay native (ADR-0019); the boundary canonicalizes them. col_to_field = {_column_of(f, self._column_naming): f.name for f in entity.fields()} - self.last_column_oids = { - col_to_field.get(c, c): oid for c, oid in result.column_oids.items() - } + self.last_column_oids = self._type_codes(result, col_to_field, entity.fields()) # int-backed enums come back as integers; hand the caller the symbol. fields_by_name = {f.name: f for f in entity.fields()} return [ - _decode_read_row(fields_by_name, {col_to_field.get(k, k): v for k, v in row.items()}) + _decode_read_row(fields_by_name, {col_to_field.get(k, k): v for k, v in row.items()}, self._d) for row in result.rows ] @@ -564,10 +661,10 @@ def count(self, entity_name: str, filter: Filter | None = None) -> int: # #214: count reads through the same source as find_many (the replica view for a # write-through entity — a 1:1 replica, so the count is unchanged). table = self._read_source_name(entity) - sql = f"SELECT COUNT(*) FROM {_q(table)}" + sql = f"SELECT COUNT(*) FROM {self._d.quote(table)}" params: list[Any] = [] filter = self._scope_filter(filter, tph_subtype_of(entity)) # FR-017 TPH subtype scope - where = _compile_filter(filter, entity, self._column_naming) if filter else None + where = _compile_filter(filter, entity, self._column_naming, self._d) if filter else None if where is not None: sql += " WHERE " + where[0] params.extend(where[1]) @@ -616,17 +713,17 @@ def _relate_n2m( # Query the junction. Symmetric unions both FK columns; directed/hetero # filter only the source side. join_table = self._table_name(junction) - select_cols = f"{_q(source_col)}, {_q(target_col)}" + select_cols = f"{self._d.quote(source_col)}, {self._d.quote(target_col)}" if desc.symmetric: sql = ( - f"SELECT {select_cols} FROM {_q(join_table)} " - f"WHERE {_q(source_col)} = %s OR {_q(target_col)} = %s" + f"SELECT {select_cols} FROM {self._d.quote(join_table)} " + f"WHERE {self._d.quote(source_col)} = {self._d.placeholder} OR {self._d.quote(target_col)} = {self._d.placeholder}" ) params: tuple[Any, ...] = (source_id, source_id) else: sql = ( - f"SELECT {select_cols} FROM {_q(join_table)} " - f"WHERE {_q(source_col)} = %s" + f"SELECT {select_cols} FROM {self._d.quote(join_table)} " + f"WHERE {self._d.quote(source_col)} = {self._d.placeholder}" ) params = (source_id,) join_rows = self._driver.select(sql, params).rows @@ -785,7 +882,10 @@ def _primary_pk_field(self, entity: MetaObject) -> str: def _compile_filter( - f: Filter | None, entity: MetaObject, column_naming: str = DEFAULT_COLUMN_NAMING + f: Filter | None, + entity: MetaObject, + column_naming: str = DEFAULT_COLUMN_NAMING, + dialect: Dialect = POSTGRES, ) -> tuple[str, list[Any]] | None: if not f: return None @@ -794,7 +894,7 @@ def _compile_filter( parts: list[str] = [] params: list[Any] = [] for child in f["and"]: - compiled = _compile_filter(child, entity, column_naming) + compiled = _compile_filter(child, entity, column_naming, dialect) if compiled is None: continue parts.append("(" + compiled[0] + ")") @@ -811,11 +911,11 @@ def _compile_filter( if not isinstance(ops, dict): # Shortcut: {field: value} → equality. Encoded like any other bound # value (see _op_clause) so an int-backed enum reaches SQL as its int. - parts.append(f"{_q(col)} = %s") - params.append(_coerce_write_value(mf, ops) if mf is not None else ops) + parts.append(f"{dialect.quote(col)} = {dialect.placeholder}") + params.append(_coerce_filter_value(mf, ops) if mf is not None else ops) continue for op, value in ops.items(): - sql, p = _op_clause(col, op, value, mf) + sql, p = _op_clause(col, op, value, mf, dialect) parts.append(sql) params.extend(p) if not parts: @@ -823,7 +923,9 @@ def _compile_filter( return " AND ".join(parts), params -def _op_clause(col: str, op: str, value: Any, field: MetaField | None = None) -> tuple[str, list[Any]]: +def _op_clause( + col: str, op: str, value: Any, field: MetaField | None = None, dialect: Dialect = POSTGRES +) -> tuple[str, list[Any]]: """Translate one operator → SQL + params. Mirrors TS/C#/Java semantics. Bound values go through the WRITE codec, exactly as an INSERT's do: the four @@ -834,27 +936,33 @@ def _op_clause(col: str, op: str, value: Any, field: MetaField | None = None) -> keeps eq/ne/in for int-backed enums precisely BECAUSE the symbol is supposed to encode to its integer before reaching SQL. """ - qc = _q(col) + qc = dialect.quote(col) + ph = dialect.placeholder + if dialect is MYSQL and field is not None and field.sub_type == "decimal" and op not in ("isNull", "like"): + # ADR-0019 W2: MySQL compares DECIMAL = 'text' as doubles; a cast operand compares as a + # decimal, so a 19-digit operand is exact. + ph = f"CAST({ph} AS DECIMAL(65,30))" if field is not None and op != "isNull": - value = ([_coerce_write_value(field, v) for v in value] - if op == "in" and value else _coerce_write_value(field, value)) - if op == "eq": return f"{qc} = %s", [value] - if op == "ne": return f"{qc} <> %s", [value] - if op == "gt": return f"{qc} > %s", [value] - if op == "gte": return f"{qc} >= %s", [value] - if op == "lt": return f"{qc} < %s", [value] - if op == "lte": return f"{qc} <= %s", [value] + value = ([_coerce_filter_value(field, v) for v in value] + if op == "in" and value else _coerce_filter_value(field, value)) + if op == "eq": return f"{qc} = {ph}", [value] + if op == "ne": return f"{qc} <> {ph}", [value] + if op == "gt": return f"{qc} > {ph}", [value] + if op == "gte": return f"{qc} >= {ph}", [value] + if op == "lt": return f"{qc} < {ph}", [value] + if op == "lte": return f"{qc} <= {ph}", [value] # `like` is case-SENSITIVE SQL LIKE — verbatim pattern, %/_ wildcards - # (ADR-0049; the de-blinded filter-like-and-ne fixture gates this). - if op == "like": return f"{qc} LIKE %s", [value] + # (ADR-0049; the de-blinded filter-like-and-ne fixture gates this). The dialect + # spells it where the engine's own LIKE is not case-sensitive. + if op == "like": return dialect.like_clause(qc, value) if op == "isNull": wants_null = bool(value) if not isinstance(value, str) else value.lower() == "true" return (f"{qc} IS NULL" if wants_null else f"{qc} IS NOT NULL"), [] if op == "in": if not value: # Empty IN list — match nothing. - return "FALSE", [] - placeholders = ", ".join("%s" for _ in value) + return dialect.false_predicate(), [] + placeholders = ", ".join(ph for _ in value) return f"{qc} IN ({placeholders})", list(value) raise ValueError(f"Unsupported filter op '{op}' on column '{col}'") @@ -876,6 +984,39 @@ def _field_is_array(field: MetaField) -> bool: return field.resolved_is_array() +_FILTER_INT_SUBTYPES = (fc.FIELD_SUBTYPE_INT, fc.FIELD_SUBTYPE_LONG, fc.FIELD_SUBTYPE_CURRENCY) +_FILTER_FLOAT_SUBTYPES = (fc.FIELD_SUBTYPE_FLOAT, fc.FIELD_SUBTYPE_DOUBLE) + + +def _coerce_filter_value(field: MetaField, value: Any) -> Any: + """A filter operand as the native type the column compares against. + + A filter value usually arrives as TEXT (the REST grammar ``filter[f][op]=v`` hands the + repository the raw URL text). Postgres lets the driver pick the column type for a + bound string; SQLite and MySQL compare a bound string against a number as TEXT, so + ``count > '0'`` is wrong there. Numeric and boolean operands are parsed from text by + the field's declared subtype, then the write codec applies as for any bound value.""" + if isinstance(value, str): + sub = field.sub_type + if sub in _FILTER_INT_SUBTYPES: + value = int(value) if _INT_TEXT.match(value) else value + elif sub in _FILTER_FLOAT_SUBTYPES: + try: + value = float(value) + except ValueError: + pass + elif sub == fc.FIELD_SUBTYPE_BOOLEAN: + lowered = value.strip().lower() + if lowered in ("true", "1"): + value = True + elif lowered in ("false", "0"): + value = False + return _coerce_write_value(field, value) + + +_INT_TEXT = _re.compile(r"^\s*-?\d+\s*$") + + def _coerce_write_value(field: MetaField, value: Any) -> Any: if value is None: return None @@ -1008,10 +1149,17 @@ def _decode_read_value(field: MetaField, value: Any) -> Any: ) -def _decode_read_row(fields_by_name: dict[str, MetaField], row: dict[str, Any]) -> dict[str, Any]: - """Apply :func:`_decode_read_value` to every field-keyed value in a mapped row.""" +def _decode_read_row( + fields_by_name: dict[str, MetaField], row: dict[str, Any], dialect: Dialect = POSTGRES +) -> dict[str, Any]: + """Apply the engine's storage decode, then :func:`_decode_read_value`, to every + field-keyed value in a mapped row.""" return { - k: (_decode_read_value(fields_by_name[k], v) if k in fields_by_name else v) + k: ( + _decode_read_value(fields_by_name[k], dialect.decode_value(fields_by_name[k], v)) + if k in fields_by_name + else v + ) for k, v in row.items() } @@ -1099,10 +1247,3 @@ def _column_of(field: MetaField | None, strategy: str = DEFAULT_COLUMN_NAMING) - if field is None: return "" return resolve_column_name(field, strategy) - - -def _q(ident: str) -> str: - if '"' in ident: - raise ValueError(f"unsafe identifier: {ident}") - return f'"{ident}"' - diff --git a/server/python/src/metaobjects/super_resolve.py b/server/python/src/metaobjects/super_resolve.py index d1552ae91..6f195acac 100644 --- a/server/python/src/metaobjects/super_resolve.py +++ b/server/python/src/metaobjects/super_resolve.py @@ -12,7 +12,7 @@ from .errors import ErrorCode, MetaError from .meta.core.identity.identity_constants import IDENTITY_UNIQUE_KEY_SUBTYPES from .meta.meta_data import MetaData -from .naming_refs import RefOverDropError, owner_candidate_keys +from .naming_refs import RefOverDropError, did_you_mean_super_hint, owner_candidate_keys from .shared.base_types import TYPE_IDENTITY from .shared.separators import PACKAGE_SEP from .source import resolved_source @@ -129,7 +129,8 @@ def resolve_node(node: MetaData) -> None: # target. Mirrors TS resolveDeferredSupers in meta-data-loader.ts. errors.append(MetaError( f"the SuperClass '{node.super_ref}' does not exist " - f"(referenced by {node.fqn()})", + f"(referenced by {node.fqn()})" + f"{did_you_mean_super_hint(root, node.super_ref)}", ErrorCode.ERR_UNRESOLVED_SUPER, path=node.fqn(), envelope=resolved_source(node.source, node.fqn(), node.super_ref), diff --git a/server/python/tests/codegen/test_cli_help_removed_flags.py b/server/python/tests/codegen/test_cli_help_removed_flags.py new file mode 100644 index 000000000..0db48701b --- /dev/null +++ b/server/python/tests/codegen/test_cli_help_removed_flags.py @@ -0,0 +1,15 @@ +"""MINOR 6: a flag that is refused is not advertised in --help.""" +from __future__ import annotations + +import pytest + +from metaobjects.cli import main + + +@pytest.mark.parametrize("command", ["gen", "verify"]) +def test_help_does_not_list_the_removed_column_naming_flag( + command: str, capsys: pytest.CaptureFixture[str] +) -> None: + with pytest.raises(SystemExit): + main([command, "--help"]) + assert "--column-naming" not in capsys.readouterr().out diff --git a/server/python/tests/codegen/test_eject_alone.py b/server/python/tests/codegen/test_eject_alone.py index be8e33e5f..ad767376e 100644 --- a/server/python/tests/codegen/test_eject_alone.py +++ b/server/python/tests/codegen/test_eject_alone.py @@ -182,7 +182,7 @@ def test_an_ejected_generator_writes_what_the_packaged_one_writes(name: str, tmp # An owned router emits its runtime into the generated package and imports it there. runtime_files = {k for k in mine if k.startswith("_runtime/")} assert runtime_files and runtime_files == { - f"_runtime/{m}.py" for m in ("__init__", "constraint_errors", "filter_parser") + f"_runtime/{m}.py" for m in ("__init__", "constraint_errors", "decimal_wire", "filter_parser") } mine = {k: v for k, v in mine.items() if k not in runtime_files} packaged = {k: v.replace("metaobjects.codegen.runtime.", "._runtime.") diff --git a/server/python/tests/codegen/test_eject_runtime.py b/server/python/tests/codegen/test_eject_runtime.py index 3774d7c40..cecb54c74 100644 --- a/server/python/tests/codegen/test_eject_runtime.py +++ b/server/python/tests/codegen/test_eject_runtime.py @@ -31,7 +31,7 @@ REPO = Path(__file__).parents[4] FITNESS = REPO / "fixtures" / "persistence-conformance" / "canonical" / "meta.fitness.json" API_CONTRACT = REPO / "fixtures" / "api-contract-conformance" / "meta.json" -RUNTIME_MODULES = ("constraint_errors", "filter_parser") +RUNTIME_MODULES = ("constraint_errors", "decimal_wire", "filter_parser") INSTALLED_RUNTIME = "metaobjects.codegen.runtime" @@ -148,8 +148,14 @@ def test_eject_never_overwrites_an_owned_runtime_copy(tmp_path, monkeypatch, cap assert "filter_parser.py: kept your copy — DIFFERS: 0 behind, 1 of your own" in kept assert "may need the newer copy" not in kept assert main(["gen", "--list"]) == 0 - assert ("runtime codegen/runtime/filter_parser.py [owned — DIFFERS: 0 behind, 1 of your own]" - in capsys.readouterr().out) + listing = capsys.readouterr().out + assert "runtime codegen/runtime/filter_parser.py [owned — DIFFERS: 0 behind, 1 of your own]" in listing + # E1: the generator is identical only if every owned companion is too; the verdict folds + # the difference in and names each differing file. + lines = listing.splitlines() + at = next(i for i, ln in enumerate(lines) if ln.startswith("routes — ")) + assert "[owned — DIFFERS: generator identical, 1 runtime module(s) differ]" in lines[at] + assert lines[at + 1] == " runtime codegen/runtime/filter_parser.py [DIFFERS: 0 behind, 1 of your own]" def test_eject_of_a_generator_without_runtime_writes_no_runtime(tmp_path, monkeypatch) -> None: diff --git a/server/python/tests/codegen/test_project_import.py b/server/python/tests/codegen/test_project_import.py index 2a9a33c1a..b14a87314 100644 --- a/server/python/tests/codegen/test_project_import.py +++ b/server/python/tests/codegen/test_project_import.py @@ -298,3 +298,26 @@ def test_a_namespace_layout_in_the_project_loses_to_a_regular_package_earlier_lo (b / "codegen" / "generators" / "thing.py").write_text("WHO = 'B'\n") assert _who(a) == "A" assert _who(b) == "A" + + +def test_build_owned_instantiates_a_generator_class(tmp_path: Path) -> None: + """MINOR 2: `module:symbol` naming a generator CLASS yields an instance, not the class.""" + from metaobjects.codegen.eject import build_owned + + pkg = tmp_path / "owned_gen_pkg" + pkg.mkdir() + (pkg / "__init__.py").write_text("") + (pkg / "mine.py").write_text( + "class Mine:\n" + " name = 'mine'\n" + " def generate(self, ctx):\n" + " return []\n" + ) + try: + gen, err = build_owned("owned_gen_pkg.mine:Mine", tmp_path, None) # type: ignore[arg-type] + assert err is None + assert not isinstance(gen, type) and gen is not None and gen.name == "mine" + assert gen.generate(None) == [] # type: ignore[arg-type] + finally: + for k in [k for k in sys.modules if k.startswith("owned_gen_pkg")]: + del sys.modules[k] diff --git a/server/python/tests/codegen/test_report_router.py b/server/python/tests/codegen/test_report_router.py index d90ed6b1d..1599bdaf3 100644 --- a/server/python/tests/codegen/test_report_router.py +++ b/server/python/tests/codegen/test_report_router.py @@ -265,3 +265,32 @@ def test_codegen_of_an_any_measure_without_of_is_a_plain_nullable_bool_row() -> "allSlow: bool", "anySlowFailure: bool | None = None", ] + + +def test_a_derived_report_field_over_an_array_or_a_map_is_neither_filterable_nor_sortable() -> None: + """A jsonb-backed derived field has no meaningful order and no operator band, so the + served report's sort and filter allowlists leave it out (cross-port with Java).""" + from metaobjects.codegen.generators.filter_allowlist_generator import _compute_filterable_ops + from metaobjects.codegen.generators.router_generator import _scalar_fields + from metaobjects.meta.core.field.field_constants import FIELD_ATTR_FILTERABLE + from metaobjects.meta.core.field.meta_field import MetaField + from metaobjects.shared.base_types import TYPE_FIELD + + for sub_type, expect_kept in (("report", False), ("entity", True)): + obj = MetaObject(TYPE_OBJECT, sub_type, "Shape") + obj.package = "acme::test" + plain = MetaField(TYPE_FIELD, "string", "label") + arr = MetaField(TYPE_FIELD, "string", "tags") + arr.set_attr("isArray", True, sub_type="boolean") + mapped = MetaField(TYPE_FIELD, "map", "extras") + for f in (plain, arr, mapped): + f.set_attr(FIELD_ATTR_FILTERABLE, True, sub_type="boolean") + obj.add_child(f) + sortable = [f.name for f in _scalar_fields(obj)] + filterable = list(_compute_filterable_ops(obj)) + assert "label" in sortable and "label" in filterable + if not expect_kept: + assert "tags" not in sortable and "extras" not in sortable + assert "tags" not in filterable and "extras" not in filterable + else: + assert "tags" in sortable # an entity's behavior is unchanged diff --git a/server/python/tests/codegen/test_router_generator.py b/server/python/tests/codegen/test_router_generator.py index 53b6acfa1..7440bec47 100644 --- a/server/python/tests/codegen/test_router_generator.py +++ b/server/python/tests/codegen/test_router_generator.py @@ -120,9 +120,9 @@ def test_per_entity_sort_allowlist_constant_emitted() -> None: def test_with_count_returns_envelope_bare_list_otherwise() -> None: out = _require(_author()) # withCount=1 → {"rows": ..., "total": N} - assert 'return {"rows": rows, "total": total}' in out + assert 'return {"rows": to_wire(rows), "total": total}' in out # default → bare list - assert "return rows" in out + assert "return to_wire(rows)" in out # The 404 envelope and 400 invalid_sort envelope are part of the contract. assert '"error": "not_found"' in out # invalid_sort NAMES the rejected field — required cross-port on every @@ -237,7 +237,8 @@ def test_router_list_handler_calls_parse_filter() -> None: assert "filter_result.error_envelope" in out assert "return JSONResponse(status_code=400, content=filter_result.error_envelope)" in out # Predicates thread through to the repository's list + count. - assert "predicates = filter_result.predicates" in out + # (decimal operands are parsed exactly first, ADR-0019 Amendment 1 W2) + assert "filter_result.predicates, _DECIMAL_FIELDS" in out assert "repo.list(actual_limit, actual_offset, sort_clause, predicates)" in out assert "repo.count(predicates)" in out # Repository protocol gains the filters parameter on list + count. diff --git a/server/python/tests/codegen/test_router_generator_m2m_target_tph.py b/server/python/tests/codegen/test_router_generator_m2m_target_tph.py index 3fa29813f..6bec72af5 100644 --- a/server/python/tests/codegen/test_router_generator_m2m_target_tph.py +++ b/server/python/tests/codegen/test_router_generator_m2m_target_tph.py @@ -139,7 +139,7 @@ def test_route_body_passes_the_resolved_discriminator_value_literal() -> None: " sponsor_id: int,\n" " repo: Annotated[SponsorRepository, Depends(get_repository)],\n" ") -> list[Any]:\n" - ' return repo.find_related_bridgeAuths(sponsor_id, "Bridge")' + ' return to_wire(repo.find_related_bridgeAuths(sponsor_id, "Bridge"))' ) in out diff --git a/server/python/tests/codegen/test_router_generator_tph_m2m.py b/server/python/tests/codegen/test_router_generator_tph_m2m.py index f13af41f9..2692881a8 100644 --- a/server/python/tests/codegen/test_router_generator_tph_m2m.py +++ b/server/python/tests/codegen/test_router_generator_tph_m2m.py @@ -254,7 +254,7 @@ def test_base_declared_m2m_mounted_at_base_path_with_no_subtype_gate() -> None: " auth_id: int,\n" " repo: Annotated[AuthRepository, Depends(get_repository)],\n" ") -> list[Any]:\n" - " return repo.find_related_tags(None, auth_id)" + " return to_wire(repo.find_related_tags(None, auth_id))" ) in out # No gate anywhere near this handler — find_by_id is never called with None. assert 'repo.find_by_id(None, auth_id) is None' not in out @@ -275,7 +275,7 @@ def _gated_body( return ( f' if repo.find_by_id("{subtype}", {pk_param}) is None:\n' " return []\n" - f' return repo.find_related_{relation}("{subtype}", {pk_param}{tail})' + f' return to_wire(repo.find_related_{relation}("{subtype}", {pk_param}{tail}))' ) @@ -322,7 +322,7 @@ def test_priority_tickets_target_discriminator_independent_of_source_gate() -> N " auth_id: int,\n" " repo: Annotated[AuthRepository, Depends(get_repository)],\n" ") -> list[Any]:\n" - ' return repo.find_related_priorityTickets(None, auth_id, "Priority")' + ' return to_wire(repo.find_related_priorityTickets(None, auth_id, "Priority"))' ) in out # Inherited by every subtype too (rule b) — each subtype's OWN gate applies, # while the target literal stays fixed at "Priority" regardless. diff --git a/server/python/tests/codegen/test_router_generator_tph_m2m_shadowed.py b/server/python/tests/codegen/test_router_generator_tph_m2m_shadowed.py index 9bad9710b..c61f8fc2d 100644 --- a/server/python/tests/codegen/test_router_generator_tph_m2m_shadowed.py +++ b/server/python/tests/codegen/test_router_generator_tph_m2m_shadowed.py @@ -181,18 +181,18 @@ def test_shadowed_name_seam_threads_target_subtype_for_the_whole_name() -> None: def test_every_mount_of_the_shadowed_name_passes_the_threaded_parameter() -> None: # Base mount (rule a — ungated): explicit None literal, not a dropped argument. out = _render() - assert " return repo.find_related_helpers(None, auth_id, None)" in out + assert " return to_wire(repo.find_related_helpers(None, auth_id, None))" in out # Non-shadowing subtype (inherits the base's vanilla target): gated, None literal. assert ( ' if repo.find_by_id("Copay", auth_id) is None:\n' " return []\n" - ' return repo.find_related_helpers("Copay", auth_id, None)' + ' return to_wire(repo.find_related_helpers("Copay", auth_id, None))' ) in out # Shadowing subtype: its OWN target's discriminator literal. assert ( ' if repo.find_by_id("Bridge", auth_id) is None:\n' " return []\n" - ' return repo.find_related_helpers("Bridge", auth_id, "Priority")' + ' return to_wire(repo.find_related_helpers("Bridge", auth_id, "Priority"))' ) in out diff --git a/server/python/tests/codegen/test_tph_codegen.py b/server/python/tests/codegen/test_tph_codegen.py index 9b405a9ac..93528e27d 100644 --- a/server/python/tests/codegen/test_tph_codegen.py +++ b/server/python/tests/codegen/test_tph_codegen.py @@ -122,7 +122,7 @@ def test_base_router_polymorphic_plus_per_subtype(entities, index) -> None: assert f'@router.post("/{seg}", status_code=status.HTTP_201_CREATED)' in src assert f'@router.get("/{seg}/{{auth_id}}")' in src assert f'@router.delete("/{seg}/{{auth_id}}"' in src - assert f'return repo.create("{val}", dto)' in src + assert f'return to_wire(repo.create("{val}", dto))' in src # the per-subtype literal routes are emitted before the polymorphic /{id} assert src.index('@router.get("/bridge")') < src.index('@router.get("/{auth_id}")') diff --git a/server/python/tests/config/test_neutral_config_unknown_key.py b/server/python/tests/config/test_neutral_config_unknown_key.py new file mode 100644 index 000000000..d684590ed --- /dev/null +++ b/server/python/tests/config/test_neutral_config_unknown_key.py @@ -0,0 +1,22 @@ +"""An unknown top-level key in ``.metaobjects/config.json`` (one a newer release added) is +ignored: the neutral subset reads only what it knows.""" +from __future__ import annotations + +import json +import warnings +from pathlib import Path + +from metaobjects.config.neutral_config import read_neutral_config + + +def test_an_unknown_top_level_key_is_ignored(tmp_path: Path) -> None: + (tmp_path / ".metaobjects").mkdir() + (tmp_path / ".metaobjects" / "config.json").write_text( + json.dumps({"schema_version": 1, "sources": [{"path": "m"}], "someFutureKey": {"x": 1}}), + encoding="utf-8", + ) + with warnings.catch_warnings(): + warnings.simplefilter("error") + cfg = read_neutral_config(tmp_path) + assert cfg is not None + assert cfg.sources == [{"path": "m"}] diff --git a/server/python/tests/conformance/conformance_adapter.py b/server/python/tests/conformance/conformance_adapter.py index 4345d1855..f49f406da 100644 --- a/server/python/tests/conformance/conformance_adapter.py +++ b/server/python/tests/conformance/conformance_adapter.py @@ -1,6 +1,8 @@ """Bridge the corpus to the Python loader.""" from __future__ import annotations +import dataclasses + from dataclasses import dataclass from pathlib import Path from typing import Iterable, Optional @@ -136,6 +138,8 @@ class ErrorEnvelopeRecord: json_path: Optional[str] referrer: Optional[str] = None target: Optional[str] = None + #: The error's message text; asserted only through ``messageIncludes``. + message: str = "" def _relativize(file_path: str, input_dir: Path) -> str: @@ -268,7 +272,10 @@ def load_fixture_with_envelopes( return [compose_code], [compose_env], [], [], "" assert result is not None codes = [e.code.name for e in result.errors] - envelopes = [_build_envelope(e, input_dir) for e in result.errors] + envelopes = [ + dataclasses.replace(_build_envelope(e, input_dir), message=str(e.message)) + for e in result.errors + ] warning_envelopes = [ _build_warning_envelope(w, input_dir) for w in result.get_envelope_warnings() ] diff --git a/server/python/tests/conformance/test_conformance.py b/server/python/tests/conformance/test_conformance.py index 0d55ced85..c10cd3dc1 100644 --- a/server/python/tests/conformance/test_conformance.py +++ b/server/python/tests/conformance/test_conformance.py @@ -51,6 +51,7 @@ def _parse_expected_errors(raw: object) -> tuple[list[dict], list[dict], bool]: errors.append({ "code": e["code"], "source": src if isinstance(src, dict) else None, + "messageIncludes": e.get("messageIncludes") or [], }) raw_warnings = raw.get("warnings", []) warnings: list[dict] = [] @@ -128,6 +129,11 @@ def _run_checks(fix: Fixture) -> tuple[bool, str]: failures.append( f"envelope[{i}].code: expected '{w['code']}', got '{g.code}'") continue + for needle in w.get("messageIncludes", []): + if needle not in g.message: + failures.append( + f"envelope[{i}].message: expected to include {needle!r}, " + f"got {g.message!r}") src = w["source"] if src is None: continue diff --git a/server/python/tests/integration/api_contract_assertions.py b/server/python/tests/integration/api_contract_assertions.py index 772dccb93..11a6f3344 100644 --- a/server/python/tests/integration/api_contract_assertions.py +++ b/server/python/tests/integration/api_contract_assertions.py @@ -10,6 +10,8 @@ """ from __future__ import annotations +import json +from decimal import Decimal from typing import Any @@ -212,16 +214,30 @@ def assert_response( ) +def parse_body(text: str) -> Any: + """A response body as JSON, every non-integer number as an exact `Decimal`.""" + return json.loads(text, parse_float=Decimal) + + +def _is_number(v: Any) -> bool: + return isinstance(v, (int, float, Decimal)) and not isinstance(v, bool) + + +def _exact(v: Any) -> Decimal: + return Decimal(repr(v)) if isinstance(v, float) else Decimal(v) + + def _structural_equals(a: Any, b: Any) -> bool: - """Recursively compare; ints and floats with equal long value match.""" + """Recursively compare; numbers exactly.""" if a is b: return True if a is None or b is None: return a is None and b is None - # Numbers: compare as int when both are numeric (mirror Java's longValue compare). - if isinstance(a, (int, float)) and not isinstance(a, bool) \ - and isinstance(b, (int, float)) and not isinstance(b, bool): - return int(a) == int(b) + # Numbers compare EXACTLY (ADR-0019 Amendment 1): a body is parsed with + # `parse_float=Decimal` (see `parse_body`), and a float from a scenario file is read as + # the decimal it was written as. Truncating to int would pass `0.4` against `0.9`. + if _is_number(a) and _is_number(b): + return _exact(a) == _exact(b) if isinstance(a, list) and isinstance(b, list): if len(a) != len(b): return False diff --git a/server/python/tests/integration/conftest.py b/server/python/tests/integration/conftest.py new file mode 100644 index 000000000..553eb0cb6 --- /dev/null +++ b/server/python/tests/integration/conftest.py @@ -0,0 +1,22 @@ +"""Session fixtures for the integration lanes.""" +from __future__ import annotations + +from collections.abc import Iterator + +import pytest + +from .mysql_container import MySqlContainer, MySqlInfo, reset_database + + +@pytest.fixture(scope="session") +def mysql_server() -> Iterator[MySqlInfo]: + """ONE MySQL 8.4 server for the whole session, started on first use. Callers get an + EMPTY database each (:func:`mysql_empty`), so sharing the server shares no state.""" + with MySqlContainer() as server: + yield server.info() + + +@pytest.fixture +def mysql_empty(mysql_server: MySqlInfo) -> MySqlInfo: + reset_database(mysql_server) + return mysql_server diff --git a/server/python/tests/integration/engines.py b/server/python/tests/integration/engines.py new file mode 100644 index 000000000..31f204d0d --- /dev/null +++ b/server/python/tests/integration/engines.py @@ -0,0 +1,87 @@ +"""The SQL engine a persistence scenario runs on. + +Postgres is the corpus' own engine: the scenario YAML holds its seed SQL and its wire-form +``expect``. SQLite and MySQL run the same scenarios against their own TypeScript-produced +schema artifact (``canonical/schema..sql``), with the seed re-spelled for the engine +where its spelling differs. The ``expect`` blocks are NEVER loosened: the runtime hands back +native Python types decoded from the field's declared type, and the same serialization +boundary (``normalization.py``) maps them onto the wire form on the actual side only. + +Mirrors the Kotlin ``ScenarioEngine`` and the C# ``ScenarioEngine`` runner type. +""" +from __future__ import annotations + +import re +from enum import Enum + +from .scenarios import QueryScenario + + +class ScenarioEngine(Enum): + POSTGRES = "postgres" + SQLITE = "sqlite" + MYSQL = "mysql" + + @property + def schema_artifact(self) -> str: + """The schema artifact, relative to the corpus root.""" + return f"canonical/schema.{self.value}.sql" + + def seed(self, scenario: QueryScenario) -> str | None: + """The scenario's seed SQL spelled for this engine, or ``None``.""" + own = (scenario.seed_data_engine or {}).get(self.value) + if own is not None: + return own + if self is ScenarioEngine.MYSQL and scenario.seed_data: + return to_mysql_seed(scenario.seed_data) + return scenario.seed_data + + +_UTC_INSTANT = re.compile(r"'(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?)Z'") + + +def to_mysql_seed(seed: str) -> str: + """Postgres-quoted seed SQL to MySQL. + + Walks the text once so a double quote INSIDE a single-quoted string (a JSON payload) is + left alone and only a double-quoted identifier is rewritten to backticks; a UTC instant + literal loses its ``Z`` (a DATETIME holds the UTC wall clock). + """ + out: list[str] = [] + in_string = False + in_ident = False + i = 0 + while i < len(seed): + c = seed[i] + if in_string: + out.append(c) + if c == "'": + if i + 1 < len(seed) and seed[i + 1] == "'": + out.append("'") + i += 1 + else: + in_string = False + elif in_ident: + if c == '"': + out.append("`") + in_ident = False + else: + out.append(c) + elif c == "'": + in_string = True + out.append(c) + elif c == '"': + in_ident = True + out.append("`") + else: + out.append(c) + i += 1 + return _UTC_INSTANT.sub(r"'\1'", "".join(out)) + + +def split_statements(script: str) -> list[str]: + """The statements of a script: split on a ``;`` at the end of a line (no body carries one + mid-line), after dropping comment-only lines. SQLite and MySQL drivers run one statement + per call.""" + kept = "\n".join(ln for ln in script.splitlines() if not ln.lstrip().startswith("--")) + return [s.strip() for s in re.split(r";[ \t]*(?:\n|$)", kept) if s.strip()] diff --git a/server/python/tests/integration/generated_projection_app.py b/server/python/tests/integration/generated_projection_app.py index fb7f976ce..011e98142 100644 --- a/server/python/tests/integration/generated_projection_app.py +++ b/server/python/tests/integration/generated_projection_app.py @@ -25,6 +25,7 @@ import sys import tempfile import uuid +from decimal import Decimal from pathlib import Path from typing import Any @@ -36,6 +37,7 @@ from metaobjects.codegen.runtime.filter_parser import FilterPredicate from metaobjects.meta.core.object.meta_object import MetaObject from metaobjects.shared.base_types import TYPE_OBJECT +from metaobjects.meta.core.field.field_constants import FIELD_SUBTYPE_DECIMAL # Each corpus projection: the columns its view returns (view column -> `invoices` column), # and the view column its generated seam addresses a row by (None: no declared identity, @@ -140,7 +142,7 @@ def build_generated_projection_app( sys.modules[f"{pkg_name}.{snake}_router"] = router_mod spec.loader.exec_module(router_mod) - repo = InMemoryProjectionRepository(columns, key) + repo = InMemoryProjectionRepository(columns, key, _decimal_fields([projections[name]])) # The generated Protocol is the contract: a keyless projection must not have # grown a `find_by_id`, and a keyed one must have kept it. has_find = "find_by_id" in inspect.getsource(router_mod.__dict__[f"{name}Repository"]) @@ -156,6 +158,20 @@ def build_generated_projection_app( return app, repos +def _decimal_fields(objects: list[MetaObject]) -> frozenset[str]: + """The ``field.decimal`` names across *objects*: a real driver hands those back as + ``Decimal``, so the in-memory repo holds them that way rather than as JSON floats.""" + return frozenset( + f.name for o in objects for f in o.fields() if f.sub_type == FIELD_SUBTYPE_DECIMAL + ) + + +def _as_driver_row(row: dict[str, Any], decimal_fields: frozenset[str]) -> dict[str, Any]: + return { + k: (Decimal(str(v)) if k in decimal_fields and v is not None else v) for k, v in row.items() + } + + class InMemoryProjectionRepository: """In-memory impl of a GENERATED read-only ``Repository``. @@ -165,16 +181,22 @@ class InMemoryProjectionRepository: generated Protocol. """ - def __init__(self, columns: dict[str, str], key: str | None) -> None: + def __init__( + self, columns: dict[str, str], key: str | None, decimal_fields: frozenset[str] = frozenset() + ) -> None: self._columns = columns self._key = key + self._decimal_fields = decimal_fields self._rows: list[dict[str, Any]] = [] def reset(self) -> None: self._rows = [] def seed(self, rows: list[dict[str, Any]]) -> None: - self._rows = [{col: r[src] for col, src in self._columns.items()} for r in rows] + self._rows = [ + _as_driver_row({col: r[src] for col, src in self._columns.items()}, self._decimal_fields) + for r in rows + ] def _coerce(self, field: str, raw: str) -> Any: for r in self._rows: @@ -186,6 +208,8 @@ def _coerce(self, field: str, raw: str) -> Any: return int(raw) if isinstance(v, float): return float(raw) + if isinstance(v, Decimal): + return Decimal(raw) break return raw diff --git a/server/python/tests/integration/generated_report_app.py b/server/python/tests/integration/generated_report_app.py index 2aa3ddb9e..c3dfdde16 100644 --- a/server/python/tests/integration/generated_report_app.py +++ b/server/python/tests/integration/generated_report_app.py @@ -1,31 +1,39 @@ -"""FR-044 — boot the GENERATED read-only report routers over HTTP. +"""FR-044 — boot the GENERATED read-only report routers over HTTP, over a REAL database. Peer of ``generated_projection_app.py``, for the ``report/`` api-contract corpus (``fixtures/api-contract-conformance/report/``). Runs the REAL generation path (``run_gen``, the one ``metaobjects gen`` takes) for the served reports, writes the emitted package to a temp dir, imports each generated router UNMODIFIED, and mounts it -behind an in-memory repository. +behind a repository that reads the REAL view through the ``ObjectManager``. The generated routers are the artifact under test: what is under test is whether this -port's GENERATOR emits a keyless read-only surface for a view-backed ``object.report``. - -Python emits no SQL for a report (ADR-0015: view SQL is TypeScript-only), so the in-memory -repository stands in for the view. It is seeded with the ``reports`` half of the corpus -``seed.json``, which is what those views return for the base-table rows (the other -top-level keys, which this lane does not load: it has no tables). Only -``list`` / ``count`` exist, matching the generated ``Protocol``: a report has no -``find_by_id``. +port's GENERATOR emits a keyless read-only surface for a view-backed ``object.report``, +and whether that surface answers correctly over the view's rows. Python emits no SQL for a +report (ADR-0015: view SQL is TypeScript-only), so the view comes from the committed, +TypeScript-produced ``report/schema..sql``, executed verbatim on Postgres, SQLite +and MySQL, with the base tables seeded from the corpus ``seed.json`` (every key but +``reports``). The ``reports`` half is what the views are expected to return: the lane +cross-checks the views against it instead of serving it. + +The repository is the adopter-owned seam of a generated router (the generated ``Protocol`` +consumer implements with their persistence layer). :class:`ObjectManagerReportRepository` is +that implementation: it turns the parsed filter predicates, sort and paging into +``find_many`` / ``count``. Only ``list`` / ``count`` exist, matching the generated +``Protocol``: a report has no ``find_by_id``. """ from __future__ import annotations -import datetime +import functools import importlib.util import shutil import sys import tempfile +import threading import uuid -from decimal import Decimal +from collections.abc import Callable +from dataclasses import dataclass from pathlib import Path +from types import ModuleType from typing import Any from fastapi import FastAPI @@ -35,8 +43,8 @@ from metaobjects.codegen.generator_registry import GeneratorBuildContext, list_generators from metaobjects.codegen.runner import run_gen from metaobjects.codegen.runtime.filter_parser import FilterPredicate - -from .generated_router_app import _like_to_regex +from metaobjects.naming import attach_naming +from metaobjects.runtime import Filter, ObjectManager #: The served reports, as (report name, generated module stem). ``ProductRevenue`` #: declares ``@spine``: its seeded rows include the product with no sale. @@ -50,6 +58,7 @@ "SalesByCategory": "sales_by_category", "SalesBySku": "sales_by_sku", "ProductRefunds": "product_refunds", + "TicketsByTags": "tickets_by_tags", } #: The sourceless report: generated nowhere, mounted nowhere. UNSERVED_REPORT = "InvoiceDays" @@ -77,14 +86,33 @@ def _import(name: str, path: Path, **kwargs: Any): return module -def build_generated_report_app( - corpus_root: Path, -) -> tuple[FastAPI, dict[str, "InMemoryReportRepository"], set[str]]: - """Generate the report routers, import them, mount them, and wire the seams. +@dataclass(frozen=True) +class GeneratedReports: + """The generated package: its router modules by report, and the files emitted.""" - Returns ``(app, repos by report name, names of the files generation emitted)``. - """ + routers: dict[str, ModuleType] + emitted: set[str] + + +@dataclass(frozen=True) +class LaneModel: + """What a corpus serves: the view-backed reports and, optionally, writable entities, each + as (name, generated module stem), plus the sourceless report that must mount nothing.""" + + reports: tuple[tuple[str, str], ...] + entities: tuple[tuple[str, str], ...] = () + unserved: str | None = None + + +REPORT_MODEL = LaneModel(tuple(SERVED_REPORTS.items()), (), UNSERVED_REPORT) + + +@functools.cache +def generate_reports(corpus_root: Path, model: LaneModel = REPORT_MODEL) -> GeneratedReports: + """Generate the routers (the real ``run_gen`` path) and import each UNMODIFIED.""" root = _load_root(corpus_root / "meta.json") + served = dict(model.reports) + writable = dict(model.entities) pkg_name = f"genreport_{uuid.uuid4().hex[:8]}" tmp = Path(tempfile.mkdtemp(prefix="apic-report-gen-")) out = tmp / pkg_name @@ -99,112 +127,107 @@ def build_generated_report_app( GenConfig(out_dir=str(out)), root, generators=generators, - entity_filter=[*SERVED_REPORTS, UNSERVED_REPORT], + entity_filter=[*served, *writable, *([model.unserved] if model.unserved else [])], ) emitted = {p.name for p in out.rglob("*.py")} sys.path.insert(0, str(tmp)) _import(pkg_name, out / "__init__.py", submodule_search_locations=[str(out)]) + routers = { + name: _import(f"{pkg_name}.{stem}_router", out / f"{stem}_router.py") + for name, stem in {**served, **writable}.items() + } + return GeneratedReports(routers, emitted) + + +def load_runtime_root(corpus_root: Path) -> Any: + """The corpus model for the ObjectManager: the project's pinned naming attached (the + sub-corpus lives under ``api-contract-conformance/``, whose ``.metaobjects/config.json`` + pins literal columns, which is what the committed schemas spell).""" + return attach_naming(_load_root(corpus_root / "meta.json"), corpus_root.parent) + +def build_generated_report_app( + corpus_root: Path, om: ObjectManager, model: LaneModel = REPORT_MODEL +) -> tuple[FastAPI, set[str]]: + """Mount the generated routers behind ObjectManager-backed repositories. + + Returns ``(app, names of the files generation emitted)``. + """ + generated = generate_reports(corpus_root, model) app = FastAPI() - repos: dict[str, InMemoryReportRepository] = {} - for report, stem in SERVED_REPORTS.items(): - router_mod = _import(f"{pkg_name}.{stem}_router", out / f"{stem}_router.py") - repo = InMemoryReportRepository() - repos[report] = repo + lock = threading.Lock() # one connection serves the app; the threadpool runs handlers + writable = dict(model.entities) + for report, router_mod in generated.routers.items(): + repo: Any = ( + ObjectManagerEntityRepository(om, report, lock) + if report in writable + else ObjectManagerReportRepository(om, report, lock) + ) app.include_router(router_mod.router) - app.dependency_overrides[router_mod.get_repository] = lambda r=repo: r - return app, repos, emitted - - -def seed_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any]]: - """What a driver hands back for the view's columns: a ``date`` column is a ``date`` - and a ``numeric`` one a ``Decimal``. The seed spells both as strings so no float sits - between the corpus and the repository.""" - out: list[dict[str, Any]] = [] - for row in rows: - typed = dict(row) - if isinstance(typed.get("issuedOnMonth"), str): - typed["issuedOnMonth"] = datetime.date.fromisoformat(typed["issuedOnMonth"]) - if isinstance(typed.get("paidShare"), str): - typed["paidShare"] = Decimal(typed["paidShare"]) - out.append(typed) - return out - - -class InMemoryReportRepository: - """In-memory impl of a GENERATED read-only ``Repository`` (``list`` / ``count``).""" - - def __init__(self) -> None: - self._rows: list[dict[str, Any]] = [] - - def reset(self) -> None: - self._rows = [] - - def seed(self, rows: list[dict[str, Any]]) -> None: - self._rows = [dict(r) for r in rows] - - def _coerce(self, field: str, raw: str) -> Any: - for r in self._rows: - v = r.get(field) - if v is None: - continue - if isinstance(v, bool): - return raw == "true" - if isinstance(v, int): - return int(raw) - if isinstance(v, float): - return float(raw) - if isinstance(v, Decimal): - return Decimal(raw) - if isinstance(v, datetime.date): - return datetime.date.fromisoformat(raw) - break - return raw - - def _matches(self, row: dict[str, Any], p: FilterPredicate) -> bool: - actual = row.get(p.field) - if p.op == "isNull": - return (actual is None) == bool(p.value) - if actual is None: - return False - if p.op == "in": - return actual in {self._coerce(p.field, str(v)) for v in p.value} - want = self._coerce(p.field, str(p.value)) - if p.op == "eq": - return actual == want - if p.op == "ne": - return actual != want - if p.op == "like": - # One SQL-LIKE translation for the port: the helper the generated-router seam - # uses. A second in-memory seam re-deriving it here would let the two drift. - return _like_to_regex(str(want)).match(str(actual)) is not None - if p.op == "gt": - return actual > want - if p.op == "gte": - return actual >= want - if p.op == "lt": - return actual < want - if p.op == "lte": - return actual <= want - raise ValueError(f"unsupported op: {p.op}") - - def _filtered(self, filters: list[FilterPredicate]) -> list[dict[str, Any]]: - rows = self._rows - for p in filters: - rows = [r for r in rows if self._matches(r, p)] - return rows - - # --- the GENERATED read-only Protocol surface --- + # No default-valued parameter: FastAPI would read it as a query parameter and + # deepcopy its default, and a repository holds a live connection. + app.dependency_overrides[router_mod.get_repository] = _provide(repo) + return app, generated.emitted + + +def _provide(repo: Any) -> Callable[[], Any]: + def provider() -> Any: + return repo + + return provider + + +class ObjectManagerReportRepository: + """The adopter-owned seam of a GENERATED read-only ``Repository`` (``list`` / + ``count``), backed by the ``ObjectManager`` reading the report's real view. + + The generated router parses the query string into :class:`FilterPredicate` records + (values still TEXT: ``"0"``, ``"true"``, a date) and a sort clause; this class only + translates them to the ObjectManager's filter dict and ``sort`` pairs. Typing the text + is the runtime's job (it parses a filter operand by the field's declared type), and an + operator is the ObjectManager's own, one for one. + """ + + def __init__(self, om: ObjectManager, report: str, lock: threading.Lock) -> None: + self._om = om + self._report = report + self._lock = lock + + @staticmethod + def _filter(predicates: list[FilterPredicate]) -> Filter | None: + if not predicates: + return None + return {"and": [{p.field: {p.op: p.value}} for p in predicates]} + def list(self, limit: int, offset: int, sort: Any, filters: list[FilterPredicate]) -> list[Any]: - rows = self._filtered(filters) - if sort is not None: - rows = sorted( - rows, - key=lambda r: (r.get(sort.field) is None, r.get(sort.field)), - reverse=(sort.direction == "desc"), + order = [(sort.field, sort.direction)] if sort is not None else None + with self._lock: + return self._om.find_many( + self._report, self._filter(filters), sort=order, limit=limit, offset=offset ) - return [dict(r) for r in rows[offset : offset + limit]] def count(self, filters: list[FilterPredicate]) -> int: - return len(self._filtered(filters)) + with self._lock: + return self._om.count(self._report, self._filter(filters)) + + +class ObjectManagerEntityRepository(ObjectManagerReportRepository): + """The seam of a GENERATED writable ``Repository``: the report seam plus + ``find_by_id`` / ``create`` / ``update`` / ``delete``, over the ObjectManager.""" + + def find_by_id(self, id: Any) -> Any: + with self._lock: + return self._om.find_by_id(self._report, id) + + def create(self, dto: Any) -> Any: + with self._lock: + return self._om.create(self._report, dto) + + def update(self, id: Any, dto: Any) -> Any: + with self._lock: + return self._om.update(self._report, id, dto) + + def delete(self, id: Any) -> bool: + with self._lock: + return self._om.delete(self._report, id) diff --git a/server/python/tests/integration/generated_router_app.py b/server/python/tests/integration/generated_router_app.py index b9299c750..4b7d272db 100644 --- a/server/python/tests/integration/generated_router_app.py +++ b/server/python/tests/integration/generated_router_app.py @@ -28,6 +28,7 @@ import sys import tempfile import uuid +from decimal import Decimal from pathlib import Path from typing import Any @@ -161,6 +162,8 @@ def coerce(field: str, raw: str) -> Any: return int(raw) if t is float: return float(raw) + if t is Decimal: + return Decimal(raw) return raw def matches(row: dict[str, Any], p: FilterPredicate) -> bool: diff --git a/server/python/tests/integration/generated_tph_app.py b/server/python/tests/integration/generated_tph_app.py index e87ddddcb..3bb431004 100644 --- a/server/python/tests/integration/generated_tph_app.py +++ b/server/python/tests/integration/generated_tph_app.py @@ -23,6 +23,7 @@ import sys import tempfile import uuid +from decimal import Decimal from pathlib import Path from typing import Any @@ -36,6 +37,7 @@ from metaobjects.codegen.generators.tph_plan import tph_plan_for from metaobjects.meta.core.object.meta_object import MetaObject from metaobjects.shared.base_types import TYPE_OBJECT +from metaobjects.meta.core.field.field_constants import FIELD_SUBTYPE_DECIMAL from .generated_router_app import filter_rows @@ -78,7 +80,7 @@ def build_generated_tph_app(tph_dir: Path) -> tuple[FastAPI, "InMemoryTphReposit raise RuntimeError("no TPH discriminator base found in tph meta.json") plan = tph_plan_for(base, index) assert plan is not None - repo = InMemoryTphRepository(plan.discriminator_field) + repo = InMemoryTphRepository(plan.discriminator_field, _decimal_fields(list(entities.values()))) snake = _snake(base.name) pkg_name = f"gentph_{uuid.uuid4().hex[:8]}" @@ -126,6 +128,20 @@ def build_generated_tph_app(tph_dir: Path) -> tuple[FastAPI, "InMemoryTphReposit return app, repo +def _decimal_fields(objects: list[MetaObject]) -> frozenset[str]: + """The ``field.decimal`` names across *objects*: a real driver hands those back as + ``Decimal``, so the in-memory repo holds them that way rather than as JSON floats.""" + return frozenset( + f.name for o in objects for f in o.fields() if f.sub_type == FIELD_SUBTYPE_DECIMAL + ) + + +def _as_driver_row(row: dict[str, Any], decimal_fields: frozenset[str]) -> dict[str, Any]: + return { + k: (Decimal(str(v)) if k in decimal_fields and v is not None else v) for k, v in row.items() + } + + # --------------------------------------------------------------------------- # In-memory single-table repo behind the GENERATED subtype-keyed seam. Replays # the TPH discriminator scope: inject-on-create / scope-on-read / strip-on-update. @@ -134,8 +150,9 @@ def build_generated_tph_app(tph_dir: Path) -> tuple[FastAPI, "InMemoryTphReposit class InMemoryTphRepository: - def __init__(self, discriminator_field: str) -> None: + def __init__(self, discriminator_field: str, decimal_fields: frozenset[str] = frozenset()) -> None: self._disc = discriminator_field + self._decimal_fields = decimal_fields self._rows: list[dict[str, Any]] = [] self._next_id = 1 @@ -145,7 +162,7 @@ def reset(self) -> None: self._next_id = 1 def seed(self, rows: list[dict[str, Any]]) -> None: - self._rows = [dict(r) for r in rows] + self._rows = [_as_driver_row(r, self._decimal_fields) for r in rows] self._next_id = max((int(r["id"]) for r in self._rows), default=0) + 1 # --- generated subtype-keyed Protocol surface --- diff --git a/server/python/tests/integration/mysql_container.py b/server/python/tests/integration/mysql_container.py new file mode 100644 index 000000000..3f05807aa --- /dev/null +++ b/server/python/tests/integration/mysql_container.py @@ -0,0 +1,176 @@ +"""A throwaway MySQL 8.4 server for the MySQL conformance lanes. + +Shells out to the ``docker`` CLI exactly like :mod:`postgres_container` (no +python-testcontainers, whose docker-API negotiation lags current Docker engines). One +server is started for a whole pytest session (see ``conftest.py``); every scenario starts +from an EMPTY schema via :func:`reset_database`, which is what the Java and Kotlin lanes do. + +Set ``METAOBJECTS_TEST_MYSQL_URL`` (``mysql://user:password@host:port/database``) to run +against an existing server instead: the tests then OWN that database and drop everything in it. +""" +from __future__ import annotations + +import os +import socket +import subprocess +import time +import uuid +from contextlib import closing +from dataclasses import dataclass +from typing import Any +from urllib.parse import unquote, urlsplit + +SHARED_MYSQL_URL_ENV = "METAOBJECTS_TEST_MYSQL_URL" + + +@dataclass(frozen=True) +class MySqlInfo: + host: str + port: int + database: str + user: str + password: str + + +def connect(info: MySqlInfo, **kwargs: Any) -> Any: + """A PyMySQL connection to *info*. ``charset`` is utf8mb4 and the session is UTC, which is + the wall clock the corpus stores in a ``DATETIME``.""" + import pymysql # local import: PyMySQL lives in the `integration` / `mysql` extras + + return pymysql.connect( + host=info.host, port=info.port, user=info.user, password=info.password, + database=info.database, charset="utf8mb4", init_command="SET time_zone = '+00:00'", + **kwargs, + ) + + +def reset_database(info: MySqlInfo) -> None: + """Drop every view and table, leaving the database empty.""" + with closing(connect(info)) as conn, closing(conn.cursor()) as cur: + cur.execute( + "SELECT TABLE_NAME, TABLE_TYPE FROM information_schema.TABLES WHERE TABLE_SCHEMA = %s", + (info.database,), + ) + objects = list(cur.fetchall()) + cur.execute("SET FOREIGN_KEY_CHECKS = 0") + for name, kind in objects: + keyword = "VIEW" if kind == "VIEW" else "TABLE" + cur.execute(f"DROP {keyword} IF EXISTS `{name}`") + cur.execute("SET FOREIGN_KEY_CHECKS = 1") + conn.commit() + + +class MySqlContainer: + IMAGE = "mysql:8.4" + DATABASE = "mo_test" + USER = "root" + PASSWORD = "test" + START_ATTEMPTS = 3 + + def __init__(self) -> None: + shared = os.environ.get(SHARED_MYSQL_URL_ENV) + if shared: + parts = urlsplit(shared) + self._name: str | None = None + self._info = MySqlInfo( + host=parts.hostname or "localhost", port=parts.port or 3306, + database=parts.path.lstrip("/") or self.DATABASE, + user=unquote(parts.username or self.USER), password=unquote(parts.password or ""), + ) + return + self._name = f"metaobjects-mysql-py-{uuid.uuid4().hex[:8]}" + last_failure: Exception | None = None + for _attempt in range(self.START_ATTEMPTS): + port = _pick_free_port() + self._info = MySqlInfo("localhost", port, self.DATABASE, self.USER, self.PASSWORD) + try: + _docker( + "run", "-d", "--name", self._name, + "-e", f"MYSQL_ROOT_PASSWORD={self.PASSWORD}", + "-e", f"MYSQL_DATABASE={self.DATABASE}", + "-p", f"{port}:3306", self.IMAGE, + ) + except Exception as e: # noqa: BLE001 -- docker refused the bind; retry on a new port + last_failure = e + self._force_remove() + continue + try: + self._wait_ready() + return + except Exception as e: # noqa: BLE001 -- died on startup; retry on a new port + last_failure = e + self._force_remove() + raise RuntimeError( + f"mysql container '{self._name}' failed to start in {self.START_ATTEMPTS} attempts; " + f"last err: {last_failure}" + ) + + def info(self) -> MySqlInfo: + return self._info + + def stop(self) -> None: + if self._name is not None: + self._force_remove() + + def __enter__(self) -> "MySqlContainer": + return self + + def __exit__(self, *exc: object) -> None: + self.stop() + + def _force_remove(self) -> None: + try: + _docker("rm", "-f", str(self._name)) + except Exception: # noqa: BLE001 -- already gone + pass + + def _state(self) -> str: + r = subprocess.run( + ["docker", "inspect", "-f", "{{.State.Status}}", str(self._name)], + capture_output=True, text=True, + ) + return r.stdout.strip() if r.returncode == 0 else "gone" + + def _logs(self, *args: str) -> str: + r = subprocess.run(["docker", "logs", *args, str(self._name)], capture_output=True, text=True) + return (r.stdout + r.stderr).strip() if r.returncode == 0 else "(docker logs unavailable)" + + # MySQL's entrypoint runs a TEMPORARY server to initialise the data directory (listening + # on "port: 0"), stops it, then starts the real one. Ready needs the real server's log + # line AND a client connection from the host. + def _wait_ready(self) -> None: + timeout_s = int(os.environ.get("MO_MYSQL_READY_TIMEOUT_S", "180")) + deadline = time.monotonic() + timeout_s + last_err: Exception | None = None + while time.monotonic() < deadline: + state = self._state() + if state != "running": + raise RuntimeError( + f"mysql container '{self._name}' is '{state}', not running -- it died during " + f"startup rather than being slow. docker logs:\n{self._logs('--tail', '40')}" + ) + logs = self._logs() + if "ready for connections" in logs and "port: 3306" in logs: + try: + connect(self._info).close() + return + except Exception as e: # noqa: BLE001 -- the driver may throw anything mid-boot + last_err = e + time.sleep(0.5) + raise RuntimeError( + f"mysql container '{self._name}' did not become ready within {timeout_s}s " + f"(state={self._state()}); last err: {last_err}. docker logs:\n{self._logs('--tail', '40')}" + ) + + +def _docker(*args: str) -> str: + r = subprocess.run(["docker", *args], capture_output=True, text=True) + if r.returncode != 0: + raise subprocess.CalledProcessError(r.returncode, ["docker", *args], output=r.stdout, stderr=r.stderr) + return r.stdout.strip() + + +def _pick_free_port() -> int: + with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s: + s.bind(("127.0.0.1", 0)) + return int(s.getsockname()[1]) diff --git a/server/python/tests/integration/normalization.py b/server/python/tests/integration/normalization.py index 4c70d586e..067b0edc0 100644 --- a/server/python/tests/integration/normalization.py +++ b/server/python/tests/integration/normalization.py @@ -36,19 +36,24 @@ def normalize_row( - row: Mapping[str, Any], column_oids: Mapping[str, int] | None = None + row: Mapping[str, Any], column_oids: Mapping[str, int] | None = None, *, actual: bool = False ) -> dict[str, Any]: """Canonicalize a native-typed row to its wire form. *column_oids* (field-name → Postgres type OID) supplies the int4-vs-int8 discriminator the value alone can't carry; when absent (top-level scalar / expected-block normalization) the value-only path applies. + + *actual* marks the runtime's own output. A decimal column must come back as a + ``Decimal``, so on the actual side a decimal-looking STRING is not canonicalized (it + stays a string and fails the comparison); only the expected literals, which the + scenario files write as strings, are canonicalized. """ oids = column_oids or {} - return {k: normalize_value(v, oids.get(k)) for k, v in row.items()} + return {k: normalize_value(v, oids.get(k), actual=actual) for k, v in row.items()} -def normalize_value(v: Any, column_oid: int | None = None) -> Any: +def normalize_value(v: Any, column_oid: int | None = None, *, actual: bool = False) -> Any: if v is None: return None if isinstance(v, bool): @@ -89,31 +94,31 @@ def normalize_value(v: Any, column_oid: int | None = None) -> Any: if isinstance(v, datetime.time): return v.strftime("%H:%M:%S") + _ms_suffix(v.microsecond) if isinstance(v, str): - if _DECIMAL_RE.match(v) and "." in v: + if not actual and _DECIMAL_RE.match(v) and "." in v: return _canonical_decimal(decimal.Decimal(v)) if _UUID_RE.match(v): return v.lower() return v if isinstance(v, Mapping): - return {k: normalize_value(val) for k, val in sorted(v.items())} + return {k: normalize_value(val, actual=actual) for k, val in sorted(v.items())} if isinstance(v, (list, tuple)): - return [normalize_value(x) for x in v] + return [normalize_value(x, actual=actual) for x in v] return str(v) def canonical_rows_json( - rows: list[Mapping[str, Any]], column_oids: Mapping[str, int] | None = None + rows: list[Mapping[str, Any]], column_oids: Mapping[str, int] | None = None, *, actual: bool = False ) -> str: """Stable JSON: sort object keys, no whitespace, normalize each row first.""" return json.dumps( - [normalize_row(r, column_oids) for r in rows], + [normalize_row(r, column_oids, actual=actual) for r in rows], sort_keys=True, separators=(",", ":"), ) def canonical_row_set_json( - rows: list[Mapping[str, Any]], column_oids: Mapping[str, int] | None = None + rows: list[Mapping[str, Any]], column_oids: Mapping[str, int] | None = None, *, actual: bool = False ) -> str: """Order-independent canonical JSON for a row *set* (M:N `relate` results). @@ -122,7 +127,7 @@ def canonical_row_set_json( navigation contract. Mirrors the TS runner's ``canonicalRowSet``. """ each = sorted( - json.dumps(normalize_row(r, column_oids), sort_keys=True, separators=(",", ":")) + json.dumps(normalize_row(r, column_oids, actual=actual), sort_keys=True, separators=(",", ":")) for r in rows ) return "[" + ",".join(each) + "]" diff --git a/server/python/tests/integration/query_runner.py b/server/python/tests/integration/query_runner.py index 55c5f3930..6d4abc641 100644 --- a/server/python/tests/integration/query_runner.py +++ b/server/python/tests/integration/query_runner.py @@ -1,7 +1,10 @@ """Run a QueryScenario end-to-end.""" from __future__ import annotations +import decimal + import json +import sqlite3 from contextlib import closing from pathlib import Path from typing import Any @@ -10,8 +13,11 @@ from metaobjects import load_directory from metaobjects.naming import attach_naming -from metaobjects.runtime import ObjectManager, PostgresDriver +from metaobjects.runtime import MySqlDriver, ObjectManager, PostgresDriver, SqliteDriver +from .engines import ScenarioEngine, split_statements +from .mysql_container import MySqlInfo +from .mysql_container import connect as mysql_connect from .normalization import canonical_row_set_json, canonical_rows_json, normalize_row from .postgres_container import PostgresContainer from .scenarios import QueryScenario, QuerySpec @@ -43,22 +49,66 @@ def run(scenario: QueryScenario, pg: PostgresContainer, canonical_dir: Path) -> )) as conn: driver = PostgresDriver(conn) om = ObjectManager(root_for_bootstrap, driver) - for spec in scenario.queries: - # FR-017 TPH: an op marked `expect-error: true` (a cross-subtype write) - # MUST be rejected by the runtime — a raise is the pass (mirrors TS/C#). - if spec.expect_error: - try: - _execute_spec(om, spec) - except Exception: - continue # rejected as required - raise AssertionError( - f"{scenario.source_path} / {spec.name}: expected the {spec.op} to " - f"FAIL (expect-error: true) but it succeeded" - ) - actual = _execute_spec(om, spec) - # The OID map from the just-run query supplies the int4-vs-int8 - # wire discriminator the native row values can't carry (ADR-0019). - _assert_result(scenario.source_path, spec, actual, om.last_column_oids) + _run_queries(scenario, om) + + +def run_sqlite(scenario: QueryScenario, db_path: Path, canonical_dir: Path) -> None: + """Run *scenario* on SQLite (stdlib ``sqlite3``): execute ``canonical/schema.sqlite.sql`` + verbatim into the empty database file *db_path*, seed it, and run the queries through the + ObjectManager with the SQLite driver. The expectations are the Postgres wire ones.""" + engine = ScenarioEngine.SQLITE + with closing(sqlite3.connect(str(db_path))) as conn: + _exec_script(conn, (canonical_dir.parent / engine.schema_artifact).read_text()) + seed = engine.seed(scenario) + if seed and seed.strip(): + _exec_script(conn, seed) + om = ObjectManager(_load(canonical_dir), SqliteDriver(conn)) + _run_queries(scenario, om) + + +def run_mysql(scenario: QueryScenario, info: MySqlInfo, canonical_dir: Path) -> None: + """Run *scenario* on MySQL: *info* names an EMPTY database; ``canonical/schema.mysql.sql`` + is executed verbatim, the seed is re-spelled for MySQL, and the queries run through the + ObjectManager with the MySQL driver. The expectations are the Postgres wire ones.""" + engine = ScenarioEngine.MYSQL + with closing(mysql_connect(info)) as conn: + _exec_script(conn, (canonical_dir.parent / engine.schema_artifact).read_text()) + seed = engine.seed(scenario) + if seed and seed.strip(): + _exec_script(conn, seed) + om = ObjectManager(_load(canonical_dir), MySqlDriver(conn)) + _run_queries(scenario, om) + + +def _exec_script(conn: Any, script: str) -> None: + """Run a multi-statement script one statement per call (neither driver batches).""" + cur = conn.cursor() + try: + for stmt in split_statements(script): + cur.execute(stmt) + conn.commit() + finally: + cur.close() + + +def _run_queries(scenario: QueryScenario, om: ObjectManager) -> None: + for spec in scenario.queries: + # FR-017 TPH: an op marked `expect-error: true` (a cross-subtype write) + # MUST be rejected by the runtime — a raise is the pass (mirrors TS/C#). + if spec.expect_error: + try: + _execute_spec(om, spec) + except Exception: + continue # rejected as required + raise AssertionError( + f"{scenario.source_path} / {spec.name}: expected the {spec.op} to " + f"FAIL (expect-error: true) but it succeeded" + ) + actual = _execute_spec(om, spec) + # The type-code map from the just-run query supplies the int4-vs-int8 + # wire discriminator the native row values can't carry (ADR-0019). + _assert_typed_decimals(scenario.source_path, spec, actual, om) + _assert_result(scenario.source_path, spec, actual, om.last_column_oids) # ---------------------------------------------------------------------------- @@ -194,6 +244,28 @@ def _execute(info: Any, sql: str) -> None: cur.close() +def _assert_typed_decimals(scenario_path: str, spec: QuerySpec, actual: Any, om: ObjectManager) -> None: + """The typed decimal gate (ADR-0019 Amendment 1): BEFORE any normalization, every field the + model declares decimal (an entity field, or a report's derived decimal measure) must be + ``None`` or a ``Decimal``. A float or a string fails, so the boundary can never launder a + runtime that read a decimal as a double or as text.""" + if spec.op in ("count", "delete") or actual is None or isinstance(actual, bool): + return + decimal_fields = [ + f.name for f in om._require_entity(spec.entity).fields() if f.sub_type == "decimal" + ] + if not decimal_fields: + return + rows = actual if isinstance(actual, list) else [actual] + for row in rows: + for name in decimal_fields: + if name in row and row[name] is not None and not isinstance(row[name], decimal.Decimal): + raise AssertionError( + f"{scenario_path} / {spec.name}: decimal field {spec.entity}.{name} came back " + f"as {type(row[name]).__name__} {row[name]!r}, not a Decimal" + ) + + def _assert_result( scenario_path: str, spec: QuerySpec, actual: Any, column_oids: dict[str, int] ) -> None: @@ -243,9 +315,9 @@ def _canonicalize_actual(actual: Any, op: str, column_oids: dict[str, int]) -> s return "null" if op != "relate" else "[]" if op in ("get", "roundtrip", "update", "create"): return json.dumps( - normalize_row(actual, column_oids), sort_keys=True, separators=(",", ":") + normalize_row(actual, column_oids, actual=True), sort_keys=True, separators=(",", ":") ) if op == "relate": rows = actual if isinstance(actual, list) else [actual] - return canonical_row_set_json(rows, column_oids) - return canonical_rows_json(actual, column_oids) + return canonical_row_set_json(rows, column_oids, actual=True) + return canonical_rows_json(actual, column_oids, actual=True) diff --git a/server/python/tests/integration/report_lane.py b/server/python/tests/integration/report_lane.py new file mode 100644 index 000000000..c02a7cab5 --- /dev/null +++ b/server/python/tests/integration/report_lane.py @@ -0,0 +1,131 @@ +"""One database engine of the api-contract ``report/`` lane: schema, seed, ObjectManager. + +The corpus ships a TypeScript-produced schema per engine (``report/schema..sql``: the +four base tables and the nine report views). A lane executes it verbatim into an EMPTY +database, seeds the base tables from ``seed.json`` (every key but ``reports``; the keys are +listed parents first), and reads the views through the ObjectManager with the engine's driver. +""" +from __future__ import annotations + +import json +import sqlite3 +from collections.abc import Iterator +from contextlib import closing, contextmanager +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import pg8000.dbapi as pg8000 + +from metaobjects.runtime import MySqlDriver, ObjectManager, PostgresDriver, SqliteDriver +from metaobjects.runtime.dialect import Dialect, MYSQL, POSTGRES, SQLITE + +from .engines import split_statements +from .generated_report_app import load_runtime_root +from .mysql_container import MySqlInfo +from .mysql_container import connect as mysql_connect +from .postgres_container import PostgresContainer + +ENGINES = ("postgres", "sqlite", "mysql") + +_DIALECTS: dict[str, Dialect] = {"postgres": POSTGRES, "sqlite": SQLITE, "mysql": MYSQL} + + +@dataclass +class ReportLane: + engine: str + conn: Any + om: ObjectManager + #: base tables in seed order (parents first): name -> rows + base_tables: dict[str, list[dict[str, Any]]] + #: what the views are expected to return for the full seed + expected_reports: dict[str, list[dict[str, Any]]] + + def reseed(self, *, empty: bool) -> None: + """Delete every base row (children first), then insert the seed unless *empty*.""" + d = _DIALECTS[self.engine] + cur = self.conn.cursor() + try: + for table in reversed(list(self.base_tables)): + cur.execute(f"DELETE FROM {d.quote(table)}") + if not empty: + for table, rows in self.base_tables.items(): + for row in rows: + cols = ", ".join(d.quote(c) for c in row) + marks = ", ".join(d.placeholder for _ in row) + cur.execute( + f"INSERT INTO {d.quote(table)} ({cols}) VALUES ({marks})", + tuple(row.values()), + ) + if self.engine == "postgres": + # An explicit id does not advance an identity column: continue after the seed + # (or restart at 1 when the table was left empty), as the other engines do. + for table, seed_rows in self.base_tables.items(): + if _has_id(seed_rows): + cur.execute( + f"SELECT setval(pg_get_serial_sequence('{d.quote(table)}', 'id'), " + f"COALESCE((SELECT MAX({d.quote('id')}) FROM {d.quote(table)}), 0) + 1, false)" + ) + self.conn.commit() + finally: + cur.close() + + +def _has_id(rows: list[dict[str, Any]]) -> bool: + return bool(rows) and "id" in rows[0] + + +def load_seed(corpus_root: Path) -> tuple[dict[str, list[dict[str, Any]]], dict[str, list[dict[str, Any]]]]: + parsed = json.loads((corpus_root / "seed.json").read_text()) + reports = parsed.get("reports") + if not isinstance(reports, dict): + raise RuntimeError("report seed.json: missing 'reports' object") + base = {name: rows for name, rows in parsed.items() if name != "reports"} + return base, reports + + +def _exec_script(conn: Any, script: str) -> None: + cur = conn.cursor() + try: + for stmt in split_statements(script): + cur.execute(stmt) + conn.commit() + finally: + cur.close() + + +@contextmanager +def open_lane( + engine: str, + corpus_root: Path, + *, + sqlite_path: Path | None = None, + mysql: MySqlInfo | None = None, +) -> Iterator[ReportLane]: + """Provision *engine* (a fresh Postgres, the SQLite file *sqlite_path*, or the EMPTY MySQL + database *mysql*), execute the corpus schema into it, and yield a lane over it.""" + schema = (corpus_root / f"schema.{engine}.sql").read_text() + base, reports = load_seed(corpus_root) + root = load_runtime_root(corpus_root) + if engine == "postgres": + with PostgresContainer() as pg: + info = pg.info() + with closing(pg8000.connect( + host=info.host, port=info.port, user=info.user, + password=info.password, database=info.database, + )) as conn: + _exec_script(conn, schema) + yield ReportLane(engine, conn, ObjectManager(root, PostgresDriver(conn)), base, reports) + elif engine == "sqlite": + assert sqlite_path is not None + # check_same_thread=False: the generated sync routes run on the framework's threadpool. + with closing(sqlite3.connect(str(sqlite_path), check_same_thread=False)) as conn: + _exec_script(conn, schema) + yield ReportLane(engine, conn, ObjectManager(root, SqliteDriver(conn)), base, reports) + elif engine == "mysql": + assert mysql is not None + with closing(mysql_connect(mysql)) as conn: + _exec_script(conn, schema) + yield ReportLane(engine, conn, ObjectManager(root, MySqlDriver(conn)), base, reports) + else: + raise ValueError(f"unknown engine {engine!r}") diff --git a/server/python/tests/integration/scenarios.py b/server/python/tests/integration/scenarios.py index 742cf0f56..cc7855f6d 100644 --- a/server/python/tests/integration/scenarios.py +++ b/server/python/tests/integration/scenarios.py @@ -59,6 +59,31 @@ class QueryScenario: source_path: str seed_data: str | None queries: list[QuerySpec] + #: ``seed-data-engine:`` -- a seed spelled for one engine (keyed ``sqlite`` / ``mysql``), + #: which wins over ``seed-data`` for that engine. Only a seed that cannot be shared + #: (an instant relative to the database clock) carries one. + seed_data_engine: dict[str, str] | None = None + #: ``engines:`` -- EXACT when present: only the listed engines run the file. Absent keeps + #: the historical rule (Postgres runs every scenario, the others the ``report-*`` ones). + engines: tuple[str, ...] | None = None + + def runs_on(self, engine: str) -> bool: + if self.engines is not None: + return engine in self.engines + return engine == "postgres" or self.name.startswith("report-") + +KNOWN_ENGINES = frozenset({"postgres", "mysql", "sqlite", "d1"}) + + +def _parse_engines(raw: object, path: Path) -> tuple[str, ...] | None: + if raw is None: + return None + if not isinstance(raw, list) or not raw or not all(isinstance(e, str) for e in raw): + raise ValueError(f"{path.name}: engines must be a non-empty list of engine names, got {raw!r}") + unknown = [e for e in raw if e not in KNOWN_ENGINES] + if unknown: + raise ValueError(f"{path.name}: unknown engine(s) {unknown}; known: {sorted(KNOWN_ENGINES)}") + return tuple(raw) # ---------------------------------------------------------------------------- @@ -107,4 +132,6 @@ def _parse_query(path: Path) -> QueryScenario: source_path=str(path), seed_data=raw.get("seed-data"), queries=queries, + seed_data_engine=raw.get("seed-data-engine"), + engines=_parse_engines(raw.get("engines"), path), ) diff --git a/server/python/tests/integration/test_api_contract.py b/server/python/tests/integration/test_api_contract.py index 7bca7a7c2..7dea0c593 100644 --- a/server/python/tests/integration/test_api_contract.py +++ b/server/python/tests/integration/test_api_contract.py @@ -131,6 +131,6 @@ def _parse_response_body(text: str | None) -> Any: if text is None or text == "": return None try: - return json.loads(text) + return api_contract_assertions.parse_body(text) except json.JSONDecodeError: return text diff --git a/server/python/tests/integration/test_api_contract_decimal.py b/server/python/tests/integration/test_api_contract_decimal.py new file mode 100644 index 000000000..163cc2b4f --- /dev/null +++ b/server/python/tests/integration/test_api_contract_decimal.py @@ -0,0 +1,139 @@ +"""ADR-0019 Amendment 1 -- cross-port API contract for ``field.decimal`` (W1 to W4). + +Drives ``fixtures/api-contract-conformance/decimal/`` against the GENERATED routers (the +writable ``Entry`` router and the served ``EntryTotals`` report router) on a REAL database: +the corpus' TypeScript-produced ``schema..sql`` runs verbatim on Postgres, SQLite and +MySQL 8.4, ``entries`` is seeded from ``seed.json`` (amounts are strings, never numbers), and +each generated consumer seam is an ``ObjectManager`` repository. A scenario's top-level +``engines:`` is EXACT (an unknown name is an error); without it a scenario runs on every +engine the lane has. + +Beyond the scenario's own assertions every response is held to the typed decimal gate: a field +the model declares decimal must be ``null`` or a canonical string, never a JSON number. +""" +from __future__ import annotations + +import json +import re +from collections.abc import Iterator +from decimal import Decimal +from pathlib import Path +from typing import Any + +import pytest +import yaml +from fastapi.testclient import TestClient + +from . import api_contract_assertions +from .generated_report_app import LaneModel, build_generated_report_app +from .mysql_container import MySqlInfo +from .report_lane import ENGINES, ReportLane, open_lane + +_CORPUS = next( + p / "fixtures" / "api-contract-conformance" / "decimal" + for p in [Path(__file__).resolve(), *Path(__file__).resolve().parents] + if (p / "fixtures" / "api-contract-conformance" / "decimal").is_dir() +) +_MODEL = LaneModel(reports=(("EntryTotals", "entry_totals"),), entities=(("Entry", "entry"),)) +_KNOWN_ENGINES = {"postgres", "mysql", "sqlite", "d1"} +#: The fields the model declares decimal: Entry.amount and the EntryTotals measures that +#: sum, average, take a maximum or divide a decimal. +_DECIMAL_FIELDS = ("amount", "total", "totalOrZero", "mean", "top", "flaggedShare", "perFlagged") +_CANONICAL = re.compile(r"^-?(0|[1-9][0-9]*)(\.[0-9]*[1-9])?$") + + +def _load() -> list[tuple[str, dict[str, Any]]]: + out = [] + for path in sorted((_CORPUS / "scenarios").glob("*.yaml")): + raw = yaml.safe_load(path.read_text()) + engines = raw.get("engines") + if engines is not None: + if not isinstance(engines, list) or not engines or set(engines) - _KNOWN_ENGINES: + raise RuntimeError(f"{path.name}: unknown or empty engines: {engines!r}") + out.append((raw["name"], raw)) + return out + + +_SCENARIOS = _load() + + +class _Lane: + def __init__(self, lane: ReportLane) -> None: + self.lane = lane + app, _ = build_generated_report_app(_CORPUS, lane.om, _MODEL) + self.client = TestClient(app) + + +@pytest.fixture(scope="module", params=ENGINES) +def lane(request: pytest.FixtureRequest, tmp_path_factory: pytest.TempPathFactory) -> Iterator[_Lane]: + engine: str = request.param + mysql: MySqlInfo | None = None + if engine == "mysql": + from .mysql_container import reset_database + + mysql = request.getfixturevalue("mysql_server") + reset_database(mysql) + with open_lane( + engine, _CORPUS, sqlite_path=tmp_path_factory.mktemp("decimal") / "decimal.db", mysql=mysql + ) as opened: + yield _Lane(opened) + + +def _check_typed(scenario: str, rid: str, body: Any) -> None: + rows = body if isinstance(body, list) else [body] if isinstance(body, dict) else [] + for row in rows: + if not isinstance(row, dict): + continue + for name in _DECIMAL_FIELDS: + if name in row: + v = row[name] + assert v is None or (isinstance(v, str) and _CANONICAL.match(v)), ( + f"{scenario} / {rid}: decimal field {name!r} must be null or a canonical string, got {v!r}" + ) + + +@pytest.mark.parametrize("name,scenario", _SCENARIOS, ids=[n for n, _ in _SCENARIOS]) +def test_decimal_scenario(lane: _Lane, name: str, scenario: dict[str, Any]) -> None: + engines = scenario.get("engines") + if engines is not None and lane.lane.engine not in engines: + pytest.skip(f"{name} does not run on {lane.lane.engine}") + lane.lane.reseed(empty=bool((scenario.get("setup") or {}).get("truncate"))) + for req in scenario.get("requests", []): + kwargs: dict[str, Any] = {} + if req.get("body") is not None: + kwargs["content"] = json.dumps(req["body"]) + kwargs["headers"] = {"content-type": "application/json"} + response = lane.client.request(str(req["method"]).upper(), str(req["path"]), **kwargs) + text = response.text + parsed = api_contract_assertions.parse_body(text) if text else None + rid = str(req.get("id", "?")) + expect = req["expect"] + _check_typed(name, rid, parsed) + api_contract_assertions.assert_response( + scenario_name=name, + request_id=rid, + expect_status=int(expect["status"]), + expect_body=expect.get("body"), + status=response.status_code, + body=parsed, + ) + + +@pytest.mark.parametrize("engine", ["postgres", "mysql"]) +def test_the_views_return_the_expected_reports(lane: _Lane, engine: str) -> None: + """On the exact engines the real view returns ``seed.json`` ``reports``, compared by value.""" + if lane.lane.engine != engine: + pytest.skip("other engine") + lane.lane.reseed(empty=False) + actual = {r["bucket"]: r for r in lane.lane.om.find_many("EntryTotals")} + for want in lane.lane.expected_reports["EntryTotals"]: + got = actual[want["bucket"]] + for k, v in want.items(): + if k == "bucket": + assert got[k] == v + elif isinstance(v, str): + assert Decimal(str(got[k])) == Decimal(v), (want["bucket"], k, got[k], v) + elif v is None: + assert got[k] is None, (want["bucket"], k) + else: + assert got[k] == v, (want["bucket"], k) diff --git a/server/python/tests/integration/test_api_contract_generated.py b/server/python/tests/integration/test_api_contract_generated.py index be7aaa8b3..c5f60e0b4 100644 --- a/server/python/tests/integration/test_api_contract_generated.py +++ b/server/python/tests/integration/test_api_contract_generated.py @@ -110,6 +110,6 @@ def _parse_response_body(text: str | None) -> Any: if text is None or text == "": return None try: - return json.loads(text) + return api_contract_assertions.parse_body(text) except json.JSONDecodeError: return text diff --git a/server/python/tests/integration/test_api_contract_jsonb.py b/server/python/tests/integration/test_api_contract_jsonb.py index 7b0ad55b3..db18cb114 100644 --- a/server/python/tests/integration/test_api_contract_jsonb.py +++ b/server/python/tests/integration/test_api_contract_jsonb.py @@ -148,6 +148,6 @@ def _parse_response_body(text: str | None) -> Any: if text is None or text == "": return None try: - return json.loads(text) + return api_contract_assertions.parse_body(text) except json.JSONDecodeError: return text diff --git a/server/python/tests/integration/test_api_contract_m2m.py b/server/python/tests/integration/test_api_contract_m2m.py index 2945569a7..602743c3c 100644 --- a/server/python/tests/integration/test_api_contract_m2m.py +++ b/server/python/tests/integration/test_api_contract_m2m.py @@ -88,6 +88,6 @@ def _parse_response_body(text: str | None) -> Any: if text is None or text == "": return None try: - return json.loads(text) + return api_contract_assertions.parse_body(text) except json.JSONDecodeError: return text diff --git a/server/python/tests/integration/test_api_contract_m2m_generated.py b/server/python/tests/integration/test_api_contract_m2m_generated.py index d6efa772e..4fe202767 100644 --- a/server/python/tests/integration/test_api_contract_m2m_generated.py +++ b/server/python/tests/integration/test_api_contract_m2m_generated.py @@ -93,6 +93,6 @@ def _parse_response_body(text: str | None) -> Any: if text is None or text == "": return None try: - return json.loads(text) + return api_contract_assertions.parse_body(text) except json.JSONDecodeError: return text diff --git a/server/python/tests/integration/test_api_contract_projection.py b/server/python/tests/integration/test_api_contract_projection.py index 6385f9e72..80edeb74e 100644 --- a/server/python/tests/integration/test_api_contract_projection.py +++ b/server/python/tests/integration/test_api_contract_projection.py @@ -111,6 +111,6 @@ def _parse_response_body(text: str | None) -> Any: if text is None or text == "": return None try: - return json.loads(text) + return api_contract_assertions.parse_body(text) except json.JSONDecodeError: return text diff --git a/server/python/tests/integration/test_api_contract_report.py b/server/python/tests/integration/test_api_contract_report.py index ffec72969..f4959a276 100644 --- a/server/python/tests/integration/test_api_contract_report.py +++ b/server/python/tests/integration/test_api_contract_report.py @@ -1,22 +1,26 @@ """FR-044 — cross-port API contract conformance for a VIEW-BACKED ``object.report``. Drives ``fixtures/api-contract-conformance/report/`` against the GENERATED routers for the -served reports (the deployed artifact), with a read-only in-memory repo behind each -generated consumer seam, seeded from ``seed.json``'s ``reports`` half. +served reports (the deployed artifact), on a REAL database: the corpus' TypeScript-produced +``schema..sql`` (four tables and nine views) is executed verbatim on Postgres, SQLite +and MySQL 8.4, the base tables are seeded from ``seed.json``, and the repository behind each +generated consumer seam is the adopter-owned ``ObjectManager`` implementation reading the +real view. Every scenario runs on every engine. -Generated lane only, by design — see the corpus README. The thing under test is whether -Python's GENERATOR emits a keyless read-only surface for a report: row model, filter -allowlist, router. A hand-rolled reference server would answer every scenario by -construction and prove nothing. +Generated lane only, by design -- see the corpus README. The thing under test is whether +Python's GENERATOR emits a keyless read-only surface for a report (row model, filter +allowlist, router) and whether it answers correctly over the view's rows. A hand-rolled +reference server would answer every scenario by construction and prove nothing. Run on-demand: cd server/python - uv run --extra integration pytest tests/integration/test_api_contract_report.py -v + uv run --extra dev --extra integration pytest tests/integration/test_api_contract_report.py -v """ from __future__ import annotations import json +from collections.abc import Iterator from pathlib import Path from typing import Any @@ -29,8 +33,11 @@ SERVED_REPORTS, UNSERVED_REPORT, build_generated_report_app, - seed_rows, + generate_reports, ) +from .mysql_container import MySqlInfo +from .normalization import canonical_row_set_json +from .report_lane import ENGINES, ReportLane, open_lane def _find_corpus_root(start: Path | None = None) -> Path: @@ -59,39 +66,68 @@ def _load_scenarios() -> list[tuple[str, dict[str, Any]]]: return out -def _load_seed_reports() -> dict[str, list[dict[str, Any]]]: - """The seam lane's half of the seed: what the views return for the base rows.""" - parsed = json.loads((_CORPUS / "seed.json").read_text()) - reports = parsed.get("reports") - if not isinstance(reports, dict): - raise RuntimeError("report seed.json: missing 'reports' object") - return {name: seed_rows(reports[name]) for name in SERVED_REPORTS} +_SCENARIOS = _load_scenarios() -_SEED = _load_seed_reports() -_SCENARIOS = _load_scenarios() +class _Lane: + """A provisioned engine, its seeded tables, and a client over the generated routers.""" + + def __init__(self, lane: ReportLane) -> None: + self.lane = lane + app, self.emitted = build_generated_report_app(_CORPUS, lane.om) + self.client = TestClient(app) + -_APP, _REPOS, _EMITTED = build_generated_report_app(_CORPUS) -_CLIENT = TestClient(_APP) +@pytest.fixture(scope="module", params=ENGINES) +def lane(request: pytest.FixtureRequest, tmp_path_factory: pytest.TempPathFactory) -> Iterator[_Lane]: + """One database per engine for the module; a Postgres container, a SQLite file, or the + session's MySQL server reset to an empty database. Every test of the module runs once + per engine.""" + engine: str = request.param + mysql: MySqlInfo | None = None + if engine == "mysql": + from .mysql_container import reset_database + + mysql = request.getfixturevalue("mysql_server") + assert mysql is not None + reset_database(mysql) + with open_lane( + engine, _CORPUS, sqlite_path=tmp_path_factory.mktemp("report") / "report.db", mysql=mysql + ) as opened: + yield _Lane(opened) def test_exactly_the_served_reports_are_generated() -> None: """The sourceless report generates nothing; each served one generates its row model, allowlist and router (and the names module is not a route concern, so it is not asked for here: only the three generators the lane runs).""" - assert len(SERVED_REPORTS) == 9 + emitted = generate_reports(_CORPUS).emitted + assert len(SERVED_REPORTS) == 10 for report, stem in SERVED_REPORTS.items(): - assert f"{report}.py" in _EMITTED - assert f"{stem}_filter_allowlist.py" in _EMITTED - assert f"{stem}_router.py" in _EMITTED - assert not any(UNSERVED_REPORT in f or "invoice_days" in f for f in _EMITTED), _EMITTED + assert f"{report}.py" in emitted + assert f"{stem}_filter_allowlist.py" in emitted + assert f"{stem}_router.py" in emitted + assert not any(UNSERVED_REPORT in f or "invoice_days" in f for f in emitted), emitted # The base entities were not asked for, so no writable router sits beside the reports. for base in ("category", "invoice", "product", "sale"): - assert f"{base}_router.py" not in _EMITTED + assert f"{base}_router.py" not in emitted + + +def test_the_unserved_report_mounts_no_route(lane: _Lane) -> None: + assert lane.client.get("/api/invoice_days").status_code == 404 -def test_the_unserved_report_mounts_no_route() -> None: - assert _CLIENT.get("/api/invoice_days").status_code == 404 +def test_the_views_return_what_the_corpus_expects(lane: _Lane) -> None: + """The seed's ``reports`` half is what the views return for the base rows: the lane reads + the REAL views and holds them to it, so the scenarios below stand on the seed and not on + a view that drifted from it. Order is not part of the comparison; a decimal is compared + by value (its spelling is each engine's own).""" + lane.lane.reseed(empty=False) + for report in SERVED_REPORTS: + actual = lane.lane.om.find_many(report) + assert canonical_row_set_json(actual) == canonical_row_set_json( + lane.lane.expected_reports.get(report, []) + ), f"{report} on {lane.lane.engine}" @pytest.mark.parametrize( @@ -99,17 +135,14 @@ def test_the_unserved_report_mounts_no_route() -> None: _SCENARIOS, ids=[name for name, _ in _SCENARIOS], ) -def test_report_scenario(scenario_name: str, scenario: dict[str, Any]) -> None: - """Run one report api-contract scenario against the GENERATED routers.""" - for report, repo in _REPOS.items(): - repo.reset() - if not (scenario.get("setup") or {}).get("truncate"): - repo.seed(_SEED[report]) +def test_report_scenario(lane: _Lane, scenario_name: str, scenario: dict[str, Any]) -> None: + """Run one report api-contract scenario against the GENERATED routers, over the views.""" + lane.lane.reseed(empty=bool((scenario.get("setup") or {}).get("truncate"))) for req in scenario.get("requests", []): - _run_request(scenario_name, req) + _run_request(lane.client, scenario_name, req) -def _run_request(scenario_name: str, req: dict[str, Any]) -> None: +def _run_request(client: TestClient, scenario_name: str, req: dict[str, Any]) -> None: method = str(req["method"]).upper() path = str(req["path"]) body = req.get("body") @@ -118,7 +151,7 @@ def _run_request(scenario_name: str, req: dict[str, Any]) -> None: kwargs: dict[str, Any] = {} if body is not None: kwargs["json"] = body - response = _CLIENT.request(method, path, **kwargs) + response = client.request(method, path, **kwargs) api_contract_assertions.assert_response( scenario_name=scenario_name, request_id=str(req.get("id", "?")), @@ -133,6 +166,6 @@ def _parse_response_body(text: str | None) -> Any: if text is None or text == "": return None try: - return json.loads(text) + return api_contract_assertions.parse_body(text) except json.JSONDecodeError: return text diff --git a/server/python/tests/integration/test_api_contract_tph_generated.py b/server/python/tests/integration/test_api_contract_tph_generated.py index 7e4c25331..7305eb6b7 100644 --- a/server/python/tests/integration/test_api_contract_tph_generated.py +++ b/server/python/tests/integration/test_api_contract_tph_generated.py @@ -95,6 +95,6 @@ def _parse_response_body(text: str | None) -> Any: if text is None or text == "": return None try: - return json.loads(text) + return api_contract_assertions.parse_body(text) except json.JSONDecodeError: return text diff --git a/server/python/tests/integration/test_generated_decimal_wire.py b/server/python/tests/integration/test_generated_decimal_wire.py new file mode 100644 index 000000000..a90f3abed --- /dev/null +++ b/server/python/tests/integration/test_generated_decimal_wire.py @@ -0,0 +1,192 @@ +"""The decimal wire rule (ADR-0019 Amendment 1) through a GENERATED router over HTTP. + +The router is the real generator's output, mounted unmodified over an ObjectManager on an +in-memory SQLite database, so the three seams are exercised together: the request body (W2), +the filter operand (W2) and the SQLite read (W4) in, and the canonical string (W1) out. +""" +from __future__ import annotations + +import importlib.util +import json +import sqlite3 +import sys +import tempfile +import threading +import uuid +from decimal import Decimal +from pathlib import Path +from typing import Any + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from metaobjects import load_string +from metaobjects.codegen.generators.entity_model import render_entity_model +from metaobjects.codegen.generators.filter_allowlist_generator import render_filter_allowlist +from metaobjects.codegen.generators.router_generator import render_router +from metaobjects.codegen.runtime.filter_parser import FilterPredicate +from metaobjects.runtime import ObjectManager, SqliteDriver + +_META = { + "metadata.root": { + "package": "acme::shop", + "children": [ + {"object.entity": {"name": "Entry", "children": [ + {"source.rdb": {"@table": "entries"}}, + {"field.long": {"name": "id"}}, + {"field.decimal": {"name": "amount", "@precision": 30, "@scale": 20, "@required": True, "@filterable": True}}, + {"field.decimal": {"name": "note", "@precision": 12, "@scale": 2, "@filterable": True}}, + {"identity.primary": {"name": "pk", "@fields": ["id"], "@generation": "increment"}}, + ]}} + ], + } +} + + +class _Repo: + def __init__(self, om: ObjectManager) -> None: + self.om = om + self.lock = threading.Lock() + + @staticmethod + def _filter(predicates: list[FilterPredicate]) -> Any: + return {"and": [{p.field: {p.op: p.value}} for p in predicates]} if predicates else None + + def list(self, limit: int, offset: int, sort: Any, filters: list[FilterPredicate]) -> list[Any]: + with self.lock: + return self.om.find_many("Entry", self._filter(filters), sort=[("id", "asc")], limit=limit, offset=offset) + + def count(self, filters: list[FilterPredicate]) -> int: + with self.lock: + return self.om.count("Entry", self._filter(filters)) + + def find_by_id(self, id: int) -> Any: + with self.lock: + return self.om.find_by_id("Entry", id) + + def create(self, dto: Any) -> Any: + with self.lock: + return self.om.create("Entry", dto) + + def update(self, id: int, dto: Any) -> Any: + with self.lock: + return self.om.update("Entry", id, dto) + + def delete(self, id: int) -> bool: + with self.lock: + return self.om.delete("Entry", id) + + +@pytest.fixture() +def client() -> TestClient: + root = load_string(json.dumps(_META)).root + entry = next(c for c in root.children() if c.name == "Entry") + pkg = f"gendec_{uuid.uuid4().hex[:8]}" + tmp = Path(tempfile.mkdtemp(prefix="gendec-")) + pkg_dir = tmp / pkg + pkg_dir.mkdir() + (pkg_dir / "__init__.py").write_text("") + (pkg_dir / "entry_filter_allowlist.py").write_text(render_filter_allowlist(entry) or "") + (pkg_dir / "Entry.py").write_text(render_entity_model(entry)) + (pkg_dir / "entry_router.py").write_text(render_router(entry) or "") + sys.path.insert(0, str(tmp)) + try: + mod = importlib.import_module(f"{pkg}.entry_router") + finally: + sys.path.remove(str(tmp)) + db = sqlite3.connect(":memory:", check_same_thread=False) + db.execute("CREATE TABLE entries (id INTEGER PRIMARY KEY AUTOINCREMENT, amount NUMERIC, note NUMERIC)") + repo = _Repo(ObjectManager(root, SqliteDriver(db))) # type: ignore[arg-type] + app = FastAPI() + app.include_router(mod.router) + app.dependency_overrides[mod.get_repository] = lambda: repo + return TestClient(app) + + +def _post(client: TestClient, body: str) -> Any: + return client.post("/api/entries", content=body, headers={"content-type": "application/json"}) + + +def test_a_decimal_is_written_in_its_canonical_form(client: TestClient) -> None: + created = _post(client, '{"amount": "12.50", "note": null}') + assert created.status_code == 201, created.text + assert created.text == '{"id":1,"amount":"12.5","note":null}' + assert client.get("/api/entries/1").json()["amount"] == "12.5" + assert client.get("/api/entries").json() == [{"id": 1, "amount": "12.5", "note": None}] + assert client.get("/api/entries?withCount=1").json()["rows"][0]["amount"] == "12.5" + + +def test_a_json_number_is_accepted_without_a_float_detour(client: TestClient) -> None: + created = _post(client, '{"amount": 0.1}') + assert created.status_code == 201, created.text + assert created.json()["amount"] == "0.1" + + +def test_put_and_patch_take_decimals_too(client: TestClient) -> None: + _post(client, '{"amount": "1"}') + patched = client.patch("/api/entries/1", content='{"amount": "2.250"}', headers={"content-type": "application/json"}) + assert patched.status_code == 200 and patched.json()["amount"] == "2.25" + bad = client.put("/api/entries/1", content='{"amount": "abc"}', headers={"content-type": "application/json"}) + assert bad.status_code == 400 and bad.json() == {"error": "validation"} + + +@pytest.mark.parametrize("bad", ['""', '"abc"', '"1,5"', '" 1.5"', '"NaN"', '"Infinity"', '"0x10"', "true", "[1]", '"1e999999999"']) +def test_garbage_is_a_400_validation(client: TestClient, bad: str) -> None: + resp = _post(client, '{"amount": ' + bad + "}") + assert resp.status_code == 400, resp.text + assert resp.json() == {"error": "validation"} + + +@pytest.mark.parametrize("body", ["{", "", "not json", "[1]", '"text"', "null"]) +def test_a_malformed_or_non_object_json_body_is_a_400_validation(client: TestClient, body: str) -> None: + """Cross-port (Java, Kotlin): the body that is not a JSON object answers 400 validation, + not the framework's 422.""" + for send in ( + lambda: _post(client, body), + lambda: client.put("/api/entries/1", content=body, headers={"content-type": "application/json"}), + lambda: client.patch("/api/entries/1", content=body, headers={"content-type": "application/json"}), + ): + resp = send() + assert resp.status_code == 400, (body, resp.text) + assert resp.json() == {"error": "validation"} + + +def test_a_decimal_filter_operand_is_validated_and_exact(client: TestClient) -> None: + _post(client, '{"amount": "12.5"}') + _post(client, '{"amount": "100.25"}') + ok = client.get("/api/entries", params={"filter[amount][gt]": "12.5"}) + assert [r["amount"] for r in ok.json()] == ["100.25"] + eq = client.get("/api/entries", params={"filter[amount]": "12.50"}) + assert [r["amount"] for r in eq.json()] == ["12.5"] + for bad in ("abc", "", " 1", "0x10", "NaN"): + resp = client.get("/api/entries", params={"filter[amount][gt]": bad}) + assert resp.status_code == 400, (bad, resp.text) + assert resp.json() == {"error": "invalid_filter_value", "field": "amount"} + resp = client.get("/api/entries", params={"filter[amount][in]": "12.5,oops"}) + assert resp.status_code == 400 and resp.json()["error"] == "invalid_filter_value" + assert client.get("/api/entries", params={"filter[note][isNull]": "true"}).status_code == 200 + + +def test_the_sqlite_read_is_exact_to_15_significant_digits(client: TestClient) -> None: + # 0.1 + 0.2 as a double; the stored REAL reads back as the decimal it was meant to be + created = _post(client, '{"amount": "0.30000000000000004"}') + assert created.json()["amount"] == "0.3" + assert client.get("/api/entries/1").json()["amount"] == "0.3" + + +def test_the_runtime_itself_returns_a_decimal() -> None: + root = load_string(json.dumps(_META)).root + db = sqlite3.connect(":memory:") + db.execute("CREATE TABLE entries (id INTEGER PRIMARY KEY AUTOINCREMENT, amount NUMERIC, note NUMERIC)") + om = ObjectManager(root, SqliteDriver(db)) # type: ignore[arg-type] + row = om.create("Entry", {"amount": Decimal("0.1") + Decimal("0.2")}) + assert row["amount"] == Decimal("0.3") and isinstance(row["amount"], Decimal) + + +def test_a_value_beyond_the_declared_range_is_a_400_and_nothing_is_rounded(client: TestClient) -> None: + """W3: @precision 30, @scale 20 here, so 21 fraction digits and 11 integer digits fail.""" + for bad in ('"0.123456789012345678901"', '"12345678901.5"'): + resp = _post(client, '{"amount": ' + bad + "}") + assert resp.status_code == 400 and resp.json() == {"error": "validation"} + assert client.get("/api/entries").json() == [] diff --git a/server/python/tests/integration/test_normalization.py b/server/python/tests/integration/test_normalization.py index c7797325f..c9ae34d7e 100644 --- a/server/python/tests/integration/test_normalization.py +++ b/server/python/tests/integration/test_normalization.py @@ -25,3 +25,15 @@ def test_out_of_band_float_raises(): _canonical_float(1.0e-10) with pytest.raises(ValueError): normalize_value(1.0e-10) + + +def test_an_actual_decimal_slot_holding_a_string_is_not_canonicalized() -> None: + """A decimal column must come back as a Decimal: on the actual side a decimal-looking + string stays a string, so it cannot match the expected literal's canonical form.""" + from decimal import Decimal + + from .normalization import normalize_row + + assert normalize_row({"amount": Decimal("12.50")}, actual=True) == {"amount": "12.5"} + assert normalize_row({"amount": "12.50"}, actual=True) == {"amount": "12.50"} + assert normalize_row({"amount": "12.50"}) == {"amount": "12.5"} # an expected literal diff --git a/server/python/tests/integration/test_query_scenarios.py b/server/python/tests/integration/test_query_scenarios.py index bbcf368fb..a7bf2a8a8 100644 --- a/server/python/tests/integration/test_query_scenarios.py +++ b/server/python/tests/integration/test_query_scenarios.py @@ -1,14 +1,26 @@ -"""Parameterized: one test per query .yaml. +"""Parameterized: one test per query .yaml, on Postgres; the FR-044 ``report-*`` scenarios +also on SQLite and MySQL. -Each scenario provisions a fresh Testcontainers Postgres, applies the committed -TS-produced ``canonical/schema.postgres.sql`` verbatim (schema migrations are -TS-only — ADR-0015), then runs the query DSL through the ObjectManager runtime. +Postgres: each scenario provisions a fresh Postgres, applies the committed TS-produced +``canonical/schema.postgres.sql`` verbatim (schema migrations are TS-only -- ADR-0015), then +runs the query DSL through the ObjectManager runtime. + +SQLite and MySQL 8.4: the same ``report-*`` scenarios run against the engine's own +TS-produced artifact (``canonical/schema.sqlite.sql`` / ``schema.mysql.sql``), through the +ObjectManager's SQLite and MySQL drivers. The ``expect`` blocks stay the Postgres wire ones; +the runtime decodes each engine's values to the declared field types and the same +serialization boundary canonicalizes them. ``report-relative-date`` carries a per-engine seed +(``seed-data-engine``). One MySQL server serves the whole session; each scenario starts from +an empty schema. """ from __future__ import annotations +from pathlib import Path + import pytest from . import query_runner +from .mysql_container import MySqlInfo from .postgres_container import PostgresContainer from .scenarios import QueryScenario, find_corpus_root, load_queries @@ -29,10 +41,43 @@ def _scenarios() -> list[QueryScenario]: return load_queries(_CORPUS / "queries") -@pytest.mark.parametrize("scenario", _scenarios(), ids=lambda s: s.name) +def _for(engine: str) -> list[QueryScenario]: + return [s for s in _scenarios() if s.runs_on(engine)] + + +@pytest.mark.parametrize("scenario", _for("postgres"), ids=lambda s: s.name) def test_query_scenario(scenario: QueryScenario) -> None: reason = EXPECTED_FAILURES.get(scenario.name) if reason: pytest.skip(reason) with PostgresContainer() as pg: query_runner.run(scenario, pg, _CORPUS / "canonical") + + +def test_the_report_corpus_is_present() -> None: + """A rename of the ``report-`` prefix must not silently empty the extra-engine lanes.""" + assert len([s for s in _scenarios() if s.name.startswith("report-")]) == 24 + assert {s.name for s in _for("sqlite")} >= {"decimal-exact-15", "report-decimal-exact-15"} + assert "decimal-exact" not in {s.name for s in _for("sqlite")} + assert {"decimal-exact", "report-decimal-exact"} <= {s.name for s in _for("mysql")} + + +def test_an_unknown_engine_name_is_a_runner_error(tmp_path: Path) -> None: + from .scenarios import load_queries + + (tmp_path / "x.yaml").write_text("name: x\nengines: [oracle]\nqueries: []\n") + with pytest.raises(ValueError, match="unknown engine"): + load_queries(tmp_path) + (tmp_path / "x.yaml").write_text("name: x\nengines: []\nqueries: []\n") + with pytest.raises(ValueError, match="non-empty"): + load_queries(tmp_path) + + +@pytest.mark.parametrize("scenario", _for("sqlite"), ids=lambda s: s.name) +def test_report_scenario_on_sqlite(scenario: QueryScenario, tmp_path: Path) -> None: + query_runner.run_sqlite(scenario, tmp_path / "scenario.db", _CORPUS / "canonical") + + +@pytest.mark.parametrize("scenario", _for("mysql"), ids=lambda s: s.name) +def test_report_scenario_on_mysql(scenario: QueryScenario, mysql_empty: MySqlInfo) -> None: + query_runner.run_mysql(scenario, mysql_empty, _CORPUS / "canonical") diff --git a/server/python/tests/loader/test_naming_discovery.py b/server/python/tests/loader/test_naming_discovery.py new file mode 100644 index 000000000..11048907d --- /dev/null +++ b/server/python/tests/loader/test_naming_discovery.py @@ -0,0 +1,104 @@ +"""B1: the loaders find the project's `naming` the way codegen and verify do.""" +from __future__ import annotations + +import json +import warnings +from pathlib import Path + +import pytest + +from metaobjects import load_directory, load_string, load_uris +from metaobjects.naming import NamingRules, attach_naming + +_META = { + "metadata.root": { + "package": "acme::shop", + "children": [ + {"object.entity": {"name": "Item", "children": [ + {"source.rdb": {"@table": "items"}}, + {"field.long": {"name": "id"}}, + {"field.string": {"name": "unitPrice"}}, + {"identity.primary": {"name": "pk", "@fields": ["id"]}}, + ]}} + ], + } +} + + +def _project(tmp_path: Path, naming: dict | None, nested: str = "metaobjects") -> Path: + if naming is not None: + (tmp_path / ".metaobjects").mkdir() + (tmp_path / ".metaobjects" / "config.json").write_text( + json.dumps({"schema_version": 1, "naming": naming}) + ) + meta = tmp_path / nested + meta.mkdir(parents=True) + (meta / "meta.shop.json").write_text(json.dumps(_META)) + return meta + + +def test_load_directory_reads_the_project_naming(tmp_path: Path) -> None: + meta = _project(tmp_path, {"columns": "literal"}) + with warnings.catch_warnings(): + warnings.simplefilter("error") + root = load_directory(meta).root + assert root.naming == NamingRules(columns="literal") # type: ignore[attr-defined] + + +def test_the_search_walks_up_from_a_nested_metadata_directory(tmp_path: Path) -> None: + meta = _project(tmp_path, {"columns": "literal"}, nested="src/models/metaobjects") + assert load_directory(meta).root.naming.columns == "literal" # type: ignore[attr-defined] + + +def test_load_uris_reads_the_project_naming_from_the_first_file_uri(tmp_path: Path) -> None: + meta = _project(tmp_path, {"columns": "literal"}) + root = load_uris([(meta / "meta.shop.json").as_uri()]).root + assert root.naming.columns == "literal" # type: ignore[attr-defined] + + +def test_load_string_takes_a_project_dir(tmp_path: Path) -> None: + _project(tmp_path, {"columns": "literal"}) + root = load_string(json.dumps(_META), project_dir=tmp_path).root + assert root.naming.columns == "literal" # type: ignore[attr-defined] + + +def test_load_string_without_a_project_keeps_the_default_quietly() -> None: + with warnings.catch_warnings(): + warnings.simplefilter("error") + root = load_string(json.dumps(_META)).root + assert root.naming == NamingRules() # type: ignore[attr-defined] + + +def test_no_config_warns_once_naming_the_setting_and_attach_naming(tmp_path: Path) -> None: + meta = _project(tmp_path, None) + with pytest.warns(UserWarning) as rec: + root = load_directory(meta).root + messages = [str(w.message) for w in rec if "snake_case" in str(w.message)] + assert len(messages) == 1 + assert '"naming"' in messages[0] and "attach_naming" in messages[0] + assert root.naming == NamingRules() # type: ignore[attr-defined] + + +def test_load_uris_without_config_warns(tmp_path: Path) -> None: + meta = _project(tmp_path, None) + with pytest.warns(UserWarning, match="attach_naming"): + load_uris([(meta / "meta.shop.json").as_uri()]) + + +def test_an_explicit_attach_naming_wins(tmp_path: Path) -> None: + meta = _project(tmp_path, {"columns": "literal"}) + other = tmp_path / "other" + (other / ".metaobjects").mkdir(parents=True) + (other / ".metaobjects" / "config.json").write_text( + json.dumps({"schema_version": 1, "naming": {"columns": "snake_case", "tables": "literal"}}) + ) + root = attach_naming(load_directory(meta).root, other) + assert root.naming == NamingRules(columns="snake_case", tables="literal") # type: ignore[attr-defined] + + +def test_discovery_can_be_turned_off(tmp_path: Path) -> None: + meta = _project(tmp_path, {"columns": "literal"}) + with warnings.catch_warnings(): + warnings.simplefilter("error") + root = load_directory(meta, discover_naming=False).root + assert root.naming == NamingRules() # type: ignore[attr-defined] diff --git a/server/python/tests/runtime/test_decimal_wire.py b/server/python/tests/runtime/test_decimal_wire.py new file mode 100644 index 000000000..a23ec5c86 --- /dev/null +++ b/server/python/tests/runtime/test_decimal_wire.py @@ -0,0 +1,132 @@ +"""The Python half of the decimal wire rule against the shared vectors (ADR-0019 Amendment 1).""" +from __future__ import annotations + +import json +from decimal import Decimal +from pathlib import Path + +import pytest + +from metaobjects.codegen.runtime.decimal_wire import ( + coerce_decimal_fields, + decimal_from_double15, + format_decimal, + parse_decimal, + parse_decimal_operand, + parse_json_object, + to_wire, +) + +_VECTORS = json.loads( + (Path(__file__).resolve().parents[4] / "fixtures" / "decimal-wire" / "vectors.json").read_text() +) + + +@pytest.mark.parametrize("v", _VECTORS["format"], ids=lambda v: v["in"]) +def test_format(v: dict[str, str]) -> None: + assert format_decimal(Decimal(v["in"])) == v["out"] + + +def _body_value(text: str) -> object: + return json.loads(text, parse_float=Decimal) + + +@pytest.mark.parametrize("v", _VECTORS["parse"], ids=lambda v: v["json"]) +def test_parse_body(v: dict[str, object]) -> None: + raw = _body_value(str(v["json"])) + if v.get("error"): + with pytest.raises(ValueError): + parse_decimal(raw) + elif v["value"] is None: + assert raw is None + else: + assert parse_decimal(raw) == Decimal(str(v["value"])) + + +@pytest.mark.parametrize("v", _VECTORS["filterOperand"], ids=lambda v: repr(v["text"])) +def test_parse_filter_operand(v: dict[str, object]) -> None: + if v.get("error"): + with pytest.raises(ValueError): + parse_decimal_operand(str(v["text"])) + else: + assert parse_decimal_operand(str(v["text"])) == Decimal(str(v["value"])) + + +@pytest.mark.parametrize("v", _VECTORS["double15"], ids=lambda v: v["double"]) +def test_double15(v: dict[str, str]) -> None: + assert format_decimal(decimal_from_double15(float(v["double"]))) == v["out"] + + +def test_a_body_number_is_never_a_float() -> None: + body = parse_json_object('{"a": 12345678901234567.89, "b": 0.1}') + assert body["a"] == Decimal("12345678901234567.89") and body["b"] == Decimal("0.1") + + +def test_the_sum_of_two_doubles_reads_as_the_decimal_they_were_written_as() -> None: + assert decimal_from_double15(0.1 + 0.2) == Decimal("0.3") + + +def test_out_of_range_is_refused_not_rounded() -> None: + with pytest.raises(ValueError): + parse_decimal("1e999999999") + with pytest.raises(ValueError): + parse_decimal_operand("1e5000") + with pytest.raises(ValueError): + format_decimal(Decimal("1e5000")) + + +def test_non_finite_and_non_numbers_are_refused() -> None: + for bad in (Decimal("NaN"), Decimal("Infinity"), float("nan"), True, [1], {}, b"1"): + with pytest.raises(ValueError): + parse_decimal(bad) + + +def test_parse_json_object_refuses_non_objects_and_garbage() -> None: + for bad in ("[1]", "1", "{", b"\xff"): + with pytest.raises(ValueError): + parse_json_object(bad) + + +def test_coerce_decimal_fields_names_the_bad_field() -> None: + dto = {"amount": "12.50", "note": None, "label": "x"} + assert coerce_decimal_fields(dto, frozenset({"amount", "note"})) == { + "amount": Decimal("12.5"), "note": None, "label": "x", + } + with pytest.raises(ValueError, match="amount"): + coerce_decimal_fields({"amount": "abc"}, frozenset({"amount"})) + + +def test_to_wire_canonicalises_every_decimal_in_a_response() -> None: + assert to_wire({"rows": [{"a": Decimal("12.50"), "n": None, "i": 3}], "total": 1}) == { + "rows": [{"a": "12.5", "n": None, "i": 3}], "total": 1, + } + assert to_wire(Decimal("0E-18")) == "0" + + +# --- W3: the column's range --------------------------------------------------------------- + + +@pytest.mark.parametrize( + "text,ok", + [ + ("12345678901234567.8901234567", True), # 18 integer + 10 fraction digits in (28, 10) + ("0.1234567890", True), + ("0.12345678901", False), # 11th fraction digit + ("1234567890123456789", False), # 19 integer digits + ("100.2500", True), # trailing zeros do not count + ("1.5e1", True), + ("-0.0000000001", True), + ("0", True), + ("1e19", False), + ], +) +def test_w3_range_is_enforced_and_nothing_is_rounded(text: str, ok: bool) -> None: + from metaobjects.codegen.runtime.decimal_wire import coerce_decimal_fields + + fields = frozenset({"amount"}) + limits = {"amount": (28, 10)} + if ok: + assert coerce_decimal_fields({"amount": text}, fields, limits)["amount"] == Decimal(text) + else: + with pytest.raises(ValueError): + coerce_decimal_fields({"amount": text}, fields, limits) diff --git a/server/python/tests/runtime/test_literal_naming_sqlite.py b/server/python/tests/runtime/test_literal_naming_sqlite.py new file mode 100644 index 000000000..29063182e --- /dev/null +++ b/server/python/tests/runtime/test_literal_naming_sqlite.py @@ -0,0 +1,57 @@ +"""B1, end to end on SQLite: `naming.columns: literal` in the project's +`.metaobjects/config.json`, found by `load_directory`, makes the ObjectManager read a +table whose columns are the literal field names.""" +from __future__ import annotations + +import json +import sqlite3 +from pathlib import Path + +from metaobjects import load_directory +from metaobjects.runtime import ObjectManager, SqliteDriver + +_META = { + "metadata.root": { + "package": "acme::shop", + "children": [ + {"object.entity": {"name": "Item", "children": [ + {"source.rdb": {"@table": "items"}}, + {"field.long": {"name": "id"}}, + {"field.string": {"name": "unitPrice"}}, + {"identity.primary": {"name": "pk", "@fields": ["id"]}}, + ]}} + ], + } +} + + +def _project(tmp_path: Path, naming: dict[str, str]) -> Path: + (tmp_path / ".metaobjects").mkdir() + (tmp_path / ".metaobjects" / "config.json").write_text( + json.dumps({"schema_version": 1, "naming": naming}) + ) + (tmp_path / "metaobjects").mkdir() + (tmp_path / "metaobjects" / "meta.shop.json").write_text(json.dumps(_META)) + return tmp_path / "metaobjects" + + +def _db() -> sqlite3.Connection: + db = sqlite3.connect(":memory:") + db.execute('CREATE TABLE "items" ("id" INTEGER PRIMARY KEY, "unitPrice" TEXT)') + db.execute("INSERT INTO items VALUES (1, 'ten')") + return db + + +def test_literal_columns_are_read_through_the_object_manager(tmp_path: Path) -> None: + meta = _project(tmp_path, {"columns": "literal"}) + om = ObjectManager(load_directory(meta).root, SqliteDriver(_db())) # type: ignore[arg-type] + assert om.find_many("Item") == [{"id": 1, "unitPrice": "ten"}] + assert om.create("Item", {"unitPrice": "five"})["unitPrice"] == "five" + + +def test_without_the_setting_the_runtime_does_not_read_the_literal_column(tmp_path: Path) -> None: + meta = _project(tmp_path, {}) + om = ObjectManager(load_directory(meta).root, SqliteDriver(_db())) # type: ignore[arg-type] + # SQLite reads an unknown double-quoted identifier as a string literal, so the wrong + # name does not error: it answers with the name itself, and the stored value is never read. + assert "ten" not in om.find_many("Item")[0].values() diff --git a/server/python/tests/runtime/test_object_manager_dialects.py b/server/python/tests/runtime/test_object_manager_dialects.py new file mode 100644 index 000000000..9e2f5f5fb --- /dev/null +++ b/server/python/tests/runtime/test_object_manager_dialects.py @@ -0,0 +1,417 @@ +"""The ObjectManager on SQLite and MySQL: the SQL each driver's dialect emits, and the values +it decodes. + +SQLite runs for real (stdlib ``sqlite3``, in memory) with a recording connection wrapped around +it. MySQL has no embedded engine, so a scripted double stands in for PyMySQL and the assertions +are on the SQL and the parameters it was handed; the live MySQL lane is +``tests/integration/test_query_scenarios.py`` and ``test_api_contract_report.py``. +""" +from __future__ import annotations + +import datetime as dt +import decimal +import json +import sqlite3 +import uuid +from typing import Any + +import pytest + +from metaobjects import load_string +from metaobjects.naming import NamingRules +from metaobjects.runtime import ( + MYSQL, + POSTGRES, + SQLITE, + MySqlDriver, + ObjectManager, + PostgresDriver, + SqliteDriver, +) + +_META = { + "metadata.root": { + "package": "acme::shop", + "children": [ + { + "object.entity": { + "name": "Item", + "children": [ + {"source.rdb": {"@table": "items"}}, + {"field.long": {"name": "id"}}, + {"field.string": {"name": "label", "@required": True}}, + {"field.int": {"name": "qty"}}, + {"field.boolean": {"name": "active"}}, + {"field.decimal": {"name": "price", "@precision": 9, "@scale": 2}}, + {"field.uuid": {"name": "ref"}}, + {"field.date": {"name": "madeOn"}}, + {"field.timestamp": {"name": "seenAt"}}, + {"identity.primary": {"name": "pk", "@fields": ["id"], "@generation": "increment"}}, + ], + } + } + ], + } +} + + +def _root() -> Any: + root = load_string(json.dumps(_META)).root + root.naming = NamingRules(columns="literal") + return root + + +class Recording: + """A DB-API connection that records every statement and delegates to a real one.""" + + def __init__(self, real: Any) -> None: + self.real = real + self.sql: list[tuple[str, tuple[Any, ...]]] = [] + + def cursor(self) -> Any: + outer = self + cur = self.real.cursor() + + class _Cur: + def execute(self, sql: str, params: tuple[Any, ...] = ()) -> Any: + outer.sql.append((sql, tuple(params))) + return cur.execute(sql, params) + + def __getattr__(self, name: str) -> Any: + return getattr(cur, name) + + return _Cur() + + def commit(self) -> None: + self.real.commit() + + def close(self) -> None: + self.real.close() + + +_SQLITE_DDL = ( + 'CREATE TABLE "items" ("id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, "label" TEXT NOT NULL, ' + '"qty" INTEGER, "active" INTEGER, "price" NUMERIC, "ref" TEXT, "madeOn" TEXT, "seenAt" TEXT)' +) + + +def _sqlite_om() -> tuple[ObjectManager, Recording]: + real = sqlite3.connect(":memory:") + real.execute(_SQLITE_DDL) + rec = Recording(real) + return ObjectManager(_root(), SqliteDriver(rec)), rec # type: ignore[arg-type] + + +# ---------------------------------------------------------------------------- +# SQLite: emitted SQL and a real round trip +# ---------------------------------------------------------------------------- + + +def test_sqlite_read_uses_qmark_placeholders_and_double_quotes() -> None: + om, rec = _sqlite_om() + om.find_many("Item", {"qty": {"gte": 2}, "label": "a"}, sort=[("qty", "desc")], limit=5, offset=10) + sql, params = rec.sql[-1] + assert sql == ( + 'SELECT "id", "label", "qty", "active", "price", "ref", "madeOn", "seenAt" FROM "items" ' + 'WHERE "qty" >= ? AND "label" = ? ORDER BY "qty" DESC LIMIT 5 OFFSET 10' + ) + assert params == (2, "a") + + +def test_sqlite_offset_without_limit_adds_an_unbounded_limit() -> None: + om, rec = _sqlite_om() + om.find_many("Item", offset=3) + assert rec.sql[-1][0].endswith("LIMIT -1 OFFSET 3") + + +def test_sqlite_like_is_case_sensitive_through_glob() -> None: + om, rec = _sqlite_om() + om.create("Item", {"label": "Alpha"}) + om.create("Item", {"label": "alpha"}) + om.create("Item", {"label": "a*b"}) + assert [r["label"] for r in om.find_many("Item", {"label": {"like": "A%"}})] == ["Alpha"] + # `*` in the pattern is a literal, not GLOB's wildcard + assert [r["label"] for r in om.find_many("Item", {"label": {"like": "a*_"}})] == ["a*b"] + assert 'GLOB ?' in rec.sql[-1][0] + + +def test_sqlite_empty_in_matches_nothing() -> None: + om, rec = _sqlite_om() + om.create("Item", {"label": "x"}) + assert om.find_many("Item", {"id": {"in": []}}) == [] + assert "0 = 1" in rec.sql[-1][0] + + +def test_sqlite_write_path_returns_the_row_decoded_to_declared_types() -> None: + om, rec = _sqlite_om() + created = om.create( + "Item", + { + "label": "pen", "qty": 3, "active": True, "price": "12.50", + "ref": "AAAAAAAA-AAAA-4AAA-8AAA-AAAAAAAAAAAA", "madeOn": "2026-05-01", + "seenAt": "2026-05-04T03:30:00Z", + }, + ) + assert rec.sql[0][0].startswith('INSERT INTO "items"') and "RETURNING" in rec.sql[0][0] + assert created["active"] is True + assert created["price"] == decimal.Decimal("12.5") + assert created["ref"] == uuid.UUID("aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa") + assert created["madeOn"] == dt.date(2026, 5, 1) + assert created["seenAt"] == dt.datetime(2026, 5, 4, 3, 30, tzinfo=dt.timezone.utc) + # what was BOUND is the engine's spelling, not a Python object sqlite3 would mis-adapt + _, params = rec.sql[0] + assert "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa" in params + assert "2026-05-04T03:30:00.000Z" in params + assert "2026-05-01" in params + assert 12.5 in params + + updated = om.update("Item", created["id"], {"active": False, "qty": 4}) + assert updated is not None and updated["active"] is False and updated["qty"] == 4 + assert om.update("Item", 999, {"qty": 1}) is None + assert om.delete("Item", created["id"]) is True + assert om.delete("Item", created["id"]) is False + + +def test_sqlite_without_returning_rereads_by_key(monkeypatch: pytest.MonkeyPatch) -> None: + """SQLite older than 3.35 has no RETURNING: the write path inserts, then reads by key.""" + monkeypatch.setattr("metaobjects.runtime.dialect.sqlite_supports_returning", lambda: False) + om, rec = _sqlite_om() + created = om.create("Item", {"label": "pen", "qty": 1}) + assert created["id"] == 1 and created["label"] == "pen" + assert not any("RETURNING" in sql for sql, _ in rec.sql) + assert rec.sql[-1][0].startswith('SELECT "id", "label"') + updated = om.update("Item", 1, {"qty": 2}) + assert updated is not None and updated["qty"] == 2 + assert om.update("Item", 7, {"qty": 2}) is None + + +def test_sqlite_type_codes_follow_the_declared_subtype() -> None: + om, _ = _sqlite_om() + om.create("Item", {"label": "x", "qty": 1}) + om.find_many("Item") + # long -> int8 (a BIGINT on the wire is a string); int -> int4 (stays a number) + assert om.last_column_oids["id"] == 20 + assert om.last_column_oids["qty"] == 23 + assert "label" not in om.last_column_oids + + +# ---------------------------------------------------------------------------- +# MySQL: a scripted PyMySQL double +# ---------------------------------------------------------------------------- + + +class _MyCursor: + def __init__(self, conn: "FakeMySql") -> None: + self._conn = conn + self.description: Any = None + self.rowcount = 0 + self.lastrowid = 0 + self._rows: list[tuple[Any, ...]] = [] + + def execute(self, sql: str, params: tuple[Any, ...] = ()) -> None: + self._conn.sql.append((sql, tuple(params))) + columns, rows, rowcount, lastrowid = self._conn.script(sql) + self.description = [(c, None, None, None, None, None, None) for c in columns] + self._rows = rows + self.rowcount = rowcount + self.lastrowid = lastrowid + + def fetchall(self) -> list[tuple[Any, ...]]: + return self._rows + + def close(self) -> None: + pass + + +class FakeMySql: + def __init__(self) -> None: + self.sql: list[tuple[str, tuple[Any, ...]]] = [] + self.row: tuple[Any, ...] | None = None + self.update_rowcount = 1 + self.insert_id = 42 + + COLS = ["id", "label", "qty", "active", "price", "ref", "madeOn", "seenAt"] + + def script(self, sql: str) -> tuple[list[str], list[tuple[Any, ...]], int, int]: + if sql.startswith("INSERT"): + return [], [], 1, self.insert_id + if sql.startswith(("UPDATE", "DELETE")): + return [], [], self.update_rowcount, 0 + if sql.startswith("SELECT COUNT"): + return ["COUNT(*)"], [(7,)], 0, 0 + return self.COLS, [self.row] if self.row else [], 0, 0 + + def cursor(self) -> _MyCursor: + return _MyCursor(self) + + def commit(self) -> None: + pass + + def close(self) -> None: + pass + + +def _mysql_om() -> tuple[ObjectManager, FakeMySql]: + conn = FakeMySql() + conn.row = (42, "pen", 3, 1, decimal.Decimal("12.50"), "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", + dt.date(2026, 5, 1), dt.datetime(2026, 5, 4, 3, 30)) + return ObjectManager(_root(), MySqlDriver(conn)), conn # type: ignore[arg-type] + + +def test_mysql_read_uses_backticks_and_percent_placeholders() -> None: + om, conn = _mysql_om() + om.find_many("Item", {"qty": {"gte": 2}, "label": {"like": "A%"}}, sort=[("qty", "asc")], offset=4) + sql, params = conn.sql[-1] + assert sql == ( + "SELECT `id`, `label`, `qty`, `active`, `price`, `ref`, `madeOn`, `seenAt` FROM `items` " + "WHERE `qty` >= %s AND `label` LIKE BINARY %s ORDER BY `qty` ASC " + "LIMIT 18446744073709551615 OFFSET 4" + ) + assert params == (2, "A%") + assert '"' not in sql # never relies on ANSI_QUOTES + + +def test_mysql_count_and_empty_in() -> None: + om, conn = _mysql_om() + assert om.count("Item", {"id": {"in": []}}) == 7 + assert conn.sql[-1][0] == "SELECT COUNT(*) FROM `items` WHERE FALSE" + + +def test_mysql_decodes_to_declared_types() -> None: + om, _ = _mysql_om() + (row,) = om.find_many("Item") + assert row["active"] is True + assert row["price"] == decimal.Decimal("12.50") + assert row["ref"] == uuid.UUID("aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa") + assert row["seenAt"] == dt.datetime(2026, 5, 4, 3, 30, tzinfo=dt.timezone.utc) # a DATETIME is UTC + + +def test_mysql_create_inserts_then_rereads_by_the_generated_key() -> None: + om, conn = _mysql_om() + created = om.create("Item", {"label": "pen"}) + insert, select = conn.sql[0], conn.sql[1] + assert insert[0] == "INSERT INTO `items` (`label`) VALUES (%s)" + assert "RETURNING" not in insert[0] + assert select == ( + "SELECT `id`, `label`, `qty`, `active`, `price`, `ref`, `madeOn`, `seenAt` " + "FROM `items` WHERE `id` = %s", + (42,), + ) + assert created["id"] == 42 and created["label"] == "pen" + + +def test_mysql_create_with_a_supplied_key_rereads_by_it() -> None: + om, conn = _mysql_om() + conn.insert_id = 0 + om.create("Item", {"id": 9, "label": "pen"}) + assert conn.sql[1][1] == (9,) + + +def test_mysql_create_with_no_key_at_all_is_refused() -> None: + om, conn = _mysql_om() + conn.insert_id = 0 + with pytest.raises(ValueError, match="supply a value for 'id'"): + om.create("Item", {"label": "pen"}) + + +def test_mysql_default_values_insert() -> None: + om, conn = _mysql_om() + om.create("Item", {}) + assert conn.sql[0][0] == "INSERT INTO `items` () VALUES ()" + + +def test_mysql_update_reads_back_even_when_nothing_changed() -> None: + om, conn = _mysql_om() + conn.update_rowcount = 0 # MySQL reports CHANGED rows, so a no-op update reads as 0 + updated = om.update("Item", 42, {"qty": 3}) + assert updated is not None and updated["id"] == 42 + assert conn.sql[0][0] == "UPDATE `items` SET `qty` = %s WHERE `id` = %s" + assert conn.sql[1][0].startswith("SELECT `id`") + + +def test_mysql_update_of_a_missing_row_is_none() -> None: + om, conn = _mysql_om() + conn.row = None + assert om.update("Item", 1, {"qty": 3}) is None + with pytest.raises(KeyError): + om.update("Item", 1, {"qty": 3}, if_missing="throw") + + +def test_mysql_delete_counts_rows() -> None: + om, conn = _mysql_om() + assert om.delete("Item", 1) is True + assert conn.sql[-1][0] == "DELETE FROM `items` WHERE `id` = %s" + conn.update_rowcount = 0 + assert om.delete("Item", 1) is False + + +def test_postgres_is_unchanged() -> None: + class Conn(FakeMySql): + pass + + conn = Conn() + om = ObjectManager(_root(), PostgresDriver(conn)) # type: ignore[arg-type] + om.find_many("Item", {"label": {"like": "A%"}}, limit=1) + assert conn.sql[-1][0] == ( + 'SELECT "id", "label", "qty", "active", "price", "ref", "madeOn", "seenAt" FROM "items" ' + 'WHERE "label" LIKE %s LIMIT 1' + ) + + +# ---------------------------------------------------------------------------- +# Dialect value hooks and the shared pieces +# ---------------------------------------------------------------------------- + + +def test_identifier_quoting_refuses_the_quote_character() -> None: + with pytest.raises(ValueError): + POSTGRES.quote('a"b') + with pytest.raises(ValueError): + SQLITE.quote('a"b') + with pytest.raises(ValueError): + MYSQL.quote("a`b") + assert MYSQL.quote('a"b') == '`a"b`' + + +def test_mysql_binds_an_aware_datetime_as_the_utc_wall_clock() -> None: + aware = dt.datetime(2026, 5, 4, 5, 30, tzinfo=dt.timezone(dt.timedelta(hours=2))) + assert MYSQL.encode_param(aware) == dt.datetime(2026, 5, 4, 3, 30) + assert MYSQL.encode_param(uuid.UUID(int=1)) == "00000000-0000-0000-0000-000000000001" + + +def test_sqlite_binds_the_engines_spelling() -> None: + assert SQLITE.encode_param(True) == 1 + assert SQLITE.encode_param(decimal.Decimal("3")) == 3 + assert SQLITE.encode_param(decimal.Decimal("0.5")) == 0.5 + assert SQLITE.encode_param(dt.datetime(2026, 5, 4, 3, 30)) == "2026-05-04T03:30:00.000" + assert SQLITE.encode_param(dt.time(1, 2, 3)) == "01:02:03" + + +def test_a_filter_operand_given_as_text_is_typed_by_the_field() -> None: + from metaobjects.runtime.object_manager import _compile_filter + + root = _root() + item = next(c for c in root.own_children() if c.name == "Item") + sql, params = _compile_filter( + {"qty": {"gt": "2"}, "active": "true", "price": {"in": ["1.5", "2"]}, "label": {"eq": "7"}}, + item, "literal", SQLITE, + ) or ("", []) + assert sql == '"qty" > ? AND "active" = ? AND "price" IN (?, ?) AND "label" = ?' + assert params == [2, True, decimal.Decimal("1.5"), decimal.Decimal("2"), "7"] + + +def test_a_mysql_decimal_operand_is_cast_to_a_decimal() -> None: + """ADR-0019 W2: MySQL compares DECIMAL = 'text' as doubles, so the operand is cast.""" + from decimal import Decimal + + from metaobjects.meta.core.field.meta_field import MetaField + from metaobjects.runtime.dialect import MYSQL, POSTGRES + from metaobjects.runtime.object_manager import _op_clause + from metaobjects.shared.base_types import TYPE_FIELD + + f = MetaField(TYPE_FIELD, "decimal", "amount") + sql, params = _op_clause("amount", "eq", Decimal("1.5"), f, MYSQL) + assert sql == "`amount` = CAST(%s AS DECIMAL(65,30))" and params == [Decimal("1.5")] + sql, _ = _op_clause("amount", "in", [Decimal("1"), Decimal("2")], f, MYSQL) + assert sql.count("CAST(%s AS DECIMAL(65,30))") == 2 + assert _op_clause("amount", "eq", Decimal("1.5"), f, POSTGRES)[0] == '"amount" = %s' diff --git a/server/python/tests/test_report_shape.py b/server/python/tests/test_report_shape.py index 7d44edbc6..b1db6e3be 100644 --- a/server/python/tests/test_report_shape.py +++ b/server/python/tests/test_report_shape.py @@ -122,9 +122,9 @@ def test_derived_shapes_byte_match_the_committed_artifact() -> None: assert generate_report_shapes_json(_root()) == expected -def test_the_canonical_model_has_twenty_three_reports() -> None: +def test_the_canonical_model_has_twenty_five_reports() -> None: reports = [c for c in _root().children() if c.type == TYPE_OBJECT and c.sub_type == OBJECT_SUBTYPE_REPORT] - assert len(reports) == 23 + assert len(reports) == 25 assert all(isinstance(r, MetaObject) for r in reports) diff --git a/server/python/tests/unit/test_reporting_validation_spine_default.py b/server/python/tests/unit/test_reporting_validation_spine_default.py index 9a420c269..9d5e70fd7 100644 --- a/server/python/tests/unit/test_reporting_validation_spine_default.py +++ b/server/python/tests/unit/test_reporting_validation_spine_default.py @@ -784,19 +784,17 @@ def test_m8_default_on_a_max_of_a_timestamp_is_refused() -> None: ) -def test_a_non_integer_default_on_a_count_reports_m7_and_the_attribute_type_error() -> None: +def test_a_non_numeric_default_on_a_count_reports_only_the_attribute_type_error() -> None: + """MINOR 4 Rule B: one mistake, one error. M7 runs only for a valid integral number.""" errors = _errors(_edit(lambda x: _patch_member(x, "Purchase", "purchases", {"@default": "zero"}))) - assert sorted(e.code.value for e in errors) == ["ERR_BAD_ATTR_VALUE", "ERR_INVALID_MEASURE"] - m7 = next(e for e in errors if e.code.value == "ERR_INVALID_MEASURE") - assert "measure 'purchases' on entity 'acme::shop::Purchase': @default cannot apply to @agg: count." in m7.message + assert [e.code.value for e in errors] == ["ERR_BAD_ATTR_VALUE"] -def test_a_boolean_default_on_a_count_reports_m7_and_the_attribute_type_error() -> None: +def test_a_boolean_default_on_a_count_reports_only_the_attribute_type_error() -> None: """``bool`` is a subclass of ``int`` in Python, so ``true`` must not pass for the - integer 1: it is the attribute type error, not a fraction, so M7 still reads the - attribute as present (the "zero" case above, for this port's bool trap).""" + integer 1: it is the attribute type error and M7 does not also report.""" errors = _errors(_edit(lambda x: _patch_member(x, "Purchase", "purchases", {"@default": True}))) - assert sorted(e.code.value for e in errors) == ["ERR_BAD_ATTR_VALUE", "ERR_INVALID_MEASURE"] + assert [e.code.value for e in errors] == ["ERR_BAD_ATTR_VALUE"] def _fractional(entity: str, measure: str, value: float) -> MetaError: diff --git a/server/python/uv.lock b/server/python/uv.lock index 360547541..ea407e428 100644 --- a/server/python/uv.lock +++ b/server/python/uv.lock @@ -267,6 +267,10 @@ integration = [ { name = "fastapi" }, { name = "httpx" }, { name = "pg8000" }, + { name = "pymysql" }, +] +mysql = [ + { name = "pymysql" }, ] [package.dev-dependencies] @@ -283,11 +287,13 @@ requires-dist = [ { name = "mypy", marker = "extra == 'dev'", specifier = ">=1.10" }, { name = "pg8000", marker = "extra == 'integration'", specifier = ">=1.31" }, { name = "pydantic", marker = "extra == 'dev'", specifier = ">=2" }, + { name = "pymysql", marker = "extra == 'integration'", specifier = ">=1.1" }, + { name = "pymysql", marker = "extra == 'mysql'", specifier = ">=1.1" }, { name = "pytest", marker = "extra == 'dev'", specifier = ">=8" }, { name = "pyyaml", specifier = ">=6.0" }, { name = "ruff", marker = "extra == 'dev'", specifier = ">=0.6" }, ] -provides-extras = ["dev", "integration"] +provides-extras = ["dev", "integration", "mysql"] [package.metadata.requires-dev] dev = [ @@ -522,6 +528,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" }, ] +[[package]] +name = "pymysql" +version = "1.2.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b1/d4/c15b459e25a23767d2f4065ef40968920320f04e302889574310c21c96a3/pymysql-1.2.3.tar.gz", hash = "sha256:d5b288529782e536ae171866df3ca9dc4f6cbfb3cc2f18e6f837fbb90dbc262b", size = 50629, upload-time = "2026-09-17T12:22:49.146Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a9/4b/0a906d8184f011ff8dbd4722743783867589b33269d2c5fff238d636fdcb/pymysql-1.2.3-py3-none-any.whl", hash = "sha256:14f1c68e2ed859243ae5ca41ffbe677027fc46bc136a9f0be8a4e928e5e7415a", size = 46740, upload-time = "2026-09-17T12:22:47.826Z" }, +] + [[package]] name = "pytest" version = "9.0.3" diff --git a/server/typescript/packages/cli/README.md b/server/typescript/packages/cli/README.md index 350cb465d..232d4358e 100644 --- a/server/typescript/packages/cli/README.md +++ b/server/typescript/packages/cli/README.md @@ -312,7 +312,7 @@ Precedence for `meta migrate`: CLI flag > env var (`DATABASE_URL` only) > `.meta ### Metadata sources (`sources`) `sources` is the single authority on **where metadata lives**. Every read command -(`gen`, `migrate`, `verify`, `docs`, `export`) resolves it. When the key is absent or +(`gen`, `migrate`, `verify`, `export`) resolves it. When the key is absent or empty — which is what `meta init` scaffolds — it takes its default value, the `metaobjects/` directory beside the config, so existing projects are unaffected. @@ -357,7 +357,7 @@ means everything; `exclude` applies after `include`; matching is case-sensitive. applies to `meta gen` and to `meta verify --codegen` (which regenerates under the same scope, so a scoped `gen` is not reported as drift). The documentation generators are not scoped: the model pages, the api reference, the agent pages and the site document the -whole loaded collection, as `meta docs` did. `meta export` is deliberately **not** scoped — it +whole loaded collection, whatever `scope` says. `meta export` is deliberately **not** scoped — it inspects the loaded collection. `meta gen ` arguments intersect with `scope`; both must pass. diff --git a/server/typescript/packages/cli/src/commands/eject.ts b/server/typescript/packages/cli/src/commands/eject.ts index 318889ab4..65d318024 100644 --- a/server/typescript/packages/cli/src/commands/eject.ts +++ b/server/typescript/packages/cli/src/commands/eject.ts @@ -417,6 +417,21 @@ interface GeneratorListRow { package: string; /** `available` = not ejected; `owned` = ejected but the reference could not be read. */ status: "available" | "owned" | "identical" | "reformatted" | "differs"; + /** Lines behind / of your own in the GENERATOR file alone. */ + behind: number; + own: number; + /** + * The renderers this generator owns under `codegen/templates/` (those it imports that + * exist in the project), each against the installed package. A generator is `identical` + * only when its file AND every one of these is; `status` is `differs` when any differs. + * Absent when the project owns none of its renderers. + */ + templates?: GeneratorTemplateRow[]; +} + +interface GeneratorTemplateRow { + path: string; + status: "identical" | "reformatted" | "differs"; behind: number; own: number; } @@ -429,6 +444,9 @@ interface GeneratorListGroup { async function generatorListGroups(cwd: string): Promise { const groups: GeneratorListGroup[] = []; + // Every renderer the project holds, against the package: attributed to a generator by what + // that generator imports, so a renderer another generator owns never flags this one. + const templateStatus = await templateCopyStatus(cwd, packagedTemplates()); for (const source of SOURCES) { const rows: GeneratorListRow[] = []; groups.push({ packageName: source.packageName, rows }); @@ -452,6 +470,19 @@ async function generatorListGroups(cwd: string): Promise { row.behind = cmp.referenceOnly; row.own = cmp.localOnly; } + const reached = new Set(closureOf(name).templates.map((f) => f.path)); + const renderers: GeneratorTemplateRow[] = []; + for (const t of templateStatus) { + if (!reached.has(t.path) || t.verdict === "not-in-package") continue; + renderers.push({ + path: t.path, + status: t.verdict === "identical" || t.verdict === "reformatted" ? t.verdict : "differs", + behind: t.referenceOnly, + own: t.localOnly, + }); + } + if (renderers.length > 0) row.templates = renderers; + if (renderers.some((r) => r.status === "differs")) row.status = "differs"; } } return groups; @@ -476,6 +507,7 @@ async function listOutput(cwd: string): Promise { lines.push(""); const groups = await generatorListGroups(cwd); const anyStale = groups.some((g) => g.rows.some((r) => r.status === "differs")); + const generatorFileDiffers = (r: GeneratorListRow): boolean => r.behind > 0 || r.own > 0; for (const group of groups) { lines.push(`${group.packageName}:`); for (const r of group.rows) { @@ -484,10 +516,15 @@ async function listOutput(cwd: string): Promise { else if (r.status === "identical") lines.push(` ${r.name} [owned — identical to the reference]`); else if (r.status === "reformatted") lines.push(` ${r.name} [owned — same content as the reference, your formatting]`); else { + const differing = (r.templates ?? []).filter((t) => t.status === "differs"); lines.push( - ` ${r.name} [owned — DIFFERS: ${r.behind} line(s) behind, ` + - `${r.own} line(s) of your own]`, + generatorFileDiffers(r) + ? ` ${r.name} [owned — DIFFERS: ${r.behind} line(s) behind, ${r.own} line(s) of your own]` + : ` ${r.name} [owned — DIFFERS: generator identical, ${differing.length} renderer(s) differ]`, ); + for (const t of differing) { + lines.push(` renderer ${t.path} [DIFFERS: ${t.behind} line(s) behind, ${t.own} line(s) of your own]`); + } } } lines.push(""); diff --git a/server/typescript/packages/cli/src/commands/init.ts b/server/typescript/packages/cli/src/commands/init.ts index 64a4f7037..96262db78 100644 --- a/server/typescript/packages/cli/src/commands/init.ts +++ b/server/typescript/packages/cli/src/commands/init.ts @@ -12,10 +12,12 @@ import { reportIgnoredScaffold } from "../lib/ignored-scaffold-check.js"; import { parseInitArgs } from "../lib/args.js"; import { log } from "../lib/log.js"; import { cliVersion, isPrereleaseVersion } from "../lib/version.js"; +import { currentContentHash } from "../lib/agent-context-staleness.js"; import { findWranglerConfig, parseWranglerConfig } from "@metaobjectsdev/migrate-ts"; import { DEFAULT_DOCS_DIR } from "@metaobjectsdev/codegen-ts"; import { describeError } from "../lib/error-text.js"; import { SUBTYPE_ROOT, TYPE_METADATA, TYPE_SUBTYPE_SEPARATOR } from "@metaobjectsdev/metadata/constants"; +import { docsRefForVersion } from "@metaobjectsdev/metadata"; // ADR-0034 scaffold-and-own — where `meta eject` and `meta generator new` put a generator // the consumer OWNS. `meta init` scaffolds it empty; metaobjects.config.ts imports from it @@ -311,10 +313,11 @@ async function writeAgentContext(opts: InitOptions, result: InitResult): Promise warnIfMonorepoSubdir(opts, result); const prior = await readManifest(opts.cwd); const stack = await stackForAgentContext(opts, prior); - let assembled = assemble({ contentRoot: resolveAgentContextRoot(), stack }); + let assembled = assemble({ contentRoot: resolveAgentContextRoot(), stack, docsRef: docsRefForVersion(cliVersion()) }); if (opts.noSkills) assembled = assembled.filter((f) => !f.path.startsWith(".claude/skills/")); const generatedBy = cliVersion(); + const contentHash = currentContentHash().currentContentHash; const decision = planScaffold({ stack, assembled, prior, readCurrent: (rel) => { @@ -322,6 +325,7 @@ async function writeAgentContext(opts: InitOptions, result: InitResult): Promise return existsSyncWrap(abs) ? readFileSyncWrap(abs, "utf8") : undefined; }, generatedBy, + ...(contentHash !== undefined && { contentHash }), }); // A pre-release CLI stamps its own pre-release version into the committed // .metaobjects/.agent-context.json ("generatedBy") and the skills/docs it scaffolds may diff --git a/server/typescript/packages/cli/src/commands/migrate.ts b/server/typescript/packages/cli/src/commands/migrate.ts index 8c693a475..eb907fae5 100644 --- a/server/typescript/packages/cli/src/commands/migrate.ts +++ b/server/typescript/packages/cli/src/commands/migrate.ts @@ -65,6 +65,7 @@ import { import { buildProjectionViews } from "@metaobjectsdev/codegen-ts"; import { tokensToAllowOptions, blockedEntriesFor, blockedHintLines, viewRecreateNote } from "../lib/allow.js"; import { reportLoadError } from "../lib/load-error.js"; +import { decimalPrecisionWarnings } from "../lib/decimal-precision-advisory.js"; import { scanForReferentialActionConflicts } from "../lib/referential-action-advisory.js"; import type { MetaData } from "@metaobjectsdev/metadata"; import { describeError } from "../lib/error-text.js"; @@ -106,7 +107,7 @@ MIGRATE FLAGS: .metaobjects/config.json. --out-dir Migration directory (default: ./.metaobjects/migrations; flyway: src/main/resources/db/migration) - --slug Required when changes are present (e.g., --slug add-user-shipping) + --slug Required to WRITE a migration (not for --dry-run) (e.g., --slug add-user-shipping) --allow Comma-separated destructive-change permissions: drop-column,drop-table,type-change,drop-index,drop-fk, drop-check,drop-view,drop-view-cascade, @@ -287,6 +288,13 @@ function warnReferentialActionConflicts(metadata: MetaData, collection: Collecti for (const f of findings) log.warn(`migrate: ${f.message}`); } +/** W4 (ADR-0019 Amendment 1): warn once per wide decimal on a dialect that keeps 15 digits. */ +function warnDecimalPrecision(metadata: MetaData, collection: Collection, dialect: string): void { + for (const w of decimalPrecisionWarnings(metadata, dialect, { skip: collection.imported })) { + log.warn(`migrate: ${w}`); + } +} + function emitStructuredError(error: string, hint: string, fmt: OutputFormat): void { const payload = { error, hint }; if (fmt === "json") { @@ -753,6 +761,7 @@ export async function migrateCommand( log.error(`migrate: ${describeError(err)}`); return 2; } + warnDecimalPrecision(metadata, collection, kysely.dialect); let exitCode = 0; let writtenPaths: string[] = []; @@ -933,29 +942,30 @@ export async function migrateCommand( } if (exitCode === 0 && emitted) { - if (config.slug === undefined) { - log.error(`migrate: --slug required when there are changes (e.g., --slug add-user-shipping)`); - emitStructuredError( - "migrate: --slug required when there are changes", - "re-run with --slug ; for a new database whose migrations are already committed, run `meta migrate apply-pending --db `", - fmt, - ); - // The common way to land here is a NEW database (a fresh clone, CI, another - // environment) whose migrations are already committed: the diff against an empty - // database is every table, so it asks to author a new migration. That database - // needs the committed chain replayed, not a new migration. - log.error( - ` New database, migrations already committed? Apply them instead: ` + - `meta migrate apply-pending --db `, - ); - await kysely.close(); - return 2; - } - + notes.push(...(emitted.notices ?? [])); if (config.dryRun) { dryRunSql = { up: emitted.up, down: emitted.down }; if (fmt === "text") log.info(sqlPreview(dryRunSql)); } else { + // A preview names no migration; only a write needs the slug. + if (config.slug === undefined) { + log.error(`migrate: --slug required when there are changes (e.g., --slug add-user-shipping)`); + emitStructuredError( + "migrate: --slug required when there are changes", + "re-run with --slug ; for a new database whose migrations are already committed, run `meta migrate apply-pending --db `", + fmt, + ); + // The common way to land here is a NEW database (a fresh clone, CI, another + // environment) whose migrations are already committed: the diff against an empty + // database is every table, so it asks to author a new migration. That database + // needs the committed chain replayed, not a new migration. + log.error( + ` New database, migrations already committed? Apply them instead: ` + + `meta migrate apply-pending --db `, + ); + await kysely.close(); + return 2; + } const outDir = resolveFormatOutDir(config, metaRoot); await mkdir(outDir, { recursive: true }); // #192 — only the envelope differs; the emitted SQL is identical. @@ -1369,6 +1379,7 @@ export async function runOfflineGenerate( return 1; } warnReferentialActionConflicts(metadata, collection); + warnDecimalPrecision(metadata, collection, config.dialect); const offlineDialect = config.dialect; const offlineViews = buildProjectionViews(metadata, { dialect: offlineDialect }); @@ -1469,11 +1480,6 @@ export async function runOfflineGenerate( finishMigrate(fmt, [`migrate: no changes`], migrateResultToData(offlineResult({}))); return 0; } - if (config.slug === undefined) { - log.error(`migrate: --slug required when there are changes (e.g., --slug add-user-shipping)`); - emitStructuredError("migrate: --slug required when there are changes", "re-run with --slug ", fmt); - return 2; - } const offlineWarnings = warnDataHazards(diffResult.hazards); const emitResult = emit(diffResult.changes, { @@ -1483,6 +1489,9 @@ export async function runOfflineGenerate( ...(snapshot.meta ? { actualMeta: snapshot.meta } : {}), }); + // The emitter's own notices (e.g. a view dropped and recreated, its grants carried over). + offlineNotes.push(...(emitResult.notices ?? [])); + if (config.dryRun) { const sql = { up: emitResult.up, down: emitResult.down }; finishMigrate( @@ -1493,6 +1502,13 @@ export async function runOfflineGenerate( return 0; } + // A preview (above) names no migration; only a write needs the slug. + if (config.slug === undefined) { + log.error(`migrate: --slug required when there are changes (e.g., --slug add-user-shipping)`); + emitStructuredError("migrate: --slug required when there are changes", "re-run with --slug ", fmt); + return 2; + } + // #192 — the MIGRATION FILES follow the active format's layout, while the // snapshot stays in `outDir` (it is metaobjects' own state). With the default // out-dir that separates them: snapshot in .metaobjects/migrations/, migrations @@ -1663,6 +1679,7 @@ async function runD1Migrate( return 1; } warnReferentialActionConflicts(metadata, collection); + warnDecimalPrecision(metadata, collection, "d1"); // 4. Build expected schema + introspect actual. const expectedViews = buildProjectionViews(metadata, { dialect: "d1" }); @@ -1749,12 +1766,6 @@ async function runD1Migrate( return 0; } - if (config.slug === undefined) { - log.error(`migrate: --slug required when there are changes`); - emitStructuredError("migrate: --slug required when there are changes", "re-run with --slug ", fmt); - return 2; - } - // 6. Emit (with D1 safety pass) + write Wrangler migration files. let emitResult; try { @@ -1767,6 +1778,7 @@ async function runD1Migrate( throw err; } + d1Notes.push(...(emitResult.notices ?? [])); const combinedUp = emitResult.up; const combinedDown = emitResult.down; @@ -1780,6 +1792,13 @@ async function runD1Migrate( return 0; } + // A preview (above) names no migration; only a write needs the slug. + if (config.slug === undefined) { + log.error(`migrate: --slug required when there are changes`); + emitStructuredError("migrate: --slug required when there are changes", "re-run with --slug ", fmt); + return 2; + } + const writeResult = await writeMigrationD1( { up: combinedUp, down: combinedDown }, { dir: migrationsDir, slug: config.slug }, diff --git a/server/typescript/packages/cli/src/commands/verify.ts b/server/typescript/packages/cli/src/commands/verify.ts index 827190a81..63accb24e 100644 --- a/server/typescript/packages/cli/src/commands/verify.ts +++ b/server/typescript/packages/cli/src/commands/verify.ts @@ -711,7 +711,7 @@ export async function verifyCommand( // --no-requirement-lint has below, so a CI job can turn it on without editing a // command line it shares with local runs. const scan = scanRequirements(root, { - coverable: collection.inScope, + coverable: collection.inCoverage, requireImplementers: flags.requireImplementers || process.env.META_REQUIRE_IMPLEMENTERS === "1", }); const diags = [...checkRequirements(root, scan)]; diff --git a/server/typescript/packages/cli/src/lib/agent-context-staleness.ts b/server/typescript/packages/cli/src/lib/agent-context-staleness.ts index feb941752..10f785784 100644 --- a/server/typescript/packages/cli/src/lib/agent-context-staleness.ts +++ b/server/typescript/packages/cli/src/lib/agent-context-staleness.ts @@ -6,10 +6,20 @@ import { join } from "node:path"; // star one, so the root barrel now hands out the dependency manifest under this // name. `init.ts` already imports these from the subpath; this was the last site // reading them from the root. -import { AGENT_CONTEXT_MANIFEST_PATH, agentContextStaleness, type Manifest } from "@metaobjectsdev/sdk/agent-context"; +import { AGENT_CONTEXT_MANIFEST_PATH, agentContextStaleness, hashAgentContextRoot, resolveAgentContextRoot, type Manifest } from "@metaobjectsdev/sdk/agent-context"; +import { docsRefForVersion } from "@metaobjectsdev/metadata"; import { cliVersion } from "./version.js"; import { log } from "./log.js"; +/** The installed content's hash, or nothing when the content tree cannot be found (version compare then). */ +export function currentContentHash(): { currentContentHash?: string } { + try { + return { currentContentHash: hashAgentContextRoot(resolveAgentContextRoot(), docsRefForVersion(cliVersion())) }; + } catch { + return {}; + } +} + /** * Advisory: if a scaffolded MetaObjects agent context predates this CLI version, * print a one-line nudge to re-scaffold. Never throws, never blocks — an absent or @@ -28,7 +38,7 @@ export function warnIfAgentContextStale(cwd: string): string | undefined { return undefined; // unreadable/corrupt — say nothing } } - const msg = agentContextStaleness({ manifest, currentVersion: cliVersion() }); + const msg = agentContextStaleness({ manifest, currentVersion: cliVersion(), ...currentContentHash() }); if (msg === null) return undefined; log.warn(msg); return msg; diff --git a/server/typescript/packages/cli/src/lib/decimal-precision-advisory.ts b/server/typescript/packages/cli/src/lib/decimal-precision-advisory.ts new file mode 100644 index 000000000..2a4d60e21 --- /dev/null +++ b/server/typescript/packages/cli/src/lib/decimal-precision-advisory.ts @@ -0,0 +1,50 @@ +// ADR-0019 Amendment 1, W4 — the SQLite and D1 tier has no decimal type. A text value with +// NUMERIC affinity is stored as an INTEGER or REAL, keeping only its first 15 significant +// digits, so a `field.decimal` declaring `@precision` above 15 is stored lossily. +// Advisory, never a gate: `meta migrate` warns once per field, on those dialects only. + +import { + FIELD_ATTR_PRECISION, + FIELD_SUBTYPE_DECIMAL, + OBJECT_SUBTYPE_ENTITY, + TYPE_FIELD, + TYPE_OBJECT, + type MetaData, +} from "@metaobjectsdev/metadata"; + +/** Significant digits a SQLite or D1 NUMERIC column keeps exactly. */ +export const SQLITE_DECIMAL_EXACT_DIGITS = 15; + +const LOSSY_DECIMAL_DIALECTS: readonly string[] = ["sqlite", "d1"]; + +export interface DecimalPrecisionOptions { + /** Skip an entity by FQN — imported (dependency) metadata is not the adopter's to fix. */ + skip?: (fqn: string) => boolean; +} + +/** One message per `field.decimal` with `@precision` above 15, or [] on any other dialect. */ +export function decimalPrecisionWarnings( + root: MetaData, + dialect: string, + opts: DecimalPrecisionOptions = {}, +): string[] { + if (!LOSSY_DECIMAL_DIALECTS.includes(dialect)) return []; + const out: string[] = []; + for (const obj of root.children()) { + if (obj.type !== TYPE_OBJECT || obj.subType !== OBJECT_SUBTYPE_ENTITY || obj.isAbstract) continue; + const fqn = obj.resolutionKey(); + if (opts.skip?.(fqn) === true) continue; + // ADR-0039: resolving — fields (and their @precision) inherited via extends count. + for (const f of obj.children()) { + if (f.type !== TYPE_FIELD || f.subType !== FIELD_SUBTYPE_DECIMAL) continue; + const precision = f.attr(FIELD_ATTR_PRECISION); + if (typeof precision !== "number" || precision <= SQLITE_DECIMAL_EXACT_DIGITS) continue; + out.push( + `${fqn}.${f.name} is a decimal with @precision ${precision}, but ${dialect} keeps only ` + + `${SQLITE_DECIMAL_EXACT_DIGITS} significant digits of a decimal (it stores NUMERIC as an integer or ` + + `a double). Digits beyond the ${SQLITE_DECIMAL_EXACT_DIGITS}th are lost; use Postgres for exact wide decimals.`, + ); + } + } + return out; +} diff --git a/server/typescript/packages/cli/src/lib/docs-removed.ts b/server/typescript/packages/cli/src/lib/docs-removed.ts index c232c647c..923694b19 100644 --- a/server/typescript/packages/cli/src/lib/docs-removed.ts +++ b/server/typescript/packages/cli/src/lib/docs-removed.ts @@ -2,9 +2,10 @@ // project lists in `generators`, and `meta gen` runs them. The docs command, `meta verify // --docs` and the `docs` config block are gone, and each says so in the amendment's fixed // words, pointing at the migration section an agent can apply by hand. +import { guideUrl } from "@metaobjectsdev/metadata/release-docs"; /** The migration guide every message below points at, and the section for this port. */ -export const DOCS_MIGRATION_GUIDE = "https://github.com/metaobjectsdev/metaobjects/blob/main/docs/features/migrations/docs-are-generators.md"; +export const DOCS_MIGRATION_GUIDE = guideUrl("docs/features/migrations/docs-are-generators.md"); const SECTION = `section "TypeScript"`; /** Where a TypeScript project declares a docs generator. */ const WHERE = "in `generators` in metaobjects.config.ts"; diff --git a/server/typescript/packages/cli/src/lib/load-metaobjects-config.ts b/server/typescript/packages/cli/src/lib/load-metaobjects-config.ts index 379507fcb..848b1e4c3 100644 --- a/server/typescript/packages/cli/src/lib/load-metaobjects-config.ts +++ b/server/typescript/packages/cli/src/lib/load-metaobjects-config.ts @@ -1,4 +1,5 @@ import { existsSync, readFileSync, readdirSync, unlinkSync } from "node:fs"; +import { physicalNamingGuideUrl } from "@metaobjectsdev/metadata/release-docs"; import { readFile, writeFile } from "node:fs/promises"; import { createRequire } from "node:module"; import { dirname, join, resolve } from "node:path"; @@ -506,7 +507,7 @@ export async function loadMetaobjectsConfig(projectRoot: string): Promise(); for (const { node, path } of addressed) { - const pkg = node.package ?? node.fileDefaultPackage ?? ""; + const pkg = effectiveRequirementPackage(node); if (pkg !== "") keyed.set(`${pkg}::${path}`, node); // Bare path is registered too, so a single-package ledger can reference // without repeating its own package on every line. @@ -311,7 +312,7 @@ export interface RequirementScan { function projectAuthoredRequirements(addressed: readonly AddressedRequirement[]): boolean { const libPkgs = libraryPackages(); return addressed.some((r) => { - const pkg = r.node.package ?? r.node.fileDefaultPackage ?? ""; + const pkg = effectiveRequirementPackage(r.node); return !libPkgs.has(pkg); }); } @@ -361,7 +362,7 @@ function claimedObjectKeys(root: MetaData, reqs: MetaRequirement[]): Set // referrerPkg is the requirement's own effective package, so a bare ref // binds package-locally under the ADR-0042 contract — the loader's own // resolver, never a parallel name scan (#228). - const referrerPkg = req.package ?? req.fileDefaultPackage ?? ""; + const referrerPkg = effectiveRequirementPackage(req); for (const ref of req.implementedBy()) { const { owner, path } = splitMemberRef(ref); const node = resolveClaimTarget(root, owner, referrerPkg); @@ -479,7 +480,7 @@ export function checkRequirements(root: MetaData, scan: RequirementScan = scanRe // referrerPkg is the requirement's own effective package, so a bare ref // binds package-locally under the ADR-0042 contract — the loader's own // resolver, never a parallel name scan (#228). - const referrerPkg = req.package ?? req.fileDefaultPackage ?? ""; + const referrerPkg = effectiveRequirementPackage(req); const node = resolveClaimTarget(root, owner, referrerPkg); const isObjectRef = path.length === 0; @@ -544,7 +545,7 @@ export function checkRequirements(root: MetaData, scan: RequirementScan = scanRe // requirement's own effective package, exactly as @implementedBy does. const superseded = req.supersededBy(); if (superseded !== undefined) { - const referrerPkg = req.package ?? req.fileDefaultPackage ?? ""; + const referrerPkg = effectiveRequirementPackage(req); if (resolveRequirementRef(addressed, superseded, referrerPkg) === undefined) { out.push({ severity: "error", code: ERR_REQUIREMENT_DANGLING_REF, path: reqPath, diff --git a/server/typescript/packages/cli/test/eject-template-handover.test.ts b/server/typescript/packages/cli/test/eject-template-handover.test.ts index 95e1fa29d..f0a3ea2d6 100644 --- a/server/typescript/packages/cli/test/eject-template-handover.test.ts +++ b/server/typescript/packages/cli/test/eject-template-handover.test.ts @@ -3,7 +3,7 @@ // install line asks for. The mechanism is template-eject.test.ts; what it achieves for every // generator is eject-templates-contract.test.ts. import { afterEach, beforeEach, describe, expect, test } from "bun:test"; -import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { appendFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { ejectCommand, ejectGenerator } from "../src/commands/eject.js"; @@ -131,8 +131,46 @@ describe("meta eject --list reports the copies a project owns", () => { expect(text).toContain("codegen/templates/names-decl.ts [DIFFERS:"); expect(text).toContain("codegen/templates/extra.ts [not in the package — yours, or removed upstream]"); expect(text).toContain("diff -u node_modules/@metaobjectsdev/codegen-ts/src/templates/names-decl.ts codegen/templates/names-decl.ts"); - // The generator copy is compared too, against the owned form of the reference. - expect(text).toContain("names [owned — identical to the reference]"); + // The generator file itself is untouched, but the generator is not "identical": one of + // the renderers it owns differs, and the row says which. + expect(text).not.toContain("names [owned — identical to the reference]"); + expect(text).toContain("names [owned — DIFFERS: generator identical, 1 renderer(s) differ]"); + expect(text).toContain("renderer codegen/templates/names-decl.ts [DIFFERS:"); + }); + + test("a generator is identical only when its file AND every renderer it owns are (text and JSON)", async () => { + await ejectCommand(["names"], cwd, "text"); + logged = []; + await ejectCommand(["--list"], cwd, "text"); + expect(logged.join("\n")).toContain("names [owned — identical to the reference]"); + + // A stale renderer, the generator file untouched: the E1 repro. + appendFileSync(join(cwd, "codegen/templates/names-decl.ts"), "\n// stale line from an older release\n"); + logged = []; + await ejectCommand(["--list"], cwd, "text"); + const text = logged.join("\n"); + expect(text).not.toContain("names [owned — identical to the reference]"); + expect(text).toContain("names [owned — DIFFERS: generator identical, 1 renderer(s) differ]"); + expect(text).toContain("renderer codegen/templates/names-decl.ts [DIFFERS: 0 line(s) behind, 1 line(s) of your own]"); + + logged = []; + await ejectCommand(["--list"], cwd, "json"); + const doc = JSON.parse(logged.join("\n")) as { + generators: { name: string; status: string; templates?: { path: string; status: string; own: number }[] }[]; + }; + const row = doc.generators.find((g) => g.name === "names")!; + expect(row.status).toBe("differs"); + expect(row.templates!.find((t) => t.path === "codegen/templates/names-decl.ts")).toMatchObject({ status: "differs", own: 1 }); + }); + + test("a renderer another generator owns does not flag this one", async () => { + await ejectCommand(["names"], cwd, "text"); + await ejectCommand(["barrel"], cwd, "text"); + appendFileSync(join(cwd, "codegen/templates/names-decl.ts"), "\n// mine\n"); + logged = []; + await ejectCommand(["--list"], cwd, "text"); + const text = logged.join("\n"); + expect(text).toContain("barrel [owned — identical to the reference]"); }); test("a project with no template copy prints no section for them", async () => { diff --git a/server/typescript/packages/cli/test/gen-docs-generators.test.ts b/server/typescript/packages/cli/test/gen-docs-generators.test.ts index 96277512f..2e775d2b7 100644 --- a/server/typescript/packages/cli/test/gen-docs-generators.test.ts +++ b/server/typescript/packages/cli/test/gen-docs-generators.test.ts @@ -5,6 +5,9 @@ // // Driven through the real CLI entry (`run`), against small projects written into the // workspace so a config resolves `@metaobjectsdev/*` the way a project's does. +import { guideUrl } from "@metaobjectsdev/metadata/release-docs"; +// The guide links name the tag of the running release (a prerelease keeps main). +const BLOB = guideUrl(""); import { describe, test, expect, beforeEach, afterEach } from "bun:test"; import { existsSync, mkdtempSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs"; import { join, relative, resolve } from "node:path"; @@ -122,7 +125,7 @@ describe("the removed docs commands say where they went (ADR-0034 Amendment 9)", expect(stderr()).toContain(REMOVED_DOCS_COMMAND); expect(REMOVED_DOCS_COMMAND).toBe( "`meta docs` was removed in 1.1.0. `meta gen` writes documentation now: declare each docs generator " + - "in `generators` in metaobjects.config.ts. See https://github.com/metaobjectsdev/metaobjects/blob/main/docs/features/migrations/docs-are-generators.md, " + + "in `generators` in metaobjects.config.ts. See " + BLOB + "docs/features/migrations/docs-are-generators.md, " + 'section "TypeScript".', ); expect(existsSync(join(root, "docs"))).toBe(false); @@ -134,7 +137,7 @@ describe("the removed docs commands say where they went (ADR-0034 Amendment 9)", expect(stderr()).toContain(REMOVED_VERIFY_DOCS); expect(REMOVED_VERIFY_DOCS).toBe( "`meta verify --docs` was removed in 1.1.0. `meta verify --codegen` checks documentation now, " + - "with everything else `generators` declares. See https://github.com/metaobjectsdev/metaobjects/blob/main/docs/features/migrations/docs-are-generators.md, " + + "with everything else `generators` declares. See " + BLOB + "docs/features/migrations/docs-are-generators.md, " + 'section "TypeScript".', ); }); @@ -158,7 +161,7 @@ describe("meta gen refuses a config it cannot read the list from", () => { const before = filesUnder(root); expect(await run(["gen", "--cwd", root])).toBe(2); expect(stderr()).toContain(NO_CONFIG_MESSAGE); - expect(NO_CONFIG_MESSAGE).toContain('A 1.0 project that ran `meta docs` with no config: see https://github.com/metaobjectsdev/metaobjects/blob/main/docs/features/migrations/docs-are-generators.md, section "TypeScript".'); + expect(NO_CONFIG_MESSAGE).toContain('A 1.0 project that ran `meta docs` with no config: see ' + BLOB + 'docs/features/migrations/docs-are-generators.md, section "TypeScript".'); expect(filesUnder(root)).toEqual(before); }); @@ -170,7 +173,7 @@ describe("meta gen refuses a config it cannot read the list from", () => { expect(legacyDocsConfigMessage("meta gen")).toBe( "meta gen: found 1.0 docs configuration (the docs block in metaobjects.config.ts). Nothing reads it any " + "more, so this run would write no documentation. Declare each docs generator in `generators` in " + - 'metaobjects.config.ts. See https://github.com/metaobjectsdev/metaobjects/blob/main/docs/features/migrations/docs-are-generators.md, section "TypeScript".', + 'metaobjects.config.ts. See ' + BLOB + 'docs/features/migrations/docs-are-generators.md, section "TypeScript".', ); expect(filesUnder(root)).toEqual(before); }); diff --git a/server/typescript/packages/cli/test/migrate-decimal-precision.test.ts b/server/typescript/packages/cli/test/migrate-decimal-precision.test.ts new file mode 100644 index 000000000..63e9dbdea --- /dev/null +++ b/server/typescript/packages/cli/test/migrate-decimal-precision.test.ts @@ -0,0 +1,71 @@ +// ADR-0019 Amendment 1, W4: `meta migrate` warns once per wide decimal on SQLite and D1. +import { describe, test, expect, afterAll } from "bun:test"; +import { mkdtemp, rm, mkdir, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { runBaseline, runOfflineGenerate } from "../src/commands/migrate.js"; +import { decimalPrecisionWarnings } from "../src/lib/decimal-precision-advisory.js"; +import { loadMemory } from "@metaobjectsdev/sdk"; + +const dirs: string[] = []; +afterAll(async () => { for (const d of dirs) await rm(d, { recursive: true, force: true }); }); + +const MODEL = { + "metadata.root": { + children: [{ + "object.entity": { + name: "Ledger", + children: [ + { "field.long": { name: "id" } }, + { "field.decimal": { name: "wide", "@precision": 20, "@scale": 4 } }, + { "field.decimal": { name: "narrow", "@precision": 15, "@scale": 2 } }, + { "field.decimal": { name: "bare" } }, + { "source.rdb": { name: "src", "@table": "ledgers" } }, + { "identity.primary": { name: "pk", "@fields": ["id"], "@generation": "increment" } }, + ], + }, + }], + }, +}; + +async function project(): Promise { + const root = await mkdtemp(join(tmpdir(), "mts-dec-")); + dirs.push(root); + await mkdir(join(root, "metaobjects"), { recursive: true }); + await writeFile(join(root, "metaobjects", "meta.ledger.json"), JSON.stringify(MODEL), "utf8"); + return root; +} + +describe("decimalPrecisionWarnings", () => { + test("warns once per field above 15 on sqlite and d1, never on postgres", async () => { + const root = await project(); + const md = await loadMemory(root); + for (const d of ["sqlite", "d1"]) { + const w = decimalPrecisionWarnings(md, d); + expect(w).toHaveLength(1); + expect(w[0]).toContain("Ledger.wide"); + expect(w[0]).toContain("15 significant digits"); + } + expect(decimalPrecisionWarnings(md, "postgres")).toEqual([]); + }); +}); + +describe("meta migrate prints the W4 warning", () => { + test("offline generate on sqlite warns about the wide decimal only", async () => { + const root = await project(); + const cfg = { dialect: "sqlite", outDir: "./.metaobjects/migrations", onAmbiguous: "abort", allow: [], slug: "auto", dryRun: true } as any; + const lines: string[] = []; + const orig = console.error; + console.error = (...a: unknown[]) => { lines.push(a.map(String).join(" ")); }; + try { + await runBaseline({ ...cfg, dryRun: false }, root); + await runOfflineGenerate(cfg, root); + } finally { + console.error = orig; + } + const w4 = lines.filter((l) => l.includes("15 significant digits")); + expect(w4.length).toBeGreaterThanOrEqual(1); + expect(w4.every((l) => l.includes("Ledger.wide"))).toBe(true); + expect(lines.some((l) => l.includes("Ledger.narrow") || l.includes("Ledger.bare"))).toBe(false); + }); +}); diff --git a/server/typescript/packages/cli/test/migrate-dry-run-no-slug.test.ts b/server/typescript/packages/cli/test/migrate-dry-run-no-slug.test.ts new file mode 100644 index 000000000..c30ee4804 --- /dev/null +++ b/server/typescript/packages/cli/test/migrate-dry-run-no-slug.test.ts @@ -0,0 +1,80 @@ +/** + * A `--dry-run` preview names no migration: `--slug` is required to WRITE one, not to + * look at the SQL. The check used to sit before the dry-run branch, so a preview of a + * pending change failed with "--slug required". + */ +import { describe, test, expect, afterAll, beforeEach, afterEach, spyOn } from "bun:test"; +import { mkdtemp, rm, mkdir, writeFile, readdir } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { migrateCommand } from "../src/commands/migrate.js"; +import { log } from "../src/lib/log.js"; +import { snapshotPath, writeSnapshot } from "@metaobjectsdev/migrate-ts"; + +const dirs: string[] = []; +afterAll(async () => { for (const d of dirs) await rm(d, { recursive: true, force: true }); }); + +const MODEL = JSON.stringify({ + "metadata.root": { + package: "acme::platform", + children: [{ + "object.entity": { + name: "Job", + children: [ + { "source.rdb": { name: "src", "@table": "jobs" } }, + { "field.long": { name: "id" } }, + { "identity.primary": { name: "pk", "@fields": ["id"], "@generation": "increment" } }, + ], + }, + }], + }, +}); + +async function project(): Promise { + const root = await mkdtemp(join(tmpdir(), "dry-run-slug-")); + dirs.push(root); + await mkdir(join(root, "metaobjects"), { recursive: true }); + await writeFile(join(root, "metaobjects", "meta.platform.json"), MODEL, "utf8"); + await mkdir(join(root, ".metaobjects", "migrations"), { recursive: true }); + await writeFile(join(root, ".metaobjects", "config.json"), JSON.stringify({ schema_version: 1, migrate: { dialect: "sqlite" } }), "utf8"); + return root; +} + +const files = async (root: string) => + (await readdir(join(root, ".metaobjects", "migrations"))).filter((e) => !e.startsWith(".")); + +let out: string[] = []; +let infoSpy: ReturnType; +let errSpy: ReturnType; +beforeEach(() => { + out = []; + infoSpy = spyOn(log, "info").mockImplementation((m: string) => { out.push(String(m)); }); + errSpy = spyOn(log, "error").mockImplementation((m: string) => { out.push(String(m)); }); +}); +afterEach(() => { infoSpy.mockRestore(); errSpy.mockRestore(); }); + +describe("meta migrate --dry-run needs no --slug", () => { + test("live (--from-db): previews the SQL, exits 0, writes nothing", async () => { + const root = await project(); + expect(await migrateCommand(["--from-db", "--db", `file:${join(root, "t.db")}`, "--dialect", "sqlite", "--dry-run"], root)).toBe(0); + expect(out.join("\n")).toContain("CREATE TABLE"); + expect(out.join("\n")).not.toContain("--slug"); + expect(await files(root)).toEqual([]); + }); + + test("offline (against the committed snapshot): previews, exits 0, writes nothing", async () => { + const root = await project(); + await writeSnapshot(snapshotPath(join(root, ".metaobjects", "migrations"), "sqlite"), { tables: [], views: [] }); + expect(await migrateCommand(["--dialect", "sqlite", "--dry-run"], root)).toBe(0); + expect(out.join("\n")).toContain("CREATE TABLE"); + expect(await files(root)).toEqual([]); + expect(await migrateCommand(["--dialect", "sqlite"], root)).toBe(2); + }); + + test("the same change WITHOUT --dry-run still requires --slug", async () => { + const root = await project(); + expect(await migrateCommand(["--from-db", "--db", `file:${join(root, "t.db")}`, "--dialect", "sqlite"], root)).toBe(2); + expect(out.join("\n")).toContain("--slug required"); + expect(await files(root)).toEqual([]); + }); +}); diff --git a/server/typescript/packages/cli/test/migrate-emit-notices.test.ts b/server/typescript/packages/cli/test/migrate-emit-notices.test.ts new file mode 100644 index 000000000..b219ba92e --- /dev/null +++ b/server/typescript/packages/cli/test/migrate-emit-notices.test.ts @@ -0,0 +1,70 @@ +// The emitter's notices (EmitResult.notices) reach `meta migrate` output, dry run and write. +import { describe, test, expect, afterAll } from "bun:test"; +import { mkdtemp, rm, mkdir, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { runBaseline, runOfflineGenerate } from "../src/commands/migrate.js"; + +const dirs: string[] = []; +afterAll(async () => { for (const d of dirs) await rm(d, { recursive: true, force: true }); }); + +const model = (viewFields: string[]) => ({ + "metadata.root": { + children: [ + { + "object.entity": { + name: "Thing", + children: [ + { "field.long": { name: "id" } }, + { "field.string": { name: "name" } }, + { "field.string": { name: "note" } }, + { "source.rdb": { name: "src", "@table": "things" } }, + { "identity.primary": { name: "pk", "@fields": ["id"], "@generation": "increment" } }, + ], + }, + }, + { + "object.projection": { + name: "ThingView", + children: [ + { "source.rdb": { "@kind": "view", "@table": "v_things" } }, + { "field.long": { name: "id", extends: "Thing.id" } }, + { "identity.primary": { name: "pk", extends: "Thing.pk" } }, + ...viewFields.map((f) => ({ "field.string": { name: f, extends: `Thing.${f}` } })), + ], + }, + }, + ], + }, +}); + +async function write(root: string, viewFields: string[]): Promise { + await writeFile(join(root, "metaobjects", "meta.things.json"), JSON.stringify(model(viewFields)), "utf8"); +} + +const cfg = (dryRun: boolean) => + ({ dialect: "postgres", outDir: "./.metaobjects/migrations", onAmbiguous: "abort", allow: [], slug: "recreate", dryRun } as any); + +async function capture(fn: () => Promise): Promise { + const lines: string[] = []; + const oe = console.error, ol = console.log; + console.error = (...a: unknown[]) => { lines.push(a.map(String).join(" ")); }; + console.log = (...a: unknown[]) => { lines.push(a.map(String).join(" ")); }; + try { await fn(); } finally { console.error = oe; console.log = ol; } + return lines.join("\n"); +} + +describe("meta migrate prints the emitter's notices", () => { + for (const dryRun of [true, false]) { + test(`offline ${dryRun ? "dry run" : "write"} says the recreated view's grants are carried over`, async () => { + const root = await mkdtemp(join(tmpdir(), "mts-notice-")); + dirs.push(root); + await mkdir(join(root, "metaobjects"), { recursive: true }); + await write(root, ["name"]); + await runBaseline(cfg(false), root); + await write(root, ["note"]); + const out = await capture(() => runOfflineGenerate(cfg(dryRun), root)); + expect(out).toContain("dropped and recreated; its grants are carried over"); + }); + } +}); diff --git a/server/typescript/packages/cli/test/migrate-load-failure-exit.test.ts b/server/typescript/packages/cli/test/migrate-load-failure-exit.test.ts new file mode 100644 index 000000000..c1ef93aae --- /dev/null +++ b/server/typescript/packages/cli/test/migrate-load-failure-exit.test.ts @@ -0,0 +1,34 @@ +// docs/features/cli.md: every command exits 1 on a metadata load failure (2 is a usage error). +import { describe, test, expect, afterAll } from "bun:test"; +import { mkdtemp, rm, mkdir, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { runBaseline, runOfflineGenerate } from "../src/commands/migrate.js"; +import { run } from "../src/index.js"; + +const dirs: string[] = []; +afterAll(async () => { for (const d of dirs) await rm(d, { recursive: true, force: true }); }); + +async function brokenProject(): Promise { + const root = await mkdtemp(join(tmpdir(), "mts-loadfail-")); + dirs.push(root); + await mkdir(join(root, "metaobjects"), { recursive: true }); + await writeFile(join(root, "metaobjects", "meta.orders.json"), "{ not json", "utf8"); + return root; +} +const cfg = () => + ({ dialect: "sqlite", outDir: "./.metaobjects/migrations", onAmbiguous: "abort", allow: [], slug: "auto", dryRun: false } as any); + +describe("meta migrate on metadata that does not load", () => { + test("offline generate exits 1", async () => { + expect(await runOfflineGenerate(cfg(), await brokenProject())).toBe(1); + }); + test("baseline exits 1", async () => { + expect(await runBaseline(cfg(), await brokenProject())).toBe(1); + }); + test("the CLI entry exits 1 (offline and --db)", async () => { + const root = await brokenProject(); + expect(await run(["migrate", "--dialect", "sqlite", "--cwd", root])).toBe(1); + expect(await run(["migrate", "--dialect", "sqlite", "--db", `file:${join(root, "x.sqlite")}`, "--cwd", root])).toBe(1); + }); +}); diff --git a/server/typescript/packages/cli/test/verify-requirements-scope-no-deps.test.ts b/server/typescript/packages/cli/test/verify-requirements-scope-no-deps.test.ts index 148331cf4..2b90ea131 100644 --- a/server/typescript/packages/cli/test/verify-requirements-scope-no-deps.test.ts +++ b/server/typescript/packages/cli/test/verify-requirements-scope-no-deps.test.ts @@ -1,12 +1,10 @@ -// FR-023 §11.1 item 2 (2026-09-11 qualification) — `Collection.inScope` is -// `matchesScope(fqn, scope) && (...)`, so declaring `scope.include` narrows -// the requirements ledger's denominator even for a project with ZERO -// dependencies — a real behaviour change the CHANGELOG records ("A project -// that declares scope.include now sees its requirements-ledger denominator -// narrow to that scope — even with zero dependencies"), and nothing exercised -// it: `verify-requirements-imported.test.ts` only covers the WITH-a- -// dependency case (the import-exclusion half of the predicate); this covers -// the other half — `matchesScope` alone, no dependency involved at all. +// Requirement coverage and `scope.include` (1.1.0 release review): coverage +// counts every non-abstract project entity except one imported from a +// dependency, and `scope` filters generated output only — it does not narrow +// the ledger's denominator. FR-023 §11.1 item 2 once narrowed it by scope even +// with zero dependencies; this pins the ruling that it does not, the half of +// the predicate `verify-requirements-imported.test.ts` (the dependency-import +// exclusion) does not reach. import { describe, test, expect, beforeEach, afterEach, afterAll } from "bun:test"; import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; @@ -108,7 +106,7 @@ afterEach(() => { console.error = origErr; }); -describe("meta verify — scope.include narrows the ledger denominator with ZERO dependencies (FR-023)", () => { +describe("meta verify — scope.include does not narrow the ledger denominator (scope is an output filter)", () => { test("no scope declared: both entities are coverable, only one is claimed", async () => { const root = project(); const exit = await run(["verify", "--format", "text", "--cwd", root]); @@ -119,16 +117,15 @@ describe("meta verify — scope.include narrows the ledger denominator with ZERO expect(all).toContain("other::Thing"); }); - test("scope.include naming only app:: narrows the denominator to 1/1 — other::Thing drops out entirely", async () => { + test("scope.include does NOT narrow coverage: scope is an output filter, only the dependency-import rule applies", async () => { const root = project({ scopeInclude: ["app::**"] }); const exit = await run(["verify", "--format", "text", "--cwd", root]); expect(exit).toBe(0); const all = [...out, ...err].join("\n"); - // Denominator narrows to the declared scope alone — other::Thing is - // excluded from the COUNT, not merely reported as unclaimed. - expect(all).toContain("1/1 entities claimed"); - expect(all).not.toContain("other::Thing"); - expect(all).not.toContain("WARN_REQUIREMENT_OBJECT_UNCLAIMED"); + // Coverage counts every own entity regardless of the project's `scope` + // patterns (T18 ruling, matches the Python port). + expect(all).toContain("1/2 entities claimed"); + expect(all).toContain("other::Thing"); }); }); diff --git a/server/typescript/packages/codegen-ts-react/src/templates/form-file.ts b/server/typescript/packages/codegen-ts-react/src/templates/form-file.ts index 3f6663526..c074d0ddf 100644 --- a/server/typescript/packages/codegen-ts-react/src/templates/form-file.ts +++ b/server/typescript/packages/codegen-ts-react/src/templates/form-file.ts @@ -31,12 +31,6 @@ import { IDENTITY_ATTR_FIELDS, FIELD_ATTR_DEFAULT, FIELD_SUBTYPE_OBJECT, - FIELD_SUBTYPE_INT, - FIELD_SUBTYPE_LONG, - FIELD_SUBTYPE_DOUBLE, - FIELD_SUBTYPE_FLOAT, - FIELD_SUBTYPE_DECIMAL, - FIELD_SUBTYPE_CURRENCY, FIELD_SUBTYPE_BOOLEAN, FIELD_SUBTYPE_DATE, FIELD_SUBTYPE_TIME, @@ -67,7 +61,7 @@ import { VIEW_CONTEXT_FORM, effectivePackage, } from "@metaobjectsdev/codegen-ts/engine"; -import { humanize, valueObjectFor } from "@metaobjectsdev/codegen-ts/templates/field-meta"; +import { humanize, valueObjectFor, numericInputControl } from "@metaobjectsdev/codegen-ts/templates/field-meta"; import { tphDiscriminatorPin } from "@metaobjectsdev/codegen-ts/engine"; function primaryFieldNames(entity: MetaObject): Set { @@ -131,13 +125,6 @@ function resolveValueObject(field: MetaField, ctx: RenderContext): MetaObject | /** A real HTML for a scalar field subtype, or undefined (text). */ function htmlTypeForSubType(subType: string): string | undefined { switch (subType) { - case FIELD_SUBTYPE_INT: - case FIELD_SUBTYPE_LONG: - case FIELD_SUBTYPE_DOUBLE: - case FIELD_SUBTYPE_FLOAT: - case FIELD_SUBTYPE_DECIMAL: - case FIELD_SUBTYPE_CURRENCY: - return "number"; case FIELD_SUBTYPE_BOOLEAN: return "checkbox"; case FIELD_SUBTYPE_DATE: @@ -189,8 +176,12 @@ function renderNestedField( const vo = resolveValueObject(field, ctx); if (vo === undefined) { // Scalar / enum / unresolved object → a single bound input. - const htmlType = htmlTypeForSubType(field.subType); - const typeAttr = htmlType !== undefined ? ` type=${JSON.stringify(htmlType)}` : ""; + const numeric = numericInputControl(field.subType); + const htmlType = numeric?.htmlType ?? htmlTypeForSubType(field.subType); + const typeAttr = + (htmlType !== undefined ? ` type=${JSON.stringify(htmlType)}` : "") + + (numeric?.step !== undefined ? ` step=${JSON.stringify(numeric.step)}` : "") + + (numeric?.inputMode !== undefined ? ` inputMode=${JSON.stringify(numeric.inputMode)}` : ""); return { jsx: `
    diff --git a/server/typescript/packages/codegen-ts-react/test/nested-value-object-form.test.ts b/server/typescript/packages/codegen-ts-react/test/nested-value-object-form.test.ts index f66b548c0..4d860f52c 100644 --- a/server/typescript/packages/codegen-ts-react/test/nested-value-object-form.test.ts +++ b/server/typescript/packages/codegen-ts-react/test/nested-value-object-form.test.ts @@ -31,6 +31,8 @@ async function loadModel(): Promise<{ root: MetaRoot; agent: MetaObject }> { children: [ { "field.string": { name: "model", "@required": true } }, { "field.double": { name: "temperature" } }, + { "field.decimal": { name: "budget", "@precision": 12, "@scale": 2 } }, + { "field.int": { name: "retries" } }, ], }, }, @@ -98,6 +100,16 @@ describe("issue #95 — nested value-object sub-forms", () => { expect(out).toContain('form.register("llmConfig.temperature")'); }); + test("fractional sub-fields are submittable: double step=any, decimal text/decimal, int unchanged", async () => { + const { root, agent } = await loadModel(); + const out = renderFormFile(agent, ctxFor(root)); + const line = (name: string) => out.match(new RegExp(`]*register\\("llmConfig\\.${name}"\\)[^>]*>`))?.[0].replace(/\s+/g, " ") ?? ""; + expect(line("temperature")).toContain('type="number" step="any"'); + expect(line("budget")).toContain('type="text" inputMode="decimal"'); + expect(line("retries")).toContain('type="number"'); + expect(line("retries")).not.toContain("step"); + }); + test("an array of value objects renders a useFieldArray repeatable group", async () => { const { root, agent } = await loadModel(); const out = renderFormFile(agent, ctxFor(root)); diff --git a/server/typescript/packages/codegen-ts/src/api-surface.ts b/server/typescript/packages/codegen-ts/src/api-surface.ts index 1875c9406..c2a918a56 100644 --- a/server/typescript/packages/codegen-ts/src/api-surface.ts +++ b/server/typescript/packages/codegen-ts/src/api-surface.ts @@ -113,21 +113,30 @@ export function servesClientTier(entity: MetaObject): boolean { } /** - * The decimal fields a report's read-only mount must send as strings: a report on SQLite. + * The names of an object's `field.decimal` fields, which its generated routes send as canonical + * strings (ADR-0019 Amendment 1, W1 and W4). * - * A decimal is a string on the wire. SQLite has no decimal, so the REAL a report's view - * computes (a ratio, an average, a sum of a decimal field) reaches the route as a JS number - * where Postgres and MySQL return a string, and the read schema says string for all of them. - * The generated route passes these names to the mount as `decimalColumns` and the mount sends - * a number under one of them as its string. Empty for every other object and dialect, which - * keeps their routes byte-identical. A projection on SQLite is not covered: it is released - * behaviour and its decimals keep reaching the wire as numbers. + * A decimal is a canonical string on the wire on EVERY dialect and for every kind of object (an + * entity, a projection, a report): Postgres `numeric` and MySQL `DECIMAL` hand back the column's + * scale (`"100.5000"`), and SQLite hands back a JS number (the REAL a report's view computes, or + * the number a NUMERIC-affinity column stores). The generated route passes these names to the + * mount as `decimalColumns` and the mount rewrites each row under them. Which keys are decimals + * comes from the model, never from the shape of a value. Empty for an object with no decimal, + * which keeps its route byte-identical. An array of decimals is not a decimal column. */ -export function reportDecimalColumns(entity: MetaObject, dialect: string): readonly string[] { - if (dialect !== "sqlite" || !isReport(entity)) return []; - // ADR-0039: resolving. A report read model's fields are its own, but the resolving call is - // the rule everywhere a field set is read. - return entity.fields().filter((f) => f.subType === FIELD_SUBTYPE_DECIMAL).map((f) => f.name); +export function decimalFieldNames(entity: MetaObject): readonly string[] { + // ADR-0039: resolving. A decimal may be inherited via extends, and a report read model's + // derived ratio, average or sum is a field of the report itself. + return entity.fields().filter((f) => f.subType === FIELD_SUBTYPE_DECIMAL && !f.resolvedIsArray()).map((f) => f.name); +} + +/** `\ndecimalColumns: [...],` for a mount option list, or "" when the object has no decimal. + * Several objects share one mount when a single-table (TPH) base's polymorphic route serves its + * subtypes' rows too: pass them all, so a decimal only a subtype declares is still a decimal. */ +export function decimalColumnsLine(entity: MetaObject | readonly MetaObject[], indent: string): string { + const objects: readonly MetaObject[] = Array.isArray(entity) ? entity : [entity as MetaObject]; + const names = [...new Set(objects.flatMap((o) => decimalFieldNames(o)))]; + return names.length > 0 ? `\n${indent}decimalColumns: ${JSON.stringify(names)},` : ""; } /** diff --git a/server/typescript/packages/codegen-ts/src/cube/build-cube-model.ts b/server/typescript/packages/codegen-ts/src/cube/build-cube-model.ts index 4c03a1a05..cd7b4a1b7 100644 --- a/server/typescript/packages/codegen-ts/src/cube/build-cube-model.ts +++ b/server/typescript/packages/codegen-ts/src/cube/build-cube-model.ts @@ -737,12 +737,12 @@ class CubeModelBuilder { const draft = this.reportCube(report, shape, "rollup"); if (draft === undefined) return; const key = report.resolutionKey(); - const { scope, rollup } = reportContribution(shape, draft.name, this.mc); + const { scope, rollups } = reportContribution(shape, draft.name, this.mc); if (scope !== undefined) { draft.namespace.add(scope.name, `the segment '${scope.name}' the exporter adds for the @filter of report '${key}'`); draft.segments.push(scope); } - if (rollup !== undefined) { + for (const rollup of rollups ?? []) { // Cube reports a pre-aggregation named like a member as "defined more than once". draft.namespace.add(rollup.name, `the rollup '${rollup.name}' of report '${key}'`); draft.preAggregations.push(rollup); diff --git a/server/typescript/packages/codegen-ts/src/cube/cube-members.ts b/server/typescript/packages/codegen-ts/src/cube/cube-members.ts index e61f3b4f1..439c97684 100644 --- a/server/typescript/packages/codegen-ts/src/cube/cube-members.ts +++ b/server/typescript/packages/codegen-ts/src/cube/cube-members.ts @@ -22,7 +22,6 @@ import { FIELD_SUBTYPE_INT, FIELD_SUBTYPE_LONG, FIELD_SUBTYPE_MAP, - FIELD_SUBTYPE_OBJECT, FIELD_SUBTYPE_STRING, FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, @@ -115,11 +114,9 @@ export function dimensionColumn( ); // ADR-0039: resolving isArray and @objectRef, so an inherited one is seen. if (field.resolvedIsArray()) throw unmappable(`field.${field.subType}, isArray`); - const objectRef = field.objectRef; - if (objectRef !== undefined) throw unmappable(`field.${field.subType} @objectRef '${objectRef}'`); - if (field.subType === FIELD_SUBTYPE_OBJECT || field.subType === FIELD_SUBTYPE_MAP) { - throw unmappable(`field.${field.subType}`); - } + // A field.object, or a field with an @objectRef, never reaches here: the loader refuses it as a + // report dimension (D5). A field.map is a valid dimension that Cube cannot hold. + if (field.subType === FIELD_SUBTYPE_MAP) throw unmappable(`field.${field.subType}`); const d = mc.dialect; const col = cubeColumn(sourceColumnNameFor(field, mc.extract), d, renderer); if (field.subType === FIELD_SUBTYPE_ENUM) { diff --git a/server/typescript/packages/codegen-ts/src/cube/cube-reports.ts b/server/typescript/packages/codegen-ts/src/cube/cube-reports.ts index 14b22009f..1e5ece24d 100644 --- a/server/typescript/packages/codegen-ts/src/cube/cube-reports.ts +++ b/server/typescript/packages/codegen-ts/src/cube/cube-reports.ts @@ -29,8 +29,14 @@ export interface ReportContribution { * The rollup. Absent when the report's `@filter`, its `@segment`, or the condition of any * measure it lists holds a relative date: a rollup is built at refresh time, so it would * answer with the "now" of its build, where the view answers with the query's. + * + * One rollup, named after the report, unless the report lists a time dimension at SEVERAL + * grains: a Cube query names a time dimension once, at one granularity, so a rollup repeating + * the dimension would match no query. Then there is one rollup per choice of grain for each + * such dimension, named `_[_]`, each a single-granularity rollup a Cube + * query at that grain can match. */ - readonly rollup?: CubeRollupSpec; + readonly rollups?: readonly CubeRollupSpec[]; } /** How many grouping columns a rollup has: its attribute dimensions plus its time dimensions. */ @@ -150,13 +156,27 @@ export function reportContribution(shape: ReportShape, cube: string, mc: MemberC if (conditions.some(hasRelativeDate)) return scope === undefined ? {} : { scope }; const segments = [...(segmentName === undefined ? [] : [segmentName]), ...(scope === undefined ? [] : [scope.name])]; - const base = { name: report.name, type: "rollup" as const, measures, dimensions, segments }; - // One time dimension takes Cube's documented form; two or more, the list form (Table F). - const rollup: CubeRollupSpec = - times.length === 0 - ? base - : times.length === 1 - ? { ...base, timeDimension: times[0]!.dimension, granularity: times[0]!.granularity } - : { ...base, timeDimensions: times }; - return { ...(scope === undefined ? {} : { scope }), rollup }; + const base = { type: "rollup" as const, measures, dimensions, segments }; + // The time dimensions in first-listed order, each with the grains the report lists for it. + const grainsOf = new Map(); + for (const t of times) grainsOf.set(t.dimension, [...(grainsOf.get(t.dimension) ?? []), t.granularity]); + // One choice of grain per time dimension: the cartesian product of the grains, a dimension + // listed at one grain contributing that grain alone. + let choices: TimeGrain[][] = [[]]; + for (const grains of grainsOf.values()) choices = choices.flatMap((c) => grains.map((g) => [...c, g])); + const several = [...grainsOf.values()].some((g) => g.length > 1); + const dims = [...grainsOf.keys()]; + const rollups = choices.map((choice): CubeRollupSpec => { + // The grains of the dimensions that have several, in listed order, name the rollup. + const suffix = dims.flatMap((d, i) => ((grainsOf.get(d) ?? []).length > 1 ? [choice[i] as string] : [])); + const name = several ? `${report.name}_${suffix.join("_")}` : report.name; + const entries: CubeRollupTimeDimension[] = dims.map((d, i) => ({ dimension: d, granularity: choice[i] as TimeGrain })); + // One time dimension takes Cube's documented form; two or more, the list form (Table F). + return entries.length === 0 + ? { ...base, name } + : entries.length === 1 + ? { ...base, name, timeDimension: entries[0]!.dimension, granularity: entries[0]!.granularity } + : { ...base, name, timeDimensions: entries }; + }); + return { ...(scope === undefined ? {} : { scope }), rollups }; } diff --git a/server/typescript/packages/codegen-ts/src/generators/agent-docs-file.ts b/server/typescript/packages/codegen-ts/src/generators/agent-docs-file.ts index 63ea1610f..c92c870d2 100644 --- a/server/typescript/packages/codegen-ts/src/generators/agent-docs-file.ts +++ b/server/typescript/packages/codegen-ts/src/generators/agent-docs-file.ts @@ -36,6 +36,7 @@ // These files are READ, never imported. There is no three-way merge and no hand-edit // preservation to think about: regenerate and the page is current. +import { physicalNamingGuideUrl } from "@metaobjectsdev/metadata/release-docs"; import { CARDINALITY_MANY, CARDINALITY_ONE, @@ -214,7 +215,7 @@ export const agentDocsFile = function agentDocsFile(opts?: AgentDocsFileOpts): G // option is refused rather than dropped: its value may be the one matching the database. const removedNaming = (opts as { columnNamingStrategy?: unknown } | undefined)?.columnNamingStrategy; if (removedNaming !== undefined) { - throw new Error(physicalNamingRemovedMessage("the columnNamingStrategy option of agentDocsFile()", String(removedNaming))); + throw new Error(physicalNamingRemovedMessage("the columnNamingStrategy option of agentDocsFile()", String(removedNaming), physicalNamingGuideUrl())); } const dir = (opts?.subDir ?? DEFAULT_AGENT_DIR).replace(/\/$/, ""); diff --git a/server/typescript/packages/codegen-ts/src/generators/agent-ui-page.ts b/server/typescript/packages/codegen-ts/src/generators/agent-ui-page.ts index 6e77ee4c7..59558098a 100644 --- a/server/typescript/packages/codegen-ts/src/generators/agent-ui-page.ts +++ b/server/typescript/packages/codegen-ts/src/generators/agent-ui-page.ts @@ -92,7 +92,11 @@ function controlCell(f: UiFieldDescriptor): string { /** The HTML type cell — empty for a nested sub-form, which is not an input at all. */ function htmlTypeCell(f: UiFieldDescriptor): string { if (f.nested !== undefined || f.htmlType === undefined) return ""; - return `\`${f.htmlType}\``; + const extras = [ + ...(f.step !== undefined ? [`step \`${f.step}\``] : []), + ...(f.inputMode !== undefined ? [`inputMode \`${f.inputMode}\``] : []), + ]; + return extras.length === 0 ? `\`${f.htmlType}\`` : `\`${f.htmlType}\` (${extras.join(", ")})`; } /** `yes` / `` — an empty cell reads better than a column of "no". */ diff --git a/server/typescript/packages/codegen-ts/src/index.ts b/server/typescript/packages/codegen-ts/src/index.ts index 3b8be78ac..15cbe507f 100644 --- a/server/typescript/packages/codegen-ts/src/index.ts +++ b/server/typescript/packages/codegen-ts/src/index.ts @@ -142,7 +142,7 @@ export { renderAgentApi, renderApiIndex, renderEntityApiPage } from "./generator export { isProjection, isWriteThrough } from "./projection/projection-detector.js"; export { isAbstract, emitsInstanceArtifacts, emitsWriteArtifacts } from "./instance-artifacts.js"; // The UI tier asks THESE — "is there an endpoint?" — never the storage predicates. -export { hasGeneratedForm, hasItemRoute, itemRouteField, reportDecimalColumns, restPath, servesClientHooks, servesClientTier, servesReadApi, servesWriteApi } from "./api-surface.js"; +export { hasGeneratedForm, hasItemRoute, decimalColumnsLine, decimalFieldNames, itemRouteField, restPath, servesClientHooks, servesClientTier, servesReadApi, servesWriteApi } from "./api-surface.js"; // #356 — every emitter selects a field's view by the SURFACE it renders, never by // declaration position. An owned generator (FR-040) composing the render layer must // use this too, or it reinstates the order-dependence in its own copy. @@ -151,7 +151,7 @@ export { viewForContext, VIEW_CONTEXT_FORM, VIEW_CONTEXT_GRID } from "./view-con // generator and the `agent/ui.md` docs page must both ask — it decides whether a field is // rendered as an input at all — and `humanize` is the one body behind every generated // label. Both are exported so the react tier reads them rather than keeping a copy. -export { humanize, inferViewKind, labelFor, currencyMetaFor, valueObjectFor } from "./templates/field-meta.js"; +export { humanize, inferViewKind, labelFor, currencyMetaFor, valueObjectFor, numericInputControl } from "./templates/field-meta.js"; // The DB-free descriptor module the UI tier imports from (see entity-meta-file.ts). export { renderEntityMetaFile, entityMetaFileName, entityMetaSpecifier } from "./templates/entity-meta-file.js"; // FR-017 TPH helpers — used by the per-framework codegen packages (tanstack, diff --git a/server/typescript/packages/codegen-ts/src/metaobjects-config.ts b/server/typescript/packages/codegen-ts/src/metaobjects-config.ts index 995951f78..68dc4ffa4 100644 --- a/server/typescript/packages/codegen-ts/src/metaobjects-config.ts +++ b/server/typescript/packages/codegen-ts/src/metaobjects-config.ts @@ -1,4 +1,5 @@ import { physicalNamingRemovedMessage, type MetaDataTypeProvider } from "@metaobjectsdev/metadata"; +import { physicalNamingGuideUrl } from "@metaobjectsdev/metadata/release-docs"; import type { RoutesExpose } from "./routes-expose.js"; import type { Generator } from "./generator.js"; import type { ExtStyle } from "./render-context.js"; @@ -371,7 +372,7 @@ export function normalizeConfig(config: MetaobjectsGenConfig): NormalizedMetaobj // because the value it held may be the one that matches this project's database. const removedNaming = (config as { columnNamingStrategy?: unknown }).columnNamingStrategy; if (removedNaming !== undefined) { - throw new Error(physicalNamingRemovedMessage("columnNamingStrategy in metaobjects.config.ts", String(removedNaming))); + throw new Error(physicalNamingRemovedMessage("columnNamingStrategy in metaobjects.config.ts", String(removedNaming), physicalNamingGuideUrl())); } const dialect = config.dialect ?? DEFAULT_DIALECT; return { diff --git a/server/typescript/packages/codegen-ts/src/reference/agent-docs.ts b/server/typescript/packages/codegen-ts/src/reference/agent-docs.ts index 877ccbcd6..77e6da4c0 100644 --- a/server/typescript/packages/codegen-ts/src/reference/agent-docs.ts +++ b/server/typescript/packages/codegen-ts/src/reference/agent-docs.ts @@ -43,6 +43,7 @@ // each is built from what the toolchain itself reads rather than from a derivation of its // own. An empty page is no file. +import { physicalNamingGuideUrl } from "@metaobjectsdev/metadata/release-docs"; import { CARDINALITY_MANY, CARDINALITY_ONE, @@ -225,7 +226,7 @@ export const agentDocsFile = function agentDocsFile(opts?: AgentDocsFileOpts): G // option is refused rather than dropped: its value may be the one matching the database. const removedNaming = (opts as { columnNamingStrategy?: unknown } | undefined)?.columnNamingStrategy; if (removedNaming !== undefined) { - throw new Error(physicalNamingRemovedMessage("the columnNamingStrategy option of agentDocsFile()", String(removedNaming))); + throw new Error(physicalNamingRemovedMessage("the columnNamingStrategy option of agentDocsFile()", String(removedNaming), physicalNamingGuideUrl())); } const dir = (opts?.subDir ?? DEFAULT_AGENT_DIR).replace(/\/$/, ""); diff --git a/server/typescript/packages/codegen-ts/src/reference/routes-hono.ts b/server/typescript/packages/codegen-ts/src/reference/routes-hono.ts index 91ae2f5a5..821d095ed 100644 --- a/server/typescript/packages/codegen-ts/src/reference/routes-hono.ts +++ b/server/typescript/packages/codegen-ts/src/reference/routes-hono.ts @@ -58,7 +58,8 @@ import { isWriteThrough, isReport, itemRouteField, - reportDecimalColumns, + decimalColumnsLine, + decimalFieldNames, servesReadApi, formatTs, entityOutputPath, @@ -136,8 +137,8 @@ function renderRoutesHono( // The mount addresses `id` by default. A projection keyed on another field names it // (the view's key for that column, which is the field name), so `GET /:id` reads the // same column the by-id query does. Absent for `id`, which keeps that output's bytes. - // A decimal a SQLite report computes is a REAL; the mount sends it as the string it is elsewhere. - const decimalColumns = reportDecimalColumns(entity, ctx.dialect); + // A decimal is a canonical string on the wire on every dialect (ADR-0019 Amendment 1). + const decimalColumns = decimalFieldNames(entity); const keylessOpts = (indent: string): string => (idField !== undefined && idField !== DEFAULT_ID_FIELD ? `\n${indent}idColumn: ${JSON.stringify(idField)},` @@ -232,7 +233,7 @@ export function ${handlerName}(app: ${HonoSym}, deps: { db: unkno updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${exposeLineHono} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ")}${exposeLineHono} }); } `; diff --git a/server/typescript/packages/codegen-ts/src/reference/routes.ts b/server/typescript/packages/codegen-ts/src/reference/routes.ts index c6f8a3d89..d9cc73bbf 100644 --- a/server/typescript/packages/codegen-ts/src/reference/routes.ts +++ b/server/typescript/packages/codegen-ts/src/reference/routes.ts @@ -68,7 +68,8 @@ import { isWriteThrough, isReport, itemRouteField, - reportDecimalColumns, + decimalColumnsLine, + decimalFieldNames, servesReadApi, formatTs, entityOutputPath, @@ -154,8 +155,8 @@ function renderRoutes( // The mount addresses `id` by default. A projection keyed on another field names it // (the view's key for that column, which is the field name), so `GET /:id` reads the // same column the by-id query does. Absent for `id`, which keeps that output's bytes. - // A decimal a SQLite report computes is a REAL; the mount sends it as the string it is elsewhere. - const decimalColumns = reportDecimalColumns(entity, ctx.dialect); + // A decimal is a canonical string on the wire on every dialect (ADR-0019 Amendment 1). + const decimalColumns = decimalFieldNames(entity); const keylessOpts = (indent: string): string => (idField !== undefined && idField !== DEFAULT_ID_FIELD ? `\n${indent}idColumn: ${JSON.stringify(idField)},` @@ -297,7 +298,7 @@ export async function ${handlerName}(fastify: ${FastifyInstanceSym}) { updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${exposeLinePrefixed} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ")}${exposeLinePrefixed} }); ${m2mMountsPrefixed} }, { prefix: ${JSON.stringify(ctx.apiPrefix)} }); } @@ -322,7 +323,7 @@ export async function ${handlerName}(fastify: ${FastifyInstanceSym}) { updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${exposeLineFlat} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ")}${exposeLineFlat} }); ${m2mMountsFlat}} `; @@ -478,7 +479,7 @@ function renderM2mMount( sourceColumn: ${sourceColumn}, targetColumn: ${targetColumn}, targetPkColumn: ${targetPkColumn}, - symmetric: ${entry.symmetric ? "true" : "false"},${discriminatorLine}${sourceDiscriminatorLine} + symmetric: ${entry.symmetric ? "true" : "false"},${decimalColumnsLine(entry.target, " ")}${discriminatorLine}${sourceDiscriminatorLine} });`; } @@ -579,7 +580,7 @@ function renderTphRoutes( updateSchema: ${baseUpdateSym}, filterAllowlist: ${baseFilterSym}, sortAllowlist: ${baseSortSym}, - dialect: ${dialectLit},${polymorphicExposeLine} + dialect: ${dialectLit},${decimalColumnsLine([base, ...plan.subtypes.map((st) => st.entity)], " ")}${polymorphicExposeLine} });`; // FR-018 x FR-017 — M:N traversal inside a TPH hierarchy. This file never consulted @@ -622,7 +623,7 @@ function renderTphRoutes( updateSchema: ${subUpdateSym}, filterAllowlist: ${subFilterSym}, sortAllowlist: ${subSortSym}, - dialect: ${dialectLit}, + dialect: ${dialectLit},${decimalColumnsLine(sub, " ")} discriminator: { column: ${JSON.stringify(discField)}, value: ${JSON.stringify(value)} },${exposeLine(expose, " ")} });`; // This subtype's own M:N navigations, mounted beneath its segment and gated on the diff --git a/server/typescript/packages/codegen-ts/src/requirement-walk.ts b/server/typescript/packages/codegen-ts/src/requirement-walk.ts index ae0c0197e..028fd89bb 100644 --- a/server/typescript/packages/codegen-ts/src/requirement-walk.ts +++ b/server/typescript/packages/codegen-ts/src/requirement-walk.ts @@ -22,6 +22,7 @@ import { REQUIREMENT_ATTR_COUNTEREXAMPLE, REQUIREMENT_STATUSES_REQUIRING_LIVE_NODES, resolveClaim, + effectiveRequirementPackage, } from "@metaobjectsdev/metadata"; import type { MetaData, MetaRequirement } from "@metaobjectsdev/metadata"; @@ -84,7 +85,7 @@ export function walkRequirements(root: MetaData): WalkedRequirement[] { const req = node as MetaRequirement; // Same referrer-package basis the CLI's checks use, so a bare reference binds // package-locally under the ADR-0042 contract. - const referrerPkg = node.package ?? node.fileDefaultPackage ?? ""; + const referrerPkg = effectiveRequirementPackage(node); const targets: ResolvedClaim[] = []; for (const ref of req.implementedBy()) { diff --git a/server/typescript/packages/codegen-ts/src/templates/decimal-wire-helper.ts b/server/typescript/packages/codegen-ts/src/templates/decimal-wire-helper.ts new file mode 100644 index 000000000..c2b3a8422 --- /dev/null +++ b/server/typescript/packages/codegen-ts/src/templates/decimal-wire-helper.ts @@ -0,0 +1,47 @@ +// The wire rule for a field.decimal WRITE (ADR-0019 Amendment 1, W2 and W3), as text a generated +// model module carries as a local function. Generated model code imports nothing from +// MetaObjects, so the rule is emitted, not imported; runtime-ts/src/decimal-wire.ts holds the +// same grammar and both are held to fixtures/decimal-wire/vectors.json. + +export const DECIMAL_WRITE_HELPER = "decimalWrite"; + +/** The Zod expression of a decimal WRITE field: a string in the JSON-number grammar or a JSON + * number, rewritten to the canonical string, refused when the column cannot hold it (W3). */ +export function zodDecimalWriteExpr(precision: number | undefined, scale: number | undefined): string { + const args = precision === undefined ? "" : `, ${precision}, ${scale ?? 0}`; + return `z.union([z.string(), z.number()]).transform((v, ctx) => ${DECIMAL_WRITE_HELPER}(v, ctx${args}))`; +} + +export const DECIMAL_WRITE_HELPER_DECL = [ + "/** A field.decimal on the wire: a string in the JSON-number grammar (an exponent is allowed) or a JSON", + " * number, rewritten to its canonical string (plain notation, no trailing fraction zeros, zero as \"0\").", + " * Garbage, or a value the column cannot hold without rounding, is a validation issue. Generated. */", + `function ${DECIMAL_WRITE_HELPER}(v: string | number, ctx: z.RefinementCtx, precision?: number, scale = 0): string {`, + ' const text = typeof v === "number" ? (Number.isFinite(v) ? v.toPrecision(15) : "") : v;', + " const m = /^(-?)(0|[1-9][0-9]*)(?:\\.([0-9]+))?(?:[eE]([+-]?[0-9]+))?$/.exec(text);", + " const e = m === null || m[4] === undefined ? 0 : Number(m[4]);", + " if (m === null || !Number.isSafeInteger(e) || Math.abs(e) > 1000) {", + ' ctx.addIssue({ code: "custom", message: "Not a decimal number" });', + " return z.NEVER;", + " }", + ' const whole = m[2] ?? "";', + ' let digits = whole + (m[3] ?? "");', + " let point = whole.length + e;", + ' const lead = digits.length - digits.replace(/^0+/, "").length;', + " digits = digits.slice(lead).replace(/0+$/, \"\");", + " point -= lead;", + ' if (digits.length === 0) return "0";', + " const plain = point <= 0", + ' ? `0.${"0".repeat(-point)}${digits}`', + " : point >= digits.length", + ' ? digits + "0".repeat(point - digits.length)', + " : `${digits.slice(0, point)}.${digits.slice(point)}`;", + " const fraction = point >= digits.length ? 0 : digits.length - Math.max(point, 0) + Math.max(-point, 0);", + " const integer = Math.max(point, 0);", + " if (plain.length > 1000 || (precision !== undefined && (fraction > scale || integer > precision - scale))) {", + ' ctx.addIssue({ code: "custom", message: "Decimal out of range for the column" });', + " return z.NEVER;", + " }", + ' return m[1] === "-" ? `-${plain}` : plain;', + "}", +].join("\n"); diff --git a/server/typescript/packages/codegen-ts/src/templates/entity-constants.ts b/server/typescript/packages/codegen-ts/src/templates/entity-constants.ts index 2574532d5..1e26a0f46 100644 --- a/server/typescript/packages/codegen-ts/src/templates/entity-constants.ts +++ b/server/typescript/packages/codegen-ts/src/templates/entity-constants.ts @@ -74,6 +74,8 @@ function renderFieldEntry(f: UiFieldDescriptor): string { `view: ${JSON.stringify(f.view)}`, ]; if (f.htmlType !== undefined) entries.push(`htmlType: ${JSON.stringify(f.htmlType)}`); + if (f.step !== undefined) entries.push(`step: ${JSON.stringify(f.step)}`); + if (f.inputMode !== undefined) entries.push(`inputMode: ${JSON.stringify(f.inputMode)}`); if (f.rules.length > 0) entries.push(`rules: { ${f.rules.map(renderRule).join(", ")} }`); // A `field.enum`'s member symbols. Emitted whenever the field has them, because the diff --git a/server/typescript/packages/codegen-ts/src/templates/entity-ui-descriptor.ts b/server/typescript/packages/codegen-ts/src/templates/entity-ui-descriptor.ts index f8525bb2b..1d3036365 100644 --- a/server/typescript/packages/codegen-ts/src/templates/entity-ui-descriptor.ts +++ b/server/typescript/packages/codegen-ts/src/templates/entity-ui-descriptor.ts @@ -52,7 +52,7 @@ import { pluralize, toSnakeCase, } from "@metaobjectsdev/metadata"; -import { inferViewKind, currencyMetaFor, labelFor, humanize, valueObjectFor } from "./field-meta.js"; +import { inferViewKind, currencyMetaFor, labelFor, humanize, valueObjectFor, numericInputControl } from "./field-meta.js"; import { VIEW_CONTEXT_FORM } from "../view-context.js"; import { enumValues } from "../enum-meta.js"; // `restPath` lives HERE rather than in api-surface.ts, which is where it used to sit and @@ -79,6 +79,10 @@ export interface UiFieldDescriptor { /** Present only when the view maps to a real HTML ``. Derived from the * view subtype — there is no authored override (see `htmlTypeFromView`). */ readonly htmlType?: string | undefined; + /** `"any"` on a double/float number input so `12.5` passes native validation. */ + readonly step?: string | undefined; + /** `"decimal"` on a decimal's text input (see `numericInputControl`). */ + readonly inputMode?: string | undefined; /** Empty when the field declares nothing to validate. */ readonly rules: readonly UiRule[]; /** Present only for a `field.currency`. */ @@ -180,7 +184,7 @@ function htmlTypeFromView(view: string, fieldSubType: string): string | undefine case VIEW_SUBTYPE_TEXT: return "text"; case VIEW_SUBTYPE_NUMBER: - return "number"; + return numericInputControl(fieldSubType)?.htmlType ?? "number"; case VIEW_SUBTYPE_DATE: // The `date` VIEW is the default for all three calendar/clock subtypes, but only a // calendar date fits ``: a timestamp needs a time as well and a @@ -209,6 +213,14 @@ function htmlTypeFromView(view: string, fieldSubType: string): string | undefine } } +function stepAttrs(subType: string): { step?: string; inputMode?: string } { + const c = numericInputControl(subType); + return { + ...(c?.step !== undefined ? { step: c.step } : {}), + ...(c?.inputMode !== undefined ? { inputMode: c.inputMode } : {}), + }; +} + /** * The field's validation rules, in DECLARATION ORDER — see the ordering note at the top * of this file. @@ -293,6 +305,8 @@ export function buildUiFieldDescriptor(field: MetaField, root?: MetaRoot): UiFie label: labelFor(field, VIEW_CONTEXT_FORM), view, htmlType: htmlTypeFromView(view, field.subType), + // Only the number view carries them: a field authored with another view has its own control. + ...(view === VIEW_SUBTYPE_NUMBER ? stepAttrs(field.subType) : {}), rules: buildFieldRules(field), currency: currencyMeta === null ? undefined diff --git a/server/typescript/packages/codegen-ts/src/templates/field-meta.ts b/server/typescript/packages/codegen-ts/src/templates/field-meta.ts index b27631db7..921296073 100644 --- a/server/typescript/packages/codegen-ts/src/templates/field-meta.ts +++ b/server/typescript/packages/codegen-ts/src/templates/field-meta.ts @@ -283,3 +283,39 @@ export function humanize(s: string): string { .replace(/\b\w/g, (c) => c.toUpperCase()) .trim(); } + +/** The `` attributes a numeric field subtype needs. */ +export interface NumericInputControl { + readonly htmlType: "number" | "text"; + /** `"any"` for a binary float: the browser's implicit step of 1 refuses `12.5`. */ + readonly step?: "any"; + /** `"decimal"` for an exact decimal rendered as a text input. */ + readonly inputMode?: "decimal"; +} + +/** + * The one answer both form tiers give for a numeric field subtype (the `` meta that + * `useEntityForm` spreads, and the nested sub-form in codegen-ts-react), or undefined for a + * non-numeric subtype. + * + * `` has an implicit step of 1, so native validation blocks `12.5`. + * double/float therefore get `step="any"`. A decimal is an exact value that travels as a + * string: a number input would parse it to a double for validation and min/max (wrong past + * 15 digits), so it is `type="text" inputmode="decimal"` and RHF/Zod own the validation. + * int, long and currency keep step 1. + */ +export function numericInputControl(subType: string): NumericInputControl | undefined { + switch (subType) { + case FIELD_SUBTYPE_INT: + case FIELD_SUBTYPE_LONG: + case FIELD_SUBTYPE_CURRENCY: + return { htmlType: "number" }; + case FIELD_SUBTYPE_DOUBLE: + case FIELD_SUBTYPE_FLOAT: + return { htmlType: "number", step: "any" }; + case FIELD_SUBTYPE_DECIMAL: + return { htmlType: "text", inputMode: "decimal" }; + default: + return undefined; + } +} diff --git a/server/typescript/packages/codegen-ts/src/templates/filter-allowlist.ts b/server/typescript/packages/codegen-ts/src/templates/filter-allowlist.ts index 9616528e6..649bf11e7 100644 --- a/server/typescript/packages/codegen-ts/src/templates/filter-allowlist.ts +++ b/server/typescript/packages/codegen-ts/src/templates/filter-allowlist.ts @@ -69,6 +69,7 @@ const VALUE_FORMAT_SUBTYPES = new Set([ FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_UUID, + FIELD_SUBTYPE_DECIMAL, ]); /** Subtypes whose values are whole numbers — `field.currency` is integer minor units. */ diff --git a/server/typescript/packages/codegen-ts/src/templates/routes-file-hono.ts b/server/typescript/packages/codegen-ts/src/templates/routes-file-hono.ts index dc75026d7..3a34b9d70 100644 --- a/server/typescript/packages/codegen-ts/src/templates/routes-file-hono.ts +++ b/server/typescript/packages/codegen-ts/src/templates/routes-file-hono.ts @@ -31,7 +31,7 @@ import { entityModuleSpecifier } from "../import-path.js"; import { GENERATED_HEADER, GENERATED_EDIT_NOTE, sidecarLine } from "../constants.js"; import { isProjection, isWriteThrough } from "../projection/projection-detector.js"; import { isReport } from "../source-detect.js"; -import { itemRouteField, reportDecimalColumns } from "../api-surface.js"; +import { decimalColumnsLine, decimalFieldNames, itemRouteField } from "../api-surface.js"; import { DEFAULT_ID_FIELD } from "../pk-resolver.js"; import { authSeamJsDoc, type CrudVerb, exposeLine } from "../routes-expose.js"; import { effectivePackage } from "../docs-paths.js"; @@ -94,8 +94,8 @@ export function renderRoutesFileHono( // The mount addresses `id` by default. A projection keyed on another field names it // (the view's key for that column, which is the field name), so `GET /:id` reads the // same column the by-id query does. Absent for `id`, which keeps that output's bytes. - // A decimal a SQLite report computes is a REAL; the mount sends it as the string it is elsewhere. - const decimalColumns = reportDecimalColumns(entity, ctx.dialect); + // A decimal is a canonical string on the wire on every dialect (ADR-0019 Amendment 1). + const decimalColumns = decimalFieldNames(entity); const keylessOpts = (indent: string): string => (idField !== undefined && idField !== DEFAULT_ID_FIELD ? `\n${indent}idColumn: ${JSON.stringify(idField)},` @@ -190,7 +190,7 @@ export function ${handlerName}(app: ${HonoSym}, deps: { db: unkno updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${exposeLineHono} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ")}${exposeLineHono} }); } `; diff --git a/server/typescript/packages/codegen-ts/src/templates/routes-file.ts b/server/typescript/packages/codegen-ts/src/templates/routes-file.ts index 91602f7f2..c44c52d06 100644 --- a/server/typescript/packages/codegen-ts/src/templates/routes-file.ts +++ b/server/typescript/packages/codegen-ts/src/templates/routes-file.ts @@ -29,7 +29,7 @@ import { GENERATED_HEADER, GENERATED_EDIT_NOTE, sidecarLine } from "../constants import { routesHandlerName } from "../naming.js"; import { isProjection, isWriteThrough } from "../projection/projection-detector.js"; import { isReport } from "../source-detect.js"; -import { itemRouteField, reportDecimalColumns } from "../api-surface.js"; +import { decimalColumnsLine, decimalFieldNames, itemRouteField } from "../api-surface.js"; import { DEFAULT_ID_FIELD, pkInfoOf } from "../pk-resolver.js"; import type { RelationEntry } from "../relation-resolver.js"; import { isTphDiscriminatorBase, tphPlan } from "./tph-discriminator.js"; @@ -93,8 +93,8 @@ export function renderRoutesFile( // The mount addresses `id` by default. A projection keyed on another field names it // (the view's key for that column, which is the field name), so `GET /:id` reads the // same column the by-id query does. Absent for `id`, which keeps that output's bytes. - // A decimal a SQLite report computes is a REAL; the mount sends it as the string it is elsewhere. - const decimalColumns = reportDecimalColumns(entity, ctx.dialect); + // A decimal is a canonical string on the wire on every dialect (ADR-0019 Amendment 1). + const decimalColumns = decimalFieldNames(entity); const keylessOpts = (indent: string): string => (idField !== undefined && idField !== DEFAULT_ID_FIELD ? `\n${indent}idColumn: ${JSON.stringify(idField)},` @@ -236,7 +236,7 @@ export async function ${handlerName}(fastify: ${FastifyInstanceSym}) { updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${exposeLinePrefixed} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ")}${exposeLinePrefixed} }); ${m2mMountsPrefixed} }, { prefix: ${JSON.stringify(ctx.apiPrefix)} }); } @@ -261,7 +261,7 @@ export async function ${handlerName}(fastify: ${FastifyInstanceSym}) { updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${exposeLineFlat} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ")}${exposeLineFlat} }); ${m2mMountsFlat}} `; @@ -412,7 +412,7 @@ function renderM2mMount( sourceColumn: ${sourceColumn}, targetColumn: ${targetColumn}, targetPkColumn: ${targetPkColumn}, - symmetric: ${entry.symmetric ? "true" : "false"},${discriminatorLine}${sourceDiscriminatorLine} + symmetric: ${entry.symmetric ? "true" : "false"},${decimalColumnsLine(entry.target, " ")}${discriminatorLine}${sourceDiscriminatorLine} });`; } @@ -510,7 +510,7 @@ function renderTphRoutesFile( updateSchema: ${baseUpdateSym}, filterAllowlist: ${baseFilterSym}, sortAllowlist: ${baseSortSym}, - dialect: ${dialectLit},${polymorphicExposeLine} + dialect: ${dialectLit},${decimalColumnsLine([base, ...plan.subtypes.map((st) => st.entity)], " ")}${polymorphicExposeLine} });`; // FR-018 x FR-017 — M:N traversal inside a TPH hierarchy. This file never consulted @@ -553,7 +553,7 @@ function renderTphRoutesFile( updateSchema: ${subUpdateSym}, filterAllowlist: ${subFilterSym}, sortAllowlist: ${subSortSym}, - dialect: ${dialectLit}, + dialect: ${dialectLit},${decimalColumnsLine(sub, " ")} discriminator: { column: ${JSON.stringify(discField)}, value: ${JSON.stringify(value)} },${exposeLine(expose, " ")} });`; // This subtype's own M:N navigations, mounted beneath its segment and gated on the diff --git a/server/typescript/packages/codegen-ts/src/templates/zod-validators.ts b/server/typescript/packages/codegen-ts/src/templates/zod-validators.ts index 47305ddc2..c8fc21241 100644 --- a/server/typescript/packages/codegen-ts/src/templates/zod-validators.ts +++ b/server/typescript/packages/codegen-ts/src/templates/zod-validators.ts @@ -13,7 +13,7 @@ import { code, joinCode, imp, type Code } from "ts-poet"; import { MetaObject, MetaField, type MetaRoot, stripPackage } from "@metaobjectsdev/metadata"; import { FIELD_SUBTYPE_STRING, FIELD_SUBTYPE_INT, FIELD_SUBTYPE_LONG, FIELD_SUBTYPE_CURRENCY, - FIELD_SUBTYPE_BOOLEAN, FIELD_SUBTYPE_DOUBLE, FIELD_SUBTYPE_FLOAT, + FIELD_SUBTYPE_BOOLEAN, FIELD_SUBTYPE_DECIMAL, FIELD_ATTR_PRECISION, FIELD_ATTR_SCALE, FIELD_SUBTYPE_DOUBLE, FIELD_SUBTYPE_FLOAT, FIELD_SUBTYPE_DATE, FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_ENUM, FIELD_SUBTYPE_OBJECT, FIELD_SUBTYPE_MAP, FIELD_SUBTYPE_UUID, FIELD_SUBTYPE_URI, FIELD_SUBTYPE_INET, @@ -43,6 +43,7 @@ import { ZOD_TIMESTAMP_UTC_EXPR, } from "./date-time-regex.js"; import { renderDocsFor } from "./jsdoc.js"; +import { DECIMAL_WRITE_HELPER_DECL, zodDecimalWriteExpr } from "./decimal-wire-helper.js"; import { sharedEnumForField } from "../enum-shared.js"; import { sharedEnumImportSpecifier } from "../enum-import.js"; import { sharedEnumZodConstName } from "./enums-file.js"; @@ -74,6 +75,13 @@ function utcTimestampHelperSince(since: number): string { return utcTimestampUses > since ? `${UTC_TIMESTAMP_HELPER_DECL}\n\n` : ""; } +// The same device for the decimal write rule (decimal-wire-helper.ts). +let decimalWriteUses = 0; + +function decimalHelperSince(since: number): string { + return decimalWriteUses > since ? `${DECIMAL_WRITE_HELPER_DECL}\n\n` : ""; +} + /** True when this object declares at least one @autoSet timestamp field * (onCreate or onUpdate). Drives whether codegen emits the #203 @@ -259,6 +267,7 @@ function isInsertRequiredPk(field: MetaField, assignedPk: Set): boolean export function renderInsertSchemaOnly(obj: MetaObject, ctx?: RenderContext): Code { const z = imp("z@zod"); const utcUsesBefore = utcTimestampUses; + const decimalUsesBefore = decimalWriteUses; const autoGenPkFields = autoGenPkFieldNames(obj); const assignedPkFields = assignedPkFieldNames(obj); const tphPin = tphDiscriminatorPin(obj); @@ -310,7 +319,7 @@ export function renderInsertSchemaOnly(obj: MetaObject, ctx?: RenderContext): Co const docsPrefix = docs ? `${docs}\n` : ""; return code` -${utcTimestampHelperSince(utcUsesBefore)}${docsPrefix}export const ${insertSchemaName} = ${z}.object({ +${utcTimestampHelperSince(utcUsesBefore)}${decimalHelperSince(decimalUsesBefore)}${docsPrefix}export const ${insertSchemaName} = ${z}.object({ ${joinCode(insertFieldLines, { on: ",\n" })} }); `; @@ -418,6 +427,7 @@ export function updateSchemaFields(obj: MetaObject): SchemaFieldShape[] { export function renderZodValidators(obj: MetaObject, ctx?: RenderContext): Code { const z = imp("z@zod"); const utcUsesBefore = utcTimestampUses; + const decimalUsesBefore = decimalWriteUses; const autoGenPkFields = autoGenPkFieldNames(obj); const pkFields = primaryKeyFieldNames(obj); const assignedPkFields = assignedPkFieldNames(obj); @@ -540,7 +550,7 @@ export type ${objName}CreatePreserving = ${z}.input { + const res = await new MetaDataLoader().load([ + new InMemoryStringSource(JSON.stringify(model), { id: "meta.json", format: "json" }), + ]); + expect(res.errors).toEqual([]); + return res.root; +} + +// `includeUiTier` defaults TRUE here because every model below describes a project whose +// suite emits forms and hooks; the page is gated on the RUN, not on the metadata, so the +// helper has to state which run it is. The false case is its own test. +function makeCtx(root: MetaRoot, apiPrefix = "", includeUiTier = true): GenContext { + return { + entities: root.objects(), + loadedRoot: root, + matches: () => true, + config: { outDir: "/tmp", extStyle: "none", dbImport: "~/db", dialect: "sqlite", includeUiTier } as never, + renderContext: makeRenderContext({ + dialect: "sqlite", + loadedRoot: root, + outDir: "/tmp", + dbImport: "~/db", + apiPrefix, + pkMap: buildPkMap(root), + relationMap: buildRelationMap(root), + }), + warn: () => {}, + }; +} + +async function emit( + root: MetaRoot, + opts?: Parameters[0], + apiPrefix = "", + includeUiTier = true, +): Promise> { + const files = await agentDocsFile(opts).generate(makeCtx(root, apiPrefix, includeUiTier)); + return new Map(files.map((f) => [f.path, f.content])); +} + +const MODEL = { + "metadata.root": { + package: "acme::shop", + children: [ + { + "object.entity": { + name: "Item", + children: [ + { "source.rdb": { "@table": "items" } }, + { "field.long": { name: "id", "@required": true } }, + { "field.int": { name: "qty" } }, + { "field.double": { name: "weight" } }, + { "field.decimal": { name: "price", "@precision": 12, "@scale": 2 } }, + { "identity.primary": { name: "pk", "@fields": "id", "@generation": "increment" } }, + ], + }, + }, + ], + }, +}; + +describe("agent/ui.md — the HTML type cell carries step and inputMode", () => { + test("decimal is text with inputMode, double is number with step, int is plain number", async () => { + const ui = (await emit(await load(MODEL))).get("agent/ui.md") ?? ""; + const cells = (name: string): string[] => + (ui.split("\n").find((l) => l.startsWith(`| \`${name}\``)) ?? "").split("|").map((c) => c.trim()); + expect(cells("price")[4]).toBe("`text` (inputMode `decimal`)"); + expect(cells("weight")[4]).toBe("`number` (step `any`)"); + expect(cells("qty")[4]).toBe("`number`"); + }); +}); diff --git a/server/typescript/packages/codegen-ts/test/cube/build-cube-model.test.ts b/server/typescript/packages/codegen-ts/test/cube/build-cube-model.test.ts index a8cb487fa..1bea40ae4 100644 --- a/server/typescript/packages/codegen-ts/test/cube/build-cube-model.test.ts +++ b/server/typescript/packages/codegen-ts/test/cube/build-cube-model.test.ts @@ -367,7 +367,6 @@ describe("Table C — dimension types", () => { test.each([ ["isArray", { "field.string": { name: "x", isArray: true } }, /field\.string, isArray/], - ["object", { "field.object": { name: "x", "@objectRef": "Label" } }, /field\.object/], ["map", { "field.map": { name: "x", "@valueType": "string" } }, /field\.map/], ])("%s → ERR_CUBE_UNMAPPABLE_DIMENSION naming the dimension and its field", async (_label, field, detail) => { const err = await buildError([ @@ -1494,7 +1493,7 @@ describe("Table H — the canonical model", () => { // join-path cubes carry no report of Table H's; only their presence is asserted here. expect(model.cubes.map((c) => c.name)).toEqual([ "Program", "Week", "Node", "Session", "Measurement", "Asset", "Auth", "BridgeAuth", "CopayAuth", "PriorAuthAuth", - "ReferralAuth", "AllTypes", "Node_fkParent", "ProgramRosterFacts", "ProgramLongWeeksFacts", + "ReferralAuth", "AllTypes", "Gadget", "GadgetEvent", "Entry", "Node_fkParent", "ProgramRosterFacts", "ProgramLongWeeksFacts", "ProgramFlagRosterFacts", "ProgramSessionsFacts", "ProgramSessions_fkWeek", ]); const tableH = new Set(["Program", "Week", "Asset", "ProgramRosterFacts", "ProgramLongWeeksFacts", "ProgramFlagRosterFacts"]); @@ -1550,22 +1549,25 @@ describe("Table H — the canonical model", () => { name: "LiveIntroPrograms", type: "rollup", measures: ["programs", "notDrafts", "listValue"], dimensions: ["status"], segments: ["introOrFree", "liveIntroProgramsScope"], }, + { + name: "ProgramsOverTime_year", type: "rollup", measures: ["programs", "avgPrice", "cheapest", "priciest"], + dimensions: [], segments: [], timeDimension: "createdAt", granularity: "year", + }, + { + name: "ProgramsOverTime_quarter", type: "rollup", measures: ["programs", "avgPrice", "cheapest", "priciest"], + dimensions: [], segments: [], timeDimension: "createdAt", granularity: "quarter", + }, { name: "ProgramsByWeek", type: "rollup", measures: ["programs"], dimensions: [], segments: ["published"], timeDimension: "createdAt", granularity: "week", }, { - name: "ProgramsByMonth", type: "rollup", measures: ["programs", "listValue"], dimensions: ["status"], - segments: [], timeDimension: "createdAt", granularity: "month", + name: "ProgramsOverTime_day", type: "rollup", measures: ["programs", "avgPrice", "cheapest", "priciest"], + dimensions: [], segments: [], timeDimension: "createdAt", granularity: "day", }, { - name: "ProgramsOverTime", type: "rollup", measures: ["programs", "avgPrice", "cheapest", "priciest"], - dimensions: [], segments: [], - timeDimensions: [ - { dimension: "createdAt", granularity: "day" }, - { dimension: "createdAt", granularity: "quarter" }, - { dimension: "createdAt", granularity: "year" }, - ], + name: "ProgramsByMonth", type: "rollup", measures: ["programs", "listValue"], dimensions: ["status"], + segments: [], timeDimension: "createdAt", granularity: "month", }, ], }, diff --git a/server/typescript/packages/codegen-ts/test/cube/cube-model-canonical.test.ts b/server/typescript/packages/codegen-ts/test/cube/cube-model-canonical.test.ts index b254e5f7d..f541f5c45 100644 --- a/server/typescript/packages/codegen-ts/test/cube/cube-model-canonical.test.ts +++ b/server/typescript/packages/codegen-ts/test/cube/cube-model-canonical.test.ts @@ -19,6 +19,9 @@ describe("the cube-model canonical golden", () => { "model/cubes/Auth.yml", "model/cubes/BridgeAuth.yml", "model/cubes/CopayAuth.yml", + "model/cubes/Entry.yml", + "model/cubes/Gadget.yml", + "model/cubes/GadgetEvent.yml", "model/cubes/Measurement.yml", "model/cubes/Node.yml", "model/cubes/Node_fkParent.yml", diff --git a/server/typescript/packages/codegen-ts/test/cube/cube-yaml.test.ts b/server/typescript/packages/codegen-ts/test/cube/cube-yaml.test.ts index a05da9aef..c385a78eb 100644 --- a/server/typescript/packages/codegen-ts/test/cube/cube-yaml.test.ts +++ b/server/typescript/packages/codegen-ts/test/cube/cube-yaml.test.ts @@ -166,28 +166,33 @@ cubes: measures: [CUBE.programs, CUBE.notDrafts, CUBE.listValue] dimensions: [CUBE.status] segments: [CUBE.introOrFree, CUBE.liveIntroProgramsScope] + - name: ProgramsOverTime_year + type: rollup + measures: [CUBE.programs, CUBE.avgPrice, CUBE.cheapest, CUBE.priciest] + time_dimension: CUBE.createdAt + granularity: year + - name: ProgramsOverTime_quarter + type: rollup + measures: [CUBE.programs, CUBE.avgPrice, CUBE.cheapest, CUBE.priciest] + time_dimension: CUBE.createdAt + granularity: quarter - name: ProgramsByWeek type: rollup measures: [CUBE.programs] segments: [CUBE.published] time_dimension: CUBE.createdAt granularity: week + - name: ProgramsOverTime_day + type: rollup + measures: [CUBE.programs, CUBE.avgPrice, CUBE.cheapest, CUBE.priciest] + time_dimension: CUBE.createdAt + granularity: day - name: ProgramsByMonth type: rollup measures: [CUBE.programs, CUBE.listValue] dimensions: [CUBE.status] time_dimension: CUBE.createdAt granularity: month - - name: ProgramsOverTime - type: rollup - measures: [CUBE.programs, CUBE.avgPrice, CUBE.cheapest, CUBE.priciest] - time_dimensions: - - dimension: CUBE.createdAt - granularity: day - - dimension: CUBE.createdAt - granularity: quarter - - dimension: CUBE.createdAt - granularity: year `; const WEEK_YML = `${HEADER} @@ -1150,7 +1155,7 @@ describe("parse oracle", () => { const model = await canonicalModel(); expect(model.cubes.map((c) => c.name)).toEqual([ "Program", "Week", "Node", "Session", "Measurement", "Asset", "Auth", "BridgeAuth", "CopayAuth", "PriorAuthAuth", - "ReferralAuth", "AllTypes", "Node_fkParent", "ProgramRosterFacts", "ProgramLongWeeksFacts", + "ReferralAuth", "AllTypes", "Gadget", "GadgetEvent", "Entry", "Node_fkParent", "ProgramRosterFacts", "ProgramLongWeeksFacts", "ProgramFlagRosterFacts", "ProgramSessionsFacts", "ProgramSessions_fkWeek", ]); for (const c of model.cubes) expect(parsed(c)).toEqual({ cubes: [expectedCube(c)] }); diff --git a/server/typescript/packages/codegen-ts/test/cube/report-rollups.test.ts b/server/typescript/packages/codegen-ts/test/cube/report-rollups.test.ts index 3040b3583..229ebaf4a 100644 --- a/server/typescript/packages/codegen-ts/test/cube/report-rollups.test.ts +++ b/server/typescript/packages/codegen-ts/test/cube/report-rollups.test.ts @@ -182,13 +182,26 @@ describe("Table F — a rollup per served report, on its @from cube", () => { expect(Object.keys(c.preAggregations[0]!).sort()).toEqual(["dimensions", "measures", "name", "segments", "timeDimensions", "type"]); }); - test("one time dimension at two grains: the list form, one entry per grain", async () => { + test("one time dimension at two grains: one single-granularity rollup per grain (a Cube query names a time dimension once)", async () => { const c = await programCube( report("ProgramGrains", { "@from": "Program", "@dimensions": ["createdAt:week", "createdAt:month"], "@measures": ["programs"] }), ); + // Coarsest first; no rollup repeats the dimension, so each is one a Cube query at that grain matches. + expect(c.preAggregations).toEqual([ + { name: "ProgramGrains_month", type: "rollup", measures: ["programs"], dimensions: [], segments: [], timeDimension: "createdAt", granularity: "month" }, + { name: "ProgramGrains_week", type: "rollup", measures: ["programs"], dimensions: [], segments: [], timeDimension: "createdAt", granularity: "week" }, + ]); + for (const p of c.preAggregations) expect(Object.keys(p)).not.toContain("timeDimensions"); + }); + + test("two time dimensions, one at two grains: a rollup per grain choice, each naming both dimensions once", async () => { + const c = await programCube( + report("ProgramMixed", { "@from": "Program", "@dimensions": ["createdAt:week", "createdAt:month", "publishedOn:week"], "@measures": ["programs"] }), + ); + expect(c.preAggregations.map((p) => p.name)).toEqual(["ProgramMixed_month", "ProgramMixed_week"]); expect(c.preAggregations[0]!.timeDimensions).toEqual([ - { dimension: "createdAt", granularity: "week" }, { dimension: "createdAt", granularity: "month" }, + { dimension: "publishedOn", granularity: "week" }, ]); }); @@ -418,13 +431,12 @@ describe("Table F — no rollup where a relative date would be frozen", () => { // Reports that contribute nothing // --------------------------------------------------------------------------- +// An abstract report and a non-view source are refused at load (R10, R11), so only a sourceless one remains. describe("Table F — a report that is not served contributes nothing", () => { const attrs: Json = { "@from": "Program", "@dimensions": ["status"], "@measures": ["programs"], "@filter": { status: "DRAFT" } }; test.each([ ["a sourceless report", report("Unserved", attrs, null)], - ["an abstract report", report("Unserved", attrs, "view", { abstract: true })], - ["a materializedView report", report("Unserved", attrs, "materializedView")], ])("%s", async (_label, unserved) => { const withReport = await build([program(), unserved]); expect(withReport).toEqual(await build([program()])); diff --git a/server/typescript/packages/codegen-ts/test/decimal-wire-e2e.test.ts b/server/typescript/packages/codegen-ts/test/decimal-wire-e2e.test.ts new file mode 100644 index 000000000..3a4eaed31 --- /dev/null +++ b/server/typescript/packages/codegen-ts/test/decimal-wire-e2e.test.ts @@ -0,0 +1,119 @@ +// A field.decimal end to end on SQLite, GENERATED code executed (ADR-0019 Amendment 1). +// +// The Zod module and the filter allowlist rendered from metadata, mounted with runtime-ts's Hono +// helper over bun:sqlite with the `decimalColumns` option the generated route passes. Covers the +// write rule (W2 accept, W3 range), the canonical read (W1) and the 15-digit rendering of a +// double (W4) - none of which the Postgres-backed lanes can see, since SQLite is where a decimal +// is a double. Exactness beyond 15 digits is the Postgres and MySQL lanes' to prove. + +import { describe, test, expect, beforeAll, afterAll } from "bun:test"; +import { mkdtempSync, writeFileSync, rmSync } from "node:fs"; +import { join } from "node:path"; +import { pathToFileURL } from "node:url"; +import { Database } from "bun:sqlite"; +import { Hono } from "hono"; +import { drizzle } from "drizzle-orm/bun-sqlite"; +import { sqliteTable, integer, text, numeric } from "drizzle-orm/sqlite-core"; +import { MetaDataLoader, InMemoryStringSource, type MetaObject } from "@metaobjectsdev/metadata"; +import { mountCrudRoutes } from "@metaobjectsdev/runtime-ts/hono"; +import { renderZodValidators } from "../src/templates/zod-validators.js"; +import { renderFilterAllowlist } from "../src/templates/filter-allowlist.js"; +import { decimalColumnsLine, decimalFieldNames } from "../src/api-surface.js"; + +const MODEL = { + "metadata.root": { package: "ledger", children: [ + { "object.entity": { name: "Entry", children: [ + { "source.rdb": { "@table": "entries" } }, + { "field.long": { name: "id" } }, + { "field.string": { name: "bucket", "@required": true, "@filterable": true } }, + { "field.decimal": { name: "amount", "@precision": 9, "@scale": 4, "@filterable": true, "@sortable": true } }, + { "field.decimal": { name: "weight", "@required": true } }, + { "identity.primary": { name: "pk", "@fields": "id", "@generation": "increment" } }, + ]}}, + ]}, +}; + +const entries = sqliteTable("entries", { + id: integer("id").primaryKey({ autoIncrement: true }), + bucket: text("bucket").notNull(), + amount: numeric("amount"), + weight: numeric("weight").notNull(), +}); + +const dir = mkdtempSync(join(import.meta.dir, "tmp-decimal-")); +afterAll(() => rmSync(dir, { recursive: true, force: true })); + +async function execute(name: string, source: string): Promise> { + const file = join(dir, `${name}.ts`); + writeFileSync(file, source); + return import(pathToFileURL(file).href); +} + +let app: Hono; +let entry: MetaObject; +let moduleText: string; +beforeAll(async () => { + const { root, errors } = await new MetaDataLoader({ strict: true }).load([ + new InMemoryStringSource(JSON.stringify(MODEL), { id: "ledger.json" }), + ]); + expect(errors).toEqual([]); + entry = root.objects().find((o) => o.name === "Entry") as MetaObject; + moduleText = renderZodValidators(entry).toString(); + const schemas = await execute("schemas", moduleText); + const allowlists = await execute("allowlist", renderFilterAllowlist(entry).toString()); + + const sqlite = new Database(":memory:"); + sqlite.run("CREATE TABLE entries (id INTEGER PRIMARY KEY AUTOINCREMENT, bucket TEXT NOT NULL, amount NUMERIC, weight NUMERIC NOT NULL)"); + app = new Hono(); + mountCrudRoutes({ + app, path: "/entries", db: drizzle(sqlite), table: entries, + insertSchema: schemas["EntryInsertSchema"] as never, + updateSchema: schemas["EntryUpdateSchema"] as never, + filterAllowlist: allowlists["EntryFilterAllowlist"] as never, + sortAllowlist: {}, dialect: "sqlite", + decimalColumns: decimalFieldNames(entry), + }); +}); + +const post = (body: unknown) => + app.request("/entries", { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify(body) }); + +describe("generated decimal write schema and route, SQLite", () => { + test("the module carries one local helper, and imports nothing from MetaObjects", () => { + expect(moduleText.match(/function decimalWrite/g)?.length).toBe(1); + expect(moduleText).not.toContain("@metaobjectsdev"); + }); + + test("the route is given every decimal field name", () => { + expect(decimalFieldNames(entry)).toEqual(["amount", "weight"]); + expect(decimalColumnsLine(entry, " ")).toBe('\n decimalColumns: ["amount","weight"],'); + }); + + test("a string and a number are accepted, and answered as the canonical string", async () => { + for (const [sent, wire] of [["0.1", "0.1"], [0.3, "0.3"], ["100.2500", "100.25"], ["1.5e1", "15"], ["-0.0", "0"]] as const) { + const res = await post({ bucket: "a", amount: sent, weight: "1" }); + expect(res.status).toBe(201); + expect(((await res.json()) as { amount: unknown }).amount).toBe(wire); + } + }); + + test("a required decimal reads back canonical too", async () => { + const res = await post({ bucket: "w", weight: "2.50" }); + expect(((await res.json()) as { weight: unknown }).weight).toBe("2.5"); + }); + + test("a value the column cannot hold, or that is not a number, is a 400", async () => { + for (const amount of ["12abc", "NaN", "0.12345", "100000", "1e999999999", true, {}]) { + const res = await post({ bucket: "bad", amount, weight: "1" }); + expect(res.status).toBe(400); + expect(((await res.json()) as { error: string }).error).toBe("validation"); + } + }); + + test("a decimal list answers strings, never numbers, and filters by number", async () => { + const rows = (await (await app.request("/entries?filter[bucket][eq]=a")).json()) as Array<{ amount: unknown }>; + expect(rows.map((r) => r.amount)).toEqual(["0.1", "0.3", "100.25", "15", "0"]); + const gt = (await (await app.request("/entries?filter[bucket][eq]=a&filter[amount][gt]=15")).json()) as Array<{ amount: unknown }>; + expect(gt.map((r) => r.amount)).toEqual(["100.25"]); + }); +}); diff --git a/server/typescript/packages/codegen-ts/test/decimal-write-schema.test.ts b/server/typescript/packages/codegen-ts/test/decimal-write-schema.test.ts new file mode 100644 index 000000000..13338d490 --- /dev/null +++ b/server/typescript/packages/codegen-ts/test/decimal-write-schema.test.ts @@ -0,0 +1,72 @@ +// The generated Zod write schema of a field.decimal (ADR-0019 Amendment 1, W2 and W3). +// The module-local helper a generated model carries is EVALUATED here against +// fixtures/decimal-wire/vectors.json, so the emitted text, not a re-implementation of it, is what +// the vectors hold to the rule. + +import { describe, test, expect } from "bun:test"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import { z } from "zod"; +import { DECIMAL_WRITE_HELPER_DECL, zodDecimalWriteExpr } from "../src/templates/decimal-wire-helper.js"; + +interface Vectors { + format: Array<{ in: string; out: string }>; + parse: Array<{ json: string; value?: string | null; error?: boolean }>; + double15: Array<{ double: string; out: string }>; +} +const vectors = JSON.parse( + readFileSync(join(import.meta.dir, "..", "..", "..", "..", "..", "fixtures", "decimal-wire", "vectors.json"), "utf8"), +) as Vectors; + +/** Build the schema exactly as a generated module spells it: the helper declaration, then the expression. */ +function schemaFor(precision?: number, scale?: number): z.ZodType { + const js = DECIMAL_WRITE_HELPER_DECL + .replace(/\(v: string \| number, ctx: z\.RefinementCtx, precision\?: number, scale = 0\): string/, "(v, ctx, precision, scale = 0)") + .replace(/^\/\*\*[\s\S]*?\*\/\n/, ""); + const body = `${js}\nreturn ${zodDecimalWriteExpr(precision, scale)};`; + return new Function("z", body)(z) as z.ZodType; +} + +describe("generated decimal write schema", () => { + test("the expression for a column with and without a declared precision", () => { + expect(zodDecimalWriteExpr(undefined, undefined)).toBe( + "z.union([z.string(), z.number()]).transform((v, ctx) => decimalWrite(v, ctx))", + ); + expect(zodDecimalWriteExpr(28, 10)).toContain("decimalWrite(v, ctx, 28, 10)"); + }); + + const open = schemaFor(); + for (const v of vectors.format) { + test(`format ${v.in}`, () => { + expect(open.parse(v.in)).toBe(v.out); + }); + } + for (const v of vectors.parse) { + if (v.value === null) continue; // null is the nullable wrapper's, not the decimal's + test(`parse ${v.json}`, () => { + const parsed: unknown = JSON.parse(v.json); + const r = open.safeParse(parsed); + if (v.error === true) expect(r.success).toBe(false); + else expect(r.success ? r.data : r.error).toBe(v.value as string); + }); + } + for (const v of vectors.double15) { + test(`double15 ${v.double}`, () => { + expect(open.parse(Number(v.double))).toBe(v.out); + }); + } + + test("W3: a value the declared column cannot hold is refused, never rounded", () => { + const col = schemaFor(28, 10); + expect(col.parse("12345678901234567.89")).toBe("12345678901234567.89"); + expect(col.safeParse("0.12345678901").success).toBe(false); + expect(col.safeParse("1234567890123456789").success).toBe(false); + expect(col.parse("123456789012345678")).toBe("123456789012345678"); + expect(col.parse("1.5e2")).toBe("150"); + expect(col.parse("0.0000000001")).toBe("0.0000000001"); + expect(col.safeParse("1e-11").success).toBe(false); + expect(col.safeParse("1e999999999").success).toBe(false); + expect(schemaFor(9, 4).parse("-99999.9999")).toBe("-99999.9999"); + expect(schemaFor(9, 4).safeParse("100000").success).toBe(false); + }); +}); diff --git a/server/typescript/packages/codegen-ts/test/form-fractional-input.test.ts b/server/typescript/packages/codegen-ts/test/form-fractional-input.test.ts new file mode 100644 index 000000000..dafb5e56c --- /dev/null +++ b/server/typescript/packages/codegen-ts/test/form-fractional-input.test.ts @@ -0,0 +1,76 @@ +// A fractional field must be submittable from the generated form. +// +// `` with no `step` has an implicit step of 1, so the browser's native +// validation refuses `12.5` ("enter a valid value"). Both form tiers (the `` meta +// spread by `useEntityForm`, and the nested sub-form in codegen-ts-react) now agree: +// double / float -> type="number" step="any" +// decimal -> type="text" inputMode="decimal" (a number input parses to a double, which +// is wrong past 15 digits; the value stays a string on the wire) +// int / long / currency -> type="number", step left at its default of 1. + +import { describe, test, expect, afterAll } from "bun:test"; +import { mkdtempSync, writeFileSync, rmSync } from "node:fs"; +import { join } from "node:path"; +import { pathToFileURL } from "node:url"; +import { MetaDataLoader, InMemoryStringSource } from "@metaobjectsdev/metadata"; +import { renderEntityConstants } from "../src/templates/entity-constants.js"; +import { numericInputControl } from "../src/templates/field-meta.js"; + +const TEMP_DIRS: string[] = []; +afterAll(() => { + for (const d of TEMP_DIRS) rmSync(d, { recursive: true, force: true }); +}); + +const MODEL = { + "metadata.root": { + package: "shop", + children: [ + { + "object.entity": { + name: "Item", + children: [ + { "source.rdb": { "@table": "items" } }, + { "field.long": { name: "id" } }, + { "field.int": { name: "qty" } }, + { "field.double": { name: "weight" } }, + { "field.float": { name: "ratio" } }, + { "field.decimal": { name: "price", "@precision": 12, "@scale": 2 } }, + { "identity.primary": { name: "pk", "@fields": ["id"], "@generation": "increment" } }, + ], + }, + }, + ], + }, +}; + +type Meta = { htmlType?: string; step?: string; inputMode?: string }; + +describe("form descriptor — fractional inputs", () => { + test("emitted meta carries step / inputMode by subtype", async () => { + const loaded = await new MetaDataLoader().load([new InMemoryStringSource(JSON.stringify(MODEL))]); + expect(loaded.errors.map((e) => e.message)).toEqual([]); + const item = loaded.root.findObject("Item"); + if (item === undefined) throw new Error("Item not loaded"); + const dir = mkdtempSync(join(import.meta.dir, "tmp-form-frac-")); + TEMP_DIRS.push(dir); + const file = join(dir, "Item.constants.ts"); + writeFileSync(file, renderEntityConstants(item, "/api").toString()); + const mod = (await import(pathToFileURL(file).href)) as { Item: Record }; + expect(mod.Item.qty).toMatchObject({ htmlType: "number" }); + expect(mod.Item.qty?.step).toBeUndefined(); + expect(mod.Item.weight).toMatchObject({ htmlType: "number", step: "any" }); + expect(mod.Item.ratio).toMatchObject({ htmlType: "number", step: "any" }); + expect(mod.Item.price).toMatchObject({ htmlType: "text", inputMode: "decimal" }); + expect(mod.Item.price?.step).toBeUndefined(); + }); + + test("the shared helper is what both tiers call", () => { + expect(numericInputControl("int")).toEqual({ htmlType: "number" }); + expect(numericInputControl("long")).toEqual({ htmlType: "number" }); + expect(numericInputControl("currency")).toEqual({ htmlType: "number" }); + expect(numericInputControl("double")).toEqual({ htmlType: "number", step: "any" }); + expect(numericInputControl("float")).toEqual({ htmlType: "number", step: "any" }); + expect(numericInputControl("decimal")).toEqual({ htmlType: "text", inputMode: "decimal" }); + expect(numericInputControl("string")).toBeUndefined(); + }); +}); diff --git a/server/typescript/packages/codegen-ts/test/metaobjects-config.test.ts b/server/typescript/packages/codegen-ts/test/metaobjects-config.test.ts index 7ce4198ce..219b1bede 100644 --- a/server/typescript/packages/codegen-ts/test/metaobjects-config.test.ts +++ b/server/typescript/packages/codegen-ts/test/metaobjects-config.test.ts @@ -1,4 +1,7 @@ import { describe, test, expect, expectTypeOf } from "bun:test"; +import { guideUrl } from "@metaobjectsdev/metadata/release-docs"; +// The guide links name the tag of the running release (a prerelease keeps main). +const BLOB = guideUrl(""); import { defineConfig, normalizeConfig, resolveTargets, DEFAULT_TARGET_NAME, type MetaobjectsGenConfig, type ResolvedGenConfig, type Dialect, type ExtStyle } from "../src/metaobjects-config.js"; import type { Generator } from "../src/generator.js"; @@ -122,7 +125,7 @@ describe("columnNamingStrategy is refused (ADR-0058)", () => { test("a config still carrying it is refused, naming the value and the new setting", () => { const cfg = { outDir: "out", dbImport: "../db", dialect: "sqlite", generators: [], columnNamingStrategy: "literal" } as unknown as MetaobjectsGenConfig; expect(() => normalizeConfig(cfg)).toThrow( - 'columnNamingStrategy in metaobjects.config.ts was removed in 1.1.0. Column and table naming is one project setting now, read by every port: "naming" in .metaobjects/config.json, for example "naming": { "columns": "literal" }. Move it there and delete columnNamingStrategy in metaobjects.config.ts. See https://github.com/metaobjectsdev/metaobjects/blob/main/docs/features/migrations/physical-naming.md', + 'columnNamingStrategy in metaobjects.config.ts was removed in 1.1.0. Column and table naming is one project setting now, read by every port: "naming" in .metaobjects/config.json, for example "naming": { "columns": "literal" }. Move it there and delete columnNamingStrategy in metaobjects.config.ts. See ' + BLOB + 'docs/features/migrations/physical-naming.md', ); }); test("a config without it normalizes as before", () => { diff --git a/server/typescript/packages/codegen-ts/test/physical-naming-refusals.test.ts b/server/typescript/packages/codegen-ts/test/physical-naming-refusals.test.ts index 25f8fb475..26d8737b1 100644 --- a/server/typescript/packages/codegen-ts/test/physical-naming-refusals.test.ts +++ b/server/typescript/packages/codegen-ts/test/physical-naming-refusals.test.ts @@ -2,12 +2,15 @@ // refused, in the packaged generator and in the reference copy `meta eject` hands out, with the // same words every port prints. +import { guideUrl } from "@metaobjectsdev/metadata/release-docs"; +// The guide links name the tag of the running release (a prerelease keeps main). +const BLOB = guideUrl(""); import { describe, expect, test } from "bun:test"; import { agentDocsFile as builtinAgentDocs } from "../src/generators/agent-docs-file.js"; import { agentDocsFile as refAgentDocs } from "../src/reference/agent-docs.js"; const REFUSAL = - 'the columnNamingStrategy option of agentDocsFile() was removed in 1.1.0. Column and table naming is one project setting now, read by every port: "naming" in .metaobjects/config.json, for example "naming": { "columns": "kebab-case" }. Move it there and delete the columnNamingStrategy option of agentDocsFile(). See https://github.com/metaobjectsdev/metaobjects/blob/main/docs/features/migrations/physical-naming.md'; + 'the columnNamingStrategy option of agentDocsFile() was removed in 1.1.0. Column and table naming is one project setting now, read by every port: "naming" in .metaobjects/config.json, for example "naming": { "columns": "kebab-case" }. Move it there and delete the columnNamingStrategy option of agentDocsFile(). See ' + BLOB + 'docs/features/migrations/physical-naming.md'; describe("agentDocsFile({ columnNamingStrategy }) is refused (ADR-0058)", () => { for (const [label, factory] of [["packaged", builtinAgentDocs], ["reference copy", refAgentDocs]] as const) { diff --git a/server/typescript/packages/codegen-ts/test/projection/build-projection-views.test.ts b/server/typescript/packages/codegen-ts/test/projection/build-projection-views.test.ts index a02d5a334..64eeb8c70 100644 --- a/server/typescript/packages/codegen-ts/test/projection/build-projection-views.test.ts +++ b/server/typescript/packages/codegen-ts/test/projection/build-projection-views.test.ts @@ -358,17 +358,6 @@ describe("buildReportViews — view-backed reports (FR-044 Plan 2, Table A)", () expect(buildReportViews(root, PG)).toEqual([]); }); - test("a non-view report source kind yields nothing", async () => { - const root = await loadModel( - shop("with", (children) => { - const r = children.find((c) => (c["object.report"] as Json | undefined)?.name === "StoreTotals"); - const kids = (r!["object.report"] as { children: Json[] }).children; - (kids[0]!["source.rdb"] as Json)["@kind"] = "materializedView"; - }), - ); - expect(buildReportViews(root, PG)).toEqual([]); - }); - test("an @sql report source keeps the author's body and depends on the @from table", async () => { const body = "SELECT COUNT(*) AS purchases FROM purchases"; const root = await loadModel( diff --git a/server/typescript/packages/codegen-ts/test/projection/report-via-redeclared-field.test.ts b/server/typescript/packages/codegen-ts/test/projection/report-via-redeclared-field.test.ts new file mode 100644 index 000000000..e48577ff9 --- /dev/null +++ b/server/typescript/packages/codegen-ts/test/projection/report-via-redeclared-field.test.ts @@ -0,0 +1,76 @@ +// The rule for a `@via` dimension: the field is the EFFECTIVE field of the entity the `@via` +// reaches. When the reached subtype redeclares a field the base also declares, and `@of` names +// the base, the redeclaration wins: its column in the view, its type in the report shape. + +import { describe, test, expect } from "bun:test"; +import { MetaDataLoader, InMemoryStringSource, reportShape } from "@metaobjectsdev/metadata"; +import { buildProjectionViews } from "../../src/projection/build-projection-views.js"; +import { extractReportSpec } from "../../src/projection/extract-report-spec.js"; + +const model = { + "metadata.root": { + package: "a", + children: [ + { + "object.entity": { + name: "Base", + abstract: true, + children: [ + { "field.long": { name: "id" } }, + { "field.string": { name: "label", "@column": "base_label" } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + ], + }, + }, + { + "object.entity": { + name: "Sub", + extends: "Base", + children: [ + { "source.rdb": { "@table": "subs" } }, + { "field.int": { name: "label", "@column": "sub_label" } }, + ], + }, + }, + { + "object.entity": { + name: "Fact", + children: [ + { "source.rdb": { "@table": "facts" } }, + { "field.long": { name: "id" } }, + { "field.long": { name: "subId", "@required": true } }, + { "identity.primary": { name: "pk", "@fields": ["id"] } }, + { "identity.reference": { name: "fkSub", "@fields": ["subId"], "@references": "Sub" } }, + { "dimension.attribute": { name: "label", "@of": "Base.label", "@via": "Fact.fkSub" } }, + { "measure.aggregate": { name: "facts", "@agg": "count", "@of": "Fact.id" } }, + ], + }, + }, + { + "object.report": { + name: "FactLabels", + "@from": "Fact", + "@dimensions": ["label"], + "@measures": ["facts"], + children: [{ "source.rdb": { "@kind": "view", "@view": "v_fact_labels" } }], + }, + }, + ], + }, +}; + +describe("a @via dimension whose @of names the base of a subtype that redeclares the field", () => { + test("shape, spec and view all use the redeclared field of the reached entity", async () => { + const { root, errors } = await new MetaDataLoader().load([new InMemoryStringSource(JSON.stringify(model))]); + expect(errors).toEqual([]); + const rep = root.objects().find((o) => o.name === "FactLabels")!; + const shape = reportShape(rep, root); + expect(shape.fields.find((f) => f.name === "label")!.subType).toBe("int"); + const spec = extractReportSpec(rep, root, { columnNamingStrategy: "snake_case" }); + const col = spec.columns.find((c) => c.fieldName === "label")!; + expect(col.kind === "dimension" && col.ref.endsWith(".sub_label")).toBe(true); + const sql = buildProjectionViews(root, { dialect: "postgres" }).find((v) => v.name === "v_fact_labels")!.sql!; + expect(sql).toContain("sub_label"); + expect(sql).not.toContain("base_label"); + }); +}); diff --git a/server/typescript/packages/codegen-ts/test/projection/routes-file.test.ts b/server/typescript/packages/codegen-ts/test/projection/routes-file.test.ts index 5efc28265..ffd1a92d2 100644 --- a/server/typescript/packages/codegen-ts/test/projection/routes-file.test.ts +++ b/server/typescript/packages/codegen-ts/test/projection/routes-file.test.ts @@ -10,7 +10,7 @@ import { MetaDataLoader, InMemoryStringSource, loadUris, reportReadModel } from import type { MetaObject, MetaRoot } from "@metaobjectsdev/metadata"; import { renderRoutesFile } from "../../src/templates/routes-file.js"; import { renderRoutesFileHono } from "../../src/templates/routes-file-hono.js"; -import { hasGeneratedForm, hasItemRoute, itemRouteField, reportDecimalColumns, servesClientHooks, servesClientTier, servesReadApi } from "../../src/api-surface.js"; +import { hasGeneratedForm, decimalFieldNames, hasItemRoute, itemRouteField, servesClientHooks, servesClientTier, servesReadApi } from "../../src/api-surface.js"; import { renderQueriesFile } from "../../src/templates/queries-file.js"; import { servedReport } from "../../src/source-detect.js"; import { hasUiSurface } from "../../src/generators/agent-ui-page.js"; @@ -698,36 +698,33 @@ describe("renderRoutesFile — a served report (FR-044 Plan 3)", () => { pkMap: buildPkMap(root), relationMap: buildRelationMap(root), }); - test("every decimal field of the report is named, on both route flavours and on SQLite only", async () => { + test("every decimal field of the report is named, on both route flavours and on every dialect", async () => { const root = await ratioModel(); const model = reportReadModel(declared(root, "InvoiceTotals"), root); // Not vacuous: the ratio and the average really are decimal fields of the read model. expect(model.findField("paidShare")?.subType).toBe("decimal"); expect(model.findField("avgAmount")?.subType).toBe("decimal"); - expect(reportDecimalColumns(model, "sqlite")).toEqual(["avgAmount", "paidShare"]); - const ctx = sqliteCtx(root); - for (const out of [renderRoutesFile(model, ctx), renderRoutesFileHono(model, ctx)]) { - expect(out).toContain('decimalColumns: ["avgAmount", "paidShare"],'); - } - for (const dialect of ["postgres", "mysql"] as const) { - expect(reportDecimalColumns(model, dialect)).toEqual([]); - const other = makeRenderContext({ + expect(decimalFieldNames(model)).toEqual(["avgAmount", "paidShare"]); + // ADR-0019 Amendment 1: a decimal is a canonical string on EVERY dialect (Postgres and + // MySQL return the column's scale), so the names are passed whatever the dialect. + for (const dialect of ["sqlite", "postgres", "mysql"] as const) { + const ctx = makeRenderContext({ dialect, loadedRoot: root, outDir: "/x", dbImport: "~/db", pkMap: buildPkMap(root), relationMap: buildRelationMap(root), }); - expect(renderRoutesFile(model, other)).not.toContain("decimalColumns"); - expect(renderRoutesFileHono(model, other)).not.toContain("decimalColumns"); + for (const out of [renderRoutesFile(model, ctx), renderRoutesFileHono(model, ctx)]) { + expect(out).toContain('decimalColumns: ["avgAmount", "paidShare"],'); + } } }); - test("a report with no decimal field, a projection and an entity pass none", async () => { + test("a report with no decimal field passes none; a projection and an entity name their decimals", async () => { const root = await loadReportingModel(); const storeTotals = reportReadModel(declared(root, "StoreTotals"), root); // StoreTotals' measures are counts and a currency sum: no decimal. - expect(reportDecimalColumns(storeTotals, "sqlite")).toEqual([]); + expect(decimalFieldNames(storeTotals)).toEqual([]); expect(renderRoutesFile(storeTotals, sqliteCtx(root))).not.toContain("decimalColumns"); - // A projection on SQLite is released behaviour and is not covered. - expect(reportDecimalColumns(declared(root, "Program"), "sqlite")).toEqual([]); + expect(decimalFieldNames(declared(root, "Program"))).toEqual([]); }); }); diff --git a/server/typescript/packages/codegen-ts/test/reference-byte-identical.test.ts b/server/typescript/packages/codegen-ts/test/reference-byte-identical.test.ts index 71ca27e50..b70e788ff 100644 --- a/server/typescript/packages/codegen-ts/test/reference-byte-identical.test.ts +++ b/server/typescript/packages/codegen-ts/test/reference-byte-identical.test.ts @@ -780,7 +780,10 @@ describe("ADR-0034 — the cube-model reference is byte-identical to the built-i // facts cube; ProgramSessions reaches its facts through the Session -> Week reference. expect("tree" in a ? a.tree.map(([p]) => p) : a).toEqual([ "model/cubes/AllTypes.yml", "model/cubes/Asset.yml", "model/cubes/Auth.yml", "model/cubes/BridgeAuth.yml", - "model/cubes/CopayAuth.yml", "model/cubes/Measurement.yml", "model/cubes/Node.yml", "model/cubes/Node_fkParent.yml", + "model/cubes/CopayAuth.yml", + "model/cubes/Entry.yml", + "model/cubes/Gadget.yml", + "model/cubes/GadgetEvent.yml", "model/cubes/Measurement.yml", "model/cubes/Node.yml", "model/cubes/Node_fkParent.yml", "model/cubes/PriorAuthAuth.yml", "model/cubes/Program.yml", "model/cubes/ProgramFlagRosterFacts.yml", "model/cubes/ProgramLongWeeksFacts.yml", "model/cubes/ProgramRosterFacts.yml", "model/cubes/ProgramSessionsFacts.yml", "model/cubes/ProgramSessions_fkWeek.yml", "model/cubes/ReferralAuth.yml", "model/cubes/Session.yml", diff --git a/server/typescript/packages/codegen-ts/test/templates/tph-tier2-routes.test.ts b/server/typescript/packages/codegen-ts/test/templates/tph-tier2-routes.test.ts index 9736d4e63..a1dd54056 100644 --- a/server/typescript/packages/codegen-ts/test/templates/tph-tier2-routes.test.ts +++ b/server/typescript/packages/codegen-ts/test/templates/tph-tier2-routes.test.ts @@ -101,6 +101,15 @@ describe("FR-017 Tier 2 — TPH routes file", () => { expect(out).toContain("BridgeAuthInsertSchema.omit({ type: true })"); }); + test("the polymorphic mount sends a decimal only a subtype declares as its canonical string", async () => { + const { root, base } = await loadTph(); + const out = renderRoutesFile(base, ctxFor(root)); + // The base declares no decimal, but its list/get serve CopayAuth rows, so the polymorphic + // mount must rewrite `copayAmount` too (ADR-0019 Amendment 1; Postgres returns "12.50"). + const polymorphic = out.slice(out.indexOf("path: Auth.$path,"), out.indexOf('Auth.$path + "/')); + expect(polymorphic).toContain('decimalColumns: ["copayAmount"]'); + }); + test("routesFile generator skips TPH subtypes", async () => { const { bridge, base } = await loadTph(); const gen = routesFile(); diff --git a/server/typescript/packages/conformance/src/adapter.ts b/server/typescript/packages/conformance/src/adapter.ts index 2e77e73d1..49ee548d0 100644 --- a/server/typescript/packages/conformance/src/adapter.ts +++ b/server/typescript/packages/conformance/src/adapter.ts @@ -25,6 +25,12 @@ export type TreeHandle = unknown; */ export interface ErrorEnvelopeRecord { readonly code: string; + /** + * The error's human-readable message. Optional: only needed when a fixture's + * expected entry carries `messageIncludes` (the one case message text is + * asserted; see spec/conformance-tests.md). + */ + readonly message?: string; readonly source: { readonly format: string; readonly files: readonly string[]; diff --git a/server/typescript/packages/conformance/src/operation-script.ts b/server/typescript/packages/conformance/src/operation-script.ts index 18038cb2b..7dbb43f07 100644 --- a/server/typescript/packages/conformance/src/operation-script.ts +++ b/server/typescript/packages/conformance/src/operation-script.ts @@ -24,6 +24,11 @@ export interface ExpectedError { readonly code: string; /** Present in the FR5a envelope shape; absent in legacy `[{code}]` form. */ readonly source?: ExpectedErrorSource; + /** + * Optional. Every listed substring MUST appear (case-sensitive) in the + * matching error's message. The only place message text is asserted. + */ + readonly messageIncludes?: readonly string[]; } export interface ExpectedErrorsEnvelope { @@ -84,6 +89,14 @@ function parseSource(raw: unknown, i: number): ExpectedErrorSource | undefined { return out; } +function parseMessageIncludes(raw: unknown, i: number): readonly string[] | undefined { + if (raw === undefined) return undefined; + if (!Array.isArray(raw) || raw.length === 0 || !raw.every((x) => typeof x === "string" && x !== "")) { + throw new Error(`expected-errors.json entry ${i}: 'messageIncludes' must be a non-empty string[] of non-empty strings`); + } + return raw as string[]; +} + /** * Parse + validate a parsed expected-errors.json value. * @@ -128,7 +141,12 @@ export function parseExpectedErrors(raw: unknown): ExpectedErrorsEnvelope { throw new Error(`expected-errors.json entry ${i} missing string 'code' field`); } const source = parseSource(r.source, i); - return source !== undefined ? { code: r.code, source } : { code: r.code }; + const messageIncludes = parseMessageIncludes(r.messageIncludes, i); + return { + code: r.code, + ...(source !== undefined ? { source } : {}), + ...(messageIncludes !== undefined ? { messageIncludes } : {}), + }; }); const rawWarnings = Array.isArray(env.warnings) ? env.warnings : []; const warnings = rawWarnings.map((item, i): ExpectedError => { diff --git a/server/typescript/packages/conformance/src/runner.ts b/server/typescript/packages/conformance/src/runner.ts index 18f1edb0f..4db8b8c33 100644 --- a/server/typescript/packages/conformance/src/runner.ts +++ b/server/typescript/packages/conformance/src/runner.ts @@ -71,6 +71,15 @@ export async function runFixture( }); continue; } + for (const needle of w.messageIncludes ?? []) { + if (!(g.message ?? "").includes(needle)) { + checks.push({ + kind: "expected-errors", + passed: false, + detail: `envelope[${i}].messageIncludes: message does not contain '${needle}' (got '${g.message ?? ""}')`, + }); + } + } if (w.source === undefined) continue; if (w.source.format !== g.source.format) { checks.push({ diff --git a/server/typescript/packages/conformance/test/runner.test.ts b/server/typescript/packages/conformance/test/runner.test.ts index b87785a83..c6fe663cf 100644 --- a/server/typescript/packages/conformance/test/runner.test.ts +++ b/server/typescript/packages/conformance/test/runner.test.ts @@ -380,3 +380,53 @@ test("expected-errors warnings: count mismatch → fail with warnings count deta const detail = report.checks.filter((c) => !c.passed).map((c) => c.detail).join("; "); expect(detail).toMatch(/warnings count: expected 2, got 1/); }); + +// ── messageIncludes — optional per-error message-substring assertion ──────── + +function messageAdapter(message: string | undefined): ConformanceAdapter { + return { + ...fake, + async loadFixture() { + return { + errorCodes: ["ERR_UNRESOLVED_SUPER"], + warnings: [], + errors: [{ + code: "ERR_UNRESOLVED_SUPER", + ...(message !== undefined ? { message } : {}), + source: { format: "resolved", files: ["meta.a.json"], referrer: "W", target: "B" }, + }], + }; + }, + }; +} + +async function messageFixture() { + const root = await fixture({ + "input/m.json": "{}", + "expected-errors.json": JSON.stringify({ + errors: [{ + code: "ERR_UNRESOLVED_SUPER", + messageIncludes: ["app::a::B", "extends"], + source: { format: "resolved", files: ["meta.a.json"], referrer: "W", target: "B" }, + }], + warnings: [], + }), + }); + return (await discoverFixtures(root))[0]!; +} + +test("messageIncludes: every substring present → pass", async () => { + const report = await runFixture(await messageFixture(), messageAdapter('exists in: app::a::B. write extends: "app::a::B"')); + expect(report.status).toBe("pass"); +}); + +test("messageIncludes: a missing substring → fail naming it", async () => { + const report = await runFixture(await messageFixture(), messageAdapter("the SuperClass 'B' does not exist")); + expect(report.status).toBe("fail"); + expect(report.checks.find((c) => !c.passed)!.detail).toContain("app::a::B"); +}); + +test("messageIncludes: adapter reporting no message → fail", async () => { + const report = await runFixture(await messageFixture(), messageAdapter(undefined)); + expect(report.status).toBe("fail"); +}); diff --git a/server/typescript/packages/integration-tests/cube-live/cube-live-support.ts b/server/typescript/packages/integration-tests/cube-live/cube-live-support.ts index ad7e4a6a5..0feed68ac 100644 --- a/server/typescript/packages/integration-tests/cube-live/cube-live-support.ts +++ b/server/typescript/packages/integration-tests/cube-live/cube-live-support.ts @@ -13,7 +13,18 @@ import { type MetaObject, type MetaRoot, } from "@metaobjectsdev/metadata"; -import type { CubeStack } from "../src/cube-container.ts"; + +/** + * Where a Cube API answers: a development stack (no auth) or a production one (a JWT in + * `headers`). `CubeStack` (src/cube-container.ts) is one; cube-prod-stack.ts's `ProdCube` is the other. + */ +export interface CubeEndpoint { + /** `http://127.0.0.1:/cubejs-api/v1`. */ + readonly apiBase: string; + /** Request headers the API requires; absent in development mode. */ + readonly headers?: Readonly>; + cubeLogs(lines?: number): string; +} /** One HTTP request to Cube; a stalled socket is aborted and counts as a transport error. */ export const REQUEST_TIMEOUT_MS = 60_000; @@ -42,6 +53,8 @@ export interface CubeQuery { readonly dimensions: readonly string[]; readonly timeDimensions: readonly CubeTimeDimension[]; readonly segments: readonly string[]; + /** Cube query filters (the MySQL cases pick rows with them). */ + readonly filters?: readonly { readonly member: string; readonly operator: string; readonly values: readonly string[] }[]; /** Table I fixes the query time zone at UTC (Table J: another zone re-buckets). */ readonly timezone: "UTC"; } @@ -280,16 +293,29 @@ export interface MetaResponse { export interface LoadResponse { readonly data: Array>; readonly usedPreAggregations?: Record; + /** True when Cube answered from Cube Store (a rollup built there) rather than from the data source. */ + readonly external?: boolean; + /** The external store that answered: `cubestore` in production mode. */ + readonly extDbType?: string; } /** A request that never produced a JSON answer: refused, reset, aborted, or a body that is not JSON. */ export class TransportError extends Error {} -export async function getJson(url: string, timeoutMs = REQUEST_TIMEOUT_MS): Promise<{ status: number; body: unknown }> { +export async function getJson( + url: string, + timeoutMs = REQUEST_TIMEOUT_MS, + headers: Readonly> = {}, + post?: unknown, +): Promise<{ status: number; body: unknown }> { let res: Response; let text: string; try { - res = await fetch(url, { signal: AbortSignal.timeout(Math.max(1000, timeoutMs)) }); + res = await fetch(url, { + headers: post === undefined ? { ...headers } : { ...headers, "Content-Type": "application/json" }, + ...(post === undefined ? {} : { method: "POST", body: JSON.stringify(post) }), + signal: AbortSignal.timeout(Math.max(1000, timeoutMs)), + }); text = await res.text(); } catch (e) { throw new TransportError(`${url.split("?")[0]}: ${e instanceof Error ? `${e.name}: ${e.message}` : String(e)}`, { cause: e }); @@ -307,8 +333,8 @@ export function errorOf(body: unknown): string | undefined { return e === undefined ? undefined : String(e); } -export async function getMeta(s: CubeStack, timeoutMs = REQUEST_TIMEOUT_MS): Promise { - const { body } = await getJson(`${s.apiBase}/meta`, timeoutMs); +export async function getMeta(s: CubeEndpoint, timeoutMs = REQUEST_TIMEOUT_MS): Promise { + const { body } = await getJson(`${s.apiBase}/meta`, timeoutMs, s.headers); const error = errorOf(body); if (error !== undefined) return { error, cubes: [] }; const cubes = (body as { cubes?: unknown }).cubes; @@ -316,7 +342,7 @@ export async function getMeta(s: CubeStack, timeoutMs = REQUEST_TIMEOUT_MS): Pro return { cubes: cubes as MetaCube[] }; } -export function queryUrl(s: CubeStack, path: "load" | "sql", query: CubeQuery): string { +export function queryUrl(s: CubeEndpoint, path: "load" | "sql", query: CubeQuery): string { return `${s.apiBase}/${path}?query=${encodeURIComponent(JSON.stringify(query))}`; } @@ -325,13 +351,13 @@ export function queryUrl(s: CubeStack, path: "load" | "sql", query: CubeQuery): * with backoff after a transport error (at most LOAD_TRANSPORT_RETRIES in a row), all within * LOAD_DEADLINE_MS. */ -export async function load(s: CubeStack, query: CubeQuery): Promise { +export async function load(s: CubeEndpoint, query: CubeQuery): Promise { const deadline = Date.now() + LOAD_DEADLINE_MS; let transportErrors = 0; for (;;) { let answer: { status: number; body: unknown }; try { - answer = await getJson(queryUrl(s, "load", query)); + answer = await getJson(queryUrl(s, "load", query), REQUEST_TIMEOUT_MS, s.headers); transportErrors = 0; } catch (e) { if (!(e instanceof TransportError)) throw e; @@ -359,8 +385,8 @@ export async function load(s: CubeStack, query: CubeQuery): Promise { - const { status, body } = await getJson(queryUrl(s, "sql", query)); +export async function generatedSql(s: CubeEndpoint, query: CubeQuery): Promise { + const { status, body } = await getJson(queryUrl(s, "sql", query), REQUEST_TIMEOUT_MS, s.headers); const sql = (body as { sql?: { sql?: unknown } }).sql?.sql; if (!Array.isArray(sql) || typeof sql[0] !== "string") { throw new Error(`/v1/sql answered HTTP ${status} without SQL: ${JSON.stringify(body).slice(0, 2000)}`); @@ -369,7 +395,7 @@ export async function generatedSql(s: CubeStack, query: CubeQuery): Promise { +export async function cubeSql(s: CubeEndpoint, query: CubeQuery): Promise { try { return await generatedSql(s, query); } catch (e) { @@ -397,11 +423,144 @@ export interface ExpectedReport { * and the view NOT to, so the difference stays an executed check rather than a skipped one. */ readonly cubeRows?: readonly Record[]; - /** - * The model declares a rollup for this report and Cube never matches it, so it answers from the - * source table (`preAggregation` is then undefined). Today that is a report listing ONE time - * dimension at several grains: the `time_dimensions` entry repeats the dimension and Cube 1.7.43 - * matches no query to it (a query names a time dimension once, at one granularity). - */ - readonly unusedRollup?: true; +} + +/** + * Every served canonical report, its view, the rollup Cube answers it from (undefined = the + * source tables) and its row count on the seed. Postgres and MySQL agree on all three, so both + * lanes, in both Cube modes, read this one table. `preAggregation` is written for development + * mode; {@link productionPreAggregation} names the same rollup in production mode. + */ +export const CANONICAL_REPORTS: readonly ExpectedReport[] = [ + { report: "ProgramMinutes", view: "v_program_minutes", preAggregation: "dev_pre_aggregations.week__program_minutes", rows: 3 }, + { report: "FitnessTotals", view: "v_fitness_totals", preAggregation: "dev_pre_aggregations.week__fitness_totals", rows: 1 }, + { report: "ProgramsByMonth", view: "v_programs_by_month", preAggregation: "dev_pre_aggregations.program__programs_by_month", rows: 5 }, + { report: "ProgramsByWeek", view: "v_programs_by_week", preAggregation: "dev_pre_aggregations.program__programs_by_week", rows: 5 }, + { report: "RecentPrograms", view: "v_recent_programs", preAggregation: undefined, rows: 1 }, + { report: "AssetActivity", view: "v_asset_activity", preAggregation: "dev_pre_aggregations.asset__asset_activity", rows: 2 }, + { report: "ProgramRoster", view: "v_program_roster", preAggregation: undefined, rows: 11 }, + { report: "ProgramLongWeeks", view: "v_program_long_weeks", preAggregation: undefined, rows: 11 }, + { report: "ProgramFlags", view: "v_program_flags", preAggregation: "dev_pre_aggregations.week__program_flags", rows: 3 }, + { report: "ProgramFlagRoster", view: "v_program_flag_roster", preAggregation: undefined, rows: 11 }, + { report: "FitnessTotalsFilled", view: "v_fitness_totals_filled", preAggregation: "dev_pre_aggregations.week__fitness_totals_filled", rows: 1 }, + { report: "MeasurementStats", view: "v_measurement_stats", preAggregation: "dev_pre_aggregations.measurement__measurement_stats", rows: 1 }, + { report: "MeasurementStatsFilled", view: "v_measurement_stats_filled", preAggregation: "dev_pre_aggregations.measurement__measurement_stats_filled", rows: 1 }, + { report: "AssetDays", view: "v_asset_days", preAggregation: "dev_pre_aggregations.asset__asset_days", rows: 1 }, + { report: "AllTypesBreakdown", view: "v_all_types_breakdown", preAggregation: undefined, rows: 2 }, + // A report listing createdAt at day, quarter and year has one rollup per grain; the Table F query + // asks the finest (day), so it is answered from the day rollup. The other two are proven by the + // production-mode lane's per-grain queries. + { report: "ProgramsOverTime", view: "v_programs_over_time", preAggregation: "dev_pre_aggregations.program__programs_over_time_day", rows: 7 }, + { report: "LiveIntroPrograms", view: "v_live_intro_programs", preAggregation: "dev_pre_aggregations.program__live_intro_programs", rows: 2 }, + // No dimension, no row in scope: the view reads one row with the count 0 and the default; Cube + // sums the (empty) rollup, which is null for every measure, the count and the default included. + { report: "ProgramsEmptyScope", view: "v_programs_empty_scope", preAggregation: "dev_pre_aggregations.program__programs_empty_scope", rows: 1, cubeRows: [{ avgPriceOrMinusOne: null, programs: null }] }, + { report: "WeekShapes", view: "v_week_shapes", preAggregation: "dev_pre_aggregations.week__week_shapes", rows: 3 }, + { report: "SessionsByProgram", view: "v_sessions_by_program", preAggregation: "dev_pre_aggregations.session__sessions_by_program", rows: 3 }, + { report: "ProgramSessions", view: "v_program_sessions", preAggregation: undefined, rows: 11 }, + { report: "NodesByParent", view: "v_nodes_by_parent", preAggregation: "dev_pre_aggregations.node__nodes_by_parent", rows: 3 }, + { report: "AuthsByKind", view: "v_auths_by_kind", preAggregation: "dev_pre_aggregations.auth__auths_by_kind", rows: 3 }, + // A dimension reached through an abstract parent's redeclared field (Gadget.label, column gadget_code): 3 gadgets with events. + { report: "GadgetEventsByLabel", view: "v_gadget_events_by_label", preAggregation: "dev_pre_aggregations.gadget_event__gadget_events_by_label", rows: 3 }, + // Exact-decimal measures over four buckets (A, B, C with all-null amounts, D). + { report: "EntryTotals", view: "v_entry_totals", preAggregation: "dev_pre_aggregations.entry__entry_totals", rows: 4 }, +]; + +/** The rollup table Cube's production mode names for a development-mode name: `prod_pre_aggregations.`. */ +export function productionPreAggregation(dev: string): string { + return dev.replace(/^dev_pre_aggregations\./, "prod_pre_aggregations."); +} + +/** + * `SELECT * FROM ` through psql, every value as the TEXT Postgres prints for it (a numeric + * keeps its scale, a timestamptz its offset), in a UTC session. + */ +export function selectViewPg(s: { psql(sqlText: string): string }, view: string): Array> { + return selectSqlPg(s, `SELECT * FROM "${view}"`); +} + +/** The rows of any `SELECT` through psql, as {@link selectViewPg} prints them. */ +export function selectSqlPg(s: { psql(sqlText: string): string }, select: string): Array> { + const out = s.psql( + `SET TIME ZONE 'UTC';\n` + + `SELECT coalesce(json_agg(t.o), '[]'::json)::text FROM (\n` + + ` SELECT (SELECT json_object_agg(e.key, e.value) FROM json_each_text(row_to_json(v)) e) AS o\n` + + ` FROM (${select}) v\n` + + `) t;\n`, + ); + const line = out.trim().split("\n").filter((l) => l.trim() !== "").pop() ?? ""; + const parsed: unknown = JSON.parse(line); + if (!Array.isArray(parsed)) throw new Error(`psql did not print a JSON array: ${line.slice(0, 200)}`); + return parsed as Array>; +} + +/** + * `SELECT * FROM ` through the mysql client, every value as the text MySQL prints for it. A + * DATETIME prints `2026-05-04 03:30:00.000`; the space becomes `T` so the shared normalization + * reads it as a UTC instant, as it does Cube's `2026-05-04T03:30:00.000`. + */ +export function selectViewMysql(s: { mysql(sqlText: string): string }, view: string): Array> { + const columns = s.mysql(`SHOW COLUMNS FROM \`${view}\`;`).trim().split("\n").map((l) => l.split("\t")[0] ?? ""); + return selectSqlMysql(s, `SELECT ${columns.map((c) => `\`${c}\``).join(", ")} FROM \`${view}\``, columns); +} + +/** The rows of any `SELECT` through the mysql client, named `columns` in order. */ +export function selectSqlMysql(s: { mysql(sqlText: string): string }, select: string, columns: readonly string[]): Array> { + const out = s.mysql(`SET time_zone = '+00:00'; ${select};`); + return out + .split("\n") + .filter((l) => l !== "") + .map((line) => { + const cells = line.split("\t"); + const row: Record = {}; + columns.forEach((c, i) => { + const v = cells[i] ?? "NULL"; + row[c] = v === "NULL" ? null : /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}/.test(v) ? v.replace(" ", "T") : v; + }); + return row; + }); +} + +// --------------------------------------------------------------------------------------------- +// Production mode: building rollups. +// --------------------------------------------------------------------------------------------- + +/** One pre-aggregation build job as the jobs API reports it. */ +interface BuildJob { + readonly table?: string; + readonly status?: string; +} + +/** + * Build every pre-aggregation of the model into Cube Store through the pre-aggregations jobs API + * (`POST /v1/pre-aggregations/jobs`, `action: post`, then `action: get` until each job is `done`). + * Production mode answers from a rollup only once it is built; this makes "built" a fact the test + * waits for rather than a race it hopes to win. Returns the tables built (`schema.table`). + */ +export async function buildAllPreAggregations(s: CubeEndpoint, timeoutMs = 600_000): Promise { + const url = `${s.apiBase}/pre-aggregations/jobs`; + const started = await getJson( + url, + REQUEST_TIMEOUT_MS, + s.headers, + { action: "post", selector: { contexts: [{ securityContext: {} }], timezones: ["UTC"] } }, + ); + const tokens = started.body; + if (!Array.isArray(tokens) || tokens.length === 0 || tokens.some((t) => typeof t !== "string")) { + throw new Error(`the jobs API did not hand back job tokens (HTTP ${started.status}): ${JSON.stringify(started.body).slice(0, 500)}\n${s.cubeLogs(40)}`); + } + const deadline = Date.now() + timeoutMs; + for (;;) { + const { body } = await getJson(url, REQUEST_TIMEOUT_MS, s.headers, { action: "get", resType: "object", tokens }); + const jobs = Object.values((body ?? {}) as Record); + const failed = jobs.filter((j) => typeof j.status === "string" && j.status.startsWith("failure")); + if (failed.length > 0) { + throw new Error(`pre-aggregation build failed: ${failed.map((j) => `${j.table}: ${j.status}`).join("; ")}\n${s.cubeLogs(60)}`); + } + if (jobs.length === tokens.length && jobs.every((j) => j.status === "done")) return jobs.map((j) => String(j.table)).sort(); + if (Date.now() > deadline) { + throw new Error(`pre-aggregations were not built within ${timeoutMs / 1000}s: ${JSON.stringify(body).slice(0, 800)}\n${s.cubeLogs(60)}`); + } + await Bun.sleep(2000); + } } diff --git a/server/typescript/packages/integration-tests/cube-live/cube-model-mysql-cases.live.ts b/server/typescript/packages/integration-tests/cube-live/cube-model-mysql-cases.live.ts new file mode 100644 index 000000000..a88ad5d11 --- /dev/null +++ b/server/typescript/packages/integration-tests/cube-live/cube-model-mysql-cases.live.ts @@ -0,0 +1,388 @@ +/** + * The three MySQL corpus cases, EXECUTED on a real MySQL 8.4 through Cube. + * + * `fixtures/cube-model/{dimension-time-mysql,measure-mysql,measure-boolean-mysql}` pin the SQL the + * exporter writes for MySQL where it differs from Postgres: a date as `CAST(… AS DATETIME)`, a + * tuple distinct count as `COUNT(DISTINCT a, b)` (and with a condition as a `CASE` on the first + * component), a ratio with no cast, a boolean as `MAX`/`MIN` with `COALESCE(…, 0)`. cube-model.live.ts + * loads them against its Postgres data source, which proves they COMPILE (a compile runs no SQL); + * until this file, nothing ran their SQL, only goldens held it. + * + * Here each case's generated tree (`fixtures/cube-model//expected/model`, which + * `cube-model-corpus.test.ts` holds equal to what the generator writes) is loaded into the pinned + * Cube over a private MySQL 8.4, the tables the case's model names are created and filled below, and + * each query is run through Cube. + * + * WHY HAND-CHECKED ROWS, NOT A VIEW. The canonical lane (cube-model-mysql.live.ts) compares Cube's + * rows with `SELECT *` from a report view `buildReportViews` lowers. These cases declare no + * `object.report`, so there is no view to compare with. The oracle is instead the rows worked out + * by hand from the seed data below (each expected value carries its derivation), which is + * independent of the generator and of Cube, and which a wrong spelling of the SQL (a tuple count + * that counts a `NULL` component, a boolean that comes back as a number, a date bucketed in the + * wrong zone) cannot satisfy. + * + * `*.live.ts`: runs only by its path, from `bun run test:cube`. Without docker every test is + * SKIPPED behind a banner. + */ + +import { afterAll, beforeAll, describe, expect, test } from "bun:test"; +import { YAML } from "bun"; +import { cpSync, existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, renameSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { CUBE_IMAGE, CUBE_MYSQL_IMAGE, cubeStackStartBudgetMs, dockerUnavailableReason, startCubeStack, type CubeStack } from "../src/cube-container.ts"; +import { type CubeQuery, canonicalDecimal, getMeta, load } from "./cube-live-support.ts"; + +const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "..", "..", "..", "..", ".."); +const CORPUS = join(REPO_ROOT, "fixtures", "cube-model"); +const TEST_TIMEOUT_MS = 300_000; + +/** The tables the three cases' models name (`events`, `weeks`, `calls`), snake_case columns as the cases declare. */ +const SCHEMA_AND_SEED = ` +CREATE TABLE events ( + id BIGINT NOT NULL PRIMARY KEY, + occurred_at DATETIME(3) NOT NULL, + on_date DATE NOT NULL, + starts_on DATE NOT NULL +); +INSERT INTO events (id, occurred_at, on_date, starts_on) VALUES + (1, '2026-05-01 08:30:00.000', '2026-05-10', '2026-06-15'), + (2, '2026-05-01 09:15:00.000', '2026-05-20', '2026-06-15'), + (3, '2026-05-02 23:59:59.000', '2026-06-03', '2026-07-01'), + (4, '2026-05-02 00:00:00.000', '2027-01-05', '2026-07-01'); + +CREATE TABLE weeks ( + id BIGINT NOT NULL PRIMARY KEY, + program_id BIGINT NOT NULL, + duration_minutes INT NOT NULL, + label VARCHAR(50) NULL +); +INSERT INTO weeks (id, program_id, duration_minutes, label) VALUES + (1, 1, 30, 'a'), + (2, 1, 60, 'a'), + (3, 2, 60, 'b'), + (4, 2, 90, 'c'), + (5, 3, 90, NULL), + (6, 3, 30, 'b'), + (7, 1, 30, 'a'); + +CREATE TABLE calls ( + id BIGINT NOT NULL PRIMARY KEY, + failed TINYINT(1) NOT NULL, + duration_ms INT NOT NULL +); +INSERT INTO calls (id, failed, duration_ms) VALUES + (1, 0, 50), + (2, 1, 2000), + (3, 0, 1500), + (4, 0, 80); +`; + +type Row = Record; + +interface Check { + readonly name: string; + readonly query: Omit; + /** The rows by hand, keyed by the Cube member as Cube names it in a row; order is not significant. */ + readonly rows: readonly Row[]; +} + +interface Case { + readonly name: string; + /** The cube the case declares (the model is loaded when /v1/meta lists exactly it). */ + readonly cube: string; + readonly checks: readonly Check[]; +} + +const q = (parts: Partial>): Omit => ({ + measures: [], dimensions: [], timeDimensions: [], segments: [], ...parts, +}); + +const CASES: readonly Case[] = [ + { + // events: occurred_at (DATETIME), on_date and starts_on (DATE). MySQL's CAST has no TIMESTAMP + // target, so a date dimension is `CAST(… AS DATETIME)`: a date reads as that day at midnight. + name: "dimension-time-mysql", + cube: "Event", + checks: [ + { + name: "occurredAt by hour (a DATETIME column, bucketed in UTC)", + query: q({ timeDimensions: [{ dimension: "Event.occurredAt", granularity: "hour" }] }), + // 08:30 and 09:15 on May 1 are two hours; 23:59:59 and 00:00:00 on May 2 are two more. + rows: [ + { "Event.occurredAt.hour": "2026-05-01T08:00:00" }, + { "Event.occurredAt.hour": "2026-05-01T09:00:00" }, + { "Event.occurredAt.hour": "2026-05-02T00:00:00" }, + { "Event.occurredAt.hour": "2026-05-02T23:00:00" }, + ], + }, + { + name: "occurredAt by day", + query: q({ timeDimensions: [{ dimension: "Event.occurredAt", granularity: "day" }] }), + rows: [{ "Event.occurredAt.day": "2026-05-01T00:00:00" }, { "Event.occurredAt.day": "2026-05-02T00:00:00" }], + }, + { + name: "onDate by month (a DATE column, CAST AS DATETIME)", + query: q({ timeDimensions: [{ dimension: "Event.onDate", granularity: "month" }] }), + // 2026-05-10 and 2026-05-20 are May; 2026-06-03 is June; 2027-01-05 is January 2027. + rows: [ + { "Event.onDate.month": "2026-05-01T00:00:00" }, + { "Event.onDate.month": "2026-06-01T00:00:00" }, + { "Event.onDate.month": "2027-01-01T00:00:00" }, + ], + }, + { + name: "onDate by year", + query: q({ timeDimensions: [{ dimension: "Event.onDate", granularity: "year" }] }), + rows: [{ "Event.onDate.year": "2026-01-01T00:00:00" }, { "Event.onDate.year": "2027-01-01T00:00:00" }], + }, + { + name: "startsOn as an attribute dimension (a date is a Cube time member: the day at midnight)", + query: q({ dimensions: ["Event.startsOn"] }), + rows: [{ "Event.startsOn": "2026-06-15T00:00:00" }, { "Event.startsOn": "2026-07-01T00:00:00" }], + }, + ], + }, + { + // weeks: 7 rows. program_id/duration pairs: (1,30) (1,60) (2,60) (2,90) (3,90) (3,30) (1,30); + // labels a a b c NULL b a. + name: "measure-mysql", + cube: "Week", + checks: [ + { + name: "every measure over all rows", + query: q({ measures: ["Week.weeks", "Week.labels", "Week.slots", "Week.longSlots", "Week.totalMinutes", "Week.longWeeks", "Week.longShare"] }), + rows: [ + { + "Week.weeks": "7", // count(id) + "Week.labels": "3", // distinct label: a, b, c (the NULL is not counted) + "Week.slots": "6", // distinct (program, minutes): the repeated (1,30) counts once + "Week.longSlots": "4", // distinct pairs with minutes >= 60: (1,60) (2,60) (2,90) (3,90); the CASE is NULL for the rest, and COUNT(DISTINCT) skips a NULL component + "Week.totalMinutes": "390", // 30+60+60+90+90+30+30 + "Week.longWeeks": "4", // rows with minutes >= 60: ids 2, 3, 4, 5 + "Week.longShare": "0.5714", // 4 / 7, MySQL's DECIMAL division + }, + ], + }, + { + name: "the long segment", + query: q({ measures: ["Week.weeks"], segments: ["Week.long"] }), + rows: [{ "Week.weeks": "4" }], + }, + { + name: "a ratio over an empty scope is NULL, not a division error", + query: q({ + measures: ["Week.weeks", "Week.longShare"], + filters: [{ member: "Week.id", operator: "equals", values: ["999"] }], + }), + // count is 0; the ratio divides by NULLIF(0, 0), so it is NULL. + rows: [{ "Week.weeks": "0", "Week.longShare": null }], + }, + ], + }, + { + // calls: (id, failed, duration_ms) = (1,0,50) (2,1,2000) (3,0,1500) (4,0,80). MySQL has no + // boolean type: Cube hands a boolean measure back as the string "1" or "0". + name: "measure-boolean-mysql", + cube: "Call", + checks: [ + { + name: "all rows", + query: q({ measures: ["Call.anyFailed", "Call.allFailed", "Call.anySlowFailed", "Call.anyFastCall", "Call.anyFailedOrFalse", "Call.allFailedOrTrue"] }), + rows: [ + { + "Call.anyFailed": "1", // MAX(failed): call 2 failed + "Call.allFailed": "0", // MIN(failed): the others did not + "Call.anySlowFailed": "1", // calls over 1000 ms are 2 (failed) and 3 (not): MAX = 1 + "Call.anyFastCall": "1", // calls 1 and 4 are under 100 ms + "Call.anyFailedOrFalse": "1", + "Call.allFailedOrTrue": "0", + }, + ], + }, + { + name: "call 1 (fast, not failed): a condition that selects no row is NULL, a default fills it", + query: q({ + measures: ["Call.anyFailed", "Call.allFailed", "Call.anySlowFailed", "Call.anyFastCall", "Call.anyFailedOrFalse", "Call.allFailedOrTrue"], + filters: [{ member: "Call.id", operator: "equals", values: ["1"] }], + }), + rows: [ + { + "Call.anyFailed": "0", + "Call.allFailed": "0", + "Call.anySlowFailed": null, // 50 ms is not slow: MAX(CASE … END) over no matching value is NULL + "Call.anyFastCall": "1", + "Call.anyFailedOrFalse": "0", + "Call.allFailedOrTrue": "0", + }, + ], + }, + { + name: "call 3 (slow, not failed)", + query: q({ + measures: ["Call.anySlowFailed", "Call.anyFastCall"], + filters: [{ member: "Call.id", operator: "equals", values: ["3"] }], + }), + // 1500 ms is slow, so the CASE yields failed (0); not fast, so COALESCE(0 < ..., 0) is 0. + rows: [{ "Call.anySlowFailed": "0", "Call.anyFastCall": "0" }], + }, + { + name: "no rows at all: the aggregates are NULL and the @default measures are their defaults", + query: q({ + measures: ["Call.anyFailed", "Call.allFailed", "Call.anyFailedOrFalse", "Call.allFailedOrTrue"], + filters: [{ member: "Call.id", operator: "equals", values: ["999"] }], + }), + rows: [{ "Call.anyFailed": null, "Call.allFailed": null, "Call.anyFailedOrFalse": "0", "Call.allFailedOrTrue": "1" }], + }, + ], + }, +]; + +const unavailable = dockerUnavailableReason(); +if (unavailable !== undefined) { + const bar = "!".repeat(78); + console.warn( + `\n${bar}\n!! CUBE MYSQL CASES LIVE CHECK SKIPPED: docker unavailable (${unavailable})\n` + + `!! Nothing was checked against ${CUBE_IMAGE} on ${CUBE_MYSQL_IMAGE}. Every test in this file reports SKIP.\n${bar}\n`, + ); +} +const skip = unavailable !== undefined; + +let work = ""; +let mount = ""; +let stack: CubeStack | undefined; +const timings: string[] = []; + +beforeAll(async () => { + if (skip) return; + work = mkdtempSync(join(tmpdir(), "mo-cube-mysql-cases-")); + process.on("exit", cleanUp); + try { + mount = join(work, "model"); + mkdirSync(mount); + // Cube needs a model directory to start; the first case's tree is in it until a test swaps it. + cpSync(caseModel(CASES[0]!.name), mount, { recursive: true }); + const t0 = Date.now(); + const started = await startCubeStack(mount, { dataSource: "mysql", initSql: [SCHEMA_AND_SEED] }); + if (started.kind === "skipped") throw new Error(`docker answered a moment ago and no longer does: ${started.reason}`); + stack = started.stack; + timings.push(`stack up (MySQL + case tables, Cube /meta): ${Date.now() - t0} ms`); + } catch (e) { + cleanUp(); + throw e; + } +}, cubeStackStartBudgetMs() + 120_000); + +afterAll(() => { + cleanUp(); + process.off("exit", cleanUp); + if (timings.length > 0) console.log(`cube mysql cases timings:\n ${timings.join("\n ")}`); +}, 180_000); + +function cleanUp(): void { + stack?.stop(); + stack = undefined; + if (work !== "") rmSync(work, { recursive: true, force: true }); + work = ""; +} + +function caseModel(name: string): string { + const dir = join(CORPUS, name, "expected", "model"); + if (!existsSync(dir)) throw new Error(`corpus case '${name}' has no expected/model tree`); + return dir; +} + +/** Every cube a case's tree declares, by name. */ +function declaredCubeNames(dir: string): string[] { + const names: string[] = []; + const walk = (at: string): void => { + for (const entry of readdirSync(at, { withFileTypes: true })) { + const full = join(at, entry.name); + if (entry.isDirectory()) walk(full); + else if (entry.name.endsWith(".yml")) { + const doc = YAML.parse(readFileSync(full, "utf8")) as { cubes?: { name?: unknown }[] }; + for (const c of doc.cubes ?? []) names.push(String(c.name)); + } + } + }; + walk(dir); + return names.sort(); +} + +let swaps = 0; + +/** Replace the mounted model with a case's tree and wait until Cube lists exactly that case's cubes. */ +async function loadCase(s: CubeStack, name: string): Promise { + const n = swaps++; + for (const entry of readdirSync(mount)) renameSync(join(mount, entry), join(work, `retired-${n}-${entry}`)); + const stage = join(work, `stage-${n}`); + cpSync(caseModel(name), stage, { recursive: true }); + for (const entry of readdirSync(stage)) renameSync(join(stage, entry), join(mount, entry)); + const want = declaredCubeNames(caseModel(name)); + const deadline = Date.now() + 120_000; + let last = "no answer"; + while (Date.now() < deadline) { + try { + const meta = await getMeta(s, 30_000); + last = meta.error ?? meta.cubes.map((c) => c.name).sort().join(", "); + if (meta.error === undefined && Bun.deepEquals(meta.cubes.map((c) => c.name).sort(), want)) return; + } catch (e) { + last = e instanceof Error ? e.message : String(e); + } + await Bun.sleep(500); + } + throw new Error(`Cube did not list [${want.join(", ")}] for case '${name}' within 120s (last /meta: ${last})`); +} + +/** A Cube value as comparable text: a time to the second, a number to four decimals (a ratio is DECIMAL(…,4)), a flag as it is. */ +function normal(key: string, v: unknown): string | null { + if (v === null || v === undefined) return null; + const s = String(v); + if (/^\d{4}-\d{2}-\d{2}T/.test(s)) return s.slice(0, 19); + if (/^-?\d+(\.\d+)?$/.test(s)) return /\./.test(s) && !/^-?\d+\.0+$/.test(s) ? Number(s).toFixed(4) : canonicalDecimal(s); + throw new Error(`${key}: unexpected value ${JSON.stringify(v)}`); +} + +function sorted(rows: readonly Row[]): Row[] { + return rows.slice().sort((a, b) => (JSON.stringify(a) < JSON.stringify(b) ? -1 : 1)); +} + +describe(`cube-model MySQL corpus cases, executed (${CUBE_IMAGE} over ${CUBE_MYSQL_IMAGE})`, () => { + test.skipIf(skip)("the cases below are exactly the corpus's MySQL cases", () => { + const mysqlCases = readdirSync(CORPUS) + .filter((d) => existsSync(join(CORPUS, d, "case.json")) && existsSync(join(CORPUS, d, "expected"))) + .filter((d) => (JSON.parse(readFileSync(join(CORPUS, d, "case.json"), "utf8")) as { dialect?: string }).dialect === "mysql") + .sort(); + expect(mysqlCases).toEqual(CASES.map((c) => c.name).sort()); + }); + + for (const c of CASES) { + describe(c.name, () => { + test.skipIf(skip)("Cube compiles the case's tree over MySQL", async () => { + if (stack === undefined) throw new Error("the cube stack did not start (see beforeAll)"); + await loadCase(stack, c.name); + }, TEST_TIMEOUT_MS); + + for (const check of c.checks) { + test.skipIf(skip)(`${check.name}: the rows Cube runs on MySQL are the hand-checked ones`, async () => { + if (stack === undefined) throw new Error("the cube stack did not start (see beforeAll)"); + const result = await load(stack, { ...check.query, timezone: "UTC" }); + const wantKeys = new Set(check.rows.flatMap((r) => Object.keys(r))); + const got = sorted( + result.data.map((row) => { + const out: Row = {}; + for (const k of wantKeys) { + if (!(k in row)) throw new Error(`Cube's row has no '${k}': ${JSON.stringify(row)}`); + out[k] = normal(k, row[k]); + } + return out; + }), + ); + const want = sorted(check.rows.map((r) => Object.fromEntries(Object.entries(r).map(([k, v]) => [k, v === null ? null : normal(k, v)])))); + expect(got).toEqual(want); + }, TEST_TIMEOUT_MS); + } + }); + } +}); diff --git a/server/typescript/packages/integration-tests/cube-live/cube-model-mysql.live.ts b/server/typescript/packages/integration-tests/cube-live/cube-model-mysql.live.ts index 431248425..3266fb9ec 100644 --- a/server/typescript/packages/integration-tests/cube-live/cube-model-mysql.live.ts +++ b/server/typescript/packages/integration-tests/cube-live/cube-model-mysql.live.ts @@ -39,7 +39,7 @@ import { import { loadMetadataFile } from "../src/load-metadata.ts"; import { CANONICAL_DIR } from "../src/paths.ts"; import { readCanonicalMysqlSchemaSql } from "../src/canonical-schema-mysql.ts"; -import { cubeQueryFor, type ExpectedReport, fromCubeRow, fromViewRow, getMeta, load, sortRows } from "./cube-live-support.ts"; +import { cubeQueryFor, CANONICAL_REPORTS, selectViewMysql, fromCubeRow, fromViewRow, getMeta, load, sortRows } from "./cube-live-support.ts"; const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "..", "..", "..", "..", ".."); const CANONICAL_MODEL = join(CANONICAL_DIR, "meta.fitness.json"); @@ -48,35 +48,8 @@ const CANONICAL_SEED = join(REPO_ROOT, "fixtures", "cube-model", "canonical", "s const GEN_CONFIG = { dialect: "mysql" } as const; const TEST_TIMEOUT_MS = 600_000; -/** - * Each served canonical report, its view, the rollup Cube answers from (the same table Table I - * names on Postgres; undefined = the source tables) and its row count on seed.mysql.sql. - */ -const REPORTS: readonly ExpectedReport[] = [ - { report: "ProgramMinutes", view: "v_program_minutes", preAggregation: "dev_pre_aggregations.week__program_minutes", rows: 3 }, - { report: "FitnessTotals", view: "v_fitness_totals", preAggregation: "dev_pre_aggregations.week__fitness_totals", rows: 1 }, - { report: "ProgramsByMonth", view: "v_programs_by_month", preAggregation: "dev_pre_aggregations.program__programs_by_month", rows: 5 }, - { report: "ProgramsByWeek", view: "v_programs_by_week", preAggregation: "dev_pre_aggregations.program__programs_by_week", rows: 5 }, - { report: "RecentPrograms", view: "v_recent_programs", preAggregation: undefined, rows: 1 }, - { report: "AssetActivity", view: "v_asset_activity", preAggregation: "dev_pre_aggregations.asset__asset_activity", rows: 2 }, - { report: "ProgramRoster", view: "v_program_roster", preAggregation: undefined, rows: 11 }, - { report: "ProgramLongWeeks", view: "v_program_long_weeks", preAggregation: undefined, rows: 11 }, - { report: "ProgramFlags", view: "v_program_flags", preAggregation: "dev_pre_aggregations.week__program_flags", rows: 3 }, - { report: "ProgramFlagRoster", view: "v_program_flag_roster", preAggregation: undefined, rows: 11 }, - { report: "FitnessTotalsFilled", view: "v_fitness_totals_filled", preAggregation: "dev_pre_aggregations.week__fitness_totals_filled", rows: 1 }, - { report: "MeasurementStats", view: "v_measurement_stats", preAggregation: "dev_pre_aggregations.measurement__measurement_stats", rows: 1 }, - { report: "MeasurementStatsFilled", view: "v_measurement_stats_filled", preAggregation: "dev_pre_aggregations.measurement__measurement_stats_filled", rows: 1 }, - { report: "AssetDays", view: "v_asset_days", preAggregation: "dev_pre_aggregations.asset__asset_days", rows: 1 }, - { report: "AllTypesBreakdown", view: "v_all_types_breakdown", preAggregation: undefined, rows: 2 }, - { report: "ProgramsOverTime", view: "v_programs_over_time", preAggregation: undefined, rows: 7 }, - { report: "LiveIntroPrograms", view: "v_live_intro_programs", preAggregation: "dev_pre_aggregations.program__live_intro_programs", rows: 2 }, - { report: "ProgramsEmptyScope", view: "v_programs_empty_scope", preAggregation: "dev_pre_aggregations.program__programs_empty_scope", rows: 1, cubeRows: [{ avgPriceOrMinusOne: null, programs: null }] }, - { report: "WeekShapes", view: "v_week_shapes", preAggregation: "dev_pre_aggregations.week__week_shapes", rows: 3 }, - { report: "SessionsByProgram", view: "v_sessions_by_program", preAggregation: "dev_pre_aggregations.session__sessions_by_program", rows: 3 }, - { report: "ProgramSessions", view: "v_program_sessions", preAggregation: undefined, rows: 11 }, - { report: "NodesByParent", view: "v_nodes_by_parent", preAggregation: "dev_pre_aggregations.node__nodes_by_parent", rows: 3 }, - { report: "AuthsByKind", view: "v_auths_by_kind", preAggregation: "dev_pre_aggregations.auth__auths_by_kind", rows: 3 }, -]; +/** Each served canonical report, its view, the rollup Cube answers from and its row count: one table for both lanes. */ +const REPORTS = CANONICAL_REPORTS; const unavailable = dockerUnavailableReason(); if (unavailable !== undefined) { @@ -147,28 +120,6 @@ function findReport(name: string): MetaObject { return report; } -/** - * `SELECT * FROM ` through the mysql client, every value as the text MySQL prints for it. A - * DATETIME prints `2026-05-04 03:30:00.000`; the space becomes `T` so the shared normalization - * reads it as a UTC instant, as it does Cube's `2026-05-04T03:30:00.000`. - */ -function selectView(s: CubeStack, view: string): Array> { - const columns = s.mysql(`SHOW COLUMNS FROM \`${view}\`;`).trim().split("\n").map((l) => l.split("\t")[0] ?? ""); - const out = s.mysql(`SET time_zone = '+00:00'; SELECT ${columns.map((c) => `\`${c}\``).join(", ")} FROM \`${view}\`;`); - return out - .split("\n") - .filter((l) => l !== "") - .map((line) => { - const cells = line.split("\t"); - const row: Record = {}; - columns.forEach((c, i) => { - const v = cells[i] ?? "NULL"; - row[c] = v === "NULL" ? null : /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}/.test(v) ? v.replace(" ", "T") : v; - }); - return row; - }); -} - describe(`cube-model MySQL live check (${CUBE_IMAGE} over ${CUBE_MYSQL_IMAGE}, development mode)`, () => { test.skipIf(skip)("Table coverage: the reports below are exactly the canonical model's served reports", () => { const served = requireCanonical().objects().filter(servedReport).map((r) => r.name).sort(); @@ -191,7 +142,7 @@ describe(`cube-model MySQL live check (${CUBE_IMAGE} over ${CUBE_MYSQL_IMAGE}, d const result = await load(s, plan.query); const used = Object.keys(result.usedPreAggregations ?? {}).sort(); const cubeRows = sortRows(result.data.map((row) => fromCubeRow(row, plan, GEN_CONFIG.dialect)), plan); - const viewRows = sortRows(selectView(s, want.view).map((row) => fromViewRow(row, plan, want.view)), plan); + const viewRows = sortRows(selectViewMysql(s, want.view).map((row) => fromViewRow(row, plan, want.view)), plan); timings.push(`${want.report}: ${cubeRows.length} row(s) compared, served from ${used[0] ?? "the source table"}, load ${Date.now() - t0} ms`); expect(viewRows.length).toBe(want.rows); if (want.cubeRows !== undefined) { diff --git a/server/typescript/packages/integration-tests/cube-live/cube-model-production.live.ts b/server/typescript/packages/integration-tests/cube-live/cube-model-production.live.ts new file mode 100644 index 000000000..ddba58b13 --- /dev/null +++ b/server/typescript/packages/integration-tests/cube-live/cube-model-production.live.ts @@ -0,0 +1,338 @@ +/** + * The cube-model live check in Cube's PRODUCTION mode — the gate cube-model.live.ts (development + * mode) cannot be: there the rollups are built in the data source's own schema, on demand, and Cube + * runs without auth. Production mode is what an adopter deploys, and it differs in every way that + * matters to a rollup: + * + * * `CUBEJS_DEV_MODE=false`: every request carries a signed JWT; + * * rollups live in a real Cube Store (`cubejs/cubestore`, the Cube image's version), so a query + * answered from one reports `external: true` under `prod_pre_aggregations`; + * * a rollup answers only once it is BUILT. The lane builds every one through the + * pre-aggregations jobs API and waits for each job to be `done` before it queries. + * + * For each dialect (Postgres, then MySQL) it generates the canonical model with `cubeModel()`, + * starts the stack, builds the rollups, and then, for every served report: + * + * * runs the Table F query and requires the rows to equal `SELECT *` from the report's view and + * the answer to come FROM the report's rollup (`usedPreAggregations` names it and says + * `external: true`) — or from the source table when the report has no rollup; + * * for a report that lists a time dimension at several grains (one rollup per grain), runs the + * query at EACH grain and requires that grain's rollup to answer it, with rows equal to the + * same grouping computed directly from the source table. + * + * Last, it requires every rollup the generated model declares to have answered at least one query: + * a rollup no Cube query can match (the defect "Several grains" used to be) fails here. + * + * `*.live.ts`: runs only by its path, from `bun run test:cube`. Without docker every test is + * SKIPPED behind a banner. + */ + +import { afterAll, beforeAll, describe, expect, test } from "bun:test"; +import { YAML } from "bun"; +import { mkdtempSync, readdirSync, readFileSync, rmSync, statSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { cubeModel, defineConfig, runGen, servedReport } from "@metaobjectsdev/codegen-ts"; +import { reportShape, type MetaObject, type MetaRoot } from "@metaobjectsdev/metadata"; +import { + CUBE_IMAGE, + CUBE_MYSQL_IMAGE, + CUBE_POSTGRES_IMAGE, + cubeStackStartBudgetMs, + dockerUnavailableReason, + startCubeStack, + type CubeDataSource, + type CubeStack, +} from "../src/cube-container.ts"; +import { loadMetadataFile } from "../src/load-metadata.ts"; +import { CANONICAL_DIR } from "../src/paths.ts"; +import { readCanonicalMysqlSchemaSql } from "../src/canonical-schema-mysql.ts"; +import { CUBE_STORE_IMAGE, startProductionCube, type ProdCube } from "./cube-prod-stack.ts"; +import { + CANONICAL_REPORTS, + type NormalRow, + buildAllPreAggregations, + cubeQueryFor, + fromCubeRow, + fromViewRow, + getMeta, + load, + productionPreAggregation, + selectSqlMysql, + selectSqlPg, + selectViewMysql, + selectViewPg, + sortRows, + type LoadResponse, + type QueryPlan, +} from "./cube-live-support.ts"; + +const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "..", "..", "..", "..", ".."); +const CANONICAL_MODEL = join(CANONICAL_DIR, "meta.fitness.json"); +const PG_SCHEMA = join(CANONICAL_DIR, "schema.postgres.sql"); +const PG_SEED = join(REPO_ROOT, "fixtures", "cube-model", "canonical", "seed.sql"); +const MYSQL_SEED = join(REPO_ROOT, "fixtures", "cube-model", "canonical", "seed.mysql.sql"); +const TEST_TIMEOUT_MS = 600_000; + +const unavailable = dockerUnavailableReason(); +if (unavailable !== undefined) { + const bar = "!".repeat(78); + console.warn( + `\n${bar}\n!! CUBE PRODUCTION-MODE LIVE CHECK SKIPPED: docker unavailable (${unavailable})\n` + + `!! Nothing was checked against ${CUBE_IMAGE} and ${CUBE_STORE_IMAGE}. Every test in this file reports SKIP.\n${bar}\n`, + ); +} +const skip = unavailable !== undefined; + +/** `ProgramsOverTime_day` → `programs_over_time_day`: Cube's snake case for the table of a cube or rollup. */ +function snake(name: string): string { + return name.replace(/([a-z0-9])([A-Z])/g, "$1_$2").toLowerCase(); +} + +/** Every rollup a model tree declares: `{ cube, name }` for each `pre_aggregations` entry. */ +function declaredRollups(modelDir: string): { cube: string; name: string }[] { + const out: { cube: string; name: string }[] = []; + const walk = (dir: string): void => { + for (const entry of readdirSync(dir).sort()) { + const full = join(dir, entry); + if (statSync(full).isDirectory()) { + walk(full); + continue; + } + if (!entry.endsWith(".yml")) continue; + const doc = YAML.parse(readFileSync(full, "utf8")) as { cubes?: { name?: unknown; pre_aggregations?: { name?: unknown }[] }[] }; + for (const c of doc.cubes ?? []) { + for (const p of c.pre_aggregations ?? []) out.push({ cube: String(c.name), name: String(p.name) }); + } + } + }; + walk(modelDir); + return out; +} + +/** The rollup table (`prod_pre_aggregations.__`) Cube names for a declared rollup. */ +function rollupTable(r: { cube: string; name: string }): string { + return `prod_pre_aggregations.${snake(r.cube)}__${snake(r.name)}`; +} + +const GRAINS = ["day", "quarter", "year"] as const; +type Grain = (typeof GRAINS)[number]; + +/** + * The source-table grouping `ProgramsOverTime` is reproduced by at one grain: what its view + * groups, minus the other grains, written by hand against the `programs` table (the view cannot be + * re-aggregated: an average of averages is not the average). + */ +function overTimeSql(dialect: CubeDataSource, grain: Grain): { select: string; columns: string[] } { + const field = `createdAt${grain.charAt(0).toUpperCase()}${grain.slice(1)}`; + if (dialect === "postgres") { + return { + select: + `SELECT CAST(date_trunc('${grain}', p."created_ts") AS DATE) AS "${field}", COUNT(p."id") AS "programs", ` + + `AVG(p."priceCents") AS "avgPrice", MIN(p."priceCents") AS "cheapest", MAX(p."priceCents") AS "priciest" ` + + `FROM "programs" p GROUP BY 1`, + columns: [], + }; + } + const bucket = { + day: "DATE(p.`created_ts`)", + quarter: "MAKEDATE(YEAR(p.`created_ts`), 1) + INTERVAL (QUARTER(p.`created_ts`) - 1) QUARTER", + year: "MAKEDATE(YEAR(p.`created_ts`), 1)", + }[grain]; + return { + select: + `SELECT ${bucket} AS \`${field}\`, COUNT(p.\`id\`) AS \`programs\`, AVG(p.\`priceCents\`) AS \`avgPrice\`, ` + + `MIN(p.\`priceCents\`) AS \`cheapest\`, MAX(p.\`priceCents\`) AS \`priciest\` FROM \`programs\` p GROUP BY 1`, + columns: [field, "programs", "avgPrice", "cheapest", "priciest"], + }; +} + +const MULTI_GRAIN_REPORT = "ProgramsOverTime"; + +for (const dialect of ["postgres", "mysql"] as const) { + describe(`cube-model production-mode live check (${CUBE_IMAGE} + ${CUBE_STORE_IMAGE} over ${dialect === "mysql" ? CUBE_MYSQL_IMAGE : CUBE_POSTGRES_IMAGE})`, () => { + let work = ""; + let prod: ProdCube | undefined; + let canonical: MetaRoot | undefined; + /** The base stack: its database is the one the views and source tables are read from. */ + let baseStack: CubeStack | undefined; + let builtTables: string[] = []; + let modelDir = ""; + /** The rollup tables queries were answered from, across this block's tests. */ + const answeredFrom = new Set(); + const timings: string[] = []; + + function cleanUp(): void { + prod?.stop(); + prod = undefined; + baseStack = undefined; + if (work !== "") rmSync(work, { recursive: true, force: true }); + work = ""; + } + + beforeAll(async () => { + if (skip) return; + work = mkdtempSync(join(tmpdir(), `mo-cube-prod-${dialect}-`)); + process.on("exit", cleanUp); + try { + canonical = await loadMetadataFile(CANONICAL_MODEL); + const outDir = join(work, "out"); + await runGen({ + config: defineConfig({ outDir, dialect, generators: [cubeModel()] }), + metadata: canonical, + genStateDir: join(work, ".gen-state"), + }); + modelDir = join(outDir, "model"); + const t0 = Date.now(); + const started = await startCubeStack(modelDir, { + dataSource: dialect, + initSql: + dialect === "mysql" + ? [readCanonicalMysqlSchemaSql(), readFileSync(MYSQL_SEED, "utf8")] + : [readFileSync(PG_SCHEMA, "utf8"), readFileSync(PG_SEED, "utf8")], + }); + if (started.kind === "skipped") throw new Error(`docker answered a moment ago and no longer does: ${started.reason}`); + const base = started.stack; + try { + prod = await startProductionCube(base, modelDir, dialect); + } catch (e) { + base.stop(); + throw e; + } + baseStack = base; + timings.push(`stack up (database, Cube Store, production Cube): ${Date.now() - t0} ms`); + const t1 = Date.now(); + builtTables = await buildAllPreAggregations(prod); + timings.push(`${builtTables.length} pre-aggregation(s) built into Cube Store: ${Date.now() - t1} ms`); + } catch (e) { + cleanUp(); + throw e; + } + }, cubeStackStartBudgetMs() + 900_000); + + afterAll(() => { + cleanUp(); + process.off("exit", cleanUp); + if (timings.length > 0) console.log(`cube production-mode ${dialect} timings:\n ${timings.join("\n ")}`); + }, 180_000); + + const requireProd = (): ProdCube => { + if (prod === undefined) throw new Error("the production Cube stack did not start (see beforeAll)"); + return prod; + }; + const requireCanonical = (): MetaRoot => { + if (canonical === undefined) throw new Error("the canonical model did not load (see beforeAll)"); + return canonical; + }; + const findReport = (name: string): MetaObject => { + const report = requireCanonical().objects().find((o) => servedReport(o) && o.name === name); + if (report === undefined) throw new Error(`the canonical model serves no report '${name}'`); + return report; + }; + const requireBase = (): CubeStack => { + if (baseStack === undefined) throw new Error("the database stack did not start (see beforeAll)"); + return baseStack; + }; + const viewRows = (view: string): Array> => + dialect === "mysql" ? selectViewMysql(requireBase(), view) : selectViewPg(requireBase(), view); + + /** + * Cube answered from exactly the rollup `want` (or from no rollup, when undefined). A rollup's + * answer is Cube Store's: the response says `external: true` and `extDbType: "cubestore"`, and + * `usedPreAggregations` names the rollup under production mode's schema. + */ + function expectAnsweredFrom(report: string, result: LoadResponse, want: string | undefined): void { + const keys = Object.keys(result.usedPreAggregations ?? {}).sort(); + expect({ report, usedPreAggregations: keys }).toEqual({ report, usedPreAggregations: want === undefined ? [] : [want] }); + if (want === undefined) { + expect({ report, external: result.external === true }).toEqual({ report, external: false }); + return; + } + expect({ report, table: want, external: result.external, extDbType: result.extDbType }).toEqual({ report, table: want, external: true, extDbType: "cubestore" }); + answeredFrom.add(want); + } + + test.skipIf(skip)("production mode is on: the API refuses a request without the token, and the rollups are in Cube Store", async () => { + const p = requireProd(); + const bare = await fetch(`${p.apiBase}/meta`, { signal: AbortSignal.timeout(30_000) }); + expect(bare.status).toBe(403); + expect((await getMeta(p)).error).toBeUndefined(); + const declared = declaredRollups(modelDir); + expect(declared.length).toBeGreaterThan(0); + // Every declared rollup was built, each under production mode's schema. + for (const r of declared) { + const table = rollupTable(r); + expect({ rollup: r.name, built: builtTables.some((t) => t.startsWith(table)) }).toEqual({ rollup: r.name, built: true }); + } + expect(builtTables.every((t) => t.startsWith("prod_pre_aggregations."))).toBe(true); + }, TEST_TIMEOUT_MS); + + test.skipIf(skip)("the reports below are exactly the canonical model's served reports", () => { + const served = requireCanonical().objects().filter(servedReport).map((r) => r.name).sort(); + expect(served).toEqual(CANONICAL_REPORTS.map((e) => e.report).sort()); + }); + + for (const want of CANONICAL_REPORTS) { + test.skipIf(skip)( + `${want.report}: the Table F query equals ${want.view}${want.preAggregation === undefined ? " (source table)" : " (from its rollup in Cube Store)"}`, + async () => { + const p = requireProd(); + const plan = cubeQueryFor(findReport(want.report), requireCanonical()); + const t0 = Date.now(); + const result = await load(p, plan.query); + const cubeRows = sortRows(result.data.map((row) => fromCubeRow(row, plan, dialect)), plan); + const expected = sortRows(viewRows(want.view).map((row) => fromViewRow(row, plan, want.view)), plan); + timings.push(`${want.report}: ${cubeRows.length} row(s), ${Object.keys(result.usedPreAggregations ?? {})[0] ?? "source table"}, ${Date.now() - t0} ms`); + expectAnsweredFrom(want.report, result, want.preAggregation === undefined ? undefined : productionPreAggregation(want.preAggregation)); + expect(expected.length).toBe(want.rows); + if (want.cubeRows !== undefined) { + expect(cubeRows).toEqual([...want.cubeRows]); + expect(cubeRows).not.toEqual(expected); + } else { + expect(cubeRows).toEqual(expected); + } + }, + TEST_TIMEOUT_MS, + ); + } + + for (const grain of GRAINS) { + test.skipIf(skip)(`${MULTI_GRAIN_REPORT} at ${grain}: its ${grain} rollup answers a query at that grain, with the source table's rows`, async () => { + const p = requireProd(); + const report = findReport(MULTI_GRAIN_REPORT); + const base: QueryPlan = cubeQueryFor(report, requireCanonical()); + // The report lists createdAt at every one of GRAINS: the model must carry one rollup for each. + const listed = reportShape(report, requireCanonical()).fields.filter((f) => f.grain !== undefined).map((f) => f.grain); + expect(listed.sort()).toEqual([...GRAINS].sort()); + const field = `createdAt${grain.charAt(0).toUpperCase()}${grain.slice(1)}`; + const plan: QueryPlan = { + query: { ...base.query, timeDimensions: [{ dimension: "Program.createdAt", granularity: grain }] }, + columns: [ + { field, cubeKey: `Program.createdAt.${grain}`, kind: "time" }, + ...base.columns.filter((c) => c.kind === "measure"), + ], + }; + const result = await load(p, plan.query); + const sql = overTimeSql(dialect, grain); + const sourceRows: Array> = + dialect === "mysql" ? selectSqlMysql(requireBase(), sql.select, sql.columns) : selectSqlPg(requireBase(), sql.select); + const cubeRows: NormalRow[] = sortRows(result.data.map((row) => fromCubeRow(row, plan, dialect)), plan); + const expected: NormalRow[] = sortRows(sourceRows.map((row) => fromViewRow(row, plan, `programs@${grain}`)), plan); + expectAnsweredFrom(`${MULTI_GRAIN_REPORT}@${grain}`, result, `prod_pre_aggregations.program__programs_over_time_${grain}`); + expect(expected.length).toBeGreaterThan(0); + // Cube Store keeps a rolled-up average to 5 decimal places (`2322.11111`), where the source + // database prints 16 (`2322.1111111111111111`): same number, different scale. Compare the + // average at 4 places; every other column is compared as it is. + const atFourPlaces = (rows: NormalRow[]): NormalRow[] => + rows.map((r) => ({ ...r, avgPrice: r.avgPrice === null || r.avgPrice === undefined ? null : Number(r.avgPrice).toFixed(4) })); + expect(atFourPlaces(cubeRows)).toEqual(atFourPlaces(expected)); + }, TEST_TIMEOUT_MS); + } + + test.skipIf(skip)("every rollup the generated model declares answered a query: none is a rollup no Cube query can match", () => { + const declared = declaredRollups(modelDir).map(rollupTable).sort(); + expect([...answeredFrom].sort()).toEqual(declared); + }); + }); +} diff --git a/server/typescript/packages/integration-tests/cube-live/cube-model-shapes.live.ts b/server/typescript/packages/integration-tests/cube-live/cube-model-shapes.live.ts new file mode 100644 index 000000000..ad4711382 --- /dev/null +++ b/server/typescript/packages/integration-tests/cube-live/cube-model-shapes.live.ts @@ -0,0 +1,357 @@ +/** + * The shapes the other Cube lane files only COMPILE, EXECUTED on Postgres through Cube. + * + * `cube-model.live.ts` loads every corpus case that has an expected tree, and Cube accepts it, but + * a compile runs no SQL. Five shapes had no live query crossing them (docs/features/cube-export.md + * used to list them under "Known limits"): + * + * join-alias-cubes two alias cubes onto one table (`Match_homeRef`, `Match_awayRef` over `teams`) + * entity-tph-subtype a TPH subtype's cube whose `sql` is `SELECT * FROM … WHERE "type" = 'Bridge'` + * join-one-to-one a `one_to_one` join and its `many_to_one` mirror, read from either end + * dimension-int-enum an int-backed enum's `CASE … WHEN 1 THEN 'LOW' …` dimension + * report-spine-multi-hop a `@spine` report through two hops (`Session.fkWeek.fkProgram`), whose + * Cube view must give the rows of the SQL view `buildReportViews` lowers, + * spine rows with no facts included + * + * For each, the case's generated tree (`fixtures/cube-model//expected/model`, which + * `cube-model-corpus.test.ts` holds equal to what the generator writes) is loaded into the pinned + * Cube over a private Postgres 16 holding the tables below, and every query is run through Cube and + * compared with the rows an independent SQL statement returns from the same database. The spine + * case's oracle is the lowered report view, as in the canonical lane; the others have no + * `object.report`, so their oracle is a plain `SELECT` written here, which neither the generator + * nor Cube produced. + * + * `*.live.ts`: runs only by its path, from `bun run test:cube`. Without docker every test is + * SKIPPED behind a banner. + */ + +import { afterAll, beforeAll, describe, expect, test } from "bun:test"; +import { YAML } from "bun"; +import { cpSync, existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, renameSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { buildReportViews } from "@metaobjectsdev/codegen-ts"; +import type { MetaRoot } from "@metaobjectsdev/metadata"; +import { CUBE_IMAGE, cubeStackStartBudgetMs, dockerUnavailableReason, startCubeStack, type CubeStack } from "../src/cube-container.ts"; +import { loadMetadataFile } from "../src/load-metadata.ts"; +import { type CubeQuery, cubeQueryFor, fromCubeRow, fromViewRow, getMeta, load, selectSqlPg, selectViewPg, sortRows } from "./cube-live-support.ts"; + +const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "..", "..", "..", "..", ".."); +const CORPUS = join(REPO_ROOT, "fixtures", "cube-model"); +const TEST_TIMEOUT_MS = 300_000; + +const SPINE_CASE = "report-spine-multi-hop"; +const SPINE_REPORT = "ProgramSessions"; + +/** Every table the five cases' models name, snake_case columns as the cases declare. */ +const SCHEMA_AND_SEED = ` +CREATE TABLE teams (id BIGINT PRIMARY KEY, name TEXT); +INSERT INTO teams (id, name) VALUES (1, 'Reds'), (2, 'Blues'), (3, 'Greens'); +CREATE TABLE matches (id BIGINT PRIMARY KEY, home_team_id BIGINT, away_team_id BIGINT); +INSERT INTO matches (id, home_team_id, away_team_id) VALUES + (1, 1, 2), (2, 1, 2), (3, 2, 1), (4, 3, 1), (5, 3, 2); + +CREATE TABLE auths (id BIGINT PRIMARY KEY, type TEXT NOT NULL); +INSERT INTO auths (id, type) VALUES (1, 'Bridge'), (2, 'Copay'), (3, 'Bridge'), (4, 'Copay'), (5, 'Copay'), (6, 'Bridge'), (7, 'Bridge'); + +-- programs: 1 and 3 have a profile, 2 has none; program 4 has sessions through no week (spine row, no facts). +CREATE TABLE programs (id BIGINT PRIMARY KEY, title TEXT); +INSERT INTO programs (id, title) VALUES (1, 'Alpha'), (2, 'Beta'), (3, 'Gamma'), (4, 'Delta'); +CREATE TABLE profiles (id BIGINT PRIMARY KEY, program_id BIGINT); +INSERT INTO profiles (id, program_id) VALUES (10, 1), (11, 3); + +CREATE TABLE tickets (id BIGINT PRIMARY KEY, priority INT NOT NULL); +INSERT INTO tickets (id, priority) VALUES (1, 1), (2, 2), (3, 2), (4, 1), (5, 2); + +-- weeks 1,2 belong to Alpha; 3 to Beta; 4 to Gamma (all of Gamma's sessions are under 10 minutes). +CREATE TABLE weeks (id BIGINT PRIMARY KEY, program_id BIGINT NOT NULL); +INSERT INTO weeks (id, program_id) VALUES (1, 1), (2, 1), (3, 2), (4, 3); +CREATE TABLE sessions (id BIGINT PRIMARY KEY, week_id BIGINT NOT NULL, minutes INT); +INSERT INTO sessions (id, week_id, minutes) VALUES + (1, 1, 30), (2, 1, 45), (3, 2, 10), (4, 2, 5), (5, 3, 60), (6, 4, 9), (7, 4, 2); +`; + +type Row = Record; + +interface Check { + readonly name: string; + readonly query: Omit; + /** A SELECT whose columns are aliased to the Cube row keys the query returns. */ + readonly oracle: string; + /** Rows the oracle must return (non-vacuity): a count, so a query that returned nothing cannot pass. */ + readonly rows: number; +} + +interface Case { + readonly name: string; + readonly cubes: readonly string[]; + readonly checks: readonly Check[]; +} + +const q = (parts: Partial>): Omit => ({ + measures: [], dimensions: [], timeDimensions: [], segments: [], ...parts, +}); + +const CASES: readonly Case[] = [ + { + name: "join-alias-cubes", + cubes: ["Match", "Match_awayRef", "Match_homeRef"], + checks: [ + { + name: "matches by the home team's name (the Match_homeRef alias)", + query: q({ measures: ["Match.matches"], dimensions: ["Match.homeName"] }), + oracle: `SELECT t.name AS "Match.homeName", count(m.id) AS "Match.matches" FROM matches m LEFT JOIN teams t ON m.home_team_id = t.id GROUP BY t.name`, + rows: 3, + }, + { + name: "matches by the away team's name (the Match_awayRef alias)", + query: q({ measures: ["Match.matches"], dimensions: ["Match.awayName"] }), + oracle: `SELECT t.name AS "Match.awayName", count(m.id) AS "Match.matches" FROM matches m LEFT JOIN teams t ON m.away_team_id = t.id GROUP BY t.name`, + rows: 2, + }, + { + name: "both aliases in one query: each fixture is counted under its own home and away pair", + query: q({ measures: ["Match.matches"], dimensions: ["Match.homeName", "Match.awayName"] }), + oracle: + `SELECT h.name AS "Match.homeName", a.name AS "Match.awayName", count(m.id) AS "Match.matches" FROM matches m ` + + `LEFT JOIN teams h ON m.home_team_id = h.id LEFT JOIN teams a ON m.away_team_id = a.id GROUP BY h.name, a.name`, + rows: 4, + }, + ], + }, + { + name: "entity-tph-subtype", + cubes: ["BridgeAuth"], + checks: [ + { + name: "the subtype's measure counts only its discriminator's rows", + query: q({ measures: ["BridgeAuth.bridges"] }), + oracle: `SELECT count(id) AS "BridgeAuth.bridges" FROM auths WHERE type = 'Bridge'`, + rows: 1, + }, + { + name: "the subtype's rows are only the Bridge rows", + query: q({ dimensions: ["BridgeAuth.id"] }), + oracle: `SELECT id AS "BridgeAuth.id" FROM auths WHERE type = 'Bridge'`, + rows: 4, + }, + ], + }, + { + name: "join-one-to-one", + cubes: ["Profile", "Program"], + checks: [ + { + name: "Program.programs and Profile.profiles side by side across the one_to_one join", + query: q({ measures: ["Program.programs", "Profile.profiles"] }), + oracle: `SELECT count(p.id) AS "Program.programs", count(pr.id) AS "Profile.profiles" FROM programs p LEFT JOIN profiles pr ON pr.program_id = p.id`, + rows: 1, + }, + { + name: "profiles per program, read from the one_to_one end (a program with no profile reads 0)", + query: q({ measures: ["Profile.profiles"], dimensions: ["Program.id"] }), + oracle: `SELECT p.id AS "Program.id", count(pr.id) AS "Profile.profiles" FROM programs p LEFT JOIN profiles pr ON pr.program_id = p.id GROUP BY p.id`, + rows: 4, + }, + { + name: "programs per profile, read from the many_to_one end", + query: q({ measures: ["Program.programs"], dimensions: ["Profile.id"] }), + oracle: `SELECT pr.id AS "Profile.id", count(p.id) AS "Program.programs" FROM profiles pr LEFT JOIN programs p ON pr.program_id = p.id GROUP BY pr.id`, + rows: 2, + }, + ], + }, + { + name: "dimension-int-enum", + cubes: ["Ticket"], + checks: [ + { + name: "the int-backed enum reads as its member symbol", + query: q({ dimensions: ["Ticket.priority"] }), + // Independent of the exporter's CASE: the stored integers mapped by the model's @intValueMap. + oracle: `SELECT DISTINCT CASE WHEN priority = 1 THEN 'LOW' WHEN priority = 2 THEN 'HIGH' END AS "Ticket.priority" FROM tickets`, + rows: 2, + }, + { + name: "a filter on the symbol selects the rows of its integer", + query: q({ dimensions: ["Ticket.id"], filters: [{ member: "Ticket.priority", operator: "equals", values: ["HIGH"] }] }), + oracle: `SELECT id AS "Ticket.id" FROM tickets WHERE priority = 2`, + rows: 3, + }, + ], + }, +]; + +const unavailable = dockerUnavailableReason(); +if (unavailable !== undefined) { + const bar = "!".repeat(78); + console.warn( + `\n${bar}\n!! CUBE SHAPES LIVE CHECK SKIPPED: docker unavailable (${unavailable})\n` + + `!! Nothing was checked against ${CUBE_IMAGE}. Every test in this file reports SKIP.\n${bar}\n`, + ); +} +const skip = unavailable !== undefined; + +let work = ""; +let mount = ""; +let stack: CubeStack | undefined; +let spineRoot: MetaRoot | undefined; +const timings: string[] = []; + +function caseModel(name: string): string { + const dir = join(CORPUS, name, "expected", "model"); + if (!existsSync(dir)) throw new Error(`corpus case '${name}' has no expected/model tree`); + return dir; +} + +beforeAll(async () => { + if (skip) return; + work = mkdtempSync(join(tmpdir(), "mo-cube-shapes-")); + process.on("exit", cleanUp); + try { + mount = join(work, "model"); + mkdirSync(mount); + cpSync(caseModel(CASES[0]!.name), mount, { recursive: true }); + + // The spine case's oracle is the view the report lowers to. + spineRoot = await loadMetadataFile(join(CORPUS, SPINE_CASE, "meta.json")); + const views = buildReportViews(spineRoot, { dialect: "postgres" }); + const viewSql = views.map((v) => `CREATE VIEW "${v.name}" AS ${v.sql};`).join("\n"); + + const t0 = Date.now(); + const started = await startCubeStack(mount, { initSql: [SCHEMA_AND_SEED, viewSql] }); + if (started.kind === "skipped") throw new Error(`docker answered a moment ago and no longer does: ${started.reason}`); + stack = started.stack; + timings.push(`stack up (Postgres + shape tables, Cube /meta): ${Date.now() - t0} ms`); + } catch (e) { + cleanUp(); + throw e; + } +}, cubeStackStartBudgetMs() + 120_000); + +afterAll(() => { + cleanUp(); + process.off("exit", cleanUp); + if (timings.length > 0) console.log(`cube shapes timings:\n ${timings.join("\n ")}`); +}, 180_000); + +function cleanUp(): void { + stack?.stop(); + stack = undefined; + if (work !== "") rmSync(work, { recursive: true, force: true }); + work = ""; +} + +function requireStack(): CubeStack { + if (stack === undefined) throw new Error("the cube stack did not start (see beforeAll)"); + return stack; +} + +function declaredCubeNames(dir: string): string[] { + const names: string[] = []; + const walk = (at: string): void => { + for (const entry of readdirSync(at, { withFileTypes: true })) { + const full = join(at, entry.name); + if (entry.isDirectory()) walk(full); + else if (entry.name.endsWith(".yml")) { + const doc = YAML.parse(readFileSync(full, "utf8")) as { cubes?: { name?: unknown }[]; views?: { name?: unknown }[] }; + for (const c of [...(doc.cubes ?? []), ...(doc.views ?? [])]) names.push(String(c.name)); + } + } + }; + walk(dir); + return names.sort(); +} + +let swaps = 0; + +/** Replace the mounted model with a case's tree and wait until Cube lists exactly that case's cubes. */ +async function loadCase(s: CubeStack, name: string): Promise { + const n = swaps++; + for (const entry of readdirSync(mount)) renameSync(join(mount, entry), join(work, `retired-${n}-${entry}`)); + const stage = join(work, `stage-${n}`); + cpSync(caseModel(name), stage, { recursive: true }); + for (const entry of readdirSync(stage)) renameSync(join(stage, entry), join(mount, entry)); + const want = declaredCubeNames(caseModel(name)); + const deadline = Date.now() + 120_000; + let last = "no answer"; + while (Date.now() < deadline) { + try { + const meta = await getMeta(s, 30_000); + last = meta.error ?? meta.cubes.map((c) => c.name).sort().join(", "); + if (meta.error === undefined && Bun.deepEquals(meta.cubes.map((c) => c.name).sort(), want)) return; + } catch (e) { + last = e instanceof Error ? e.message : String(e); + } + await Bun.sleep(500); + } + throw new Error(`Cube did not list [${want.join(", ")}] for case '${name}' within 120s (last /meta: ${last})`); +} + +function sorted(rows: readonly Row[]): Row[] { + return rows.slice().sort((a, b) => (JSON.stringify(a) < JSON.stringify(b) ? -1 : 1)); +} + +describe(`cube-model shapes, executed (${CUBE_IMAGE} over Postgres)`, () => { + for (const c of CASES) { + describe(c.name, () => { + test.skipIf(skip)("Cube lists exactly the cubes the case's tree declares", async () => { + const s = requireStack(); + await loadCase(s, c.name); + const meta = await getMeta(s); + expect(meta.cubes.map((x) => x.name).sort()).toEqual([...c.cubes]); + }, TEST_TIMEOUT_MS); + + for (const check of c.checks) { + test.skipIf(skip)(`${check.name}: the rows Cube runs are the rows plain SQL returns`, async () => { + const s = requireStack(); + const want = sorted(selectSqlPg(s, check.oracle)); + expect(want.length).toBe(check.rows); + const result = await load(s, { ...check.query, timezone: "UTC" }); + const keys = Object.keys(want[0] ?? {}); + const got = sorted( + result.data.map((row) => { + const out: Row = {}; + for (const k of keys) { + if (!(k in row)) throw new Error(`Cube's row has no '${k}': ${JSON.stringify(row)}`); + const v = row[k]; + out[k] = v === null || v === undefined ? null : String(v); + } + return out; + }), + ); + expect(got).toEqual(want); + }, TEST_TIMEOUT_MS); + } + }); + } + + describe(SPINE_CASE, () => { + test.skipIf(skip)("the two-hop @spine Cube view answers the rows of the SQL view, spine rows with no facts included", async () => { + const s = requireStack(); + await loadCase(s, SPINE_CASE); + const root = spineRoot; + if (root === undefined) throw new Error("the spine model did not load (see beforeAll)"); + const report = root.objects().find((o) => o.name === SPINE_REPORT); + if (report === undefined) throw new Error(`${SPINE_REPORT} is not in the case's model`); + const plan = cubeQueryFor(report, root); + const result = await load(s, plan.query); + const cubeRows = sortRows(result.data.map((row) => fromCubeRow(row, plan, "postgres")), plan); + // This case declares no physical naming, so the view's columns are snake_case (`program_key`); + // the plan names report fields (`programKey`). + const camel = (k: string): string => k.replace(/_([a-z])/g, (_, c: string) => c.toUpperCase()); + const viewRows = sortRows( + selectViewPg(s, "v_program_sessions").map((row) => + fromViewRow(Object.fromEntries(Object.entries(row).map(([k, v]) => [camel(k), v])), plan, "v_program_sessions"), + ), + plan, + ); + // Non-vacuity: all four programs are spine rows, and Delta (no weeks) and Gamma (only + // sessions under the report's 10-minute filter) are the ones with no facts. + expect(viewRows.length).toBe(4); + expect(viewRows.filter((r) => r.sessions === "0").length).toBe(2); + expect(cubeRows).toEqual(viewRows); + }, TEST_TIMEOUT_MS); + }); +}); diff --git a/server/typescript/packages/integration-tests/cube-live/cube-model.live.ts b/server/typescript/packages/integration-tests/cube-live/cube-model.live.ts index 6affc1120..29f9b9587 100644 --- a/server/typescript/packages/integration-tests/cube-live/cube-model.live.ts +++ b/server/typescript/packages/integration-tests/cube-live/cube-model.live.ts @@ -14,10 +14,10 @@ * `SELECT * FROM ` under Table I's normalization. A report with `@spine` is a Cube * view, queried by its own name, and its rows include the spine rows with no facts; * 4. loads every case of the mapping corpus that has an expected tree into the same Cube and - * requires each to compile, which turns the shapes no live query reaches (alias cubes, a TPH - * subtype's `sql`, one-to-one joins, Jinja-escaped text) into executed checks. The MySQL + * requires each to compile (alias cubes, a TPH subtype's `sql`, one-to-one joins and the int-backed + * enum's `CASE` are then EXECUTED by cube-model-shapes.live.ts; Jinja-escaped text is read back here). The MySQL * cases are loaded too: Cube's `/v1/meta` compile runs no SQL, so the Postgres data source - * does not stop a MySQL model from compiling; + * does not stop a MySQL model from compiling (cube-model-mysql-cases.live.ts then EXECUTES them on MySQL); * 5. reads the `escaping` case's SQL back through `/v1/sql` and requires each of its literals, * and its braced column, to reach the SQL exactly as the report view writes them (Cube * compiles every `sql` as a template literal, so a missed escape changes the value); @@ -25,7 +25,8 @@ * operand that declares `@default` to divide that operand's COALESCE, as the view does. * * It is `*.live.ts`, so no directory-walking `bun test` picks it up: it runs only by its path, - * from `bun run test:cube` (the `cube` lane of scripts/ci-local.sh). Without docker every test is + * from `bun run test:cube` (the `cube` lane of scripts/ci-local.sh), as do the MySQL, MySQL-cases and + * production-mode files beside it. Without docker every test is * SKIPPED behind a banner; nothing here ever passes without having run. */ @@ -75,7 +76,8 @@ import { generatedSql, fromCubeRow, fromViewRow, - type ExpectedReport, + CANONICAL_REPORTS, + selectViewPg, getJson, getMeta, load, @@ -109,33 +111,7 @@ const CORPUS_CASES = corpusCasesWithTree(); const CORPUS_BUDGET_MS = CORPUS_CASES.length * CASE_BUDGET_MS; -const TABLE_I: readonly ExpectedReport[] = [ - { report: "ProgramMinutes", view: "v_program_minutes", preAggregation: "dev_pre_aggregations.week__program_minutes", rows: 3 }, - { report: "FitnessTotals", view: "v_fitness_totals", preAggregation: "dev_pre_aggregations.week__fitness_totals", rows: 1 }, - { report: "ProgramsByMonth", view: "v_programs_by_month", preAggregation: "dev_pre_aggregations.program__programs_by_month", rows: 5 }, - { report: "ProgramsByWeek", view: "v_programs_by_week", preAggregation: "dev_pre_aggregations.program__programs_by_week", rows: 5 }, - { report: "RecentPrograms", view: "v_recent_programs", preAggregation: undefined, rows: 1 }, - { report: "AssetActivity", view: "v_asset_activity", preAggregation: "dev_pre_aggregations.asset__asset_activity", rows: 2 }, - { report: "ProgramRoster", view: "v_program_roster", preAggregation: undefined, rows: 11 }, - { report: "ProgramLongWeeks", view: "v_program_long_weeks", preAggregation: undefined, rows: 11 }, - { report: "ProgramFlags", view: "v_program_flags", preAggregation: "dev_pre_aggregations.week__program_flags", rows: 3 }, - { report: "ProgramFlagRoster", view: "v_program_flag_roster", preAggregation: undefined, rows: 11 }, - { report: "FitnessTotalsFilled", view: "v_fitness_totals_filled", preAggregation: "dev_pre_aggregations.week__fitness_totals_filled", rows: 1 }, - { report: "MeasurementStats", view: "v_measurement_stats", preAggregation: "dev_pre_aggregations.measurement__measurement_stats", rows: 1 }, - { report: "MeasurementStatsFilled", view: "v_measurement_stats_filled", preAggregation: "dev_pre_aggregations.measurement__measurement_stats_filled", rows: 1 }, - { report: "AssetDays", view: "v_asset_days", preAggregation: "dev_pre_aggregations.asset__asset_days", rows: 1 }, - { report: "AllTypesBreakdown", view: "v_all_types_breakdown", preAggregation: undefined, rows: 2 }, - { report: "ProgramsOverTime", view: "v_programs_over_time", preAggregation: undefined, rows: 7, unusedRollup: true }, - { report: "LiveIntroPrograms", view: "v_live_intro_programs", preAggregation: "dev_pre_aggregations.program__live_intro_programs", rows: 2 }, - // No dimension, no row in scope: the view reads one row with the count 0 and the default; Cube - // sums the (empty) rollup, which is null for every measure, the count and the default included. - { report: "ProgramsEmptyScope", view: "v_programs_empty_scope", preAggregation: "dev_pre_aggregations.program__programs_empty_scope", rows: 1, cubeRows: [{ avgPriceOrMinusOne: null, programs: null }] }, - { report: "WeekShapes", view: "v_week_shapes", preAggregation: "dev_pre_aggregations.week__week_shapes", rows: 3 }, - { report: "SessionsByProgram", view: "v_sessions_by_program", preAggregation: "dev_pre_aggregations.session__sessions_by_program", rows: 3 }, - { report: "ProgramSessions", view: "v_program_sessions", preAggregation: undefined, rows: 11 }, - { report: "NodesByParent", view: "v_nodes_by_parent", preAggregation: "dev_pre_aggregations.node__nodes_by_parent", rows: 3 }, - { report: "AuthsByKind", view: "v_auths_by_kind", preAggregation: "dev_pre_aggregations.auth__auths_by_kind", rows: 3 }, -]; +const TABLE_I = CANONICAL_REPORTS; // --------------------------------------------------------------------------------------------- // Skip semantics: never a silent pass. @@ -269,7 +245,9 @@ describe(`cube-model live check (${CUBE_IMAGE}, development mode)`, () => { // Table I's rollup column agrees with the golden: a rollup named after the report on its // `@from` cube exactly when Table I expects Cube to answer from one. const cube = reportShape(report, requireCanonical()).from.name; - expect(declaredRollups(golden, cube).includes(want.report)).toBe(want.preAggregation !== undefined || want.unusedRollup === true); + // (A report listing one time dimension at several grains has one rollup per grain: `_`.) + const hasRollup = declaredRollups(golden, cube).some((n) => n === want.report || n.startsWith(`${want.report}_`)); + expect(hasRollup).toBe(want.preAggregation !== undefined); // And a Cube view named after the report exactly when it declares @spine. expect(views.includes(want.report)).toBe(reportSpine(report) !== undefined); } @@ -286,7 +264,7 @@ describe(`cube-model live check (${CUBE_IMAGE}, development mode)`, () => { // Rows first, so a rollup mismatch can say whether the numbers were right anyway. const cubeRows = sortRows(result.data.map((row) => fromCubeRow(row, plan, GEN_CONFIG.dialect)), plan); - const viewRows = sortRows(selectView(s, want.view).map((row) => fromViewRow(row, plan, want.view)), plan); + const viewRows = sortRows(selectViewPg(s, want.view).map((row) => fromViewRow(row, plan, want.view)), plan); const rowsMatch = Bun.deepEquals(cubeRows, viewRows); const used = Object.keys(result.usedPreAggregations ?? {}).sort(); const wantUsed = want.preAggregation === undefined ? [] : [want.preAggregation]; @@ -424,24 +402,6 @@ function findReport(name: string): MetaObject { return report; } -/** - * `SELECT * FROM ` through psql, every value as the TEXT Postgres prints for it (a numeric - * keeps its scale, a timestamptz its offset), in a UTC session. - */ -function selectView(s: CubeStack, view: string): Array> { - const out = s.psql( - `SET TIME ZONE 'UTC';\n` + - `SELECT coalesce(json_agg(t.o), '[]'::json)::text FROM (\n` + - ` SELECT (SELECT json_object_agg(e.key, e.value) FROM json_each_text(row_to_json(v)) e) AS o\n` + - ` FROM "${view}" v\n` + - `) t;\n`, - ); - const line = out.trim().split("\n").filter((l) => l.trim() !== "").pop() ?? ""; - const parsed: unknown = JSON.parse(line); - if (!Array.isArray(parsed)) throw new Error(`${view}: psql did not print a JSON array: ${line.slice(0, 200)}`); - return parsed as Array>; -} - // --------------------------------------------------------------------------------------------- // What the files declare, and what Cube lists. // --------------------------------------------------------------------------------------------- @@ -590,7 +550,7 @@ function declaredRollups(tree: ReadonlyMap, cube: string): strin /** * Mapping-corpus cases with an `expected/` tree, of either dialect. Compiling a model runs no SQL * (`/v1/meta` neither queries the data source nor builds a rollup), so a MySQL case compiles - * against the Postgres data source as well. Its SQL is held by the golden, not executed. + * against the Postgres data source as well. Its SQL is executed by cube-model-mysql-cases.live.ts. */ function corpusCasesWithTree(): string[] { return readdirSync(CUBE_CORPUS_DIR) diff --git a/server/typescript/packages/integration-tests/cube-live/cube-prod-stack.ts b/server/typescript/packages/integration-tests/cube-live/cube-prod-stack.ts new file mode 100644 index 000000000..bd9ccf439 --- /dev/null +++ b/server/typescript/packages/integration-tests/cube-live/cube-prod-stack.ts @@ -0,0 +1,151 @@ +// cube-prod-stack.ts — Cube's PRODUCTION mode for the live lane: `CUBEJS_DEV_MODE=false`, a signed +// JWT on every request, and a real Cube Store container that holds the rollups. +// +// It adds to a stack src/cube-container.ts already started (the private network, the seeded +// database, the model directory) rather than replacing it: startCubeStack owns the network and the +// database, and `stop()` here removes the two containers it adds before handing the rest to the +// base stack's own `stop()`. Everything stays on the private network; the one published port is +// the API's, ephemeral and bound to 127.0.0.1. + +import { spawnSync } from "node:child_process"; +import { createHmac, randomUUID } from "node:crypto"; +import { CUBE_IMAGE, type CubeDataSource, type CubeStack } from "../src/cube-container.ts"; +import type { CubeEndpoint } from "./cube-live-support.ts"; + +/** Cube Store, pinned to the release of the Cube image (a rollup is written and read through it). */ +export const CUBE_STORE_IMAGE = "cubejs/cubestore:v1.7.43"; + +/** The credentials src/cube-container.ts gives the throwaway database it starts. */ +const DB_PASSWORD = "test"; +const MYSQL_DATABASE = "mo_test"; + +const READY_TIMEOUT_S = 300; +const PULL_TIMEOUT_MS = 15 * 60_000; + +export interface ProdCube extends CubeEndpoint { + readonly headers: Readonly>; + readonly storeContainer: string; + readonly cubeContainer: string; + /** Remove Cube, Cube Store and then the base stack. Idempotent. */ + stop(): void; +} + +/** An HS256 JWT over an empty security context, valid for a day: what production mode requires. */ +function signJwt(secret: string): string { + const b64 = (o: unknown): string => Buffer.from(JSON.stringify(o)).toString("base64url"); + const now = Math.floor(Date.now() / 1000); + const body = `${b64({ alg: "HS256", typ: "JWT" })}.${b64({ iat: now, exp: now + 86_400 })}`; + return `${body}.${createHmac("sha256", secret).update(body).digest("base64url")}`; +} + +function docker(args: string[]): string { + const r = spawnSync("docker", args, { encoding: "utf8", timeout: 120_000 }); + if (r.status !== 0) throw new Error(`docker ${args.join(" ")} failed: ${r.stderr || r.stdout || r.error?.message || ""}`); + return r.stdout.trim(); +} + +function tail(name: string, lines: number): string { + const r = spawnSync("docker", ["logs", "--tail", String(lines), name], { encoding: "utf8", timeout: 30_000 }); + return r.status === 0 ? `${r.stdout}${r.stderr}`.trim() : "(docker logs unavailable)"; +} + +function ensureImage(image: string): void { + if (spawnSync("docker", ["image", "inspect", image], { stdio: "ignore", timeout: 60_000 }).status === 0) return; + const r = spawnSync("docker", ["pull", "--quiet", image], { encoding: "utf8", timeout: PULL_TIMEOUT_MS }); + if (r.status !== 0) throw new Error(`docker pull ${image} failed: ${r.stderr || r.stdout}`); +} + +/** + * Start Cube Store and a production-mode Cube on `base`'s network, reading `base`'s database and the + * model in `modelDir`. The base stack's development-mode Cube is removed first: it has no further + * use, and only one Cube should read the model. + */ +export async function startProductionCube(base: CubeStack, modelDir: string, dataSource: CubeDataSource): Promise { + ensureImage(CUBE_STORE_IMAGE); + spawnSync("docker", ["rm", "-f", "-v", base.cubeContainer], { stdio: "ignore", timeout: 120_000 }); + + const storeContainer = `${base.network}-store`; + const cubeContainer = `${base.network}-prodcube`; + let stopped = false; + const stop = (): void => { + if (stopped) return; + stopped = true; + process.off("exit", stop); + process.off("SIGINT", onSignal); + process.off("SIGTERM", onSignal); + for (const name of [cubeContainer, storeContainer]) spawnSync("docker", ["rm", "-f", "-v", name], { stdio: "ignore", timeout: 120_000 }); + base.stop(); + }; + const onSignal = (signal: NodeJS.Signals): void => { + stop(); + process.exit(signal === "SIGINT" ? 130 : 143); + }; + // Ahead of the base stack's own handlers: its network cannot be removed while ours are attached. + process.prependListener("exit", stop); + process.prependListener("SIGINT", onSignal); + process.prependListener("SIGTERM", onSignal); + + try { + docker(["run", "-d", "--name", storeContainer, "--network", base.network, CUBE_STORE_IMAGE]); + const secret = randomUUID(); + const dbEnv = + dataSource === "mysql" + ? ["CUBEJS_DB_TYPE=mysql", `CUBEJS_DB_HOST=${base.pgContainer}`, "CUBEJS_DB_PORT=3306", + `CUBEJS_DB_NAME=${MYSQL_DATABASE}`, "CUBEJS_DB_USER=root", `CUBEJS_DB_PASS=${DB_PASSWORD}`] + : ["CUBEJS_DB_TYPE=postgres", `CUBEJS_DB_HOST=${base.pgContainer}`, "CUBEJS_DB_PORT=5432", + "CUBEJS_DB_NAME=postgres", "CUBEJS_DB_USER=postgres", `CUBEJS_DB_PASS=${DB_PASSWORD}`]; + const env = [ + ...dbEnv, + "CUBEJS_DEV_MODE=false", + "NODE_ENV=production", + "CUBEJS_TELEMETRY=false", + `CUBEJS_API_SECRET=${secret}`, + `CUBEJS_CUBESTORE_HOST=${storeContainer}`, + "CUBEJS_CUBESTORE_PORT=3030", + // The refresh worker runs in this process too; the lane also builds explicitly through the + // jobs API (buildAllPreAggregations) and waits for every job, so nothing races a timer. + "CUBEJS_REFRESH_WORKER=true", + // The jobs API is off by default (`API scope is missing: jobs`). + "CUBEJS_DEFAULT_API_SCOPES=meta,data,jobs", + ]; + docker([ + "run", "-d", "--name", cubeContainer, "--network", base.network, + "-p", "127.0.0.1::4000", + ...env.flatMap((e) => ["-e", e]), + "-v", `${modelDir}:/cube/conf/model:ro`, + CUBE_IMAGE, + ]); + const out = docker(["port", cubeContainer, "4000/tcp"]); + const m = /:(\d+)\s*$/m.exec(out.split("\n")[0] ?? ""); + if (m === null) throw new Error(`docker port ${cubeContainer} 4000/tcp printed no port: '${out}'`); + const apiBase = `http://127.0.0.1:${Number(m[1])}/cubejs-api/v1`; + const headers = { Authorization: signJwt(secret) }; + const cubeLogs = (lines = 60): string => `cube:\n${tail(cubeContainer, lines)}\ncube store:\n${tail(storeContainer, Math.min(lines, 30))}`; + + // Ready when /meta answers 200 with the token: production mode answers 403 without one. + const deadline = Date.now() + READY_TIMEOUT_S * 1000; + let last = "no answer yet"; + for (;;) { + if (Date.now() > deadline) throw new Error(`production Cube did not answer ${apiBase}/meta within ${READY_TIMEOUT_S}s (last: ${last})`); + const state = spawnSync("docker", ["inspect", "-f", "{{.State.Status}}", cubeContainer], { encoding: "utf8", timeout: 30_000 }).stdout.trim(); + if (state !== "running") throw new Error(`production Cube container is '${state}', not running`); + try { + const res = await fetch(`${apiBase}/meta`, { headers, signal: AbortSignal.timeout(10_000) }); + const text = await res.text(); + if (res.status === 200) { + JSON.parse(text); + break; + } + last = `HTTP ${res.status}: ${text.slice(0, 200)}`; + } catch (e) { + last = e instanceof Error ? e.message : String(e); + } + await Bun.sleep(1000); + } + return { apiBase, headers, storeContainer, cubeContainer, cubeLogs, stop }; + } catch (e) { + const logs = `cube:\n${tail(cubeContainer, 60)}\ncube store:\n${tail(storeContainer, 30)}`; + stop(); + throw new Error(`production Cube failed to start: ${e instanceof Error ? e.message : String(e)}\n${logs}`, { cause: e }); + } +} diff --git a/server/typescript/packages/integration-tests/package.json b/server/typescript/packages/integration-tests/package.json index c70762423..4b8e2eb6e 100644 --- a/server/typescript/packages/integration-tests/package.json +++ b/server/typescript/packages/integration-tests/package.json @@ -6,9 +6,10 @@ "type": "module", "scripts": { "test": "bun test", - "test:cube": "bun test ./cube-live/cube-model.live.ts ./cube-live/cube-model-mysql.live.ts", + "test:cube": "bun test ./cube-live/cube-model.live.ts ./cube-live/cube-model-mysql.live.ts ./cube-live/cube-model-mysql-cases.live.ts ./cube-live/cube-model-production.live.ts ./cube-live/cube-model-shapes.live.ts", "gen:schema": "bun run src/gen-canonical-schema.ts", "gen:report-api-schema": "bun run src/gen-api-contract-report-schema.ts", + "gen:decimal-api-schema": "bun run src/gen-api-contract-decimal-schema.ts", "gen:report-shapes": "bun run src/gen-report-shapes.ts", "oracle": "bun run src/oracle-cli.ts" }, diff --git a/server/typescript/packages/integration-tests/src/api-contract-decimal-schema.ts b/server/typescript/packages/integration-tests/src/api-contract-decimal-schema.ts new file mode 100644 index 000000000..29a522a13 --- /dev/null +++ b/server/typescript/packages/integration-tests/src/api-contract-decimal-schema.ts @@ -0,0 +1,37 @@ +// api-contract-decimal-schema.ts — the schema artifacts of the api-contract `decimal/` sub-corpus +// (ADR-0019 Amendment 1): one table, `entries`, and its report view `v_entry_totals`. +// +// Postgres and SQLite: `meta migrate`'s own output. MySQL: the adopter's hand-written table +// (ADR-0015) and the view `buildReportViews(root, { dialect: "mysql" })` lowers. + +import type { MetaRoot } from "@metaobjectsdev/metadata"; + +import { generateCanonicalSchemaSql } from "./canonical-schema.ts"; +import { renderMysqlSchemaArtifact } from "./canonical-schema-mysql.ts"; + +export type DecimalApiSchemaDialect = "postgres" | "sqlite" | "mysql"; + +const LABEL: Record = { postgres: "Postgres", sqlite: "SQLite", mysql: "MySQL 8.4" }; + +function header(dialect: DecimalApiSchemaDialect): string { + return [ + "-- @generated by @metaobjectsdev/integration-tests — DO NOT EDIT.", + `-- api-contract decimal sub-corpus schema (${LABEL[dialect]}, literal column naming).`, + "-- Regenerate: `bun run gen:decimal-api-schema` in server/typescript/packages/integration-tests.", + ].join("\n"); +} + +const MYSQL_TABLES = [ + `CREATE TABLE \`entries\` ( + \`id\` BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY, + \`bucket\` VARCHAR(8) NOT NULL, + \`amount\` DECIMAL(28,10), + \`flagged\` BOOLEAN NOT NULL +)`, +]; + +/** The bytes of decimal/schema..sql for a loaded decimal/meta.json. */ +export async function generateDecimalApiSchemaSql(root: MetaRoot, dialect: DecimalApiSchemaDialect): Promise { + if (dialect === "mysql") return renderMysqlSchemaArtifact(header(dialect), MYSQL_TABLES, root); + return generateCanonicalSchemaSql(root, { header: header(dialect), dialect }); +} diff --git a/server/typescript/packages/integration-tests/src/api-contract-decimal-server.ts b/server/typescript/packages/integration-tests/src/api-contract-decimal-server.ts new file mode 100644 index 000000000..ca0f445a0 --- /dev/null +++ b/server/typescript/packages/integration-tests/src/api-contract-decimal-server.ts @@ -0,0 +1,201 @@ +// api-contract-decimal-server.ts — boots the GENERATED routes of the decimal sub-corpus +// (api-contract-conformance/decimal/) on Postgres, MySQL, SQLite and Cloudflare D1's local runtime +// and drives them over HTTP (ADR-0019 Amendment 1). +// +// The model is one writable entity (`Entry`, a decimal(28,10) `amount`) and one served report +// (`EntryTotals`, whose measures are decimal sums, an average, a maximum and two ratios). The +// server runs the real codegen over decimal/meta.json for the engine's dialect, executes the +// COMMITTED schema artifact for that engine (the table and the view the real lowering produces), +// imports each EMITTED .routes.ts unmodified and mounts it. Nothing is hand-rolled +// between the HTTP request and the database, which is the point: the wire rule is a property of +// the deployed artifact. + +import Fastify, { type FastifyInstance } from "fastify"; +import { randomUUID } from "node:crypto"; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { runGen, defineConfig } from "@metaobjectsdev/codegen-ts"; +import { entityFile, routesFile } from "@metaobjectsdev/test-generators"; +import mysql from "mysql2/promise"; +import pg from "pg"; +import { startLocalD1 } from "./query-scenario-d1.ts"; +import { loadMetadataFile } from "./load-metadata.ts"; +import { seedInserts, type ReportSeed } from "./api-contract-report-generated-server.ts"; +import { + API_CONTRACT_DECIMAL_MYSQL_SCHEMA_SQL_PATH, + API_CONTRACT_DECIMAL_SCHEMA_SQL_PATH, + API_CONTRACT_DECIMAL_SQLITE_SCHEMA_SQL_PATH, +} from "./paths.ts"; +import { splitStatements } from "./sql-script.ts"; + +export type DecimalEngine = "postgres" | "mysql" | "sqlite" | "d1"; + +/** The served objects: the emitted routes file and the route the corpus calls. */ +const SERVED = ["Entry", "EntryTotals"] as const; + +export interface DecimalServerHandle { + baseUrl: string; + applySeed(seed: ReportSeed): Promise; + close(): Promise; +} + +interface EngineClient { + /** Run one statement; `?` placeholders (translated to `$n` on Postgres). */ + run(sql: string, params?: unknown[]): Promise; + /** Make the next generated id follow the highest explicit one (Postgres identity does not). */ + afterSeed(): Promise; + close(): Promise; +} + +const SCHEMA_PATH: Record = { + postgres: API_CONTRACT_DECIMAL_SCHEMA_SQL_PATH, + mysql: API_CONTRACT_DECIMAL_MYSQL_SCHEMA_SQL_PATH, + sqlite: API_CONTRACT_DECIMAL_SQLITE_SCHEMA_SQL_PATH, + d1: API_CONTRACT_DECIMAL_SQLITE_SCHEMA_SQL_PATH, +}; + +export async function startDecimalServer( + engine: DecimalEngine, + metaPath: string, + /** The Postgres or MySQL connection URL; unused by SQLite and D1. */ + connectionUrl?: string, +): Promise { + const here = dirname(fileURLToPath(import.meta.url)); + const genTmpRoot = join(here, "..", ".gen-tmp"); + mkdirSync(genTmpRoot, { recursive: true }); + const tmp = mkdtempSync(join(genTmpRoot, `api-contract-decimal-${engine}-`)); + const fail = (message: string): never => { + rmSync(tmp, { recursive: true, force: true }); + throw new Error(message); + }; + + // The root carries the corpus's literal naming (the nearest .metaobjects/config.json), the + // spelling the committed schema artifact uses, so the emitted tables read it unmodified. + const root = await loadMetadataFile(metaPath); + const lr = await runGen({ + config: defineConfig({ + outDir: tmp, + extStyle: "none", + dbImport: "./db", + dialect: engine === "d1" ? "sqlite" : engine, // codegen has no d1 dialect: D1 is SQLite to Drizzle + apiPrefix: "/api", + generators: [entityFile(), routesFile()], + }), + metadata: root, + }); + for (const name of SERVED) if (!existsSync(join(tmp, `${name}.routes.ts`))) fail(`codegen emitted no ${name}.routes.ts`); + if (lr.warnings.length > 0) fail(`codegen produced warnings: ${lr.warnings.join("; ")}`); + + const client = await openEngine(engine, tmp, connectionUrl); + for (const stmt of splitStatements(readFileSync(SCHEMA_PATH[engine], "utf8"))) await client.run(stmt); + + const fastify = Fastify(); + for (const name of SERVED) { + const mod = (await import(pathToFileURL(join(tmp, `${name}.routes.ts`)).href)) as Record; + const registrar = Object.values(mod).find((v): v is (f: FastifyInstance) => Promise => typeof v === "function"); + if (registrar === undefined) fail(`${name}.routes.ts exports no registrar`); + await fastify.register(registrar!); + } + await fastify.ready(); + const baseUrl = await fastify.listen({ host: "127.0.0.1", port: 0 }); + + return { + baseUrl, + applySeed: async (seed: ReportSeed) => { + await client.run(engine === "postgres" ? `TRUNCATE TABLE "entries" RESTART IDENTITY` : engine === "mysql" ? "TRUNCATE TABLE `entries`" : `DELETE FROM "entries"`); + for (const r of seedInserts(root, seed)) { + const q = engine === "mysql" ? "`" : '"'; + await client.run( + `INSERT INTO ${q}${r.table}${q} (${r.columns.map((c) => `${q}${c}${q}`).join(",")}) ` + + `VALUES (${r.columns.map(() => "?").join(",")})`, + r.values, + ); + } + await client.afterSeed(); + }, + close: async () => { + await fastify.close(); + await client.close(); + rmSync(tmp, { recursive: true, force: true }); + }, + }; +} + +/** Write the `db` module the emitted routes import and open the engine behind it. */ +async function openEngine(engine: DecimalEngine, tmp: string, url: string | undefined): Promise { + if (engine === "postgres") { + if (url === undefined) throw new Error("postgres needs a connection URL"); + const bigintTypes = pathToFileURL(join(dirname(fileURLToPath(import.meta.url)), "pg-bigint-number-types.ts")).href; + writeFileSync(join(tmp, "db.ts"), ` +import { drizzle } from "drizzle-orm/node-postgres"; +import pg from "pg"; +import { bigintAsNumberTypes } from ${JSON.stringify(bigintTypes)}; +export const pool = new pg.Pool({ connectionString: ${JSON.stringify(url)}, types: bigintAsNumberTypes }); +export const db = drizzle(pool); +`, "utf8"); + const mod = (await import(pathToFileURL(join(tmp, "db.ts")).href)) as { pool: pg.Pool }; + const admin = new pg.Pool({ connectionString: url }); + return { + run: async (sql, params = []) => { + let n = 0; + await admin.query(sql.replace(/\?/g, () => `$${++n}`), params); + }, + afterSeed: async () => { + await admin.query(`SELECT setval(pg_get_serial_sequence('"entries"', 'id'), (SELECT COALESCE(MAX("id"), 1) FROM "entries"))`); + }, + close: async () => { await mod.pool.end(); await admin.end(); }, + }; + } + if (engine === "mysql") { + if (url === undefined) throw new Error("mysql needs a connection URL"); + writeFileSync(join(tmp, "db.ts"), ` +import { drizzle } from "drizzle-orm/mysql2"; +import { createPool } from "mysql2"; +export const pool = createPool({ uri: ${JSON.stringify(url)}, timezone: "Z", supportBigNumbers: true }); +export const db = drizzle(pool); +`, "utf8"); + const mod = (await import(pathToFileURL(join(tmp, "db.ts")).href)) as { pool: { end(cb: () => void): void } }; + const admin = await mysql.createConnection({ uri: url, timezone: "Z" }); + // A previous server on this database may have left the objects behind. + await admin.query("DROP VIEW IF EXISTS `v_entry_totals`"); + await admin.query("DROP TABLE IF EXISTS `entries`"); + return { + run: async (sql, params = []) => { await admin.query(sql, params); }, + afterSeed: async () => {}, + close: async () => { + await new Promise((done) => mod.pool.end(done)); + await admin.query("DROP VIEW IF EXISTS `v_entry_totals`"); + await admin.query("DROP TABLE IF EXISTS `entries`"); + await admin.end(); + }, + }; + } + if (engine === "d1") { + const local = await startLocalD1(); + // One key per server, so two servers in flight never read each other's binding. + const key = `__decimalD1_${randomUUID().replaceAll("-", "")}`; + (globalThis as Record)[key] = local.d1; + writeFileSync(join(tmp, "db.ts"), ` +import { drizzle } from "drizzle-orm/d1"; +export const db = drizzle((globalThis as Record)[${JSON.stringify(key)}]); +`, "utf8"); + return { + run: async (sql, params = []) => { await local.d1.prepare(sql).bind(...params).all(); }, + afterSeed: async () => {}, + close: async () => { delete (globalThis as Record)[key]; await local.dispose(); }, + }; + } + writeFileSync(join(tmp, "db.ts"), ` +import { Database } from "bun:sqlite"; +import { drizzle } from "drizzle-orm/bun-sqlite"; +export const client = new Database(":memory:"); +export const db = drizzle(client); +`, "utf8"); + const mod = (await import(pathToFileURL(join(tmp, "db.ts")).href)) as { client: import("bun:sqlite").Database }; + return { + run: async (sql, params = []) => { mod.client.run(sql, params as never); }, + afterSeed: async () => {}, + close: async () => { mod.client.close(); }, + }; +} diff --git a/server/typescript/packages/integration-tests/src/api-contract-report-generated-server.ts b/server/typescript/packages/integration-tests/src/api-contract-report-generated-server.ts index 0a6a8be7b..10ed3dc8b 100644 --- a/server/typescript/packages/integration-tests/src/api-contract-report-generated-server.ts +++ b/server/typescript/packages/integration-tests/src/api-contract-report-generated-server.ts @@ -138,6 +138,15 @@ const BASE_TABLES: ReadonlyArray<{ name: string; ddl: string }> = [ "refunded" boolean );`, }, + { + name: "tickets", + ddl: ` + CREATE TABLE IF NOT EXISTS "tickets" ( + "id" bigserial PRIMARY KEY, + "tags" text[], + "attrs" jsonb + );`, + }, ]; export interface GeneratedReportServerHandle { @@ -158,6 +167,7 @@ export const SERVED_REPORTS = [ { name: "SalesByCategory", registrar: "salesByCategoryRoutes" }, { name: "SalesBySku", registrar: "salesBySkuRoutes" }, { name: "ProductRefunds", registrar: "productRefundsRoutes" }, + { name: "TicketsByTags", registrar: "ticketsByTagsRoutes" }, ] as const; export async function startGeneratedReportServer( diff --git a/server/typescript/packages/integration-tests/src/api-contract-report-mysql-schema.ts b/server/typescript/packages/integration-tests/src/api-contract-report-mysql-schema.ts index a560bd28b..375145f35 100644 --- a/server/typescript/packages/integration-tests/src/api-contract-report-mysql-schema.ts +++ b/server/typescript/packages/integration-tests/src/api-contract-report-mysql-schema.ts @@ -47,6 +47,11 @@ const TABLES = [ \`amountCents\` BIGINT NOT NULL, \`refunded\` BOOLEAN, FOREIGN KEY (\`productId\`) REFERENCES \`products\` (\`id\`) +)`, + `CREATE TABLE \`tickets\` ( + \`id\` BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY, + \`tags\` JSON, + \`attrs\` JSON )`, ]; diff --git a/server/typescript/packages/integration-tests/src/api-contract-report-mysql-server.ts b/server/typescript/packages/integration-tests/src/api-contract-report-mysql-server.ts index 42d3737e6..9741a3b7e 100644 --- a/server/typescript/packages/integration-tests/src/api-contract-report-mysql-server.ts +++ b/server/typescript/packages/integration-tests/src/api-contract-report-mysql-server.ts @@ -64,6 +64,14 @@ const BASE_TABLES: ReadonlyArray<{ name: string; ddl: string }> = [ FOREIGN KEY (\`product_id\`) REFERENCES \`products\` (\`id\`) )`, }, + { + name: "tickets", + ddl: `CREATE TABLE \`tickets\` ( + \`id\` BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY, + \`tags\` JSON, + \`attrs\` JSON + )`, + }, ]; export async function startMysqlReportServer(mysqlUrl: string, metaPath: string): Promise { diff --git a/server/typescript/packages/integration-tests/src/api-contract-report-sqlite-server.ts b/server/typescript/packages/integration-tests/src/api-contract-report-sqlite-server.ts index 525a89efd..9ba53fb6e 100644 --- a/server/typescript/packages/integration-tests/src/api-contract-report-sqlite-server.ts +++ b/server/typescript/packages/integration-tests/src/api-contract-report-sqlite-server.ts @@ -80,6 +80,15 @@ const BASE_TABLES: ReadonlyArray<{ name: string; ddl: string }> = [ "refunded" INTEGER )`, }, + { + name: "tickets", + ddl: ` + CREATE TABLE "tickets" ( + "id" INTEGER PRIMARY KEY AUTOINCREMENT, + "tags" TEXT, + "attrs" TEXT + )`, + }, ]; export async function startSqliteReportServer( diff --git a/server/typescript/packages/integration-tests/src/api-contract-scenario.ts b/server/typescript/packages/integration-tests/src/api-contract-scenario.ts index cbe87d59d..e3b345064 100644 --- a/server/typescript/packages/integration-tests/src/api-contract-scenario.ts +++ b/server/typescript/packages/integration-tests/src/api-contract-scenario.ts @@ -56,8 +56,14 @@ export interface ApiRequest { export interface ApiScenario { name: string; - description?: string; - setup?: { truncate?: boolean }; + description?: string | undefined; + setup?: { truncate?: boolean } | undefined; + /** + * The engines that run this scenario (`postgres`, `mysql`, `sqlite`, `d1`), or undefined for + * every engine the lane has. A scenario whose subject an engine cannot hold (an exact decimal + * on SQLite, W4) says so here, once, instead of a runner skipping it by name. + */ + engines?: string[] | undefined; requests: ApiRequest[]; sourcePath: string; } @@ -75,6 +81,7 @@ function parseScenario(file: string): ApiScenario { name: String(raw["name"] ?? file), description: raw["description"] as string | undefined, setup: raw["setup"] as { truncate?: boolean } | undefined, + engines: raw["engines"] as string[] | undefined, requests: requests.map((r) => ({ id: String(r["id"]), method: String(r["method"]).toUpperCase() as HttpMethod, @@ -102,12 +109,43 @@ export async function runScenario(scenario: ApiScenario, baseUrl: string): Promi const bodyText = await res.text(); let body: unknown = null; if (bodyText.length > 0) { - try { body = JSON.parse(bodyText); } catch { body = bodyText; } + try { body = parseBodyExact(bodyText); } catch { body = bodyText; } } assertResponse(scenario.name, req, res.status, body); } } +/** Stands in for a JSON number whose token a double cannot hold; it equals no expectation. */ +export interface InexactNumber { readonly inexactNumber: string } + +/** + * Whether `value` (the double a parser made of `source`) is the number `source` spells. A decimal + * point and trailing fraction zeros are spelling; a digit the double dropped is not. + */ +function isExact(value: number, source: string): boolean { + if (Number.isSafeInteger(value) && /^-?\d+$/.test(source)) return true; + const plain = String(value); + if (/[eE]/.test(plain) || /[eE]/.test(source)) return false; + const trimmed = source.includes(".") ? source.replace(/0+$/, "").replace(/\.$/, "") : source; + return plain === trimmed; +} + +/** + * JSON.parse that does not let a number lose digits unnoticed. The assertion engine compares + * parsed values, so `12345678901234567.89` arriving as a JSON number would parse to a double and + * equal an expectation written with the same double, however wrong the server's digits were. A + * number whose token the double cannot reproduce is returned as an {@link InexactNumber}, which + * equals nothing the scenarios can write, so the comparison fails and names the token. + */ +export function parseBodyExact(text: string): unknown { + return JSON.parse(text, function (this: unknown, _key: string, value: unknown, context?: { source?: string }) { + if (typeof value === "number" && context?.source !== undefined && !isExact(value, context.source)) { + return { inexactNumber: context.source } satisfies InexactNumber; + } + return value; + } as Parameters[1]); +} + // --------------------------------------------------------------------------- // Assertion engine // --------------------------------------------------------------------------- diff --git a/server/typescript/packages/integration-tests/src/api-contract-tph-server.ts b/server/typescript/packages/integration-tests/src/api-contract-tph-server.ts index 7da66a1f4..d28b0e67a 100644 --- a/server/typescript/packages/integration-tests/src/api-contract-tph-server.ts +++ b/server/typescript/packages/integration-tests/src/api-contract-tph-server.ts @@ -14,6 +14,7 @@ import { pgTable, bigserial, text, varchar, integer, numeric, timestamp, check } import { z } from "zod"; import { mountCrudRoutes, + parseDecimalInput, type FilterAllowlist, type SortAllowlist, } from "@metaobjectsdev/runtime-ts/drizzle-fastify"; @@ -89,9 +90,27 @@ const writeOnceInsertShape = { issuedCurrency: z.string().max(3).nullable().optional(), }; +// CopayAuth.copayAmount is decimal(10,2): a string or a JSON number in, its canonical string +// stored, and a value the column would round or overflow refused (ADR-0019 Amendment 1, W2/W3), +// as the generated write schema does it. +const copayAmount = z + .union([z.string(), z.number()]) + .transform((v, ctx) => { + const c = parseDecimalInput(v); + const m = c === undefined ? null : /^-?(0|[1-9][0-9]{0,7})(?:\.([0-9]{1,2}))?$/.exec(c); + if (c === undefined || m === null) { + ctx.addIssue({ code: "custom", message: "Not a decimal(10,2) value" }); + return z.NEVER; + } + return c; + }) + .nullable() + .optional(); +const DECIMAL_COLUMNS = ["copayAmount"] as const; + // Per-subtype insert schemas (discriminator pinned, omitted at the route boundary). const BridgeInsert = z.object({ reference: z.string().min(1).max(80), quantity: z.number().int(), ...writeOnceInsertShape, ...autoSetInsertShape }); -const CopayInsert = z.object({ reference: z.string().min(1).max(80), copayAmount: z.string().nullable().optional(), ...writeOnceInsertShape, ...autoSetInsertShape }); +const CopayInsert = z.object({ reference: z.string().min(1).max(80), copayAmount, ...writeOnceInsertShape, ...autoSetInsertShape }); const PriorAuthInsert = z.object({ reference: z.string().min(1).max(80), approver: z.string().max(80).nullable().optional(), ...writeOnceInsertShape, ...autoSetInsertShape }); // Per-subtype update schemas — NOT derived via `.partial()` (that would @@ -99,7 +118,7 @@ const PriorAuthInsert = z.object({ reference: z.string().min(1).max(80), approve // creation timestamp on every update — the exact lost-update bug this gate // exists to catch). Declared explicitly instead, omitting autoCreatedAt. const BridgeUpdate = z.object({ reference: z.string().min(1).max(80).optional(), quantity: z.number().int().optional(), ...autoSetUpdateShape }); -const CopayUpdate = z.object({ reference: z.string().min(1).max(80).optional(), copayAmount: z.string().nullable().optional(), ...autoSetUpdateShape }); +const CopayUpdate = z.object({ reference: z.string().min(1).max(80).optional(), copayAmount, ...autoSetUpdateShape }); const PriorAuthUpdate = z.object({ reference: z.string().min(1).max(80).optional(), approver: z.string().max(80).nullable().optional(), ...autoSetUpdateShape }); const numberOps = ["eq", "ne", "gt", "gte", "lt", "lte", "in", "isNull"] as const; @@ -112,7 +131,7 @@ const BridgeFilterAllowlist = { const CopayFilterAllowlist = { id: { ops: numberOps, subType: "number" as const, leadingWildcard: false }, reference: { ops: stringOps, subType: "string" as const, leadingWildcard: false }, - copayAmount: { ops: numberOps, subType: "number" as const, leadingWildcard: false }, + copayAmount: { ops: numberOps, subType: "number" as const, leadingWildcard: false, format: "decimal" as const }, } as const satisfies FilterAllowlist; const PriorAuthFilterAllowlist = { id: { ops: numberOps, subType: "number" as const, leadingWildcard: false }, @@ -147,7 +166,7 @@ export async function startTphServer(connectionUri: string): Promise> { return shapes; } -/** Canonical decimal text: no trailing fractional zeros, no bare `.` — the normalization.ts rule. */ -function decimalText(v: unknown): unknown { - if (typeof v === "number") return Number.isInteger(v) ? String(v) : canonicalFloat(v); - if (typeof v === "bigint") return v.toString(); - if (typeof v === "string" && DECIMAL_RE.test(v)) { - return v.includes(".") ? canonicalDecimal(v) : v; +/** + * `Entity.field` -> subType of every plain entity field of the canonical model, for the rows of + * a non-report entity (the W4 decimal scenarios read `Entry`). Only `long` and `currency` are + * mapped from it: the corpus' wire form for a BIGINT is a string, which node-postgres delivers + * and SQLite (one INTEGER storage class) hands over as a number. + */ +let entityLongFields: Map> | undefined; +function entityIntegralFields(entity: string): Set | undefined { + if (!entityLongFields) { + entityLongFields = new Map(); + const root = JSON.parse(readFileSync(join(CANONICAL_DIR, "meta.fitness.json"), "utf8")) as { "metadata.root": { children?: unknown[] } }; + for (const node of root["metadata.root"].children ?? []) { + const [type, body] = Object.entries(node as Record)[0] ?? []; + if (type !== "object.entity" || body?.name === undefined) continue; + const names = new Set(); + for (const child of body.children ?? []) { + const [ctype, attrs] = Object.entries(child as Record>)[0] ?? []; + if ((ctype === "field.long" || ctype === "field.currency") && typeof attrs?.["name"] === "string") names.add(attrs["name"]); + } + entityLongFields.set(body.name, names); + } } - return v; + return entityLongFields.get(entity); } const pad = (n: number): string => n.toString().padStart(2, "0"); @@ -104,10 +116,22 @@ function instantText(engine: WireEngine, v: unknown, localTime: boolean): unknow ); } +/** The decimal-typed field names of report `entity`, from report-shapes.json; empty for any other entity. */ +export function reportDecimalFields(entity: string): readonly string[] { + const fields = reportShapes().get(entity); + return fields === undefined ? [] : [...fields].filter(([, f]) => f.subType === "decimal").map(([n]) => n); +} + /** Map one row of report `entity` from `engine`'s driver spelling onto the corpus' wire form. */ export function toWireRow(engine: WireEngine, entity: string, row: Record): Record { const fields = reportShapes().get(entity); - if (!fields) return row; + if (!fields) { + const integral = entityIntegralFields(entity); + if (integral === undefined) return row; + return Object.fromEntries( + Object.entries(row).map(([k, v]) => [k, integral.has(k) && (typeof v === "number" || typeof v === "bigint") ? String(v) : v]), + ); + } const out: Record = {}; for (const [name, value] of Object.entries(row)) { if (value === null || value === undefined) { out[name] = value; continue; } @@ -118,7 +142,10 @@ export function toWireRow(engine: WireEngine, entity: string, row: Record { + const root = await loadMetadataFile(join(API_CONTRACT_DECIMAL_DIR, "meta.json")); + writeArtifact(API_CONTRACT_DECIMAL_SCHEMA_SQL_PATH, await generateDecimalApiSchemaSql(root, "postgres")); + writeArtifact(API_CONTRACT_DECIMAL_SQLITE_SCHEMA_SQL_PATH, await generateDecimalApiSchemaSql(root, "sqlite")); + writeArtifact(API_CONTRACT_DECIMAL_MYSQL_SCHEMA_SQL_PATH, await generateDecimalApiSchemaSql(root, "mysql")); +} + +main().catch((err: unknown) => { + // eslint-disable-next-line no-console + console.error(err); + process.exit(1); +}); diff --git a/server/typescript/packages/integration-tests/src/paths.ts b/server/typescript/packages/integration-tests/src/paths.ts index 67b0cacc7..db7053aa1 100644 --- a/server/typescript/packages/integration-tests/src/paths.ts +++ b/server/typescript/packages/integration-tests/src/paths.ts @@ -8,7 +8,9 @@ const here = dirname(fileURLToPath(import.meta.url)); // src → integration-tests → packages → typescript → server → repo-root const repoRoot = resolve(here, "..", "..", "..", "..", ".."); -export const CORPUS_DIR = resolve(repoRoot, "fixtures", "persistence-conformance"); +// MO_CORPUS_DIR / MO_API_CONTRACT_DIR point the artifact generators at a working copy of a corpus, +// so a proposed corpus change can be regenerated and diffed without touching the committed one. +export const CORPUS_DIR = process.env["MO_CORPUS_DIR"] ?? resolve(repoRoot, "fixtures", "persistence-conformance"); export const CANONICAL_DIR = resolve(CORPUS_DIR, "canonical"); export const MIGRATIONS_DIR = resolve(CORPUS_DIR, "migrations"); export const QUERIES_DIR = resolve(CORPUS_DIR, "queries"); @@ -17,7 +19,7 @@ export const QUERIES_DIR = resolve(CORPUS_DIR, "queries"); export const REPORT_SHAPES_PATH = resolve(CORPUS_DIR, "report-shapes.json"); // fixtures/api-contract-conformance/ — cross-port REST API contract corpus. -export const API_CONTRACT_DIR = resolve(repoRoot, "fixtures", "api-contract-conformance"); +export const API_CONTRACT_DIR = process.env["MO_API_CONTRACT_DIR"] ?? resolve(repoRoot, "fixtures", "api-contract-conformance"); export const API_CONTRACT_SCENARIOS_DIR = resolve(API_CONTRACT_DIR, "scenarios"); // fixtures/api-contract-conformance/m2m/ — FR-018 many-to-many traversal corpus @@ -52,3 +54,11 @@ export const API_CONTRACT_REPORT_MYSQL_SCHEMA_SQL_PATH = resolve(API_CONTRACT_RE // fixtures/validation-conformance/ — cross-port generated input-validation corpus. export const VALIDATION_DIR = resolve(repoRoot, "fixtures", "validation-conformance"); + +// fixtures/api-contract-conformance/decimal/ — ADR-0019 Amendment 1: a decimal is exact on the wire +// (read, filter, sort, write, report) over HTTP, generated lane only. +export const API_CONTRACT_DECIMAL_DIR = resolve(API_CONTRACT_DIR, "decimal"); +export const API_CONTRACT_DECIMAL_SCENARIOS_DIR = resolve(API_CONTRACT_DECIMAL_DIR, "scenarios"); +export const API_CONTRACT_DECIMAL_SCHEMA_SQL_PATH = resolve(API_CONTRACT_DECIMAL_DIR, "schema.postgres.sql"); +export const API_CONTRACT_DECIMAL_SQLITE_SCHEMA_SQL_PATH = resolve(API_CONTRACT_DECIMAL_DIR, "schema.sqlite.sql"); +export const API_CONTRACT_DECIMAL_MYSQL_SCHEMA_SQL_PATH = resolve(API_CONTRACT_DECIMAL_DIR, "schema.mysql.sql"); diff --git a/server/typescript/packages/integration-tests/src/query-scenario.ts b/server/typescript/packages/integration-tests/src/query-scenario.ts index d4060f177..4ab454afd 100644 --- a/server/typescript/packages/integration-tests/src/query-scenario.ts +++ b/server/typescript/packages/integration-tests/src/query-scenario.ts @@ -16,14 +16,14 @@ import { ObjectManager, type Filter } from "@metaobjectsdev/runtime-ts"; import { kyselyDriver } from "@metaobjectsdev/runtime-ts/drivers"; import { type MetaRoot, - TYPE_OBJECT, TYPE_IDENTITY, + TYPE_OBJECT, TYPE_IDENTITY, TYPE_FIELD, FIELD_SUBTYPE_DECIMAL, IDENTITY_SUBTYPE_PRIMARY, IDENTITY_ATTR_FIELDS, } from "@metaobjectsdev/metadata"; import { Kysely, PostgresDialect } from "kysely"; import { Pool } from "pg"; import { readCanonicalSchemaSql } from "./canonical-schema.ts"; -import { toWireRow, type WireEngine } from "./engine-wire.ts"; +import { reportDecimalFields, toWireRow, type WireEngine } from "./engine-wire.ts"; import { loadMetadataDir } from "./load-metadata.ts"; import { canonicalJson, normalizeRow } from "./normalization.ts"; import { executeSql } from "./postgres-sql.ts"; @@ -105,10 +105,55 @@ export async function runScenarioQueries( continue; } const actual = toEngineWire(engine, spec, await execute(om, root, spec)); + assertDecimalsTyped(scenario.sourcePath, spec, actual, root, label ?? engine); assertResult(scenario.sourcePath, spec, actual, label ?? engine); } } +/** ADR-0019 Amendment 1, W1 — written out here, not imported, so the gate does not trust the code under test. */ +const DECIMAL_CANONICAL_RE = /^-?(0|[1-9][0-9]*)(\.[0-9]*[1-9])?$/; + +/** + * The typed decimal gate (ADR-0019 Amendment 1, W1). `normalizeRow` routes by the SHAPE of a + * value, so a decimal that arrived as a binary float (or as `"100.50"`) would be rewritten into + * the expected string and pass. Before normalizing, every field the MODEL declares `decimal` + * (an entity's own, or a report's, from report-shapes.json) must already be null or a string in + * the canonical spelling the runtime owes the wire. + */ +function assertDecimalsTyped(scenarioPath: string, spec: QuerySpec, actual: unknown, root: MetaRoot, engine: string): void { + if (spec.op === "count" || spec.op === "delete" || actual === null || actual === undefined) return; + const rows = (Array.isArray(actual) ? actual : [actual]) as Record[]; + const names = decimalFieldNames(root, spec.entity); + if (names.length === 0) return; + for (const row of rows) { + for (const name of names) { + const v = row[name]; + if (v === undefined || v === null) continue; + if (typeof v !== "string") { + throw new Error( + `${scenarioPath} / ${spec.name}: decimal ${spec.entity}.${name} arrived as a binary ${typeof v} (${String(v)}) on ${engine}; ` + + `a decimal is a canonical string at the runtime boundary`, + ); + } + if (!DECIMAL_CANONICAL_RE.test(v)) { + throw new Error( + `${scenarioPath} / ${spec.name}: decimal ${spec.entity}.${name} is ${JSON.stringify(v)} on ${engine}, not canonical ` + + `(plain notation, no trailing fraction zeros)`, + ); + } + } + } +} + +function decimalFieldNames(root: MetaRoot, entity: string): readonly string[] { + const own = root.children().find((c) => c.type === TYPE_OBJECT && c.name === entity); + const declared = own === undefined + ? [] + // ADR-0039: resolving. A decimal may be inherited via extends. + : own.children().filter((c) => c.type === TYPE_FIELD && c.subType === FIELD_SUBTYPE_DECIMAL).map((c) => c.name); + return [...new Set([...declared, ...reportDecimalFields(entity)])]; +} + /** A list result from a non-Postgres engine, mapped from its driver's spelling onto the corpus' wire form. */ function toEngineWire(engine: "postgres" | WireEngine, spec: QuerySpec, actual: unknown): unknown { if (engine === "postgres" || spec.op !== "list" || !Array.isArray(actual)) return actual; diff --git a/server/typescript/packages/integration-tests/src/scenario.ts b/server/typescript/packages/integration-tests/src/scenario.ts index 7df13df4a..7254ce221 100644 --- a/server/typescript/packages/integration-tests/src/scenario.ts +++ b/server/typescript/packages/integration-tests/src/scenario.ts @@ -24,6 +24,9 @@ export interface MigrationExpect { readonly applyUpThenQuery: { sql: string; rows: ReadonlyArray> } | null; } +export const QUERY_ENGINES = ["postgres", "mysql", "sqlite", "d1"] as const; +export type QueryEngine = (typeof QUERY_ENGINES)[number]; + export interface QueryScenario { readonly kind: "query"; readonly name: string; @@ -37,6 +40,14 @@ export interface QueryScenario { * calls the database clock, and every engine spells that differently). */ readonly seedDataEngine: Readonly> | null; + /** + * The engines that run this scenario (`postgres`, `mysql`, `sqlite`, `d1`), or null when the + * file does not say. Absent keeps the historical rule: Postgres runs every scenario and the + * other engines run the `report-*` ones. Present is EXACT: only the engines listed run it, so + * a scenario whose subject an engine cannot hold (an exact decimal on SQLite, a double) is + * declared once, in the file, instead of skipped by a runner. + */ + readonly engines: ReadonlyArray | null; readonly queries: ReadonlyArray; } @@ -97,12 +108,12 @@ export function loadQueries(dir: string): QueryScenario[] { } /** - * The `report-*` query scenarios — the only corpus subset the non-Postgres schema artifacts - * provision (their tables exist for what the reports read). One home for the predicate so a - * renamed or added dialect-gated scenario cannot be silently skipped by some lanes. + * The query scenarios `engine` runs: exactly the ones whose `engines:` lists it, and for a file + * with no `engines:` key, everything on Postgres and the `report-*` scenarios elsewhere. */ -export function loadReportQueries(dir: string): QueryScenario[] { - return loadQueries(dir).filter((s) => s.name.startsWith("report-")); +export function loadQueriesFor(dir: string, engine: QueryEngine): QueryScenario[] { + return loadQueries(dir).filter((s) => + s.engines !== null ? s.engines.includes(engine) : engine === "postgres" || s.name.startsWith("report-")); } export function loadMigration(yamlPath: string): MigrationScenario { @@ -142,6 +153,7 @@ export function loadQuery(yamlPath: string): QueryScenario { description: raw.description ?? "", sourcePath: yamlPath, seedData: raw["seed-data"] ?? null, + engines: parseEngines(raw.engines, yamlPath), seedDataEngine: (raw["seed-data-engine"] as Record | undefined) ?? null, queries: (raw.queries ?? []).map((q) => ({ name: required(q.name, yamlPath, "query.name"), @@ -203,6 +215,7 @@ interface QueryYaml { description?: string; "seed-data"?: string; "seed-data-engine"?: unknown; + engines?: unknown; queries?: { name?: string; op?: string; @@ -219,3 +232,11 @@ interface QueryYaml { expect?: unknown; }[]; } + +function parseEngines(raw: unknown, yamlPath: string): QueryEngine[] | null { + if (raw === undefined || raw === null) return null; + if (!Array.isArray(raw) || raw.length === 0 || !raw.every((e) => (QUERY_ENGINES as readonly unknown[]).includes(e))) { + throw new Error(`${yamlPath}: 'engines' must be a non-empty list drawn from ${QUERY_ENGINES.join(", ")}`); + } + return raw as QueryEngine[]; +} diff --git a/server/typescript/packages/integration-tests/test/api-contract-decimal.test.ts b/server/typescript/packages/integration-tests/test/api-contract-decimal.test.ts new file mode 100644 index 000000000..289337c38 --- /dev/null +++ b/server/typescript/packages/integration-tests/test/api-contract-decimal.test.ts @@ -0,0 +1,83 @@ +// ADR-0019 Amendment 1 api-contract conformance: a decimal is exact on the wire, GENERATED lane. +// +// Drives fixtures/api-contract-conformance/decimal/ over HTTP against the EMITTED routes of one +// writable entity and one served report, on Postgres, MySQL 8.4, SQLite and Cloudflare D1's local +// runtime. A scenario's `engines:` key says which engines run it (W4: only Postgres and MySQL hold +// 12345678901234567.89 exactly), and the `-15` scenarios are the subset every engine runs. +// Requires Docker for Postgres and MySQL. + +import { afterAll, beforeAll, describe, expect, test } from "bun:test"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import { API_CONTRACT_DECIMAL_DIR, API_CONTRACT_DECIMAL_SCENARIOS_DIR } from "../src/paths.ts"; +import { loadScenarios, runScenario } from "../src/api-contract-scenario.ts"; +import { startDecimalServer, type DecimalEngine } from "../src/api-contract-decimal-server.ts"; +import type { ReportSeed } from "../src/api-contract-report-generated-server.ts"; +import { startMysql, type MysqlContainerHandle } from "../src/mysql-container.ts"; +import { startPostgres } from "../src/postgres-container.ts"; + +const SEED = JSON.parse(readFileSync(join(API_CONTRACT_DECIMAL_DIR, "seed.json"), "utf8")) as ReportSeed; +const META_PATH = join(API_CONTRACT_DECIMAL_DIR, "meta.json"); +const ENGINES: readonly DecimalEngine[] = ["postgres", "mysql", "sqlite", "d1"]; + +let mysqlContainer: MysqlContainerHandle | undefined; +beforeAll(async () => { mysqlContainer = await startMysql(); }, 240_000); +afterAll(() => { mysqlContainer?.stop(); }, 60_000); + +/** Run `fn` against a fresh server on `engine`, seeded with `seed`. Postgres gets its own container. */ +async function withServer( + engine: DecimalEngine, + fn: (baseUrl: string) => Promise, + seed: ReportSeed = SEED, +): Promise { + if (engine === "postgres") { + const pg = await startPostgres(); + try { + const server = await startDecimalServer(engine, META_PATH, pg.connectionUri); + try { await server.applySeed(seed); await fn(server.baseUrl); } finally { await server.close(); } + } finally { + await pg.stop(); + } + return; + } + const server = await startDecimalServer(engine, META_PATH, engine === "mysql" ? mysqlContainer!.url : undefined); + try { await server.applySeed(seed); await fn(server.baseUrl); } finally { await server.close(); } +} + +const scenarios = loadScenarios(API_CONTRACT_DECIMAL_SCENARIOS_DIR); + +for (const engine of ENGINES) describe(`api contract decimal (ADR-0019 Amendment 1) — GENERATED routes on ${engine}`, () => { + for (const scenario of scenarios) { + if (scenario.engines !== undefined && !scenario.engines.includes(engine)) continue; + test(scenario.name, async () => { + await withServer(engine, (baseUrl) => runScenario(scenario, baseUrl)); + }, { timeout: 120_000 }); + } + + // seed.json's `reports` half is what the seam lanes of the other ports serve in place of the view. + // On the engines that hold the full seed exactly, the view's own answer must be that half. + if (engine === "postgres" || engine === "mysql") { + test("seed.json `reports` is exactly what the view answers", async () => { + await withServer(engine, async (baseUrl) => { + const res = await fetch(`${baseUrl}/api/entry_totals?sort=bucket:asc`); + expect(res.status).toBe(200); + expect(await res.json()).toEqual(SEED.reports!["EntryTotals"]!); + }); + }, { timeout: 120_000 }); + } + + test("a decimal is never a JSON number on the wire", async () => { + await withServer(engine, async (baseUrl) => { + const rows = (await (await fetch(`${baseUrl}/api/entries?filter[bucket][in]=A,D,E,F`)).json()) as Array>; + expect(rows.length).toBe(7); + for (const row of rows) expect(row["amount"] === null || typeof row["amount"] === "string").toBe(true); + const totals = (await (await fetch(`${baseUrl}/api/entry_totals?filter[bucket][in]=A,D,E,F`)).json()) as Array>; + for (const row of totals) { + for (const k of ["total", "totalOrZero", "mean", "top", "flaggedShare", "perFlagged"]) { + expect(row[k] === null || typeof row[k] === "string").toBe(true); + } + expect(typeof row["entries"]).toBe("number"); + } + }); + }, { timeout: 120_000 }); +}); diff --git a/server/typescript/packages/integration-tests/test/api-contract-generated.test.ts b/server/typescript/packages/integration-tests/test/api-contract-generated.test.ts index 7e7887402..31ac7d28b 100644 --- a/server/typescript/packages/integration-tests/test/api-contract-generated.test.ts +++ b/server/typescript/packages/integration-tests/test/api-contract-generated.test.ts @@ -15,7 +15,7 @@ import { describe, test } from "bun:test"; import { readFileSync } from "node:fs"; import { join } from "node:path"; import { API_CONTRACT_DIR, API_CONTRACT_SCENARIOS_DIR } from "../src/paths.ts"; -import { loadScenarios, assertResponse, type ApiScenario } from "../src/api-contract-scenario.ts"; +import { loadScenarios, assertResponse, type ApiScenario, parseBodyExact } from "../src/api-contract-scenario.ts"; import { startGeneratedServer, type GeneratedServerHandle, @@ -56,7 +56,7 @@ async function executeRequests(scenario: ApiScenario, server: GeneratedServerHan const bodyText = await res.text(); let body: unknown = null; if (bodyText.length > 0) { - try { body = JSON.parse(bodyText); } catch { body = bodyText; } + try { body = parseBodyExact(bodyText); } catch { body = bodyText; } } assertResponse(scenario.name, req, res.status, body); } diff --git a/server/typescript/packages/integration-tests/test/api-contract-jsonb.test.ts b/server/typescript/packages/integration-tests/test/api-contract-jsonb.test.ts index 926f831ef..0562be504 100644 --- a/server/typescript/packages/integration-tests/test/api-contract-jsonb.test.ts +++ b/server/typescript/packages/integration-tests/test/api-contract-jsonb.test.ts @@ -15,7 +15,7 @@ import { describe, test } from "bun:test"; import { readFileSync } from "node:fs"; import { join } from "node:path"; import { API_CONTRACT_JSONB_DIR, API_CONTRACT_JSONB_SCENARIOS_DIR } from "../src/paths.ts"; -import { loadScenarios, assertResponse, type ApiScenario } from "../src/api-contract-scenario.ts"; +import { loadScenarios, assertResponse, type ApiScenario, parseBodyExact } from "../src/api-contract-scenario.ts"; import { startPostgres } from "../src/postgres-container.ts"; import { loadMetadataFile } from "../src/load-metadata.ts"; import { startJsonbServer, type JsonbServerHandle, type DocumentRow } from "../src/api-contract-jsonb-server.ts"; @@ -75,7 +75,7 @@ async function runScenario(scenario: ApiScenario, server: Lane): Promise { const bodyText = await res.text(); let body: unknown = null; if (bodyText.length > 0) { - try { body = JSON.parse(bodyText); } catch { body = bodyText; } + try { body = parseBodyExact(bodyText); } catch { body = bodyText; } } assertResponse(scenario.name, req, res.status, body); } diff --git a/server/typescript/packages/integration-tests/test/api-contract-m2m.test.ts b/server/typescript/packages/integration-tests/test/api-contract-m2m.test.ts index aa12c0b1b..3a393b7a5 100644 --- a/server/typescript/packages/integration-tests/test/api-contract-m2m.test.ts +++ b/server/typescript/packages/integration-tests/test/api-contract-m2m.test.ts @@ -14,7 +14,7 @@ import { describe, test } from "bun:test"; import { readFileSync } from "node:fs"; import { join } from "node:path"; import { API_CONTRACT_M2M_DIR, API_CONTRACT_M2M_SCENARIOS_DIR } from "../src/paths.ts"; -import { loadScenarios, assertResponse, type ApiScenario } from "../src/api-contract-scenario.ts"; +import { loadScenarios, assertResponse, type ApiScenario, parseBodyExact } from "../src/api-contract-scenario.ts"; import { startPostgres } from "../src/postgres-container.ts"; import { startM2mServer, type M2mServerHandle } from "../src/api-contract-m2m-server.ts"; import { @@ -73,7 +73,7 @@ async function runScenario(scenario: ApiScenario, server: Lane): Promise { const bodyText = await res.text(); let body: unknown = null; if (bodyText.length > 0) { - try { body = JSON.parse(bodyText); } catch { body = bodyText; } + try { body = parseBodyExact(bodyText); } catch { body = bodyText; } } assertResponse(scenario.name, req, res.status, body); } diff --git a/server/typescript/packages/integration-tests/test/api-contract-projection.test.ts b/server/typescript/packages/integration-tests/test/api-contract-projection.test.ts index 1107eaa3e..2997b2015 100644 --- a/server/typescript/packages/integration-tests/test/api-contract-projection.test.ts +++ b/server/typescript/packages/integration-tests/test/api-contract-projection.test.ts @@ -17,7 +17,7 @@ import { API_CONTRACT_PROJECTION_DIR, API_CONTRACT_PROJECTION_SCENARIOS_DIR, } from "../src/paths.ts"; -import { loadScenarios, assertResponse, type ApiScenario } from "../src/api-contract-scenario.ts"; +import { loadScenarios, assertResponse, type ApiScenario, parseBodyExact } from "../src/api-contract-scenario.ts"; import { startPostgres } from "../src/postgres-container.ts"; import { startGeneratedProjectionServer, @@ -61,7 +61,7 @@ async function runScenario( const bodyText = await res.text(); let body: unknown = null; if (bodyText.length > 0) { - try { body = JSON.parse(bodyText); } catch { body = bodyText; } + try { body = parseBodyExact(bodyText); } catch { body = bodyText; } } assertResponse(scenario.name, req, res.status, body); } diff --git a/server/typescript/packages/integration-tests/test/api-contract-report-corpus.test.ts b/server/typescript/packages/integration-tests/test/api-contract-report-corpus.test.ts index 36a726c42..62330dab5 100644 --- a/server/typescript/packages/integration-tests/test/api-contract-report-corpus.test.ts +++ b/server/typescript/packages/integration-tests/test/api-contract-report-corpus.test.ts @@ -26,20 +26,20 @@ describe("api-contract report corpus", () => { const reports = root.objects().filter((o) => o.subType === OBJECT_SUBTYPE_REPORT); expect(reports.map((r) => r.name)).toEqual( ["InvoiceStatusTotals", "InvoicesByMonth", "InvoiceTotals", "InvoiceCalendar", "InvoiceDisputes", "InvoiceDays", - "ProductRevenue", "SalesByCategory", "SalesBySku", "ProductRefunds"]); + "ProductRevenue", "SalesByCategory", "SalesBySku", "ProductRefunds", "TicketsByTags"]); expect(reports.filter((r) => reportReadSource(r) !== undefined).map((r) => r.name)).toEqual( ["InvoiceStatusTotals", "InvoicesByMonth", "InvoiceTotals", "InvoiceCalendar", "InvoiceDisputes", - "ProductRevenue", "SalesByCategory", "SalesBySku", "ProductRefunds"]); + "ProductRevenue", "SalesByCategory", "SalesBySku", "ProductRefunds", "TicketsByTags"]); }); test("the route segments are the ones the scenarios call", async () => { const root = await loadMetadataFile(join(API_CONTRACT_REPORT_DIR, "meta.json")); const segment = (name: string): string => pluralize(toSnakeCase(name)); expect(["InvoiceStatusTotals", "InvoicesByMonth", "InvoiceTotals", "InvoiceCalendar", "InvoiceDisputes", - "ProductRevenue", "SalesByCategory", "SalesBySku", "ProductRefunds"].map(segment)).toEqual( + "ProductRevenue", "SalesByCategory", "SalesBySku", "ProductRefunds", "TicketsByTags"].map(segment)).toEqual( ["invoice_status_totals", "invoices_by_months", "invoice_totals", "invoice_calendars", "invoice_disputes", - "product_revenues", "sales_by_categories", "sales_by_skus", "product_refunds"]); - expect(root.objects().length).toBe(14); + "product_revenues", "sales_by_categories", "sales_by_skus", "product_refunds", "tickets_by_tags"]); + expect(root.objects().length).toBe(16); }); test("each seeded report row has exactly the report's derived fields", async () => { @@ -77,9 +77,9 @@ describe("api-contract report corpus", () => { } }); - test("every scenario parses, and there are twenty-five", () => { + test("every scenario parses, and there are twenty-seven", () => { const scenarios = loadScenarios(API_CONTRACT_REPORT_SCENARIOS_DIR); - expect(scenarios.length).toBe(25); + expect(scenarios.length).toBe(27); for (const s of scenarios) expect(s.requests.length).toBeGreaterThan(0); }); diff --git a/server/typescript/packages/integration-tests/test/api-contract-report.test.ts b/server/typescript/packages/integration-tests/test/api-contract-report.test.ts index f6e1d8526..1c5268759 100644 --- a/server/typescript/packages/integration-tests/test/api-contract-report.test.ts +++ b/server/typescript/packages/integration-tests/test/api-contract-report.test.ts @@ -9,7 +9,7 @@ import { describe, expect, test } from "bun:test"; import { readFileSync } from "node:fs"; import { join } from "node:path"; import { API_CONTRACT_REPORT_DIR, API_CONTRACT_REPORT_SCENARIOS_DIR } from "../src/paths.ts"; -import { loadScenarios, assertResponse, type ApiScenario } from "../src/api-contract-scenario.ts"; +import { loadScenarios, assertResponse, type ApiScenario, parseBodyExact } from "../src/api-contract-scenario.ts"; import { startPostgres } from "../src/postgres-container.ts"; import { startGeneratedReportServer, @@ -106,7 +106,7 @@ async function runScenario( const bodyText = await res.text(); let body: unknown = null; if (bodyText.length > 0) { - try { body = JSON.parse(bodyText); } catch { body = bodyText; } + try { body = parseBodyExact(bodyText); } catch { body = bodyText; } } assertResponse(scenario.name, req, res.status, body); } diff --git a/server/typescript/packages/integration-tests/test/api-contract-scenario.test.ts b/server/typescript/packages/integration-tests/test/api-contract-scenario.test.ts new file mode 100644 index 000000000..57807e856 --- /dev/null +++ b/server/typescript/packages/integration-tests/test/api-contract-scenario.test.ts @@ -0,0 +1,34 @@ +// The assertion engine's number handling: a JSON number must not lose digits unnoticed. + +import { describe, expect, test } from "bun:test"; +import { assertResponse, parseBodyExact, type ApiRequest } from "../src/api-contract-scenario.ts"; + +describe("parseBodyExact", () => { + test("a number a double holds parses as that number", () => { + expect(parseBodyExact('{"a":1,"b":0.5,"c":-12.25,"d":9007199254740991,"e":1.0,"f":0.10}')).toEqual({ + a: 1, b: 0.5, c: -12.25, d: 9007199254740991, e: 1, f: 0.1, + }); + }); + + test("a number a double cannot hold is marked, not rounded", () => { + expect(parseBodyExact('{"amount":12345678901234567.89}')).toEqual({ amount: { inexactNumber: "12345678901234567.89" } }); + expect(parseBodyExact("[9007199254740993]")).toEqual([{ inexactNumber: "9007199254740993" }]); + expect(parseBodyExact("[1e400]")).toEqual([{ inexactNumber: "1e400" }]); + }); + + test("strings are untouched, so an exact decimal string compares as the string it is", () => { + expect(parseBodyExact('{"amount":"12345678901234567.89"}')).toEqual({ amount: "12345678901234567.89" }); + }); +}); + +describe("assertResponse with an inexact number", () => { + const req = (equals: unknown): ApiRequest => ({ id: "r1", method: "GET", path: "/x", expect: { status: 200, body: { equals } } }); + + test("a number does not equal the string the contract owes", () => { + expect(() => assertResponse("s", req([{ amount: "0.3" }]), 200, [{ amount: 0.3 }])).toThrow(/mismatch/); + }); + + test("a number beyond a double fails against the double an expectation can spell", () => { + expect(() => assertResponse("s", req([{ amount: 12345678901234567.89 }]), 200, parseBodyExact('[{"amount":12345678901234567.89}]'))).toThrow(/inexactNumber/); + }); +}); diff --git a/server/typescript/packages/integration-tests/test/api-contract-tph.test.ts b/server/typescript/packages/integration-tests/test/api-contract-tph.test.ts index 9e75000e4..cf1c55009 100644 --- a/server/typescript/packages/integration-tests/test/api-contract-tph.test.ts +++ b/server/typescript/packages/integration-tests/test/api-contract-tph.test.ts @@ -15,7 +15,7 @@ import { describe, test } from "bun:test"; import { readFileSync } from "node:fs"; import { join } from "node:path"; import { API_CONTRACT_TPH_DIR, API_CONTRACT_TPH_SCENARIOS_DIR } from "../src/paths.ts"; -import { loadScenarios, assertResponse, type ApiScenario } from "../src/api-contract-scenario.ts"; +import { loadScenarios, assertResponse, type ApiScenario, parseBodyExact } from "../src/api-contract-scenario.ts"; import { startPostgres } from "../src/postgres-container.ts"; import { startTphServer, type TphServerHandle } from "../src/api-contract-tph-server.ts"; import { @@ -80,7 +80,7 @@ async function runScenario(scenario: ApiScenario, server: Lane): Promise { const bodyText = await res.text(); let body: unknown = null; if (bodyText.length > 0) { - try { body = JSON.parse(bodyText); } catch { body = bodyText; } + try { body = parseBodyExact(bodyText); } catch { body = bodyText; } } assertResponse(scenario.name, req, res.status, body); } diff --git a/server/typescript/packages/integration-tests/test/api-contract-write-through.test.ts b/server/typescript/packages/integration-tests/test/api-contract-write-through.test.ts index da76514d6..9b73b6627 100644 --- a/server/typescript/packages/integration-tests/test/api-contract-write-through.test.ts +++ b/server/typescript/packages/integration-tests/test/api-contract-write-through.test.ts @@ -18,7 +18,7 @@ import { API_CONTRACT_WRITE_THROUGH_DIR, API_CONTRACT_WRITE_THROUGH_SCENARIOS_DIR, } from "../src/paths.ts"; -import { loadScenarios, assertResponse, type ApiScenario } from "../src/api-contract-scenario.ts"; +import { loadScenarios, assertResponse, type ApiScenario, parseBodyExact } from "../src/api-contract-scenario.ts"; import { startPostgres } from "../src/postgres-container.ts"; import { startGeneratedWriteThroughServer, @@ -62,7 +62,7 @@ async function runScenario( const bodyText = await res.text(); let body: unknown = null; if (bodyText.length > 0) { - try { body = JSON.parse(bodyText); } catch { body = bodyText; } + try { body = parseBodyExact(bodyText); } catch { body = bodyText; } } assertResponse(scenario.name, req, res.status, body); } diff --git a/server/typescript/packages/integration-tests/test/api-contract.test.ts b/server/typescript/packages/integration-tests/test/api-contract.test.ts index eefecc1b4..10339468a 100644 --- a/server/typescript/packages/integration-tests/test/api-contract.test.ts +++ b/server/typescript/packages/integration-tests/test/api-contract.test.ts @@ -14,7 +14,7 @@ import { describe, test } from "bun:test"; import { readFileSync } from "node:fs"; import { join } from "node:path"; import { API_CONTRACT_DIR, API_CONTRACT_SCENARIOS_DIR } from "../src/paths.ts"; -import { loadScenarios, assertResponse, type ApiScenario } from "../src/api-contract-scenario.ts"; +import { loadScenarios, assertResponse, type ApiScenario, parseBodyExact } from "../src/api-contract-scenario.ts"; import { startServer, type ServerHandle, type AuthorRow } from "../src/api-contract-server.ts"; import { loadMetadataFile } from "../src/load-metadata.ts"; import { startPostgres } from "../src/postgres-container.ts"; @@ -55,7 +55,7 @@ async function executeRequests(scenario: ApiScenario, server: ServerHandle): Pro const bodyText = await res.text(); let body: unknown = null; if (bodyText.length > 0) { - try { body = JSON.parse(bodyText); } catch { body = bodyText; } + try { body = parseBodyExact(bodyText); } catch { body = bodyText; } } assertResponse(scenario.name, req, res.status, body); } diff --git a/server/typescript/packages/integration-tests/test/engine-wire.test.ts b/server/typescript/packages/integration-tests/test/engine-wire.test.ts index 5d56fc65e..9c4e1af81 100644 --- a/server/typescript/packages/integration-tests/test/engine-wire.test.ts +++ b/server/typescript/packages/integration-tests/test/engine-wire.test.ts @@ -9,18 +9,18 @@ import { toMysqlSeed } from "../src/query-scenario-mysql.ts"; import { splitStatements } from "../src/sql-script.ts"; describe("toWireRow", () => { - test("SQLite: a count and a currency are strings, a ratio is a canonical decimal string, min/max stay numbers", () => { + test("SQLite: a count and a currency are strings; a decimal, min and max pass through untouched", () => { expect(toWireRow("sqlite", "ProgramMinutes", { - program: 1, weeks: 4, avgMinutes: 60, longShare: 0.75, minMinutes: 30, programTitle: "7", + program: 1, weeks: 4, avgMinutes: "60", longShare: "0.75", minMinutes: 30, programTitle: "7", })).toEqual({ program: "1", weeks: "4", avgMinutes: "60", longShare: "0.75", minMinutes: 30, programTitle: "7", }); }); - test("MySQL: a DECIMAL keeps its value and loses its scale; a BIGINT string is untouched", () => { + test("a decimal is never rewritten here: the runtime owes the canonical string, the typed gate checks it", () => { expect(toWireRow("mysql", "ProgramMinutes", { weeks: "4", avgMinutes: "60.0000", longShare: "0.6667" })) - .toEqual({ weeks: "4", avgMinutes: "60", longShare: "0.6667" }); - expect(toWireRow("mysql", "ProgramMinutes", { longShare: "0.0000" })).toEqual({ longShare: "0" }); + .toEqual({ weeks: "4", avgMinutes: "60.0000", longShare: "0.6667" }); + expect(toWireRow("sqlite", "ProgramMinutes", { longShare: 0.75 })).toEqual({ longShare: 0.75 }); }); test("an instant becomes a UTC ...Z string: SQLite's stored text and MySQL's naive wall clock", () => { @@ -33,7 +33,9 @@ describe("toWireRow", () => { test("null, a date bucket and an unknown entity pass through", () => { expect(toWireRow("sqlite", "ProgramMinutes", { totalMinutes: null })).toEqual({ totalMinutes: null }); expect(toWireRow("sqlite", "AssetActivity", { asOfDateWeek: "2026-04-27" })).toEqual({ asOfDateWeek: "2026-04-27" }); - expect(toWireRow("mysql", "Program", { id: 1 })).toEqual({ id: 1 }); + expect(toWireRow("mysql", "NoSuchEntity", { id: 1 })).toEqual({ id: 1 }); + // A known entity's long is a string on the wire, whatever number the driver read. + expect(toWireRow("mysql", "Program", { id: 1 })).toEqual({ id: "1" }); }); test("a wrong value is still wrong: only the spelling is mapped", () => { diff --git a/server/typescript/packages/integration-tests/test/mysql-generated-app.test.ts b/server/typescript/packages/integration-tests/test/mysql-generated-app.test.ts index aa05217dc..b5817ecff 100644 --- a/server/typescript/packages/integration-tests/test/mysql-generated-app.test.ts +++ b/server/typescript/packages/integration-tests/test/mysql-generated-app.test.ts @@ -147,7 +147,7 @@ describe("generated TypeScript tier on MySQL", () => { expect(body.rank).toBe(10); expect(body.active).toBe(true); expect(body.tags).toEqual(["poetry", "essays"]); - expect(body.score).toBe("12.50"); + expect(body.score).toBe("12.5"); // canonical (ADR-0019 Amendment 1): no trailing fraction zero expect(body.externalId).toBe("0f8fad5b-d9cb-469f-a165-70867728950e"); expect(body.createdAt).toMatch(/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/); diff --git a/server/typescript/packages/integration-tests/test/query-d1.test.ts b/server/typescript/packages/integration-tests/test/query-d1.test.ts index 57675daff..7a138cc47 100644 --- a/server/typescript/packages/integration-tests/test/query-d1.test.ts +++ b/server/typescript/packages/integration-tests/test/query-d1.test.ts @@ -8,7 +8,7 @@ import { beforeAll, describe, test } from "bun:test"; import { CANONICAL_DIR, QUERIES_DIR } from "../src/paths.ts"; import { generateCanonicalSchemaSql } from "../src/canonical-schema.ts"; import { loadMetadataDir } from "../src/load-metadata.ts"; -import { loadReportQueries } from "../src/scenario.ts"; +import { loadQueriesFor } from "../src/scenario.ts"; import { runQueryScenarioD1, startLocalD1 } from "../src/query-scenario-d1.ts"; // The D1 schema has no committed artifact (see query-scenario-d1.ts), but it depends only on the @@ -19,7 +19,7 @@ beforeAll(async () => { }, 120_000); describe("persistence conformance — report query scenarios on Cloudflare D1 (local runtime)", () => { - for (const scenario of loadReportQueries(QUERIES_DIR)) { + for (const scenario of loadQueriesFor(QUERIES_DIR, "d1")) { test(scenario.name, async () => { const local = await startLocalD1(); try { diff --git a/server/typescript/packages/integration-tests/test/query-mysql.test.ts b/server/typescript/packages/integration-tests/test/query-mysql.test.ts index b5cdc0ea8..50ca7529f 100644 --- a/server/typescript/packages/integration-tests/test/query-mysql.test.ts +++ b/server/typescript/packages/integration-tests/test/query-mysql.test.ts @@ -5,7 +5,7 @@ import { afterAll, beforeAll, describe, test } from "bun:test"; import { CANONICAL_DIR, QUERIES_DIR } from "../src/paths.ts"; -import { loadReportQueries } from "../src/scenario.ts"; +import { loadQueriesFor } from "../src/scenario.ts"; import { runQueryScenarioMysql } from "../src/query-scenario-mysql.ts"; import { startMysql, type MysqlContainerHandle } from "../src/mysql-container.ts"; @@ -14,7 +14,7 @@ beforeAll(async () => { container = await startMysql(); }, 240_000); afterAll(() => { container?.stop(); }, 60_000); describe("persistence conformance — report query scenarios on MySQL", () => { - for (const scenario of loadReportQueries(QUERIES_DIR)) { + for (const scenario of loadQueriesFor(QUERIES_DIR, "mysql")) { test(scenario.name, async () => { await runQueryScenarioMysql(scenario, container.url, CANONICAL_DIR); }, { timeout: 120_000 }); diff --git a/server/typescript/packages/integration-tests/test/query-sqlite.test.ts b/server/typescript/packages/integration-tests/test/query-sqlite.test.ts index 912a5513f..8e89bf33b 100644 --- a/server/typescript/packages/integration-tests/test/query-sqlite.test.ts +++ b/server/typescript/packages/integration-tests/test/query-sqlite.test.ts @@ -7,11 +7,11 @@ import { describe, test } from "bun:test"; import { CANONICAL_DIR, QUERIES_DIR } from "../src/paths.ts"; -import { loadReportQueries } from "../src/scenario.ts"; +import { loadQueriesFor } from "../src/scenario.ts"; import { runQueryScenarioSqlite } from "../src/query-scenario-sqlite.ts"; describe("persistence conformance — report query scenarios on SQLite", () => { - for (const scenario of loadReportQueries(QUERIES_DIR)) { + for (const scenario of loadQueriesFor(QUERIES_DIR, "sqlite")) { test(scenario.name, async () => { await runQueryScenarioSqlite(scenario, CANONICAL_DIR); }, { timeout: 60_000 }); diff --git a/server/typescript/packages/integration-tests/test/query.test.ts b/server/typescript/packages/integration-tests/test/query.test.ts index 5cc616dda..3697b4310 100644 --- a/server/typescript/packages/integration-tests/test/query.test.ts +++ b/server/typescript/packages/integration-tests/test/query.test.ts @@ -2,12 +2,12 @@ import { describe, test } from "bun:test"; import { CANONICAL_DIR, QUERIES_DIR } from "../src/paths.ts"; -import { loadQueries } from "../src/scenario.ts"; +import { loadQueriesFor } from "../src/scenario.ts"; import { runQueryScenario } from "../src/query-scenario.ts"; import { startPostgres } from "../src/postgres-container.ts"; describe("persistence conformance — query scenarios", () => { - for (const scenario of loadQueries(QUERIES_DIR)) { + for (const scenario of loadQueriesFor(QUERIES_DIR, "postgres")) { test(scenario.name, async () => { const pg = await startPostgres(); try { diff --git a/server/typescript/packages/integration-tests/test/report-shapes-artifact.test.ts b/server/typescript/packages/integration-tests/test/report-shapes-artifact.test.ts index f74938a66..277345307 100644 --- a/server/typescript/packages/integration-tests/test/report-shapes-artifact.test.ts +++ b/server/typescript/packages/integration-tests/test/report-shapes-artifact.test.ts @@ -61,6 +61,8 @@ describe("canonical report-shapes artifact (report-shapes.json)", () => { ["fitness::ProgramSessions", "v_program_sessions"], ["fitness::NodesByParent", "v_nodes_by_parent"], ["fitness::AuthsByKind", "v_auths_by_kind"], + ["fitness::GadgetEventsByLabel", "v_gadget_events_by_label"], + ["fitness::EntryTotals", "v_entry_totals"], ]); }); @@ -97,6 +99,18 @@ describe("canonical report-shapes artifact (report-shapes.json)", () => { ["totalMinutesOrZero", "measure", "long", true], ["longShareOrZero", "measure", "decimal", true], ]); + // The decimal wire reports (ADR-0019 Amendment 1): every decimal measure is typed, so a runner + // knows to hold it to the canonical string. + expect(fields("fitness::EntryTotals")).toEqual([ + ["bucket", "dimension", "string", true], + ["entries", "measure", "long", true], + ["total", "measure", "decimal", false], + ["totalOrZero", "measure", "decimal", true], + ["mean", "measure", "decimal", false], + ["top", "measure", "decimal", false], + ["flaggedShare", "measure", "decimal", false], + ["perFlagged", "measure", "decimal", false], + ]); }); test("`view` is the source the lowering names and the runtime reads: primary, else first", async () => { diff --git a/server/typescript/packages/integration-tests/test/report-views-pg.test.ts b/server/typescript/packages/integration-tests/test/report-views-pg.test.ts index 80c7f2a76..5c8b1dcea 100644 --- a/server/typescript/packages/integration-tests/test/report-views-pg.test.ts +++ b/server/typescript/packages/integration-tests/test/report-views-pg.test.ts @@ -16,6 +16,7 @@ * what the conformance runner's wire normalisation does too. */ +import { splitSqlStatements } from "@metaobjectsdev/migrate-ts"; import { describe, test, expect, beforeAll, afterAll, beforeEach } from "bun:test"; import { buildExpectedSchema, diff, emit, introspectPostgres, postgresViewReplaceProbe, collectUnmanagedNames, @@ -63,7 +64,7 @@ beforeEach(async () => { }); async function applyRaw(text: string): Promise { - for (const stmt of text.split(/;\s*\n/).map((s) => s.trim()).filter(Boolean)) { + for (const stmt of splitSqlStatements(text)) { await sql.raw(stmt.endsWith(";") ? stmt : `${stmt};`).execute(k); } } diff --git a/server/typescript/packages/integration-tests/test/runtime-return-types.test.ts b/server/typescript/packages/integration-tests/test/runtime-return-types.test.ts index 84e27737d..2410ba99c 100644 --- a/server/typescript/packages/integration-tests/test/runtime-return-types.test.ts +++ b/server/typescript/packages/integration-tests/test/runtime-return-types.test.ts @@ -16,7 +16,7 @@ // // * Measurement.id (BIGINT) → string ("1") — node-postgres' int8 default; // JS has no safe 64-bit int, so bigint→string. -// * Measurement.preciseKg (NUMERIC) → string ("12.5000") — TS has no native exact +// * Measurement.preciseKg (NUMERIC) → string ("12.5", canonical) — TS has no native exact // decimal type; string preserves precision (ADR-0019). // * Asset.recordedAt (TIMESTAMPTZ) → Date — a native JS temporal, NOT a re-canonicalized // string. runtime-ts hands back the driver's Date; the @@ -98,11 +98,10 @@ describe("SP-D Unit 4 — runtime returns native types, not wire-strings", () => expect(measurement.id).toBe("1"); // preciseKg is NUMERIC → string (ADR-0019: TS has no native exact-decimal type; - // string preserves full precision). Crucially it is NOT pre-canonicalized: the - // raw scale-preserving DB text "12.5000" comes through, NOT the wire form "12.5" - // (which is produced only by the boundary normalizer). + // string preserves full precision). Amendment 1 (W1): the runtime reads a decimal in its + // canonical spelling, so the column's scale ("12.5000") is not carried to the caller. expect(typeof measurement.preciseKg).toBe("string"); - expect(measurement.preciseKg).toBe("12.5000"); + expect(measurement.preciseKg).toBe("12.5"); // --- Asset: native temporal (Date) + native jsonb (object) ---------------- const a = await om.findById("Asset", "11111111-1111-4111-8111-111111111111"); diff --git a/server/typescript/packages/integration-tests/test/view-lifecycle-pg.test.ts b/server/typescript/packages/integration-tests/test/view-lifecycle-pg.test.ts index b8e090eea..539b3000c 100644 --- a/server/typescript/packages/integration-tests/test/view-lifecycle-pg.test.ts +++ b/server/typescript/packages/integration-tests/test/view-lifecycle-pg.test.ts @@ -20,6 +20,7 @@ * see the round trip cannot see this class of bug. */ +import { splitSqlStatements } from "@metaobjectsdev/migrate-ts"; import { describe, test, expect, beforeAll, afterAll, beforeEach } from "bun:test"; import { buildExpectedSchema, diff, emit, introspectPostgres, postgresViewReplaceProbe, collectUnmanagedNames, @@ -193,7 +194,7 @@ async function createDownstreamDependent(): Promise { async function applyRaw(text: string): Promise { // Statement-split must not shred the CASCADE warning banner or a view body — split on // ";\n" boundaries the emitter actually produces, then drop empties. - for (const stmt of text.split(/;\s*\n/).map((s) => s.trim()).filter(Boolean)) { + for (const stmt of splitSqlStatements(text)) { await sql.raw(stmt.endsWith(";") ? stmt : `${stmt};`).execute(k); } } diff --git a/server/typescript/packages/integration-tests/test/view-recreate-drift-pg.test.ts b/server/typescript/packages/integration-tests/test/view-recreate-drift-pg.test.ts index 35a66a6ec..cd8714610 100644 --- a/server/typescript/packages/integration-tests/test/view-recreate-drift-pg.test.ts +++ b/server/typescript/packages/integration-tests/test/view-recreate-drift-pg.test.ts @@ -6,6 +6,7 @@ * the fingerprint in the view's COMMENT, not from the text. The pair stays in the migration * SQL; it says the view is unchanged, a drift report leaves it out, and it converges. */ +import { splitSqlStatements } from "@metaobjectsdev/migrate-ts"; import { describe, test, expect, beforeAll, afterAll, beforeEach } from "bun:test"; import { buildExpectedSchema, collectUnmanagedNames, computeDriftFromActual, diff, emit, introspectPostgres, postgresViewReplaceProbe, @@ -68,7 +69,7 @@ const expectedForRun = (): SchemaSnapshot => { }; async function applyRaw(text: string): Promise { - for (const stmt of text.split(/;\s*\n/).map((s) => s.trim()).filter(Boolean)) { + for (const stmt of splitSqlStatements(text)) { await sql.raw(stmt.endsWith(";") ? stmt : `${stmt};`).execute(k); } } diff --git a/server/typescript/packages/integration-tests/test/view-recreate-grants-pg.test.ts b/server/typescript/packages/integration-tests/test/view-recreate-grants-pg.test.ts new file mode 100644 index 000000000..d57f59d70 --- /dev/null +++ b/server/typescript/packages/integration-tests/test/view-recreate-grants-pg.test.ts @@ -0,0 +1,146 @@ +/** + * f. Postgres: a report view that `meta migrate` drops and creates again keeps its GRANTs. + * + * `DROP VIEW` discards the privileges granted on the view. The emitted migration therefore + * captures them from `relacl` before the drop and grants them again after the create, to the + * same grantees and WITH GRANT OPTION where it was held, using nothing a non-superuser lacks. + * Two roads lead to a drop and create: a view over a column whose type changes (drop-view + + * create-view, which the planner picks when the live probe shows a replace is illegal or when + * it plans offline with no probe). A replace the probe proved legal is `CREATE OR REPLACE`, which + * Postgres itself keeps the grants through; the emitter's unproven replace-view branch is covered + * in migrate-ts/test/emit-postgres-view-grants.test.ts. + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from "bun:test"; +import { + buildExpectedSchema, collectUnmanagedNames, diff, emit, introspectPostgres, postgresViewReplaceProbe, + splitSqlStatements, viewFingerprint, type SchemaSnapshot, +} from "@metaobjectsdev/migrate-ts"; +import { buildProjectionViews } from "@metaobjectsdev/codegen-ts"; +import type { MetaRoot } from "@metaobjectsdev/metadata"; +import { Kysely, PostgresDialect, sql } from "kysely"; +import { Pool } from "pg"; +import { startPostgres, type RunningPg } from "../src/postgres-container.ts"; +import { loadMetadataDir } from "../src/load-metadata.ts"; +import { CANONICAL_DIR } from "../src/paths.ts"; + +const VIEW = "v_program_minutes"; +let pg: RunningPg; +let k: Kysely>; +let canonical: MetaRoot; + +beforeAll(async () => { + pg = await startPostgres(); + k = new Kysely>({ + dialect: new PostgresDialect({ pool: new Pool({ connectionString: pg.connectionUri, max: 1 }) }), + }); + canonical = await loadMetadataDir(CANONICAL_DIR); + // The project setting `"naming": { "columns": "literal" }` (ADR-0058), set where the project-aware load sets it. + (canonical as unknown as { physicalNaming: unknown }).physicalNaming = { columns: "literal", tables: "snake_case" }; +}, 120_000); + +afterAll(async () => { + await k.destroy(); + await pg.stop(); +}, 60_000); + +beforeEach(async () => { + await sql.raw("DROP SCHEMA public CASCADE").execute(k); + await sql.raw("CREATE SCHEMA public").execute(k); + await sql.raw("GRANT USAGE ON SCHEMA public TO PUBLIC").execute(k); + for (const role of ["mo_reader", "mo_admin"]) { + await sql.raw(`DROP ROLE IF EXISTS ${role}`).execute(k); + await sql.raw(`CREATE ROLE ${role}`).execute(k); + } + widened = false; + renamed = false; +}); + +let widened = false; +let renamed = false; +const expectedSchema = (): SchemaSnapshot => { + const schema = buildExpectedSchema(canonical, { views: buildProjectionViews(canonical, { dialect: "postgres" }) }); + if (widened) { + schema.tables.find((t) => t.name === "weeks")!.columns.find((c) => c.name === "durationMinutes")!.sqlType = + { kind: "integer", bits: 64 }; + } + if (renamed) { + // A renamed output column: `CREATE OR REPLACE VIEW` refuses it, so it is written as DROP + CREATE. + const v = schema.views.find((x) => x.name === VIEW)!; + v.sql = v.sql!.replace(' AS "totalMinutes"', ' AS "totalMinutesRenamed"'); + v.fingerprint = viewFingerprint(v.sql!); + } + return schema; +}; + +async function apply(text: string): Promise { + await sql.raw("BEGIN").execute(k); + try { + for (const stmt of splitSqlStatements(text)) await sql.raw(stmt).execute(k); + await sql.raw("COMMIT").execute(k); + } catch (e) { + await sql.raw("ROLLBACK").execute(k); + throw e; + } +} + +async function migrate(probe = true): Promise<{ up: string; notices: readonly string[]; kinds: string[] }> { + const result = await diff({ + expected: expectedSchema(), actual: await introspectPostgres(k), dialect: "postgres", + ...(probe ? { viewReplaceProbe: postgresViewReplaceProbe(k) } : {}), allow: { typeChange: true }, + unmanagedNames: collectUnmanagedNames(canonical), + }); + expect(result.blocked).toEqual([]); + const emitted = emit(result.changes, { dialect: "postgres" }); + if (emitted.up.trim().length > 0) await apply(emitted.up); + return { up: emitted.up, notices: emitted.notices ?? [], kinds: result.changes.map((c) => c.kind) }; +} + +async function grantsOn(view: string): Promise { + const rows = await sql<{ g: string }>` + SELECT (CASE WHEN a.grantee = 0 THEN 'PUBLIC' ELSE pg_get_userbyid(a.grantee) END) || ':' || a.privilege_type || + (CASE WHEN a.is_grantable THEN ':grant' ELSE '' END) AS g + FROM pg_class c, aclexplode(c.relacl) a + WHERE c.oid = to_regclass(${view}) AND a.grantee <> c.relowner ORDER BY 1`.execute(k); + return rows.rows.map((r) => r.g); +} + +async function grantThenCheck(): Promise { + await sql.raw(`GRANT SELECT ON ${VIEW} TO mo_reader`).execute(k); + await sql.raw(`GRANT SELECT ON ${VIEW} TO mo_admin WITH GRANT OPTION`).execute(k); + await sql.raw(`GRANT SELECT ON ${VIEW} TO PUBLIC`).execute(k); + expect(await grantsOn(VIEW)).toEqual(["PUBLIC:SELECT", "mo_admin:SELECT:grant", "mo_reader:SELECT"]); +} + +async function readerCanSelect(): Promise { + await sql.raw("SET ROLE mo_reader").execute(k); + try { + await sql.raw(`SELECT * FROM ${VIEW}`).execute(k); + } finally { + await sql.raw("RESET ROLE").execute(k); + } +} + +describe("a recreated report view keeps its grants (postgres)", () => { + test("drop-view + create-view around a column type change", async () => { + await migrate(); + await grantThenCheck(); + widened = true; + const { up, notices, kinds } = await migrate(); + expect(kinds).toContain("drop-view"); + expect(up).toContain(`CREATE VIEW "${VIEW}"`); + expect(notices).toContain(`view public.${VIEW}: dropped and recreated; its grants are carried over.`); + expect(await grantsOn(VIEW)).toEqual(["PUBLIC:SELECT", "mo_admin:SELECT:grant", "mo_reader:SELECT"]); + await readerCanSelect(); + }, 60_000); + + test("a report whose body changes, planned offline with no probe (drop + create)", async () => { + await migrate(); + await grantThenCheck(); + renamed = true; + const { up, kinds } = await migrate(false); + expect(kinds).toContain("drop-view"); + expect(up).toContain(`DROP VIEW IF EXISTS "${VIEW}"`); + expect(await grantsOn(VIEW)).toEqual(["PUBLIC:SELECT", "mo_admin:SELECT:grant", "mo_reader:SELECT"]); + await readerCanSelect(); + }, 60_000); +}); diff --git a/server/typescript/packages/integration-tests/test/view-replace-probe-pg.test.ts b/server/typescript/packages/integration-tests/test/view-replace-probe-pg.test.ts index 0f501ef5c..5f6fad472 100644 --- a/server/typescript/packages/integration-tests/test/view-replace-probe-pg.test.ts +++ b/server/typescript/packages/integration-tests/test/view-replace-probe-pg.test.ts @@ -16,6 +16,7 @@ * cannot tell apart: json and jsonb, smallint and integer, text and character varying. */ +import { splitSqlStatements } from "@metaobjectsdev/migrate-ts"; import { describe, test, expect, beforeAll, afterAll, beforeEach } from "bun:test"; import { randomUUID } from "node:crypto"; import { @@ -237,7 +238,7 @@ async function load(host: unknown, programFields: unknown[] = [], others: unknow } async function applyRaw(text: string, db: Db = k): Promise { - for (const stmt of text.split(/;\s*\n/).map((s) => s.trim()).filter(Boolean)) { + for (const stmt of splitSqlStatements(text)) { await sql.raw(stmt.endsWith(";") ? stmt : `${stmt};`).execute(db); } } diff --git a/server/typescript/packages/metadata/package.json b/server/typescript/packages/metadata/package.json index 9a8b0d116..f8f0642ee 100644 --- a/server/typescript/packages/metadata/package.json +++ b/server/typescript/packages/metadata/package.json @@ -26,6 +26,11 @@ "types": "./dist/constants.d.ts", "default": "./dist/constants.js" }, + "./release-docs": { + "bun": "./src/release-docs.ts", + "types": "./dist/release-docs.d.ts", + "default": "./dist/release-docs.js" + }, "./vocabulary-rewrite-yaml": { "bun": "./src/vocabulary-rewrite-yaml.ts", "types": "./dist/vocabulary-rewrite-yaml.d.ts", diff --git a/server/typescript/packages/metadata/src/core/attr/meta-attr.ts b/server/typescript/packages/metadata/src/core/attr/meta-attr.ts index 8952e456e..530dddac6 100644 --- a/server/typescript/packages/metadata/src/core/attr/meta-attr.ts +++ b/server/typescript/packages/metadata/src/core/attr/meta-attr.ts @@ -111,7 +111,14 @@ export class MetaAttr extends MetaData implements DataTypeAware { if (dt === DATA_TYPE_STRING) { return typeof value === "string" ? [] : [this._typeError("string", value)]; } - if (dt === DATA_TYPE_INT || dt === DATA_TYPE_LONG || dt === DATA_TYPE_DOUBLE) { + if (dt === DATA_TYPE_INT || dt === DATA_TYPE_LONG) { + if (typeof value !== "number") return [this._typeError("number", value)]; + // Rule A: an integral number (4, 4.0, 1e2) is an int; a fraction is not. + return Number.isInteger(value) + ? [] + : [{ message: `attribute '@${this.name}' must be of type '${this.subType}' but got the fraction ${value}` }]; + } + if (dt === DATA_TYPE_DOUBLE) { return typeof value === "number" ? [] : [this._typeError("number", value)]; } if (dt === DATA_TYPE_BOOLEAN) { diff --git a/server/typescript/packages/metadata/src/core/reporting/report-describe.ts b/server/typescript/packages/metadata/src/core/reporting/report-describe.ts index 049d5789a..d6db43947 100644 --- a/server/typescript/packages/metadata/src/core/reporting/report-describe.ts +++ b/server/typescript/packages/metadata/src/core/reporting/report-describe.ts @@ -15,7 +15,6 @@ import type { MetaData } from "../../shared/meta-data.js"; import type { MetaRoot } from "../../shared/meta-root.js"; import { TYPE_DIMENSION, TYPE_MEASURE, TYPE_SEGMENT } from "../../shared/base-types.js"; -import { SOURCE_KIND_VIEW } from "../../persistence/source/source-constants.js"; import { AGG_ALL } from "../../persistence/origin/origin-constants.js"; import type { MetaObject } from "../object/meta-object.js"; import { OBJECT_REPORT_ATTR_FILTER, OBJECT_REPORT_ATTR_SEGMENT, OBJECT_REPORT_ATTR_SPINE } from "../object/object-constants.js"; @@ -177,17 +176,12 @@ export function reportFieldTypeName(field: ReportField): string { } /** - * Why a report is not served, or undefined when it is (Plan 3 Table A: served means not - * abstract, with a read source of `@kind: view`). + * Why a report is not served, or undefined when it is. The loader refuses an abstract + * report (R11) and a read source whose `@kind` is not `view` (R10), so the one way a loaded + * report goes unserved is declaring no read source at all (Plan 3 Table A). */ export function reportNotServedReason(report: MetaObject): string | undefined { - if (report.isAbstract === true) return "Not served: the report is abstract"; - const source = reportReadSource(report); - if (source === undefined) return "Not served: declares no view source"; - if (source.effectiveKind !== SOURCE_KIND_VIEW) { - return `Not served: its source is a ${source.effectiveKind}, not a view`; - } - return undefined; + return reportReadSource(report) === undefined ? "Not served: declares no view source" : undefined; } const DIMENSION_ATTRS = [REPORTING_ATTR_OF, REPORTING_ATTR_VIA] as const; diff --git a/server/typescript/packages/metadata/src/core/reporting/report-read-model.ts b/server/typescript/packages/metadata/src/core/reporting/report-read-model.ts index 7649e9b2c..0e079bc83 100644 --- a/server/typescript/packages/metadata/src/core/reporting/report-read-model.ts +++ b/server/typescript/packages/metadata/src/core/reporting/report-read-model.ts @@ -36,12 +36,12 @@ import { FIELD_ATTR_FILTERABLE, FIELD_ATTR_INT_VALUE_MAP, FIELD_ATTR_MAX_LENGTH, - FIELD_ATTR_OBJECT_REF, FIELD_ATTR_PRECISION, FIELD_ATTR_REQUIRED, FIELD_ATTR_SCALE, FIELD_ATTR_STORAGE, FIELD_ATTR_VALUES, + FIELD_SUBTYPE_MAP, } from "../field/field-constants.js"; import { opsForField } from "../query/query-constants.js"; import { reportShape, type ReportField } from "./report-shape.js"; @@ -61,10 +61,14 @@ const CARRIED_ATTRS = [ FIELD_ATTR_PRECISION, FIELD_ATTR_SCALE, FIELD_ATTR_LOCAL_TIME, - FIELD_ATTR_OBJECT_REF, FIELD_ATTR_STORAGE, ] as const; +/** True for a derived field whose source field is an array or a `field.map`. */ +function isArrayOrMap(field: MetaField): boolean { + return field.resolvedIsArray() || field.subType === FIELD_SUBTYPE_MAP; +} + function derivedField(f: ReportField): MetaField { const field = new MetaField(new TypeId(TYPE_FIELD, f.subType), f.name); // From the derived shape, never from the type source: a `min` of a required column @@ -89,9 +93,13 @@ function derivedField(f: ReportField): MetaField { if (src.resolvedIsArray()) field.setIsArray(true); } // Table C (Plan 3): a report author has no node to put @filterable on, so every - // derived field that has a filter band is filterable. Set on this detached model - // only; no vocabulary is added and the declared tree is not touched. - if (opsForField(field).length > 0) field.setAttr(FIELD_ATTR_FILTERABLE, true); + // derived field that has a filter band is filterable, and (the sort allowlist follows + // `@filterable`) sortable. Set on this detached model only; no vocabulary is added and the + // declared tree is not touched. EXCEPT an array or a map: an array column cannot be compared + // with a scalar (`text[] = text` is a Postgres error), and the entity rule already refuses + // `@filterable` on an array field, so a derived field whose source is either is neither + // filterable nor sortable, in every port. + if (!isArrayOrMap(field) && opsForField(field).length > 0) field.setAttr(FIELD_ATTR_FILTERABLE, true); return field; } diff --git a/server/typescript/packages/metadata/src/core/requirement/resolve-claim.ts b/server/typescript/packages/metadata/src/core/requirement/resolve-claim.ts index 1a26b1ff7..2515ff891 100644 --- a/server/typescript/packages/metadata/src/core/requirement/resolve-claim.ts +++ b/server/typescript/packages/metadata/src/core/requirement/resolve-claim.ts @@ -96,3 +96,23 @@ export function resolveClaim( if (owner === undefined || segs.length === 1) return owner; return resolveMember(owner, segs.slice(1)); } + +/** + * The package a requirement node resolves in (ADR-0059 D3/D10): its own declared + * `package`, else the nearest enclosing node that declares one, else the file default + * package of the top-level node it sits under, else "". One definition for the gate, the summary and + * codegen's requirement-test fan-out, so nesting and overlay contribution cannot + * make two call sites disagree. + */ +export function effectiveRequirementPackage(node: MetaData): string { + let cur: MetaData = node; + for (;;) { + if (cur.package !== undefined && cur.package !== "") return cur.package; + const parent: MetaData | undefined = cur.parent; + // The tree root is not an "enclosing node": stop at the top-level node, whose file + // default package is the answer (a child contributed by another document inherits + // the merge target's, matching the overlay-declared package case). + if (parent === undefined || parent.parent === undefined) return cur.fileDefaultPackage ?? ""; + cur = parent; + } +} diff --git a/server/typescript/packages/metadata/src/docs-ref.ts b/server/typescript/packages/metadata/src/docs-ref.ts new file mode 100644 index 000000000..e0c37533a --- /dev/null +++ b/server/typescript/packages/metadata/src/docs-ref.ts @@ -0,0 +1,41 @@ +// docs-ref.ts — where a guide URL points, derived from the running version. +// +// A guide URL that says `blob/main` moves after the release is cut, and no installed +// package ships the guides, so the CLI and the agent context name the TAG of the release +// they belong to: `blob/v1.1.0/...`. A prerelease version has no tag, so it keeps `main`. +// The repository URL and the rule live here once; nothing else spells either. This file is +// pure (browser-safe, reachable from the root entry); reading the running version is the +// Node-side `release-docs.ts`. + +/** The public repository the guides live in. */ +export const DOCS_REPO_URL = "https://github.com/metaobjectsdev/metaobjects"; + +/** The branch a prerelease or an unresolvable version links to. */ +export const DOCS_DEFAULT_REF = "main"; + +const RELEASE_VERSION = /^\d+\.\d+\.\d+$/; + +/** + * The git ref a guide URL pins to: `v` for a release (`1.1.0` → `v1.1.0`), + * `main` for anything with no tag — a prerelease (`1.1.0-rc.7`), the `0.0.0` sentinel a + * port stamps when it cannot resolve its own version, or text that is not a version. + */ +export function docsRefForVersion(version: string): string { + const v = version.trim(); + return RELEASE_VERSION.test(v) && v !== "0.0.0" ? `v${v}` : DOCS_DEFAULT_REF; +} + +/** `https://github.com//blob//`. */ +export function docsBlobUrl(repoRelativePath: string, ref: string = DOCS_DEFAULT_REF): string { + return `${DOCS_REPO_URL}/blob/${ref}/${repoRelativePath.replace(/^\/+/, "")}`; +} + +/** + * Rewrite every `…/blob/main/` link into this repository, inside `text`, to `ref`. + * Authors of shipped prose write `main` as the placeholder; the pin is applied once, at + * render time, so no template carries a version. + */ +export function pinDocsRefs(text: string, ref: string): string { + if (ref === DOCS_DEFAULT_REF) return text; + return text.split(`${DOCS_REPO_URL}/blob/${DOCS_DEFAULT_REF}/`).join(`${DOCS_REPO_URL}/blob/${ref}/`); +} diff --git a/server/typescript/packages/metadata/src/index.ts b/server/typescript/packages/metadata/src/index.ts index f0200021d..25fdc5357 100644 --- a/server/typescript/packages/metadata/src/index.ts +++ b/server/typescript/packages/metadata/src/index.ts @@ -87,6 +87,7 @@ export { // Shared `@implementedBy` resolution — one resolver for the CLI's requirement // checks and codegen's requirement-test fan-out (FR-038). export { + effectiveRequirementPackage, resolveClaim, resolveClaimTarget, resolveMember, @@ -308,6 +309,7 @@ export { splitChildTail, RefOverDropError, didYouMeanHint, + didYouMeanSuperHint, REF_BEARING_ATTR_NAMES, } from "./naming-refs.js"; export { resolveReferences } from "./loader/resolve-references.js"; @@ -358,13 +360,18 @@ export type { Scope, CompiledScope } from "./scope.js"; export { validateAttrSchema } from "./attr-schema-validate.js"; export type { AttrSchemaValidationResult } from "./attr-schema-validate.js"; +// Guide URLs pinned to the running release's tag (prerelease → main). +export { + DOCS_REPO_URL, DOCS_DEFAULT_REF, docsRefForVersion, docsBlobUrl, pinDocsRefs, +} from "./docs-ref.js"; + // Naming — hoisted from runtime-ts in v0.2.3 so multiple consumers (runtime-ts, migrate-ts, codegen-ts) // share identical name resolution. See spec §4.1. export { toSnakeCase, toKebabCase, pluralize, pluralizeTableNameLegacy, applyColumnNamingStrategy, DEFAULT_COLUMN_NAMING_STRATEGY, COLUMN_NAMING_STRATEGIES, TABLE_NAMING_RULES, DEFAULT_PHYSICAL_NAMING, physicalNamingOf, applyTableNamingRule, - PHYSICAL_NAMING_MIGRATION_GUIDE, physicalNamingRemovedMessage, + PHYSICAL_NAMING_MIGRATION_GUIDE, PHYSICAL_NAMING_MIGRATION_GUIDE_PATH, physicalNamingRemovedMessage, resolveTableName, resolveColumnName, resolveTableSchema, resolveIndexName, primaryRdbSource, sourceAddressKey, buildNameMap, diff --git a/server/typescript/packages/metadata/src/loader/meta-data-loader.ts b/server/typescript/packages/metadata/src/loader/meta-data-loader.ts index b02b53e26..9285fab03 100644 --- a/server/typescript/packages/metadata/src/loader/meta-data-loader.ts +++ b/server/typescript/packages/metadata/src/loader/meta-data-loader.ts @@ -45,6 +45,7 @@ import type { MetaDataFormat, MetaDataSource } from "./meta-data-source.js"; import { InMemoryStringSource } from "./meta-data-source.js"; import type { ParseOptions, ParseResult, PendingOverlay } from "../parser-core.js"; import { expandPackageForPath, applyPendingOverlays } from "../parser-core.js"; +import { didYouMeanSuperHint } from "../naming-refs.js"; // Local mirror of DirectorySource's options shape. Deliberately inlined here // (instead of `import type`'d from ./sources/directory-source.js) so the @@ -606,7 +607,7 @@ export class MetaDataLoader { } errors.push( new ParseError( - `the SuperClass '${failure.ref}' does not exist (referenced by ${failure.nodeFqn})`, + `the SuperClass '${failure.ref}' does not exist (referenced by ${failure.nodeFqn})${didYouMeanSuperHint(root, failure.ref)}`, { code: "ERR_UNRESOLVED_SUPER", source: resolvedSource(failure.source, failure.nodeFqn, failure.ref), diff --git a/server/typescript/packages/metadata/src/loader/reporting-validation.ts b/server/typescript/packages/metadata/src/loader/reporting-validation.ts index ad043f3c1..f202e5127 100644 --- a/server/typescript/packages/metadata/src/loader/reporting-validation.ts +++ b/server/typescript/packages/metadata/src/loader/reporting-validation.ts @@ -56,6 +56,7 @@ import { FIELD_SUBTYPE_MAP, FIELD_SUBTYPE_DATE, FIELD_SUBTYPE_TIMESTAMP, + FIELD_ATTR_OBJECT_REF, } from "../core/field/field-constants.js"; import { IDENTITY_SUBTYPE_REFERENCE, @@ -76,7 +77,8 @@ import { opsForField, } from "../core/query/query-constants.js"; import { ATTR_SUBTYPE_FILTER } from "../core/attr/attr-constants.js"; -import { isMetaObject, isMetaRoot, isWritableSource } from "../shared/node-guards.js"; +import { isMetaObject, isMetaRoot, isReadOnlySource, isWritableSource } from "../shared/node-guards.js"; +import { SOURCE_KIND_VIEW } from "../persistence/source/source-constants.js"; import { AGG_ALL, AGG_ANY } from "../persistence/origin/origin-constants.js"; import { FILTER_RELATIVE_NOW, @@ -137,7 +139,6 @@ const ERR_INVALID_MEASURE: ErrorCode = "ERR_INVALID_MEASURE"; const ERR_INVALID_REPORT: ErrorCode = "ERR_INVALID_REPORT"; const ERR_REPORT_FOREIGN_MEASURE: ErrorCode = "ERR_REPORT_FOREIGN_MEASURE"; const ERR_BAD_ATTR_FILTER: ErrorCode = "ERR_BAD_ATTR_FILTER"; -const ERR_BAD_ATTR_VALUE: ErrorCode = "ERR_BAD_ATTR_VALUE"; // --------------------------------------------------------------------------- // Entry point @@ -231,6 +232,11 @@ function childOfType(obj: MetaData, type: string, name: string): MetaData | unde return obj.children().find((c) => c.type === type && c.name === name); } +/** A field's `@objectRef`, read with the resolving accessor (ADR-0039). */ +function objectRefOf(field: MetaData): unknown { + return field.attr(FIELD_ATTR_OBJECT_REF); +} + function fieldOf(obj: MetaData, name: string): MetaData | undefined { return childOfType(obj, TYPE_FIELD, name); } @@ -392,9 +398,31 @@ function checkDimension(ctx: MemberCtx, dim: MetaDimension): void { return; } const fieldName = parts.path[0]!; - const field = fieldOf(ofEntity, fieldName); + // With @via the field is looked up on the entity @of NAMES (resolving: its own plus + // inherited), not on the @via terminal: a base the terminal extends must itself carry the field, or the + // dimension names a column the base table lacks and `meta migrate` fails on it. Without @via + // the host IS the checked entity: a dimension inherited from a base is checked against the + // inheritor's own (possibly overriding) field. + const lookup = via === undefined ? ofEntity : named!; + const field = fieldOf(lookup, fieldName); if (field === undefined) { - err(`@of '${of}' names no field '${fieldName}' on '${shown(ctx, ofEntity)}'.`); + const declaredOnTerminal = lookup !== ofEntity && fieldOf(ofEntity, fieldName) !== undefined; + err( + declaredOnTerminal + ? `@of '${of}' names no field '${fieldName}' on '${shown(ctx, lookup)}': only '${shown(ctx, ofEntity)}', ` + + `the entity @via reaches, declares it. Name it '${ofEntity.name}.${fieldName}'.` + : `@of '${of}' names no field '${fieldName}' on '${shown(ctx, lookup)}'.`, + ); + return; + } + + // D5 — a dimension groups by a scalar. A field.object (or a field carrying @objectRef) is a JSON + // document: no port serves a report over one, and a document is not a value to group rows by. + if (field.subType === FIELD_SUBTYPE_OBJECT || objectRefOf(field) !== undefined) { + err( + `@of '${of}' is a field.${field.subType}${objectRefOf(field) !== undefined ? " with an @objectRef" : ""}, ` + + `which a report cannot group by. Group by a scalar field of it.`, + ); return; } @@ -536,21 +564,12 @@ function checkMeasure(ctx: MemberCtx, measure: MetaMeasure): void { const err = (message: string): void => ctx.sink.push(measure, ERR_INVALID_MEASURE, ctx.label, `: ${message}`, ctx.suffix); - // The type rule — a measure's @default is a whole number. Checked here, not by - // tightening attr.int, whose other users (validator @min/@max) take a fraction. - // A non-number is the attribute type check's error and is not repeated. It runs - // first; when it fires, M7/M8 are skipped (one mistake, one error). + // A measure's @default is a whole number: attr.int refuses a fraction (Rule A) and a + // non-number, so that is the one error. M7-M9 run only when @default is an integral + // number (Rule B: one mistake, one error). const declaredDefault = measure.attr(REPORTING_ATTR_DEFAULT); - const fractionalDefault = typeof declaredDefault === "number" && !Number.isInteger(declaredDefault); - if (fractionalDefault) { - ctx.sink.push( - measure, - ERR_BAD_ATTR_VALUE, - ctx.label, - `: @default '${declaredDefault}' is not an integer. A measure's @default is a whole number (for example 0).`, - ctx.suffix, - ); - } + const invalidDefault = + declaredDefault !== undefined && !(typeof declaredDefault === "number" && Number.isInteger(declaredDefault)); if (measure.isRatio()) { checkRatioOperands(ctx, measure, err); @@ -563,9 +582,8 @@ function checkMeasure(ctx: MemberCtx, measure: MetaMeasure): void { const ofField = checked?.field; // M7 / M8 — where a @default can apply. Only when M0–M4 passed and the @default is - // not a fraction: one mistake, one error. The presence test reads the raw attribute, - // so a non-number value on a count still reports M7 beside the attribute type error. - if (clean && !fractionalDefault && declaredDefault !== undefined) { + // an integral number: one mistake, one error (the attribute type check reports a bad value). + if (clean && !invalidDefault && declaredDefault !== undefined) { const agg = measure.agg(); if (agg === AGG_COUNT) { err( @@ -662,6 +680,15 @@ function checkAggregateColumns( err(`@of '${item}' names no field '${parts.path[0]!}' on '${ctx.declaring.resolutionKey()}'.`); return undefined; } + // M10 — a measure aggregates scalars. A field.object (or a field carrying @objectRef) is a JSON + // document, which no port aggregates in a report. + if (field.subType === FIELD_SUBTYPE_OBJECT || objectRefOf(field) !== undefined) { + err( + `@of '${item}' is a field.${field.subType}${objectRefOf(field) !== undefined ? " with an @objectRef" : ""}, ` + + `which a report cannot aggregate. Aggregate a scalar field of it.`, + ); + return undefined; + } fields.push(field); } @@ -847,6 +874,32 @@ function checkReport(root: MetaData, report: MetaData, sink: ErrorSink): void { } } + // R10 — a report is read through a view. The other read-only kinds are not read by a plain + // SELECT (a stored procedure and a table function are called, not selected from), and no port + // lowers or serves them. A view or materialized view the adopter creates is declared + // `@kind: view` with `@unmanaged: true`; a SELECT reads either one. + // ADR-0039: resolving, as R5 reads it. + for (const source of report.children().filter(isReadOnlySource)) { + if (source.effectiveKind !== SOURCE_KIND_VIEW) { + err( + `: ${childLabel(source)} has @kind '${source.effectiveKind}'; a report is read through a view, so its ` + + `source must be @kind: view. For a view or materialized view you create yourself, declare ` + + `@kind: view with @unmanaged: true.`, + source, + ); + } + } + + // R11 — a report cannot be abstract: its fields are derived from its own @dimensions and + // @measures, no port reads or serves an abstract report, and report inheritance is not part of + // the vocabulary. + if (report.isAbstract) { + err( + `: is abstract; a report cannot be abstract, because its fields are derived from its own @dimensions ` + + `and @measures and no port reads or serves an abstract report. Remove abstract: true.`, + ); + } + // R1 — @from resolves to an object.entity. Without it, R2/R3/R6/R7, R8/R9 and // the @filter have nothing to resolve against, so they are skipped. const fromRef = reportFrom(report); diff --git a/server/typescript/packages/metadata/src/naming-refs.ts b/server/typescript/packages/metadata/src/naming-refs.ts index e447325e7..94b576d4c 100644 --- a/server/typescript/packages/metadata/src/naming-refs.ts +++ b/server/typescript/packages/metadata/src/naming-refs.ts @@ -310,6 +310,27 @@ export function referrerPackageFor(node: MetaData, attrName: string): string { * suggest). Appended to the per-attr unresolved-ref error message. */ export function didYouMeanHint(root: MetaData, ref: string): string { + const found = sameNamedObjects(root, ref); + if (found === undefined) return ""; + return ` An object named "${found.shortName}" exists in: ${found.candidates.join(", ")}. Qualify it with its package (FQN).`; +} + +/** + * The did-you-mean suffix for an UNRESOLVED `extends` ref: the same candidate list + * as {@link didYouMeanHint}, plus the exact spelling to write. A bare `extends` + * resolves in the referrer's own package (ADR-0042/0059), so a base declared in + * another package must be named by its FQN. Returns "" when nothing is named alike. + */ +export function didYouMeanSuperHint(root: MetaData, ref: string): string { + const found = sameNamedObjects(root, ref); + if (found === undefined) return ""; + return ` An object named "${found.shortName}" exists in: ${found.candidates.join(", ")}. Qualify it with its package (FQN): write extends: "${found.candidates[0]}".`; +} + +function sameNamedObjects( + root: MetaData, + ref: string, +): { shortName: string; candidates: string[] } | undefined { const { owner } = splitChildTail(ref); const sep = owner.lastIndexOf(PACKAGE_SEPARATOR); const shortName = sep === -1 ? owner : owner.slice(sep + PACKAGE_SEPARATOR.length); @@ -317,8 +338,7 @@ export function didYouMeanHint(root: MetaData, ref: string): string { .children() .filter((c) => c.type === TYPE_OBJECT && c.name === shortName) .map((c) => c.resolutionKey()); - if (candidates.length === 0) return ""; - return ` An object named "${shortName}" exists in: ${candidates.join(", ")}. Qualify it with its package (FQN).`; + return candidates.length === 0 ? undefined : { shortName, candidates }; } /** diff --git a/server/typescript/packages/metadata/src/naming.ts b/server/typescript/packages/metadata/src/naming.ts index 8bf038904..794335465 100644 --- a/server/typescript/packages/metadata/src/naming.ts +++ b/server/typescript/packages/metadata/src/naming.ts @@ -1,3 +1,4 @@ +import { docsBlobUrl } from "./docs-ref.js"; import type { MetaData } from "./shared/meta-data.js"; import { TYPE_FIELD } from "./shared/base-types.js"; import { PACKAGE_SEPARATOR } from "./shared/structural.js"; @@ -101,19 +102,25 @@ export function physicalNamingOf(node: MetaData): PhysicalNaming { } /** The migration guide every refusal of a former naming source names (ADR-0058). */ -export const PHYSICAL_NAMING_MIGRATION_GUIDE = - "https://github.com/metaobjectsdev/metaobjects/blob/main/docs/features/migrations/physical-naming.md"; +export const PHYSICAL_NAMING_MIGRATION_GUIDE_PATH = "docs/features/migrations/physical-naming.md"; + +/** The guide on `main`: the default when a caller has no release to pin to. */ +export const PHYSICAL_NAMING_MIGRATION_GUIDE = docsBlobUrl(PHYSICAL_NAMING_MIGRATION_GUIDE_PATH); /** * What a former column-naming source prints when it is still used (ADR-0058): the one * setting, where it goes, and the guide. Same words in every port. `what` names the source * exactly; `value` is the rule it gave, carried into the example so the move is mechanical. */ -export function physicalNamingRemovedMessage(what: string, value: string = "literal"): string { +export function physicalNamingRemovedMessage( + what: string, + value: string = "literal", + guide: string = PHYSICAL_NAMING_MIGRATION_GUIDE, +): string { return ( `${what} was removed in 1.1.0. Column and table naming is one project setting now, read by ` + `every port: "naming" in .metaobjects/config.json, for example "naming": { "columns": "${value}" }. ` + - `Move it there and delete ${what}. See ${PHYSICAL_NAMING_MIGRATION_GUIDE}` + `Move it there and delete ${what}. See ${guide}` ); } diff --git a/server/typescript/packages/metadata/src/release-docs.ts b/server/typescript/packages/metadata/src/release-docs.ts new file mode 100644 index 000000000..94b2c1b75 --- /dev/null +++ b/server/typescript/packages/metadata/src/release-docs.ts @@ -0,0 +1,48 @@ +// release-docs.ts — guide URLs pinned to the release this toolchain belongs to. +// +// Node-only (reads package.json), so it sits behind its own subpath +// (`@metaobjectsdev/metadata/release-docs`) and is never reachable from the root entry, +// which must stay browser-safe. The pure rule (`docsRefForVersion`) is in `docs-ref.ts`. + +import { existsSync, readFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { docsBlobUrl, docsRefForVersion } from "./docs-ref.js"; +import { PHYSICAL_NAMING_MIGRATION_GUIDE_PATH } from "./naming.js"; + +let cachedVersion: string | undefined; + +/** + * The version of this package, read from its own package.json (walking up from this + * module). All `@metaobjectsdev/*` npm packages release in lockstep, so it is also the + * version of the toolchain a message is printed by. "0.0.0" when not found. + */ +export function metadataPackageVersion(): string { + if (cachedVersion !== undefined) return cachedVersion; + let dir = dirname(fileURLToPath(import.meta.url)); + for (let i = 0; i < 6; i++) { + const candidate = join(dir, "package.json"); + if (existsSync(candidate)) { + try { + const pkg = JSON.parse(readFileSync(candidate, "utf8")) as { name?: string; version?: string }; + if (pkg.name === "@metaobjectsdev/metadata" && pkg.version) return (cachedVersion = pkg.version); + } catch { + // not our manifest / unreadable — keep walking up + } + } + const parent = dirname(dir); + if (parent === dir) break; + dir = parent; + } + return (cachedVersion = "0.0.0"); +} + +/** A guide URL pinned to the release this package belongs to. */ +export function guideUrl(repoRelativePath: string): string { + return docsBlobUrl(repoRelativePath, docsRefForVersion(metadataPackageVersion())); +} + +/** The physical-naming migration guide, pinned: what `physicalNamingRemovedMessage` is given. */ +export function physicalNamingGuideUrl(): string { + return guideUrl(PHYSICAL_NAMING_MIGRATION_GUIDE_PATH); +} diff --git a/server/typescript/packages/metadata/test/conformance/adapter.ts b/server/typescript/packages/metadata/test/conformance/adapter.ts index f5a65e5a1..b5f1bdc4a 100644 --- a/server/typescript/packages/metadata/test/conformance/adapter.ts +++ b/server/typescript/packages/metadata/test/conformance/adapter.ts @@ -205,9 +205,9 @@ export const tsAdapter: ConformanceAdapter = { ...(referrer !== undefined ? { referrer } : {}), ...(target !== undefined ? { target } : {}), }; - return { code: err.code, source }; + return { code: err.code, message: err.message, source }; } - return { code: err.code, source: { format: src.format, files: [] } }; + return { code: err.code, message: err.message, source: { format: src.format, files: [] } }; } const code = (err as { code?: unknown }).code; return { diff --git a/server/typescript/packages/metadata/test/docs-ref.test.ts b/server/typescript/packages/metadata/test/docs-ref.test.ts new file mode 100644 index 000000000..8f3c17935 --- /dev/null +++ b/server/typescript/packages/metadata/test/docs-ref.test.ts @@ -0,0 +1,29 @@ +import { describe, expect, test } from "bun:test"; +import { docsRefForVersion, docsBlobUrl, pinDocsRefs } from "../src/docs-ref.js"; +import { guideUrl, metadataPackageVersion, physicalNamingGuideUrl } from "../src/release-docs.js"; + +describe("docs refs pin to the release tag", () => { + test("a release pins to v", () => { + expect(docsRefForVersion("1.1.0")).toBe("v1.1.0"); + expect(docsRefForVersion(" 1.0.13 ")).toBe("v1.0.13"); + }); + test("a prerelease, the unresolved sentinel and junk keep main", () => { + for (const v of ["1.1.0-rc.7", "1.1.0-SNAPSHOT", "0.0.0", "", "latest"]) expect(docsRefForVersion(v)).toBe("main"); + }); + test("docsBlobUrl and pinDocsRefs", () => { + expect(docsBlobUrl("docs/a.md", "v1.1.0")).toBe("https://github.com/metaobjectsdev/metaobjects/blob/v1.1.0/docs/a.md"); + const text = "see https://github.com/metaobjectsdev/metaobjects/blob/main/docs/a.md and https://example.com/blob/main/x"; + expect(pinDocsRefs(text, "v1.1.0")).toBe( + "see https://github.com/metaobjectsdev/metaobjects/blob/v1.1.0/docs/a.md and https://example.com/blob/main/x", + ); + expect(pinDocsRefs(text, "main")).toBe(text); + }); + test("guideUrl derives from this package's version, never a literal", () => { + expect(guideUrl("docs/a.md")).toBe(docsBlobUrl("docs/a.md", docsRefForVersion(metadataPackageVersion()))); + expect(metadataPackageVersion()).not.toBe("0.0.0"); + }); +}); + +test("the physical-naming guide is pinned the same way", () => { + expect(physicalNamingGuideUrl()).toBe(guideUrl("docs/features/migrations/physical-naming.md")); +}); diff --git a/server/typescript/packages/metadata/test/report-array-map-dimension.test.ts b/server/typescript/packages/metadata/test/report-array-map-dimension.test.ts new file mode 100644 index 000000000..18c610ef4 --- /dev/null +++ b/server/typescript/packages/metadata/test/report-array-map-dimension.test.ts @@ -0,0 +1,47 @@ +// Rule: a derived report field whose source field is an array (isArray) or a field.map is neither +// filterable nor sortable. The read model carries no @filterable on it, and every allowlist built +// from the read model (filter, sort) therefore leaves it out. + +import { describe, expect, test } from "bun:test"; +import { FIELD_ATTR_FILTERABLE, InMemoryStringSource, MetaDataLoader, reportReadModel } from "../src/index.js"; + +const model = { + "metadata.root": { + package: "acme", + children: [ + { + "object.entity": { + name: "Event", + children: [ + { "source.rdb": { "@table": "events" } }, + { "field.long": { name: "id" } }, + { "field.string": { name: "kind" } }, + { "field.string": { name: "tags", isArray: true } }, + { "field.map": { name: "attrs", "@valueType": "string" } }, + { "identity.primary": { name: "id", "@fields": ["id"] } }, + { "dimension.attribute": { name: "kind", "@of": "Event.kind" } }, + { "dimension.attribute": { name: "tags", "@of": "Event.tags" } }, + { "dimension.attribute": { name: "attrs", "@of": "Event.attrs" } }, + { "measure.aggregate": { name: "events", "@agg": "count", "@of": "Event.id" } }, + ], + }, + }, + { + "object.report": { + name: "EventCounts", "@from": "Event", "@dimensions": ["kind", "tags", "attrs"], "@measures": ["events"], + children: [{ "source.rdb": { "@kind": "view", "@view": "v_event_counts" } }], + }, + }, + ], + }, +}; + +describe("array- and map-valued report dimensions", () => { + test("only the scalar dimension and the measure are filterable", async () => { + const { root, errors } = await new MetaDataLoader().load([new InMemoryStringSource(JSON.stringify(model))]); + expect(errors).toEqual([]); + const rm = reportReadModel(root.objects().find((o) => o.name === "EventCounts")!, root); + const filterable = Object.fromEntries(rm.fields().map((f) => [f.name, f.attr(FIELD_ATTR_FILTERABLE) === true])); + expect(filterable).toEqual({ kind: true, tags: false, attrs: false, events: true }); + }); +}); diff --git a/server/typescript/packages/metadata/test/report-describe.test.ts b/server/typescript/packages/metadata/test/report-describe.test.ts index acf16c1d2..6244e59a2 100644 --- a/server/typescript/packages/metadata/test/report-describe.test.ts +++ b/server/typescript/packages/metadata/test/report-describe.test.ts @@ -67,10 +67,6 @@ const MODEL = { "@dimensions": ["issuedOn:month", "customerJoinedAt:month", "region", "tags"], "@measures": ["invoices", "paidShare"], children: [view("v_by_month")] } }, { "object.report": { name: "Sourceless", "@from": "Invoice", "@measures": ["invoices"] } }, - { "object.report": { name: "Materialized", "@from": "Invoice", "@measures": ["invoices"], - children: [view("mv_invoices", "materializedView")] } }, - { "object.report": { name: "AbstractReport", abstract: true, "@from": "Invoice", "@measures": ["invoices"], - children: [view("v_abstract")] } }, { "object.report": { name: "PaidTotals", "@from": "Invoice", "@measures": ["invoices"], "@segment": "paid" } }, { "object.report": { name: "PaidByRegion", "@from": "Invoice", "@spine": "Invoice.customer", "@dimensions": ["region"], "@measures": ["invoices", "liveCentsOrZero"], "@segment": "paid" } }, @@ -223,13 +219,9 @@ describe("why a report is not served (Plan 3 Table A)", () => { expect(reportNotServedReason(object(root, "ByMonth"))).toBeUndefined(); }); - test("no source, a source that is not a view, and an abstract report each say why", async () => { + test("a report with no source says why (an abstract report and a non-view source never load)", async () => { const root = await load(); expect(reportNotServedReason(object(root, "Sourceless"))).toBe("Not served: declares no view source"); - expect(reportNotServedReason(object(root, "Materialized"))) - .toBe("Not served: its source is a materializedView, not a view"); - // Abstract wins over the source it declares. - expect(reportNotServedReason(object(root, "AbstractReport"))).toBe("Not served: the report is abstract"); }); }); diff --git a/server/typescript/packages/metadata/test/reporting-refusals.test.ts b/server/typescript/packages/metadata/test/reporting-refusals.test.ts new file mode 100644 index 000000000..c8ead28bd --- /dev/null +++ b/server/typescript/packages/metadata/test/reporting-refusals.test.ts @@ -0,0 +1,86 @@ +// FR-044 — what a report refuses at load, with the message text every port copies: +// D5 / M10 a dimension or measure over a field.object (or a field carrying @objectRef) +// R10 a report source whose @kind is not `view` +// R11 an abstract report +// The conformance fixtures (error-dimension-over-object, error-measure-over-object, +// error-report-source-kind-*, error-report-abstract) gate the codes and the node; this file +// pins the text. + +import { describe, expect, test } from "bun:test"; +import { InMemoryStringSource, MetaDataLoader } from "../src/index.js"; + +const value = { "object.value": { name: "Tags", children: [{ "field.string": { name: "label" } }] } }; +const event = (extra: unknown[]) => ({ + "object.entity": { + name: "Event", + children: [ + { "source.rdb": { "@table": "events" } }, + { "field.long": { name: "id" } }, + { "field.string": { name: "kind" } }, + { "field.object": { name: "payload", "@objectRef": "acme::Tags", "@storage": "jsonb" } }, + { "identity.primary": { name: "id", "@fields": ["id"] } }, + ...extra, + ], + }, +}); +const ok = [ + { "dimension.attribute": { name: "kind", "@of": "Event.kind" } }, + { "measure.aggregate": { name: "events", "@agg": "count", "@of": "Event.id" } }, +]; +const report = (source: Record | undefined, extra: Record = {}) => ({ + "object.report": { + name: "EventCounts", "@from": "Event", "@dimensions": ["kind"], "@measures": ["events"], ...extra, + ...(source ? { children: [{ "source.rdb": source }] } : {}), + }, +}); + +async function errorsOf(children: unknown[]): Promise<{ code: string; message: string }[]> { + const src = new InMemoryStringSource(JSON.stringify({ "metadata.root": { package: "acme", children } })); + const { errors } = await new MetaDataLoader().load([src]); + return errors.map((e) => ({ code: (e as Error & { code: string }).code, message: e.message })); +} + +describe("a report over a field.object", () => { + test("D5: a dimension over it is ERR_INVALID_DIMENSION", async () => { + const errs = await errorsOf([value, event([...ok, { "dimension.attribute": { name: "payload", "@of": "Event.payload" } }])]); + expect(errs.map((e) => e.code)).toEqual(["ERR_INVALID_DIMENSION"]); + expect(errs[0]!.message).toContain("@of 'Event.payload' is a field.object with an @objectRef, which a report cannot group by. Group by a scalar field of it."); + }); + test("M10: a measure over it is ERR_INVALID_MEASURE, whatever the aggregate", async () => { + for (const agg of ["count", "max"]) { + const errs = await errorsOf([value, event([...ok, { "measure.aggregate": { name: "p", "@agg": agg, "@of": "Event.payload" } }])]); + expect(errs.map((e) => e.code)).toEqual(["ERR_INVALID_MEASURE"]); + expect(errs[0]!.message).toContain("@of 'Event.payload' is a field.object with an @objectRef, which a report cannot aggregate. Aggregate a scalar field of it."); + } + }); +}); + +describe("a report source", () => { + test("R10: only @kind view is read; the others are ERR_INVALID_REPORT on the source", async () => { + const kinds: [string, Record][] = [ + ["materializedView", { "@kind": "materializedView", "@materializedView": "mv" }], + ["storedProc", { "@kind": "storedProc", "@proc": "sp" }], + ["tableFunction", { "@kind": "tableFunction", "@function": "fn" }], + ]; + for (const [kind, src] of kinds) { + const errs = await errorsOf([value, event(ok), report(src)]); + expect(errs.map((e) => e.code)).toEqual(["ERR_INVALID_REPORT"]); + expect(errs[0]!.message).toContain(`has @kind '${kind}'; a report is read through a view, so its source must be @kind: view.`); + expect(errs[0]!.message).toContain("declare @kind: view with @unmanaged: true."); + } + }); + test("R10: a view, an @unmanaged view and an @sql view load", async () => { + for (const src of [ + { "@kind": "view", "@view": "v" }, + { "@kind": "view", "@view": "v", "@unmanaged": true }, + { "@kind": "view", "@view": "v", "@sql": "SELECT 1" }, + ]) { + expect(await errorsOf([value, event(ok), report(src)])).toEqual([]); + } + }); + test("R11: an abstract report is ERR_INVALID_REPORT", async () => { + const errs = await errorsOf([value, event(ok), report({ "@kind": "view", "@view": "v" }, { abstract: true })]); + expect(errs.map((e) => e.code)).toEqual(["ERR_INVALID_REPORT"]); + expect(errs[0]!.message).toContain("a report cannot be abstract"); + }); +}); diff --git a/server/typescript/packages/metadata/test/reporting-validation.test.ts b/server/typescript/packages/metadata/test/reporting-validation.test.ts index ca8a8da50..eddd4dcea 100644 --- a/server/typescript/packages/metadata/test/reporting-validation.test.ts +++ b/server/typescript/packages/metadata/test/reporting-validation.test.ts @@ -366,6 +366,33 @@ describe("validateReporting — dimensions", () => { expect(msg).toContain("'acme::shop::Program'"); }); + test("D1: @of naming a base that lacks a field only the @via terminal subtype declares is refused at load", async () => { + const m: Model = { + "metadata.root": { + package: PKG, + children: [ + { "object.entity": { name: "Item", children: [{ "source.rdb": { "@table": "items" } }, field("long", "id"), primary()] } }, + { "object.entity": { name: "Special", extends: "Item", children: [field("string", "tag")] } }, + { + "object.entity": { + name: "Order", + children: [ + { "source.rdb": { "@table": "orders" } }, + field("long", "id"), + primary(), + { "relationship.association": { name: "item", "@objectRef": "Special", "@cardinality": "one" } }, + { "dimension.attribute": { name: "itemTag", "@of": "Item.tag", "@via": "Order.item" } }, + ], + }, + }, + ], + }, + } as unknown as Model; + const msg = await single(m, "ERR_INVALID_DIMENSION"); + expect(msg).toContain("only 'acme::shop::Special'"); + expect(msg).toContain("Special.tag"); + }); + test("D1: @of not of the form Entity.field is refused", async () => { const m = edit((x) => patchMember(x, "Purchase", "program", { "@of": "programId" })); await single(m, "ERR_INVALID_DIMENSION"); @@ -1502,23 +1529,11 @@ describe("validateReporting — measure @default (M7, M8)", () => { ); }); - test("a non-integer @default on a count reports M7 and the attribute type error", async () => { - const m = edit((x) => patchMember(x, "Purchase", "purchases", { "@default": "zero" })); - const { errors } = await loadInline(m); - expect(errors.map((e) => e.code).sort()).toEqual(["ERR_BAD_ATTR_VALUE", "ERR_INVALID_MEASURE"]); - expect(errors.find((e) => e.code === "ERR_INVALID_MEASURE")!.message).toContain( - "measure 'purchases' on entity 'acme::shop::Purchase': @default cannot apply to @agg: count.", - ); - }); - test("a fractional @default on a sum is ERR_BAD_ATTR_VALUE on the measure node, once", async () => { const m = edit((x) => patchMember(x, "Purchase", "revenue", { "@default": 0.5 })); const { errors } = await loadInline(m); expect(errors.map((e) => e.code)).toEqual(["ERR_BAD_ATTR_VALUE"]); - expect(errors[0]!.message).toBe( - "measure 'revenue' on entity 'acme::shop::Purchase': @default '0.5' is not an integer. A measure's " + - "@default is a whole number (for example 0).", - ); + expect(errors[0]!.message).toContain("@default"); expect(jsonPathOf(errors[0]!.source)).toContain("['measure.aggregate']"); }); @@ -1526,37 +1541,34 @@ describe("validateReporting — measure @default (M7, M8)", () => { const m = edit((x) => patchMember(x, "WorkoutEvent", "avgDaysPerStarter", { "@default": 0.5 })); const { errors } = await loadInline(m); expect(errors.map((e) => e.code)).toEqual(["ERR_BAD_ATTR_VALUE"]); - expect(errors[0]!.message).toBe( - "measure 'avgDaysPerStarter' on entity 'acme::shop::WorkoutEvent': @default '0.5' is not an integer. " + - "A measure's @default is a whole number (for example 0).", - ); expect(jsonPathOf(errors[0]!.source)).toContain("['measure.ratio']"); }); - test("the integer check runs first: a fractional @default on a count skips M7", async () => { + test("a fractional @default on a count reports only the attribute error (M7 skipped)", async () => { const m = edit((x) => patchMember(x, "Purchase", "purchases", { "@default": 0.5 })); - const msg = await single(m, "ERR_BAD_ATTR_VALUE"); - expect(msg).toContain("@default '0.5' is not an integer"); + await single(m, "ERR_BAD_ATTR_VALUE"); + }); + + test("an integral float @default on a count reports only M7 (Rule A: 0.0 is the integer 0)", async () => { + const m = edit((x) => patchMember(x, "Purchase", "purchases", { "@default": 0.0 })); + await single(m, "ERR_INVALID_MEASURE"); }); - test("the integer check runs first: a fractional @default on a max of a timestamp skips M8", async () => { + test("a non-numeric @default on a count reports only the attribute error (Rule B)", async () => { + const m = edit((x) => patchMember(x, "Purchase", "purchases", { "@default": "zero" })); + await single(m, "ERR_BAD_ATTR_VALUE"); + }); + + test("a fractional @default on a max of a timestamp skips M8", async () => { const m = edit((x) => patchMember(x, "WorkoutEvent", "lastActivityAt", { "@default": -1.5 })); - const msg = await single(m, "ERR_BAD_ATTR_VALUE"); - expect(msg).toBe( - "measure 'lastActivityAt' on entity 'acme::shop::WorkoutEvent': @default '-1.5' is not an integer. " + - "A measure's @default is a whole number (for example 0).", - ); + await single(m, "ERR_BAD_ATTR_VALUE"); }); test("a fractional @default on a sum declared on an abstract base is reported ONCE", async () => { const m = inheritedModel(); const base = objectBody(m, "BaseEvent").children as Wrapper[]; base[3] = { "measure.aggregate": { name: "events", "@agg": "sum", "@of": "BaseEvent.id", "@default": 0.5 } }; - const msg = await single(m, "ERR_BAD_ATTR_VALUE"); - expect(msg).toBe( - "measure 'events' on entity 'acme::shop::BaseEvent': @default '0.5' is not an integer. A measure's " + - "@default is a whole number (for example 0).", - ); + await single(m, "ERR_BAD_ATTR_VALUE"); }); test("@default on a sum, an avg, a min of an int and a ratio is fine", async () => { diff --git a/server/typescript/packages/metadata/test/unresolved-super-hint.test.ts b/server/typescript/packages/metadata/test/unresolved-super-hint.test.ts new file mode 100644 index 000000000..1fca9b71d --- /dev/null +++ b/server/typescript/packages/metadata/test/unresolved-super-hint.test.ts @@ -0,0 +1,31 @@ +import { describe, expect, test } from "bun:test"; +import { MetaDataLoader } from "../src/loader/meta-data-loader.js"; +import { InMemoryStringSource } from "../src/loader/meta-data-source.js"; + +function doc(pkg: string, name: string, extendsRef?: string): InMemoryStringSource { + return new InMemoryStringSource( + JSON.stringify({ + "metadata.root": { + package: pkg, + children: [{ "object.value": { name, ...(extendsRef !== undefined ? { extends: extendsRef } : {}) } }], + }, + }), + { id: `meta.${pkg.replace(/::/g, ".")}.${name}.json`, format: "json" }, + ); +} + +describe("ERR_UNRESOLVED_SUPER names the fix", () => { + test("a bare cross-package extends lists the candidate and the qualified spelling", async () => { + const res = await new MetaDataLoader().load([doc("app::a", "Base"), doc("app::b", "Widget", "Base")]); + const err = res.errors.find((e) => (e as { code?: string }).code === "ERR_UNRESOLVED_SUPER"); + expect(err).toBeDefined(); + expect(err!.message).toContain("app::a::Base"); + expect(err!.message).toContain('extends: "app::a::Base"'); + }); + + test("no same-named object anywhere: no hint", async () => { + const res = await new MetaDataLoader().load([doc("app::b", "Widget", "Nope")]); + const err = res.errors.find((e) => (e as { code?: string }).code === "ERR_UNRESOLVED_SUPER"); + expect(err!.message).not.toContain("exists in"); + }); +}); diff --git a/server/typescript/packages/migrate-ts/src/emit/postgres.ts b/server/typescript/packages/migrate-ts/src/emit/postgres.ts index aaf653f02..a0346aacc 100644 --- a/server/typescript/packages/migrate-ts/src/emit/postgres.ts +++ b/server/typescript/packages/migrate-ts/src/emit/postgres.ts @@ -54,10 +54,12 @@ export function renderPostgres(changes: Change[]): EmitResult { const sorted = [...changes].sort((a, b) => STAGE_ORDER[a.kind] - STAGE_ORDER[b.kind]); const upStmts: string[] = []; const downStmts: string[] = []; + const recreated = recreatedViews(sorted); + const recreatedDown = recreatedViewsOnDown(sorted); for (const c of sorted) { const hazard = dataHazardOf(c); - upStmts.push(hazard === undefined ? renderUp(c) : `${hazardComment(hazard)}\n${renderUp(c)}`); - downStmts.push(renderDown(c)); + upStmts.push(hazard === undefined ? renderUp(c, recreated) : `${hazardComment(hazard)}\n${renderUp(c, recreated)}`); + downStmts.push(renderDown(c, recreatedDown)); } // Down runs in reverse order (so creates undo correctly w.r.t. FKs). const upHelpers = conversionHelpers(sorted, "up"); @@ -72,9 +74,118 @@ export function renderPostgres(changes: Change[]): EmitResult { ...[...downStmts].reverse(), ...downHelpers.map(dropHelper), ].join("\n\n"), recreatedTables: new Set(), // postgres alters in place; no recreate-and-copy + notices: [...recreated].sort().map((k) => `view ${k}: dropped and recreated; its grants are carried over.`), }; } +/** The key a recreated view is tracked by: `.`, the default schema spelled out. */ +function viewKey(name: string, schema: string | undefined): string { + return `${schema ?? DEFAULT_DB_SCHEMA_POSTGRES}.${name}`; +} + +/** + * The views this migration DROPS and creates again: a `drop-view` that a `create-view` or + * `replace-view` of the same view follows, and a `replace-view` the planner could not prove + * legal for `CREATE OR REPLACE` (written as DROP + CREATE). A DROP discards the privileges + * granted on the view, so each of these captures them before the drop and grants them again + * after the create (see {@link renderCaptureViewGrants}). A proven `CREATE OR REPLACE VIEW` + * keeps the view's privileges by itself and is not listed. + */ +function recreatedViews(changes: readonly Change[]): Set { + const dropped = new Set(); + const created = new Set(); + const out = new Set(); + for (const c of changes) { + if (c.kind === "drop-view") dropped.add(viewKey(c.view, c.schema)); + else if (c.kind === "create-view") created.add(viewKey(c.view.name, c.schema ?? c.view.schema)); + else if (c.kind === "replace-view") { + const key = viewKey(c.view.name, c.schema); + created.add(key); + if (c.proven === undefined) out.add(key); + } + } + for (const k of dropped) if (created.has(k)) out.add(k); + return out; +} + +/** + * The views the DOWN migration drops and creates again. The paired drop-view + create-view + * is the same set as UP (down drops the view UP created, then builds the one UP dropped). A + * `replace-view` is different: UP may be a proven CREATE OR REPLACE (grants kept) while its + * down cannot be (`proven.restorable` false), so down is DROP + CREATE and needs the + * capture and re-grant too. A replace whose down is an in-place replace, or has no body + * to restore, is not listed. + */ +function recreatedViewsOnDown(changes: readonly Change[]): Set { + const out = recreatedViews(changes.filter((c) => c.kind !== "replace-view")); + for (const c of changes) { + if (c.kind !== "replace-view") continue; + const hasBody = c.restore?.sql !== undefined && c.restore.sql.trim().length > 0; + if (hasBody && c.proven?.restorable !== true) out.add(viewKey(c.view.name, c.schema)); + } + return out; +} + +/** Session-scoped holding table for the grants of a view that is dropped and created again. */ +const GRANTS_TABLE = "_mo_view_grants"; + +/** + * Remember the privileges granted on a view the migration is about to drop. Reads `relacl` + * through `aclexplode`, skipping the owner's own implicit privileges (the recreated view's + * owner is whoever applies the migration). The holding table is a session temp table, so + * nothing outlives the migration's connection, and the capture needs no privilege beyond + * reading `pg_class`. + */ +function renderCaptureViewGrants(name: string, schema: string | undefined): string { + const qualified = quoteQualifiedView(name, schema).replace(/'/g, "''"); + const key = viewKey(name, schema).replace(/'/g, "''"); + const grantee = (col: string): string => + `CASE WHEN ${col} = 0 THEN 'PUBLIC' ELSE quote_ident(pg_get_userbyid(${col})) END`; + return [ + `CREATE TEMP TABLE IF NOT EXISTS ${GRANTS_TABLE} (view_key text, grantee text, privilege text, grantable boolean, column_name text);`, + // The view's own privileges (relacl), then the column-level ones (pg_attribute.attacl). + `INSERT INTO ${GRANTS_TABLE} (view_key, grantee, privilege, grantable, column_name)\n` + + `SELECT '${key}', ${grantee("a.grantee")}, a.privilege_type, a.is_grantable, NULL\n` + + `FROM pg_class c, aclexplode(c.relacl) a\n` + + `WHERE c.oid = to_regclass('${qualified}') AND c.relacl IS NOT NULL AND a.grantee <> c.relowner;`, + `INSERT INTO ${GRANTS_TABLE} (view_key, grantee, privilege, grantable, column_name)\n` + + `SELECT '${key}', ${grantee("x.grantee")}, x.privilege_type, x.is_grantable, t.attname\n` + + `FROM pg_class c JOIN pg_attribute t ON t.attrelid = c.oid, aclexplode(t.attacl) x\n` + + `WHERE c.oid = to_regclass('${qualified}') AND t.attacl IS NOT NULL AND NOT t.attisdropped AND x.grantee <> c.relowner;`, + ].join("\n"); +} + +/** + * Grant again, after the CREATE, what {@link renderCaptureViewGrants} captured: to the same + * grantees, WITH GRANT OPTION where it was held. The applier owns the new view, so this needs + * no superuser. Raises a NOTICE so the apply log shows it happened. + */ +function renderRestoreViewGrants(name: string, schema: string | undefined): string { + const qualified = quoteQualifiedView(name, schema).replace(/'/g, "''"); + const key = viewKey(name, schema).replace(/'/g, "''"); + return ( + `-- grants on ${quoteQualifiedView(name, schema)} are carried over from before the drop\n` + + `DO $mo_grants$\nDECLARE r record; n integer := 0;\nBEGIN\n` + + ` IF to_regclass('pg_temp.${GRANTS_TABLE}') IS NOT NULL THEN\n` + + ` FOR r IN SELECT grantee, privilege, grantable, column_name FROM ${GRANTS_TABLE} WHERE view_key = '${key}' LOOP\n` + + ` IF r.column_name IS NULL THEN\n` + + ` EXECUTE format('GRANT %s ON %s TO %s%s', r.privilege, '${qualified}', r.grantee,\n` + + ` CASE WHEN r.grantable THEN ' WITH GRANT OPTION' ELSE '' END);\n` + + ` n := n + 1;\n` + + // A column grant is carried over only to a column the new view still has. + ` ELSIF EXISTS (SELECT 1 FROM pg_attribute WHERE attrelid = to_regclass('${qualified}')\n` + + ` AND attname = r.column_name AND attnum > 0 AND NOT attisdropped) THEN\n` + + ` EXECUTE format('GRANT %s (%I) ON %s TO %s%s', r.privilege, r.column_name, '${qualified}', r.grantee,\n` + + ` CASE WHEN r.grantable THEN ' WITH GRANT OPTION' ELSE '' END);\n` + + ` n := n + 1;\n` + + ` END IF;\n` + + ` END LOOP;\n` + + ` DELETE FROM ${GRANTS_TABLE} WHERE view_key = '${key}';\n` + + ` END IF;\n` + + ` RAISE NOTICE 'view ${qualified}: % privilege(s) carried over', n;\n` + + `END\n$mo_grants$;` + ); +} /** * `CREATE SCHEMA IF NOT EXISTS` for every non-default schema this migration creates * an object in, ahead of everything else it emits. @@ -105,7 +216,7 @@ function createSchemaStmts(sorted: readonly Change[]): string[] { return [...schemas].sort().map((s) => `CREATE SCHEMA IF NOT EXISTS ${quote(s)};`); } -function renderUp(c: Change): string { +function renderUp(c: Change, recreated: ReadonlySet = new Set()): string { switch (c.kind) { case "create-table": return renderCreateTable(c.table); // #313 — every FORWARD drop is `IF EXISTS`. A committed chain must apply to a @@ -173,9 +284,16 @@ function renderUp(c: Change): string { case "add-check": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} ADD CONSTRAINT ${quote(c.check.name)} CHECK (${c.check.expression});`; // Same `ALTER TABLE IF EXISTS` gap as `drop-fk` above. case "drop-check": return `ALTER TABLE IF EXISTS ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT IF EXISTS ${quote(c.check)};`; - case "create-view": return renderCreateView(c.view, c.schema, /* orReplace */ false); - case "drop-view": return renderDropView(c); - case "replace-view": return renderReplaceView(c); + case "create-view": { + const key = viewKey(c.view.name, c.schema ?? c.view.schema); + const create = renderCreateView(c.view, c.schema, /* orReplace */ false); + return recreated.has(key) ? `${create}\n${renderRestoreViewGrants(c.view.name, c.schema ?? c.view.schema)}` : create; + } + case "drop-view": { + const drop = renderDropView(c); + return recreated.has(viewKey(c.view, c.schema)) ? `${renderCaptureViewGrants(c.view, c.schema)}\n${drop}` : drop; + } + case "replace-view": return renderReplaceView(c, recreated); } } @@ -217,7 +335,7 @@ function renderRenameTable(c: Extract, dir: "u return (up ? lines : [...lines].reverse()).join("\n\n"); } -function renderDown(c: Change): string { +function renderDown(c: Change, recreated: ReadonlySet = new Set()): string { switch (c.kind) { case "create-table": return `DROP TABLE ${quoteQualified(c.table.name, c.table.schema)};`; case "drop-table": { @@ -282,11 +400,26 @@ function renderDown(c: Change): string { return c.restore ? `ALTER TABLE ${quoteQualified(c.table, c.schema)} ADD CONSTRAINT ${quote(c.restore.name)} CHECK (${c.restore.expression});` : `-- WARNING: down migration cannot restore the original CHECK definition`; - case "create-view": return `DROP VIEW ${quoteQualifiedView(c.view.name, c.schema)};`; + case "create-view": { + const name = c.view.name; + const schema = c.schema ?? c.view.schema; + const drop = `DROP VIEW ${quoteQualifiedView(name, c.schema)};`; + // Down drops the view UP created, then builds the one UP dropped: keep the grants across it. + return recreated.has(viewKey(name, schema)) ? `${renderCaptureViewGrants(name, c.schema)}\n${drop}` : drop; + } // The deparsed body introspection captured IS a valid restore payload. - case "drop-view": return renderRestoreView(c.restore, c.view, c.schema, "create"); - case "replace-view": - return renderRestoreView(c.restore, c.view.name, c.schema, c.proven?.restorable === true ? "replace" : "drop-and-create"); + case "drop-view": { + const restore = renderRestoreView(c.restore, c.view, c.schema, "create"); + return recreated.has(viewKey(c.view, c.schema)) && c.restore?.sql !== undefined + ? `${restore}\n${renderRestoreViewGrants(c.view, c.schema)}` + : restore; + } + case "replace-view": { + const restore = renderRestoreView(c.restore, c.view.name, c.schema, c.proven?.restorable === true ? "replace" : "drop-and-create"); + return recreated.has(viewKey(c.view.name, c.schema)) + ? `${renderCaptureViewGrants(c.view.name, c.schema)}\n${restore}\n${renderRestoreViewGrants(c.view.name, c.schema)}` + : restore; + } } } @@ -633,10 +766,16 @@ function renderCreateView(v: ViewDescriptor, schema: string | undefined, orRepla * DROP is not `CASCADE`: if something depends on the view, Postgres says so instead of * destroying it. */ -function renderReplaceView(c: Extract): string { +function renderReplaceView(c: Extract, recreated: ReadonlySet): string { if (c.proven !== undefined) return renderCreateView(c.view, c.schema, /* orReplace */ true); const qualified = quoteQualifiedView(c.view.name, c.schema); - return `DROP VIEW IF EXISTS ${qualified};\n${renderCreateView(c.view, c.schema, /* orReplace */ false)}`; + const create = renderCreateView(c.view, c.schema, /* orReplace */ false); + // A DROP discards the view's privileges: capture them first, grant them again after the CREATE. + if (!recreated.has(viewKey(c.view.name, c.schema))) return `DROP VIEW IF EXISTS ${qualified};\n${create}`; + return ( + `${renderCaptureViewGrants(c.view.name, c.schema)}\nDROP VIEW IF EXISTS ${qualified};\n${create}\n` + + renderRestoreViewGrants(c.view.name, c.schema) + ); } function renderViewComment(qualifiedView: string, comment: string | null): string { diff --git a/server/typescript/packages/migrate-ts/src/index.ts b/server/typescript/packages/migrate-ts/src/index.ts index fcd2a29ee..f6f0ad991 100644 --- a/server/typescript/packages/migrate-ts/src/index.ts +++ b/server/typescript/packages/migrate-ts/src/index.ts @@ -151,3 +151,5 @@ export { type D1Binding, type WranglerConfig, } from "./wrangler-config.js"; +export { splitSqlStatements } from "./sql/split-statements.js"; +export { viewFingerprint } from "./view-fingerprint.js"; diff --git a/server/typescript/packages/migrate-ts/src/types.ts b/server/typescript/packages/migrate-ts/src/types.ts index 098074af1..4a808836e 100644 --- a/server/typescript/packages/migrate-ts/src/types.ts +++ b/server/typescript/packages/migrate-ts/src/types.ts @@ -562,6 +562,12 @@ export interface EmitResult { * mid-recreate source table. */ recreatedTables: ReadonlySet; + /** + * One line per notable thing the emitted SQL does that a reader of the plan should hear + * about (postgres: each view dropped and recreated, whose grants are carried over). The CLI + * prints them; absent or empty when there is nothing to say. + */ + notices?: readonly string[]; } export type Dialect = "postgres" | "sqlite" | "d1"; diff --git a/server/typescript/packages/migrate-ts/test/emit-postgres-view-grants-recreate.test.ts b/server/typescript/packages/migrate-ts/test/emit-postgres-view-grants-recreate.test.ts new file mode 100644 index 000000000..0e6edfb76 --- /dev/null +++ b/server/typescript/packages/migrate-ts/test/emit-postgres-view-grants-recreate.test.ts @@ -0,0 +1,77 @@ +// A recreated view keeps its grants in BOTH directions and at BOTH levels: the DOWN migration's +// recreate carries them over too, and column privileges (pg_attribute.attacl) ride with the +// view's own — re-granted only for columns the new view still has. Runs on real Postgres +// semantics (PGlite): the SQL is applied and the privileges read back. +import { describe, expect, test } from "bun:test"; +import { PGlite } from "@electric-sql/pglite"; +import { emit } from "../src/emit/index.js"; +import type { Change, ViewDescriptor } from "../src/types.js"; + +const ALLOWED = { kind: "allowed" } as unknown as Change extends { status: infer S } ? S : never; +const oldView: ViewDescriptor = { name: "v_r", sql: "SELECT a, b FROM t", dependsOn: [] }; +const newView: ViewDescriptor = { name: "v_r", sql: "SELECT a, c FROM t", dependsOn: [] }; + +// Unproven replace: UP is DROP + CREATE; DOWN (restore present, not restorable) is too. +const change: Change = { kind: "replace-view", view: newView, restore: oldView, status: ALLOWED } as unknown as Change; + +async function setup(): Promise { + const db = new PGlite(); + await db.exec(` + CREATE ROLE reader; CREATE ROLE colreader; + CREATE TABLE t (a int, b int, c int); + CREATE VIEW v_r AS SELECT a, b FROM t; + GRANT SELECT ON v_r TO reader; + GRANT SELECT (a) ON v_r TO colreader; + GRANT SELECT (b) ON v_r TO colreader; + `); + return db; +} + +async function grants(db: PGlite): Promise { + const rel = await db.query<{ g: string }>( + `SELECT pg_get_userbyid(a.grantee) || ':' || a.privilege_type AS g + FROM pg_class c, aclexplode(c.relacl) a WHERE c.oid = 'v_r'::regclass AND a.grantee <> c.relowner`, + ); + const col = await db.query<{ g: string }>( + `SELECT pg_get_userbyid(x.grantee) || ':' || x.privilege_type || '(' || t.attname || ')' AS g + FROM pg_attribute t, aclexplode(t.attacl) x WHERE t.attrelid = 'v_r'::regclass AND t.attacl IS NOT NULL AND x.grantee <> (SELECT relowner FROM pg_class WHERE oid = 'v_r'::regclass)`, + ); + return [...rel.rows, ...col.rows].map((r) => r.g).sort(); +} + +describe("postgres: recreated view grants, column level and the DOWN direction", () => { + test("UP keeps relation grants and re-grants columns that still exist, dropping the removed one", async () => { + const db = await setup(); + await db.exec(emit([change], { dialect: "postgres" }).up); + expect(await grants(db)).toEqual(["colreader:SELECT(a)", "reader:SELECT"]); + }); + + test("DOWN recreates the old view and carries the grants over again", async () => { + const db = await setup(); + const sql = emit([change], { dialect: "postgres" }); + await db.exec(sql.up); + await db.exec("GRANT SELECT (c) ON v_r TO colreader"); + await db.exec(sql.down); + // c no longer exists in the old view: its grant is dropped; a and the relation grant stay. + expect(await grants(db)).toEqual(["colreader:SELECT(a)", "reader:SELECT"]); + }); + + test("DOWN of a drop-view + create-view pair keeps the grants too", async () => { + const db = await setup(); + const pair: Change[] = [ + { kind: "drop-view", view: "v_r", restore: oldView, status: ALLOWED } as unknown as Change, + { kind: "create-view", view: newView, status: ALLOWED } as unknown as Change, + ]; + const sql = emit(pair, { dialect: "postgres" }); + await db.exec(sql.up); + expect(await grants(db)).toEqual(["colreader:SELECT(a)", "reader:SELECT"]); + await db.exec(sql.down); + expect(await grants(db)).toEqual(["colreader:SELECT(a)", "reader:SELECT"]); + }); + + test("a proven restorable replace needs no capture in DOWN", () => { + const proven = { ...change, proven: { restorable: true } } as unknown as Change; + const r = emit([proven], { dialect: "postgres" }); + expect(r.down).not.toContain("aclexplode"); + }); +}); diff --git a/server/typescript/packages/migrate-ts/test/emit-postgres-view-grants.test.ts b/server/typescript/packages/migrate-ts/test/emit-postgres-view-grants.test.ts new file mode 100644 index 000000000..195e68225 --- /dev/null +++ b/server/typescript/packages/migrate-ts/test/emit-postgres-view-grants.test.ts @@ -0,0 +1,44 @@ +import { describe, expect, test } from "bun:test"; +import { emit } from "../src/emit/index.js"; +import type { Change, ViewDescriptor } from "../src/types.js"; + +const view: ViewDescriptor = { name: "v_r", sql: "SELECT 1 AS a", dependsOn: [] }; +const ALLOWED = { kind: "allowed" } as unknown as Change extends { status: infer S } ? S : never; + +describe("postgres: a view dropped and created again carries its grants over", () => { + test("an unproven replace-view captures before the DROP and grants again after the CREATE", () => { + const r = emit([{ kind: "replace-view", view, status: ALLOWED }], { dialect: "postgres" }); + const i = (needle: string): number => r.up.indexOf(needle); + expect(i("aclexplode")).toBeGreaterThan(-1); + expect(i("aclexplode")).toBeLessThan(i('DROP VIEW IF EXISTS "v_r"')); + expect(i('DROP VIEW IF EXISTS "v_r"')).toBeLessThan(i('CREATE VIEW "v_r"')); + expect(i('CREATE VIEW "v_r"')).toBeLessThan(i("GRANT %s ON %s TO %s%s")); + expect(r.up).toContain("WITH GRANT OPTION"); + expect(r.notices).toEqual(["view public.v_r: dropped and recreated; its grants are carried over."]); + }); + + test("a drop-view paired with a create-view of the same view does the same", () => { + const r = emit( + [ + { kind: "drop-view", view: "v_r", status: ALLOWED }, + { kind: "create-view", view, status: ALLOWED }, + ], + { dialect: "postgres" }, + ); + expect(r.up.indexOf("aclexplode")).toBeLessThan(r.up.indexOf('CREATE VIEW "v_r"')); + expect(r.up).toContain("GRANT %s ON %s TO %s%s"); + expect(r.notices).toHaveLength(1); + }); + + test("a plain create, a plain drop and a proven CREATE OR REPLACE add nothing", () => { + for (const changes of [ + [{ kind: "create-view", view, status: ALLOWED }], + [{ kind: "drop-view", view: "v_r", status: ALLOWED }], + [{ kind: "replace-view", view, proven: { restorable: true }, status: ALLOWED }], + ] as Change[][]) { + const r = emit(changes, { dialect: "postgres" }); + expect(r.up).not.toContain("aclexplode"); + expect(r.notices).toEqual([]); + } + }); +}); diff --git a/server/typescript/packages/migrate-ts/test/integration/pg-adopt-view-239.test.ts b/server/typescript/packages/migrate-ts/test/integration/pg-adopt-view-239.test.ts index a9e9c4882..41c95bc07 100644 --- a/server/typescript/packages/migrate-ts/test/integration/pg-adopt-view-239.test.ts +++ b/server/typescript/packages/migrate-ts/test/integration/pg-adopt-view-239.test.ts @@ -20,13 +20,14 @@ import { Kysely, PostgresDialect, sql } from "kysely"; import { introspectPostgres } from "../../src/introspect/postgres.js"; import { diff } from "../../src/diff/index.js"; import { emit } from "../../src/emit/index.js"; +import { splitSqlStatements } from "../../src/sql/split-statements.js"; import { viewFingerprint } from "../../src/view-fingerprint.js"; import type { SchemaSnapshot } from "../../src/types.js"; const PG_URL = process.env["MIGRATE_TS_PG_URL"]; async function applyRaw(k: Kysely>, sqlText: string): Promise { - for (const stmt of sqlText.split(";").map((s) => s.trim()).filter((s) => s.length > 0)) { + for (const stmt of splitSqlStatements(sqlText)) { await sql.raw(stmt).execute(k); } } diff --git a/server/typescript/packages/runtime-ts/src/decimal-wire.ts b/server/typescript/packages/runtime-ts/src/decimal-wire.ts index c71e7034c..fab6c35d5 100644 --- a/server/typescript/packages/runtime-ts/src/decimal-wire.ts +++ b/server/typescript/packages/runtime-ts/src/decimal-wire.ts @@ -1,21 +1,120 @@ -// The wire spelling of a field.decimal that a read-only mount SENDS. +// The wire form of a field.decimal — ADR-0019 Amendment 1 (W1-W4). The ONE decimal boundary of +// the TypeScript runtime: every decimal column of an entity, a projection or a report, on every +// dialect, crosses to a client through here, and every decimal a client sends comes in through +// here (filter operands) or through the same grammar (the generated Zod write schemas, which +// carry a local copy of `canonicalDecimal` because generated model code imports nothing from +// MetaObjects; test/decimal-wire.test.ts holds both to fixtures/decimal-wire/vectors.json). // -// A decimal is a string on the wire: it is the string Postgres `numeric` and MySQL `DECIMAL` -// really return, and the string the TypeScript read schema types it as. SQLite has no -// decimal. A decimal field on a table is a `text` column and stays a string, but the value -// a report's view computes (a ratio, an average, a sum of a decimal field) is a REAL, which -// a SQLite driver hands back as a JS number. Sent as it is, the same route answers -// `"paidShare": "0.4"` on Postgres and `"paidShare": 0.4` on SQLite, and the type says -// `string` for both. +// W1 A decimal is a JSON string in canonical form, `^-?(0|[1-9][0-9]*)(\.[0-9]*[1-9])?$`: +// plain notation, no exponent, no trailing fractional zeros, zero as "0". The server never +// formats a scale; a client that wants `12.50` formats with the field's `@scale`. +// W2 Input is a string in the JSON-number grammar (an exponent is allowed on input only) or a +// JSON number. Anything else is refused: `parseDecimalInput` answers `undefined`. +// W3 A value the declared column cannot hold is an error, never a silent rounding +// (`decimalFits`, applied by the generated write schemas). +// W4 SQLite and D1 have no decimal type: a decimal on them is a double, exact to 15 significant +// digits. The read path renders the double with 15 significant digits, then canonicalizes +// it, so `0.1 + 0.2` reads `"0.3"` (`decimalFromDouble15`). // -// So a mount takes the NAMES of its decimal fields (`decimalColumns`, which the generated -// report route passes on SQLite) and sends a number among them as its string. Which keys are -// decimals comes from the caller, never from the shape of a value: an integer column is not -// a decimal because it is a number. A string, a null and anything else pass through. The -// digits are JavaScript's own shortest round-trip spelling of the double (`1.6666666666666667`); -// like every decimal's spelling on the wire, they are the engine's and not part of the contract. - -/** Rewrite a number under any of `columns` to its string. Returns `undefined` when there is nothing to do. */ +// A string is never turned into a number on the way (no `Number()` on a decimal anywhere): an +// operand or a body value stays text, and is bound to SQL as text. + +/** W2: the JSON-number grammar, exponent allowed. No whitespace, no leading `+`, no bare point. */ +export const DECIMAL_INPUT_RE = /^(-?)(0|[1-9][0-9]*)(?:\.([0-9]+))?(?:[eE]([+-]?[0-9]+))?$/; + +/** W1: the canonical spelling a decimal takes on the wire. */ +export const DECIMAL_CANONICAL_RE = /^-?(0|[1-9][0-9]*)(\.[0-9]*[1-9])?$/; + +/** An expanded value longer than this many digits is out of range (W3): `1e999999999` must not allocate. */ +const MAX_EXPANDED_DIGITS = 1000; + +/** + * W1 + W2: parse `text` (the JSON-number grammar) and answer its canonical string, or `undefined` + * when it is not a number in that grammar or is too large to expand. Exact: it works on the digits + * and the exponent, never on a double. + */ +export function canonicalDecimal(text: string): string | undefined { + const m = DECIMAL_INPUT_RE.exec(text); + if (m === null) return undefined; + const negative = m[1] === "-"; + const intPart = m[2] ?? ""; + const fracPart = m[3] ?? ""; + const exponent = m[4] === undefined ? 0 : Number(m[4]); + if (!Number.isSafeInteger(exponent) || Math.abs(exponent) > MAX_EXPANDED_DIGITS) return undefined; + let digits = intPart + fracPart; + let point = intPart.length + exponent; // the decimal point sits after `point` digits + const lead = digits.length - digits.replace(/^0+/, "").length; + digits = digits.slice(lead); + point -= lead; + digits = digits.replace(/0+$/, ""); + if (digits.length === 0) return "0"; + let plain: string; + if (point <= 0) plain = `0.${"0".repeat(-point)}${digits}`; + else if (point >= digits.length) plain = digits + "0".repeat(point - digits.length); + else plain = `${digits.slice(0, point)}.${digits.slice(point)}`; + if (plain.length > MAX_EXPANDED_DIGITS) return undefined; + return negative ? `-${plain}` : plain; +} + +/** + * W4: the canonical string of an IEEE double at 15 significant digits (SQLite's own text + * precision). `undefined` for NaN and the infinities. + */ +export function decimalFromDouble15(value: number): string | undefined { + if (!Number.isFinite(value)) return undefined; + return canonicalDecimal(value.toPrecision(15)); +} + +/** + * W2 for a value read off a parsed JSON request body: a string in the JSON-number grammar, or a + * JSON number (exact to 15 significant digits, W4). `undefined` for anything else. + */ +export function parseDecimalInput(value: unknown): string | undefined { + if (typeof value === "string") return canonicalDecimal(value); + if (typeof value === "number") return decimalFromDouble15(value); + return undefined; +} + +/** W2 for a query-string filter operand: the same grammar as a body string, compared exactly. */ +export function parseDecimalOperand(text: string): string | undefined { + return canonicalDecimal(text); +} + +/** + * W3: whether the canonical decimal `canonical` fits a column declared `@precision` / `@scale` + * (at most `precision - scale` integer digits and `scale` fraction digits). A column rounds an + * excess fraction silently and refuses an excess integer part, so neither may reach it. + */ +export function decimalFits(canonical: string, precision: number | undefined, scale: number | undefined): boolean { + if (precision === undefined) return true; + const s = scale ?? 0; + const unsigned = canonical.startsWith("-") ? canonical.slice(1) : canonical; + const dot = unsigned.indexOf("."); + const intDigits = dot === -1 ? unsigned : unsigned.slice(0, dot); + const fracDigits = dot === -1 ? 0 : unsigned.length - dot - 1; + const intLen = intDigits === "0" ? 0 : intDigits.length; + return fracDigits <= s && intLen <= precision - s; +} + +/** + * W1/W4 on a value a driver hands back for a decimal column: a string (Postgres `numeric`, MySQL + * `DECIMAL`, a SQLite text value) is canonicalized, a number (a SQLite or D1 REAL or INTEGER) is + * rendered at 15 significant digits and canonicalized, a bigint is its digits. Null passes + * through. A value in none of those shapes is left for the caller to see. + */ +export function decimalToWire(value: unknown): unknown { + if (typeof value === "string") return canonicalDecimal(value) ?? value; + if (typeof value === "number") return decimalFromDouble15(value) ?? value; + if (typeof value === "bigint") return value.toString(); + return value; +} + +/** + * Rewrite the values under `columns` to their canonical string. Returns `undefined` when there is + * nothing to do. Which keys are decimals comes from the caller (the generated route passes the + * names of the entity's `field.decimal` fields), never from the shape of a value: an integer + * column is not a decimal because it is a number. + */ export function decimalWire( columns: readonly string[] | undefined, ): ((row: unknown) => unknown) | undefined { @@ -24,9 +123,17 @@ export function decimalWire( if (row === null || typeof row !== "object") return row; const out = { ...(row as Record) }; for (const c of columns) { - const v = out[c]; - if (typeof v === "number") out[c] = String(v); + if (c in out) out[c] = decimalToWire(out[c]); } return out; }; } + +/** A wire mapper with the decimal rewrite of `columns` applied after `base` (which may be a timestamp mapper). */ +export function withDecimals( + base: (row: unknown) => unknown, + columns: readonly string[] | undefined, +): (row: unknown) => unknown { + const decimals = decimalWire(columns); + return decimals === undefined ? base : (row) => decimals(base(row)); +} diff --git a/server/typescript/packages/runtime-ts/src/drizzle-fastify/filter-allowlist.ts b/server/typescript/packages/runtime-ts/src/drizzle-fastify/filter-allowlist.ts index e5f043aa2..386b10e1e 100644 --- a/server/typescript/packages/runtime-ts/src/drizzle-fastify/filter-allowlist.ts +++ b/server/typescript/packages/runtime-ts/src/drizzle-fastify/filter-allowlist.ts @@ -11,6 +11,7 @@ import { OPS_BY_SUBTYPE, opsForSubType, FIELD_SUBTYPE_DATE, + FIELD_SUBTYPE_DECIMAL, FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_UUID, @@ -28,6 +29,7 @@ export type FilterSubType = "string" | "number" | "boolean" | "datetime"; */ export const FILTER_VALUE_FORMATS = [ FIELD_SUBTYPE_DATE, + FIELD_SUBTYPE_DECIMAL, FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_UUID, @@ -52,7 +54,9 @@ export interface FilterFieldRule { readonly dateValues?: boolean; /** * The field's exact wire format, set by codegen for `field.date` / `field.time` / - * `field.timestamp` / `field.uuid`. The filter parser rejects a value that does not + * `field.timestamp` / `field.uuid` / `field.decimal`. A `decimal` operand is parsed with the + * JSON-number grammar and bound as an exact string, never through a double (ADR-0019 + * Amendment 1, W2). The filter parser rejects a value that does not * match it with `invalid_filter_value` instead of binding it — on SQLite a malformed * date compared as text and silently matched nothing; on Postgres it was a driver * error. Absent (an allowlist generated before this existed), a `datetime` value is diff --git a/server/typescript/packages/runtime-ts/src/drizzle-fastify/filter-parser.ts b/server/typescript/packages/runtime-ts/src/drizzle-fastify/filter-parser.ts index 11c20e585..76abece6c 100644 --- a/server/typescript/packages/runtime-ts/src/drizzle-fastify/filter-parser.ts +++ b/server/typescript/packages/runtime-ts/src/drizzle-fastify/filter-parser.ts @@ -15,7 +15,8 @@ import { ANY_TEMPORAL_EXPECTED, FORMAT_EXPECTED, PLUS_OFFSET_HINT, looksLikeUnencodedPlusOffset, matchesAnyTemporal, matchesFormat, utcIsoIfZoned, utcIsoInstant, } from "./filter-value-format.js"; -import { FIELD_SUBTYPE_TIMESTAMP } from "@metaobjectsdev/metadata"; +import { FIELD_SUBTYPE_DECIMAL, FIELD_SUBTYPE_TIMESTAMP } from "@metaobjectsdev/metadata"; +import { canonicalDecimal } from "../decimal-wire.js"; // biome-ignore lint/suspicious/noExplicitAny: dynamic dispatch over user's Drizzle table type AnyTable = any; @@ -215,14 +216,14 @@ function compileOp( throw new FilterParseError("filter.unsupported_op", `Op "${op}" not supported for field "${field}".`, { field, op, allowed: rule.ops }); } switch (op as FilterOp) { - case "eq": return eq(col as any, coerce(value, rule, col, field, op)); - case "ne": return ne(col as any, coerce(value, rule, col, field, op)); - case "gt": return gt(col as any, coerce(value, rule, col, field, op)); - case "gte": return gte(col as any, coerce(value, rule, col, field, op)); - case "lt": return lt(col as any, coerce(value, rule, col, field, op)); - case "lte": return lte(col as any, coerce(value, rule, col, field, op)); + case "eq": return eq(col as any, coerce(value, rule, col, field, op, dialect)); + case "ne": return ne(col as any, coerce(value, rule, col, field, op, dialect)); + case "gt": return gt(col as any, coerce(value, rule, col, field, op, dialect)); + case "gte": return gte(col as any, coerce(value, rule, col, field, op, dialect)); + case "lt": return lt(col as any, coerce(value, rule, col, field, op, dialect)); + case "lte": return lte(col as any, coerce(value, rule, col, field, op, dialect)); case "in": { - const list = String(value).split(",").map((v) => coerce(v.trim(), rule, col, field, op)); + const list = String(value).split(",").map((v) => coerce(v.trim(), rule, col, field, op, dialect)); if (list.length > maxInList) { throw new FilterParseError("filter.in_too_large", `In-list size ${list.length} exceeds limit ${maxInList}.`, { field, limit: maxInList }); } @@ -327,10 +328,22 @@ function assertWellFormed(s: string, rule: FilterFieldRule, col: unknown, field: } /** Check a comparison value against the field's rule, then coerce it for binding. */ -function coerce(value: unknown, rule: FilterFieldRule, col: unknown, field: string, op: string): unknown { +function coerce(value: unknown, rule: FilterFieldRule, col: unknown, field: string, op: string, dialect: SqlDialect): unknown { if (value === null || value === undefined) return null; const s = typeof value === "string" ? value : String(value); assertWellFormed(s, rule, col, field, op); + // A decimal operand is exact text, never a number (ADR-0019 Amendment 1, W2): parsed with + // the JSON-number grammar and bound as its canonical string. MySQL compares a DECIMAL column + // with a string operand as DOUBLES, so there the operand is cast to a wide DECIMAL first. + // SQLite and D1 hold a decimal as a double (W4) and order every number before any text, so a + // text operand against a computed column (a view's sum or average has no NUMERIC affinity) + // would compare false in the wrong direction: the operand is bound as the number it is, exact + // to the 15 digits W4 promises. + if (rule.format === FIELD_SUBTYPE_DECIMAL) { + const canonical = canonicalDecimal(s) as string; + if (dialect === "mysql") return sql`CAST(${canonical} AS DECIMAL(65,30))`; + return dialect === "sqlite" ? Number(canonical) : canonical; + } // A timestamp bound is compared in the same UTC spelling the generated schemas store: // on SQLite/D1 the column is TEXT and compares as text. An instant field (`instant`, i.e. // not @localTime) reads a zoneless or date-only bound as UTC, as its write schema does — diff --git a/server/typescript/packages/runtime-ts/src/drizzle-fastify/filter-value-format.ts b/server/typescript/packages/runtime-ts/src/drizzle-fastify/filter-value-format.ts index fec7b54a7..5def0974f 100644 --- a/server/typescript/packages/runtime-ts/src/drizzle-fastify/filter-value-format.ts +++ b/server/typescript/packages/runtime-ts/src/drizzle-fastify/filter-value-format.ts @@ -13,11 +13,13 @@ import { FIELD_SUBTYPE_DATE, + FIELD_SUBTYPE_DECIMAL, FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_UUID, } from "@metaobjectsdev/metadata"; import type { FilterValueFormat } from "./filter-allowlist.js"; +import { canonicalDecimal } from "../decimal-wire.js"; const DATE_RE = /^(\d{4})-(\d{2})-(\d{2})$/; const TIME_RE = /^(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d{1,9})?)?$/; @@ -45,6 +47,7 @@ const CHECKS: Readonly boolean>> = { [FIELD_SUBTYPE_TIME]: (s) => TIME_RE.test(s), [FIELD_SUBTYPE_TIMESTAMP]: isTimestamp, [FIELD_SUBTYPE_UUID]: (s) => UUID_RE.test(s), + [FIELD_SUBTYPE_DECIMAL]: (s) => canonicalDecimal(s) !== undefined, }; /** The `expected` member of the `invalid_filter_value` envelope, per format. */ @@ -53,6 +56,7 @@ export const FORMAT_EXPECTED: Readonly> = { [FIELD_SUBTYPE_TIME]: "time (HH:MM[:SS[.fff]])", [FIELD_SUBTYPE_TIMESTAMP]: "timestamp (YYYY-MM-DD[THH:MM[:SS[.fff]]][Z|±HH:MM])", [FIELD_SUBTYPE_UUID]: "uuid (8-4-4-4-12 hex)", + [FIELD_SUBTYPE_DECIMAL]: "decimal number (digits, optional fraction, optional exponent)", }; /** `expected` for a temporal rule that does not say WHICH temporal format it is. */ diff --git a/server/typescript/packages/runtime-ts/src/drizzle-fastify/index.ts b/server/typescript/packages/runtime-ts/src/drizzle-fastify/index.ts index 734594217..d159a176a 100644 --- a/server/typescript/packages/runtime-ts/src/drizzle-fastify/index.ts +++ b/server/typescript/packages/runtime-ts/src/drizzle-fastify/index.ts @@ -29,12 +29,13 @@ export type { FilterAllowlist, SortAllowlist } from "./filter-allowlist.js"; import { parseFilterParams, parsePageBound, FilterParseError } from "./filter-parser.js"; import { isTruthyFlag, contractErrorCode, coerceIdForColumn, firstRow, extractRowCount } from "./util.js"; import { timestampWire } from "../timestamp-wire.js"; +import { withDecimals } from "../decimal-wire.js"; import { supportsReturning, type SqlDialect } from "../sql-dialect.js"; import { withContractErrorHandler } from "./route-error-handler.js"; import { validationErrorBody } from "../route-errors.js"; export { isTruthyFlag, contractErrorCode, parseId, coerceIdForColumn } from "./util.js"; export { timestampWire, canonicalTimestamp } from "../timestamp-wire.js"; -export { decimalWire } from "../decimal-wire.js"; +export { decimalWire, withDecimals, canonicalDecimal, parseDecimalInput, parseDecimalOperand } from "../decimal-wire.js"; // --------------------------------------------------------------------------- // Loose types — we don't bind to a specific Drizzle backend so the helper @@ -85,6 +86,13 @@ export interface CrudRoutesOptions { updateMethod?: "patch" | "put"; filterAllowlist?: FilterAllowlist; sortAllowlist?: SortAllowlist; + /** + * The names of the entity's `field.decimal` fields. A decimal is a canonical string on the + * wire on every dialect (ADR-0019 Amendment 1): Postgres and MySQL return the column's scale + * (`"100.5000"`) and SQLite hands back a number, so each row the routes send is rewritten + * under these keys. The generated route passes them; absent, rows are sent as read. + */ + decimalColumns?: readonly string[]; /** Dialect — required if filterAllowlist or sortAllowlist is set (for dialect-specific `like` lowering: SQLite lowers to GLOB to stay case-sensitive; ADR-0049). */ dialect?: SqlDialect; /** @@ -123,7 +131,7 @@ function readSource(opts: VerbOptions): AnyTable { * non-timestamp of the same name. */ function readWire(opts: VerbOptions): (row: unknown) => unknown { - return timestampWire(readSource(opts)); + return withDecimals(timestampWire(readSource(opts)), opts.decimalColumns); } /** @@ -132,7 +140,7 @@ function readWire(opts: VerbOptions): (row: unknown) => unknown { * against the union of both sources. */ function echoWire(opts: VerbOptions): (row: unknown) => unknown { - return timestampWire(opts.table, opts.readView); + return withDecimals(timestampWire(opts.table, opts.readView), opts.decimalColumns); } /** #214 — after a write to the base table, re-read the row THROUGH the replica diff --git a/server/typescript/packages/runtime-ts/src/drizzle-fastify/mount-m2m.ts b/server/typescript/packages/runtime-ts/src/drizzle-fastify/mount-m2m.ts index cd6ea8320..31e1e4e8c 100644 --- a/server/typescript/packages/runtime-ts/src/drizzle-fastify/mount-m2m.ts +++ b/server/typescript/packages/runtime-ts/src/drizzle-fastify/mount-m2m.ts @@ -34,6 +34,7 @@ import type { FastifyInstance, RouteShorthandOptions } from "fastify"; import { and, eq, or, inArray } from "drizzle-orm"; import { coerceIdForColumn } from "./util.js"; import { timestampWire } from "../timestamp-wire.js"; +import { withDecimals } from "../decimal-wire.js"; import { withContractErrorHandler } from "./route-error-handler.js"; // Loose Drizzle types — the helper works across libsql / better-sqlite3 / pg. @@ -60,6 +61,8 @@ export interface M2mRouteOptions { targetColumn: string; /** Target entity PK column (physical column name, e.g. "id"). Defaults to "id". */ targetPkColumn?: string; + /** The names of the TARGET entity's `field.decimal` fields, sent as canonical strings (ADR-0019 Amendment 1). */ + decimalColumns?: readonly string[]; /** Undirected self-join: union both junction FK columns on read. */ symmetric: boolean; /** @@ -87,7 +90,7 @@ export function mountM2mRoute(opts: M2mRouteOptions): void { const route = `${opts.path}/:id/${opts.relationName}`; // Route-scoped contract error handler — see route-error-handler.ts. const ro = withContractErrorHandler(opts.routeOptions); - const toWire = timestampWire(opts.targetTable); + const toWire = withDecimals(timestampWire(opts.targetTable), opts.decimalColumns); opts.fastify.get(route, ro, async (req, reply) => { const { id } = req.params as { id: string }; diff --git a/server/typescript/packages/runtime-ts/src/drizzle-fastify/mount-read-only.ts b/server/typescript/packages/runtime-ts/src/drizzle-fastify/mount-read-only.ts index b4f926228..471334410 100644 --- a/server/typescript/packages/runtime-ts/src/drizzle-fastify/mount-read-only.ts +++ b/server/typescript/packages/runtime-ts/src/drizzle-fastify/mount-read-only.ts @@ -6,7 +6,7 @@ import { parseFilterParams, parsePageBound, RAW_VIEW_MAX_LIMIT, FilterParseError import type { FilterAllowlist, SortAllowlist } from "./filter-allowlist.js"; import { isTruthyFlag, contractErrorCode, coerceIdForColumn, rawIdLiteral, viewBaseConfig } from "./util.js"; import { timestampWire } from "../timestamp-wire.js"; -import { decimalWire } from "../decimal-wire.js"; +import { withDecimals } from "../decimal-wire.js"; import { withContractErrorHandler } from "./route-error-handler.js"; // biome-ignore lint/suspicious/noExplicitAny: dynamic dispatch over user-supplied views @@ -51,11 +51,11 @@ export interface MountReadOnlyOptions { /** The noun in the 405 message, which is free prose. Default "projection". */ readonly resource?: "projection" | "report"; /** - * The names of the view's decimal fields, for a dialect that has no decimal. SQLite hands - * a computed decimal (a report's ratio, average or sum) back as a REAL, a JS number, where - * Postgres and MySQL return the string the read schema types it as. A number under one of - * these keys is sent as its string, so the route answers the same on every engine. The - * generated route passes it for a report on SQLite and omits it otherwise. Default none. + * The names of the view's `field.decimal` fields. A decimal is a canonical string on the wire + * on every dialect (ADR-0019 Amendment 1): Postgres and MySQL return the column's scale + * (`"100.5000"`), and SQLite hands back a number, a REAL for a computed one (a report's ratio, + * average or sum). Each row is rewritten under these keys. The generated route passes them; + * absent, rows are sent as read. Default none. */ readonly decimalColumns?: readonly string[]; } @@ -160,8 +160,7 @@ export function mountReadOnlyCrudRoutes(opts: MountReadOnlyOptions): void { const useRawSql = isEmptyColumnView(view) && !!viewName; // The raw-SQL branch has no declared columns, so nothing names a timestamp there. const timestamps = timestampWire(view); - const decimals = decimalWire(opts.decimalColumns); - const toWire = decimals === undefined ? timestamps : (row: unknown) => decimals(timestamps(row)); + const toWire = withDecimals(timestamps, opts.decimalColumns); // ── List ────────────────────────────────────────────────────────────────── fastify.get(path, ro, async (req, reply) => { diff --git a/server/typescript/packages/runtime-ts/src/hono/index.ts b/server/typescript/packages/runtime-ts/src/hono/index.ts index 93b6a1298..f403a7f6e 100644 --- a/server/typescript/packages/runtime-ts/src/hono/index.ts +++ b/server/typescript/packages/runtime-ts/src/hono/index.ts @@ -45,6 +45,7 @@ export type { import { isTruthyFlag, coerceIdForColumn, firstRow, extractRowCount } from "../drizzle-fastify/util.js"; import { supportsReturning, type SqlDialect } from "../sql-dialect.js"; import { timestampWire } from "../timestamp-wire.js"; +import { withDecimals } from "../decimal-wire.js"; // Every handler below is wrapped in guardRoute (an unexpected error answers // `500 { error: "internal" }`, logged server-side) and reads its body through // readJsonBody (a malformed body answers `400 { error: "invalid_json" }`). @@ -53,7 +54,7 @@ import { guardRoute, readJsonBody } from "./route-guard.js"; // shared (deprecated) helper so the three adapters can't silently diverge. export { parseId } from "../drizzle-fastify/util.js"; export { timestampWire, canonicalTimestamp } from "../timestamp-wire.js"; -export { decimalWire } from "../decimal-wire.js"; +export { decimalWire, withDecimals, canonicalDecimal, parseDecimalInput, parseDecimalOperand } from "../decimal-wire.js"; // --------------------------------------------------------------------------- // Loose types — we don't bind to a specific Drizzle backend so the helper @@ -105,6 +106,13 @@ export interface CrudRoutesOptions { updateMethod?: "patch" | "put"; filterAllowlist?: FilterAllowlist; sortAllowlist?: SortAllowlist; + /** + * The names of the entity's `field.decimal` fields. A decimal is a canonical string on the + * wire on every dialect (ADR-0019 Amendment 1): Postgres and MySQL return the column's scale + * (`"100.5000"`) and SQLite hands back a number, so each row the routes send is rewritten + * under these keys. The generated route passes them; absent, rows are sent as read. + */ + decimalColumns?: readonly string[]; /** Dialect — required if filterAllowlist or sortAllowlist is set (for dialect-specific `like` lowering: SQLite lowers to GLOB to stay case-sensitive; ADR-0049). */ dialect?: SqlDialect; } @@ -151,7 +159,7 @@ function readSource(opts: VerbOptions): AnyTable { * non-timestamp of the same name. */ function readWire(opts: VerbOptions): (row: unknown) => unknown { - return timestampWire(readSource(opts)); + return withDecimals(timestampWire(readSource(opts)), opts.decimalColumns); } /** @@ -160,7 +168,7 @@ function readWire(opts: VerbOptions): (row: unknown) => unknown { * against the union of both sources. */ function echoWire(opts: VerbOptions): (row: unknown) => unknown { - return timestampWire(opts.table, opts.readView); + return withDecimals(timestampWire(opts.table, opts.readView), opts.decimalColumns); } /** Re-read a just-written row through the replica view so the response carries the diff --git a/server/typescript/packages/runtime-ts/src/hono/mount-read-only.ts b/server/typescript/packages/runtime-ts/src/hono/mount-read-only.ts index b598469dd..bda2cdec2 100644 --- a/server/typescript/packages/runtime-ts/src/hono/mount-read-only.ts +++ b/server/typescript/packages/runtime-ts/src/hono/mount-read-only.ts @@ -13,7 +13,7 @@ import type { } from "../drizzle-fastify/filter-allowlist.js"; import { isTruthyFlag, coerceIdForColumn, rawIdLiteral, contractErrorCode, viewBaseConfig } from "../drizzle-fastify/util.js"; import { timestampWire } from "../timestamp-wire.js"; -import { decimalWire } from "../decimal-wire.js"; +import { withDecimals } from "../decimal-wire.js"; // An unexpected error on a mounted route answers `500 { error: "internal" }`. import { guardRoute } from "./route-guard.js"; @@ -50,11 +50,11 @@ export interface MountReadOnlyOptions { /** The noun in the 405 message, which is free prose. Default "projection". */ readonly resource?: "projection" | "report"; /** - * The names of the view's decimal fields, for a dialect that has no decimal. SQLite hands - * a computed decimal (a report's ratio, average or sum) back as a REAL, a JS number, where - * Postgres and MySQL return the string the read schema types it as. A number under one of - * these keys is sent as its string, so the route answers the same on every engine. The - * generated route passes it for a report on SQLite and omits it otherwise. Default none. + * The names of the view's `field.decimal` fields. A decimal is a canonical string on the wire + * on every dialect (ADR-0019 Amendment 1): Postgres and MySQL return the column's scale + * (`"100.5000"`), and SQLite hands back a number, a REAL for a computed one (a report's ratio, + * average or sum). Each row is rewritten under these keys. The generated route passes them; + * absent, rows are sent as read. Default none. */ readonly decimalColumns?: readonly string[]; } @@ -139,8 +139,7 @@ export function mountReadOnlyCrudRoutes(opts: MountReadOnlyOptions): void { const useRawSql = isEmptyColumnView(view) && !!viewName; // The raw-SQL branch has no declared columns, so nothing names a timestamp there. const timestamps = timestampWire(view); - const decimals = decimalWire(opts.decimalColumns); - const toWire = decimals === undefined ? timestamps : (row: unknown) => decimals(timestamps(row)); + const toWire = withDecimals(timestamps, opts.decimalColumns); // ── List ────────────────────────────────────────────────────────────────── app.get(path, guardRoute(async (c) => { diff --git a/server/typescript/packages/runtime-ts/src/object-manager.ts b/server/typescript/packages/runtime-ts/src/object-manager.ts index c298a44e7..8437a1068 100644 --- a/server/typescript/packages/runtime-ts/src/object-manager.ts +++ b/server/typescript/packages/runtime-ts/src/object-manager.ts @@ -1,4 +1,5 @@ import type { MetaData } from "@metaobjectsdev/metadata"; +import { physicalNamingGuideUrl } from "@metaobjectsdev/metadata/release-docs"; import { TYPE_OBJECT, TYPE_FIELD, OBJECT_SUBTYPE_REPORT, @@ -72,7 +73,7 @@ export class ObjectManager { // The 1.0 option is refused rather than dropped: its value may be the one that matches. const removedNaming = (opts as { columnNamingStrategy?: unknown }).columnNamingStrategy; if (removedNaming !== undefined) { - throw new Error(physicalNamingRemovedMessage("the ObjectManager columnNamingStrategy option", String(removedNaming))); + throw new Error(physicalNamingRemovedMessage("the ObjectManager columnNamingStrategy option", String(removedNaming), physicalNamingGuideUrl())); } this.columnNamingStrategy = physicalNamingOf(this.metadata).columns; } diff --git a/server/typescript/packages/runtime-ts/src/type-coercer.ts b/server/typescript/packages/runtime-ts/src/type-coercer.ts index 03e15f6a3..c3c684b9e 100644 --- a/server/typescript/packages/runtime-ts/src/type-coercer.ts +++ b/server/typescript/packages/runtime-ts/src/type-coercer.ts @@ -3,6 +3,10 @@ // - SQLite has no native boolean: booleans are stored as 0/1 ints, so we map // boolean↔int on the way in/out. MySQL's BOOLEAN is TINYINT(1) and mysql2 reads it // back as 0/1, so it takes the same mapping. +// - field.decimal is the canonical decimal STRING on every dialect (ADR-0019 Amendment 1): a +// string Postgres/MySQL hand back loses its scale (`100.5000` -> `100.5`), a number SQLite/D1 +// hand back is rendered at 15 significant digits, and a value written as a number or in +// exponent form is bound as its exact canonical string. // - JSONB-backed object/map fields: the driver (node-postgres) does NOT // serialize a plain JS object to a jsonb column — it must arrive as a JSON // string, or the write fails / writes "[object Object]". A `field.object` @@ -18,6 +22,7 @@ import type { MetaData } from "@metaobjectsdev/metadata"; import { TYPE_FIELD, FIELD_SUBTYPE_BOOLEAN, + FIELD_SUBTYPE_DECIMAL, FIELD_SUBTYPE_ENUM, FIELD_SUBTYPE_OBJECT, FIELD_SUBTYPE_TIMESTAMP, @@ -29,10 +34,11 @@ import { DB_COLUMN_TYPE_JSONB, } from "@metaobjectsdev/metadata"; import type { Dialect, Row } from "./persistence-driver.js"; +import { decimalToWire, parseDecimalInput } from "./decimal-wire.js"; export function coerceRowOnRead(entity: MetaData, row: Row, dialect: Dialect): Row { const hydrated = parseJsonTextArrays(entity, deserializeJsonbObjectFields(entity, row)); - const decoded = decodeIntBackedEnums(entity, hydrated); + const decoded = canonicalizeDecimals(entity, decodeIntBackedEnums(entity, hydrated)); if (dialect !== "sqlite" && dialect !== "mysql") return decoded; return mapBooleansFromInt(entity, decoded); } @@ -134,7 +140,7 @@ export function coerceRowOnWrite(entity: MetaData, row: Row, dialect: Dialect): // Dialect-independent: an int-backed enum's column is INTEGER on every dialect // (SQLite has one integer storage class), so the symbol→int encode is not gated // on the dialect the way the boolean mapping below is. - const encoded = encodeIntBackedEnums(entity, jsonbColumned); + const encoded = encodeIntBackedEnums(entity, bindDecimals(entity, jsonbColumned)); if (dialect === "mysql") return mapBooleansToInt(entity, isoTimestampsToDates(entity, encoded)); if (dialect !== "sqlite") return encoded; return mapBooleansToInt(entity, encoded); @@ -281,3 +287,44 @@ function parseJsonTextArrays(entity: MetaData, row: Row): Row { } return out ?? row; } + +/** Read side: every `field.decimal` the row carries becomes its canonical string. */ +function canonicalizeDecimals(entity: MetaData, row: Row): Row { + let out: Row | null = null; + // Effective children (ADR-0039): a decimal may be inherited via extends, and a report's + // derived ratio / average / sum is a decimal field of the report itself. + for (const child of entity.children()) { + if (child.type !== TYPE_FIELD || child.subType !== FIELD_SUBTYPE_DECIMAL) continue; + if (!(child.name in row)) continue; + const v = row[child.name]; + if (v === null || v === undefined) continue; + const wire = decimalToWire(v); + if (wire === v) continue; + out ??= { ...row }; + out[child.name] = wire; + } + return out ?? row; +} + +/** + * Write side: a `field.decimal` value is bound as its exact canonical string. A string in the + * JSON-number grammar or a number is accepted; anything else is refused here rather than + * reaching a column that would round it, reject it with a driver error, or store garbage. + */ +function bindDecimals(entity: MetaData, row: Row): Row { + let out: Row | null = null; + for (const child of entity.children()) { + if (child.type !== TYPE_FIELD || child.subType !== FIELD_SUBTYPE_DECIMAL) continue; + if (!(child.name in row)) continue; + const v = row[child.name]; + if (v === null || v === undefined) continue; + const canonical = parseDecimalInput(v); + if (canonical === undefined) { + throw new Error(`field.decimal '${child.name}' cannot be written: ${JSON.stringify(v)} is not a decimal number (a string in the JSON-number grammar, or a number)`); + } + if (canonical === v) continue; + out ??= { ...row }; + out[child.name] = canonical; + } + return out ?? row; +} diff --git a/server/typescript/packages/runtime-ts/test/decimal-wire-vectors.test.ts b/server/typescript/packages/runtime-ts/test/decimal-wire-vectors.test.ts new file mode 100644 index 000000000..21530b3a8 --- /dev/null +++ b/server/typescript/packages/runtime-ts/test/decimal-wire-vectors.test.ts @@ -0,0 +1,80 @@ +// L1 of the decimal wire rule (ADR-0019 Amendment 1): every vector in +// fixtures/decimal-wire/vectors.json against the runtime's PRODUCTION helpers, the ones the +// mounts and the filter parser call. No database. The generated Zod write schemas carry a local +// copy of canonicalDecimal; codegen-ts/test/decimal-write-schema.test.ts holds that copy to the +// same file. + +import { describe, test, expect } from "bun:test"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import { + canonicalDecimal, decimalFromDouble15, decimalToWire, decimalFits, parseDecimalInput, parseDecimalOperand, + DECIMAL_CANONICAL_RE, +} from "../src/decimal-wire.js"; + +interface Vectors { + format: Array<{ in: string; out: string }>; + parse: Array<{ json: string; value?: string | null; error?: boolean }>; + filterOperand: Array<{ text: string; value?: string; error?: boolean }>; + double15: Array<{ double: string; out: string }>; +} +const vectors = JSON.parse( + readFileSync(join(import.meta.dir, "..", "..", "..", "..", "..", "fixtures", "decimal-wire", "vectors.json"), "utf8"), +) as Vectors; + +describe("decimal wire vectors (production helpers)", () => { + for (const v of vectors.format) { + test(`format ${v.in} -> ${v.out}`, () => { + expect(canonicalDecimal(v.in)).toBe(v.out); + expect(DECIMAL_CANONICAL_RE.test(v.out)).toBe(true); + // a driver's string goes through the same path + expect(decimalToWire(v.in)).toBe(v.out); + }); + } + for (const v of vectors.parse) { + test(`parse ${v.json}`, () => { + const parsed: unknown = JSON.parse(v.json); + if (v.error === true) { + expect(parseDecimalInput(parsed)).toBeUndefined(); + } else if (v.value === null) { + expect(parsed).toBeNull(); // null is not parsed as a decimal: it is the absence of one + } else { + expect(parseDecimalInput(parsed)).toBe(v.value as string); + } + }); + } + for (const v of vectors.filterOperand) { + test(`filter operand ${JSON.stringify(v.text)}`, () => { + if (v.error === true) expect(parseDecimalOperand(v.text)).toBeUndefined(); + else expect(parseDecimalOperand(v.text)).toBe(v.value as string); + }); + } + for (const v of vectors.double15) { + test(`double15 ${v.double} -> ${v.out}`, () => { + expect(decimalFromDouble15(Number(v.double))).toBe(v.out); + expect(decimalToWire(Number(v.double))).toBe(v.out); + }); + } +}); + +describe("decimal wire beyond the vectors", () => { + test("a huge exponent is refused, not expanded", () => { + expect(canonicalDecimal("1e999999999")).toBeUndefined(); + expect(canonicalDecimal("1e-999999999")).toBeUndefined(); + }); + test("NaN and the infinities have no decimal", () => { + expect(decimalFromDouble15(Number.NaN)).toBeUndefined(); + expect(parseDecimalInput(Number.POSITIVE_INFINITY)).toBeUndefined(); + }); + test("W3: a value must fit the declared precision and scale", () => { + expect(decimalFits("12345678901234567.89", 28, 10)).toBe(true); + expect(decimalFits("0.12345678901", 28, 10)).toBe(false); // 11 fraction digits would be rounded + expect(decimalFits("123456789012345678.5", 28, 10)).toBe(true); // 18 integer digits = 28 - 10 + expect(decimalFits("1234567890123456789.5", 28, 10)).toBe(false); + expect(decimalFits("0", 9, 4)).toBe(true); + expect(decimalFits("-99999.9999", 9, 4)).toBe(true); + expect(decimalFits("100000", 9, 4)).toBe(false); + expect(decimalFits("5.5", 9, undefined)).toBe(false); // numeric(9) is scale 0 + expect(decimalFits("5.5", undefined, undefined)).toBe(true); + }); +}); diff --git a/server/typescript/packages/runtime-ts/test/decimal-wire.test.ts b/server/typescript/packages/runtime-ts/test/decimal-wire.test.ts index 28d48b542..19f953979 100644 --- a/server/typescript/packages/runtime-ts/test/decimal-wire.test.ts +++ b/server/typescript/packages/runtime-ts/test/decimal-wire.test.ts @@ -1,6 +1,6 @@ -// `decimalColumns` on the read-only mounts: a SQLite REAL under a declared decimal key is -// sent as its string, exactly the spelling Postgres `numeric` and MySQL `DECIMAL` already -// give. Both adapters (Fastify and Hono), plus the pure function they share. +// `decimalColumns` on the read-only mounts: a decimal under a declared key is sent as its +// canonical string on every dialect (ADR-0019 Amendment 1): a SQLite REAL at 15 significant +// digits, a Postgres/MySQL string without its trailing zeros. Both adapters (Fastify and Hono), plus the pure function they share. import { describe, test, expect, beforeAll, afterAll } from "bun:test"; import Fastify, { type FastifyInstance } from "fastify"; @@ -21,9 +21,10 @@ describe("decimalWire", () => { test("a number under a named key becomes its string; everything else passes through", () => { const wire = decimalWire(["share", "avg"])!; expect(wire({ share: 0.4, avg: 1.6666666666666667, n: 5, label: "a" })).toEqual({ - share: "0.4", avg: "1.6666666666666667", n: 5, label: "a", + share: "0.4", avg: "1.66666666666667", n: 5, label: "a", }); - expect(wire({ share: null, avg: "2.50", n: 5 })).toEqual({ share: null, avg: "2.50", n: 5 }); + // a string is canonicalized too: Postgres and MySQL hand back the column's scale + expect(wire({ share: null, avg: "2.50", n: 5 })).toEqual({ share: null, avg: "2.5", n: 5 }); expect(wire({ n: 5 })).toEqual({ n: 5 }); expect(wire(null)).toBeNull(); }); diff --git a/server/typescript/packages/runtime-ts/test/filter-parser-decimal.test.ts b/server/typescript/packages/runtime-ts/test/filter-parser-decimal.test.ts new file mode 100644 index 000000000..6ce4106bd --- /dev/null +++ b/server/typescript/packages/runtime-ts/test/filter-parser-decimal.test.ts @@ -0,0 +1,52 @@ +// A decimal filter operand is exact text (ADR-0019 Amendment 1, W2): parsed with the JSON-number +// grammar, bound as its canonical string, never turned into a double. Drizzle's own parameter +// serialization runs over the expression, so what is asserted is what reaches the driver. + +import { describe, test, expect } from "bun:test"; +import { pgTable, bigserial, numeric, PgDialect } from "drizzle-orm/pg-core"; +import { MySqlDialect } from "drizzle-orm/mysql-core"; +import { SQLiteSyncDialect } from "drizzle-orm/sqlite-core"; +import type { FilterAllowlist } from "../src/drizzle-fastify/filter-allowlist.js"; +import { parseFilterParams, FilterParseError } from "../src/drizzle-fastify/filter-parser.js"; + +const table = pgTable("entries", { + id: bigserial("id", { mode: "number" }).primaryKey(), + amount: numeric("amount", { precision: 28, scale: 10 }), +}); +const allowlist: FilterAllowlist = { + amount: { ops: ["eq", "ne", "gt", "gte", "lt", "lte", "in", "isNull"], subType: "number", format: "decimal", leadingWildcard: false }, +}; + +function params(dialect: "postgres" | "mysql" | "sqlite", query: Record): unknown[] { + const { where } = parseFilterParams({ query: { filter: query }, table, allowlist, sortAllowlist: {}, dialect }); + if (where === undefined) throw new Error("expected a WHERE expression"); + const d = dialect === "postgres" ? new PgDialect() : dialect === "mysql" ? new MySqlDialect() : new SQLiteSyncDialect(); + return d.sqlToQuery(where.getSQL()).params; +} + +describe("decimal filter operand", () => { + test("Postgres: the operand is bound as its canonical string, digits intact", () => { + expect(params("postgres", { amount: { eq: "12345678901234567.89" } })).toEqual(["12345678901234567.89"]); + expect(params("postgres", { amount: { gt: "100.500" } })).toEqual(["100.5"]); + expect(params("postgres", { amount: { eq: "1.005e2" } })).toEqual(["100.5"]); + expect(params("postgres", { amount: { in: "0.10,0.2" } })).toEqual(["0.1", "0.2"]); + }); + + test("MySQL: the operand is cast to a wide DECIMAL so the comparison is not made as doubles", () => { + const { where } = parseFilterParams({ query: { filter: { amount: { eq: "12345678901234567.89" } } }, table, allowlist, sortAllowlist: {}, dialect: "mysql" }); + const q = new MySqlDialect().sqlToQuery(where!.getSQL()); + expect(q.sql).toContain("CAST("); + expect(q.sql).toContain("DECIMAL(65,30)"); + expect(q.params).toEqual(["12345678901234567.89"]); + }); + + test("SQLite: a decimal is a double (W4), so the operand is bound as a number, not text", () => { + expect(params("sqlite", { amount: { gt: "0.25" } })).toEqual([0.25]); + }); + + test("a bad operand is refused, naming the field", () => { + for (const bad of ["abc", "1e", "0x10", "", " 1", "1,5"]) { + expect(() => params("postgres", { amount: { eq: bad } })).toThrow(FilterParseError); + } + }); +}); diff --git a/server/typescript/packages/runtime-ts/test/object-manager-naming.test.ts b/server/typescript/packages/runtime-ts/test/object-manager-naming.test.ts index 4c1308499..f2df5f214 100644 --- a/server/typescript/packages/runtime-ts/test/object-manager-naming.test.ts +++ b/server/typescript/packages/runtime-ts/test/object-manager-naming.test.ts @@ -2,6 +2,9 @@ // the loaded root, so they are the names codegen and `meta migrate` used. The 1.0 option that // set them separately is refused rather than dropped. +import { guideUrl } from "@metaobjectsdev/metadata/release-docs"; +// The guide links name the tag of the running release (a prerelease keeps main). +const BLOB = guideUrl(""); import { describe, expect, test } from "bun:test"; import { InMemoryStringSource, MetaDataLoader, type MetaRoot } from "@metaobjectsdev/metadata"; import { ObjectManager } from "../src/object-manager.js"; @@ -35,7 +38,7 @@ describe("ObjectManager column naming (ADR-0058)", () => { const root = await load(); const opts = { metadata: root, driver: inMemoryDriver(), columnNamingStrategy: "literal" }; expect(() => new ObjectManager(opts as ConstructorParameters[0])).toThrow( - 'the ObjectManager columnNamingStrategy option was removed in 1.1.0. Column and table naming is one project setting now, read by every port: "naming" in .metaobjects/config.json, for example "naming": { "columns": "literal" }. Move it there and delete the ObjectManager columnNamingStrategy option. See https://github.com/metaobjectsdev/metaobjects/blob/main/docs/features/migrations/physical-naming.md', + 'the ObjectManager columnNamingStrategy option was removed in 1.1.0. Column and table naming is one project setting now, read by every port: "naming" in .metaobjects/config.json, for example "naming": { "columns": "literal" }. Move it there and delete the ObjectManager columnNamingStrategy option. See ' + BLOB + 'docs/features/migrations/physical-naming.md', ); }); diff --git a/server/typescript/packages/runtime-ts/test/object-manager-report.test.ts b/server/typescript/packages/runtime-ts/test/object-manager-report.test.ts index 8ba04ebfb..24c0257d6 100644 --- a/server/typescript/packages/runtime-ts/test/object-manager-report.test.ts +++ b/server/typescript/packages/runtime-ts/test/object-manager-report.test.ts @@ -134,7 +134,7 @@ describe("ObjectManager reads a view-backed report (FR-044)", () => { expect(rows).toHaveLength(3); expect(rows[0]).toEqual({ program: 1, programTitle: "Alpha", weeks: 3, longWeeks: 1, labels: 3, slots: 3, - totalMinutes: 150, avgMinutes: "50.0000", minMinutes: 30, maxMinutes: 75, longShare: "0.3333", + totalMinutes: 150, avgMinutes: "50", minMinutes: 30, maxMinutes: 75, longShare: "0.3333", }); }); @@ -179,7 +179,7 @@ describe("ObjectManager reads a view-backed report (FR-044)", () => { }); root.physicalNaming = { columns: "literal", tables: "snake_case" }; const om = new ObjectManager({ metadata: root, driver }); - expect(await om.findMany("FitnessTotals")).toEqual([{ weeks: 6, totalMinutes: 310, longShare: "0.5000" }]); + expect(await om.findMany("FitnessTotals")).toEqual([{ weeks: 6, totalMinutes: 310, longShare: "0.5" }]); }); test("rows are coerced by derived subtype: an int-backed enum dimension reads and filters as its symbol", async () => { diff --git a/server/typescript/packages/sdk/src/agent-context/assemble.ts b/server/typescript/packages/sdk/src/agent-context/assemble.ts index 4a45ec6c1..b1b99f960 100644 --- a/server/typescript/packages/sdk/src/agent-context/assemble.ts +++ b/server/typescript/packages/sdk/src/agent-context/assemble.ts @@ -1,5 +1,6 @@ import { readFileSync, readdirSync, existsSync, statSync } from "node:fs"; import { join } from "node:path"; +import { pinDocsRefs } from "@metaobjectsdev/metadata"; import { SKILL_NAMES, SERVER_LANGS, CLIENT_FRAMEWORKS, CONCERN_TOKENS, type AssembledFile, type Stack, } from "./types.js"; @@ -153,9 +154,17 @@ function applyTemplate( }); } -/** Assemble the consumer files for a resolved stack. Pure given the content tree. */ -export function assemble(opts: { contentRoot: string; stack: Stack }): AssembledFile[] { +/** + * Assemble the consumer files for a resolved stack. Pure given the content tree. + * + * `docsRef` is the git ref every guide link in the shipped prose points at (default + * `main`, which is what the content tree is authored with). The scaffolder passes the tag + * of the running release (`docsRefForVersion`), so the links an agent follows are the + * guides of the version that wrote them and do not move after the cut. + */ +export function assemble(opts: { contentRoot: string; stack: Stack; docsRef?: string }): AssembledFile[] { const { contentRoot, stack } = opts; + const docsRef = opts.docsRef ?? "main"; const out: AssembledFile[] = []; // 1. Always-on (AGENTS.md + CLAUDE.md, identical contents). @@ -204,5 +213,5 @@ export function assemble(opts: { contentRoot: string; stack: Stack }): Assembled // Stable order: by path. out.sort((a, b) => (a.path < b.path ? -1 : a.path > b.path ? 1 : 0)); - return out; + return out.map((f) => ({ ...f, contents: pinDocsRefs(f.contents, docsRef) })); } diff --git a/server/typescript/packages/sdk/src/agent-context/scaffold.ts b/server/typescript/packages/sdk/src/agent-context/scaffold.ts index 2fab48001..2ef7ba685 100644 --- a/server/typescript/packages/sdk/src/agent-context/scaffold.ts +++ b/server/typescript/packages/sdk/src/agent-context/scaffold.ts @@ -1,5 +1,8 @@ import { createHash } from "node:crypto"; +import { readdirSync, readFileSync, statSync } from "node:fs"; +import { join, relative, sep } from "node:path"; import type { AssembledFile, Stack } from "./types.js"; +import { docsBlobUrl, docsRefForVersion } from "@metaobjectsdev/metadata"; /** Consumer-relative path of the sidecar manifest that tracks scaffolded files. */ export const AGENT_CONTEXT_MANIFEST_PATH = ".metaobjects/.agent-context.json"; @@ -13,6 +16,14 @@ export interface Manifest { * back-compat with manifests written before version tracking existed. */ generatedBy?: string; + /** + * Hash of the agent-context content the scaffolding CLI shipped (see + * {@link hashAgentContextRoot}). The staleness check compares THIS, so a content change + * shipped without a version change is noticed, and a version change that changed no + * content is not a reason to nudge. Optional: manifests written before it fall back to + * comparing `generatedBy`. + */ + contentHash?: string; servers: string[]; clients: string[]; /** consumer-relative path → sha256 of the contents as last scaffolded. */ @@ -23,6 +34,24 @@ export function hashContents(s: string): string { return createHash("sha256").update(s, "utf8").digest("hex"); } +/** + * A stable hash of the agent-context content tree a release ships, plus the docs ref its + * links are pinned to (the scaffolded text embeds it, so a different ref is different + * content). Independent of the project's stack and of directory order. + */ +export function hashAgentContextRoot(root: string, docsRef: string): string { + const entries: string[] = []; + const walk = (dir: string): void => { + for (const name of readdirSync(dir).sort()) { + const abs = join(dir, name); + if (statSync(abs).isDirectory()) walk(abs); + else entries.push(`${relative(root, abs).split(sep).join("/")}\0${hashContents(readFileSync(abs, "utf8"))}`); + } + }; + walk(root); + return hashContents(`${docsRef}\n${entries.sort().join("\n")}`); +} + export interface ScaffoldDecision { /** files to (over)write at their own path: new, or unmodified-since-last-scaffold. */ writes: { path: string; contents: string }[]; @@ -62,8 +91,10 @@ export function planScaffold(opts: { readCurrent: (path: string) => string | undefined; /** The MetaObjects version doing the scaffold — stamped into the manifest. */ generatedBy: string; + /** {@link hashAgentContextRoot} of the content being scaffolded — stamped into the manifest. */ + contentHash?: string; }): ScaffoldDecision { - const { stack, assembled, prior, readCurrent, generatedBy } = opts; + const { stack, assembled, prior, readCurrent, generatedBy, contentHash } = opts; const writes: ScaffoldDecision["writes"] = []; const conflicts: ScaffoldDecision["conflicts"] = []; const files: Record = {}; @@ -119,7 +150,7 @@ export function planScaffold(opts: { conflicts, // `files` holds only what this run assembled, so a pruned orphan does not survive into // the new manifest — otherwise it would be re-reported as an orphan on every run. - manifest: { version: 1, generatedBy, servers: stack.servers, clients: stack.clients, files }, + manifest: { version: 1, generatedBy, ...(contentHash !== undefined && { contentHash }), servers: stack.servers, clients: stack.clients, files }, prunes, removed, }; @@ -179,6 +210,46 @@ function contextIsAheadOfInstall(generatedBy: string | undefined, currentVersion return stamped[0] > installed[0] || (stamped[0] === installed[0] && stamped[1] > installed[1]); } +/** + * One row per minor (or major) step that has an upgrade guide. `from` and `to` are + * `major.minor` lines; `path` is repo-relative. Adding the next step's guide is one row + * here, nothing else. A patch-level step needs no row: its guide is the within-line one + * the always-on context already names. + */ +export const MINOR_STEP_GUIDES: readonly { from: string; to: string; path: string }[] = [ + { from: "1.0", to: "1.1", path: "docs/features/migrations/upgrading-1.0-to-1.1.md" }, +]; + +function releaseLine(version: string | undefined): [number, number] | null { + if (version === undefined) return null; + const m = /^(\d+)\.(\d+)\./.exec(version.trim()); + return m === null ? null : [Number(m[1]), Number(m[2])]; +} + +function lineAtMost(a: [number, number], b: [number, number]): boolean { + return a[0] < b[0] || (a[0] === b[0] && a[1] <= b[1]); +} + +/** + * The guides for every minor/major step between the stamped version and the installed + * one: a step counts when it starts at or after the stamped line and ends at or before the + * installed line. Empty for a patch-level difference or a version that cannot be read. + */ +function minorStepGuides(generatedBy: string | undefined, currentVersion: string): string[] { + const stamped = releaseLine(generatedBy); + const installed = releaseLine(currentVersion); + if (stamped === null || installed === null) return []; + const ref = docsRefForVersion(currentVersion); + const out: string[] = []; + for (const g of MINOR_STEP_GUIDES) { + const from = releaseLine(`${g.from}.`); + const to = releaseLine(`${g.to}.`); + if (from === null || to === null) continue; + if (lineAtMost(stamped, from) && lineAtMost(to, installed)) out.push(`${g.from} to ${g.to}: ${docsBlobUrl(g.path, ref)}`); + } + return out; +} + /** * A one-line nudge if the scaffolded agent context predates the installed MetaObjects * (so `gen`/`verify` can remind the user to refresh the skills after an upgrade), or @@ -188,17 +259,27 @@ function contextIsAheadOfInstall(generatedBy: string | undefined, currentVersion export function agentContextStaleness(opts: { manifest: Manifest | undefined; currentVersion: string; + /** {@link hashAgentContextRoot} of the installed content; when given and the manifest + * carries one, the two hashes decide instead of the versions. */ + currentContentHash?: string; }): string | null { - const { manifest, currentVersion } = opts; + const { manifest, currentVersion, currentContentHash } = opts; if (manifest === undefined) return null; // no agent context here → nothing to nudge // Exact-equality FIRST: ANY drift nudges (a re-scaffold is cheap + idempotent). A // prerelease/build-metadata difference is still a reason to refresh, so this is not a // semver compare — see releaseSeries() for the ONE case that is exempt. - if (manifest.generatedBy === currentVersion) return null; // in sync + const hashed = manifest.contentHash !== undefined && currentContentHash !== undefined; + // The shipped content decides when both sides recorded it: equal content has nothing to + // refresh whatever the versions say (and a content change that rode on an unchanged version + // is no longer missed). + if (hashed && manifest.contentHash === currentContentHash) return null; + if (!hashed && manifest.generatedBy === currentVersion) return null; // in sync if (contextIsAheadOfInstall(manifest.generatedBy, currentVersion)) return null; const from = manifest.generatedBy ?? "an older MetaObjects"; + const guides = minorStepGuides(manifest.generatedBy, currentVersion); return ( `MetaObjects agent context was generated by ${from}; you're on ${currentVersion}. ` + - `Re-run 'meta init --docs-only --refresh-docs' to refresh the .claude/skills docs.` + `Re-run 'meta init --docs-only --refresh-docs' to refresh the .claude/skills docs.` + + (guides.length > 0 ? ` This is a minor upgrade: read the upgrade guide first (${guides.join("; ")}).` : "") ); } diff --git a/server/typescript/packages/sdk/src/collection.ts b/server/typescript/packages/sdk/src/collection.ts index 0ebcd91d3..e2ea6d919 100644 --- a/server/typescript/packages/sdk/src/collection.ts +++ b/server/typescript/packages/sdk/src/collection.ts @@ -113,6 +113,13 @@ export interface Collection { * `compileScope`/`matchesScope` stay exported for the conformance corpus. */ readonly inScope: (fqn: string) => boolean; + /** + * Requirement-coverage filter: only the dependency-import rule of `inScope`. + * An entity from a dependency's snapshot is not this project's to claim unless + * `scope.include` names its package; the project's own `scope` patterns narrow + * codegen output and never the coverage denominator (T18 ruling, all ports). + */ + readonly inCoverage: (fqn: string) => boolean; /** Output filter for migrate/verify --db (`migrate.scope`). Undefined => the * command governs everything loaded, and that undefined is load-bearing: it * is what leaves the expected schema untouched (migrate-ts `scope.ts`). */ @@ -369,6 +376,10 @@ export async function resolveCollection( const ownFiles = ownSources.map((s) => s.file); const scope = compileScope(toScope(scopeSpec)); + const inCoverage = (fqn: string): boolean => { + const pkg = packageOfResolutionKey(fqn); + return !importedPackages.has(pkg) || explicitlyIncludes(scopeSpec?.include, pkg); + }; const migrateScope = migrateSpec === undefined ? undefined : compileScope({ include: migrateSpec }); const declaredMigrateScope = @@ -400,11 +411,8 @@ export async function resolveCollection( // (DESIGN §11.1 item 2): a dependency's node survives only when the // project's own `scope.include` NAMES its package. A wildcard reaches it // and does not name it — see `explicitlyIncludes`. - inScope: (fqn: string): boolean => { - if (!matchesScope(fqn, scope)) return false; - const pkg = packageOfResolutionKey(fqn); - return !importedPackages.has(pkg) || explicitlyIncludes(scopeSpec?.include, pkg); - }, + inScope: (fqn: string): boolean => matchesScope(fqn, scope) && inCoverage(fqn), + inCoverage, // Undefined ONLY when the project declares no `migrate.scope` AND imports // nothing — migrate-ts reads that undefined as "govern everything loaded", // and it is what leaves the expected schema untouched. With a dependency diff --git a/server/typescript/packages/sdk/src/config.ts b/server/typescript/packages/sdk/src/config.ts index bc86a5df3..f6c27ff14 100644 --- a/server/typescript/packages/sdk/src/config.ts +++ b/server/typescript/packages/sdk/src/config.ts @@ -217,12 +217,61 @@ export const DEFAULT_CONFIG: Config = ConfigSchema.parse({ schema_version: 1 }); * declarations of the same string). */ export const CONFIG_FILE = "config.json"; +/** Where an unknown-key warning goes. Replaceable so a host can route or capture it. */ +let configWarningSink: (message: string) => void = (m) => { process.stderr.write(`${m}\n`); }; +const warnedKeys = new Set(); + +/** Route unknown-key warnings elsewhere (tests, a host with its own logger). Returns the previous sink. */ +export function setConfigWarningSink(sink: (message: string) => void): (message: string) => void { + const previous = configWarningSink; + configWarningSink = sink; + return previous; +} + +/** + * Read `.metaobjects/config.json`. + * + * FORWARD-TOLERANT at the top level: a key this version does not know is a WARNING, and the + * key is ignored. The file is shared by every port and by every version in a mixed estate, so + * a newer MetaObjects adds keys (1.1 added `naming`) that an older reader must not refuse — + * a 1.0 reader's `.strict()` did exactly that, and the whole load failed on a key that was + * perfectly valid. A key this version DOES know is still validated strictly, as is everything + * nested inside it: a typo in `migrate.d1.bindingg` stays an error. The trade is that a + * misspelled TOP-LEVEL key (`scopes` for `scope`) now warns rather than fails; the warning + * names it. {@link ConfigSchema} itself stays strict, so `saveConfig` never writes an unknown key. + */ export async function loadConfig(metaRoot: string): Promise { const path = join(metaRoot, CONFIG_FILE); const raw = await readFile(path, "utf8"); - const result = ConfigSchema.safeParse(JSON.parse(raw)); - if (result.success) return result.data; - throw new Error(describeConfigError(path, result.error)); + const doc: unknown = JSON.parse(raw); + const { known, unknownKeys } = splitUnknownTopLevelKeys(doc); + const result = ConfigSchema.safeParse(known); + if (!result.success) throw new Error(describeConfigError(path, result.error)); + for (const key of unknownKeys) { + const id = `${path}\u0000${key}`; + if (warnedKeys.has(id)) continue; + warnedKeys.add(id); + configWarningSink( + `warning: ${path}: unknown top-level key "${key}" ignored. A newer MetaObjects may have written it; ` + + `upgrade this tool, or remove the key if it is a misspelling. Known keys: ${CONFIG_TOP_LEVEL_KEYS.join(", ")}.`, + ); + } + return result.data; +} + +/** The top-level keys this version reads. */ +export const CONFIG_TOP_LEVEL_KEYS: readonly string[] = Object.keys(ConfigSchema.shape); + +/** Split a parsed config document into the part this version validates and the keys it does not know. */ +export function splitUnknownTopLevelKeys(doc: unknown): { known: unknown; unknownKeys: string[] } { + if (typeof doc !== "object" || doc === null || Array.isArray(doc)) return { known: doc, unknownKeys: [] }; + const known: Record = {}; + const unknownKeys: string[] = []; + for (const [k, v] of Object.entries(doc)) { + if (CONFIG_TOP_LEVEL_KEYS.includes(k)) known[k] = v; + else unknownKeys.push(k); + } + return { known, unknownKeys }; } /** diff --git a/server/typescript/packages/sdk/src/index.ts b/server/typescript/packages/sdk/src/index.ts index 7e662ea2f..adc2b17c9 100644 --- a/server/typescript/packages/sdk/src/index.ts +++ b/server/typescript/packages/sdk/src/index.ts @@ -16,7 +16,7 @@ // shows `resolveCollection`, the documented single authority for locating a project. // Config -export { ConfigSchema, DEFAULT_CONFIG, loadConfig, saveConfig, AllowTokenEnum, physicalNamingFromConfig } from "./config.js"; +export { ConfigSchema, DEFAULT_CONFIG, loadConfig, setConfigWarningSink, splitUnknownTopLevelKeys, CONFIG_TOP_LEVEL_KEYS, saveConfig, AllowTokenEnum, physicalNamingFromConfig } from "./config.js"; export type { Config } from "./config.js"; // Metadata dependencies (FR-023) — the `dependencies` config key's schema, diff --git a/server/typescript/packages/sdk/test/agent-context/assemble.test.ts b/server/typescript/packages/sdk/test/agent-context/assemble.test.ts index d7e094451..6b4265ac1 100644 --- a/server/typescript/packages/sdk/test/agent-context/assemble.test.ts +++ b/server/typescript/packages/sdk/test/agent-context/assemble.test.ts @@ -12,6 +12,20 @@ function paths(files: { path: string }[]): string[] { return files.map((f) => f.path).sort(); } +describe("assemble pins guide links", () => { + const stack = makeStack(["typescript"], ["react"]); + test("by default the links keep the authored ref (main)", () => { + const all = assemble({ contentRoot: CONTENT_ROOT, stack }).map((f) => f.contents).join("\n"); + expect(all).toContain("metaobjects/blob/main/docs/"); + }); + test("with docsRef every repository blob link names that ref and none is left on main", () => { + const files = assemble({ contentRoot: CONTENT_ROOT, stack, docsRef: "v1.1.0" }); + const all = files.map((f) => f.contents).join("\n"); + expect(all).toContain("metaobjects/blob/v1.1.0/docs/features/migrations/"); + expect(all).not.toContain("metaobjects/blob/main/"); + }); +}); + describe("assemble", () => { test("emits AGENTS.md + CLAUDE.md with the stack line + codegen command substituted", () => { const stack = makeStack(["typescript"], ["react"]); diff --git a/server/typescript/packages/sdk/test/agent-context/scaffold.test.ts b/server/typescript/packages/sdk/test/agent-context/scaffold.test.ts index ba2d8a5d0..d5bb00786 100644 --- a/server/typescript/packages/sdk/test/agent-context/scaffold.test.ts +++ b/server/typescript/packages/sdk/test/agent-context/scaffold.test.ts @@ -1,5 +1,8 @@ +import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; import { test, expect, describe } from "bun:test"; -import { hashContents, planScaffold, agentContextStaleness, AGENT_CONTEXT_MANIFEST_PATH, type Manifest } from "../../src/agent-context/scaffold.js"; +import { hashContents, hashAgentContextRoot, planScaffold, agentContextStaleness, AGENT_CONTEXT_MANIFEST_PATH, type Manifest } from "../../src/agent-context/scaffold.js"; import { makeStack } from "../../src/agent-context/resolve.js"; import type { AssembledFile } from "../../src/agent-context/types.js"; @@ -129,6 +132,19 @@ describe("agentContextStaleness", () => { expect(w).toContain("0.9.0"); expect(w).toContain("refresh-docs"); }); + test("a MINOR step names its upgrade guide, pinned to the installed release's tag", () => { + const w = agentContextStaleness({ manifest: m("1.0.13"), currentVersion: "1.1.0" })!; + expect(w).toContain("refresh-docs"); + expect(w).toContain("https://github.com/metaobjectsdev/metaobjects/blob/v1.1.0/docs/features/migrations/upgrading-1.0-to-1.1.md"); + }); + test("a prerelease install links the guide on main (no tag exists)", () => { + const w = agentContextStaleness({ manifest: m("1.0.13"), currentVersion: "1.1.0-rc.7" })!; + expect(w).toContain("/blob/main/docs/features/migrations/upgrading-1.0-to-1.1.md"); + }); + test("a PATCH step names no minor guide", () => { + expect(agentContextStaleness({ manifest: m("1.0.12"), currentVersion: "1.0.13" })!).not.toContain("upgrading-1.0-to-1.1"); + expect(agentContextStaleness({ manifest: m("1.1.0"), currentVersion: "1.1.1" })!).not.toContain("upgrading-1.0-to-1.1"); + }); test("legacy manifest without generatedBy → warning (treated as older)", () => { const w = agentContextStaleness({ manifest: m(undefined), currentVersion: "0.9.0" }); expect(w).toContain("0.9.0"); @@ -225,3 +241,51 @@ describe("planScaffold — the manifest never records content it declined to wri expect(d.manifest.files[".metaobjects/AGENTS.md"]).toBe(hashContents("fresh v3")); }); }); + +// ── content hash (release review MINOR 8): the shipped content decides, not the stamped version ── +describe("agentContextStaleness by content hash", () => { + const mh = (generatedBy: string, contentHash?: string): Manifest => ({ + version: 1, generatedBy, ...(contentHash !== undefined && { contentHash }), servers: ["typescript"], clients: [], files: {}, + }); + + test("same version, changed shipped content → nudges (a content change shipped without a version bump)", () => { + const w = agentContextStaleness({ manifest: mh("1.1.0", "aaa"), currentVersion: "1.1.0", currentContentHash: "bbb" }); + expect(w).toContain("refresh-docs"); + }); + test("different version, identical shipped content → silent (nothing to refresh)", () => { + expect(agentContextStaleness({ manifest: mh("1.1.0", "aaa"), currentVersion: "1.1.1", currentContentHash: "aaa" })).toBeNull(); + }); + test("a minor step with changed content still names the upgrade guide", () => { + const w = agentContextStaleness({ manifest: mh("1.0.13", "aaa"), currentVersion: "1.1.0", currentContentHash: "bbb" })!; + expect(w).toContain("upgrading-1.0-to-1.1"); + }); + test("a context from a NEWER release stays silent despite a different hash (#347)", () => { + expect(agentContextStaleness({ manifest: mh("0.24.7", "aaa"), currentVersion: "0.24.4", currentContentHash: "bbb" })).toBeNull(); + }); + test("a legacy manifest without a hash falls back to comparing versions", () => { + expect(agentContextStaleness({ manifest: mh("1.1.0"), currentVersion: "1.1.0", currentContentHash: "bbb" })).toBeNull(); + expect(agentContextStaleness({ manifest: mh("1.0.9"), currentVersion: "1.1.0", currentContentHash: "bbb" })).not.toBeNull(); + }); + test("planScaffold stamps the content hash it is given", () => { + const d = planScaffold({ + stack: { servers: [], clients: [], concerns: [], tokens: new Set() }, assembled: [], prior: undefined, + readCurrent: () => undefined, generatedBy: "1.1.0", contentHash: "abc", + }); + expect(d.manifest.contentHash).toBe("abc"); + }); +}); + +describe("hashAgentContextRoot", () => { + test("changes with a file's content and with the docs ref, not with directory order", () => { + const root = mkdtempSync(join(tmpdir(), "ac-hash-")); + mkdirSync(join(root, "skills"), { recursive: true }); + writeFileSync(join(root, "skills", "a.md"), "one"); + writeFileSync(join(root, "b.md"), "two"); + const h1 = hashAgentContextRoot(root, "v1.1.0"); + expect(hashAgentContextRoot(root, "v1.1.0")).toBe(h1); + expect(hashAgentContextRoot(root, "main")).not.toBe(h1); + writeFileSync(join(root, "b.md"), "three"); + expect(hashAgentContextRoot(root, "v1.1.0")).not.toBe(h1); + rmSync(root, { recursive: true, force: true }); + }); +}); diff --git a/server/typescript/packages/sdk/test/config.test.ts b/server/typescript/packages/sdk/test/config.test.ts index e51e46982..7826d100f 100644 --- a/server/typescript/packages/sdk/test/config.test.ts +++ b/server/typescript/packages/sdk/test/config.test.ts @@ -2,7 +2,7 @@ import { describe, test, expect, beforeEach, afterEach } from "bun:test"; import { mkdtempSync, rmSync, mkdirSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { ConfigSchema, DEFAULT_CONFIG, loadConfig, saveConfig } from "../src/config.js"; +import { ConfigSchema, DEFAULT_CONFIG, loadConfig, saveConfig, setConfigWarningSink, type Config } from "../src/config.js"; let metaRoot: string; beforeEach(() => { @@ -73,6 +73,45 @@ describe("loadConfig — an invalid config says what is wrong, in words", () => }); }); +describe("loadConfig — forward-tolerant at the top level (a newer MetaObjects may have written the key)", () => { + async function load(doc: unknown): Promise<{ result?: Config; error?: Error; warnings: string[] }> { + const warnings: string[] = []; + const prev = setConfigWarningSink((m) => warnings.push(m)); + try { + await Bun.write(join(metaRoot, "config.json"), JSON.stringify(doc)); + return { result: await loadConfig(metaRoot), warnings }; + } catch (e) { + return { error: e as Error, warnings }; + } finally { + setConfigWarningSink(prev); + } + } + + test("an unknown top-level key warns, naming it and saying a newer MetaObjects may have written it", async () => { + const r = await load({ schema_version: 1, fromTheFuture: { x: 1 }, migrate: { dialect: "sqlite" } }); + expect(r.error).toBeUndefined(); + expect(r.result!.migrate?.dialect).toBe("sqlite"); + expect(r.warnings).toHaveLength(1); + expect(r.warnings[0]).toContain('"fromTheFuture"'); + expect(r.warnings[0]).toContain("newer MetaObjects"); + expect((r.result as Record).fromTheFuture).toBeUndefined(); + }); + + test("a type error on a KNOWN key stays fatal", async () => { + const r = await load({ schema_version: 1, pending_in_git: "yes", fromTheFuture: 1 }); + expect(r.error?.message).toContain("pending_in_git"); + }); + + test("a typo INSIDE a known block stays fatal", async () => { + const r = await load({ schema_version: 1, migrate: { d1: { bindingg: "DB" } } }); + expect(r.error).toBeDefined(); + }); + + test("the schema itself is still strict, so saveConfig never writes an unknown key", () => { + expect(() => ConfigSchema.parse({ schema_version: 1, foo: 1 })).toThrow(); + }); +}); + describe("loadConfig / saveConfig", () => { test("round-trips defaults", async () => { await saveConfig(metaRoot, DEFAULT_CONFIG); diff --git a/server/typescript/packages/test-generators/src/generators/routes.ts b/server/typescript/packages/test-generators/src/generators/routes.ts index c6f8a3d89..d9cc73bbf 100644 --- a/server/typescript/packages/test-generators/src/generators/routes.ts +++ b/server/typescript/packages/test-generators/src/generators/routes.ts @@ -68,7 +68,8 @@ import { isWriteThrough, isReport, itemRouteField, - reportDecimalColumns, + decimalColumnsLine, + decimalFieldNames, servesReadApi, formatTs, entityOutputPath, @@ -154,8 +155,8 @@ function renderRoutes( // The mount addresses `id` by default. A projection keyed on another field names it // (the view's key for that column, which is the field name), so `GET /:id` reads the // same column the by-id query does. Absent for `id`, which keeps that output's bytes. - // A decimal a SQLite report computes is a REAL; the mount sends it as the string it is elsewhere. - const decimalColumns = reportDecimalColumns(entity, ctx.dialect); + // A decimal is a canonical string on the wire on every dialect (ADR-0019 Amendment 1). + const decimalColumns = decimalFieldNames(entity); const keylessOpts = (indent: string): string => (idField !== undefined && idField !== DEFAULT_ID_FIELD ? `\n${indent}idColumn: ${JSON.stringify(idField)},` @@ -297,7 +298,7 @@ export async function ${handlerName}(fastify: ${FastifyInstanceSym}) { updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${exposeLinePrefixed} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ")}${exposeLinePrefixed} }); ${m2mMountsPrefixed} }, { prefix: ${JSON.stringify(ctx.apiPrefix)} }); } @@ -322,7 +323,7 @@ export async function ${handlerName}(fastify: ${FastifyInstanceSym}) { updateSchema: ${entityName}UpdateSchema, filterAllowlist: ${entityName}FilterAllowlist, sortAllowlist: ${entityName}SortAllowlist, - dialect: ${JSON.stringify(ctx.dialect)},${exposeLineFlat} + dialect: ${JSON.stringify(ctx.dialect)},${decimalColumnsLine(entity, " ")}${exposeLineFlat} }); ${m2mMountsFlat}} `; @@ -478,7 +479,7 @@ function renderM2mMount( sourceColumn: ${sourceColumn}, targetColumn: ${targetColumn}, targetPkColumn: ${targetPkColumn}, - symmetric: ${entry.symmetric ? "true" : "false"},${discriminatorLine}${sourceDiscriminatorLine} + symmetric: ${entry.symmetric ? "true" : "false"},${decimalColumnsLine(entry.target, " ")}${discriminatorLine}${sourceDiscriminatorLine} });`; } @@ -579,7 +580,7 @@ function renderTphRoutes( updateSchema: ${baseUpdateSym}, filterAllowlist: ${baseFilterSym}, sortAllowlist: ${baseSortSym}, - dialect: ${dialectLit},${polymorphicExposeLine} + dialect: ${dialectLit},${decimalColumnsLine([base, ...plan.subtypes.map((st) => st.entity)], " ")}${polymorphicExposeLine} });`; // FR-018 x FR-017 — M:N traversal inside a TPH hierarchy. This file never consulted @@ -622,7 +623,7 @@ function renderTphRoutes( updateSchema: ${subUpdateSym}, filterAllowlist: ${subFilterSym}, sortAllowlist: ${subSortSym}, - dialect: ${dialectLit}, + dialect: ${dialectLit},${decimalColumnsLine(sub, " ")} discriminator: { column: ${JSON.stringify(discField)}, value: ${JSON.stringify(value)} },${exposeLine(expose, " ")} });`; // This subtype's own M:N navigations, mounted beneath its segment and gated on the diff --git a/server/typescript/packages/test-generators/src/templates/decimal-wire-helper.ts b/server/typescript/packages/test-generators/src/templates/decimal-wire-helper.ts new file mode 100644 index 000000000..c2b3a8422 --- /dev/null +++ b/server/typescript/packages/test-generators/src/templates/decimal-wire-helper.ts @@ -0,0 +1,47 @@ +// The wire rule for a field.decimal WRITE (ADR-0019 Amendment 1, W2 and W3), as text a generated +// model module carries as a local function. Generated model code imports nothing from +// MetaObjects, so the rule is emitted, not imported; runtime-ts/src/decimal-wire.ts holds the +// same grammar and both are held to fixtures/decimal-wire/vectors.json. + +export const DECIMAL_WRITE_HELPER = "decimalWrite"; + +/** The Zod expression of a decimal WRITE field: a string in the JSON-number grammar or a JSON + * number, rewritten to the canonical string, refused when the column cannot hold it (W3). */ +export function zodDecimalWriteExpr(precision: number | undefined, scale: number | undefined): string { + const args = precision === undefined ? "" : `, ${precision}, ${scale ?? 0}`; + return `z.union([z.string(), z.number()]).transform((v, ctx) => ${DECIMAL_WRITE_HELPER}(v, ctx${args}))`; +} + +export const DECIMAL_WRITE_HELPER_DECL = [ + "/** A field.decimal on the wire: a string in the JSON-number grammar (an exponent is allowed) or a JSON", + " * number, rewritten to its canonical string (plain notation, no trailing fraction zeros, zero as \"0\").", + " * Garbage, or a value the column cannot hold without rounding, is a validation issue. Generated. */", + `function ${DECIMAL_WRITE_HELPER}(v: string | number, ctx: z.RefinementCtx, precision?: number, scale = 0): string {`, + ' const text = typeof v === "number" ? (Number.isFinite(v) ? v.toPrecision(15) : "") : v;', + " const m = /^(-?)(0|[1-9][0-9]*)(?:\\.([0-9]+))?(?:[eE]([+-]?[0-9]+))?$/.exec(text);", + " const e = m === null || m[4] === undefined ? 0 : Number(m[4]);", + " if (m === null || !Number.isSafeInteger(e) || Math.abs(e) > 1000) {", + ' ctx.addIssue({ code: "custom", message: "Not a decimal number" });', + " return z.NEVER;", + " }", + ' const whole = m[2] ?? "";', + ' let digits = whole + (m[3] ?? "");', + " let point = whole.length + e;", + ' const lead = digits.length - digits.replace(/^0+/, "").length;', + " digits = digits.slice(lead).replace(/0+$/, \"\");", + " point -= lead;", + ' if (digits.length === 0) return "0";', + " const plain = point <= 0", + ' ? `0.${"0".repeat(-point)}${digits}`', + " : point >= digits.length", + ' ? digits + "0".repeat(point - digits.length)', + " : `${digits.slice(0, point)}.${digits.slice(point)}`;", + " const fraction = point >= digits.length ? 0 : digits.length - Math.max(point, 0) + Math.max(-point, 0);", + " const integer = Math.max(point, 0);", + " if (plain.length > 1000 || (precision !== undefined && (fraction > scale || integer > precision - scale))) {", + ' ctx.addIssue({ code: "custom", message: "Decimal out of range for the column" });', + " return z.NEVER;", + " }", + ' return m[1] === "-" ? `-${plain}` : plain;', + "}", +].join("\n"); diff --git a/server/typescript/packages/test-generators/src/templates/entity-constants.ts b/server/typescript/packages/test-generators/src/templates/entity-constants.ts index 294191ce4..b6d0ecccb 100644 --- a/server/typescript/packages/test-generators/src/templates/entity-constants.ts +++ b/server/typescript/packages/test-generators/src/templates/entity-constants.ts @@ -74,6 +74,8 @@ function renderFieldEntry(f: UiFieldDescriptor): string { `view: ${JSON.stringify(f.view)}`, ]; if (f.htmlType !== undefined) entries.push(`htmlType: ${JSON.stringify(f.htmlType)}`); + if (f.step !== undefined) entries.push(`step: ${JSON.stringify(f.step)}`); + if (f.inputMode !== undefined) entries.push(`inputMode: ${JSON.stringify(f.inputMode)}`); if (f.rules.length > 0) entries.push(`rules: { ${f.rules.map(renderRule).join(", ")} }`); // A `field.enum`'s member symbols. Emitted whenever the field has them, because the diff --git a/server/typescript/packages/test-generators/src/templates/field-meta.ts b/server/typescript/packages/test-generators/src/templates/field-meta.ts index a6b30a7e1..6182a3130 100644 --- a/server/typescript/packages/test-generators/src/templates/field-meta.ts +++ b/server/typescript/packages/test-generators/src/templates/field-meta.ts @@ -283,3 +283,39 @@ export function humanize(s: string): string { .replace(/\b\w/g, (c) => c.toUpperCase()) .trim(); } + +/** The `` attributes a numeric field subtype needs. */ +export interface NumericInputControl { + readonly htmlType: "number" | "text"; + /** `"any"` for a binary float: the browser's implicit step of 1 refuses `12.5`. */ + readonly step?: "any"; + /** `"decimal"` for an exact decimal rendered as a text input. */ + readonly inputMode?: "decimal"; +} + +/** + * The one answer both form tiers give for a numeric field subtype (the `` meta that + * `useEntityForm` spreads, and the nested sub-form in codegen-ts-react), or undefined for a + * non-numeric subtype. + * + * `` has an implicit step of 1, so native validation blocks `12.5`. + * double/float therefore get `step="any"`. A decimal is an exact value that travels as a + * string: a number input would parse it to a double for validation and min/max (wrong past + * 15 digits), so it is `type="text" inputmode="decimal"` and RHF/Zod own the validation. + * int, long and currency keep step 1. + */ +export function numericInputControl(subType: string): NumericInputControl | undefined { + switch (subType) { + case FIELD_SUBTYPE_INT: + case FIELD_SUBTYPE_LONG: + case FIELD_SUBTYPE_CURRENCY: + return { htmlType: "number" }; + case FIELD_SUBTYPE_DOUBLE: + case FIELD_SUBTYPE_FLOAT: + return { htmlType: "number", step: "any" }; + case FIELD_SUBTYPE_DECIMAL: + return { htmlType: "text", inputMode: "decimal" }; + default: + return undefined; + } +} diff --git a/server/typescript/packages/test-generators/src/templates/filter-allowlist.ts b/server/typescript/packages/test-generators/src/templates/filter-allowlist.ts index 0812a0bb4..4a0cd087f 100644 --- a/server/typescript/packages/test-generators/src/templates/filter-allowlist.ts +++ b/server/typescript/packages/test-generators/src/templates/filter-allowlist.ts @@ -69,6 +69,7 @@ const VALUE_FORMAT_SUBTYPES = new Set([ FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_UUID, + FIELD_SUBTYPE_DECIMAL, ]); /** Subtypes whose values are whole numbers — `field.currency` is integer minor units. */ diff --git a/server/typescript/packages/test-generators/src/templates/zod-validators.ts b/server/typescript/packages/test-generators/src/templates/zod-validators.ts index 113186b89..643429fac 100644 --- a/server/typescript/packages/test-generators/src/templates/zod-validators.ts +++ b/server/typescript/packages/test-generators/src/templates/zod-validators.ts @@ -13,7 +13,7 @@ import { code, joinCode, imp, type Code } from "@metaobjectsdev/codegen-ts/engin import { MetaObject, MetaField, type MetaRoot, stripPackage } from "@metaobjectsdev/metadata"; import { FIELD_SUBTYPE_STRING, FIELD_SUBTYPE_INT, FIELD_SUBTYPE_LONG, FIELD_SUBTYPE_CURRENCY, - FIELD_SUBTYPE_BOOLEAN, FIELD_SUBTYPE_DOUBLE, FIELD_SUBTYPE_FLOAT, + FIELD_SUBTYPE_BOOLEAN, FIELD_SUBTYPE_DECIMAL, FIELD_ATTR_PRECISION, FIELD_ATTR_SCALE, FIELD_SUBTYPE_DOUBLE, FIELD_SUBTYPE_FLOAT, FIELD_SUBTYPE_DATE, FIELD_SUBTYPE_TIME, FIELD_SUBTYPE_TIMESTAMP, FIELD_SUBTYPE_ENUM, FIELD_SUBTYPE_OBJECT, FIELD_SUBTYPE_MAP, FIELD_SUBTYPE_UUID, FIELD_SUBTYPE_URI, FIELD_SUBTYPE_INET, @@ -43,6 +43,7 @@ import { ZOD_TIMESTAMP_UTC_EXPR, } from "./date-time-regex.js"; import { renderDocsFor } from "./jsdoc.js"; +import { DECIMAL_WRITE_HELPER_DECL, zodDecimalWriteExpr } from "./decimal-wire-helper.js"; import { sharedEnumForField } from "@metaobjectsdev/codegen-ts/engine"; import { sharedEnumImportSpecifier } from "@metaobjectsdev/codegen-ts/engine"; import { sharedEnumZodConstName } from "./enums-file.js"; @@ -74,6 +75,13 @@ function utcTimestampHelperSince(since: number): string { return utcTimestampUses > since ? `${UTC_TIMESTAMP_HELPER_DECL}\n\n` : ""; } +// The same device for the decimal write rule (decimal-wire-helper.ts). +let decimalWriteUses = 0; + +function decimalHelperSince(since: number): string { + return decimalWriteUses > since ? `${DECIMAL_WRITE_HELPER_DECL}\n\n` : ""; +} + /** True when this object declares at least one @autoSet timestamp field * (onCreate or onUpdate). Drives whether codegen emits the #203 @@ -259,6 +267,7 @@ function isInsertRequiredPk(field: MetaField, assignedPk: Set): boolean export function renderInsertSchemaOnly(obj: MetaObject, ctx?: RenderContext): Code { const z = imp("z@zod"); const utcUsesBefore = utcTimestampUses; + const decimalUsesBefore = decimalWriteUses; const autoGenPkFields = autoGenPkFieldNames(obj); const assignedPkFields = assignedPkFieldNames(obj); const tphPin = tphDiscriminatorPin(obj); @@ -310,7 +319,7 @@ export function renderInsertSchemaOnly(obj: MetaObject, ctx?: RenderContext): Co const docsPrefix = docs ? `${docs}\n` : ""; return code` -${utcTimestampHelperSince(utcUsesBefore)}${docsPrefix}export const ${insertSchemaName} = ${z}.object({ +${utcTimestampHelperSince(utcUsesBefore)}${decimalHelperSince(decimalUsesBefore)}${docsPrefix}export const ${insertSchemaName} = ${z}.object({ ${joinCode(insertFieldLines, { on: ",\n" })} }); `; @@ -418,6 +427,7 @@ export function updateSchemaFields(obj: MetaObject): SchemaFieldShape[] { export function renderZodValidators(obj: MetaObject, ctx?: RenderContext): Code { const z = imp("z@zod"); const utcUsesBefore = utcTimestampUses; + const decimalUsesBefore = decimalWriteUses; const autoGenPkFields = autoGenPkFieldNames(obj); const pkFields = primaryKeyFieldNames(obj); const assignedPkFields = assignedPkFieldNames(obj); @@ -540,7 +550,7 @@ export type ${objName}CreatePreserving = ${z}.input Date: Wed, 21 Oct 2026 13:05:11 -0400 Subject: [PATCH 2/4] no-mistakes(review): Canonicalize decimal arrays element-wise; document fractional validator bounds --- CHANGELOG.md | 13 +++-- .../migrations/upgrading-1.0-to-1.1.md | 8 +++ .../codegen/runtime/decimal-wire.ts | 6 ++- .../showcase/codegen/runtime/decimal-wire.ts | 6 ++- .../packages/codegen-ts/src/api-surface.ts | 5 +- .../codegen-ts/test/decimal-wire-e2e.test.ts | 26 ++++++++-- .../packages/runtime-ts/src/decimal-wire.ts | 6 ++- .../packages/runtime-ts/src/type-coercer.ts | 40 ++++++++++---- .../runtime-ts/test/type-coercer.test.ts | 52 ++++++++++++++++++- 9 files changed, 135 insertions(+), 27 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5d405a93d..a2a363391 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -859,10 +859,12 @@ generator catalog and the Atlas evaluation are 1.2 work. that is not numeric, is `ERR_INVALID_MEASURE`. A fractional `@default` (`0.5`) is `ERR_BAD_ATTR_VALUE` on the measure in every port: TypeScript says "@default '0.5' is not an integer. A measure's @default is a whole number (for example 0).", while C#, Java and Python - refuse it through their existing integer-attribute check with that check's message. No port - changed its general `attr.int` check, so a fraction in a validator's `@min` or `@max` loads - where it did. `@default: 0.0` (or `1.0`) loads as that integer in every port, and a - non-number `@default` reports that one error (both under **Fixed**, with fixtures). + refuse it through their integer-attribute check with that check's message. That check now + refuses a fraction in every `attr.int` and `attr.long` in TypeScript, C# and Python, as Java + already did, so a fractional `validator.numeric` `@min` or `@max` (`0.5`) is + `ERR_BAD_ATTR_VALUE` too (see the upgrade guide). `@default: 0.0` (or `1.0`) loads as that + integer in every port, and a non-number `@default` reports that one error (both under + **Fixed**, with fixtures). TypeScript lowers it: the view selects `FROM` the spine entity's table with `LEFT OUTER JOIN`s back to `@from`, and a defaulted measure is `COALESCE(…, n)` (SQLite writes `n.0` for a `decimal`, `double` or `float` measure and for a ratio, so the column keeps one storage class; @@ -1494,7 +1496,8 @@ until you regenerate. on one is `400 invalid_filter_field` and a sort `400 invalid_sort`. A Java report map dimension keeps its `@valueType`. - **An integer attribute written `4.0` loads as `4`**, and a fraction is - `ERR_BAD_ATTR_VALUE`, in every port (a requirement's `level`, a measure's `@default`). A + `ERR_BAD_ATTR_VALUE`, in every port (a requirement's `level`, a measure's `@default`, a + `validator.numeric` `@min` or `@max`). A bad `@default` reports that one error, not a second rule error beside it. - **A requirement's package is its own, else the nearest enclosing declaration, else the file's**, and an overlay that declares a package gives it to the children it adds. Coverage diff --git a/docs/features/migrations/upgrading-1.0-to-1.1.md b/docs/features/migrations/upgrading-1.0-to-1.1.md index a6392b1bd..44957ac42 100644 --- a/docs/features/migrations/upgrading-1.0-to-1.1.md +++ b/docs/features/migrations/upgrading-1.0-to-1.1.md @@ -141,6 +141,14 @@ rather than rounded. A decimal filter operand is compared exactly; a bad one is `meta migrate` warns for one declared with `@precision` above 15. An owned routes generator keeps its 1.0 decimal handling until you resync it (step 6). +### Fractional validator bounds (TypeScript, C#, Python) + +An integer attribute (`attr.int`, `attr.long`) no longer accepts a fraction. A +`validator.numeric` declares `@min` and `@max` as integers, so a bound such as `@min: 0.5` on a +`decimal` or `double` field is now `ERR_BAD_ATTR_VALUE` at load in TypeScript, C# and Python. +Java already refused it, so the five ports now agree. Write an integer bound (`@min: 0`). A +whole number written `1.0` still loads, as `1`. + ## Diffs to expect after regenerating - **References are written fully qualified (ADR-0059).** `meta export` and canonical diff --git a/examples/advanced-modeling/codegen/runtime/decimal-wire.ts b/examples/advanced-modeling/codegen/runtime/decimal-wire.ts index fab6c35d5..c7afadf45 100644 --- a/examples/advanced-modeling/codegen/runtime/decimal-wire.ts +++ b/examples/advanced-modeling/codegen/runtime/decimal-wire.ts @@ -99,10 +99,12 @@ export function decimalFits(canonical: string, precision: number | undefined, sc /** * W1/W4 on a value a driver hands back for a decimal column: a string (Postgres `numeric`, MySQL * `DECIMAL`, a SQLite text value) is canonicalized, a number (a SQLite or D1 REAL or INTEGER) is - * rendered at 15 significant digits and canonicalized, a bigint is its digits. Null passes - * through. A value in none of those shapes is left for the caller to see. + * rendered at 15 significant digits and canonicalized, a bigint is its digits, and an array (a + * `field.decimal` declared `isArray`) is rewritten element by element. Null passes through. A + * value in none of those shapes is left for the caller to see. */ export function decimalToWire(value: unknown): unknown { + if (Array.isArray(value)) return value.map(decimalToWire); if (typeof value === "string") return canonicalDecimal(value) ?? value; if (typeof value === "number") return decimalFromDouble15(value) ?? value; if (typeof value === "bigint") return value.toString(); diff --git a/examples/showcase/codegen/runtime/decimal-wire.ts b/examples/showcase/codegen/runtime/decimal-wire.ts index fab6c35d5..c7afadf45 100644 --- a/examples/showcase/codegen/runtime/decimal-wire.ts +++ b/examples/showcase/codegen/runtime/decimal-wire.ts @@ -99,10 +99,12 @@ export function decimalFits(canonical: string, precision: number | undefined, sc /** * W1/W4 on a value a driver hands back for a decimal column: a string (Postgres `numeric`, MySQL * `DECIMAL`, a SQLite text value) is canonicalized, a number (a SQLite or D1 REAL or INTEGER) is - * rendered at 15 significant digits and canonicalized, a bigint is its digits. Null passes - * through. A value in none of those shapes is left for the caller to see. + * rendered at 15 significant digits and canonicalized, a bigint is its digits, and an array (a + * `field.decimal` declared `isArray`) is rewritten element by element. Null passes through. A + * value in none of those shapes is left for the caller to see. */ export function decimalToWire(value: unknown): unknown { + if (Array.isArray(value)) return value.map(decimalToWire); if (typeof value === "string") return canonicalDecimal(value) ?? value; if (typeof value === "number") return decimalFromDouble15(value) ?? value; if (typeof value === "bigint") return value.toString(); diff --git a/server/typescript/packages/codegen-ts/src/api-surface.ts b/server/typescript/packages/codegen-ts/src/api-surface.ts index c2a918a56..c779dbf63 100644 --- a/server/typescript/packages/codegen-ts/src/api-surface.ts +++ b/server/typescript/packages/codegen-ts/src/api-surface.ts @@ -122,12 +122,13 @@ export function servesClientTier(entity: MetaObject): boolean { * the number a NUMERIC-affinity column stores). The generated route passes these names to the * mount as `decimalColumns` and the mount rewrites each row under them. Which keys are decimals * comes from the model, never from the shape of a value. Empty for an object with no decimal, - * which keeps its route byte-identical. An array of decimals is not a decimal column. + * which keeps its route byte-identical. A decimal declared `isArray` is listed too: the mount + * rewrites each element of its array. */ export function decimalFieldNames(entity: MetaObject): readonly string[] { // ADR-0039: resolving. A decimal may be inherited via extends, and a report read model's // derived ratio, average or sum is a field of the report itself. - return entity.fields().filter((f) => f.subType === FIELD_SUBTYPE_DECIMAL && !f.resolvedIsArray()).map((f) => f.name); + return entity.fields().filter((f) => f.subType === FIELD_SUBTYPE_DECIMAL).map((f) => f.name); } /** `\ndecimalColumns: [...],` for a mount option list, or "" when the object has no decimal. diff --git a/server/typescript/packages/codegen-ts/test/decimal-wire-e2e.test.ts b/server/typescript/packages/codegen-ts/test/decimal-wire-e2e.test.ts index 3a4eaed31..9f711b33c 100644 --- a/server/typescript/packages/codegen-ts/test/decimal-wire-e2e.test.ts +++ b/server/typescript/packages/codegen-ts/test/decimal-wire-e2e.test.ts @@ -28,6 +28,7 @@ const MODEL = { { "field.string": { name: "bucket", "@required": true, "@filterable": true } }, { "field.decimal": { name: "amount", "@precision": 9, "@scale": 4, "@filterable": true, "@sortable": true } }, { "field.decimal": { name: "weight", "@required": true } }, + { "field.decimal": { name: "readings", isArray: true, "@precision": 9, "@scale": 4 } }, { "identity.primary": { name: "pk", "@fields": "id", "@generation": "increment" } }, ]}}, ]}, @@ -38,6 +39,7 @@ const entries = sqliteTable("entries", { bucket: text("bucket").notNull(), amount: numeric("amount"), weight: numeric("weight").notNull(), + readings: text("readings", { mode: "json" }), }); const dir = mkdtempSync(join(import.meta.dir, "tmp-decimal-")); @@ -52,6 +54,7 @@ async function execute(name: string, source: string): Promise { const { root, errors } = await new MetaDataLoader({ strict: true }).load([ new InMemoryStringSource(JSON.stringify(MODEL), { id: "ledger.json" }), @@ -62,8 +65,8 @@ beforeAll(async () => { const schemas = await execute("schemas", moduleText); const allowlists = await execute("allowlist", renderFilterAllowlist(entry).toString()); - const sqlite = new Database(":memory:"); - sqlite.run("CREATE TABLE entries (id INTEGER PRIMARY KEY AUTOINCREMENT, bucket TEXT NOT NULL, amount NUMERIC, weight NUMERIC NOT NULL)"); + sqlite = new Database(":memory:"); + sqlite.run("CREATE TABLE entries (id INTEGER PRIMARY KEY AUTOINCREMENT, bucket TEXT NOT NULL, amount NUMERIC, weight NUMERIC NOT NULL, readings TEXT)"); app = new Hono(); mountCrudRoutes({ app, path: "/entries", db: drizzle(sqlite), table: entries, @@ -85,8 +88,8 @@ describe("generated decimal write schema and route, SQLite", () => { }); test("the route is given every decimal field name", () => { - expect(decimalFieldNames(entry)).toEqual(["amount", "weight"]); - expect(decimalColumnsLine(entry, " ")).toBe('\n decimalColumns: ["amount","weight"],'); + expect(decimalFieldNames(entry)).toEqual(["amount", "weight", "readings"]); + expect(decimalColumnsLine(entry, " ")).toBe('\n decimalColumns: ["amount","weight","readings"],'); }); test("a string and a number are accepted, and answered as the canonical string", async () => { @@ -116,4 +119,19 @@ describe("generated decimal write schema and route, SQLite", () => { const gt = (await (await app.request("/entries?filter[bucket][eq]=a&filter[amount][gt]=15")).json()) as Array<{ amount: unknown }>; expect(gt.map((r) => r.amount)).toEqual(["100.25"]); }); + + test("a decimal array is written and read element by element", async () => { + const res = await post({ bucket: "arr", weight: "1", readings: ["100.2500", 0.3, "1.5e1"] }); + expect(res.status).toBe(201); + expect(((await res.json()) as { readings: unknown }).readings).toEqual(["100.25", "0.3", "15"]); + for (const readings of [["0.12345"], ["12abc"]]) { + expect((await post({ bucket: "arr", weight: "1", readings })).status).toBe(400); + } + }); + + test("a decimal array the database holds in another spelling is answered canonical", async () => { + sqlite.run(`INSERT INTO entries (bucket, weight, readings) VALUES ('raw', 1, '["100.5000", 0.30000000000000004, "-0.0"]')`); + const rows = (await (await app.request("/entries?filter[bucket][eq]=raw")).json()) as Array<{ readings: unknown }>; + expect(rows.map((r) => r.readings)).toEqual([["100.5", "0.3", "0"]]); + }); }); diff --git a/server/typescript/packages/runtime-ts/src/decimal-wire.ts b/server/typescript/packages/runtime-ts/src/decimal-wire.ts index fab6c35d5..c7afadf45 100644 --- a/server/typescript/packages/runtime-ts/src/decimal-wire.ts +++ b/server/typescript/packages/runtime-ts/src/decimal-wire.ts @@ -99,10 +99,12 @@ export function decimalFits(canonical: string, precision: number | undefined, sc /** * W1/W4 on a value a driver hands back for a decimal column: a string (Postgres `numeric`, MySQL * `DECIMAL`, a SQLite text value) is canonicalized, a number (a SQLite or D1 REAL or INTEGER) is - * rendered at 15 significant digits and canonicalized, a bigint is its digits. Null passes - * through. A value in none of those shapes is left for the caller to see. + * rendered at 15 significant digits and canonicalized, a bigint is its digits, and an array (a + * `field.decimal` declared `isArray`) is rewritten element by element. Null passes through. A + * value in none of those shapes is left for the caller to see. */ export function decimalToWire(value: unknown): unknown { + if (Array.isArray(value)) return value.map(decimalToWire); if (typeof value === "string") return canonicalDecimal(value) ?? value; if (typeof value === "number") return decimalFromDouble15(value) ?? value; if (typeof value === "bigint") return value.toString(); diff --git a/server/typescript/packages/runtime-ts/src/type-coercer.ts b/server/typescript/packages/runtime-ts/src/type-coercer.ts index c3c684b9e..caa77dad2 100644 --- a/server/typescript/packages/runtime-ts/src/type-coercer.ts +++ b/server/typescript/packages/runtime-ts/src/type-coercer.ts @@ -29,12 +29,14 @@ import { FIELD_ATTR_STORAGE, FIELD_ATTR_DB_COLUMN_TYPE, FIELD_ATTR_INT_VALUE_MAP, + FIELD_ATTR_PRECISION, + FIELD_ATTR_SCALE, STORAGE_JSONB, STORAGE_FLATTENED, DB_COLUMN_TYPE_JSONB, } from "@metaobjectsdev/metadata"; import type { Dialect, Row } from "./persistence-driver.js"; -import { decimalToWire, parseDecimalInput } from "./decimal-wire.js"; +import { decimalFits, decimalToWire, parseDecimalInput } from "./decimal-wire.js"; export function coerceRowOnRead(entity: MetaData, row: Row, dialect: Dialect): Row { const hydrated = parseJsonTextArrays(entity, deserializeJsonbObjectFields(entity, row)); @@ -136,11 +138,11 @@ function encodeIntBackedEnums(entity: MetaData, row: Row): Row { } export function coerceRowOnWrite(entity: MetaData, row: Row, dialect: Dialect): Row { - const jsonbColumned = serializeJsonbColumns(entity, row); + const jsonbColumned = serializeJsonbColumns(entity, bindDecimals(entity, row)); // Dialect-independent: an int-backed enum's column is INTEGER on every dialect // (SQLite has one integer storage class), so the symbol→int encode is not gated // on the dialect the way the boolean mapping below is. - const encoded = encodeIntBackedEnums(entity, bindDecimals(entity, jsonbColumned)); + const encoded = encodeIntBackedEnums(entity, jsonbColumned); if (dialect === "mysql") return mapBooleansToInt(entity, isoTimestampsToDates(entity, encoded)); if (dialect !== "sqlite") return encoded; return mapBooleansToInt(entity, encoded); @@ -307,21 +309,41 @@ function canonicalizeDecimals(entity: MetaData, row: Row): Row { } /** - * Write side: a `field.decimal` value is bound as its exact canonical string. A string in the - * JSON-number grammar or a number is accepted; anything else is refused here rather than - * reaching a column that would round it, reject it with a driver error, or store garbage. + * Write side: a `field.decimal` value is bound as its exact canonical string, and a decimal + * declared `isArray` as an array of them. A string in the JSON-number grammar or a number is + * accepted; anything else, or a value the declared `@precision` / `@scale` cannot hold (W3), is + * refused here rather than reaching a column that would round it, reject it with a driver error, + * or store garbage. */ function bindDecimals(entity: MetaData, row: Row): Row { let out: Row | null = null; + // ADR-0039: resolving — a decimal, its isArray and its @precision / @scale may be inherited. for (const child of entity.children()) { if (child.type !== TYPE_FIELD || child.subType !== FIELD_SUBTYPE_DECIMAL) continue; if (!(child.name in row)) continue; const v = row[child.name]; if (v === null || v === undefined) continue; - const canonical = parseDecimalInput(v); - if (canonical === undefined) { - throw new Error(`field.decimal '${child.name}' cannot be written: ${JSON.stringify(v)} is not a decimal number (a string in the JSON-number grammar, or a number)`); + const precision = child.attr(FIELD_ATTR_PRECISION); + const scale = child.attr(FIELD_ATTR_SCALE); + const bind = (value: unknown): string => { + const canonical = parseDecimalInput(value); + if (canonical === undefined) { + throw new Error(`field.decimal '${child.name}' cannot be written: ${JSON.stringify(value)} is not a decimal number (a string in the JSON-number grammar, or a number)`); + } + if (!decimalFits(canonical, typeof precision === "number" ? precision : undefined, typeof scale === "number" ? scale : undefined)) { + throw new Error(`field.decimal '${child.name}' cannot be written: ${canonical} does not fit @precision ${String(precision)} / @scale ${String(scale ?? 0)}`); + } + return canonical; + }; + if (child.resolvedIsArray()) { + if (!Array.isArray(v)) { + throw new Error(`field.decimal '${child.name}' cannot be written: ${JSON.stringify(v)} is not an array of decimal numbers`); + } + out ??= { ...row }; + out[child.name] = v.map((element) => (element === null ? null : bind(element))); + continue; } + const canonical = bind(v); if (canonical === v) continue; out ??= { ...row }; out[child.name] = canonical; diff --git a/server/typescript/packages/runtime-ts/test/type-coercer.test.ts b/server/typescript/packages/runtime-ts/test/type-coercer.test.ts index b143cccfa..871af0ade 100644 --- a/server/typescript/packages/runtime-ts/test/type-coercer.test.ts +++ b/server/typescript/packages/runtime-ts/test/type-coercer.test.ts @@ -2,7 +2,7 @@ import { describe, test, expect } from "bun:test"; import type { MetaData } from "@metaobjectsdev/metadata"; import { TypeId, TYPE_OBJECT, TYPE_FIELD, FIELD_SUBTYPE_BOOLEAN, FIELD_SUBTYPE_LONG, FIELD_SUBTYPE_STRING, - OBJECT_SUBTYPE_ENTITY } from "@metaobjectsdev/metadata"; + OBJECT_SUBTYPE_ENTITY, MetaDataLoader, InMemoryStringSource } from "@metaobjectsdev/metadata"; import { meta } from "./_meta-build.js"; import { coerceRowOnRead, coerceRowOnWrite } from "../src/type-coercer.js"; @@ -78,3 +78,53 @@ describe("non-boolean fields untouched", () => { expect(coerceRowOnWrite(e, row, "sqlite")).toEqual(row); }); }); + +// ADR-0019 Amendment 1 for a `field.decimal` declared `isArray` (a Postgres `numeric[]`): every +// element is a canonical string on read and on write, and an element the column cannot hold is +// refused (W3), as C#'s DecimalListJsonConverter, Python's to_wire and Java's contentUsing do. +describe("field.decimal isArray", () => { + async function ledger(): Promise { + const model = { "metadata.root": { package: "ledger", children: [ + { "object.entity": { name: "Entry", children: [ + { "source.rdb": { "@table": "entries" } }, + { "field.long": { name: "id" } }, + { "field.decimal": { name: "readings", isArray: true, "@precision": 9, "@scale": 4 } }, + { "field.decimal": { name: "amount", "@precision": 9, "@scale": 4 } }, + { "identity.primary": { name: "pk", "@fields": "id" } }, + ]}}, + ]}}; + const { root, errors } = await new MetaDataLoader({ strict: true }).load([ + new InMemoryStringSource(JSON.stringify(model), { id: "ledger.json" }), + ]); + expect(errors).toEqual([]); + return root.objects().find((o) => o.name === "Entry") as MetaData; + } + + test("read: the column's scale is stripped from each element (100.5000 -> 100.5)", async () => { + const e = await ledger(); + expect(coerceRowOnRead(e, { readings: ["100.5000", "0.0000", "-2.2500", null] }, "postgres").readings) + .toEqual(["100.5", "0", "-2.25", null]); + expect(coerceRowOnRead(e, { readings: "[0.30000000000000004, 12]" }, "sqlite").readings).toEqual(["0.3", "12"]); + }); + + test("write: each element is bound as its canonical string", async () => { + const e = await ledger(); + expect(coerceRowOnWrite(e, { readings: ["100.5000", 2.5, "1.5e1", null] }, "postgres").readings) + .toEqual(["100.5", "2.5", "15", null]); + expect(coerceRowOnWrite(e, { readings: [] }, "postgres").readings).toEqual([]); + }); + + test("write: an element that is not a number, or that the column cannot hold, is refused", async () => { + const e = await ledger(); + expect(() => coerceRowOnWrite(e, { readings: ["1", "12abc"] }, "postgres")).toThrow("is not a decimal number"); + expect(() => coerceRowOnWrite(e, { readings: ["0.12345"] }, "postgres")).toThrow("does not fit"); + expect(() => coerceRowOnWrite(e, { readings: ["100000"] }, "postgres")).toThrow("does not fit"); + expect(() => coerceRowOnWrite(e, { readings: "100.5" }, "postgres")).toThrow("is not an array"); + }); + + test("write: a scalar decimal the column cannot hold is refused too", async () => { + const e = await ledger(); + expect(coerceRowOnWrite(e, { amount: "12345.1000" }, "postgres").amount).toBe("12345.1"); + expect(() => coerceRowOnWrite(e, { amount: "0.12345" }, "postgres")).toThrow("does not fit"); + }); +}); From 532c123bbf7f63f6139b6ae05165cfb6f0362941 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 21 Oct 2026 13:14:09 -0400 Subject: [PATCH 3/4] no-mistakes(review): Coerce and range-check decimal array elements in Python and Kotlin --- .../kotlin/KotlinSpringControllerGenerator.kt | 33 ++++++--- .../generator/kotlin/KotlinDecimalWireTest.kt | 73 +++++++++++++++++++ .../codegen/runtime/decimal_wire.py | 18 ++++- .../python/tests/runtime/test_decimal_wire.py | 13 ++++ 4 files changed, 124 insertions(+), 13 deletions(-) diff --git a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinSpringControllerGenerator.kt b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinSpringControllerGenerator.kt index b0a6b35e6..8002e2c97 100644 --- a/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinSpringControllerGenerator.kt +++ b/server/java/codegen-kotlin/src/main/kotlin/com/metaobjects/generator/kotlin/KotlinSpringControllerGenerator.kt @@ -1234,11 +1234,19 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase, value: String) { - if (field !is DecimalField || field.isArrayType) return + if (field !is DecimalField) return + val outOfRange = if (field.isArrayType) { + "$value?.any { !${decimalFitsExpr(field, "it")} } == true" + } else { + "!${decimalFitsExpr(field, value)}" + } append(indent) - append("if (!${decimalFitsExpr(field, value)}) return@transaction ") + append("if ($outOfRange) return@transaction ") append("ResponseEntity.badRequest().body(mapOf(\"error\" to \"validation\", \"field\" to \"${field.name}\") as Any)\n") } @@ -1252,15 +1260,20 @@ open class KotlinSpringControllerGenerator : MultiFileDirectGeneratorBase, fn: String, ktType: String): String = - if (field is DecimalField && !field.isArrayType) { - "runCatching { MetaDecimalWire.fromNode(body.get(\"$fn\")) }" + + if (field is DecimalField) { + val read = if (field.isArrayType) { + "body.get(\"$fn\").also { require(it.isArray) }.map { MetaDecimalWire.fromNode(it) }" + } else { + "MetaDecimalWire.fromNode(body.get(\"$fn\"))" + } + "runCatching { $read }" + ".getOrElse { return@transaction ResponseEntity.badRequest().body(mapOf(\"error\" to \"validation\", \"field\" to \"$fn\") as Any) }" } else { "objectMapper.treeToValue(body.get(\"$fn\"), object : TypeReference<$ktType>() {})" diff --git a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinDecimalWireTest.kt b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinDecimalWireTest.kt index 934a30c1a..f1b759667 100644 --- a/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinDecimalWireTest.kt +++ b/server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinDecimalWireTest.kt @@ -131,6 +131,79 @@ class KotlinDecimalWireTest { assertTrue("\"field\" to \"price\"" in src, src) } + @Test + fun `a decimal array is read and range-checked element by element on create and on PATCH (W2, W3)`() { + val json = model.replace( + "{ \"field.string\": { \"name\": \"name\", \"@maxLength\": 40 } },", + "{ \"field.decimal\": { \"name\": \"readings\", \"isArray\": true, \"@precision\": 9, \"@scale\": 4 } },", + ) + val files = emit(json, all()) + val controller = files.getValue("acme/shop/ItemController.kt").lines().map { it.trim() } + fun line(prefix: String): String = + controller.firstOrNull { it.startsWith(prefix) } ?: error("no generated line starting '$prefix':\n" + controller.joinToString("\n")) + // The generated create range check and the generated PATCH bind + range check, run as written. + val harness = """ + package acme.shop + + import com.fasterxml.jackson.databind.JsonNode + import java.math.BigDecimal + + class ResponseEntity(val status: Int, val body: T?) { + class Builder { fun body(b: Any): ResponseEntity = ResponseEntity(400, b) } + companion object { fun badRequest() = Builder() } + } + fun transaction(block: () -> T): T = block() + class Dto(val readings: List?) + + object Harness { + @JvmStatic fun create(dto: Dto): ResponseEntity = transaction { + ${line("if (dto.readings")} + ResponseEntity(201, dto.readings?.map { MetaDecimalWire.format(it) }) + } + @JvmStatic fun patch(body: JsonNode): ResponseEntity = transaction { + ${line("val hasReadings")} + ${line("val nullReadings")} + ${line("val vReadings")} + ${line("if (vReadings")} + ResponseEntity(200, vReadings?.map { MetaDecimalWire.format(it) }) + } + } + """.trimIndent() + val sources = files.filterKeys { it.endsWith("MetaDecimalWire.kt") }.map { (path, src) -> + com.tschuchort.compiletesting.SourceFile.kotlin(path.substringAfterLast('/'), src) + } + com.tschuchort.compiletesting.SourceFile.kotlin("Harness.kt", harness) + val result = com.tschuchort.compiletesting.KotlinCompilation().apply { + this.sources = sources + inheritClassPath = true + messageOutputStream = System.out + }.compile() + assertEquals(com.tschuchort.compiletesting.KotlinCompilation.ExitCode.OK, result.exitCode, result.messages + "\n" + harness) + val cl = result.classLoader + val h = cl.loadClass("acme.shop.Harness") + val mapper = com.fasterxml.jackson.databind.ObjectMapper() + fun outcome(r: Any?): Pair { + val c = r!!.javaClass + return c.getMethod("getStatus").invoke(r) to c.getMethod("getBody").invoke(r) + } + fun patch(body: String) = outcome(h.getMethod("patch", com.fasterxml.jackson.databind.JsonNode::class.java).invoke(null, mapper.readTree(body))) + fun create(vararg readings: String): Pair { + val dto = cl.loadClass("acme.shop.Dto").getConstructor(List::class.java) + .newInstance(readings.map { java.math.BigDecimal(it) }) + return outcome(h.getMethod("create", dto.javaClass).invoke(null, dto)) + } + val refused = mapOf("error" to "validation", "field" to "readings") + + // Oracle: ADR-0019 Amendment 1 — canonical strings (W1), exact input (W2), decimal(9, 4) range (W3). + assertEquals(200 to listOf("100.5", "0.3", "15"), patch("""{"readings": ["100.5000", 0.3, "1.5e1"]}""")) + assertEquals(400 to refused, patch("""{"readings": ["1", "0.12345"]}""")) + assertEquals(400 to refused, patch("""{"readings": ["100000"]}""")) + assertEquals(400 to refused, patch("""{"readings": ["12abc"]}""")) + assertEquals(400 to refused, patch("""{"readings": "100.5"}""")) + assertEquals(201 to listOf("100.5", "0"), create("100.5000", "0.0000")) + assertEquals(400 to refused, create("1", "0.12345")) + assertEquals(400 to refused, create("100000")) + } + @Test fun `a TPH base and its subtype enforce the declared range on their per-subtype writes`() { val src = emit(tphModel, all()).getValue("acme/shop/ChargeController.kt") diff --git a/server/python/src/metaobjects/codegen/runtime/decimal_wire.py b/server/python/src/metaobjects/codegen/runtime/decimal_wire.py index c686945ec..b9d55c71e 100644 --- a/server/python/src/metaobjects/codegen/runtime/decimal_wire.py +++ b/server/python/src/metaobjects/codegen/runtime/decimal_wire.py @@ -131,19 +131,31 @@ def coerce_decimal_fields( ) -> dict[str, Any]: """A copy of *dto* with each present decimal field parsed under W2 (``null`` stays ``None``) and held to its column's range under W3 (*limits*: name to (precision, scale)). + A list (a decimal declared ``isArray``) is parsed and range-checked element by element. Raises ``ValueError`` naming the field when one is malformed or out of range.""" out = dict(dto) for name in decimal_fields: if name in out and out[name] is not None: + limit = limits.get(name) if limits else None try: - out[name] = parse_decimal(out[name]) - if limits and name in limits: - check_decimal_range(out[name], *limits[name]) + value = out[name] + out[name] = ( + [None if v is None else _coerce_decimal(v, limit) for v in value] + if isinstance(value, list) + else _coerce_decimal(value, limit) + ) except ValueError as exc: raise ValueError(f"{name}: {exc}") from None return out +def _coerce_decimal(value: Any, limit: tuple[int, int] | None) -> Decimal: + parsed = parse_decimal(value) + if limit is not None: + check_decimal_range(parsed, *limit) + return parsed + + def coerce_decimal_predicates( predicates: Iterable[Any], decimal_fields: frozenset[str] ) -> tuple[list[Any], dict[str, str] | None]: diff --git a/server/python/tests/runtime/test_decimal_wire.py b/server/python/tests/runtime/test_decimal_wire.py index a23ec5c86..2dd52e024 100644 --- a/server/python/tests/runtime/test_decimal_wire.py +++ b/server/python/tests/runtime/test_decimal_wire.py @@ -130,3 +130,16 @@ def test_w3_range_is_enforced_and_nothing_is_rounded(text: str, ok: bool) -> Non else: with pytest.raises(ValueError): coerce_decimal_fields({"amount": text}, fields, limits) + + +def test_a_decimal_array_is_coerced_and_range_checked_element_by_element() -> None: + # ADR-0019 Amendment 1 over a ``field.decimal`` declared ``isArray``: each element is read + # exactly (W2), held to the column's range (W3), and leaves as its canonical string (W1). + fields = frozenset({"readings"}) + limits = {"readings": (9, 4)} + coerced = coerce_decimal_fields({"readings": ["100.5000", 2.5, "1.5e1", None]}, fields, limits) + assert to_wire(coerced) == {"readings": ["100.5", "2.5", "15", None]} + assert coerce_decimal_fields({"readings": []}, fields, limits) == {"readings": []} + for bad in (["1", "0.12345"], ["100000"], ["1", "12abc"], [True]): + with pytest.raises(ValueError, match="readings"): + coerce_decimal_fields({"readings": bad}, fields, limits) From d01433d6fda91f6d9027e96ff5127861108bf52e Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 21 Oct 2026 14:19:00 -0400 Subject: [PATCH 4/4] no-mistakes(document): Fixed stale decimal-array and eject docs. Review-2 CHANGELOG and upgrade-guide fixes were already present in the diff. Edited the reporting skill decimal line (source plus five fixture copies, which mirror it) to the canonical-string rule, and added the companion-file clause to the own-your-codegen eject --list bullet. --- .../skills/metaobjects-authoring/references/reporting.md | 2 +- docs/features/own-your-codegen.md | 6 ++++-- .../skills/metaobjects-authoring/references/reporting.md | 2 +- .../skills/metaobjects-authoring/references/reporting.md | 2 +- .../skills/metaobjects-authoring/references/reporting.md | 2 +- .../skills/metaobjects-authoring/references/reporting.md | 2 +- .../skills/metaobjects-authoring/references/reporting.md | 2 +- 7 files changed, 10 insertions(+), 8 deletions(-) diff --git a/agent-context/skills/metaobjects-authoring/references/reporting.md b/agent-context/skills/metaobjects-authoring/references/reporting.md index 31489594b..b8547d3fc 100644 --- a/agent-context/skills/metaobjects-authoring/references/reporting.md +++ b/agent-context/skills/metaobjects-authoring/references/reporting.md @@ -166,7 +166,7 @@ A concrete report whose read source is `@kind: view` is served by every port's g - `` is the report name, snake_cased then pluralized, like any object: `InvoicesByMonth` is `/invoices_by_months`, `StoreTotals` is `/store_totals`. There is no report-specific spelling. - **Every derived field with filter operators is filterable and sortable**, dimension and measure alike, with its type's own operators. You cannot narrow that set: a report has no field to put `@filterable` on. A field of the `@from` entity that the report does not expose is refused with a `400`. - A time dimension at `day` or coarser is `YYYY-MM-DD`; a null measure is `null` with the key present (a measure with `@default` is never null); `currency` is integer minor units. -- **A decimal (`avg`, a ratio, a `sum` of a decimal) has no agreed JSON spelling**: each port sends its own (TypeScript a string). Do not compare one across backends as text. +- **A decimal (`avg`, a ratio, a `sum` of a decimal) is one canonical string in every port** (ADR-0019 Amendment 1: `12.5`, never `12.50` or an exponent). Display formatting is a view's job. Do not parse it as a float to compare. - `meta gen` writes a model page for every report (`docsFile()`; a sourceless one is marked "Not served") and an API page for a served one (`apiDocsFile()`) — each an ordinary entry in `generators`. The files each port generates are in the `metaobjects-codegen` skill's reference for that language ("Reports"). diff --git a/docs/features/own-your-codegen.md b/docs/features/own-your-codegen.md index dd422c441..77922d8ac 100644 --- a/docs/features/own-your-codegen.md +++ b/docs/features/own-your-codegen.md @@ -126,8 +126,10 @@ is touched, so "what did upstream change?" is a diff of two files whose body is - **`meta eject --list` compares each copy** against what the installed packages would write today: `identical`, `same content, your formatting`, `DIFFERS: N line(s) only in the package, M only in yours`, or `not in the package`, with the `diff -u` command for each - that differs. Take an upgrade by merging the package's change into your copy; the two - files differ by their import lines alone. + that differs. The comparison covers the companion files a generator copies with it (its + template modules, Java's support classes, the Kotlin decimal wire), not only the generator + file. Take an upgrade by merging the package's change into your copy; the two files differ + by their import lines alone. - **A copy from before 1.1 gets no template modules it would never call.** It imports the renderers from the package, so copying them beside it would be dead code. `meta eject --force` takes the new shape; to keep your changes, merge them into the new file. diff --git a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/reporting.md b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/reporting.md index 31489594b..b8547d3fc 100644 --- a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/reporting.md +++ b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/reporting.md @@ -166,7 +166,7 @@ A concrete report whose read source is `@kind: view` is served by every port's g - `` is the report name, snake_cased then pluralized, like any object: `InvoicesByMonth` is `/invoices_by_months`, `StoreTotals` is `/store_totals`. There is no report-specific spelling. - **Every derived field with filter operators is filterable and sortable**, dimension and measure alike, with its type's own operators. You cannot narrow that set: a report has no field to put `@filterable` on. A field of the `@from` entity that the report does not expose is refused with a `400`. - A time dimension at `day` or coarser is `YYYY-MM-DD`; a null measure is `null` with the key present (a measure with `@default` is never null); `currency` is integer minor units. -- **A decimal (`avg`, a ratio, a `sum` of a decimal) has no agreed JSON spelling**: each port sends its own (TypeScript a string). Do not compare one across backends as text. +- **A decimal (`avg`, a ratio, a `sum` of a decimal) is one canonical string in every port** (ADR-0019 Amendment 1: `12.5`, never `12.50` or an exponent). Display formatting is a view's job. Do not parse it as a float to compare. - `meta gen` writes a model page for every report (`docsFile()`; a sourceless one is marked "Not served") and an API page for a served one (`apiDocsFile()`) — each an ordinary entry in `generators`. The files each port generates are in the `metaobjects-codegen` skill's reference for that language ("Reports"). diff --git a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-authoring/references/reporting.md b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-authoring/references/reporting.md index 31489594b..b8547d3fc 100644 --- a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-authoring/references/reporting.md +++ b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-authoring/references/reporting.md @@ -166,7 +166,7 @@ A concrete report whose read source is `@kind: view` is served by every port's g - `` is the report name, snake_cased then pluralized, like any object: `InvoicesByMonth` is `/invoices_by_months`, `StoreTotals` is `/store_totals`. There is no report-specific spelling. - **Every derived field with filter operators is filterable and sortable**, dimension and measure alike, with its type's own operators. You cannot narrow that set: a report has no field to put `@filterable` on. A field of the `@from` entity that the report does not expose is refused with a `400`. - A time dimension at `day` or coarser is `YYYY-MM-DD`; a null measure is `null` with the key present (a measure with `@default` is never null); `currency` is integer minor units. -- **A decimal (`avg`, a ratio, a `sum` of a decimal) has no agreed JSON spelling**: each port sends its own (TypeScript a string). Do not compare one across backends as text. +- **A decimal (`avg`, a ratio, a `sum` of a decimal) is one canonical string in every port** (ADR-0019 Amendment 1: `12.5`, never `12.50` or an exponent). Display formatting is a view's job. Do not parse it as a float to compare. - `meta gen` writes a model page for every report (`docsFile()`; a sourceless one is marked "Not served") and an API page for a served one (`apiDocsFile()`) — each an ordinary entry in `generators`. The files each port generates are in the `metaobjects-codegen` skill's reference for that language ("Reports"). diff --git a/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-authoring/references/reporting.md b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-authoring/references/reporting.md index 31489594b..b8547d3fc 100644 --- a/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-authoring/references/reporting.md +++ b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-authoring/references/reporting.md @@ -166,7 +166,7 @@ A concrete report whose read source is `@kind: view` is served by every port's g - `` is the report name, snake_cased then pluralized, like any object: `InvoicesByMonth` is `/invoices_by_months`, `StoreTotals` is `/store_totals`. There is no report-specific spelling. - **Every derived field with filter operators is filterable and sortable**, dimension and measure alike, with its type's own operators. You cannot narrow that set: a report has no field to put `@filterable` on. A field of the `@from` entity that the report does not expose is refused with a `400`. - A time dimension at `day` or coarser is `YYYY-MM-DD`; a null measure is `null` with the key present (a measure with `@default` is never null); `currency` is integer minor units. -- **A decimal (`avg`, a ratio, a `sum` of a decimal) has no agreed JSON spelling**: each port sends its own (TypeScript a string). Do not compare one across backends as text. +- **A decimal (`avg`, a ratio, a `sum` of a decimal) is one canonical string in every port** (ADR-0019 Amendment 1: `12.5`, never `12.50` or an exponent). Display formatting is a view's job. Do not parse it as a float to compare. - `meta gen` writes a model page for every report (`docsFile()`; a sourceless one is marked "Not served") and an API page for a served one (`apiDocsFile()`) — each an ordinary entry in `generators`. The files each port generates are in the `metaobjects-codegen` skill's reference for that language ("Reports"). diff --git a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/reporting.md b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/reporting.md index 31489594b..b8547d3fc 100644 --- a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/reporting.md +++ b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/reporting.md @@ -166,7 +166,7 @@ A concrete report whose read source is `@kind: view` is served by every port's g - `` is the report name, snake_cased then pluralized, like any object: `InvoicesByMonth` is `/invoices_by_months`, `StoreTotals` is `/store_totals`. There is no report-specific spelling. - **Every derived field with filter operators is filterable and sortable**, dimension and measure alike, with its type's own operators. You cannot narrow that set: a report has no field to put `@filterable` on. A field of the `@from` entity that the report does not expose is refused with a `400`. - A time dimension at `day` or coarser is `YYYY-MM-DD`; a null measure is `null` with the key present (a measure with `@default` is never null); `currency` is integer minor units. -- **A decimal (`avg`, a ratio, a `sum` of a decimal) has no agreed JSON spelling**: each port sends its own (TypeScript a string). Do not compare one across backends as text. +- **A decimal (`avg`, a ratio, a `sum` of a decimal) is one canonical string in every port** (ADR-0019 Amendment 1: `12.5`, never `12.50` or an exponent). Display formatting is a view's job. Do not parse it as a float to compare. - `meta gen` writes a model page for every report (`docsFile()`; a sourceless one is marked "Not served") and an API page for a served one (`apiDocsFile()`) — each an ordinary entry in `generators`. The files each port generates are in the `metaobjects-codegen` skill's reference for that language ("Reports"). diff --git a/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-authoring/references/reporting.md b/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-authoring/references/reporting.md index 31489594b..b8547d3fc 100644 --- a/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-authoring/references/reporting.md +++ b/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-authoring/references/reporting.md @@ -166,7 +166,7 @@ A concrete report whose read source is `@kind: view` is served by every port's g - `` is the report name, snake_cased then pluralized, like any object: `InvoicesByMonth` is `/invoices_by_months`, `StoreTotals` is `/store_totals`. There is no report-specific spelling. - **Every derived field with filter operators is filterable and sortable**, dimension and measure alike, with its type's own operators. You cannot narrow that set: a report has no field to put `@filterable` on. A field of the `@from` entity that the report does not expose is refused with a `400`. - A time dimension at `day` or coarser is `YYYY-MM-DD`; a null measure is `null` with the key present (a measure with `@default` is never null); `currency` is integer minor units. -- **A decimal (`avg`, a ratio, a `sum` of a decimal) has no agreed JSON spelling**: each port sends its own (TypeScript a string). Do not compare one across backends as text. +- **A decimal (`avg`, a ratio, a `sum` of a decimal) is one canonical string in every port** (ADR-0019 Amendment 1: `12.5`, never `12.50` or an exponent). Display formatting is a view's job. Do not parse it as a float to compare. - `meta gen` writes a model page for every report (`docsFile()`; a sourceless one is marked "Not served") and an API page for a served one (`apiDocsFile()`) — each an ordinary entry in `generators`. The files each port generates are in the `metaobjects-codegen` skill's reference for that language ("Reports").