Simplify `fiboa publish` for catalog-driven publication by ivorbosloper · Pull Request #214 · fiboa/cli

ivorbosloper · 2026-08-21T14:51:40Z

We will not merge this branch. It's a working branch to publish many datasets to source-coop and the harmonized data catalog. We will pluck commits from here to smaller PRs

Co-authored-by: Matthias Mohr <matthias@mohr.ws>

# Conflicts: # CHANGELOG.md # pixi.lock

…easing memory requirements for large data sets

`fiboa publish` now only converts, validates, builds PMTiles and writes a collection.json with relative links, file:size/file:checksum (multihash), a web-map-links v1.3.0 `pmtiles` link with `pmtiles:layers` and a `visual` asset. README/LICENSE generation, the data-survey lookup and the S3 upload are gone; catalogs such as fieldsoftheworld/harmonized-field-data-catalog own those. spdx-license-list is only needed by tests and moves to the dev feature. Also: - FiboaBaseConverter: keep the determination:datetime column that `use_variant_as_determination` adds; it was removed again as unlisted (affected dk, hr). - be_vlg: drop plots without a crop code (one in 2023 failed validation), take the determination date from the variant year, add the 2026 edition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…note Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

PRIA's GSAA WFS publishes the crop as free text and no crop code anywhere in the layer, which is why ee has crop:name and no crop:code — hcat:code is derived from the name. It does carry taotletud_maakasutus, the requested land use: arable, permanent grassland, restored grassland, permanent crops, black fallow. Every edition from 2016 has it. The converter had no test either. It has one now, on a hundred features of the 2024 layer as the service answers them. Catalog issue fieldsoftheworld/harmonized-field-data-catalog#17. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG

16 of the 165,244 rows of 2016 share a parcel id, one of them seven times. It is published as parcel_id, with the row index identifying the field in an edition where it repeats — the same shape as fr. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG

The code list is ours, at https://fiboa.org/code/ee/ee.csv: a four-digit number per Estonian crop name, frozen once published. 44 names that appear in the editions were missing from the table and are mapped to HCAT as well. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG

EPSG defines SWEREF99 TM northing first, and Jordbruksverket's shapefile honours that: AXIS["Northing", NORTH] before AXIS["Easting", EAST], with the coordinates written that way. GDAL reads them as written, GeoParquet stores x, y, and everything after that point reads such a file as (lat, lon) — the published Swedish tiles and STAC bbox put the country in the Indian Ocean, at lon 60-69, lat 1-6. The declaration alone is not enough to act on, because most sources in a lat-lon-ordered CRS (EPSG:4258, 4326) still store lon, lat — the Spanish fixture does. So the coordinates have to disagree with the CRS's own area of use as well: only a projected CRS that says northing first, whose x values fall outside its easting range and inside its northing range, is flipped. Checked against every published collection: se is the only one wrong. si and hr are in authority-order CRSs too, but their sources deliver easting first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG

localId is the LPIS parcel reference and repeats where a parcel is declared more than once — 214 of the 1,027,438 rows of 2022, one of them four times — so the 2022 edition would not convert at all. It is published as parcel_id, and the GML's own feature id, which is unique by construction, is the id. The source publishes no area, so the parcels had none: `metrics:area` is computed from the geometry now, as several other converters do. Median parcel 2.7 ha in the 2024 edition. The department's bucket serves GSAA_2022 through GSAA_2024; GSAA_2017 to GSAA_2021 answer 403 (checked 2026-09-13), which is recorded where the variants are built. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG

Carries the converter from split/lv-backfill so the catalog can build the editions; the PR against main holds the same change.

publish-portolan still had the bbox-grid de_sl, which covered part of the state, and no de_sl_block at all. main's versions page through the WFS: 54,038 and 62,500 features, both verified against the live service today.

The Saarland WFS refuses a request without a User-Agent, which the downloader only sends from 0.2.18 (vecorel/cli#30): the same page is 155 kB on 0.2.18 and a FileNotFoundError on 0.2.16. main gets this through #280. Every converter built from this branch now runs under 0.2.18's checks: bounded row drops, required properties from the declared schemas, and the id uniqueness warning.

m-mohr · 2026-09-24T12:04:31Z

Outdated. @ivorbosloper will check whether any changes still need to be migrated.

Retry the service metadata, the layer metadata, the id bounds and every page through one helper (8 attempts, backoff capped at 30 s), and read a joined layer's key field from the layer's own metadata instead of a one-row "where=1=1" probe, which es_ib refuses on its joined views. Ports 1f28b73, a1f4dda, f8aea60 and 545ada0 from #214 onto the window cache scheme of #232. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Read the per-campaign "Hranice užívania" datasets on data.slovensko.sk instead of one rolling download, rename the 2018/PLODINA_NA columns to the current schema, and normalise no-break spaces in crop names so the grassland names match the crop table. Ports f3ce959 from #214. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ivorbosloper · 2026-09-25T18:45:11Z

Checked what still had to be migrated. The converter changes went in as separate PRs (#232–#330). The rest is superseded:

multipart splitting and -N ids: Refactor ID handling #315 and vecorel-cli 0.3, which no longer explode geometries;
Hilbert sorting and the DuckDB converter: vecorel-cli;
the zero ES areas: Calculate missing area by default #277 #323;
the publish.py rewrite: Portolan does this; replacing fiboa publish is Replace fiboa publish with the Portolan workflow, documented for agents #335;
per-file ES conversion (PerFileBaseConverter: convert file by file, then merge #281): ES: convert all of Spain in one run, or per file #336, only needed if the full Spain conversion does not fit in memory.

ivorbosloper and others added 24 commits December 8, 2025 11:16

Implement DuckDB-based converter

f6d9436

Not sure if this is right

6f52d97

Update project

0a854ad

Fix tests

996cc38

Add collection to metadata

51bea87

Add required to arrow definition

2d3fc50

Support for sources with different schemas

852fb6a

Export collection and set compression

3cf3b8d

Add caching and warnings

c524142

Add todos

0d5a9f9

Fix tests

07d7aac

Implement rewrite to correct GeoParquet

78fb00d

Merge branch 'main' into duckdb_converter

dd88ed0

Update fiboa_cli/conversion/duckdb.py

6017ef0

Co-authored-by: Matthias Mohr <matthias@mohr.ws>

Merge branch 'main' into duckdb_converter

993824e

# Conflicts: # CHANGELOG.md # pixi.lock

Update pixi.lock for duckdb

920b39d

Update pixi

5aad9cb

PerFileBaseConverter: per-file migration and merging the result, decr…

892be96

…easing memory requirements for large data sets

Update ES Provinces list

06e47ea

Fix hilbert-sort bug

13aa258

Fail early for debugging

cf85e81

Don't upload part files

cea4333

Keep geo_parquet version in PerFileConverter

4ba30ca

ivorbosloper changed the title ~~Simplify for catalog-driven publication~~ Simplify fiboa publish for catalog-driven publication Aug 21, 2026

ivorbosloper requested a review from m-mohr August 21, 2026 21:44

ivorbosloper and others added 4 commits August 22, 2026 22:33

NL: new PDOK download location, 2026 concept edition; DE-TH: INSPIRE …

f8944b4

…note Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Declare beautifulsoup4 (used by es_pv, es_vc)

3b8d8f1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Drop rows without a required crop:code instead of failing the conversion

d22b603

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Look up the source column when dropping rows without a crop:code

b92251a

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

ivorbosloper and others added 15 commits September 13, 2026 15:13

LV: the yearly campaigns from data.gov.lv (#286)

e892364

Carries the converter from split/lv-backfill so the catalog can build the editions; the PR against main holds the same change.

EC-FR: the 2018 campaign, from EuroCrops (#288)

74269cb

EuroCrops: fetch from Zenodo with requests (#288)

5f080dd

EC-FR: split the multi-part parcels, and share that split (#288, #286)

8ce3908

EC-FR: distinct ids for the parts of a multi-part parcel (#288)

c499e19

DE-SL: the paged converters from main, so Saarland can be built

e3c7a48

publish-portolan still had the bbox-grid de_sl, which covered part of the state, and no de_sl_block at all. main's versions page through the WFS: 54,038 and 62,500 features, both verified against the live service today.

DE-SL: an id that survives the geometry split

f66b9f5

ES-EX, ES-NC: the recintos FEGA publishes (#292)

f1118fc

ES-EX, ES-NC: an id that survives the geometry split (#292)

806cef7

ivorbosloper added the wontfix This will not be worked on label Sep 22, 2026

m-mohr closed this Sep 24, 2026

This was referenced Sep 24, 2026

SK: editions 2018-2026 #326

Merged

EC-BE-VLG, ES and IE-LPIS convert correctly again #327

Merged

REST converters: survive a tired service #328

Merged

Tests: local HCAT fixtures and command-module smoke test #329

Merged

ivorbosloper mentioned this pull request Sep 25, 2026

Replace fiboa publish with the Portolan workflow, documented for agents #335

Open

ivorbosloper deleted the publish-portolan branch September 25, 2026 18:45

Navigation Menu

Simplify `fiboa publish` for catalog-driven publication - #214

Simplify `fiboa publish` for catalog-driven publication#214
ivorbosloper wants to merge 159 commits into
mainfrom
publish-portolan

ivorbosloper commented Aug 21, 2026 •

edited

Loading

Uh oh!

m-mohr commented Sep 24, 2026

Uh oh!

ivorbosloper commented Sep 25, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ivorbosloper commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

m-mohr commented Sep 24, 2026

Uh oh!

ivorbosloper commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ivorbosloper commented Aug 21, 2026 •

edited

Loading

ivorbosloper commented Sep 25, 2026 •

edited

Loading