Skip to content

ES: convert province by province, then merge - #338

Draft
ivorbosloper wants to merge 1 commit into
mainfrom
es-per-file
Draft

ivorbosloper wants to merge 1 commit into
mainfrom
es-per-file

Conversation

@ivorbosloper

Copy link
Copy Markdown
Collaborator

es now converts each of its 50 provincial GeoPackages on its own and merges the parts, because the whole of Spain does not fit in memory in one run. Closes #336.

  • PerFileConverterMixin (fiboa_cli/conversion/per_file.py) runs the normal convert once per source file into a temporary part, then merges the parts with vecorel-cli's DuckDBBaseConverter.merge_parquet, which checks the ids and Hilbert-sorts over the whole set. With a single source file it is plain convert. It replaces the 450-line PerFileBaseConverter of PerFileBaseConverter: convert file by file, then merge #281.
  • The merge runs on a standalone DuckDBBaseConverter rather than mixing that class in, so the converter's own convert chain stays as it is.
  • Full 2025 edition (17.8 million rows): peak RSS 10 GB, against more than 27 GB before the one-run attempt was stopped; the output validates.
  • Not in this PR: in several provinces about 0.1% of the rows repeat an id (the SIGPAC key plus the declaration line). They get the ~n suffix, as before.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@m-mohr

m-mohr commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Does it handle collectio metadata correctly? Especially creating columns for e.g. different data colimng from there? License, variant/state/kanton, etc? Ideally it mimics what vec merge does and can be used as a generic replacement as vec merge in geopandas will be an issue anyway in the long run.

@ivorbosloper

ivorbosloper commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator Author

Good point. There's a bug in vec merge due to the "collection" properties (which are not read correctly). A constant property gets into collection, and merge_parquet keeps only what all parts agree on. So admin_province_code is lost in the full ES run and should be expanded again to a column-property. I'll fix merge_parquet in vecorel-cli to hydrate those per part, as vec merge does; this PR then only bumps the pin.

@ivorbosloper

Copy link
Copy Markdown
Collaborator Author

Fix: vecorel/cli#55. Checked on two real provinces: admin_province_code is a column again.

@ivorbosloper

Copy link
Copy Markdown
Collaborator Author

I'll wait with making this a Non-Draft PR, it needs a vecorel dependency update due to vecorel/cli#55

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ES: convert all of Spain in one run, or per file

2 participants