feat: extract_text, and the five UI gaps behind a working API

[davior]

Closes six gaps found by auditing the code against docs/plan-of-attack.md rather than trusting it. Every one of them sat inside a milestone already marked done, which is why none of them was visible to anyone reading the plan.

extract_text

plan-of-attack.md listed it in M6's job table as though it existed. It never did — no KIND_ constant, no module — so enrichment/source.py raised NoSourceMaterial for every document and summarize/autotag/describe refused a PDF outright.

Format	Reader	Row per
.pdf	pypdfium2	page
.docx	python-docx (paragraphs and table cells)	chunk
.pptx	python-pptx (slides and speaker notes)	slide
.xlsx	openpyxl, read_only=True	worksheet
.txt .md .csv	stdlib decode	chunk

Dependencies. PDF needs nothing new: pypdfium2 is already pinned for first-page thumbnails, and reusing it keeps the licensing argument in ingest/thumbnails.py settled rather than reopening it for pypdf. The three Office libraries are pure-Python wheels whose only compiled transitive dependency is lxml, which ships cp313 wheels — checked on PyPI before adding them, since a source build on the pinned 3.13 is how Pillow and numpy break.

Storage. A new DocumentPage table, not a column on Asset. SQLModel selects every column of every row and the library listing selects assets by the page, so a 200-page PDF's text in an asset column would be read from disk to render a thumbnail grid that never looks at it. One row per natural unit is also the granularity page-level search will want.

The precedence detail worth reviewing. The FROM_DOCUMENT branch in gather sits above the poster branch, not down where the old refusal was. Every PDF gets a first-page thumbnail at ingest, so a branch placed lower would leave a PDF summarised from a picture of its cover — the same mistake the transcript-beats-poster rule at the top of that module exists to prevent, one format over. There is a test for exactly this.

embed is deliberately not chained after extraction: page bodies are not vectorised under this change, so it would queue a job that does no new work. The one-line change is in place and commented for when that lands.

The five UI gaps

Each was a backend endpoint or store action with no component behind it — code passing every test that no user could reach.

• Tag management. M3 built rename, delete and a nested category tree, then wired a picker that can only add. New TagPanel in Settings. tagsApi.recategorise and updateCategory had no store action at all. The two rejections the server bothers to explain — duplicate name, category moved inside itself — are shown rather than swallowed. Also guarded the store's rollback paths, which wrote a pre-await snapshot back with no staleness check, so a failure landing after sign-out repopulated an emptied store with the previous user's vocabulary.
• /a/{id}. GN-4 settles the Notes→GAM asset reference as this path, chosen over a BlockNote shortcode partly because it needed no work in Notes — reasoning that assumed this end existed. /a/anything fell through the catch-all to the library with no sign anything was wrong. openById inserts the fetched asset into the library store, because update, remove, setAssetTags and refreshAsset all map over that list and would otherwise be controls whose every write landed nowhere.
• Search asset_type and limit. Typed by searchApi.run, passed by nothing. Both now survive a reload in the URL.
• 401 mid-session. A response interceptor, injected from App rather than imported (the store imports api/auth.ts, which imports the client). Fires once — the activity store polls every 2–10s — only on 401, never on the CSRF guard's 403, and it returns whether it acted so the ordinary anonymous 401 during bootstrap does not spend the one shot.
• Sign out. signOut() had existed since M2 with no caller outside its own test.

Deliberately not in this change

Recorded in plan-of-attack.md rather than left to be rediscovered:

• Document text is not searchable. The rows are in neither segment_fts nor the vector index. A document is findable by the summary enrichment writes, because that path reindexes — which looks exactly like the body being indexed until someone searches for a phrase on page 74.
• Search paging. routers/search.py has no offset and its total is the size of the page, not the corpus. "Show more" asks for a bigger page, capped at the server's 100. A real pager is a backend change.
• Legacy and OpenDocument formats (.doc, .xls, .ppt, .rtf, .odt, .ods, .odp) refuse by name with what to do instead. Scans need OCR, which is its own project.

Testing

• Backend: 739 passed, up from 712. The 6 failures are the pre-existing ffmpeg ones — identical set before and after these changes; this sandbox has no ffmpeg, CI installs it.
• Frontend: 244 passed, up from 210. npm run build, eslint --max-warnings 0 and prettier --check all clean.
• The migration was run forward and back against a scratch database.

Not verified here: nothing touching real files in a browser, since this sandbox has no ffmpeg. The end-to-end checks are listed in the plan — the one that matters most is running Summarize on a PDF and confirming the summary reflects its contents rather than its cover page.

🤖 Generated with Claude Code

https://claude.ai/code/session_01F6D8S8oXYij748Fu5k4AAZ

---

Generated by Claude Code