Skip to content

Remove the structure filter: dead under cost-gap admission after #365 #366

Open

Description

@send

Summary

After #365 the structure filter measures its threshold from the pre-history #1. Re-measured under that rule, the pre-registered deletion check holds on all three conditions, so the filter looks dead under cost-gap admission and can be removed together with its supporting machinery.

The rule (pre-registered before #365, evaluated right after it)

Delete the filter if all of:

  1. Page 1 (first 9 candidates, n=20) differs from a filter-off oracle (structure_cost_filter = 10^9) on ≤ 1 of the corpus readings
  2. Replaying the commit log's rank>0 selections, the owner breakdown of what gets selected differs from the oracle by 0
  3. No filter-only drop (a dropped path within the cost-gap bound, which admission alone would have kept) lands in page-1 range

Measured (main 450fa55 + #365, frozen commit-log snapshot, all arms at the same time; counts only)

Condition Result
1. page 1 vs oracle differs on 0 / 142 corpus readings
2. replay selections by owner vs oracle identical, without history and with history
3. filter-only drops in page-1 range 0 (filter-only drops: 26 of 593 dropped paths, across 67 readings)

Expectation before measuring was that the filter would stay (most surfaces it lets back in are junk); with the threshold measured from the #1 instead of the population minimum, admission already removes everything the filter used to catch on page 1.

Scope of the removal PR

  • structure_cost_filter hard filter in rerank, with what exists only for it: the prefix floor (structure_prefix_floor), the per-transition cap (structure_cost_transition_cap), single-segment imputation, StructureFilter / on_drop, and explain's dropped_by_structure / structure_filter (added in fix(core): measure the structure filter from the #1, not the population minimum (#353) #365 as provisional).
  • Settings keys structure_cost_filter / structure_cost_transition_cap: keep accepting them (on-disk settings must keep loading) and warn that they no longer have an effect.
  • Identity exemption (分節崩れ・接辞付き複合語の変換ミス #263) disappears with the filter; check that the typed-kana path stays selectable through admission (it is exempt there too).
  • Re-run both accuracy corpora, the 3-width top-1 check, replay (no history / history) against main, and the cost-gap T rule (DEFAULT_CANDIDATE_MAX_COST_GAP): removing the filter changes the population admission sees.
  • Speed ±5% on the 1-best path (removing work, expected neutral or faster).

Order

Before #362 (PR-F, single_char_kanji_penalty), so PR-F is measured on the final candidate pipeline rather than one whose filter is about to go. Then #361 (PR-E2).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions