Skip to content

gh-153569: Simplify tokenizer state - #158124

Open
pablogsal wants to merge 29 commits into
python:mainfrom
pablogsal:gh-153569-tokenizer-cleanups
Open

pablogsal wants to merge 29 commits into
python:mainfrom
pablogsal:gh-153569-tokenizer-cleanups

Conversation

@pablogsal

@pablogsal pablogsal commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

@pablogsal pablogsal changed the title gh-153569: Simplify tokenizer state, input handling, and diagnostics gh-153569: Simplify tokenizer state Sep 24, 2026
@pablogsal
pablogsal force-pushed the gh-153569-tokenizer-cleanups branch 3 times, most recently from b32925e to 12de6d3 Compare September 24, 2026 17:29
@pablogsal
pablogsal force-pushed the gh-153569-tokenizer-cleanups branch from 06d93cd to cb1af47 Compare September 24, 2026 17:55
@pablogsal
pablogsal force-pushed the gh-153569-tokenizer-cleanups branch from cb1af47 to 30f8f52 Compare September 24, 2026 18:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant