gh-153569: Simplify tokenizer state - #158124

Open

pablogsal wants to merge 29 commits into

python:mainfrom

pablogsal:gh-153569-tokenizer-cleanups

Commits on Sep 24, 2026

Use diagnostic line for missing-brace errors
pablogsal
committed
a7684aa
Keep implicit prepared newline metadata in the reader
pablogsal
committed
f276100
Reuse parsed formatted string kind in lexer
pablogsal
committed
5d28759
Transfer owned tokenizer encoding strings
pablogsal
committed
3fdd498
Remove unused tokenizer state and redundant branches
pablogsal
committed
b0ab20b
Avoid rescanning partial decoded tokenizer lines
pablogsal
committed
8b0d8dd
Reuse prepared tokenizer line terminators for CR detection
pablogsal
committed
b913bf8
Reuse interactive tokenizer chunks without carriage returns
pablogsal
committed
65a1f41
Release tokenizer constructor resources on allocation failure
pablogsal
committed
d807f73
Narrow interactive newline normalization wrapper
pablogsal
committed
98a96e5
Close duplicated tokenizer descriptors when fdopen fails
pablogsal
committed
6eb2e4a
Preserve cached tokenizer lines when newline allocation fails
pablogsal
committed
f200953
Collect cycles involving tokenizer readline callbacks
pablogsal
committed
f01c604
Preserve buffered text when decoding source files
pablogsal
committed
f3e6076
Clear the transferred syntax error line before cleanup
pablogsal
committed
ec6259a
Propagate tokenizer column conversion failures
pablogsal
committed
30f8f52
Allocate parser repetition buffers on first match
pablogsal
committed
5ad3239
Allocate type comment storage only when needed
pablogsal
committed
bfa0b82
Reuse decoded multiline text when counting token columns
pablogsal
committed
96a07b2
Skip empty decoded tokenizer chunks
pablogsal
committed
930bb19
Check parser array growth before allocating
pablogsal
committed
e8eda42
Name tokenizer bounds and derive byte lengths from their data
pablogsal
committed
4fe3bd0
Set the tokenizer exception when file input size overflows
pablogsal
committed
1cb4394
gh-153569: Send exact line endings in the REPL test
pablogsal
committed
849c26b
Share parser repetition buffer growth
pablogsal
committed
59a2c99
Fold Unicode input coverage into the existing REPL test
pablogsal
committed
89806d7
Keep parser array growth policy private
pablogsal
committed
0eea3f1
Drop obsolete bitmap stress from source discard test
pablogsal
committed
e74e40b

Commits on Sep 25, 2026

Preserve input failures while scanning string literals
pablogsal
committed
b6085cd

Navigation Menu

Uh oh!

gh-153569: Simplify tokenizer state - #158124

gh-153569: Simplify tokenizer state#158124
pablogsal wants to merge 29 commits into
python:mainfrom
pablogsal:gh-153569-tokenizer-cleanups

Commits on Sep 24, 2026

Use diagnostic line for missing-brace errors

Keep implicit prepared newline metadata in the reader

Reuse parsed formatted string kind in lexer

Transfer owned tokenizer encoding strings

Remove unused tokenizer state and redundant branches

Avoid rescanning partial decoded tokenizer lines

Reuse prepared tokenizer line terminators for CR detection

Reuse interactive tokenizer chunks without carriage returns

Release tokenizer constructor resources on allocation failure

Narrow interactive newline normalization wrapper

Close duplicated tokenizer descriptors when fdopen fails

Preserve cached tokenizer lines when newline allocation fails

Collect cycles involving tokenizer readline callbacks

Preserve buffered text when decoding source files

Clear the transferred syntax error line before cleanup

Propagate tokenizer column conversion failures

Allocate parser repetition buffers on first match

Allocate type comment storage only when needed

Reuse decoded multiline text when counting token columns

Skip empty decoded tokenizer chunks

Check parser array growth before allocating

Name tokenizer bounds and derive byte lengths from their data

Set the tokenizer exception when file input size overflows

gh-153569: Send exact line endings in the REPL test

Share parser repetition buffer growth

Fold Unicode input coverage into the existing REPL test

Keep parser array growth policy private

Drop obsolete bitmap stress from source discard test

Commits on Sep 25, 2026

Preserve input failures while scanning string literals