[ecosSystem]

Reading a table front to back was slow: for example DbEval(), a count with a condition in a browse window, or a report. Counting ADRESSDB (52,000 rows, 154 columns) took about 45 s on SQL Server against 5 s with ADS. There were two causes, one commit each, plus a read-ahead.

1. OFFSET paging was quadratic

Every next page was fetched with ORDER BY ... OFFSET n ROWS FETCH NEXT PageSize ROWS, so the server stepped over all n rows for every page. For ADRESSDB that was 521 queries with OFFSET up to 52,000.

_FetchForward() now uses keyset paging. It asks for the rows that follow the last buffered row in the current order:

• the ORDER BY items plus the recno column are compared lexicographically with their values for that row,
• the values are read by subqueries on the row's recno, so the comparison uses the same expressions and collation as the ORDER BY, also for keys with functions and for descending orders,
• a leading k1 >= v1 bound lets the server seek in the index,
• the fetched rows are exactly the ones OFFSET would return, so the buffer coordinates do not change.

OFFSET remains for a non-contiguous buffer, for tables without a recno column, and when the last row has NULL in a key column.

2. Read ahead

When pages are read one after the other, every next fetch asks for twice as many rows as the previous one. This goes up to BufferSize pages and never beyond what the buffer holds without evicting the current row. Opening, paging backward or reloading at an offset starts again with one page, so browsers read as before.

After a hard seek the read-ahead starts small (PageSize / 10 rows) and doubles from there. Code that seeks and then skips while the key matches mostly reads only a few records, and a full page of a wide table costs about 10 ms to load. A seek that is followed by a long scan still reaches full pages after a few fetches.

3. Constraint indexes on every loaded page

SqlDbCommand.GetDataTable() filled pages with DataTable.Load(reader). Load takes NOT NULL from the reader's schema and validates the constraints when it finishes, which builds a sort index for every non-nullable column. Those indexes stay attached to the table, and the RDD keeps the first page table as its row buffer, so with 154 NOT NULL columns every row added to or removed from the buffer maintained 154 indexes. Profiling showed about 80 % of the time in DataTable index inserts and string compares. Pages are now loaded inside a DataSet with EnforceConstraints = FALSE. The RDD already switched the constraints off on its buffer afterwards (via reflection), but too late to avoid the indexes.

Measurements

DbEval() over ADRESSDB, 52,025 rows, SQL Server 2022 Express:

order	before	after
natural (recno)	36.3 s	2.0 s
tag REF_NR FOR TYP == 'K'	54.8 s	3.9 s
tag TYP+SUBSTR(PLZ,1,1)+KURZNAME	205 s	34 s

The last one stays slow because the server has to sort the computed key for every page. No index can serve that order without an additional computed column.

Testing

• The recno sequence of the full read is identical to the OFFSET version for natural order, tags with FOR conditions, function keys and descending tags.
• Row and key counts, GoTop/Skip/GoBottom/Skip(-n) over all 297 tables and about 1,065 tags of our ERP are unchanged.

🤖 Generated with Claude Code