waitForCassandraSecondaryIndex never waits: ALLOW FILTERING defeats its own probe #803

waitForCassandraSecondaryIndex (internal/storage/db/cassandradb/provider.go:646) exists to block startup until a secondary index is queryable, because ScyllaDB builds them asynchronously. Its probe is:

probe := fmt.Sprintf("SELECT id FROM %s.%s WHERE %s='' LIMIT 1 ALLOW FILTERING", keyspace, table, column)

ALLOW FILTERING is exactly the hint that lets Cassandra answer a query on a non-indexed column. So the probe succeeds whether or not the index exists, the function returns on its first attempt, and it never waits for anything.

Reproduction

Against a ScyllaDB container, using a table with a deliberately non-indexed column:

cqlsh> SELECT id FROM probe_t.t WHERE plain_col='' LIMIT 1 ALLOW FILTERING;
 (0 rows)                      <- succeeds with no index

cqlsh> SELECT id FROM probe_t.t WHERE plain_col='' LIMIT 1;
 InvalidRequest: Cannot execute this query as it might involve data filtering
 and thus may have unpredictable performance [...] use ALLOW FILTERING

The second form is what a real readiness probe has to be: a query that fails until the index is ready.

Impact

All 13 call sites get a no-op wait, so every one is exposed to the startup race the helper was written to prevent:

client_id, issuer_url, SAML SP org_id, SAML IdP key org_id, credential_id, organization name, org membership org_id / user_id, federated identity org_id / user_id, SCIM endpoint org_id, SCIM group org_id, org domain org_id.

Symptom would be an intermittent index-unavailable error on the first query against a freshly provisioned keyspace — most likely in CI or a new deployment, hard to reproduce once indexes have settled.

Fix

Drop ALLOW FILTERING from the probe string. One line — but it makes 13 waits genuinely block for the first time, so it needs its own testing against a cold ScyllaDB container to confirm startup latency stays acceptable. That is why it was not bundled into #802.

waitForCassandraIndexes, added in #802, already omits the hint for this reason and carries a comment explaining why.