Fix deleted chat messages reappearing as masked

[Paul Lizer (paullizer)]

Summary

• Deleted messages came back as "This message is masked." With Enable Conversation Archiving on, deleting a chat message is a soft delete. The document stays with metadata.is_deleted set and is also masked as a fail-safe. /api/get_messages still returned it, and V2 didn't filter it out (the classic chat does). So it reappeared as a masked placeholder in the re-read after the delete and every time the conversation was reopened. With archiving off the delete is permanent, which is why it only happened sometimes.
• The same documents came back in other places. Attempt switching, forks, convert-to-shared, search snippets, the inbound MCP message tool, conversation summaries, and the history sent to the model all read them. Retry and edit copied attempt 1's metadata, so a new question could start out deleted and masked. Both mask routes could also clear the fail-safe mask.
• Fix: one shared helper (functions_message_deletion.py) is applied at every read, copy, search, and model boundary. Delete, retry, edit, switch-attempt, and both mask routes return 404 for a deleted target. V2 also drops deleted messages on load and re-read. Stored documents and archive copies are unchanged. Shared conversations that already hold copies hide them at read time, so no migration is needed.
• Related attempt bugs: deleting only the latest answer of a retried turn brought an older attempt back next to the latest question, whether or not archiving was on. Now another attempt takes over only when the active attempt's question is deleted, a deleted attempt is never chosen, and exactly one attempt stays active. The V2 attempt counter now counts by position, so it can't read "3/2".

Worth a careful look:

• Attempt promotion in delete_message moved into _promote_remaining_thread_attempt. It uses a parameterized query instead of the string-built NOT IN (...) query, and deactivates whatever is left of the deleted attempt.
• switch_attempt now selects c.metadata instead of SELECT DISTINCT ... thread_attempt so it can see is_deleted. Retry's source lookup no longer uses TOP 1, so deleted attempts can be skipped.
• Lookups that use TOP (recent assistant replies, diagram-edit grounding) filter in the Cosmos query with NOT_SOFT_DELETED_COSMOS_FILTER, so TOP counts only messages that still exist.
• The search cache key gains soft_deleted_message_policy_version, so cached results that contain deleted snippets aren't served.
• Three existing tests run production functions in namespaces they build themselves: test_analyze_backend_saved_integration.py, test_content_screening_history.py, and test_group_collaboration_source_storage_fix.py. They now bind exclude_soft_deleted_messages.

Linked issue

Fixes #1649

Refs #1650 (gaps in regular masking found while fixing this, left for a follow-up)

Release Notes & Latest Features

• New Feature
• Bug Fix
• UI Enhancement
• Breaking Change
• Internal only

Is this visible to end users?

• Yes
• No

Is this admin-facing (Admin Settings, governance, deployment, config)?

• Yes
• No

Should this become a Latest Feature card?

• Yes
• No
• Already added

Screenshot needed for the card?

• Yes
• No
• Attached

Version bump

• application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only (0.261.252 to 0.261.253)
• deployers/version.txt bumped, or not needed because deployers/ was not changed (not changed)

Testing / validation

New tests:

• python -m pytest functional_tests\test_chat_soft_deleted_message_visibility_fix.py -q -p no:cacheprovider: 2 passed. The test imports the real modules in a fresh process with network access blocked and runs 22 scenarios (the helper plus 21 paths) under normal and optimized Python. python functional_tests\test_chat_soft_deleted_message_visibility_fix.py also reports 2/2 passed.
• python -m pytest .\ui_tests\test_v2_chat_deleted_messages.py -q -p no:cacheprovider: 2 passed. It runs the production message list, composer, and chat store in Chromium. Its HTTP stub returns deleted messages the way the server did before this fix.
• Regression check: with the 7 changed server files stashed, the functional probe fails all 21 path scenarios. With the 2 changed V2 files stashed, the UI test fails with two "This message is masked" placeholders on screen, and the attempt counter doesn't reach "2/2".

Builds and existing tests:

• npm --prefix application\v2_ui run typecheck: passed
• npm --prefix application\v2_ui run build -- --outDir ..\..\ui_tests\artifacts\orchestration-plan-editor: passed
• python functional_tests\test_v2_message_actions.py: 7/7 passed
• python functional_tests\test_v2_chat_phase1_fixes.py: 10/10 passed
• python functional_tests\test_v2_message_masking.py: 9/9 passed
• python functional_tests\route_tests\test_route_blueprint_policy_inventory.py: 12/12 passed
• python functional_tests\route_tests\test_route_unauthenticated_policy_contract.py: 7/7 passed
• python functional_tests\route_tests\test_route_policy_test_coverage.py: 2/2 passed
• python functional_tests\test_docs_app_surface_coverage.py: 7/7 passed
• python functional_tests\test_docs_site_quality.py: 6/6 passed
• python -m pytest <39 related functional test files> -q -p no:cacheprovider --continue-on-collection-errors -W ignore::pytest.PytestReturnNotNoneWarning -rfE (files listed below): 415 passed, 44 failed, 204 errors. I compared failing test IDs against a run of the same files on unmodified HEAD, and no test fails only with this change. The ones I traced need Azure configuration that this environment doesn't have; for example, importing config.py builds a Cosmos client.

Failures that also occur on unmodified HEAD:

• python scripts\check_broken_access_control.py <the 8 changed application .py files>: 2 findings, both on lines this PR doesn't change. HEAD reports the same two.
• python functional_tests\test_docs_release_notes_integrity.py: the generated release-notes pages are already stale on HEAD, by more than 158 versions. I didn't regenerate them here, to keep the diff focused.
• python functional_tests\test_docs_link_integrity.py: 11 broken relative links elsewhere in the docs.

Environment note: the tests that import route modules ran in a fresh virtualenv with application/single_app/requirements.txt installed. This machine's global Python has a pyOpenSSL/cryptography mismatch that breaks importing route_backend_chats.

The 39 related functional test files

test_advanced_conversation_search_matching_fix.py, test_analyze_backend_saved_integration.py, test_assist_thread_submission_id.py, test_backend_conversations_swagger_integration.py, test_chat_cited_source_tracking.py, test_chat_layered_message_masking.py, test_chat_soft_deleted_message_visibility_fix.py, test_chat_stream_history_context_fix.py, test_collaboration_conversation_summary_export_fix.py, test_collaboration_message_delete_fix.py, test_collaboration_shared_conversation_management_fix.py, test_content_screening_history.py, test_conversation_export.py, test_conversation_fork.py, test_conversations_read_ownership_authorization.py, test_enhanced_citations_blob_and_collaboration_fix.py, test_group_collaboration_source_storage_fix.py, test_image_chat_route_integration.py, test_inbound_mcp_governance_and_tools.py, test_inbound_mcp_server_shell.py, test_m365_collaboration_action_cards.py, test_m365_conversation_lifecycle.py, test_message_block_revisions.py, test_orchestration_conversation_context.py, test_orchestration_conversation_context_routes.py, test_orchestration_deliverables.py, test_orchestration_workflow_run_adapter.py, test_personal_conversation_followup_authorization.py, test_stored_xss_share_activity_and_masking_fix.py, test_v2_chat_phase1_fixes.py, test_v2_message_actions.py, test_v2_message_masking.py, test_v2_prompt_attachment_persistence.py, test_v2_shared_conversations.py, test_workflow_chat_delivery_placement_and_masking.py, test_workflow_chat_delivery_refusals.py, test_workflow_result_chat_routes.py, test_workflow_result_orchestration_lineage.py, test_workflow_result_review_paths.py

Documentation

• Release notes updated, or not needed (v0.261.253 entry in docs/explanation/release_notes.md)
• Feature documentation updated, or not needed (not needed)
• Fix documentation updated, or not needed (docs/explanation/fixes/SOFT_DELETED_MESSAGES_SHOWN_AS_MASKED_FIX.md, listed in the fixes index. The Enable Conversation Archiving row in docs/admin/data-lifecycle.md now says deleted messages are archived and hidden.)

Security checklist

• New Flask routes include @swagger_route(security=get_auth_security()) (no new routes)
• Settings sent to non-admin frontends use sanitize_settings_for_user() (no settings changes)
• Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS (V2 source changes only, no new assets)
• No secrets, keys, connection strings, or local-only artifacts are included (build output and the UI test harness bundle are gitignored)