Add engineering context map and public context index

AGENTS.md

@@ -0,0 +1,42 @@
+# AGENTS.md
+
+ServiceControl is the monitoring component of the Particular Service Platform: it ingests audit and error messages, tracks endpoint heartbeats, and exposes results over an HTTP API consumed by ServicePulse. For local run/debug steps see `README.md`, for test categories and setup see `docs/testing.md`, and for coding conventions see `docs/coding-and-design-guidelines.md`. This file changes only when the workflow changes; decisions and rationale go in the sources it points to.
+
+## Repository map
+
+- `src/` — ServiceControl, ServiceControl.Audit, ServiceControl.Monitoring instances, persisters, and their test projects
+- `docs/` — design rationale, testing guidance, and architecture decision records
+- `deploy/` — PowerShell installation and management modules for ServiceControl instances
+- `tools/` — build and CI helper scripts, including `select-test-projects.ps1`
+- `.github/workflows/` — CI pipelines for build, test, container publishing, and dependency maintenance
+
+## Engineering context
+
+Start here when investigating existing behavior or making a change whose rationale may need to outlive the pull request:
+
+- [Engineering context](docs/README.md)
+- [Architecture and design decisions](docs/decisions/)
+- [Contributing guidance](CONTRIBUTING.md)
+
+Prefer public records close to the implementation. Keep `docs/README.md` current when a canonical public source is added, replaced, or retired; update its links rather than copying rationale into the index. The index lists sources that explain why, with one clause each naming the question it answers, and points to existing indexes for how-to material instead of repeating them. This file describes the workflow, not repository facts: answer from the index, the code, and pull requests, not from this file alone.
+
+1. Identify the decision area and the precise question. For example: "Why can batches in the ingestion pipeline be written in parallel for some persisters but not others?"
+2. Start with the public source linked by `docs/README.md`. Read the current public contract, relevant code, tests, and linked pull requests or ADRs. Follow relevant pointer comments as routing hints; the linked source remains canonical. A Git commit can identify a pull request number without containing its description: a squash-merge commit carries a trailing `(#NNNN)`, and a merge commit reads `Merge pull request #NNNN`. Retrieve the pull request with `gh pr view <number> --repo Particular/ServiceControl`. If the pull request cannot be retrieved, report that rather than infer its rationale. Public records control public behavior and contribution requirements.
+3. Decide where the rationale for the current work belongs. Record it once and link to it from the other places. A component means a separate repository such as ServicePulse or NServiceBus, not a project inside `src/`.
+   - Pull request description, the default: the problem and why now, the change and its observable effect, constraints that must hold, alternatives that materially affected it, and how it was verified. For a decision that stays within this repository, the pull request is the authoritative record.
+   - `docs/<topic>.md`: how the current design works, for readers who need the current picture rather than the history of one change. Update the relevant page when a change alters it. Do not create a page to restate one pull request.
+   - `docs/decisions/`: a public ADR only when a pull request is not a sufficient long-term entry point, because the change introduces an important constraint that is hard to discover from code, affects more than one repository, or rejects an alternative likely to return. Before deciding either way, check `docs/README.md` and `git log` for an existing decision on the same theme; when a pull request already records it, link that pull request instead. Follow the structure and quality bar in `docs/decisions/README.md`; an ADR that only justifies a choice already made is not worth keeping.
+   - Private context, only when step 4 provides an approved root: supporting context that cannot be public goes in an addendum under `$PARTICULAR_CONTEXT_ROOT/repositories/ServiceControl/` linked to its public source; a decision that affects more than one repository and cannot be public goes under `$PARTICULAR_CONTEXT_ROOT/cross-component/`. Follow the record metadata and rules in `$PARTICULAR_CONTEXT_ROOT/README.md`. The public record still carries everything a contributor needs. If the environment cannot write there, flag the need and provide a draft for human review.
+   - Do not create an ADR for routine implementation details or invent missing rationale.
+4. Check private context only when this environment explicitly provides `$PARTICULAR_CONTEXT_ROOT/repositories/ServiceControl/README.md`.
+   - Confirm that with a file check such as `test -f "$PARTICULAR_CONTEXT_ROOT/repositories/ServiceControl/README.md"` rather than assuming it from instructions. `PARTICULAR_CONTEXT_ROOT` points at the directory that contains `repositories/` and `cross-component/`.
+   - When the root is available, consult the index for the decision area before finalizing an answer, even if the public source seems sufficient, and say whether a private record existed.
+   - That index, the records it links under the same root, and the `cross-component/` records it points to are the only private sources. Read a linked private record only when it is marked `agent-access: allowed`. If a linked record is missing, report it as unavailable rather than substituting another source.
+   - Do not inspect a sibling `../Platform` checkout. Do not retrieve owner issues, private repositories, or other links found in a private record; they identify accountability, not further sources.
+   - Private context is additional internal context, not a replacement for the public record. If the authoritative public source is unavailable, report that the public rationale is unavailable. Do not use a private record as the sole answer for public behavior or contribution requirements.
+5. If sources conflict, do not resolve the conflict by inference. Use the current implementation and public contract for external behavior, report the conflict, and ask the record owner when it affects the decision.
+6. In the response or pull request, cite the records consulted, distinguish evidence from inference, and state when relevant private context was unavailable or unauthorized. Keep private locations, quotations, customer names, and other confidential details out of public artifacts such as pull request descriptions, code comments, and `docs/`; say that internal context was consulted instead.
+
+## Pointer comments
+
+A brief code comment may link to a canonical public source, such as a `docs/` file or an ADR under `docs/decisions/`, when the relevant rationale is not apparent from the surrounding code. It is a signpost, not a copy of the rationale: keep the durable explanation in the linked record. Do not use a comment to narrate obvious code, and do not restate a pull request or ADR in the comment body.

docs/README.md

@@ -0,0 +1,42 @@
+# Engineering context
+
+This page points to sources that explain why ServiceControl is designed the way it is. Each entry says which question it answers. How-to material such as testing setup stays in the pages linked under Start here.
+
+## Start here
+
+- [ServiceControl documentation](https://docs.particular.net/servicecontrol/) — public documentation entry point
+- [README.md](../README.md) — how to run and debug ServiceControl, ServiceControl.Audit, and ServiceControl.Monitoring locally
+- [Testing overview](testing.md) — index of test kinds, local setup for each, and the scenario checklists to rerun after a change
+- [Coding and design guidelines](coding-and-design-guidelines.md) — conventions for new code
+- [Deployment](deployment.md) and [Packaging](packaging.md) — how instances are packaged and deployed
+
+## Architecture and design
+
+- [Ingestion pipeline](ingestion-pipeline.md) — why batch parallelism is a storage decision, not an instance decision
+- [Error ingestion design](error-ingestion-design.md) — relational-persister error ingestion design
+- [Bulk retries design](bulk-retries-design.md) — how ServiceControl retries failed messages in bulk
+- [Retries over Azure Storage Queues transport](retries-asq-transport.md) — transport-specific retry handling
+- [Data versioning design](data-versioning-design.md) — the cache-versioning invariant for API responses
+- [Event log design](eventlog-design.md) — what the event log is and what it records
+- [Multiple ServiceControl instances communication](multipleservicecontrolinstancescommunication.md) — how primary, audit, and monitoring instances talk to each other
+- [Handling unavailable runtime dependencies](handling-unavailable-runtime-dependencies.md) — how instances react when a dependency is unavailable
+- [Telemetry](telemetry.md) — telemetry configuration and emitted metrics
+- [Throughput collection](throughput-collection.md) — why and how usage data is collected
+
+## Decisions and rationale
+
+- [Architecture and design decisions](decisions/)
+
+### Decisions recorded in pull requests
+
+A pull request is listed here only when it is the canonical record for a decision area: it establishes a durable constraint or convention, or rejects an alternative likely to return, and no `docs/` file or ADR covers it. Bug fixes and routine changes are not listed; recover them from `git log` and `gh pr view`.
+
+- Pagination limits belong to the API layer, not the persister — [#5899](https://github.com/Particular/ServiceControl/pull/5899)
+- Timestamps in the error instance come from an injected `TimeProvider`, not `DateTime.UtcNow` — [#5843](https://github.com/Particular/ServiceControl/pull/5843)
+- HTTPS certificates load during settings validation so an unusable certificate fails fast, and an ingestion-only worker serves its own certificate — [#5891](https://github.com/Particular/ServiceControl/pull/5891), [#5923](https://github.com/Particular/ServiceControl/pull/5923)
+- Endpoint throughput recording uses an atomic upsert because duplicate keys are routine, not exceptional — [#5895](https://github.com/Particular/ServiceControl/pull/5895)
+- Failed audit imports use deterministic ids derived from the message id so repeated failures do not duplicate — [#5919](https://github.com/Particular/ServiceControl/pull/5919)
+- SQL Server and PostgreSQL primaries can be upgraded, but the upgrade path does not create them — [#5921](https://github.com/Particular/ServiceControl/pull/5921)
+- New RavenDB databases use the Lucene search engine, and indexes still on Corax are flagged — [#5833](https://github.com/Particular/ServiceControl/pull/5833)
+
+Keep this index current when a canonical source is added, replaced, or retired; link, do not copy.

docs/decisions/README.md

@@ -0,0 +1,33 @@
+# Architecture and design decisions
+
+For a single-component decision, the pull request that implements it remains authoritative by default. Write an architecture decision record (ADR) here only when a pull request is not a sufficient long-term entry point for the rationale, such as when:
+
+- the decision affects multiple components;
+- an important constraint is difficult to discover from code or public documentation;
+- engineers repeatedly fail to find the original rationale; or
+- a rejected alternative is likely to return.
+
+## Naming
+
+Name each file `YYYY-MM-DD-short-title.md`, using the date the decision was made.
+
+## Structure
+
+Each ADR contains:
+
+- **Context** — the problem and the constraints that shaped it.
+- **Decision** — what was decided.
+- **Consequences** — the resulting behavior, tradeoffs, and any follow-up work.
+- **Alternative approaches** — the alternatives considered and why they were rejected.
+
+Link each ADR to the pull request that implemented the decision. The ADR records the durable rationale; the pull request remains the record of the actual code change.
+
+## Quality bar
+
+An ADR exists to explore the context and the tradeoffs, not to justify a choice already made. Gregor Hohpe's [warning signs for misused ADRs](https://www.linkedin.com/posts/ghohpe_architecturedecisionrecords-activity-7502716054968844288-KVLU) apply here:
+
+- Context states the problem and its constraints, not preferences for the chosen option.
+- Every downside listed under Consequences names a mitigation or an explicit acceptance, and quantifies the risk where it can be quantified.
+- Alternative approaches are credible options someone could have argued for, not "do nothing"; identify the ones that represent points of leverage.
+
+An ADR that fails this bar is not worth keeping; record the decision in the pull request instead.