ARIE decides what enrichment evidence is worth buying, when enough evidence exists to act, when evidence is insufficient, and why it stopped.
Most lead-enrichment pipelines call every data provider for every lead, then ask a model to score whatever comes back. ARIE asks a different question first: given what I already know, is the next purchase even worth making?
Try it — no login required · Frontend repo · Docs
A sales team buys contact and company data per lookup. The usual pipeline runs every provider on every lead, because deciding which ones to skip is harder than just calling them all. Most of that spend buys nothing — the answer was already obvious three providers ago.
ARIE treats it as a stopping problem instead. It buys the cheapest evidence
first, and after each purchase asks two questions: could anything I haven't
bought yet still change this answer? and am I confident enough to act
without a person? When both say no more is needed, it stops and decides. When
the evidence genuinely isn't enough, it says so — evidence_sufficiency is a
first-class, honestly-reported field, and insufficient evidence is never
allowed to present itself to a user as a definitive rejection.
The interesting part is what happens when it isn't confident. ARIE doesn't guess. It hands the lead to a human, records what it would have done, and keeps both records side by side afterwards — so you can always see where the machine and the person disagreed.
I built this to find out whether that kind of adaptive stopping actually beats a well-tuned fixed pipeline. It does, on cost — and it costs you some accuracy. The honest numbers are below.
flowchart LR
A["New lead"] --> B["Buy cheapest<br/>useful evidence"]
B --> C{"Could more data<br/>change the answer?"}
C -->|yes, and affordable| B
C -->|no| D{"Confident enough<br/>to act alone?"}
D -->|yes| E["Route or reject<br/>automatically"]
D -->|no| F["Send to a human"]
E --> G["Decision Receipt"]
F --> G
Two separate rules, answering two different questions. Settled asks whether anything left to buy could still flip the outcome. Confidence asks whether the answer is actually right. A decision can be settled and still wrong, so neither rule replaces the other.
More detail in architecture.md.
This is the part I'd point at first.
Every lead produces a receipt: what ARIE decided, how confident it was, why it stopped buying data, what that cost, and which providers were involved. It is reconstructed from stored facts, not re-derived later, so a receipt from three months ago still explains a decision made under a policy version you've since replaced.
One rule holds it together: a machine recommendation and a human's decision never collapse into a single "outcome" field. If ARIE said reject and a reviewer approved, the receipt shows both, in order, permanently. And insufficient evidence is never presented as a definitive rejection — a lead ARIE couldn't settle reads as "worth pursuing, needs a person," not "no."
| Confident enough to act | Insufficient evidence |
|---|---|
|
 |
| Confidence cleared the threshold, so ARIE acted alone. | The reachable range still crossed a threshold — ARIE stopped short and a person decided, instead of guessing. |
The provider ledger is deliberately blunt about waste. It separates evidence bought fresh from evidence reused out of cache, and it names any provider that charged for a call and returned nothing.
Try all three outcomes yourself, with no account: /demo.
flowchart LR
U["Browser"] --> V["Vercel<br/>Next.js proxy"]
N["n8n Cloud"] --> R
V --> R["Railway — API"]
R <--> S[("Supabase<br/>Postgres")]
W["Railway — worker"] <--> S
The API writes identity resolution, the lead row and its first job in one
transaction. A worker claims jobs with SELECT ... FOR UPDATE SKIP LOCKED, so
adding workers needs no coordination between them. No Redis, no Celery, no
Temporal — Postgres already gives transactional consistency with the lead
state those jobs mutate.
The same database carries a multi-tenant layer on top of that: Supabase-issued sessions and scoped API keys for auth, row-level security per organization, BYOK provider credentials in Supabase Vault, and a self-serve commercial layer — signup, Stripe subscriptions, plan entitlements, transactional email. Entitlements only ever decide what an organization may configure; they can never grant autonomy the calibration data doesn't support, and every entitlement change goes through one signature-verified Stripe webhook rather than a browser redirect.
Full topology, environment variables and rollback path: deployment.md. Design decisions and the ones deliberately rejected: architecture.md and docs/adr/.
Everything above is proven twice: a synthetic benchmark (below) and a small, real-money validation against live vendors. Kept deliberately separate, because they answer different questions.
Verified in a disposable, fully isolated environment, purpose-built so this
could never touch production: a fresh Supabase database branch
(with_data:false), a temporary organization with its own Vault-stored
Abstract + Hunter credentials, execution_mode=live_shadow (real evidence,
real cost, zero authoritative effect). Exactly three real leads, chosen to
cover a clean match, a genuinely ambiguous identity, and a known vendor edge
case — expectations pre-registered and frozen before any provider call.
| Lead | Category | Abstract | Hunter | Outcome |
|---|---|---|---|---|
| Steli Efti · Close | Strong fit | miss | VERIFIED match, scored |
evidence scored normally |
| Hailley Griffis · Buffer | Ambiguous / role-alias email | success (firmographics) | miss — no identity found | correctly left unscored, not guessed |
| Patrick Collison · Stripe | Provider-quality edge case | success (firmographics) | MISMATCH — wrong person returned |
evidence correctly suppressed |
Total real spend: $0.01965. Branch deleted after the run; a post-deletion branch listing confirmed only the permanent production branch remained. Production database and organization were never read from or written to at any point.
The Patrick Collison case is the one worth dwelling on: Hunter returned a real
but wrong person's data — reproduced across three independent real calls on
three separate occasions. arie.identity.validation requires a VERIFIED
match verdict before any person-provider evidence can reach the scorer, so the
wrong person's title never entered the decision. That guard, and the vendor
defect it exists for, are both real — not a synthetic test case.
This was an architecture/correctness validation at n=3, not a statistical accuracy study. No accuracy, ROI, or cost-savings claim is made from it.
Ten seeds, 300 held-out test leads each, dataset regenerated and the baseline re-tuned per seed. This is the synthetic benchmark — the real-provider validation above is a separate kind of evidence, at a much smaller scale.
| policy | agreement | API $/lead | calls | autonomy |
|---|---|---|---|---|
| full enrichment (call everything) | 0.8390 | 0.4447 | 8.00 | 0.816 |
| tuned waterfall (industry baseline) | 0.8347 | 0.4205 | 7.58 | 0.795 |
| calibrated bounds ← production | 0.8113 | 0.2463 | 5.26 | 0.833 |
| adaptive EVoI | 0.8093 | 0.2906 | 2.19 | 0.786 |
The project's founding hypothesis was expected-value-of-information (EVoI) reasoning. It failed the bar set before running anything (≤1pp agreement loss at ≥20% cost reduction) and lost to a much simpler ablation on 9 of 10 seeds — the project's headline negative result, written up rather than buried: ADR 0004. Calibrated bounds was selected after the EVoI hypothesis failed that preregistered win condition; it reduces modeled API spend ~41.6% versus the tuned waterfall baseline, at ~2.3 percentage points lower synthetic-oracle agreement — a stated trade-off against the pre-registered bar, not a claim that it "won" anything. The standard deviation on that saving is 11.0pp, large next to the effect.
Method, dataset design and every parameter assumption: benchmark.md.
A third kind of evidence, larger than the n=3 real-provider validation above and closer to real usage than the synthetic benchmark: 140 real, named companies run through ARIE's actual production API — real org, real ICP-confirmation flow, real CSV upload, real job queue, real Decision Receipts — evaluating the full "buy only what's necessary" thesis end to end, not just the policy in isolation.
Reduced modeled enrichment spend 36.6% versus a tuned waterfall and 48.9% versus full enrichment across the 140-company set, at 93.6% policy-decision agreement with both — reproduced 140/140 exactly against real production receipts in an offline replay before any number was trusted. 55% of leads stopped before the full evidence cascade; 16.4% escalated to human review.
These are modeled/simulated-evidence economics, not a real-world accuracy claim. A follow-up blinded review of 30 companies — independently fit-assessed from public information, sampled by ARIE's own output tiers — found a 30% stark-disagreement rate with real-world company fit, traced to exactly how simulated evidence is generated for any company outside ARIE's benchmark corpus. A further real-provider (Abstract + Hunter) live-shadow reality check on 8 of those companies, and a set of code-level Hunter diagnostics, investigated whether real evidence closes that gap and surfaced specific, honestly-reported provider-coverage and company-identity-validation limitations rather than resolving it.
Full write-up, including what didn't hold up and why: case-study-realistic-pilot.md.
- Postgres
SKIP LOCKEDjob queue, no Redis/Celery/Temporal — a worker claims a job and commits the lead's new status in the same transaction as marking the job complete, which a separate queue technology would reopen as a dual-write hazard. Retry with backoff, dead-lettering after repeated failure, no coordination needed between workers. ADR 0002 records the trigger for revisiting this (north of ~1k jobs/sec) — nowhere near today's scale. - Org-scoped BYOK provider credentials in Supabase Vault — a real
credential is written/read exactly once per call site, and
organization_provider_configscarries only a Vault secret pointer, never the value itself. - Two-layer live-execution safety: a process-wide
PROVIDER_MODEgate (does this deployment even have the live code path available) and a per-organizationexecution_mode(simulated/live_shadow/live_human_only) — an organization set to simulated gets genuinely simulated behavior even sharing a live worker with a live organization, not a degraded live path with zero evidence. - Identity verification before person evidence can score. A person
provider's returned name/employer is checked against what was actually
requested; only a
VERIFIEDverdict allows those fields into the scorer. Directly responsible for correctly suppressing a real, reproduced Hunter wrong-person match (see Real-provider validation above) instead of silently scoring the wrong person's title. - A deterministic boundary around LLM-assisted configuration. The M7 intelligence layer lets a customer describe their business in plain English and get a targeting/scoring profile out — but the model only interprets intent. A deterministic normalizer enforces the scoring invariants (the exactly-100.0-point ICP allocation ceiling among them); the model has no code path that lets it award itself points.
- 2,077 test functions, CI-gated on every push: ruff lint + format, mypy
strict, a migration-drift check (
supabase/migrations/must stay a byte-identical generated mirror ofmigrations/), and a real-Postgres integration job — not just unit tests with everything mocked. - Decision Receipt provenance — every receipt carries the exact policy name, scorer version, and calibration method that produced it, reconstructed from persisted state rather than re-derived, so a receipt from months ago still explains itself after the policy has moved on.
Two of the surfaces this shows up in, beyond the core decision loop:
| Ask ARIE | Find Customers |
|---|---|
 |
 |
| Plain-English questions answered from what ARIE already decided — never a re-score. | Discovery Pivot: the same evidence discipline pointed upstream, screening candidates before spending. |
I built the safe diagnostic interface I'd want before letting an AI agent
anywhere near production — not a chat wrapper around psql.
A local, stdio-only Model Context Protocol server gives Claude Code 11 read-only tools for inspecting this system's runtime state — queue health, provider errors, enrichment spend, routing decisions, schema drift:
- Backed by a dedicated Postgres role that can
SELECTfrom exactly the views in one schema and nothing else — verified live: querying a base table directly through that role raisesInsufficientPrivilege, not merely "no tool exposes it." - 5-second statement timeout enforced at the role level, not just in client code, so a bug in this server's own SQL can't hang or write past that ceiling.
- Every call — success or failure — appends to a local, redacted JSONL audit log. Email addresses, connection strings, bearer tokens and API-key-shaped strings are stripped before truncation, not after.
- No write path exists. No admin endpoints, no arbitrary SQL, no control over infrastructure — out of scope for this version, not partially wired.
Used for real: a documented debugging session traced five dead-lettered jobs to their root cause in three typed, capped, audited tool calls. Full writeup: mcp-architecture.md.
Worth being precise about, because they're easy to conflate.
The public /demo and the hosted console run in simulated mode. Known example identities replay a frozen evaluation corpus; any other identity gets deterministic synthetic evidence generated from the same provider catalogue and noise model, seeded by the lead's own email and domain — so the same lead always resolves the same way. No vendor is called and no money is spent either way, so the cost figures you see are modelled cost at configured provider rates — not billed spend. Everything around it is real: real Postgres queue, real worker, real persistence, real receipts, real human-review workflow.
Two real provider integrations have made real, billed calls: Abstract
API's Company Enrichment and Hunter's Combined Enrichment — including the
validation above. A third, Apollo's People Enrichment, is implemented and
fixture-tested but has not made a real call — deliberately out of scope for
this portfolio milestone, not blocked on anything. All three sit behind the
same EnrichmentProvider interface the simulator implements.
Live mode's default optimized strategy walks providers cheapest-first
(Abstract $0.00165 → Hunter ~$0.0049 → Apollo ~$0.0196, all modelled figures)
and stops the moment existing evidence answers the question. A private
evaluation strategy deliberately calls the person providers in parallel on
controlled identities so their coverage, quality, latency, and agreement can
be measured (scripts/provider_bakeoff.py) before any waterfall order is
declared the winner.
Details of both: provider-integration.md.
Python 3.12 · FastAPI · Postgres (Supabase) · pytest · Docker Next.js 16 · React 19 · TypeScript (strict) · Tailwind CSS v4 · Motion Vitest + Testing Library · Playwright (e2e) Railway (API + worker) · Vercel (frontend) · n8n Cloud (edge workflows) Supabase Auth + Vault · Stripe · OpenTelemetry · Model Context Protocol
Reproduce the benchmark — no API keys, no network:
pip install -e ".[dev,service]"
make dataset # generate the seeded evaluation set
make bench # single-seed benchmark
python -m bench.multi_seed # 10 seedsOr run the whole stack and watch it decide, escalate, and honour an override. Needs Docker:
.\scripts\demo.ps1The demo brings up Postgres, the API and the worker, submits a few leads from the frozen corpus, and prints their receipts.
| architecture.md | How it works, the invariants, what's where in the code |
| benchmark.md | Dataset design, measured results, every assumption |
| case-study-realistic-pilot.md | 140 real companies through the real pipeline, a blinded human-fit review, and a real-provider reality check |
| deployment.md | Hosted topology, config, migrations, rollback |
| provider-integration.md | The real adapters, live verification status, and shadow mode |
| mcp-architecture.md | The read-only MCP engineering interface Claude Code connects to |
| portfolio.md | Short explanations, resume bullets, and what not to claim |
| docs/adr/ | Decision records, including the negative result |
- Live autonomy remains hard-disabled in code, not by policy switch. A
lead enriched by a real provider always terminates at a human (or at
SHADOW_EVALUATEDif run as shadow) —tauis fitted on the synthetic calibration split, and applying it to real-provider evidence with different coverage and error modes would be an unmeasured claim wearing a calibrated number's clothes. See Live V1 Foundation. - The n=3 real-provider validation is an architecture/correctness proof, not a statistical accuracy study. No accuracy, ROI, or cost-savings figure is claimed from it, and none should be inferred.
- The synthetic benchmark and the real-provider validation are separate kinds of evidence. The benchmark proves the policy against a modelled provider/noise distribution at real statistical scale (10 seeds, 3,000 leads); the real validation proves the live architecture is wired correctly at real (tiny) scale. Neither substitutes for the other.
- Apollo has not been live-validated — contract-tested against fixtures and the vendor's published documentation only.
- No large-scale customer deployment or load validation. The only concurrency proof is small: five simultaneous submissions against the same identity, all settled correctly with no duplicate processing — not load testing. No auth/tenancy beyond a single-tenant proof.
- The public
/demois simulated by design — frozen corpus and deterministic synthetic evidence, clearly labelled, no vendor called, no money spent, regardless of what you type into it. - The cheapest-first provider order is a reasoned prior awaiting the bake-off's measurements, not a result. And the EVoI result stays open: ADR 0004 names three concrete conditions under which it might actually win — none tested here.
MIT — see LICENSE.