Building async CDC engines, runtime schema guards, and local OLAP pipelines with Python, DuckDB, and Polars.
Data Infrastructure, Async Backend, Developer Tooling. Nairobi, Kenya.
async-cdc-engine: AsyncIO listener that parses CDC log events and applies state mutations without blocking the event loop.data-stream-pipeline: Multi-stage stream processor. FastAPI ingress, Polars LazyFrame transforms, DuckDB sinks.scraper-engine/scraper_tracker: Crawl4AI async crawlers with crawl state persisted in SQLite/DuckDB.
analytics-dbt-duckdb: In-process OLAP modeling with dbt-duckdb. No warehouse server, no network hop.local-elt-pipeline: End-to-end ELT orchestrated with Docker Compose. One command to bring up the stack.
schema-guard: Validates incoming JSON against structural contracts before anything reaches storage.data-ops-dashboard: Pipeline health, latency tracking, and monitoring UI.
- AsyncIO over threads. The workload is I/O-bound: a socket reading log events and small state updates. A single event loop avoids per-thread stack memory, GIL contention, and lock management around shared state. CPU-heavy steps (if any) get pushed to an executor so they do not stall the loop.
- Mutations are applied in log order. State changes are sequential per stream, which keeps replay deterministic after a reconnect.
- Cost: one blocking call in a handler stalls every consumer. Handlers must be strictly non-blocking.
- Validate at the boundary. Contract checks run before the storage load, so bad payloads are rejected or quarantined instead of corrupting tables downstream.
- Runtime enforcement over load-time failure. A drifted field is caught on the offending record, not discovered as a failed batch.
- Cost: per-record validation adds CPU on the hot path. It is the price of catching drift before it lands.
- DuckDB is in-process. Models run inside the dbt process against local files. No connection pool, no server to operate, no network latency.
- Columnar scans over row-store. Aggregations over wide tables read only the columns they touch. PostgreSQL stays the choice for concurrent writes and transactional workloads; DuckDB covers single-writer analytical work.
- Zero-copy reads from Parquet/Arrow avoid materializing intermediate copies between stages.
- Cost: single-writer model. Not a fit for many concurrent writers.
- Polars LazyFrames over Pandas. The query plan is built first and optimized (projection and predicate pushdown) before execution. Pandas executes eagerly and materializes every intermediate.
- FastAPI for ingress because it is async-native and shares the event loop with the rest of the pipeline.
- DuckDB as sink keeps the analytical output queryable in-process with no separate load step.
| Domain | Tools |
|---|---|
| Languages | Python 3.11+, SQL, TypeScript, JavaScript (ES6+), HTML5, CSS3 |
| Async & ingestion | AsyncIO, FastAPI, Crawl4AI |
| Analytical engines | DuckDB, Polars, dbt-core, PostgreSQL, SQLite |
| Infrastructure | Docker, Docker Compose, WSL (Ubuntu), Git, Vercel |
| Frontend | Next.js (App Router), React, Tailwind CSS |
Forge Digital Solutions is my design and development studio. It covers production web interfaces (Next.js, React, Tailwind, deployed on Vercel), data automation, and API integrations.
- Site: benwam.vercel.app
- LinkedIn: in/wambuabenny-007