Skip to content
View BenWam254's full-sized avatar
:atom:
Ready to work
:atom:
Ready to work

Block or report BenWam254

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
BenWam254/README.md

Benny Wambua · BenWam254

Building async CDC engines, runtime schema guards, and local OLAP pipelines with Python, DuckDB, and Polars.

Data Infrastructure, Async Backend, Developer Tooling. Nairobi, Kenya.

Python AsyncIO DuckDB Polars dbt FastAPI PostgreSQL Docker TypeScript


Systems & Architecture

Stream Processing & CDC

  • async-cdc-engine: AsyncIO listener that parses CDC log events and applies state mutations without blocking the event loop.
  • data-stream-pipeline: Multi-stage stream processor. FastAPI ingress, Polars LazyFrame transforms, DuckDB sinks.
  • scraper-engine / scraper_tracker: Crawl4AI async crawlers with crawl state persisted in SQLite/DuckDB.

Analytical Data Engines

  • analytics-dbt-duckdb: In-process OLAP modeling with dbt-duckdb. No warehouse server, no network hop.
  • local-elt-pipeline: End-to-end ELT orchestrated with Docker Compose. One command to bring up the stack.

Runtime Reliability

  • schema-guard: Validates incoming JSON against structural contracts before anything reaches storage.
  • data-ops-dashboard: Pipeline health, latency tracking, and monitoring UI.

Design Notes

async-cdc-engine

  • AsyncIO over threads. The workload is I/O-bound: a socket reading log events and small state updates. A single event loop avoids per-thread stack memory, GIL contention, and lock management around shared state. CPU-heavy steps (if any) get pushed to an executor so they do not stall the loop.
  • Mutations are applied in log order. State changes are sequential per stream, which keeps replay deterministic after a reconnect.
  • Cost: one blocking call in a handler stalls every consumer. Handlers must be strictly non-blocking.

schema-guard

  • Validate at the boundary. Contract checks run before the storage load, so bad payloads are rejected or quarantined instead of corrupting tables downstream.
  • Runtime enforcement over load-time failure. A drifted field is caught on the offending record, not discovered as a failed batch.
  • Cost: per-record validation adds CPU on the hot path. It is the price of catching drift before it lands.

analytics-dbt-duckdb

  • DuckDB is in-process. Models run inside the dbt process against local files. No connection pool, no server to operate, no network latency.
  • Columnar scans over row-store. Aggregations over wide tables read only the columns they touch. PostgreSQL stays the choice for concurrent writes and transactional workloads; DuckDB covers single-writer analytical work.
  • Zero-copy reads from Parquet/Arrow avoid materializing intermediate copies between stages.
  • Cost: single-writer model. Not a fit for many concurrent writers.

data-stream-pipeline

  • Polars LazyFrames over Pandas. The query plan is built first and optimized (projection and predicate pushdown) before execution. Pandas executes eagerly and materializes every intermediate.
  • FastAPI for ingress because it is async-native and shares the event loop with the rest of the pipeline.
  • DuckDB as sink keeps the analytical output queryable in-process with no separate load step.

Stack

Domain Tools
Languages Python 3.11+, SQL, TypeScript, JavaScript (ES6+), HTML5, CSS3
Async & ingestion AsyncIO, FastAPI, Crawl4AI
Analytical engines DuckDB, Polars, dbt-core, PostgreSQL, SQLite
Infrastructure Docker, Docker Compose, WSL (Ubuntu), Git, Vercel
Frontend Next.js (App Router), React, Tailwind CSS

Client Work

Forge Digital Solutions is my design and development studio. It covers production web interfaces (Next.js, React, Tailwind, deployed on Vercel), data automation, and API integrations.

Pinned Loading

  1. analytics-dbt-duckdb analytics-dbt-duckdb Public

    Analytics engineering with dbt-core and DuckDB: raw -> staging -> Kimball star schema marts, with dbt test and lineage docs.

  2. async-cdc-engine async-cdc-engine Public

    Async ingestion and hash-based Change Data Capture engine, asyncio and httpx scrape concurrently, Redis buffers payloads, Polars computes SHA-256 row hashes, and DuckDB gets INSERT/UPDATE/UNCHANGED…

    Python

  3. data-ops-dashboard data-ops-dashboard Public

    High-density Next.js 15 control plane UI for real-time monitoring, telemetry aggregation, and job dispatching across a distributed web ingestion ecosystem.

    TypeScript

  4. data-stream-pipeline data-stream-pipeline Public

    High-throughput real-time stream processor and micro-batching engine built with Polars, PyArrow, and aiokafka. Consumes Redpanda event streams and persists Snappy-compressed, partition-pruned Parqu…

    Python

  5. local-elt-pipeline local-elt-pipeline Public

    Automated weather ELT batch pipeline: httpx extracts from Open-Meteo, Polars transforms, and idempotent loads go into DuckDB. Runs daily on Github Actions.

    Python

  6. schema-guard schema-guard Public

    Declarative data contract enforcer and schema drift detection CLI built with Polars, Typer, and Pydantic. Validates Parquet data lakes against YAML contracts for automated data quality assurance.

    Python