DevConnect

A social platform for developers: share posts with syntax-highlighted code, follow people, and message them in real time. Built April 2026 – June 2026.

Live demo: https://www.dev-connect.online

Real-time chat across 3 Node.js gateways behind Nginx, with at-least-once delivery and crash recovery on Redis Streams consumer groups.

Results at a glance (k6 load tests; details in Testing & performance):

• 2,015 peak concurrent WebSocket connections across 3 gateways, message delivery p95 33ms
• Failover: one gateway killed mid-test, 0 failed sends across 1,740 delivered messages
• REST API: 1,000 concurrent users, 99.88% success, p95 7ms
• Found and fixed a rate-limit bug behind Nginx: 40% failed sends in the July test, 0 of 1,831 in the re-run

The project goes past CRUD. Posts and follows are here and they work, but the focus is the chat system underneath: Redis Streams for durable message delivery, consumer groups with offline replay, presence tracking, and rate limiting that stays atomic under concurrent load. This README covers that part in detail.

Stack: TypeScript end to end. React 19 + Vite on the frontend, Node/Express on the backend, PostgreSQL via Prisma, Redis for everything real-time, Socket.IO for the WebSocket layer, Docker for local infra, deployed on Fly.io (backend) and S3 + CloudFront (frontend).

---

Architecture

The chat system is designed to run behind multiple gateway instances, with Redis as the message bus between them. A single instance can't prove anything about cross-instance delivery, so the whole point is testing it with more than one.

                  ┌──▶ Node.js gateway 1 ──┐
Browser ── Nginx ─┼──▶ Node.js gateway 2 ──┼──▶ Redis ──── write-behind ───▶ PostgreSQL
 (WebSocket)      └──▶ Node.js gateway 3 ──┘   (streams,     flusher, batch    (message
                                                pub/sub,     insert every      history)
                                                cache,       250ms
                                                WB queue)

This 3-gateway setup runs locally via Docker Compose and is what the load tests below run against. Production on Fly.io currently runs a single instance (see Design Trade-offs): the horizontal-scaling story is proven in the load-test harness, not live yet.

Redis is doing two different jobs here, deliberately:

• Pub/sub for things where losing a message doesn't matter and latency does: typing indicators, presence pings.
• Streams + consumer groups for actual chat messages: persistent, acknowledged with XACK, replayable with XREADGROUP if a recipient was offline. This is the part that has to be correct: a dropped pub/sub message is a UI glitch, a dropped chat message is a bug report.

At-least-once delivery. An entry leaves the recipient's pending entries list (PEL) only after XACK. If a gateway crashes after a delivery but before the ack, the entry stays pending.

Crash recovery. All gateways read a user's group under one shared consumer name. On the user's next connect, whichever gateway gets the socket re-reads the un-acked PEL entries (XREADGROUP from id 0) and delivers them. The worst case is a rare duplicate, not a lost message (backend/src/modules/chat/chat.streams.consumer.ts).

Sending a message, end to end

1. Sender hits POST /api/messages. Per-user and per-conversation token buckets (Lua script, atomic) check the rate limit.
2. The message is written to Redis immediately: a hot cache entry (sorted set) and a write-behind queue entry, and the request returns. Postgres hasn't been touched yet.
3. A background flusher drains the write-behind queue every 250ms and batch-inserts into Postgres. If Redis is down, step 2 falls back to a direct synchronous Postgres write instead, so nothing gets silently dropped.
4. The message is also XADDed to the conversation's Redis stream, unconditionally. This is the durability path.
5. If the recipient is online (checked via a Redis presence key), the message is pushed to them directly over their socket and immediately acknowledged on the stream, so it won't be redelivered later. If the gateway crashes before that ack, the entry stays pending and is redelivered on the next connect.
6. If they're offline, the stream entry just sits there. On their next connect, they read their pending + new entries from the stream and catch up.

The reason for writing to Redis before Postgres (write-behind) instead of the other way around: an online recipient needs the message in under 100ms, and a synchronous Postgres round-trip on every send doesn't get you there reliably under load. Postgres still ends up with everything, just a couple hundred milliseconds later.

Data model

8 Prisma models: User, Post, Comment, Like (polymorphic: one table for both post and comment likes via nullable FKs), Follow (self-referential many-to-many), Message, Notification, Hashtag. Cascade deletes throughout, so removing a user cleans up everything they touched.

---

Features

Auth & security

• JWT access tokens (15min) + refresh tokens (7d), so a leaked access token has a short blast radius
• GitHub and Google OAuth via Passport, with account linking when the email matches an existing user
• TOTP 2FA (Speakeasy), works with any standard authenticator app
• Rate limiting on every route category, all backed by the same atomic Redis Lua token-bucket script. Auth, password reset, posting, following, search, and chat each have their own budget

Chat

• Redis Streams + consumer groups for delivery, with offline catch-up on reconnect (see the flow above)
• Presence via 30-second TTL heartbeats: a user counts as online only as long as their client keeps refreshing the key
• Read receipts and typing indicators pushed over Socket.IO, fanned out across gateway instances by the Socket.IO Redis adapter
• Write-behind caching to Postgres, sorted-set hot cache for the last 50 messages per conversation
• WebSocket only, no long-polling fallback. State lives in Redis instead of in-process memory, so there's no need for sticky sessions, and Nginx load-balances connections freely

Posts & social

• Code snippets with syntax highlighting (Prism), language auto-detection
• Nested/threaded comments, hashtag extraction and trending counts
• Follow system, personalized and trending feeds

Images

• Upload via Multer, processed with Sharp (resize, WebP conversion, strip metadata), stored in S3
• Falls back to local disk storage automatically if S3 isn't configured

---

Design trade-offs

Decision	Alternative	Why	Cost
Modular monolith (9 modules, one process, one Postgres DB)	Microservices	No inter-service network calls, no distributed transactions, one thing to deploy	Scales vertically only for now
Write-behind cache for chat messages	Write straight to Postgres	Sub-100ms delivery to online recipients without waiting on a DB round-trip	Up to ~250ms where a message exists only in Redis; a crash in that window before flush would lose it
Rate limiting via Redis Lua script	Per-instance in-memory counters	Atomic under concurrent requests to the same key, and the limit is shared correctly across all gateway instances	Redis becomes a dependency for rate limiting to work at all (it fails open if Redis is down, so it degrades to "no limiting" rather than blocking traffic)
Single Fly.io instance in production	Deploy the 3-gateway/Nginx cluster live	Cheaper and simpler to operate for a low-traffic portfolio app	The cross-instance routing this project is actually about isn't running in production, it's proven locally with Docker Compose + k6, not live
Polymorphic Like table	Separate PostLike / CommentLike tables	One set of like/unlike logic instead of two	Nullable FKs, relies on composite unique constraints to stay correct

---

Tech stack

Layer	Technology
Frontend	React 19, TypeScript, Vite, Tailwind CSS 4, Zustand, TanStack Query
Backend	Node.js 20, Express, TypeScript, Zod
Database	PostgreSQL 15 via Prisma
Real-time	Socket.IO + Redis adapter, Redis Streams
Cache / bus	Redis 7 (Upstash in production, TLS)
Auth	Passport.js (JWT, GitHub/Google OAuth), Speakeasy (TOTP)
Images	Sharp, AWS S3
Infra	Docker (multi-stage build), Nginx, Fly.io, AWS CloudFront
Testing	Jest, k6, ESLint, GitHub Actions

---

Testing & performance

Unit tests: 28 tests across auth, posts, feed, the rate-limit middleware, and the trusted-proxy setup.

cd backend && npm test

Load testing (k6):

General API (auth, posts, feed, REST only, single instance):

Metric	Value
Concurrent users	1,000
Total requests	2,142,599
Success rate	99.88%
p95 latency	7ms

Real-time chat (WebSocket, 3-gateway cluster via Docker Compose, target 2,000 concurrent connections; raw numbers in backend/tests/load/results/ws-concurrency-results.json):

Metric	Value
Peak WebSocket connections	2,015
Successful connections	1,996
Message delivery p95	33ms
Message delivery avg	17ms

Failover test (300 target connections, one gateway removed with docker kill mid-run): the remaining two gateways kept serving. 1,740 messages were delivered with 0 failed sends. Delivery latency held at p95 29ms overall, but the worst case was 8.2 seconds for clients that were reconnecting from the killed gateway. The results also record 200 WebSocket connect errors during that reconnect storm. Those clients retried and reconnected, and no messages were lost. The 8.2-second tail is the next thing to improve: it is the gap between "the system survives a gateway dying" and "no client notices." Raw numbers: backend/tests/load/results/ws-failover-results.json.

A bug the load test found (October 2026). The July WebSocket run had 1,718 of 4,320 send attempts fail, about 40% (2,602 succeeded). Every failure was a 429 from the per-IP rate limiter, not a crash or a lost message. The real cause: the gateways sit behind Nginx, but Express did not trust the proxy, so req.ip was Nginx's address for every user and all users shared one rate-limit bucket. Behind any proxy, one busy user could block everyone.

• Fix: configureTrustProxy (backend/src/shared/middleware/trustProxy.ts) trusts exactly TRUST_PROXY_HOPS proxies (1 in Docker Compose, 0 by default so a client cannot fake its IP when the app is exposed directly). 4 unit tests cover it.
• Proof, two users through Nginx: before the fix, user B was blocked on its first request after user A used the limit; after it, each user had its own bucket. Details and script: backend/tests/load/proofs/2026-10-21-trust-proxy.md, two_users.sh.
• Load test re-run (2026-10-22): k6 runs from one machine, so it is one client to a per-IP limiter. The re-run raised the per-IP limits for this run only (RATE_LIMIT_MAX_REQUESTS=1000000, AUTH_RATE_LIMIT_MAX_REQUESTS=1000000). All 300 test users registered. 0 of 1,831 send attempts failed (1,831 got 201), and 1,497 of 1,500 target WebSocket connections authenticated. This shows the limiter was the only cause of the July failures. Raw numbers: backend/tests/load/results/ws-concurrency-limit-raised-2026-10-22.json. The delivery latency in this run (p95 87ms) is not comparable to the July run, because a Kubernetes cluster ran on the same laptop. The run proves failure counts only. ws-concurrency-results.json is still the original July run.

cd backend && npm run test:load:ws         # concurrency test
cd backend && npm run test:load:failover   # gateway kill test
cd backend && npm run test:load:quick      # 30s smoke test

---

What's next

• Get the failover reconnect latency down. Right now a killed gateway causes a multi-second tail for clients that were mid-connection to it.
• Actually deploy the 3-gateway/Nginx setup to production instead of a single Fly.io instance, so the horizontal-scaling design is live, not just load-tested locally.
• Push the WebSocket concurrency test past 2,000 toward 5,000+ connections.
• End-to-end tests (Playwright). There are unit tests for logic and load tests for throughput, but nothing that walks a full user flow (register, post, message someone).
• Full-text search. Posts/users search is currently Prisma contains queries, no fuzzy matching or ranking.

---

Project structure

backend/src/
├── server.ts                    # entry point, Socket.IO + HTTP server setup
├── modules/
│   ├── auth/                    # registration, login, OAuth, 2FA, password reset
│   ├── users/                   # profiles, follow/unfollow
│   ├── posts/                   # CRUD, code snippets, hashtags, likes
│   ├── feed/                    # personalized, trending, following
│   ├── chat/                    # messaging, hot cache, unread counts, stream consumer
│   ├── notifications/
│   ├── search/
│   ├── uploads/                 # image pipeline
│   └── metrics/
└── shared/
    ├── redis/                   # streams client, write-behind flusher, Lua token bucket
    ├── socket/                  # Socket.IO auth, presence, Redis adapter
    ├── middleware/               # auth, rate limiting, uploads
    └── database/                 # Prisma client, Redis client

frontend/src/
├── pages/                        # 14 route-level pages
├── components/                   # ui/, posts/, chat/, profile/, common/
├── api/, hooks/, store/          # Axios clients, React Query hooks, Zustand store

nginx/nginx.conf                  # load balancer config for the 3-gateway local cluster
docker-compose.yml                # postgres, redis, gateway1-3, nginx

---

API overview

Everything's mounted under /api. Protected routes need a Bearer token.

Module	Base path	Notes
Auth	/api/auth	register, login, refresh, OAuth, 2FA, password reset
Users	/api/users	profiles, avatars, follow/unfollow
Posts	/api/posts	CRUD, likes, comments
Feed	/api/feed	personalized, trending, following
Messages	/api/messages	conversations, send, mark read, delete
Search	/api/search	users, posts, hashtags
Notifications	/api/notifications	list, unread count, mark read
Uploads	/api/uploads	images (avatar, banner, post)

62 routes total across 9 modules.

---

Running it locally

Requirements: Node 20+, Docker.

git clone https://github.com/mmerlyn/dev-connect.git
cd dev-connect

cd backend && npm install && cd ../frontend && npm install && cd ..
cp backend/.env.example backend/.env   # fill in secrets, see below

Option A: just the app, single backend instance:

docker compose up postgres redis -d
cd backend && npx prisma migrate dev && npx prisma generate
npm run dev              # backend on :4000
cd ../frontend && npm run dev   # frontend on :5173

Option B: the full 3-gateway cluster, same setup the load tests run against:

cp .env.example .env     # Docker Compose reads POSTGRES_PASSWORD etc. from here
docker compose up -d     # postgres, redis, gateway1-3, nginx, all of it

This puts the API behind Nginx on :8080, load-balanced across three backend containers. Compose sets TRUST_PROXY_HOPS=1 for each gateway (see below).

Environment variables

# Required
DATABASE_URL=postgresql://devconnect:yourpassword@localhost:5432/devconnect
JWT_SECRET=your-secret-key
JWT_REFRESH_SECRET=your-refresh-secret
FRONTEND_URL=http://localhost:5173

# Redis: chat, caching, and rate limiting all degrade gracefully without it,
# but it's required for anything to feel "real-time"
REDIS_HOST=localhost
REDIS_PORT=6379

# Optional: OAuth login buttons won't show without these
GITHUB_CLIENT_ID=...
GITHUB_CLIENT_SECRET=...
GOOGLE_CLIENT_ID=...
GOOGLE_CLIENT_SECRET=...

# Optional: falls back to local disk storage without these
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_S3_BUCKET=...

# Optional: verification emails won't send without this
SENDGRID_API_KEY=...

# Reverse-proxy hops in front of the backend. 0 (default) = no proxy, so
# X-Forwarded-For is ignored. Set 1 behind one proxy such as Nginx, or
# every user shares one per-IP rate-limit bucket. Docker Compose sets 1.
TRUST_PROXY_HOPS=0

# Rate limits per IP. Defaults are the real app values; load tests from one
# machine raise them because every virtual user shares that machine's IP.
RATE_LIMIT_MAX_REQUESTS=100
AUTH_RATE_LIMIT_MAX_REQUESTS=10

---

License

MIT

Merlyn Mercylona · GitHub · LinkedIn