Skip to main content
Staff-level questions are less “how does it work” and more “how do you reason about evolving it, what would you trade, and how do you lead through ambiguity.” Each answer demonstrates judgment, not recall.

​
Architecture evolution

At current scale the monolith is correct: one process serving REST + WS is simpler to deploy, reason about, and debug, with no inter-component network hops. Premature splitting buys distributed-systems cost (network failure modes, deploy coordination, tracing) for no benefit. Evolution, driven by measured pressure: (1) horizontally scale the same monolith behind a load balancer + Redis pub/sub for cross-instance broadcast; (2) extract a service only if a concern (e.g. event ingestion) develops independent scaling/availability needs. Trigger is data, not aesthetics.
Auth is absent by design (public demo). Add it at the edge: a provider (Clerk/Auth.js) + a users table + middleware after Arcjet, before routes. The WS upgrade validates a token during the handshake. Keep it additive: read paths can stay public, write paths gate. CSRF becomes relevant once cookie sessions exist; until then N/A. The key is that the gate points (HTTP middleware chain, WS upgrade handler) already exist as clean seams.
Today lib/types.ts mirrors the backend’s Drizzle-inferred types manually, fine for one team, but it can drift. The staff move: generate an OpenAPI spec from the existing Zod schemas (zod-to-openapi), making validation the single source of truth, then generate the client types from that spec. Contract tests in CI then fail the build on drift. This converts a discipline-dependent invariant into an enforced one.
Currently commentary.eventType is a free text string and metadata is jsonb: flexible but unvalidated per-sport. At scale you’d want per-sport event-type validation (a discriminated union in Zod keyed by sport) while keeping the storage flexible (jsonb). The tradeoff: stricter validation vs the cost of modeling every sport’s events. Stage it: validate the common types first, leave metadata open for the long tail.

​
Tradeoffs

Hand-rolled is right here: small protocol, minimal deps, full understanding, no need for presence/acks/fallbacks. Socket.IO becomes right when the protocol grows (presence, delivery acks, binary frames, transport fallback), at which point reimplementing its feature set is the worse trade. Meta-principle: adopt the dependency when its feature set exceeds what you’d responsibly build and maintain yourself.
The fix is trivial; the decision of how isn’t (extra existence query vs catching PG error 23503: a round-trip vs coupling to driver internals). Rather than rush a tradeoff, we pinned current behavior in a test with an explanatory comment, making any future fix deliberate and reviewed. A documented, test-guarded known-gap beats a hasty fix that bakes in the wrong choice. (ISSUE-004.)
It’s a classic build-vs-buy. Mintlify: zero config, polished instantly, but a ‘Powered by’ badge on the free tier and vendor dependency. Nextra: fully owned, no badge, deploys like the rest of the stack, but more setup. The deciding question isn’t the tools; it’s which constraint dominates: time-to-value (Mintlify) or ownership/white-label (Nextra). Naming the dominant constraint is the staff skill; the tool falls out of it.
Client cache helps a single user’s repeat views. It does nothing for cross-user load on a hot endpoint: 10k users each cold-fetching the same match list still hammer the DB. That’s when a server-side cache (Redis) earns its place. The tradeoff is cache invalidation complexity and an extra moving part; you pay it only when measurements show the DB is the bottleneck, not preemptively.
Treat realness as a ladder, each rung a different question. Rung 1 (the default, in CI): mock REST/WS at the network boundary so the suite is deterministic, fast, and offline: it answers ‘does the UI behave given a contract.’ Rung 2: run the same specs against a locally-running real backend to catch contract drift. Rung 3: a thin smoke subset against the deployed environment to catch infra/config issues. The same spec files point at each via PLAYWRIGHT_BASE_URL. Keep the high-realness rungs small and out of the fast feedback loop, since they trade determinism for coverage, so they belong in nightly/pre-release, not on every push. The principle: don’t make one test tier carry two jobs; let each rung answer its own question.
First reframe: ‘rerun until green’ is the team silently accepting that a failure carries no signal, and the real cost is that a genuine regression now hides in the noise. Diagnose the class of flake before patching symptoms: ours was environment (dev-server compilation under parallelism), provable by a serial run going green, and fixed at the environment (prod build + isolated port), not by sprinkling retries or waits. Establish the norm that flakiness is a P-class bug, mocked-at-the-boundary is the default for determinism, and retries exists to absorb rare true-flake, not to mask a structural one. The leadership move is restoring trust that red means red.
Consistency of tooling isn’t the goal; fit-to-purpose is. The backend is a long-running stateful server (live WS connections, DB pool, no CDN story); a container fits it. The frontend is client-rendered and leans on infrastructure Vercel manages: global CDN, image optimization, per-PR preview URLs, TLS, scaling. Self-hosting it in a container means re-owning all of that for no benefit at this scale. So we keep a gated standalone Dockerfile for own-infra/compliance optionality (and learning), but Vercel is the live deploy. The staff skill is naming the dominant constraint per workload, not applying one deployment model uniformly. Revisit if data-residency/compliance or cost-at-scale makes self-hosting worth the re-owned complexity: the image is ready.
Defensible for what it is: the goal is a portfolio demo that’s always live for any visitor with zero external infra. Running in-process is the simplest thing that works, and bypassing the public HTTP gate is correct: Arcjet exists to throttle untrusted anonymous traffic, not a trusted first-party producer; you don’t rate-limit yourself. The limit, stated honestly: co-locating the producer inside the web server is a demo shortcut. A real ingestion pipeline would be a DECOUPLED service (or a queue consumer) with its own service-level auth, so it scales and deploys independently of the API, and so a slow producer can’t tie up request threads. The tell that you’ve outgrown it: needing more than one API instance, real (non-simulated) feeds, or backpressure control. The staff move is shipping the pragmatic version while naming exactly what would replace it and when.

​
Incident response

Root cause: no graceful shutdown, so SIGTERM exits immediately, killing in-flight WS uncleanly. Short-term: client reconnect-with-backoff (already built) masks it. Real fix: a SIGTERM handler that stops accepting new connections, calls the WS layer’s close() (exists; stops heartbeat + closes server), drains HTTP, closes the DB pool, then exits. Add a deploy health-gate so traffic shifts only after the new instance is ready.
Respond: without APM, localize via Render request logs + Winston (route vs DB?), and Neon’s dashboard for slow queries / connection saturation. Prevent: this incident is the business case for the backend-observability gap, so add OpenTelemetry on Express + pg for route/query timings, then p95 alerts. The discipline: let the incident convert a known gap into prioritized, justified work rather than speculative instrumentation.
First isolate: is it delivery (WS) or persistence (DB)? Check whether the event is in the DB (it is: POST returned 201). So it’s broadcast. At a single instance, suspect dropped/slow sockets (backpressure on ws.send) or clients silently disconnected (heartbeat hasn’t reaped them yet). At multiple instances, this is the in-process-registry limitation manifesting: viewers on instances that didn’t originate the event. That symptom is itself the signal you’ve outgrown single-instance broadcast.
The HTTP path catches Arcjet errors and returns 503: it fails closed (denies traffic). For a security gate that’s defensible (don’t serve unprotected), but it couples your availability to Arcjet’s. The staff discussion: decide deliberately between fail-closed (secure, less available) and fail-open (available, briefly unprotected) per risk tolerance, and make it configurable rather than incidental.

​
Scaling deep-dive

10–1k: single instance, everything holds. 1k–10k: watch pg pool size and WS memory (each connection holds a subscription Set); still one instance. 10k–100k: need multiple instances for CPU/availability; the moment you do, in-process broadcast breaks, so introduce Redis pub/sub, and pooled DB connections become essential. 100k–1M: read replicas / caching for hot reads, possibly sharding matches across channels, CDN for static frontend, and dedicated event-ingestion if write volume demands it. Each step is triggered by a measured limit, not a calendar.
30s ping to every client is O(connections) work every 30s on one thread. At 100k connections that’s a measurable periodic stall. Tune: stagger pings across the interval, or offload to a worker, and consider raising the interval (trading faster ghost-reaping for less overhead). The principle: a constant-cost loop becomes a hotspot once the constant is large.
It breaks the moment there are more than ~100 matches: the app only ever fetches the 100 most-recent and paginates those in the browser, so older matches are simply unreachable (the route accepts only limit, no offset/cursor). It’s the right call at demo scale (few matches, and it lets a WS-created match appear on page 1 with no refetch), but it doesn’t scale. The fix is server-side cursor (keyset) pagination: ?limit=100&after=<createdAt|id> → WHERE createdAt < :cursor ORDER BY createdAt DESC. Prefer cursor over offset specifically because this is a real-time feed: new matches are constantly inserted at the top, so offset pagination would drift (duplicates/skips between pages), while a fixed cursor anchor stays stable. The nuance to name: you’d keep the client-side/optimistic-insert behaviour for the live page and layer cursor pagination underneath for depth.

​
Leadership & judgment

Validate the instinct, then redirect with principle: infrastructure is added when a measured constraint demands it, not speculatively. Walk through what would force Redis (multi-instance broadcast) and why we’re not there (one instance handles current load). Make it a teaching moment on premature complexity and the carrying cost of unused infra, and capture the trigger condition in the docs so it’s added at the right time by anyone, not lost as tribal knowledge.
Document the why and the non-obvious: decisions with alternatives (ADRs), real failures (post-mortems), and the gap between what exists and what doesn’t (project status). Don’t document what the code already states clearly. The test: would a competent engineer reconstruct this reasoning from the code alone? If no (a tradeoff, a deferred fix, a scale trigger), document it. Sportz’s handbook is built on exactly this split.
Three things, all present: a test suite that fails loudly on regressions (95 backend tests across unit/integration/WS, plus a Playwright e2e + a11y suite covering the UI), a decision log explaining why things are the way they are (so I don’t ‘fix’ a deliberate choice), and post-mortems documenting how it behaves under failure. That triad (tests, decisions, failures) is what makes a system safely evolvable by strangers. It’s the explicit design goal of this handbook.