> ## Documentation Index
> Fetch the complete documentation index at: https://sportzdocs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> What's instrumented, end to end, and what's not built yet.

Observability rests on three pillars: **logs** (what happened), **metrics** (how it's performing), **traces** (where a request went). Here is Sportz's honest coverage of each.

## The three pillars, and Sportz's coverage

| Pillar      | Backend                         | Frontend                        |
| ----------- | ------------------------------- | ------------------------------- |
| **Logs**    | ✅ Winston                       | ✅ New Relic error tracking      |
| **Metrics** | ✅ New Relic APM + PostHog       | ✅ New Relic Browser + PostHog   |
| **Traces**  | ✅ New Relic distributed tracing | ✅ New Relic distributed tracing |

## Backend logging: Winston (built)

Winston is the log-of-record. The transport strategy is environment-aware:

* **Production** → console only. Containers should log to stdout; file logs inside a container are lost on restart and (as [ISSUE-002](/issues) showed) can crash startup on permission errors.
* **Development** → console + `logs/error.log` + `logs/combined.log`.

HTTP requests are logged via Morgan piped into Winston (`combined` format in prod, `dev` in development).

## Backend APM: New Relic (built)

The `newrelic` Node agent auto-instruments `express`, `pg`, and `http`: request timing, DB query performance, error rates, and distributed traces, all without manual instrumentation in route code. See [ADR-013](/decisions) for why New Relic was chosen over an OpenTelemetry+Sentry stack, and the ESM require-order problem its config had to solve (`src/bootstrap.ts`).

Env-gated like Arcjet: `NEW_RELIC_LICENSE_KEY` unset → `bootstrap.ts` never imports the agent, the server runs exactly as if it weren't installed.

## Backend product analytics: PostHog (built)

`posthog-node`, env-gated via `POSTHOG_KEY` (`src/posthog.ts`). Two business events captured so far:

* `match_created`: `{ matchId, sport }`, fired in `POST /matches`
* `score_updated`: `{ matchId, homeScore, awayScore }`, fired in `PATCH /matches/:id/score`

Distinct ID is `'backend'` for both, since these are system-generated events (mostly from the `DEMO_MODE` simulator), not tied to a real end user. See [ADR-014](/decisions) for why this stays separate from New Relic rather than folding in.

## Frontend observability: live

Two tools are integrated as provider components in `sportz-ui`, keys set on Vercel and confirmed present in the deployed production bundle:

* **New Relic Browser**: Core Web Vitals (LCP/CLS/INP) on real users, AJAX timing, session replay, distributed tracing linked to the backend agent above. Lazy-loaded *after* hydration so the monitoring agent doesn't hurt the LCP it measures.
* **PostHog**: product analytics, autocapture (clicks, page views), plus custom events `match_watched`, `match_closed`, `ws_reconnected`. Manual `$pageview` tracking because App Router's client navigation doesn't trigger PostHog's default auto-capture.

Both are configured via `NEXT_PUBLIC_NEW_RELIC_ACCOUNT_ID`, `NEXT_PUBLIC_NEW_RELIC_AGENT_ID`, `NEXT_PUBLIC_NEW_RELIC_LICENSE_KEY`, `NEXT_PUBLIC_NEW_RELIC_APPLICATION_ID`, `NEXT_PUBLIC_POSTHOG_KEY`, and `NEXT_PUBLIC_POSTHOG_HOST` on Vercel. `NEXT_PUBLIC_*` values are baked into the JS bundle at build time, so any future change to these needs a redeploy to take effect, not just an env var update.

<Note>
  Sentry was previously wired on the frontend (errors, session replay, source maps) but has been **fully removed**: three config files, the shared config module, and the `withSentryConfig` build wrapper are all gone. New Relic now covers frontend errors as part of the same RUM agent, and the earlier "Sentry + PostHog + New Relic, three lenses" framing no longer applies. See [ADR-013](/decisions).
</Note>

## What each tool answers

```
New Relic  → "Is it healthy, and how fast?"     (errors, performance, traces)
PostHog    → "What are users actually doing?"   (product analytics)
```

Two lenses, not three: engineering health vs. product behavior. They don't overlap.

## Not built yet

* **Custom WebSocket metrics**: subscriber counts per match, broadcast frequency, fan-out size. The data is in the subscription registry; nothing emits it yet.
* **Alerting / dashboards / thresholds**: none defined. Would follow once real (non-demo) traffic makes thresholds meaningful.
* **OpenTelemetry**: explicitly decided against for now, not deferred-by-accident. See [ADR-013](/decisions): New Relic's own agent covers the same instrumentation surface without a separate SDK, at the cost of vendor lock-in over vendor neutrality.

<Note>
  Honest summary: **logging, APM, and analytics are all live end to end, backend and frontend.** WebSocket-level metrics and alerting remain the gaps worth closing next.
</Note>