Statistics, not stack traces

Surface: Sentry API tooling, Zod-issue analysis, editor normalizers · Result: 12,587 → 977 events; 30 → 11 error signatures · Stack: Node.js (zero dependencies) · Sentry REST API · Zod · Slate/Plate editor

The brief: one issue, twelve thousand events, zero insight

Every document in the platform is a JSON tree of editor nodes, and every save passes through a Zod schema that describes what a valid tree looks like. The validation was deliberately observational — log and report, never block a save — because nobody knew yet how much real-world data would fail it. The answer turned out to be: a lot. Each failing save sent one event to Sentry carrying the document ID, team ID, version, and the full array of Zod validation issues as payload.

Sentry groups events by fingerprint, and since every one of these had the same message, they collapsed into a single issue with thousands of events behind it. That grouping is exactly wrong for this data: the message carries no information, and all the variance — which property failed, which block type produced it, whose documents are affected — sits inside a JSON payload the Sentry UI only reveals one event at a time, one click each. Twelve thousand clicks was not a plan. The events had to come out of Sentry and into a format where they could be counted.

 Sentry issue — one fingerprint, 12,587 events, variance buried in payloads
      │
      │  1 · export     cursor pagination · per-event detail fetch · 429 backoff
      ▼
 out.jsonl / out.csv — one row per event: docId, teamId, version, issues[]
      │
      │  2 · flatten    recurse into Zod's nested unions → leaf errors
      │  3 · group      strip indices from paths → signatures, ranked by count
      ▼
 30 signatures with counts ──▶ two fix PRs ──▶ re-export, same window: −92%

01 · Export — a zero-dependency exporter for one Sentry issue

The extraction tool is a single Node script — no packages, just built-in fetch and streams. Configured by environment variables (org, project, issue short ID, an optional message filter, a stats period defaulting to 14 days), it resolves the issue's short ID to its numeric ID, then walks the full event list via Sentry's Link-header cursor pagination, 100 events per page.

The list endpoint only returns event summaries, so each event costs a second request for its full body. That makes rate limiting a certainty, not a risk — the fetch wrapper honors 429 Retry-After with a capped backoff and retries up to five times. Results stream straight to disk as they arrive: a JSONL file with the raw extracted rows, and a flattened CSV for spreadsheet work.

The least glamorous function earned its keep the most. Sentry has moved the "extra" payload around across API versions and event shapes, so extraction probes five locations in order:

function pickExtraField(event, key) {
  // 1. event.extra[key]              — the modern location
  // 2. entries[type="extra"].data    — the entries array
  // 3. event.context[key]            — older API shape
  // 4. event.contexts.extra[key]     — contexts object
  // 5. event.tags                    — extras sometimes promoted to tags
}

A DEBUG=1 mode dumps the first raw event to disk with its top-level keys printed — the tool for discovering which of the five shapes this particular project speaks, written after the first shape guessed wrong.

02 · Flatten — turning Zod's error trees into countable signatures

A Zod failure on a document isn't one error — it's a tree. Union types produce invalid_union issues containing nested arrays of sub-errors, one branch per union member that was tried. The analysis script recurses through that structure, accumulating path segments, until it reaches leaf errors:

function collectLeafIssues(error, parentPath = []) {
  const currentPath = [...parentPath, ...(error.path || [])]

  if (error.code === 'invalid_union' && Array.isArray(error.errors)) {
    // one nested error array per union branch — recurse into all of them
    ...
  }
  // leaf: { path, code, message, expected }
}

Then the key normalization: a leaf's path is something like nodes[4].children[3].children[2].children[0].text. The indices are noise — they say where in one particular document the error sits, not what kind of error it is. Dropping numeric segments and the container key, and keeping the last few property names, collapses that path to the signature children.children.text. Group by signature, then by error code and message, count, sort descending — and 167,738 leaf errors across 12,587 events reduce to 30 rows.

The pivot. Twelve thousand events you read one by one is anecdote. Thirty signatures with counts is a dataset. Nothing about the underlying errors changed — only the unit of analysis.

03 · Read — what the ranking said

Signature	Failure	Count
children.children.text	expected string, received undefined	67,558
children.children.id	expected string, received undefined	33,582
children.text	expected string, received undefined	16,870
data.languages.language	expected string, received null	8,586
request.bodyParams.type	invalid enum option	5,627
children.children.type	invalid enum option	3,411
data.noFollow	expected boolean, received undefined	2,709
data.tryItEnabled	expected boolean, received undefined	2,584

The ranking carried three separate diagnoses that no amount of event-by-event reading would have surfaced:

• One family dominated. The top three signatures — text leaves and node IDs missing at various nesting depths — were 70% of all leaf errors and pointed at the same root-cause family: nodes entering documents without the invariants the editor assumes.
• Some errors were the validator's fault. Enum failures on API-block parameter types traced to capitalized legacy values the schema refused but the product handled fine — a schema fix, not a data fix.
• Some were born broken. Undefined booleans like data.noFollow came from element factories and importers that never set defaults — every document they had ever created failed validation from day one.

A second script drilled into the #1 signature specifically: same flattening, but grouped by document — unique document counts per depth category, per-doc error counts, example paths, team and space IDs. That answered the operational questions: is this a handful of pathological documents or spread everywhere, and whose are they? A third variant produced per-document repair lists with deduplicated paths, for walking individual broken documents.

04 · Fix — two PRs, aimed by the counts

The fixes landed in two passes over the week following the export, shaped directly by the ranking — highest-count signatures first, schema corrections where the validator was wrong, data-source fixes where the data was wrong:

• Pass one (+342/−71 across 23 files): normalization overrides added to the editor's plugin layer — an ID normalizer for nodes missing id, plus per-plugin data normalizers for the API block, code editor, CTA button, HTML embed, image caption, and split-layout blocks — so malformed nodes are repaired the moment a document opens, not flagged forever. Importer fixes stopped one source of bad enum casing at its origin.
• Pass two (+333/−121 across 20 files): the Markdown-to-editor converter learned to always emit text leaves (the #1 signature's largest source), the code block gained a language fallback for null, image and iframe fields got coercion and defaults, and the API parameter-type schema was widened to accept the legacy capitalized values — with the now-unnecessary runtime normalization deleted.

Decision. Repair at the boundary, not in the database. Editor-side normalizers and importer defaults fix documents lazily as they're touched — no bulk migration over millions of rows, no risk window, and newly created documents are simply born valid.

05 · Re-measure — same instrument, two weeks later

The point of building the pipeline instead of clicking through events: measurement is repeatable. Two weeks after the first export, the same script with the same 14-day window ran again.

Metric	Jan 14	Jan 28	Change
Events (14-day window)	12,587	977	−92%
Leaf validation errors	167,738	10,508	−94%
Distinct signatures	30	11	−63%
data.languages.language	8,586	0	gone
request.bodyParams.type	5,627	0	gone
children.children.type	3,411	21	−99%
children.children.text	67,558	5,274	−92%

Whole signature classes went to zero; the dominant family dropped an order of magnitude, with the remainder tapering as lazily-normalized documents get touched. The second export also quietly validated the tooling itself: the first run had 1,414 JSONL lines that failed to parse — oversized payloads truncated mid-line — while the re-export had none, so the before/after comparison slightly understates the first run's error volume rather than inflating the improvement.

06 · By the numbers

Component	Measure
Exporter script	~300 lines · 0 dependencies
Extra-field fallback chain	5 payload locations probed
Export volume	12,587 events · 2 requests each
Analysis output	167,738 leaf errors → 30 signatures
Fix PRs	23 + 20 files · +675 / −192
Outcome (same 14-day window)	−92% events · −94% errors

07 · Retrospective — decisions that held up

• Observe before you enforce. Log-only validation on the save path meant discovering the real failure landscape — including the validator's own bugs — without ever blocking a user's save on a schema that wasn't ready.
• When the tool groups wrong, export. Sentry's fingerprint grouping is built for stack traces, not payload variance. Rather than fighting the UI, pull the raw events and impose the grouping the data actually needs.
• Normalize paths into signatures. Dropping array indices was the single highest-leverage line in the pipeline — it's what turned 167,738 data points into 30 decisions.
• Fix by count, not by recency. The loudest recent error is not the biggest one. Three signatures covered 70% of the volume; that's where the two PRs went, and the re-export confirmed the aim.
• Re-measure with the same instrument. The identical script, window, and grouping turned "we think it's better" into −92%, with the residual precisely itemized for the next pass.

---

All figures are measured directly from the export files, analysis outputs, and fix PRs at the time of writing. Identifying details — product name, error messages, issue IDs, and code identifiers — have been altered or omitted. Companion pieces: An AI agent that rewrites your documentation, Git-style branching for documents, and Compiling links at publish time.