Applied LLM systems / email automation
Emmy — thread-aware Gmail categorization
A production Gmail auto-categorization system that routes known senders by deterministic rule and classifies the rest with a thread-aware LLM, writing every decision back as a native Gmail label — and logging why.
The problem
A client needed incoming Gmail sorted automatically into their own operational categories, applied as native Gmail labels so the sorting lives in the inbox they already use — not a separate app. Naively sending every message to an LLM is slow, expensive, and inconsistent across a conversation; a pure-rules filter can't handle the ambiguous mail that actually needs judgment. The system also had to categorize new mail fast enough to feel live, and be auditable and correctable when it got a call wrong.
Constraints
- Near-real-time requirement ruled out cheap batch processing — new mail must be labeled promptly, not hours later.
- Operates on real user inboxes: OAuth tokens and full message bodies are sensitive and stay server-side; no inbox content is exposed.
- Every LLM call costs money and latency, so known senders must never reach the model.
- Decisions had to stay consistent within a thread — a conversation split across categories is worse than no sorting.
- Corrections needed to be cheap to act on and to influence future decisions without a retraining loop.
Approach & key decisions
Two-layer classifier: deterministic Contact Group rules first, LLM only for the rest.
Known senders are the common case and don't need judgment. Matching a sender email or domain to a single-category group short-circuits to that category with zero model cost and full confidence by construction, so the LLM budget is spent only where it buys something — a real cost/latency/accuracy trade-off, not a buzzword.
Thread-level consistency enforced end to end.
The classifier reads full conversation history and keeps every message in a thread in one category; moving one message re-routes the whole thread. Per-message drift within a conversation is the failure mode users notice most.
Structured-context prompting instead of dumping raw text at the model.
The prompt encodes concrete signals — the owner's To/CC/BCC role, internal-vs-external domain, reply/forward detection, timing, attachments, and per-category descriptions — and the model must return strict JSON with a category plus reasoning, falling back to Uncategorized on any parse or validation failure. Concrete signals and a strict contract make the output usable and safe to automate.
Rejected the OpenAI Batch API, blind thread routing, and two-step AI.
Batch's 24-hour turnaround is wrong for a near-real-time inbox; the added complexity of multi-pass AI wasn't justified by the accuracy it bought. Knowing when not to add a layer is the point.
A lightweight correction feedback loop rather than fine-tuning.
User corrections are stored as training data and the last 10 are injected into subsequent prompts. It adapts to the client's real conventions immediately, with no retraining pipeline to own.
What I built
- Built the Pub/Sub-to-background-job ingestion path: a Gmail push webhook decodes the notification and dispatches an Inngest event that categorizes mail in small batches and auto-applies labels — a real pipeline, not a synchronous loop.
- Implemented the deterministic Contact Group layer: match by sender email or domain, and short-circuit single-category groups to their category with no LLM call.
- Wrote the structured-context prompt and strict-JSON contract; used a fast single-pass configuration (reasoning effort off, low temperature) for deterministic classification, with a separate model call powering sender-to-group suggestions from a sender's recent history.
- Wrote decisions back as native Gmail labels through a label manager, including restore-all-to-inbox logic on account deletion so uninstalling leaves the inbox clean.
- Persisted a structured JSON reasoning log and confidence score per email to Postgres via Prisma, surfaced in a dashboard modal for full decision transparency.
- Raised OpenAI concurrency from 3 to 8 parallel calls under a p-limit rate limiter, the change behind the reported ~2–3× processing speedup.
- Built the Next.js dashboard: category manager, Contact Group CRUD with bulk email upload, setup wizard, prompt editor, training panel, and a thread-grouped inbox view.
Outcomes
- ~2–3× ~2–3× faster processing after raising concurrency from 3 to 8 parallel callsSelf-reported
- 0 model calls Known senders in single-category Contact Groups route with zero LLM cost and full confidence by constructionDocumented
- rejected OpenAI Batch API evaluated and rejected as unfit for near-real-time sorting (up to 24h latency) — a documented trade-offDocumented
- per-email logs Every categorization is auditable: structured reasoning log + confidence score persisted per email and viewable in-productDocumented
- 12 categories Shipped 12 default categories with configurable initial backfill (100/300/500/1000 emails)Documented
Limitations & what's next
- The ~2–3× speedup is developer-reported from a weekly update, not an independently measured benchmark — presented as reported, not guaranteed.
- Confidence scores are a system heuristic (a fixed high value for a valid LLM match, a low value on fallback), not calibrated model probabilities — they flag fallbacks, they don't quantify certainty.
- Pinecone is wired as a dependency and embedding fields exist on the schema, but the shipped categorization path is rules-plus-prompt; a live vector-search layer is scaffolded, not the current mechanism.
- The Uncategorized bucket is reduced via a forced second-pass review with thread context and a lower confidence bar, but nothing guarantees zero unsorted mail — some ambiguous messages are correctly left for a human.
- Next steps: calibrate confidence against real correction outcomes, and evaluate whether the embedding layer earns its place for near-duplicate sender/category matching.