← Production MonitoringArrows or space · F for fullscreen · Print to saveTalks

CLAUDE CODE BUILD NIGHT · PART 10

Production
Monitoring

Tonight, the alarms. Every night stands on its own.

Ship AI · TalksProduction Monitoring01/22

Who's hosting

How you do anything is how you do everything.

Santos Hernandez
Santos HernandezFounder, Ship AI
$25k+/moggbucks, built with agents
$0 to $12MARR at ZBD
EU firstMiCAR approval
THE COMMUNITY

Ship AI

Free workshops and talks in Phoenix.

shipai.club
THE STUDIO

Ship AI Studio

AI built into your business, in two weeks.

shipaistudio.com

Ship AI · TalksProduction Monitoring02/22

Join the room

SHIP AI DISCORD

Scan it now.
Prompts later.

Open the Discord invite

Ship AI · TalksProduction Monitoring03/22

Our partner

Ship AIWorkuityAI Exchange

Building Phoenix's AI community, together. Workuity hosts tonight, and we present Founder Weekend side by side.

Ship AI · TalksProduction Monitoring04/22

THE IDEA

Your users are not
your alarm system.

Ship AI · TalksProduction Monitoring05/22

Signal, not noise

LOGS YOU NEVER READ

Ten thousand lines. Nobody looking.

ALERTS YOU ACT ON

A few alerts.
Each with an owner and a first check.

Ship AI · TalksProduction Monitoring06/22

Tonight

01

Log

02

Catch

03

Alert

04

Triage

Ship AI · TalksProduction Monitoring07/22

Tonight's runbook

RUNBOOK.md
SLOs

99% of Find dinners under 15 seconds.

Alerts

Each one: who acts, first three checks.

Rollback

Roll back first. Debug second.

Done when

Logs are JSON, no PII

Down means #alerts in 3 minutes

Grok Bot drafts the triage

Ship AI · TalksProduction Monitoring08/22

Logs a machine can read

claude · ~/whats-for-dinner

✓ lib/log.ts · one JSON line, requestId on every response

{"event":"dinners_found","status":200,"ms":8412}

claudeRemoved 3 console.logs that printed ingredients.

Copy · structured logs
Add a JSON logger with Vercel's request id on every response, redact personal data, use it on the API and cron routes, and show me the lines.

Full prompt

Copy this entire prompt into Claude Code.

Read SPEC.md and SECURITY.md, especially the rule about personal data in logs.

1. Create lib/log.ts: a small logger that writes one JSON line per event with console.log (console.error for errors). Fields: ts, level, event, requestId, route, userId (the id only), status, durationMs, plus deployment from VERCEL_DEPLOYMENT_ID and commit from VERCEL_GIT_COMMIT_SHA.
2. Give every API request a requestId: Vercel's own id from the x-vercel-id request header, falling back to crypto.randomUUID() in local dev. Return it in an x-request-id response header. Using Vercel's id means vercel logs --query 'requestId:"<id>"' finds the same request; after the next preview deploy, check that it does, and if it doesn't, search the bare id as text instead and tell me.
3. Add a redact step that drops these keys anywhere in the object: email, name, ingredients, prompt, photo, token, authorization, cookie. Never log request bodies.
4. Use it in POST /api/dinners, POST /api/photo, the Resend webhook route and every cron route: one line when the request finishes (status, duration) and one on error (errorName and message, never the input). Log the error's name as errorName, because the redact step drops name.
5. Search the codebase for any other console.log that prints user data and list them before changing anything.

Run the dev server, call /api/dinners once, show me the log lines, and stop.

vercel.com/docs/logs/runtime

Ship AI · TalksProduction Monitoring09/22

Catch every error

~/whats-for-dinner
YOU RUN THE WIZARDnpx @sentry/wizard@latest -i nextjs
source maps · global-error · tunnel

Claude reviews what it wrote.

01

You sign in

The wizard creates the project and config files.

02

Claude tightens it

No PII, a sample rate, the requestId tag.

03

Prove it

A test error with a readable stack trace.

Copy · set up Sentry
I run the Sentry wizard. Claude reviews its files, strips PII, tags the requestId, monitors both crons, and proves it with one test error.

Full prompt

Copy this entire prompt into Claude Code.

Read RUNBOOK.md if it exists and SECURITY.md.

1. In my own terminal I'll run npx @sentry/wizard@latest -i nextjs. It signs me in, creates the Sentry project and writes the config. I'll also install Sentry's Vercel integration so each deploy notifies Sentry and uploads source maps. Wait until I tell you it's done.
2. Read what the wizard created (instrumentation-client.ts, sentry.server.config.ts, sentry.edge.config.ts, instrumentation.ts, app/global-error.tsx and the withSentryConfig wrapper in next.config) and explain each file in one line.
3. Make sure sendDefaultPii is off, and add a beforeSend that strips emails, cookies and request bodies. Set a modest tracesSampleRate for production and tell me the tradeoff. Set environment explicitly to VERCEL_ENV (development when it's missing) in every Sentry.init, so events say production or preview, not vercel-production, and our alert filters match. After the test error, check the Environment dropdown in Sentry shows those names.
4. Tag every event with the requestId from lib/log.ts and the user's id, never their email.
5. Wrap the daily at-risk email cron and the Sunday plan cron with Sentry.withMonitor, using the slugs at-risk-daily and sunday-plan and their real schedules. Tell me to check how many cron monitors our Sentry plan includes; the free plan has one, so if that's ours, ask me which cron gets it.
6. Check that .env.sentry-build-plugin is in .gitignore. Never print the auth token.
7. Add a temporary /api/debug-sentry route that throws, and only works when VERCEL_ENV is not production. Tell me to deploy a preview, hit it, and check that the stack trace in Sentry points at real source lines. Then remove the route and stop.

docs.sentry.io/platforms/javascript/guides/nextjs

Ship AI · TalksProduction Monitoring10/22

Is it up?

curl /api/health

{"status":"ok","db":{"ok":true,"ms":38},

"aiGateway":{"ok":true,"ms":112},"commit":"8c1e2f4"}

claudeDatabase stopped: 503 in 2.1s. Restored.

Copy · health route and uptime
A /api/health route that checks Neon and AI Gateway without spending tokens, returns 503 on failure, and a Sentry Uptime monitor I set up.

Full prompt

Copy this entire prompt into Claude Code.

Read RUNBOOK.md and SECURITY.md. Build GET /api/health.

1. Database: run select 1 with a 2-second timeout.
2. AI Gateway, without spending tokens: call its credit balance endpoint (read the AI Gateway REST API docs for GET /v1/credits, its URL and auth) with a 3-second timeout, and cache that result for 60 seconds so the route can't be used to hammer it. AI_GATEWAY_API_KEY is already in .env.local.
3. Return JSON: status (ok or degraded), ok and ms for each check, the first 7 characters of VERCEL_GIT_COMMIT_SHA and VERCEL_ENV. No hostnames, no error details, no keys. 200 when every check passes, 503 when any fails. Cache the response for about 10 seconds (Cache-Control: public, s-maxage=10, or a 10-second in-memory cache) so a burst of requests can't hit the database every time.
4. Keep it outside auth, and check it isn't blocked by any Firewall rule from SECURITY.md.
5. Ask me whether the database check should run on every ping: a check every minute keeps Neon's compute from scaling to zero. Wait for my answer.
6. Prove the 503 locally by pointing at a bad database URL, then restore it.
7. Tell me to create an Uptime Monitor in Sentry for our production /api/health at a 1-minute interval (the free plan includes one). Tell me to check our Sentry plan includes Slack alerts. If it does, the alert posts downtime issues to Slack #alerts. If it doesn't, it emails #alerts' channel email address if our Slack has one, or the on-call team. Wait until I tell you it's done, then stop.

docs.sentry.io/product/uptime-monitoring

Ship AI · TalksProduction Monitoring11/22

Is the database okay?

claude · ~/whats-for-dinner

connections · 14 active, 31 idle, max 104

slowest · favorites by user_id · 41 ms × 3,120

claudeNo index on favorites.user_id. Want a migration?

Copy · database health
I enable pg_stat_statements. Claude writes read-only health queries, a Database section for the runbook, and checks we use the pooler.

Full prompt

Copy this entire prompt into Claude Code.

Read RUNBOOK.md. We're adding database checks. Nothing in this prompt writes to the database.

1. In the Neon SQL Editor I'll run CREATE EXTENSION IF NOT EXISTS pg_stat_statements; on the production database. Wait until I tell you it's done.
2. Write scripts/db-health.sql with read-only queries: the ten slowest statements by mean_exec_time from pg_stat_statements, the ten most called, and current connections grouped by state from pg_stat_activity.
3. Tell me how to run it against production with a read-only role, and note that pg_stat_statements resets when the compute suspends or restarts. The Query performance tab on Neon's Monitoring page shows top queries without the extension.
4. Add a Database section to RUNBOOK.md: which Neon Monitoring charts to open first (Postgres connections count against the max line, pooler connections, CPU, deadlocks) and what each pattern usually means.
5. Check that serverless routes use Neon's pooled connection string.

If a slow query needs an index, write the migration but don't run it. Stop and wait.

neon.com/docs/introduction/monitoring

Ship AI · TalksProduction Monitoring12/22

Alerts worth waking for

ONE CHANNEL#alerts

Every alert has an owner
and a first check.

01

SLO

99% of Find dinners under 15 seconds.

02

Page

App down: three failed checks in a row.

03

Not an alert

One 429. One retry. One bad webhook.

Copy · SLOs and alerts
Interview me, then extend RUNBOOK.md: on call, SLOs, alerts with owners, first checks, rollback, logging, the Grok Bot routine and a postmortem template. I create the Sentry rules.

Full prompt

Copy this entire prompt into Claude Code.

Read SPEC.md, GROWTH.md, SECURITY.md and RUNBOOK.md.

Interview me one question at a time and wait for each answer: what users must always be able to do, how slow is too slow for Find dinners, who is on call, and what should wake someone up versus wait until morning.

Then extend the RUNBOOK.md from Part 8. Keep its workflow notes and move them under Per-area checks, Workflows. Add these sections:
1. On call: primary, backup, and where alerts go.
2. SLOs: plain sentences, each with a number and a 28-day window, for example "99% of Find dinners finish in under 15 seconds."
3. An Alerts table: alert, fires when, severity, who acts, first check.
4. A Not alerts list.
5. First three checks for any alert: /api/health, the Sentry issue, then vercel logs --environment production --query 'level:error' --since 1h and one request with --query 'requestId:"<id>"'.
6. Per-area checks: Functions, AI Gateway, Database (keep the section we wrote), Workflows, Email.
7. Rollback: read vercel.com/docs/instant-rollback and write the dashboard and CLI steps, what a rollback doesn't undo, and vercel promote to resume.
8. Logging rules from lib/log.ts.
9. Grok Bot SRE routine: trigger, reads, can't reach, writes, ask first and never. We fill in the brief later tonight.
10. A blameless Postmortem template: impact, timeline in Phoenix time, detection, what happened and why, what went well and didn't, actions with owners.

Rules: every alert has an owner and a first check. Nothing pages on a single event. Our own 429s and rejected webhooks are expected, so they alert only on spikes. Preview deployments never alert. Alert filters use the environment names Sentry shows: production and preview.

Then give me a click-by-click checklist for creating each rule in Sentry. Check our Sentry plan includes Slack alerts: if it does, the action is Slack #alerts; if not, email to #alerts' channel email address if our Slack has one, or to the on-call team. I'll create them myself. Stop and wait.

docs.sentry.io/organization/integrations/notification-incidents/slack

Ship AI · TalksProduction Monitoring13/22

Debug with Claude Code

claude · ~/whats-for-dinner

youFind dinners is slow since 2pm.

claudeSentry: p95 31s since release 8c1e2f4. Logs: 3 retries per request.

claudeRoll back or fix forward? Your call.

Copy · investigate an alert
I add the Sentry and Vercel MCP servers. Claude investigates an alert read-only across Sentry, Vercel logs and workflows, then writes a triage note.

Full prompt

Copy this entire prompt into Claude Code.

1. In my own terminal I'll run:
claude mcp add --transport http sentry https://mcp.sentry.dev/mcp
claude mcp add --transport http vercel https://mcp.vercel.com
Then I'll restart Claude Code and sign in to both with /mcp. Wait until I tell you it's done.
2. Read RUNBOOK.md. When I give you an alert, a Sentry issue link or a request id, investigate read-only:
- Sentry: first seen, release, event count, users affected, the stack trace.
- Vercel: runtime logs for that request id (our requestId is Vercel's x-vercel-id, so --query 'requestId:"<id>"' finds it), and which deployment was live. vercel logs --environment production --query 'level:error' --since 1h works too.
- AI Gateway: its request list isn't in the API, so tell me what to filter on its Logs page (Status 429 or 5xx, Model, Provider).
- Workflows: npx workflow inspect runs, if the Sunday plan is involved.
3. Write a triage note: what's broken, since when, who is affected, the suspected cause with evidence, and options (roll back, fix forward, wait).

Don't deploy, roll back, change environment variables or write to the database. Those are my call. Start with the latest unresolved Sentry issue in production.

mcp.sentry.dev

Ship AI · TalksProduction Monitoring14/22

An AI SRE in Grok Bot

Grok Bot · #alerts routine

Sentry · Error spike · POST /api/dinners

claudeHealth ok. Started 14:02, after commit 8c1e2f4.

→ Triage note and a Claude Code prompt drafted. Approve to post?

Copy · the Grok Bot SRE brief
Write a Grok Bot routine brief: on each #alerts message it reads the alert and /api/health, drafts a triage note and a Claude Code prompt, and asks before posting.

Full prompt

Copy this entire prompt into Claude Code.

Read RUNBOOK.md and SECURITY.md. Write grok-bot-sre.md: a routine brief I'll paste into Grok Bot, xAI's AI teammate, myself. Keep it under 300 words.

1. Trigger: any new message in Slack #alerts (a Slack trigger on that one channel). I'll test that alerts arriving in #alerts, from the Sentry app or by email, start it; if they don't, use Grok Bot's Sentry event trigger instead.
2. Reads: the alert message and its thread, our public /api/health response, and RUNBOOK.md, which I'll give it.
3. Say plainly what it can't reach: Vercel (Vercel's MCP server only accepts reviewed clients, and we won't put a Vercel token on its computer), and Sentry unless I add Sentry's MCP server as a custom connector and it works read-only.
4. Output: a draft reply in the alert's thread with what fired, who is affected, the suspected cause with evidence and confidence, the first three checks from RUNBOOK.md and what they showed, the recommended next step, and a ready-to-paste Claude Code prompt for me.
5. Ask first rules: posting anything. Never: deploy, roll back, change environment variables, write to the database, email users or post outside #alerts.

Don't sign up for or connect anything. I'll do that myself.

docs.x.ai/grok-bot

Ship AI · TalksProduction Monitoring15/22

Drafts, not deploys

AN AGENT WITH PROD KEYS

Fixes at 3 AM. Nobody reviewed it.

AN SRE THAT DRAFTS

Triage in minutes.
The rollback is your click.

Ship AI · TalksProduction Monitoring16/22

Run a fire drill

claude · ~/whats-for-dinner

✗ Sentry · TypeError in POST /api/dinners (preview)

→ #alerts · Grok Bot draft in the thread

claudePostmortem written. 2 actions, both owned.

Copy · fire drill and postmortem
Break a preview on purpose, time the path from error to #alerts to Grok Bot's draft, rehearse the rollback, and write the postmortem.

Full prompt

Copy this entire prompt into Claude Code.

Read RUNBOOK.md. We're running a fire drill on a preview deployment. Never production.

1. Add a FIRE_DRILL environment flag that makes POST /api/dinners throw a TypeError, and only when VERCEL_ENV is preview.
2. Tell me to set FIRE_DRILL=1 for one preview branch in Vercel, add a temporary Sentry alert rule for the preview environment that notifies #alerts the same way our real alerts do (Slack, or email), and deploy. Wait until I tell you it's done.
3. I'll call the route a few times. Then ask me, one at a time: did Sentry open an issue, did #alerts get it, did Grok Bot draft a triage note, and how many minutes did each take?
4. Walk me through the Rollback section of RUNBOOK.md without running anything: Instant Rollback in the dashboard or vercel rollback, what it doesn't undo, and vercel promote to resume.
5. Write postmortems/fire-drill.md from the template, with today's date and real timings, and add any fixes to RUNBOOK.md.
6. Remove the flag and tell me to delete the temporary alert rule. Stop.

Also in the series: Part 11, Admin Console. The controls to act on what monitoring finds.

Ship AI · TalksProduction Monitoring17/22

BONUS ROUND

Errors aren't the only fire.
Watch the bill.

Ship AI · TalksProduction Monitoring18/22

AI spend by feature

npx tsx scripts/ai-spend.ts

feature:dinners · 61% of spend

feature:photo · 33% · up 4x this week

claudePhoto spend jumped after Tuesday's deploy.

Copy · AI spend tracking
Tag AI calls by feature and user, set a budget with spend alerts, and print a weekly spend-by-feature table.

Full prompt

Copy this entire prompt into Claude Code.

Bonus round. Read RUNBOOK.md and SECURITY.md, then the AI Gateway Custom Reporting and Budgets docs.

1. Tag every AI Gateway call with a feature tag (feature:dinners, feature:photo, feature:sunday-plan) and the user's id as the user, using the exact fields the Custom Reporting docs show.
2. Tell me to set a monthly budget for this project in AI Gateway with spend alerts going to my email. I'll do it in the dashboard. Wait until I tell you it's done.
3. Write scripts/ai-spend.ts that calls the Custom Reporting API for the last 7 days grouped by tag and prints a table. Read AI_GATEWAY_API_KEY from .env.local and never print it.
4. Add a Spend section to RUNBOOK.md: where to look, what a normal week costs once we know, and who acts on the budget alert.

Run the script once and stop.

vercel.com/docs/ai-gateway/observability-and-spend

Ship AI · TalksProduction Monitoring19/22

Take it home

THE TALK PAGE

Every prompt,
and this deck.

Open the talk page

Ship AI · TalksProduction Monitoring20/22

Founder Weekend

FEB 6‑7, 2027 · FREE

Tonight was one hour.
Take a weekend.

An Arizona AI hackathon. Teams of 1-4.

See the hackathon

Ship AI · TalksProduction Monitoring21/22

SHIP AI DISCORD

Free and open.
Build with us.

Scan to join

Ship AI · TalksProduction Monitoring22/22
01/22