Ship AI · Talks · Coming soon
Ship an Agent Product
An agent people pay for, not a demo that works once.
How the night runs.
A short AI news briefing, then we build Deskhand live with Claude Code from one AGENT-SPEC.md, then five-minute demos from the room.
- 01
Spec
The job, the tools, what needs approval.
- 02
Tools
A chat UI and an agent that reads store data.
- 03
Approve
Refunds wait for the owner's yes in Slack.
- 04
Guard
Deploy, rate limits, cost caps, evals.
The deck.
Every slide from the night, free to share. Open it full size to step through.
Tonight's agent.
The agent spec we build from on the night: a support agent for a small online store, with its tools, its approvals and its evals. Copy it and swap in your own agent's job.
Deskhand
A support agent for small online stores. It answers order questions and asks the owner before money moves.
The jobAnswer order, return and care questions in the store's chat; refunds wait for the owner's yes in Slack.
- Chat SDK
- AI SDK
- AI Gateway
- Workflow SDK
- Next.js
- Vercel
- A wrong email on an order lookup reveals nothing
- Refunds wait for Approve in the owner's Slack
- Approve refunds exactly once
- Rapid messages get a 429
- All 13 eval cases pass
The full agent spec
# AGENT-SPEC.md · Deskhand
## Job
Deskhand answers customer questions in a small online store's chat: where is my order, can I return this, how do I care for it. It resolves what it can from store data. Anything that moves money waits for the owner's yes in Slack. First customer: one Phoenix candle shop.
## Users
- Shoppers: anonymous on the storefront chat, known by a random visitor id in a signed cookie that `proxy.ts` issues. No valid cookie, no chat (401). Order details only after the order number and the email on the order match.
- Owner: one person in Slack. Approves refunds, gets handoffs.
## Stack
- Chat SDK (`chat`): Web adapter (`@chat-adapter/web`) for the storefront chat, Slack adapter (`@chat-adapter/slack`) for the owner, Redis state (`@chat-adapter/state-redis`) in production
- Scaffold: `npm create chat-sdk@latest -- deskhand --adapter web slack memory -y`
- Chat UI: the Chat pattern from shadcn/ui AI Elements, on `useChat` from `@chat-adapter/web/react`
- AI SDK (`ai`): `ToolLoopAgent`, `tool()` with Zod schemas, `stopWhen: isStepCount(8)`, streamed with `thread.post(result.fullStream)`
- AI Gateway: model as a "provider/model" string; OIDC on Vercel, `AI_GATEWAY_API_KEY` locally
- Workflow SDK (`workflow`): the refund workflow, with `requestApproval` from `chat/workflow`
- Resend for customer emails
- Vercel: deploy, environment variables, logs, workflow runs, AI Gateway spend, Firewall rate limits
## Tools
| Tool | Reads or writes | Approval |
| --- | --- | --- |
| lookupOrder(orderNumber, email) | Reads one order, only if the email matches. On a match, stores the order number as verified for this thread in Redis state | None |
| trackShipment(trackingNumber) | Reads carrier status | None |
| searchPolicies(query) | Reads returns, shipping and care policies | None |
| handOff(summary) | Posts a summary to the owner's Slack DM | None |
| requestRefund(orderNumber, amount, reason) | Starts the refund workflow only for an order verified in this thread, with the email from that record. Never refunds directly | Owner, in Slack |
## Never
- Refund, cancel or change an order without the owner's approval
- Start a refund for an order that lookupOrder hasn't verified in this thread
- Share order details unless the order number and email match
- Promise anything that isn't in the policies
- Ask for card numbers, passwords or full addresses
## Refund workflow
`refundRequest({ orderNumber, email, amount, reason })`, "use workflow"
1. **loadOrder** (step): fetch the order. FatalError if it isn't found, the email doesn't match the order, the amount is more than the order total, or it's outside the 30-day return policy
2. **openOwnerDM** (step): `bot.openDM(OWNER_SLACK_USER_ID)`
3. **requestApproval** (workflow level): card with order, amount and reason; `approvers` is the owner only; `timeout: "24h"` (2 minutes when testing)
4. Denied: **notifyCustomer** with a polite no. Timed out: **handOff**
5. Approved: **issueRefund** (step) with idempotency key `refund/<orderNumber>`
6. **notifyCustomer** (step): Resend email with an idempotency key
## Guardrails
- Rate limit: `checkRateLimit("deskhand-chat", { request })` from `@vercel/firewall` at the top of `/api/chat`, keyed by client IP (the default, so clearing the cookie doesn't reset it), 429 when limited
- Cost: messages capped at 2,000 characters, 8 steps per turn, a daily token cap per client IP and per visitor counted from `totalUsage` in Redis, and a monthly AI Gateway project budget
- Logs: one JSON line per turn with a hashed thread id, tool names, token counts, latency and outcome. Never message text, emails, names, addresses, raw IPs or order contents
## Evals
`evals/cases.json`: 13 conversations with expected outcomes. 4 order questions (one with the wrong email), 3 policy questions, 2 refund requests (one outside policy), 1 handoff, 1 off-topic, 1 attempt to talk the agent into a refund, 1 refund request for an order this thread never verified. Run locally with `npx tsx evals/run.ts` against `data/orders.json`, with issueRefund, Resend and the Slack calls stubbed.
## Done when
- [ ] A wrong email on an order lookup reveals nothing
- [ ] A refund for an order not verified in this thread is refused
- [ ] A refund request posts an approval card in the owner's Slack, and nothing is refunded before Approve
- [ ] Approve refunds exactly once, even when clicked twice
- [ ] No answer in the wait (2 minutes when testing) hands off to the owner
- [ ] Rapid messages from one IP get a 429, even after clearing cookies
- [ ] All 13 eval cases pass locally
The prompts.
Every prompt from the deck, in order. Open one, copy it, paste it into Claude Code, Codex or any coding agent.
- 01
Spec the agent
I'm building Deskhand, a support agent for small online stores, starting with one Phoenix candle shop. Shoppers chat on the store's site. The owner approves anything that moves money from Slack. Interview me one question at a time and wait for each answer: the job, who the users are, which tools the agent may call and what each one reads or writes, what must wait for the owner's approval, what the agent must never do, which limits protect us (rate, cost, message length), what we may and may not log, and how we'll know it works. Then write AGENT-SPEC.md with these sections: Job, Users, Stack, Tools (a table: tool, reads or writes, approval), Never, Refund workflow (numbered steps), Guardrails, Evals, Done when. The stack is Chat SDK with the Web and Slack adapters, the AI SDK through Vercel AI Gateway, the Workflow SDK, and Vercel. Don't write code yet. Stop and wait for my OK.
- 02
Scaffold with Chat SDK
Read AGENT-SPEC.md. 1. In my own terminal I'll run: npx skills add vercel/chat npx skills add https://github.com/vercel/workflow --skill workflow Wait until I tell you it's done. 2. From this parent folder (where AGENT-SPEC.md is, not inside a deskhand folder), run: npm create chat-sdk@latest -- deskhand --adapter web slack memory -y Memory state is for local development only. We switch to Redis before deploying. 3. Move AGENT-SPEC.md into the deskhand folder. Show me the generated files and what each one does. 4. Tell me to restart Claude Code inside the deskhand folder so the skills load there. Stop and wait.
- 03
The storefront chat
Read AGENT-SPEC.md, src/lib/bot.ts and src/app/api/chat/route.ts. Use the Chat SDK skill and the docs in node_modules/chat/docs. 1. Issue the visitor cookie in src/proxy.ts. Next.js 16 renamed middleware to proxy, so check https://nextjs.org/docs/app/api-reference/file-conventions/proxy first. When a request has no valid cookie, create a random visitor id (no colon in it), sign it with COOKIE_SECRET, and set it on the response as httpOnly, secure and sameSite lax. Skip static files in the matcher. 2. Replace the getUser stub so it only reads and verifies that cookie: return { id } when the signature checks out, and null (a 401) when the cookie is missing or invalid. getUser never creates a cookie. Tell me to add COOKIE_SECRET to .env.local myself (a long random string) for local dev, and wait until I tell you it's done. 3. The scaffold has no pages, no root layout and no Tailwind. Add a root layout and Tailwind CSS, then run npx shadcn@latest init. Build the storefront chat in src/app/page.tsx from the Chat pattern at https://ui.shadcn.com/docs/ai-elements/base/chat.md. Its components import @ai-sdk/react, so install that too, but wire the page to useChat from @chat-adapter/web/react. 4. In bot.ts, handle onDirectMessage with a ToolLoopAgent from ai. Its instructions come from the Job and Never sections. No tools yet. Pass earlier turns with toAiMessages from chat/ai, then post the agent's fullStream to the thread with thread.signal as the abort signal. 5. Use AI Gateway with a "provider/model" string. Check the AI Gateway model list for a current model ID instead of guessing. Tell me to add AI_GATEWAY_API_KEY to .env.local myself, and wait until I tell you it's done. 6. Tell me to run npm run dev and ask it about the return policy. Stop there. - 04
Give it tools
Read AGENT-SPEC.md, sections Tools and Never. 1. Create data/orders.json with 15 realistic sample orders for a candle shop (order number, email, items, total, status, tracking number, order date) and data/policies.md with returns, shipping and candle-care policies. Put all reads behind lib/store.ts so it can call the real store API later. 2. Add tools with tool() from ai and a Zod inputSchema each: lookupOrder (returns details only when the email matches the order, otherwise a generic "I couldn't find that order"; on a match, it also stores the order number and email as verified for this thread on the server, in the bot's state, memory now and Redis once deployed, keyed by the thread id from the handler, never by anything the model or the browser sends), trackShipment, searchPolicies, and handOff (for now it logs a one-line summary). 3. Give them to the agent with stopWhen: isStepCount(8). No refund tool yet. 4. Give me five questions to try in the chat, including one with the wrong email, and tell me what a correct answer looks like for each. Stop there.
- 05
Read freely. Ask before money moves.
Read AGENT-SPEC.md and the "Create a Slack app from manifest" section of node_modules/@chat-adapter/slack/README.md. 1. Write slack-manifest.yaml from that manifest, with the app named Deskhand and both request_url fields set to https://YOUR-TUNNEL/api/webhooks/slack. 2. Tell me how to expose localhost:3000 with a tunnel. I'll create the app from the manifest at api.slack.com/apps, install it to my workspace, replace YOUR-TUNNEL, and add SLACK_BOT_TOKEN, SLACK_SIGNING_SECRET and OWNER_SLACK_USER_ID to .env.local myself. Wait until I tell you it's done. 3. Change handOff: open a DM with the owner with bot.openDM(OWNER_SLACK_USER_ID) and post the summary there. 4. Tell me to ask for a human in the storefront chat and check my Slack. Stop there.
- 06
Refunds wait for a yes
Read AGENT-SPEC.md, section Refund workflow, the Approvals guide in node_modules/chat/docs, and the Workflow SDK docs in node_modules/workflow/docs. 1. Run npm i workflow and set it up for Next.js as the docs say (withWorkflow from workflow/next in next.config). Call registerSingleton() on the Chat instance in bot.ts. 2. Create workflows/refund.ts with a "use workflow" function refundRequest({ orderNumber, email, amount, reason }). As "use step" functions: loadOrder (throw FatalError from workflow if the order isn't found, the email doesn't match the order, the amount is more than the total, or it's outside the return policy), openOwnerDM (bot.openDM with OWNER_SLACK_USER_ID), issueRefund (through lib/store.ts, idempotency key refund/<orderNumber>), and notifyCustomer (Resend email with an idempotency key). 3. At the workflow level, call requestApproval from chat/workflow on the owner's DM thread: order, amount and reason as fields, approvers set to the owner only, timeout "24h", or 2 minutes when a test env flag is set. On deny, notify the customer politely. On timeout, hand off. 4. Add a requestRefund tool. First it reads the verified orders that lookupOrder stored for this thread on the server. If the order number isn't there, it refuses and asks the shopper to look the order up with its email first. Otherwise it takes the email from that record, never from the model, starts the workflow with start from workflow/api, and tells the shopper the owner will decide within a day. The agent never refunds directly. 5. Tell me to add RESEND_API_KEY to .env.local myself and wait until I tell you it's done. Then tell me to request a refund in the chat, approve it in Slack, click Approve twice, and watch the run in npx workflow web. Stop there. - 07
Deploy it
Read AGENT-SPEC.md and the Workflow SDK deploy docs in node_modules/workflow/docs. 1. Swap the memory state for createRedisState from @chat-adapter/state-redis, reading REDIS_URL. The approval click and the workflow must share the same state. 2. In my own terminal and dashboard I'll run vercel link, add a Redis database to the project from the Vercel Marketplace, add SLACK_BOT_TOKEN, SLACK_SIGNING_SECRET, OWNER_SLACK_USER_ID, RESEND_API_KEY and COOKIE_SECRET under Environment Variables, make sure system environment variables are exposed as the Workflow SDK docs say, and run vercel env pull. On Vercel, AI Gateway authenticates with OIDC, so no gateway key is needed there. Wait until I tell you it's done. 3. Run npm run build and fix any errors, then deploy with vercel --prod. 4. Tell me to point both request_url fields in the Slack app at the project's production domain (deskhand.vercel.app, or our custom domain once it's added), not the per-deployment URL that vercel --prod prints: Standard Protection blocks that one, so Slack's requests would fail. Wait until I tell you it's done. Then show me where to find this deploy's logs, its workflow runs, and its AI Gateway requests in the Vercel dashboard. Stop there.
- 08
Guardrails
Read AGENT-SPEC.md, section Guardrails, and the Vercel rate limiting SDK docs at https://vercel.com/docs/vercel-firewall/vercel-waf/rate-limiting-sdk. 1. Rate limit: in the Vercel dashboard I'll create a Firewall rule with the @vercel/firewall condition and the rate limit ID deskhand-chat. Wait until I tell you it's done. Then npm i @vercel/firewall and call checkRateLimit("deskhand-chat", { request }) at the top of /api/chat. Leave out rateLimitKey so the key stays the client IP: a shopper can clear the visitor cookie, but not their IP. When limited, return 429 with a friendly message. 2. Cost: reject messages over 2,000 characters, keep stopWhen at 8 steps, and keep two daily token counts in Redis from the agent's totalUsage: one per client IP (ipAddress from @vercel/functions, stored hashed) and one per visitor id. Past either cap, offer handOff instead of calling the model. Give me the command to set a monthly AI Gateway project budget myself: vercel ai-gateway budgets set project deskhand --amount <limit> --refresh-period monthly. 3. Logs: one JSON line per turn with a hashed thread id, tool names, token counts, latency and outcome. Never log message text, emails, names, addresses, raw IPs or order contents. Search the codebase for any log that could leak them and fix it. 4. Deploy, then tell me to send messages quickly from one browser until I see the 429. Stop there. - 09
A small eval set
Read AGENT-SPEC.md, sections Evals and Done when. 1. Write evals/cases.json: 13 short conversations, each with its expected outcome. 4 order questions (one with the wrong email), 3 policy questions, 2 refund requests (one outside the policy), 1 request for a human, 1 off-topic message, 1 shopper trying to talk the agent into a refund without approval, and 1 shopper who verifies their own order and then asks for a refund on a different order number. 2. Write evals/run.ts. It runs locally: each case goes through the same agent and tools against data/orders.json, with memory state, and with issueRefund, Resend and the Slack calls stubbed. It checks which tools were called, that no refund ran without approval, that requestRefund refused any order not verified in that thread, and that no order details leaked on a wrong email. Use plain checks first. For tone, add a second model call with Output.object that returns pass or fail and a reason. 3. Run it with npx tsx evals/run.ts and show me a pass/fail table. Fix the agent's instructions or tools, not the cases, until all 13 pass. 4. Then check the other Done when items in AGENT-SPEC.md against the deployed app, tell me which pass, and stop.
The toolkit.
Vercel's agent stack, one job each: the chat, the model calls, the long-running work and the platform it runs on.
Chat SDK ↗
One bot for the web chat and Slack, with typed handlers, cards and streaming
npm create chat-sdk@latestChat SDK skill ↗
Teaches Claude Code the Chat SDK from the docs bundled with the package
npx skills add vercel/chatAI Elements ↗
The Chat pattern for the storefront UI, copied into your project from shadcn/ui
npx shadcn@latest initAI SDK ↗
ToolLoopAgent, tools with Zod schemas, streaming and structured output
npm install ai zodAI Gateway ↗
"provider/model" strings, request logs and spend budgets
vercel ai-gateway budgets set project deskhand --amount <limit> --refresh-period monthlyWorkflow SDK ↗
Durable steps, retries and waits that survive deploys
npm i workflowVercel CLI ↗
Link the project, pull env vars, deploy
npm i -g vercelVercel Firewall SDK ↗
Per-IP rate limits in code
npm i @vercel/firewall
Write the limits. Then charge for it.
Date to be announced. Dates go out in the Discord first.
Harbor at dusk · Phoenix, Arizona