Ask Siri

A password-gated voice agent built on ElevenLabs Conversational AI. It answers only from my resume, project history and a red-teamed knowledge base, in my cloned voice, for up to ten minutes.

Classification

Voice AI · Conversational agents · RAG grounding · Applied trust & safety

Stack

Next.js · TypeScript · ElevenLabs Conversational AI · React SDK · Vercel

Subject

A ten-minute voice conversation with one hard rule: if it isn't in the files, the agent doesn't say it.

The Brief

For the past few years, most people have used AI through a chat box. That isn't how people naturally communicate, and voice removes a lot of the friction of typing at a computer.

So I built one: a live agent visitors can talk to, grounded in a knowledge base I control, that tells you when it doesn't know something.

Design Thesis

Answer from the files, or say so.

Every claim the agent makes has to trace back to an uploaded document. If retrieval finds nothing, it says so. A made-up claim about my career is the one failure this build is designed to prevent.

The Architecture

Five stages across two systems.

A session flows left to right. Solid stages run in my own Next.js code. Dashed stages hand off entirely to ElevenLabs' infrastructure.

Stage 1

GATE

A password and a resumable 10-minute budget, checked on the server before any mic prompt or ElevenLabs call.

OUR CODE

Stage 2

SIGN

OUR CODE

Mints a short-lived ElevenLabs signed URL, as close to the moment it's used as possible.

OUR CODE

Stage 3

STREAM

The browser opens a direct WebSocket to ElevenLabs for two-way audio. My server steps out once the call is live.

ELEVENLABS INFRASTRUCTURE

Stage 4

GROUND

CORE · RAG

Every answer comes from an uploaded knowledge base, never from the model's general knowledge.

ELEVENLABS INFRASTRUCTURE

Stage 5

ESCALATE

Tool calls (log_unanswered, send_contact) hit Next.js webhooks that email me directly.

OUR CODE

GUARDRAILSSALARY · AVAILABILITY · OPINIONS ON NAMED PEOPLE — REFUSED EVERY TIMEEnforced by the agent's own rules and the scope of its knowledge base.

ON FILE

Answered

Grounded directly in the corpus, cited by document.

OFF LIMITS

Deflected

Salary, availability and opinions are refused by rule.

GAP FOUND

Escalated

Logged and emailed to me, so I can fill the gap in the knowledge base.

Solid ink = my own Next.js code Dashed oxblood = ElevenLabs infrastructure

View the full architecture diagram →

Results

What shipped.

Knowledge-base documents

Resume, timeline, six project write-ups, an FAQ and some personal background.

10 min

Session cap per invite

A deliberate cost control. The agent tells visitors about the limit up front.

60s

Silence timeout

Client-side watchdog, tracked independently of any ElevenLabs dashboard setting.

Agent tools wired to email

log_unanswered and send_contact, both tested with curl before the agent was connected to them.

What This Demonstrates

Three problems I had to solve.

Debug a failure that looked like a network problem and wasn't.

The WebSocket handshake looked the same whether the call worked or not: DevTools showed a clean 101 Switching Protocols and a conversation ID either way. The bug wasn't in the network. Next.js's dev-mode Fast Refresh was remounting the component that owned the connection mid-handshake, leaving the connection attached to a component that no longer existed. A production build, which has no Fast Refresh, connected on the first try.

Stop the model from bluffing.

The agent won't answer from general knowledge. If retrieval returns nothing, it says so. Before launch I red-teamed it against a fixed set of questions designed to draw out a confident, made-up answer.

Design around cost from the start.

Each password carries a ten-minute budget, tracked on the server and resumable across sessions. It syncs with sendBeaconwhen a tab closes, because a normal fetch isn't guaranteed to finish while a page unloads. A silence watchdog in the browser ends idle calls, since ElevenLabs' dashboard has no per-session setting for that.

FIELD NOTE · CONNECT

I rate-limited myself while debugging. The guard that stops password guessing also counted every retry, so I restarted the dev server to clear the in-memory counter.

FIELD NOTE · GROUND

There's another public Siri Rama, a dancer with a PhD in Fine Arts. The knowledge base tells the agent to ignore anything it already knows about the name, so it never answers as the wrong person.

FIELD NOTE · GATE

Signed URLs expire quickly, so they're created when a visitor taps “begin” instead of at page load or password entry.

EXHIBIT A — RESUMABLE BUDGET SYNC (TYPESCRIPT)

// Monotonically decreasing — a later report can never raise a

// password's remaining time back up.

function reportRemainingSeconds(password: string, reported: number) {

if (!SINGLE_USE_PASSWORDS.has(password)) return;

const current = budgets.get(password) ?? SESSION_SECONDS;

const next = Math.max(0, Math.min(current, Math.floor(reported))); // FIX

budgets.set(password, next);

}

Highlighted: a slow heartbeat that arrives after a newer one can't add time back. The budget only goes down.

Notes & Provenance

Visitor audio and transcripts aren't used to train anything. Conversations are transcribed for my review only.
The agent never uses pretrained knowledge about “Siri Rama”, because another, unrelated public person has the same name.
Every tool webhook fails closed: if the shared secret isn't set, the call is rejected.

Talk to it yourself →View the architecture diagram →← Back to all files