A voice agent that books real appointments is four systems working together: speech-to-text, a dialogue layer, text-to-speech, and a deterministic booking tool that writes to an actual calendar. The last one is the part almost every tutorial skips, and it is the part that separates a demo from something a business can answer its phone with. I built the voice agent that runs on withhammad.com this way, and this post is the build sequence I would follow again — including the mistakes.
What booking a real appointment actually requires
Most voice bot content stops at 'the AI understood the caller and responded naturally'. That is components one and two: STT turns audio into text, the LLM decides what to say. But a caller who wants Tuesday at 3pm does not care how natural the voice sounds — they care whether Tuesday at 3pm actually lands on the calendar, does not collide with an existing booking, and produces a confirmation they can trust. That requires a fourth component: a booking tool with real calendar write access, real conflict checking, and a hard rule that the LLM proposes and the tool disposes. If the model is allowed to 'decide' availability from conversation context, it will eventually invent a slot that does not exist. Mine did, in testing, repeatedly.
Step 1 — design the call flow as a state machine
Before writing a single prompt, I drew the call as states: greeting, intent capture, availability check, slot offer, confirmation, write, fallback. Each state has an allowed set of transitions and its own narrow prompt. This matters because a single free-form system prompt ('you are a friendly receptionist who books appointments') degrades unpredictably — the model forgets to confirm, offers slots before checking, or wanders into small talk while the caller waits. With a state machine, the code decides which state the call is in; the LLM only handles the language inside that state. It also gives you an obvious place to hang the human-fallback path: any state can transition to 'escalate' when confidence drops or the caller asks for a person.
Step 2 — wire up STT and TTS against a latency budget
Phone conversation has an unforgiving rhythm. If the round trip from the caller finishing a sentence to the agent starting its reply goes much past 1.5 seconds, the call stops feeling like a conversation and starts feeling like a hold queue. That budget has to cover STT finalisation, LLM inference, and TTS synthesis of at least the first audio chunk — so every component gets chosen for time-to-first-byte, not benchmark quality. I use Fish Audio for TTS in production: it streams fast enough to start speaking while the rest of the sentence is still synthesising, and it costs meaningfully less than ElevenLabs at comparable quality for this use case. The other half of the problem is turn-taking: callers interrupt, and your agent has to stop talking when they do. Barge-in handling — cancel TTS playback the moment STT detects new speech — is not optional for phone work.
Step 3 — give the agent a real calendar tool, not a suggestion
ONE TACTIC A WEEK
The booking layer in my build is a tool call against a Google Calendar service account. The flow is strict: when the caller states a preferred time, the code — not the model — calls the freebusy endpoint, gets back the actual open slots, and only those slots are injected into the model's context to offer aloud. The model literally cannot offer a time the API did not return. The write itself is guarded twice more: a second freebusy check immediately before the insert (two concurrent calls can both be offered the same slot — the second check plus the calendar write acting as the source of truth resolves the race), and the confirm-before-write rule described in the next step. Service accounts matter here because the agent needs its own credentials with calendar scope, not a human's OAuth session that expires mid-week.
Step 4 — confirmation and correction handling
Callers change their minds mid-call constantly — 'actually, can we do Thursday instead?' — and your state machine has to treat that as a normal transition back to availability check, not an error. The rule I hold as non-negotiable: nothing is written to the calendar until the agent has read the full booking back (date, time, name, purpose) and the caller has explicitly said yes. An explicit confirmation state costs a few seconds of call time and eliminates the single worst failure mode a booking agent has, which is a confident wrong write. Timezones belong in this layer too: normalise everything at the tool layer, in code, using the business's timezone as canonical. In the UAE, where GST has no daylight saving but many callers' devices and calendars are set elsewhere, letting the LLM do timezone arithmetic is how you book someone at 3am.
Step 5 — test with adversarial calls before going live
Before the agent touched a real calendar, I ran it against a scripted set of hostile calls: dates given in ambiguous formats ('the 6th' — of which month?), interruptions mid-sentence, background noise degrading transcription, callers who confirm and then immediately retract, and callers who never state a time at all. Each script has an expected end state — booked, escalated, or politely ended — and a run fails if the call lands anywhere else. This is boring work and it is the entire difference between a demo and a product. Point the tool at a sandbox calendar for this phase; the test suite should be free to double-book and cancel without consequence.
Common failure modes and honest gotchas
The failures I actually hit, in rough order of pain: hallucinated availability before I locked slot offers to the freebusy result; TTS getting cut off mid-sentence when the telephony layer closed the stream too early; the timezone bug above; and STT mishearing numbers, which for a booking agent is catastrophic — 'the 15th' transcribed as 'the 5th' books the wrong week, which is why the read-back confirmation exists. Cost-wise, be honest with yourself before building: you are paying per-minute for STT and TTS, per-token for the LLM, and a monthly fee plus per-minute rates for the phone number and telephony. The components are individually cheap; a high call volume is not. And every voice agent needs a human fallback path — for callers who ask, for low-confidence states, and for anything compliance-sensitive. An agent that cannot hand off is a liability, not a receptionist.