What We Learned Wiring Up an AI Receptionist

We built an AI receptionist that answers a lead the moment they submit our workshop form. It calls them, has a real ten-minute conversation, scores how good a fit they are, and books them straight into a founder's calendar if it makes sense. Neither of us touches any of it. It works now. Getting there took longer than we expected, and almost none of the delay was the AI.

This is the honest version of what that build involved: the four systems we had to get talking to each other (Twilio, ElevenLabs, our own n8n workflows, and Twenty CRM), and the specific things that went wrong. We're writing it down mostly for ourselves. Partly because if you're about to build something similar, a lot of this will save you a day or two.

You've seen the ads. Connect four boxes, click deploy, your business runs itself while you sip a cocktail somewhere. This is the version with the boring parts left in. Nobody's landing page mentions the trial-tier phone restriction that only shows up once you try calling a real customer, or the webhook checkbox buried three menus deep that fails completely silently, or an afternoon lost to a CRM field that quietly changed type without telling anyone. The tools do work. It's just that "it just works" is doing a lot of quiet, unpaid overtime in that sentence.

Twilio taught us that "trial" doesn't mean "smaller"

We assumed a Twilio trial account was the real thing with a spending cap on it. It isn't. It's a genuinely different product, and the gap between the two nearly cost us a full day of debugging something that had nothing to do with our code.

Verified numbers only, no exceptions

On a trial account, Twilio will only call numbers you've manually verified in advance. A one-time code sent to that exact number, confirmed in the console. That's a sensible anti-fraud guardrail. It's also completely incompatible with calling a lead who filled out a form five minutes ago, because there's no way to pre-verify someone you've never met.

The error message doesn't say "you're on a trial account." It says the call isn't authorized, which reads like a permissions problem. So naturally we went looking in the permissions settings.

Geo-permissions are a separate lock, not the same one

Twilio also has a country-by-country toggle for outbound voice. Australia, Low-Risk, enabled. We flipped it, certain that was the fix. Same error, word for word. Two completely independent restrictions were stacked on top of each other, and clearing one told us nothing about the other.

The actual fix, confirmed by re-reading Twilio's own upgrade page rather than guessing again, was that upgrading to a paid account removes the verified-recipients restriction entirely. Nothing in the geo-permissions screen. We'd have found this in five minutes if we'd checked the account tier before the country settings.

ElevenLabs: easy to make it sound right, harder to make it work right

Getting a conversational agent to sound convincing in a browser test is genuinely the easy part now. What surfaced problems was putting it on a real phone line, talking to a real person with no patience for an AI that talks over them.

The call that dropped after one line

The first real test call connected, the agent said its opening line, and the call ended. Nobody hung up. The agent's own turn-handling had stalled under latency. The model we'd started with simply wasn't fast enough to keep a phone conversation feeling like one. Switching to a faster model in the same family fixed it outright. The difference between usable on a phone call and not came down to milliseconds we couldn't see in the text-only testing we'd been doing.

A checkbox we didn't know existed

Once calls were going out cleanly, the next problem was quieter and much harder to find. Real qualifying calls were happening, but nothing was coming back. No score in the CRM, no call summary, nothing. That looked like an automation bug, so that's where we spent most of a day looking.

It wasn't. ElevenLabs lets you choose which events actually trigger its post-call webhook, and "Transcription completed", the one event our entire pipeline depended on, was simply unchecked. The webhook URL was correct. The signature was (eventually) correct. It was never being asked to fire in the first place.

If a step in an automated pipeline "isn't happening" and nothing anywhere is logging an error, check whether the step was ever actually subscribed to its trigger before you debug anything downstream of it.

What actually matters once it's a real phone call

A few things only became obvious once the agent was making real calls instead of browser demos.

Latency

Every extra second of thinking time before the agent responds reads as dead air to a caller on a phone, in a way it doesn't in a chat window. This is the single biggest lever on whether a call feels natural or feels like talking to a machine.

Audio quality on an actual phone line

Phone audio is narrowband and compressed in ways a laptop speaker test never reveals. Voice settings and audio presets that sounded perfectly natural in a browser needed real tuning once we were listening over an actual handset.

The glue between the tools is where the real work happened

Twilio, ElevenLabs and Twenty each behaved reasonably once we understood them individually. Almost every hour of genuinely hard debugging went into the connective tissue between them: the webhooks, the signatures, the workflow platform sitting in the middle of it all.

Writing our own encryption, because Node's crypto module was blocked

ElevenLabs signs its webhook payloads with a standard HMAC-SHA256 signature, a well-understood five-minute job in any normal Node environment. Except our self-hosted automation platform's code-execution sandbox blocks Node's built-in crypto module entirely, for security reasons that make sense in isolation and were extremely inconvenient in the moment.

The fix was writing a pure-JavaScript SHA-256 and HMAC implementation from scratch, by hand, inside a workflow step. Bit-shifting and byte arrays, just to check a header that any standard library would have handled in one line. It works. It also shouldn't have been necessary, and it's the kind of thing worth checking for before you commit to a self-hosted platform.

The outage that turned out to be us

Midway through debugging, webhooks that had been working hours earlier started returning 404s, including ones we'd personally confirmed working that same morning. Every signal pointed at an instance-wide outage. Repeated deregistration events in the logs. Our own test requests failing consistently. We got as far as planning a full service restart.

Then a real lead submitted the form and the whole pipeline ran correctly end to end, live, while our own test requests kept failing with the same 404. That contradiction was the clue. Our debugging requests were being routed through a different edge region than real traffic, a networking quirk specific to where we were running our tests from rather than anything wrong with the service.

We'd already drafted a restart plan before catching it. Lesson taken: when a real, current, successful execution log disagrees with your own reproduction, trust the log. Go looking for what's different about your own environment before you touch production.

Splitting the automation into five separate workflows

Why we split it up at all

The pipeline isn't one big workflow. It's five smaller ones, each owning a single job: taking the lead in, dispatching scheduled calls, the shared scoring logic, handling the results that come back from a call, and inbound messaging. That decomposition was the right call architecturally. Each piece is small enough to understand at a glance, and a mistake in one doesn't take the whole system down.

The rename nobody flagged

The cost showed up later, as drift. A field on a CRM record got renamed at some point. The newer workflows, written after the rename, used the new name and worked fine. An older workflow, untouched since before it, kept using the old one. Nothing about it looked wrong until it ran and failed with an error that only made sense once we compared the two workflows side by side.

Five independent workflows means five independent places that can quietly fall out of sync with the same underlying system, and none of them will tell you the other four have moved on without them.

Twenty CRM: building against a schema that's still being built

We're actively extending our own CRM's schema as the automation needs new fields, which means the shape we tested against yesterday isn't guaranteed to be the shape that's there today. Every one of the following was discovered by a failed write in production rather than by reading documentation first.

Rich text fields don't take a plain string

A field that looks like an ordinary text box in the CRM's own interface can expect a structured object behind the scenes. Sending it a string fails with an error that reads, at first glance, like the field itself is broken, until you realise it's a genuine type mismatch and not the one you'd guess.

Fields you're building around might not exist yet

We designed a scoring step that computes which qualification tier a lead falls into, long before checking whether the CRM had a field to put that result in. It didn't. Discovering a missing field mid-build, on a live call with a real lead's data sitting in the pipeline waiting to be saved, is a worse moment to find that gap than five minutes spent checking the schema up front.

What we'd tell ourselves, starting this again tomorrow

None of this was one big mistake. It was a long run of small, independent surprises, each invisible right up until it broke something with a real caller on the other end of the line. If we were starting the same build from scratch:

Before writing a single line of call logic

Upgrade the calling account first. The cost of the upgrade is nothing next to the hours lost debugging restrictions that only exist on the free tier.

Whenever a webhook step "just isn't firing"

Check whether the trigger is actually subscribed to the event you're expecting, before you go anywhere near the receiving end. It'll look identical to a real bug from every other angle.

A two-minute check that would have saved us a day

Open the webhook's own event configuration screen and confirm the checkbox is ticked. We had the URL right, the payload right and the receiving workflow right, for hours, before finding the one setting that made none of it matter.

On the CRM side

Treat every field's type as something to confirm before you write to it, not something to discover from a failed request in production. And build the destination field before you build the logic that depends on it existing.

None of it was glamorous. All of it is why the thing works now, reliably, on real calls, without either of us babysitting it.