Parley — self-hosted voice AI agent

Parley is a voice agent that answers real phone calls and holds a natural conversation — built to replace a rented platform (Retell) with a voice layer that lives inside my own Spring application. Twilio handles the telephony; Parley owns everything from the words onward. Source on GitHub.

Why self-host

Managed voice-AI platforms hide latency, turn-taking, and backend access behind a black box. I wanted to see what it feels like to own those decisions: interpret the caller, run the LLM, decide what to say or do, and reach into the same backend my other services use — without a vendor boundary in the middle.

Architecture

• Telephony: Twilio inbound call → ConversationRelay over WebSocket into Spring Boot.
• Session: Per-call state for transcript, tool context, and turn boundaries.
• Routing: Each business has its own phone number and a BusinessProfile (name, wording, facts), so one app answers for several businesses and scopes each caller to the right one.
• LLM: Spring AI ChatClient on Azure OpenAI, with prompts and tools defined in-app.
• Inventory: eBay listings imported into MongoDB and held in an in-memory index; when a caller names a card, matching stock is injected into that turn.
• Actions: Backend calls (lookups, updates) triggered from agent decisions — not hard-coded IVR trees.
• Audio path: LLM replies streamed sentence by sentence, with native barge-in and per-turn, per-stage latency logging.
• After the call: Post-call summaries persisted to MongoDB; a turn-based fallback path for resilience.

What I'm learning

Voice is unforgiving. Half-second delays feel broken. Turn-taking heuristics that work in chat fail on a phone line. Owning the stack means owning those failure modes — and that's the point of the project.

Stack

• Java, Spring Boot, Spring AI
• Twilio Programmable Voice + ConversationRelay
• Azure OpenAI via Spring AI
• MongoDB Atlas
• Azure Container Apps, Terraform

Status

Parley is live and answering real phone calls for three businesses, each on its own number: Poke Pitch Shop, JMAC Intel, and JMAC Logistics. Callers hear the answer while it is still being generated, can interrupt naturally, and every reply passes through guardrails with business facts injected into each turn. For the card shop it looks up live inventory when a caller names a card and refuses to invent prices, store hours, or order status. Ongoing work: tool reliability under real call noise and deeper observability across the audio and LLM path.