Parley — self-hosted voice AI agent

Parley is a voice agent that answers real phone calls and holds a natural conversation — built to replace a rented platform (Retell) with a voice layer that lives inside my own Spring application. Twilio handles the telephony; Parley owns everything from the words onward.

Why self-host

Managed voice-AI platforms hide latency, turn-taking, and backend access behind a black box. I wanted to see what it feels like to own those decisions: interpret the caller, run the LLM, decide what to say or do, and reach into the same backend my other services use — without a vendor boundary in the middle.

Architecture

• Telephony: Twilio inbound call → webhook into Spring Boot.
• Session: Per-call state for transcript, tool context, and turn boundaries.
• LLM: Spring AI for completion, with prompts and tools defined in-app.
• Actions: Backend calls (lookups, updates) triggered from agent decisions — not hard-coded IVR trees.
• Audio path: Streaming where possible; measure end-to-end latency per turn.

What I'm learning

Voice is unforgiving. Half-second delays feel broken. Turn-taking heuristics that work in chat fail on a phone line. Owning the stack means owning those failure modes — and that's the point of the project.

Stack

• Java, Spring Boot, Spring AI
• Twilio Voice
• LLM providers via Spring AI abstractions

Status

Inbound call handling and core conversation loop are in place. Active work: tighter turn-taking, tool reliability under real call noise, and observability across the audio and LLM path.