Tool-calling over a voice WebSocket: keeping one contract for text and speech
A voice agent that can act — pull up pages, fetch real data mid-sentence — needs its tools to live somewhere both the REST chat loop and the audio WebSocket can execute them. The seam is the tool contract, not the transport.
Adding real-time voice to an existing text assistant looks like an audio problem (PCM formats, sample rates, interruption handling). The harder design problem turned out to be architectural: the text loop executes tools server-side inside one HTTP request, while the voice session is a WebSocket that terminates in the browser — which can't run server-only code like database reads or GitHub fetches.
The shape that worked: one shared tool module (declarations + an executeTool function) used three ways. The REST chat loop calls it in-process. The voice client receives toolCall frames over the WebSocket and proxies them to a thin API route that calls the same executeTool. And the ephemeral session token bakes the same declarations into its constraints, so a leaked token can't be repurposed into a general assistant.
Ephemeral tokens matter more for voice than text: the browser must connect directly to the model provider for latency, so it must hold a credential. Locking model, system prompt and tool list into the token at mint time means the credential is only good for being Rimuru, briefly.
Interruption handling is where audio does get interesting: when the user talks over the model, the server sends an interrupted flag, and the client has to discard its queued playback buffer immediately — an audio cursor reset, or the model audibly finishes a sentence the user already cut off.
Unresolved: transcript identity. The input transcription stream and the output transcription stream arrive as separate fragments with no turn ids, so stitching a clean conversation log out of a barge-in-heavy session is still heuristic.