Talking to coding agents from the terminal

A realtime voice layer for local coding agents: open a repo, run rtd, speak, and let Codex or Claude work against the files already on disk.

I wanted to stay in the terminal, say what I needed, and watch the work start without managing a chat thread.

Open a terminal in a repo, run rtd, and speak. The app listens continuously, turns complete instructions into small coding tasks, and sends them to a local agent. Codex is the default and Claude is supported. The terminal shows the state of the work without filling the screen with a transcript.

The interaction was inspired by the work Jaytel has been sharing around voice-driven software creation: speak naturally, keep the tool close to the work, and let the interface show just enough state to stay trustworthy.

What it does

Real Time Design is a realtime voice layer for local coding agents. You can say something like Make the header smaller and run the tests. The voice layer hears that as two separate jobs: edit the header, then run tests.

The scheduler starts separate agent runs when jobs can happen independently. Corrections while a task is running are treated as a barge-in. If I say Make the header red. and then Actually make it blue instead., the first run stops and restarts with the correction.

Interruptions were the interaction I cared about most. People revise themselves, add another request, cancel, or change their mind while the machine is already working. The app has to handle that.

The shape of the system

The app has four main pieces:

The Realtime model has one job in this project: voice and intent. It listens to audio, waits for a complete thought with semantic VAD, and calls a small tool called code_intents.

{
  "intents": [
    {
      "action": "edit",
      "target": "navigation",
      "description": "Make the nav red"
    },
    {
      "action": "run",
      "target": "typecheck",
      "description": "Run the typecheck"
    }
  ]
}

After that, Real Time Design owns the workflow. It dedupes repeated events, decides whether the new instruction is a fresh task, a cancellation, or a correction, and dispatches work to the selected coding CLI.

The split stays clear: the voice model decides whether I said something actionable, and the coding agent uses the repo and its tools to make the change.

Terminal UI decisions

I used Ink, which renders React in the terminal. The app tracks the connection, microphone, transcript, queue, active and finished runs, selected model, errors, and keyboard shortcuts. React gave me normal component boundaries for all of that state.

The UI is compact: the top panel shows listening state, waveform, and latest heard text; the middle panel shows active and recent tasks; the bottom bar shows status, queue count, active model, and shortcuts.

The terminal should answer three questions fast: is it listening, what did it think I said, and what agents are working right now?

The app uses the terminal's alternate screen when possible. When I quit, the previous terminal buffer comes back cleanly.

Keeping the UI quiet

Coding agents produce a lot of output. Full logs are useful for debugging, but they make a poor main interface.

The task list shows a short inferred step such as step: editing src/components/Nav.tsx, step: running npm run typecheck, or step: finished. It is guessed from agent output, so it is imperfect, but it is enough to show whether the agent is reading, editing, testing, or done.

I wanted calm status at a glance.

Parallel work

The scheduler can run up to four local agents at once. If the Realtime layer splits one spoken request into three independent intents, all three can start together.

This is risky if two agents touch the same file, so every generated prompt reminds the worker that other agents may be running in the same workspace and that it must inspect the current tree before editing. The scheduler also keeps barge-in decisions target-aware, so actually make the nav blue tries to restart the nav task instead of killing some unrelated run.

Conflict handling is still basic: inspect the tree first, keep each task narrow, and only parallelize clearly independent work. Every agent is reminded that the workspace is shared.

Agent and model picker

The app detects installed CLIs and prefers Codex when both Codex and Claude are available. You can also pick explicitly with rtd --agent codex or rtd --agent claude.

The default model is gpt-5.3-codex-spark because it starts quickly enough for a spoken workflow. A few seconds after I finish a sentence, the app begins changing.

Inside the TUI, pressing a opens a picker for future tasks. Active jobs keep the model they started with, so changing the picker only affects the next run.

Audio and Realtime plumbing

The voice layer should feel uninterrupted. I start rtd, talk, pause, correct myself, and continue without pressing a button or managing a transcript.

Realtime handles the listening loop and waits until an instruction sounds complete. When there is a coding task, it emits structured intents for the scheduler. When I am thinking out loud, it should stay quiet.

There is no push-to-talk or separate send step. I speak and revise; the terminal decides when it has an actionable instruction.

The rough edges

This is a prototype. Barge-in matching is heuristic, reconnect behavior is basic, progress summaries are inferred from CLI output, parallel agents can still collide on overlapping edits, and the UI targets a focused terminal size.

The main loop works: talk, create structured tasks, dispatch local agents, interrupt them when needed, and keep the terminal readable.

What I learned

The main design work was deciding where each responsibility lives. Realtime handles listening and intent extraction. The scheduler handles task shape, dedupe, queueing, cancellation, and parallel dispatch. Codex or Claude works in the repo. Ink renders the stateful terminal interface.

Keeping those pieces separate makes the prototype easier to change.

I can stay in the repo, talk through small changes, correct myself, and watch several local agents work at once. That is the workflow I wanted.

P.S. This is also really good for using LLMs as a writing companion. I mostly spoke this entire post, corrected myself, asked for visualizations, and kept shaping the page out loud. Give it a try: the code is on GitHub, and you can install it with npm install -g real-time-design.