CLI Agent Orchestrator Evaluation

CLI Agent Orchestrator Evaluation (#1048)

Evaluated 2026-10-01 by reading the repository’s docs and source. Nothing was cloned, installed or run. The source tip was an X post by @dankornas; the post itself was not checked, but awslabs/cli-agent-orchestrator is live and is the only orchestrator AWS Labs publishes under that name.

Field	Value
What it is	“Multi-agent orchestration for AI coding CLIs — Claude Code, Kiro, Codex, and more, coordinated in isolated tmux sessions”
License	Apache-2.0, Python
Activity	1,366 stars, 73 contributors, pushed 2026-10-01, release v2.5.1 (2026-09-11)

Verdict: reference-only. Take no action beyond one small follow-up. CAO is a capable, well-maintained supervisor and worker runtime. Its strength, driving 18 different agent CLIs through one interface, is something BoB does not need: BoB runs on Claude Code only. Adopting it would add a second source of truth (a SQLite runtime and a Python daemon on port 9889) beside GitHub Issues and the warp-drive state machine, against the Prime Directive. It also works out an agent’s status by reading its screen, and starts Claude Code with --dangerously-skip-permissions by default. Both get around BoB’s hook-gated automation levels.

Architecture

• Runtime. Three pieces run locally:
  • the cao CLI;
  • cao-server, a FastAPI server on localhost:9889 that serves HTTP, server-sent events and a terminal WebSocket, plus a React web UI;
  • two MCP servers, one for agents inside a session and one for managing the whole fleet from outside.
• Hosting. Each agent is a full agent CLI running in its own tmux window. A provider adapter for each CLI (18 of them; Kiro is the default) owns the launch command, input, exit and status. For Claude Code, the status (idle, processing, waiting for the user, completed, error) is worked out by scanning the screen for the prompt, the spinner and the output marker.
• Lifecycle. cao launch --agents <profile> starts a session and cao shutdown ends it. When a terminal is deleted, its scrollback and a snapshot are saved under ~/.cao/logs/terminal/ for seven days, and cao terminal restore can replay them.
• Orchestration. A supervisor delegates to workers through three MCP tools:
  • handoff is synchronous: it waits for the worker to finish, returns the output, then removes the worker;
  • assign is asynchronous: the worker reports back later;
  • send_message puts a message in an inbox stored in SQLite, delivered as a paste when the receiving agent is idle.
• Inbox delivery. Three layers make it reliable: an immediate attempt, a five-second watchdog, and a 30-second sweep with a grace window, so no message stays pending forever.
• State and profiles. Sessions, terminals and the inbox live in SQLite. Agent profiles are Markdown with frontmatter, and each role maps to a list of allowed tools.
• Memory. A Markdown wiki with keyword search is injected at session start. An optional loop has a “retrospector” agent write lessons that can be promoted into a profile.
• Scheduling and workflows. cao schedule runs agent sessions on a cron schedule, behind a gate script that answers {"execute": bool, "output": {}}. cao workflow runs Python workflows that resume by replaying a journal.

Overlaps and gaps

Capability	CAO	BoB	Note
Agents in tmux	One window per agent	bob fork, segment spawning	Same substrate
Supervisor delegating to workers	handoff, assign, send_message over MCP	Claude Code’s subagents and Agent Teams; bob swarm	BoB uses Claude Code’s own primitives
Messages between running agents	SQLite inbox	None between sessions; the remote decision bridge connects a human and an agent	CAO’s one genuinely new primitive
Knowing an agent’s state	Reads the screen	Reads the warp-drive state file	BoB’s way is sturdier; #3549 keeps it that way
Worktree per agent	Only in one example	bob fork	BoB ahead
Governance: issues, gates, hook-gated commits, automation levels	None; permissions skipped by default	Core of BoB	BoB ahead
Many agent CLIs	18	Claude Code only	Not a BoB goal
Scheduled runs	cao schedule with a gate script	bob tick, bob ritual due	Equivalent
Resumable workflows	Journal replay	Claude Code’s Workflow scripts	Already covered
Output kept after teardown	Scrollback and snapshot saved	bob warp reap kills sessions without saving output	Small gap; follow-up below
Memory and lessons	Wiki, retrospector, promote	Memory conventions, /journal, /memory-audit	Overlapping
Fleet view	Web UI	bob viz, bob warp viz	Equivalent

Patterns worth keeping

1. Save a session’s scrollback before killing it (follow-up #3552). bob warp reap and the segment teardown could save tmux capture-pane -pS - to a short-lived log under .claude/ before killing the session. That keeps the evidence a harness-defect report needs.
2. Pasting into tmux, for later. BoB sends single lines with send-keys today, so it is not affected. If it ever sends multi-line text into a pane, tmux 3.7 and later clean out hand-made bracketed-paste markers, and paste-buffer -p is the safe way to do it.
3. Three-layer delivery. An immediate attempt, a watchdog and a sweep with a grace window. This is the reference design if #3550 (forked workers that start the items a landed item was blocking) ever needs messages between running workers.
4. The gate script’s {execute, output} answer. A clean shape, but bob ritual due and bob tick already make the same decision.

Not verified

• The X post itself.
• Any runtime behaviour: everything above comes from the docs and the MCP tool names, not from running it.
• Whether CAO isolates agents in worktrees outside its example.
• Whether cao-server has any authentication beyond listening on localhost.
• Whether port 9889 clashes with BoB’s dashboard port bands.