Terminal automation CLI for AI agents
Like agent-browser, but for TUI applications.

Installation • Quick Start • Commands • AI Agents

pilotty enables AI agents to interact with terminal applications (vim, htop, lazygit, dialog, etc.) through a simple CLI interface. It manages PTY sessions, captures terminal output, and provides keyboard/mouse input capabilities for navigating TUI applications.

• PTY Management: Spawn and manage terminal applications in background sessions
• Keyboard Navigation: Interact with TUIs using Tab, Enter, arrow keys, and key combos
• AI-Friendly Output: Clean JSON responses with actionable suggestions on errors
• Multi-Session: Run multiple terminal apps simultaneously in isolated sessions
• Zero Config: Daemon auto-starts on first command, auto-stops after 5 minutes idle

npm install -g pilotty

git clone https://github.com/msmps/pilotty
cd pilotty
cargo build --release
./target/release/pilotty --help

Requires Rust 1.70+.

Windows is not supported due to the use of Unix domain sockets and POSIX PTY APIs.

# Spawn a TUI application
pilotty spawn htop

# Take a snapshot of the terminal
pilotty snapshot

# Type text
pilotty type "hello world"

# Send keys
pilotty key Enter
pilotty key Ctrl+C

# Click at specific coordinates (row, col)
pilotty click 10 5

# List active sessions
pilotty list-sessions

# Stop the daemon
pilotty stop

pilotty spawn <command>           # Spawn a TUI app (e.g., pilotty spawn vim file.txt)
pilotty spawn --name myapp <cmd>  # Spawn with a custom session name
pilotty kill                      # Kill default session
pilotty kill -s myapp             # Kill specific session
pilotty list-sessions             # List all active sessions
pilotty stop                      # Stop the daemon and all sessions
pilotty daemon                    # Manually start daemon (usually auto-starts)
pilotty examples                  # Show end-to-end workflow example

pilotty snapshot                  # Full JSON with text
pilotty snapshot --format compact # JSON without text field
pilotty snapshot --format text    # Plain text with cursor indicator

pilotty type "hello"              # Type text at cursor
pilotty key Enter                 # Send Enter key
pilotty key Ctrl+C                # Send Ctrl+C
pilotty key Alt+F                 # Send Alt+F
pilotty key F1                    # Send function key
pilotty key Tab                   # Send Tab
pilotty key Escape                # Send Escape

pilotty click 10 5                # Click at row 10, col 5
pilotty scroll up                 # Scroll up 1 line
pilotty scroll down 5             # Scroll down 5 lines

pilotty resize 120 40             # Resize terminal to 120x40
pilotty wait-for "Ready"          # Wait for text to appear
pilotty wait-for "Error" --regex  # Wait for regex pattern
pilotty wait-for "Done" -t 5000   # Wait with 5s timeout

The snapshot command returns structured data about the terminal screen:

{
  "snapshot_id": 42,
  "size": { "cols": 80, "rows": 24 },
  "cursor": { "row": 5, "col": 10, "visible": true },
  "text": "... plain text content ..."
}

Use the cursor position and text content to understand the screen state and navigate using keyboard commands (Tab, Enter, arrow keys) or click at specific coordinates.

Each session is an isolated terminal with its own:

• PTY (pseudo-terminal)
• Screen buffer
• Child process

# Run multiple apps (--name must come before the command)
pilotty spawn --name monitoring htop
pilotty spawn --name editor vim file.txt

# Target specific session
pilotty snapshot -s monitoring
pilotty key -s editor Ctrl+S

# List all sessions
pilotty list-sessions

If no --session is specified, pilotty uses the default session.

Note: The first session spawned without --name is automatically named default. To run multiple sessions, give each a unique name with --name:

pilotty spawn --name monitoring htop
pilotty spawn --name editor vim

Important: The --name flag must come before the command. Everything after the command is passed as arguments to that command.

pilotty uses a daemon architecture similar to agent-browser:

┌─────────────┐     Unix Socket      ┌─────────────────┐
│   CLI       │ ──────────────────▶  │     Daemon      │
│  (pilotty)  │     JSON-line        │  (auto-started) │
└─────────────┘                      └─────────────────┘
                                              │
                                     ┌────────┴────────┐
                                     ▼                 ▼
                              ┌───────────┐     ┌───────────┐
                              │  Session  │     │  Session  │
                              │  (htop)   │     │  (vim)    │
                              └───────────┘     └───────────┘

• Auto-start: Daemon starts automatically on first command
• Auto-stop: Daemon shuts down after 5 minutes with no active sessions
• Session cleanup: Sessions are automatically removed when their process exits
• Background: Runs in background, survives terminal close
• Shared state: Multiple CLI invocations share sessions
• Clean shutdown: pilotty stop gracefully terminates all sessions

The daemon is designed for zero-maintenance operation:

1. First command (e.g., pilotty spawn vim) starts the daemon automatically
2. Session ends (e.g., vim exits after :wq) and the session is cleaned up within 500ms
3. Idle timeout: After 5 minutes with no sessions, the daemon shuts down
4. Next command starts the daemon again automatically

This means you never need to manually manage the daemon, it starts when needed and stops when idle.

The daemon socket is created at (in priority order):

1. $PILOTTY_SOCKET_DIR/{session}.sock (explicit override)
2. $XDG_RUNTIME_DIR/pilotty/{session}.sock (Linux standard)
3. ~/.pilotty/{session}.sock (home directory fallback)
4. /tmp/pilotty/{session}.sock (last resort)

All errors include AI-friendly suggestions:

{
  "code": "SESSION_NOT_FOUND",
  "message": "Session 'abc123' not found",
  "suggestion": "Run 'pilotty list-sessions' to see available sessions"
}

Add the skill to your AI coding assistant for richer context:

npx skills add msmps/pilotty

This works with Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Goose, OpenCode, and Windsurf.

The simplest approach - just tell your agent to use it:

Use pilotty to interact with vim. Run pilotty --help to see available commands.

The --help output is comprehensive and most agents can figure it out from there.

For more consistent results, add to your project or global instructions file:

## Terminal Automation

Use `pilotty` for TUI automation. Run `pilotty --help` for all commands.

Core workflow:
1. `pilotty spawn <command>` - Start a TUI application
2. `pilotty snapshot` - Get screen state with cursor position
3. `pilotty key Tab` / `pilotty type "text"` - Navigate and interact
4. Re-snapshot after screen changes

# 1. Spawn the application
pilotty spawn vim myfile.txt

# 2. Wait for it to be ready
pilotty wait-for "myfile.txt"

# 3. Take a snapshot to understand the screen
pilotty snapshot

# 4. Navigate using keyboard commands
pilotty key i                    # Enter insert mode
pilotty type "Hello, World!"
pilotty key Escape
pilotty type ":wq"
pilotty key Enter

# 5. Re-snapshot after screen changes
pilotty snapshot

Supported key formats:

Contributions welcome! Please:

1. Run cargo fmt before committing
2. Run cargo clippy --all --all-features and fix warnings
3. Add tests for new functionality
4. Update documentation as needed

MIT

• Inspired by agent-browser by Vercel Labs
• Built with vt100 for terminal emulation
• Built with portable-pty for PTY management