--- name: ocelo description: > Autonomous agent skill and CLI reference for recording native macOS apps and websites with Ocelo: orchestrate app windows, capture video, apply eased camera zooms and captions, and export MP4/HEVC/GIF files without manual GUI interaction. --- # Ocelo Agent Skill & Automation Reference Ocelo is a native macOS app that captures a window or screen and exports a finished product video with smooth eased zooms, captions, cursor smoothing, and styling. The app bundle contains the same engine as a command-line interface designed for scripts and AI agents: single process, one take, one output file, machine-readable JSON on stdout. There is no background server and nothing to install beyond the app itself. --- ## 1. Fast Reference & Binary Location The binary lives inside the installed application bundle: ```sh /Applications/Ocelo.app/Contents/MacOS/Ocelo --list-windows --json ``` Or from local development builds: - `./.build/release/Ocelo` - `$OCELO_BIN` (if exported) Two primary verbs exist: 1. `--list-windows`: Query capturable windows, displays, or apps. 2. `--record`: Record, edit, compose, and export in a single pass. Running the binary without either flag opens the standard desktop GUI. --- ## 2. The Agent Execution Protocol Autonomous agents driving Ocelo must follow this 5-step operational protocol to guarantee reliable takes: ``` [1. Pre-flight Check] ──> [2. Stage Subject] ──> [3. Roll & Sync on stderr] ──> [4. Drive Subject] ──> [5. Verify Export] --list-windows Web / Native App wait for "· recording" AppleScript / CDP ffmpeg frame check ``` ### Step 1: Pre-flight & TCC Permission Check Screen Recording is a per-application macOS TCC permission. **The CLI cannot grant it for itself.** Run: ```sh Ocelo --list-windows --json ``` - **Exit code `0`**: Permission granted. Proceed. - **Exit code `3`**: Screen Recording not granted (ScreenCaptureKit reports empty content). **Action required**: Instruct the human user to open **System Settings › Privacy & Security › Screen Recording** and enable Ocelo (or the calling Terminal app). ### Step 2: Stage the Subject (Apps vs. Websites) - **Native macOS Apps**: Use `--launch "App Name"` (or `--launch /path/to/App.app`) and `--activate`. Ocelo raises the window and re-reads its frame geometry, ensuring it is on the active macOS Space. - **Websites**: Open the URL in Chrome or Safari, size the window (e.g. 1380×860), and ensure the target tab is frontmost. ### Step 3: Roll & Synchronize on `stderr` Capturing is asynchronous. **Never use a fixed `sleep` to guess when capture begins.** 1. Start `Ocelo --record ...` in the background with stderr redirected. 2. Monitor stderr for the line: ``` · recording — for <seconds>s ``` 3. Only begin driving interactions **after** this line is emitted. ### Step 4: Camera & Timing Rules - **Explicit Zooms Required**: Automatic zooms are triggered by real hardware clicks. Because agent-driven captures do not generate hardware clicks, the auto-zoom heuristic finds nothing. **You must supply explicit `--zooms`**. - **The 1.5-Second Rule**: The zoom parameter `at` specifies when the camera has **completely settled** on the crop. The camera path calculates an ease curve *around* `at`. Set `at` at least **1.5 seconds after** the interaction or scroll finishes. Never zoom across an active page transition or fast scroll. - **Coordinate Model**: Coordinates `u` (horizontal) and `v` (vertical) are normalized fractions `[0.0, 1.0]` of the target window frame. $(0.0, 0.0)$ is top-left; $(0.5, 0.5)$ is dead center. ### Step 5: Verification Loop Never report success to the user without inspecting the output. Extract frames at key timestamps: ```sh ffmpeg -y -ss 2.0 -i output.mp4 -frames:v 1 /tmp/beat-1.png ffmpeg -y -ss 6.0 -i output.mp4 -frames:v 1 /tmp/beat-2.png ``` Inspect the images to verify that the target window was in focus, content was rendered, and no black frames or permission dialogues were recorded. --- ## 3. High-Level Automation Workflows Ocelo provides ready-made scripts in its repository for orchestrating complex recordings without writing custom bash timing loops. ### Workflow A: Declarative Storyboards (`scripts/shoot.sh`) Shoots a complete multi-shot tour defined in a single JSON storyboard file. It handles browser staging over Chrome DevTools Protocol (CDP), native app launching, synchronization, and driver execution. ```sh ./scripts/shoot.sh docs/storyboards/tour.json out/tour.mp4 ``` #### Storyboard Schema Example (Native App): ```json { "app": "Spotify", "lead": 1.5, "tail": 1.5, "shots": [ { "label": "Next Track", "travel": 1.0, "hold": 3.0, "caption": "Skipping to next track via AppleScript", "zoom": { "u": 0.5, "v": 0.88, "scale": 1.8 }, "pointer": { "u": 0.5, "v": 0.88, "travel": 0.8 }, "actions": [ { "at": 0.2, "type": "applescript", "script": "tell application \"Spotify\" to next track" } ] } ] } ``` #### Storyboard Schema Example (Web Page): ```json { "url": "https://example.com", "width": 1440, "height": 900, "shots": [ { "label": "Hero Section", "travel": 0.0, "hold": 3.5, "caption": "Welcome to Example", "zoom": { "u": 0.5, "v": 0.35, "scale": 1.6 } }, { "label": "Feature Tour", "travel": 1.2, "hold": 4.0, "caption": "Integrated analytics", "zoom": { "u": 0.6, "v": 0.5, "scale": 2.0 } } ] } ``` ### Workflow B: Web 1-Liner (`scripts/record-site.sh`) A dedicated script for fast website showcases. It stages Chrome/Safari, resizes the browser to 1380×860, handles keyboard-based scrolling (`Home`, `PageDown`), and records with zooms and captions: ```sh scripts/record-site.sh https://example.com \ --seconds 12 \ --captions '[{"at":0.5,"hold":3.0,"text":"Explore our new platform"}]' ``` --- ## 4. Driving Targets: Native Apps vs. Websites | Subject | Recommended Driving Technique | Security / Permissions Required | |---|---|---| | **Native Mac App (Scriptable)** | AppleScript (`tell application "..." to ...`) | Standard macOS Automation dialog (prompted once per app) | | **Native Mac App (UI Controls)** | `ocelo-drive` or System Events key/click simulation | macOS Accessibility permission | | **Website (Smooth scroll)** | Chrome DevTools Protocol (CDP) or `window.scrollTo` in JS | Remote debugging port or browser developer menu | | **Website (Section jumps)** | System Events key codes (`PageDown`: 121, `Home`: 115) | macOS Accessibility | > **Synthetic Mouse Scrolls Fail**: macOS browsers silently ignore synthetic `CGEvent` mouse wheel events. When scrolling web pages, use keyboard key codes (`PageDown` / `Home`) or JavaScript execution (`window.scrollTo({ top: ..., behavior: 'smooth' })`). --- ## 5. Direct CLI Reference (`Ocelo`) For custom pipelines, drive `/Applications/Ocelo.app/Contents/MacOS/Ocelo` directly. ### `--list-windows` List capturable targets in JSON format: ```sh Ocelo --list-windows --json ``` Output: ```json { "windows": [ { "id": "window-4465", "title": "Slack - General", "app": "Slack", "width": 1400, "height": 900, "kind": "window" } ] } ``` ### `--record` Flags #### Target Selection | Flag | Description | |---|---| | `--window <name>` | Window ID, app name, or title substring. Picks frontmost window if omitted. | | `--display` | Record the entire primary monitor instead of a window. | | `--launch <app>` | Launches the app (`open -g -a`), waits up to 10s for its window, then records. Implies `--activate`. | | `--activate` | Focuses and raises target window before recording, then recalculates window bounds. **Recommended** for correct $(u, v)$ coordinates. | #### Capture Settings | Flag | Default | Description | |---|---|---| | `--seconds <n>` | `10` | Total duration of the recording. | | `--fps <n>` | `60` | Export frame rate. | | `--quality <q>` | `full` | `full`, `balanced`, or `small`. | | `--settle <n>` | `0` (`2.5` with `--launch`) | Seconds to wait after the window appears before capture starts (avoids loading skeletons). | | `--countdown <n>` | `0` | Seconds of pre-roll countdown. | #### Edit Specifications All specs accept inline JSON or a file path reference `@/path/to/spec.json`. - **Zooms** (`--zooms`): ```json [ { "at": 2.0, "u": 0.5, "v": 0.4, "hold": 2.5, "scale": 2.0 } ] ``` - `at`: Second in take when camera finishes easing and arrives at the crop target. - `hold`: Duration in seconds to stay magnified (default: `2.0`). - `scale`: Magnification factor (default: `2.0`). - `u`, `v`: Center coordinates as fractions of window width and height. - **Captions** (`--captions`): ```json [ { "at": 1.0, "hold": 3.0, "text": "Real-time sync enabled" } ] ``` - **Pointer** (`--pointer`): ```json [ { "at": 1.5, "u": 0.5, "v": 0.5, "travel": 0.8 } ] ``` Warps the system cursor smoothly over `travel` seconds. *(Note: Pointer movement does not trigger mouse hover, click, or scroll events.)* #### Output Options | Flag | Default | Description | |---|---|---| | `--out <path>` | `recording.mp4` | Destination file path. | | `--format <f>` | `h264` | `h264`, `hevc`, or `gif`. | | `--aspect <a>` | `source` | `source`, `16:9`, `4:3`, `1:1`, `9:16`, or `4:5`. | | `--max-width <n>`| `1920` | Maximum pixel width of output video. | | `--no-alpha` | off | Disables transparency pipeline. **Always pass for opaque apps and browsers** to use faster H.264 path. | | `--corner-radius <n>` | none | Add rounded corners to the recorded window over background. | | `--json` | off | Outputs metadata to stdout on completion. | --- ## 6. Exit Codes Reference | Exit Code | Cause | Recommended Action | |---|---|---| | `0` | Success | Verify exported file via `ffmpeg`. | | `2` | Window list could not be read | Check OS window server state. | | `3` | **Screen Recording permission denied** | Direct human user to System Settings › Privacy & Security › Screen Recording. | | `4` | Target window not found | Verify window title with `Ocelo --list-windows --json`. Check if app is on another Space. | | `5` | Capture failed to initialize | Check if display/window is valid and visible. | | `6` | Capture failed during recording | System sleep or target window closed mid-recording. | | `7` | Video export / encoding failed | Check destination directory write permissions and disk space. | | `8` | `--zooms` JSON parse error | Validate JSON syntax or use `@path/to/file.json`. | | `9` | `--pointer` JSON parse error | Validate JSON syntax. | | `10` | `--captions` JSON parse error | Validate JSON syntax. | | `11` | `--launch` app failed to create window | Increase `--settle` or inspect app launch status. | --- ## 7. Complete Worked Example: Recording a Native App Recording Spotify playing a song with an eased zoom on album art and an explanatory caption: ```sh # 1. Start recording Spotify with a 10-second capture window /Applications/Ocelo.app/Contents/MacOS/Ocelo --record \ --window "Spotify" \ --activate \ --seconds 10 \ --no-alpha \ --zooms '[{"at": 3.0, "u": 0.15, "v": 0.85, "hold": 3.0, "scale": 2.2}]' \ --captions '[{"at": 1.0, "hold": 3.5, "text": "Now Playing: Live Session"}]' \ --out spotify-demo.mp4 \ --json > result.json 2> progress.log & RECORDER_PID=$! # 2. Await sync point on stderr while ! grep -q "recording " progress.log 2>/dev/null; do sleep 0.1 done # 3. Trigger AppleScript action osascript -e 'tell application "Spotify" to play track "spotify:track:4cOdK2wGLETKBW3PvgPWqT"' # 4. Await process exit wait $RECORDER_PID # 5. Inspect result cat result.json ffmpeg -y -ss 3.0 -i spotify-demo.mp4 -frames:v 1 /tmp/spotify-verify.png ```