Cadre Agent API — Tool Reference (v1)

All 62 tools, generated from the live zod input schemas in src/main/services/agent-api/tools/*.ts and the renderer command executor (src/renderer/agent/command-executor.ts). The Constraints column also folds in the semantic/cross-field checks from the runtime validation layer (src/main/services/agent-api/validate.ts) — ordering, sanity ceilings, path safety, and structural limits that a per-field zod schema cannot express. Every check documented below is enforced before the underlying command runs; a rejected call never reaches the editor or the filesystem.

Build status: this reference describes the current source build. Packaged and public release verification for the motion-graphics additions is still pending.

Conventions

• All times are milliseconds from the start of the source recording (not output time — cuts and speed changes remap output time, but tool inputs are always in recording time). Every time value is bounded to 0 – 86,400,000 ms (24 h) — comfortably beyond any plausible recording, while still rejecting absurd values a bare zod number() would accept as "finite" (e.g. 1e308).
• Every tool returns a single JSON block. On success it is the post-mutation entity or snapshot (so you can verify the effect without a second call). Five operations that can exceed common MCP client deadlines are the explicit exception: import_video, add_animation_overlay, generate_captions, generate_transcript, and download_caption_model return a background-job receipt immediately. Poll get_agent_job_status, then read get_agent_job_result. On failure it is { "error": { "code", "message", "hint" } } with isError: true. Validation failures always use INVALID_ARGS (or NOT_FOUND for an unloadable project manifest — see open_project); the message names the parameter, what was expected, and what was received, and hint is an actionable next step. See README.md §5 for the full error shape and code table.
• Rectangles (sourceRect) default to logical pixels of the source recording. Read recording.width / recording.height from get_timeline for the frame size, or pass "units": "normalized" and give 0–1 fractions instead — see Coordinate spaces.
• "Bridge-backed" tools mutate live editor state and require a project open in the editor; they return NO_PROJECT_OPEN / EDITOR_NOT_AVAILABLE otherwise.

Timebases

Read this before passing any timestamp to any tool.

There are two clocks in a Cadre project and they are not the same one:

They are equal only while the project has no cuts and no speed segments. After that they diverge by the total removed/rescaled duration — and the offset grows with every edit, including the edits you make.

The practical consequence: when a user says "cut 30 to 45 seconds", they are almost always reading the preview clock. If a 10-second cut already exists near the start, the material they mean lives at 40–55 s in recording time. Passing 30000, 45000 straight to add_cut removes fifteen seconds of the wrong footage, confidently and silently.

Rule: convert user-quoted timestamps with map_time (from: "output") before using them, and re-convert after each edit rather than reusing an earlier result. Timestamps you derived from get_timeline (keyframe bounds, cut bounds, caption timings) are already in recording time and need no conversion.

Coordinate spaces

sourceRect accepts either space, selected by an explicit units field:

The unit is never inferred from the magnitude of the numbers. { x: 0, y: 0, width: 1, height: 1 } is legal in both spaces — the whole frame normalised, or a single pixel at the origin — so a range heuristic would have to guess, and guessing wrong is a thousandfold error in crop size. Omitting units always means pixels, so every rect written before this field existed keeps its meaning.

Tool index

Shared shapes

Referenced by multiple tools.

LicenseStatus

{
  active: boolean;          // true when the user may export (incl. grace window)
  plan: 'free' | 'pro';
  email: string | null;     // null when unlicensed
  expiresAt: number | null; // subscription period end, epoch ms; null when
                            // unlicensed OR when the licence is perpetual
  perpetual: boolean;       // true for a one-time "Lifetime" licence that never expires
  inGrace: boolean;         // past expiresAt but inside the offline grace window
  mode: 'stub' | 'live';    // which integration backs this status
}

expiresAt: null has two meanings and perpetual is what separates them: an unlicensed app reports { active: false, perpetual: false, expiresAt: null }, a Lifetime licence reports { active: true, perpetual: true, expiresAt: null }. Never present a renewal date or a countdown for a perpetual licence — it has no expiry, and inGrace is always false for one.

ExportStatusSnapshot

{
  phase: 'idle' | 'exporting' | 'completed' | 'cancelled' | 'failed';
  progress: { percent: number; eta: number; currentFrame: number;
              totalFrames: number; stage?: string } | null;
  result:   { outputPath: string; fileSize: number; duration: number } | null;
  error: string | null;
}

Background-job receipt and status

The five potentially long operations return this receipt without waiting for FFmpeg, Whisper, or the network:

{
  jobId: string;          // opaque UUID
  jobToken: string;       // secret 256-bit capability; required for poll/cancel
  type: 'import_video' | 'add_animation_overlay' | 'generate_captions' |
        'generate_transcript' | 'download_caption_model';
  status: 'running';
  createdAt: string;      // ISO-8601
  pollAfterMs: 500;
  statusTool: 'get_agent_job_status';
  resultTool: 'get_agent_job_result';
  cancelTool: 'cancel_agent_job';
}

Keep jobId and jobToken together. The token is shown only in the initiating response. Jobs are scoped to the current app boot, run for at most their bounded operation window, retain at most 1 MiB of result data, and expire 15 minutes after settlement. At most four Agent jobs run concurrently, with stricter single-flight slots for import, transcription, and model download. See get_agent_job_status for the status shape.

ZoomKeyframe

{
  id: string;
  startTime: number; endTime: number;                 // ms
  sourceRect: { x: number; y: number; width: number; height: number };
  zoomLevel: number;                                  // 1 = full frame
  trigger: 'click' | 'typing' | 'scroll' | 'manual';
  isUserModified: boolean;
  isUserCreated: boolean;
}

Cut

{ id: string; startTime: number; endTime: number; type: 'remove' }

SpeedSegment

{ id: string; startTime: number; endTime: number;
  speed: number; rampIn: number; rampOut: number }

CaptionSegment

{ id: string; startTime: number; endTime: number;
  text: string; confidence: number; isUserEdited: boolean }

MusicTrack

{ trackId: string; filePath: string; name: string;
  volume: number; fadeInMs: number; fadeOutMs: number;
  loop: boolean; durationMs: number }

MaskSegment

{
  id: string;
  startTime: number; endTime: number;                 // ms
  type: 'blur' | 'highlight';
  rect: { x: number; y: number; width: number; height: number };  // normalised 0-1, source frame
  enabled: boolean;
  blurRadius: number;                                  // px (blur only)
  opacity: number;
}

OverlaySegment

{
  id: string;
  type: 'text' | 'svg' | 'graphic' | 'animation';
  startTime: number; endTime: number;                  // ms
  position: { x: number; y: number };                  // normalised centre, OUTPUT frame
  width: number;                                        // fraction of output width
  rotationDeg: number;
  opacity: number;
  text?: string;                                        // type 'text' only
  textStyle?: OverlayTextStyle;                         // type 'text' only
  svg?: string;                                          // type 'svg' only, sanitised markup
  graphic?: MotionGraphicSpec;                           // type 'graphic' only
  animationSource?: AnimationOverlaySource;              // type 'animation' only
  animation: OverlayAnimation;
  keyframes?: OverlayKeyframe[];                        // optional animation track, <= 60
}

MotionGraphicSpec

Every native graphic contains these canonical common fields after creation:

{
  title: string;                  // <= 80 characters
  subtitle: string;               // <= 120 characters
  theme: 'dark' | 'light';
  accentColor: string;            // #RRGGBB
  delayMs: number;                // 0-60,000, local recording-time offset
  durationMs: number;             // 0-60,000; 0 = instantaneous after delay
  easing: 'linear' | 'ease-in' | 'ease-out' | 'ease-in-out';
}

The kind selects one strict data shape; fields from another kind and unknown fields are rejected:

{ kind: 'counter'; from: number; to: number; decimals: 0|1|2|3;
  prefix: string; suffix: string; ...common }
{ kind: 'bar-chart'; bars: Array<{ label: string; value: number }>;
  maxValue?: number; suffix: string; ...common }
{ kind: 'progress-ring'; value: number; decimals: 0|1; ...common }

Counter values are finite and bounded to ±1 billion. Charts contain 1–8 non-negative values no greater than 1 billion; an explicit positive maxValue must cover every bar. Ring values are percentages from 0–100.

AnimationOverlaySource

{
  assetId: string;       // UUID naming a project-managed private directory
  frameCount: number;    // integer 1-3600
  fps: number;           // 1-60, sampled against local recording time
  width: number;         // integer 1-4096
  height: number;        // integer 1-4096
  loop: boolean;         // false holds the final frame
}

Frames live under style/assets/animations/<assetId>/frame-000000.png in natural playback order. They are managed project assets, not references to the import source.

LayoutSegment

{
  id: string;
  startTime: number; endTime: number;                  // ms, RECORDING time, end exclusive
  mode: 'fullscreen' | 'default' | 'hide' | 'split';
  layout?: { mode: 'fullscreen' | 'default' | 'hide' | 'split';
    speakerSide: 'left' | 'right'; speakerSize: number; gap: number; supportHeight: number };
  position: 'top-left' | 'top-center' | 'top-right'
          | 'middle-left' | 'middle-center' | 'middle-right'
          | 'bottom-left' | 'bottom-center' | 'bottom-right'
          | null;                                       // null = follow the global webcam position
  customSize: number | null;                            // 0.01-1 fraction of frame width; null = global size
  enabled: boolean;
}

The camera's layout over a span of time. Segments are non-overlapping, and the stretches they do not cover render the project's global style.webcam layout — so "go back to the corner pip" is a gap, not a second segment. null position / customSize mean "keep the global value", which is what a new segment gets, so it still follows the user if they move the camera later. Both are meaningful only in 'default' (Corner) mode.

OverlayKeyframe

{
  atMs: number;                                         // OFFSET from the overlay's startTime
  position?: { x: number; y: number };                  // normalised centre, OUTPUT frame
  scale?: number;                                        // multiplies the authored width
  rotationDeg?: number;
  opacity?: number;
  easing?: 'linear' | 'ease-in' | 'ease-out' | 'ease-in-out' | 'ease-out-back' | 'hold';
}

OverlayAnimation

{
  enter: 'none' | 'fade' | 'slide-up' | 'slide-down' | 'slide-left' | 'slide-right' | 'pop' | 'typewriter';
  enterDurationMs: number;
  exit: 'none' | 'fade' | 'slide-up' | 'slide-down' | 'slide-left' | 'slide-right' | 'pop';  // no typewriter on exit
  exitDurationMs: number;
  loop: 'none' | 'pulse' | 'spin' | 'bob';
  loopPeriodMs: number;
}

OverlayTextStyle

{
  fontFamily: string;
  fontSizeFrac: number;      // fraction of output frame height
  fontWeight: number;        // 100-900
  color: string;              // hex, text
  backgroundColor: string;    // hex, pill background
  backgroundOpacity: number;  // 0 removes the pill
  paddingFrac: number;        // fraction of font size
  cornerRadiusFrac: number;   // fraction of font size
  align: 'left' | 'center' | 'right';
}

App & license

get_app_state

High-level Cadre state. Call this first to orient before editing.

Parameters: none.

Returns:

{
  version: string;
  projectOpen: boolean;
  project: { id: string; name: string; path: string } | null;
  license: LicenseStatus;
  export: ExportStatusSnapshot;
}

Errors: none (always succeeds).

Example

// request
{ "name": "get_app_state", "arguments": {} }
// result
{
  "version": "<current-app-version>",
  "projectOpen": true,
  "project": { "id": "a1b2...", "name": "Onboarding demo", "path": "/Users/me/Movies/Onboarding demo.screencraft" },
  "license": { "active": true, "plan": "pro", "email": "me@example.com", "expiresAt": 1789000000000, "perpetual": false, "inGrace": false, "mode": "live" },
  "export": { "phase": "idle", "progress": null, "result": null, "error": null }
}

get_license_status

Current subscription/license status. Export is gated on active: true — check this before export_video.

Parameters: none. Returns: LicenseStatus. Errors: none.

// request
{ "name": "get_license_status", "arguments": {} }
// result
{ "active": false, "plan": "free", "email": null, "expiresAt": null, "perpetual": false, "inGrace": false, "mode": "live" }

A Lifetime (one-time) licence instead answers:

{ "active": true, "plan": "pro", "email": "me@example.com", "expiresAt": null, "perpetual": true, "inGrace": false, "mode": "live" }

Projects

list_projects

Recently modified projects, most recent first.

Returns: array of

{ id: string; name: string; path: string; modifiedAt: string;
  durationMs: number | null; thumbnailPath: string | null;
  sourceType: 'screen' | 'window' | 'ios' | 'area' | 'import' | null }

Rows whose .screencraft directory no longer exists on disk are pruned and omitted.

Errors: none.

// request
{ "name": "list_projects", "arguments": { "limit": 5 } }
// result
[
  { "id": "a1b2...", "name": "Onboarding demo", "path": "/Users/me/Movies/Onboarding demo.screencraft",
    "modifiedAt": "2026-07-19T21:03:11.000Z", "durationMs": 48200, "thumbnailPath": null, "sourceType": "screen" }
]

open_project

Open a project by its .screencraft directory path (use a path from list_projects). Loads the manifest, makes it the active project, and navigates the Cadre UI to the editor.

path is not required to end in .screencraft — the contract names the convention but does not enforce the suffix; the real guard is that project.json inside it must parse as a valid manifest (see the NOT_FOUND error below).

Returns: { id: string; name: string; path: string }.

Errors:

• INVALID_ARGS — path fails one of the path-safety checks above.
• NOT_FOUND — no loadable project at path: either nothing is readable there, or project.json does not parse as an object with a string id and name.

// request
{ "name": "open_project", "arguments": { "path": "/Users/me/Movies/Onboarding demo.screencraft" } }
// result
{ "id": "a1b2...", "name": "Onboarding demo", "path": "/Users/me/Movies/Onboarding demo.screencraft" }

save_project

Persist the editor's current live edit state to project.json. Run after a batch of edits to make them durable. (Cadre also autosaves ~3 s after any edit, so this is mostly for an explicit checkpoint.)

Parameters: none. Returns: { saved: true; path: string | null }. Errors: NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "save_project", "arguments": {} }
// result
{ "saved": true, "path": "/Users/me/Movies/Onboarding demo.screencraft" }

import_video

Import an existing video file as a new Cadre project — the entry point when a user hands the agent a raw file instead of a Cadre recording. Probes the video, scaffolds a .screencraft project, extracts the audio track, generates thumbnails, and opens it in the editor. Transcoding a non-H.264 source can take a while, so the initiating call returns a background-job receipt immediately instead of holding one MCP request open.

Returns immediately: a background-job receipt. Poll get_agent_job_status; once completed, get_agent_job_result contains result: { id: string; name: string; path: string } for the new project.

No interaction log. Unlike a Cadre recording, an imported video has no interactions.jsonl — there was no click/scroll capture — so recalculate_zooms has nothing to analyse for it. Add zoom keyframes manually with add_zoom instead.

Errors:

• INVALID_ARGS — path fails a path-safety check above, names an unsupported extension, or does not resolve to an existing regular file.

// request
{ "name": "import_video", "arguments": { "path": "/Users/me/Downloads/demo-raw.mp4" } }
// immediate result
{ "jobId": "...", "jobToken": "...", "type": "import_video", "status": "running",
  "pollAfterMs": 500, "statusTool": "get_agent_job_status",
  "resultTool": "get_agent_job_result", "cancelTool": "cancel_agent_job" }

// terminal get_agent_job_result (abridged)
{ "jobId": "...", "type": "import_video", "status": "completed",
  "resultAvailable": true,
  "result": { "id": "b7c1...", "name": "demo-raw", "path": "/Users/me/Movies/demo-raw.screencraft" } }

Recording

Seven tools that make new footage. This is the only part of the API with real-world side effects: it turns the user's camera-less screen into a file.

The whole flow, in the order an agent uses it:

list_recording_sources            # what can be recorded, and are the grants in place?
start_recording { source }        # returns only once capture is genuinely LIVE
  … sleep for as long as you want to capture …
stop_recording                    # returns { id, name, path } of the new project
get_timeline                      # the editor is already open on it — start editing

Four things to know before you call any of them.

1. Nothing records silently. start_recording runs the same code path as the app's own Record button: the countdown appears, the display flashes its blue border, the floating recording-controls bar shows up, and non-recorded displays dim. A user is always able to see (and stop) a recording an agent started. This is a product guarantee, not an implementation detail.
2. Recording and editing are free to try; export is the paywall. This is the same D-059 policy used by the app UI. start_recording does not require a subscription; export_video does.
3. There is no duration parameter and no auto-stop. You own the clock: sleep, then call stop_recording. The consequence is worth stating plainly — if your process dies mid-recording, the recording keeps running. Cadre's only backstop is the disk monitor, which force-stops the session when free space reaches the critical threshold. If you may not survive to call stop_recording, do not start a recording.
4. Success means the state machine moved. Every mutating call polls the real session state until it observes the target state, bounded by a timeout, and returns TIMEOUT otherwise. None of these tools reports success off an event it merely emitted.

Recording state machine. idle → countdown → recording ⇄ paused → finishing → completed → idle. pause_recording is legal only from recording, resume_recording only from paused, stop_recording from recording or paused, start_recording only from idle, and cancel_recording from anything except idle. Every other combination returns INVALID_STATE naming the state you are actually in — nothing silently no-ops.

Errors specific to this section (in addition to the shared codes):

Webcam capture is not available to agents. The camera is recorded by a preview window the user opens from the recording toolbar; an agent cannot conjure it, and a config that claims a webcam without one produces a recording whose camera track silently never arrives. start_recording therefore has no webcam parameter and always reports config.webcamEnabled: false. If the user wants a webcam in the take, they start that recording themselves.

list_recording_sources

Everything Cadre can record right now, plus the permission picture. Call this before start_recording and pass back an id from the result rather than guessing one.

Parameters: none.

Returns:

{
  screens: Array<{ type: 'screen'; displayId: number; displayName: string }>;
  windows: Array<{ type: 'window'; windowId: number; appName: string; windowTitle: string }>;
  iosDevices: Array<{ deviceId: string; name: string; model: string; osVersion: string;
                     connectionType: 'usb' | 'wireless'; screenWidth: number; screenHeight: number }>;
  microphones: Array<{ deviceId: string; name: string; kind: 'input'; isDefault: boolean }>;
  permissions: { screenRecording: 'not-determined' | 'granted' | 'denied' | 'restricted' | 'unknown';
                 microphone: 'not-determined' | 'granted' | 'denied' | 'restricted' | 'unknown';
                 accessibility: boolean; inputMonitoring: boolean };
  defaultSource: { type: 'screen'; displayId: number; displayName: string } | null;
  enumerationError: string | null;
}

screens[0] — and therefore defaultSource — is the main display, not whatever the OS enumerated first. That ordering is load-bearing: a QA session once recorded six minutes of an unattended external monitor because the default came from raw enumeration order (F-006).

displayId values are CGDirectDisplayIDs, the id space the capture engine uses. They are not Electron display ids and are not stable across disconnect/reconnect — re-enumerate rather than caching one.

This call never fails on a missing permission. If Screen Recording has not been granted, screens and windows come back empty with enumerationError explaining why and permissions.screenRecording showing the real status — which is more actionable than a rejection. microphones and iosDevices are enumerated independently, so one failing does not blank the others.

Errors: none.

// request
{ "name": "list_recording_sources", "arguments": {} }
// result
{
  "screens": [ { "type": "screen", "displayId": 1, "displayName": "Built-in Display" },
               { "type": "screen", "displayId": 2, "displayName": "Display 2" } ],
  "windows": [ { "type": "window", "windowId": 4242, "appName": "Safari", "windowTitle": "Safari — Cadre" } ],
  "iosDevices": [],
  "microphones": [ { "deviceId": "BuiltInMicrophoneDevice", "name": "MacBook Pro Microphone",
                     "kind": "input", "isDefault": true } ],
  "permissions": { "screenRecording": "granted", "microphone": "granted",
                   "accessibility": true, "inputMonitoring": false },
  "defaultSource": { "type": "screen", "displayId": 1, "displayName": "Built-in Display" },
  "enumerationError": null
}

get_recording_status

Where the recording state machine is, right now. Cheap, read-only, and the right thing to call after any TIMEOUT or before deciding what to do next.

Parameters: none.

Returns:

{
  state: 'idle' | 'countdown' | 'recording' | 'paused' | 'finishing' | 'completed';
  isRecording: boolean;      // true for BOTH 'recording' and 'paused' — a paused session still holds a live take
  elapsedMs: number;         // recorded duration, excluding paused spans; 0 when idle
  projectPath: string | null;// the in-progress project directory
  permissions: { screenRecording: string; microphone: string;
                 accessibility: boolean; inputMonitoring: boolean };
  stubCapture?: true;        // test builds only — see below
}

stubCapture appears only when the app is running against no-op native stubs (SCREENCRAFT_ALLOW_STUB_RECORDING=1 with the capture addon absent), which happens in Cadre's own e2e tests. If you ever see it, no pixels are being captured. It is never present in a real build.

Errors: none.

// request
{ "name": "get_recording_status", "arguments": {} }
// result
{ "state": "recording", "isRecording": true, "elapsedMs": 12480,
  "projectPath": "/Users/me/Library/Application Support/Cadre/projects/recording-8f2a….screencraft",
  "permissions": { "screenRecording": "granted", "microphone": "granted",
                   "accessibility": true, "inputMonitoring": false } }

start_recording

Start a screen recording. Returns only once the session has genuinely reached recording — through the countdown, native stream setup, and the first encoded frame.

The source is resolved, not trusted. Whatever you pass is matched against live enumeration, and the system's name for the display/window/device is what goes into the recording metadata. An id that no longer exists fails with NOT_FOUND before anything starts, rather than capturing nothing.

Microphone ids are native CoreAudio UIDs and are validated again immediately before capture. Do not fabricate input:<name> ids or cache them across device changes: a device name cannot prove hardware identity, so Cadre will require a fresh selection instead of guessing. Use the literal "default" only when the recording should follow whichever input macOS considers default at start time.

trackCursor defaults on, trackKeyboard defaults off. Cursor data is what automatic zoom is computed from, so it is worth the Accessibility grant. Keyboard tracking only feeds the keystroke overlay, and failing an otherwise fine recording over an Input Monitoring grant nobody asked for is worse than shipping without the overlay — opt in when you want it. Keyboard capture always filters ordinary typing and records only privacy-safe shortcuts; this cannot be disabled through the Agent API.

Returns:

{
  started: true;
  state: 'recording';
  projectPath: string;      // the .screencraft directory being written
  source: CaptureSource;    // the RESOLVED target, with system-reported names
  microphone: {
    status: 'disabled' | 'starting' | 'active' | 'failed';
    requestedDeviceId: string | null;
    message?: string;       // present when native mic admission failed
  };
  config: { fps; resolution; captureRetina; audioSource; microphoneDeviceId;
            trackCursor; trackKeyboard; webcamEnabled: false };
  stubCapture?: true;       // test builds only — no pixels were captured
}

started: true describes the video session. Read microphone.status separately: Cadre deliberately keeps video recording if microphone admission fails, and reports that case as failed with the native message instead of implying narration is active. The tool waits for auxiliary admission to settle for up to five seconds after video reaches recording. A rare starting result means that bound expired: it is deliberately not an active claim; the visible Cadre warning and the finalized recording remain authoritative if admission later fails.

Errors:

• PERMISSION_REQUIRED — Screen Recording is not granted; a requested microphone lacks Microphone access; trackCursor was requested without Accessibility; or trackKeyboard without Input Monitoring. The hint names the System Settings pane and the parameter you can turn off instead.
• INVALID_STATE — a recording is already running (get_recording_status, then stop_recording or cancel_recording).
• NOT_FOUND — the display/window/iOS/microphone id does not exist right now. The hint tells the caller to enumerate again; a microphone is never retargeted by a matching device name.
• INVALID_ARGS — a required id for the chosen source.type is missing, or the area rect is degenerate/absurd.
• TIMEOUT — the session never reached recording. It may be stalled mid-startup: check get_recording_status, then cancel_recording.
• INTERNAL — anything else, including insufficient disk space (Cadre needs ≥ 5 GB of headroom). The original message is preserved.

// request
{ "name": "start_recording", "arguments": {
    "source": { "type": "screen", "displayId": 1 },
    "fps": 30, "audioSource": "none", "countdownSeconds": 3 } }
// result
{ "started": true, "state": "recording",
  "projectPath": "/Users/me/Library/Application Support/Cadre/projects/recording-8f2a….screencraft",
  "source": { "type": "screen", "displayId": 1, "displayName": "Built-in Display" },
  "microphone": { "status": "disabled", "requestedDeviceId": null },
  "config": { "fps": 30, "resolution": "1080p", "captureRetina": true, "audioSource": "none",
              "microphoneDeviceId": null, "trackCursor": true, "trackKeyboard": false,
              "webcamEnabled": false } }

stop_recording

Stop the running (or paused) recording and hand back an editable project.

Under the hood: every encoder segment is finalised, its ordered path is written to the metadata and project.json manifest, and the app opens the result in the editor. There is no full-recording consolidation pass on this foreground path, so handoff does not wait for a second pass over the whole take. The call returns after the editor has loaded the project, so the id and path you get back are immediately usable.

Parameters: none.

Returns:

{
  id: string;            // project id — pass to nothing else; the editor is already on it
  name: string;
  path: string;          // the .screencraft directory
  durationMs: number;    // recorded duration, paused spans excluded
  fps: number;
  resolution: { width: number; height: number; physicalWidth: number; physicalHeight: number };
  hasSystemAudio: boolean;
  hasMicrophone: boolean;
  editorOpen: boolean;   // false = saved on disk, but the editor did not open it (call open_project)
}

This is the round trip the recording tools exist for. Once it returns you can go straight to get_timeline and edit — no open_project needed while editorOpen is true.

Errors:

• INVALID_STATE — nothing is recording, or the session is still counting down (wait, or cancel_recording).
• INTERNAL — finalization failed. The captured footage is left on disk — the message says where, and list_projects may still show it. Never assume a failed stop means a lost take.

// request
{ "name": "stop_recording", "arguments": {} }
// result
{ "id": "c4d8…", "name": "recording-8f2a…",
  "path": "/Users/me/Library/Application Support/Cadre/projects/recording-8f2a….screencraft",
  "durationMs": 30140, "fps": 30,
  "resolution": { "width": 1512, "height": 982, "physicalWidth": 3024, "physicalHeight": 1964 },
  "hasSystemAudio": false, "hasMicrophone": false, "editorOpen": true }

pause_recording

Pause a running recording. Capture stops; resume_recording continues into a new segment, and stop_recording preserves every finalized segment in timeline order for immediate editing. elapsedMs excludes paused spans, so a paused take does not inflate the project duration.

Parameters: none. Returns: the get_recording_status shape, with state: "paused". Errors: INVALID_STATE (not currently recording), TIMEOUT (the pause was issued but the session never reached paused — it may still be running).

resume_recording

Resume a paused recording into a new capture segment.

Parameters: none. Returns: the get_recording_status shape, with state: "recording". Errors: INVALID_STATE (not currently paused), TIMEOUT.

cancel_recording

DESTRUCTIVE. Abandon the in-progress recording and delete its project directory — every frame and audio sample captured so far. There is no undo and nothing goes to the Trash.

Use stop_recording to keep a take you no longer want to extend. Call this only when the user has asked to throw the recording away. It is annotated destructiveHint: true, so a well-behaved MCP client will ask the user before running it.

Parameters: none.

Returns: { cancelled: true; discardedProjectPath: string | null; state: 'idle' }.

Errors: INVALID_STATE (nothing is running), TIMEOUT (the session did not return to idle; it may still be recording), INTERNAL (a stop is already finalising this recording — let it finish).

// request
{ "name": "cancel_recording", "arguments": {} }
// result
{ "cancelled": true,
  "discardedProjectPath": "/Users/me/Library/Application Support/Cadre/projects/recording-8f2a….screencraft",
  "state": "idle" }

Worked recipe: record a 30-second demo, then cut it

// 1. Orient. Are the grants in place? (A license is needed only at export.)
{ "name": "get_app_state", "arguments": {} }
{ "name": "list_recording_sources", "arguments": {} }
//    → screens[0] is the main display; permissions.screenRecording === "granted"

// 2. Start on the main display. The user sees the countdown and the controls bar.
{ "name": "start_recording", "arguments": {
    "source": { "type": "screen", "displayId": 1 },
    "audioSource": "all", "trackCursor": true } }
//    → { started: true, state: "recording", projectPath: "…" }

// 3. YOU own the clock. Sleep ~30 s in your own runtime — there is no
//    duration parameter and Cadre will not stop itself.
//    (Optionally poll get_recording_status to watch elapsedMs.)

// 4. Stop. This returns the finished, editable project.
{ "name": "stop_recording", "arguments": {} }
//    → { id: "c4d8…", path: "…", durationMs: 30140, editorOpen: true }

// 5. Edit it immediately — the editor is already open on it.
{ "name": "get_timeline", "arguments": {} }
{ "name": "analyze_audio", "arguments": { "track": "systemAudio" } }   // find the dead air
{ "name": "add_cut", "arguments": { "startTime": 0, "endTime": 2400 } } // trim the lead-in
{ "name": "recalculate_zooms", "arguments": {} }                       // cinematic zoom from the click log
{ "name": "save_project", "arguments": {} }
{ "name": "export_video", "arguments": { "outputPath": "/Users/me/Desktop/demo.mp4" } }

If the user wanted the take discarded instead of edited, cancel_recording replaces step 4 — and deletes the footage.

Timeline — inspect

get_timeline

Snapshot of the current editor state — the primary way to perceive the project before editing. Bridge-backed: requires a project open.

Parameters: none.

Returns:

{
  project: { id: string; name: string; path: string | null };
  playhead: number;                       // ms, recording time
  // Media geometry — the frame of reference for every `sourceRect`.
  // `null` when the project carries no usable recording metadata.
  recording: {
    width: number;                        // logical px — the sourceRect space
    height: number;                       // logical px
    physicalWidth: number;                // = width * displayScaleFactor
    physicalHeight: number;
    fps: number;
    displayScaleFactor: number;           // 1 standard, 2 Retina
    sourceType: 'screen' | 'window' | 'ios' | 'area' | 'import' | 'unknown';
  } | null;
  durations: {
    recordingMs: number;                  // source capture length (recording time)
    outputMs: number;                     // rendered length after cuts + speed (output time)
  };
  timeline: {
    zoomKeyframes: ZoomKeyframe[];
    cuts: Cut[];
    speedSegments: SpeedSegment[];
    splitPoints: number[];                // ms marks
    maskSegments: MaskSegment[];
    overlaySegments: OverlaySegment[];
    layoutSegments: LayoutSegment[];      // camera layout over time; gaps = the global webcam style
    captions: { language: string; modelUsed: string;
                segmentCount: number; segments: CaptionSegment[] } | null;
    musicTrack: MusicTrack | null;
  };
  audio: { systemAudioGain: number; micAudioGain: number };
  style: { background; frame; cursor; keyboard; motion; webcam };  // full config objects
}

Note the caption shape here is the summarised form (segmentCount + segments). list_captions returns the full CaptionTrack (which also carries generatedAt and style).

timeline.layoutSegments is where update_camera_layout and delete_camera_layout get their ids, and reading it is how you find the free gaps add_camera_layout will accept. An empty list means the camera follows style.webcam for the whole video, which is the normal state.

Errors: NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "get_timeline", "arguments": {} }
// result (abridged)
{
  "project": { "id": "a1b2...", "name": "Onboarding demo", "path": "/Users/me/Movies/Onboarding demo.screencraft" },
  "playhead": 0,
  "recording": { "width": 1440, "height": 900, "physicalWidth": 2880, "physicalHeight": 1800,
                 "fps": 60, "displayScaleFactor": 2, "sourceType": "screen" },
  "durations": { "recordingMs": 48200, "outputMs": 48200 },
  "timeline": { "zoomKeyframes": [], "cuts": [], "speedSegments": [], "splitPoints": [],
                "maskSegments": [], "overlaySegments": [], "layoutSegments": [],
                "captions": null, "musicTrack": null },
  "audio": { "systemAudioGain": 1, "micAudioGain": 1 },
  "style": { "background": { "type": "gradient", "...": "..." }, "...": "..." }
}

recording is null for a project whose manifest has no usable resolution (an import, or a partially written project). Treat that as "spatial reasoning is unavailable" — pass explicit pixel rects, or ask the user — rather than assuming a default resolution. add_zoom will refuse to auto-centre in that state instead of inventing a frame.

map_time

Convert timestamps between the recording and output timebases. Bridge-backed: requires a project open. See Timebases for why this matters.

Parameters:

Returns:

{
  from: 'recording' | 'output';
  to: 'recording' | 'output';
  durations: { recordingMs: number; outputMs: number };
  results: Array<{
    inputMs: number;      // the value as supplied
    recordingMs: number;
    outputMs: number;
    isCut: boolean;       // see below
    clamped: boolean;     // input was outside the timeline and was clamped
  }>;
}

Fidelity. output → recording → output is exact: every output instant corresponds to exactly one surviving frame. The reverse is not total. A recording time that falls inside a cut has no output instant at all — it collapses onto the cut's boundary — and is flagged isCut: true. An edit anchored to such a position lands at the boundary, so treat isCut as a signal to reconsider rather than a detail to ignore.

Positions outside the timeline are clamped rather than rejected, and reported with clamped: true, so probing for the end of the timeline returns an answer.

Errors: INVALID_ARGS (empty/oversized batch, non-finite entry, unknown from, or a project with no recording duration), NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// The user says "cut 30 to 45 seconds", reading the preview clock.
// A 10 s cut already exists at the start of the recording.
// request
{ "name": "map_time", "arguments": { "timesMs": [30000, 45000], "from": "output" } }
// result
{
  "from": "output", "to": "recording",
  "durations": { "recordingMs": 60000, "outputMs": 50000 },
  "results": [
    { "inputMs": 30000, "recordingMs": 40000, "outputMs": 30000, "isCut": false, "clamped": false },
    { "inputMs": 45000, "recordingMs": 55000, "outputMs": 45000, "isCut": false, "clamped": false }
  ]
}
// -> call add_cut with 40000 / 55000, NOT 30000 / 45000.

Timeline — zoom

add_zoom

Add a manual zoom keyframe over a time range. Returns the created keyframe.

Times are in recording time — see Timebases. Prefer a normalized focusPoint: Cadre derives a bounds-clamped crop with the source's aspect ratio at exactly zoomLevel. sourceRect remains available when the caller intentionally needs a literal crop.

Returns: ZoomKeyframe (trigger: 'manual', isUserCreated: true) plus compositionReview: { required: true, timeMs, instruction }. The review time is the midpoint of the zoom. Call get_edited_frame at that exact time before save/export and reject any crop whose boundary crosses recognised text, a card, button, or another UI control. When the target cannot be isolated without fragments, leave the shot wide.

Errors: INVALID_ARGS (e.g. endTime <= startTime), NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "add_zoom", "arguments": {
    "startTime": 300, "endTime": 3700, "zoomLevel": 1.6,
    "focusPoint": { "x": 0.72, "y": 0.58 } } }
// result
{ "id": "kf-9f2c...", "startTime": 300, "endTime": 3700,
  "sourceRect": { "x": 587, "y": 281, "width": 900, "height": 562.5 },
  "zoomLevel": 1.6, "trigger": "manual", "isUserModified": false, "isUserCreated": true,
  "compositionReview": { "required": true, "timeMs": 2000,
    "instruction": "Before save or export, call get_edited_frame ..." } }

Behaviour change. Omitting sourceRect now produces a frame-centred window of 1 / zoomLevel of the source in each axis, matching what the app's own "add zoom" button computes. It previously substituted { x: 0, y: 0, width: 1, height: 1 } — read downstream as absolute pixels, i.e. a 1×1 pixel crop at the top-left corner, not the "auto-centre" this page described. Centring needs the frame size, so on a project whose recording is null the call now fails with INVALID_ARGS rather than silently producing that crop; pass an explicit pixel rect in that case.

update_zoom

Update fields of an existing zoom keyframe by id.

If both updates.startTime and updates.endTime are given, updates.endTime must exceed updates.startTime. A one-sided update (e.g. only endTime) is not cross-checked against the keyframe's stored counterpart at this boundary — an update that inverts the range against the existing value is caught downstream, not here.

An updates object with no recognised field (e.g. a misspelled key — zod silently strips unknown keys before this check runs) is rejected rather than accepted as a no-op.

Returns: the updated ZoomKeyframe. Errors: NOT_FOUND (unknown id), INVALID_ARGS, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "update_zoom", "arguments": { "id": "kf-9f2c...", "updates": { "zoomLevel": 2.5 } } }
// result
{ "id": "kf-9f2c...", "startTime": 300, "endTime": 3700, "sourceRect": { "...": "..." },
  "zoomLevel": 2.5, "trigger": "manual", "isUserModified": true, "isUserCreated": true }

delete_zoom

Delete a zoom keyframe by id.

Returns: { id: string; deleted: true }. Errors: NOT_FOUND, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "delete_zoom", "arguments": { "id": "kf-9f2c..." } }
// result
{ "id": "kf-9f2c...", "deleted": true }

recalculate_zooms

Re-run Cadre's automatic cinematic-zoom analysis over the recording's interaction log (clicks and scrolls) and replace the auto-generated zoom keyframes with the result. Manual keyframes an agent or the user created (isUserCreated: true) are preserved by the editor's merge — this only touches the automatic ones.

Cadre never records plain keystrokes, so typing is not a signal the analysis can see or a prompt can target — clicks and scrolls are the whole input.

Parameters: none.

Returns: { keyframes: ZoomKeyframe[] } on success, or { keyframes: []; reason: string } when the project has no interaction log to analyse.

Imported videos. Only projects recorded in Cadre carry an interactions.jsonl. A project opened via import_video has none, so this call returns { keyframes: [], reason: "..." } explaining that honestly instead of guessing or failing — add zoom keyframes manually with add_zoom for those. Verify the result with get_timeline afterwards either way.

Errors: NO_PROJECT_OPEN.

// request — a Cadre recording with interaction data
{ "name": "recalculate_zooms", "arguments": {} }
// result
{ "keyframes": [ { "id": "kf-...", "startTime": 1200, "endTime": 3400, "...": "...", "trigger": "click", "isUserCreated": false } ] }

// request — a project opened via import_video
{ "name": "recalculate_zooms", "arguments": {} }
// result
{ "keyframes": [], "reason": "This project has no interaction log (recording/interactions.jsonl) — imported videos never do — so auto-zoom has no click/scroll data to work from. Add zooms manually with add_zoom instead." }

Timeline — cuts

add_cut

Ripple-delete a time range from the output. Returns the resulting cut.

Times are in recording time — see Timebases. This is the tool most often given the wrong clock: a user quoting "cut 30 to 45 seconds" is reading the preview, so run those numbers through map_time (from: "output") first.

Merge semantics. A range that overlaps or abuts an existing cut is merged into it, so the returned cut may be wider than the range you asked for and may carry the older cut's id. This mirrors what the renderer already does — it coalesces overlapping cuts when building the output — so the cut list can never disagree with what is actually removed.

Adding a cut shortens durations.outputMs and shifts every later output timestamp earlier. Re-run map_time after this call rather than reusing conversions made before it.

Returns: Cut. Errors: INVALID_ARGS, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "add_cut", "arguments": { "startTime": 1000, "endTime": 2500 } }
// result
{ "id": "c-71a0...", "startTime": 1000, "endTime": 2500, "type": "remove" }

update_cut

Change the bounds of an existing cut by id.

If both bounds are given, endTime must exceed startTime. An updates object with no recognised field is rejected rather than accepted as a no-op.

Returns: the updated Cut. Errors: NOT_FOUND, INVALID_ARGS, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "update_cut", "arguments": { "id": "c-71a0...", "updates": { "endTime": 2500 } } }
// result
{ "id": "c-71a0...", "startTime": 1000, "endTime": 2500, "type": "remove" }

delete_cut

Delete a cut by id (restores that range to the output).

Returns: { id: string; deleted: true }. Errors: NOT_FOUND, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "delete_cut", "arguments": { "id": "c-71a0..." } }
// result
{ "id": "c-71a0...", "deleted": true }

Timeline — speed

set_speed

Set a playback-speed multiplier over a time range (e.g. 2.0 to speed up, 0.5 to slow down). Returns the created speed segment.

Times are in recording time — see Timebases.

Replacement semantics. set_speed replaces any speed already covering the range rather than layering on top of it:

So the call is idempotent, and get_timeline's speedSegments always describes the real timeline. Changing speed changes durations.outputMs, so re-run map_time afterwards rather than reusing earlier conversions.

Returns: SpeedSegment (with default rampIn/rampOut). Errors: INVALID_ARGS, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "set_speed", "arguments": { "startTime": 8000, "endTime": 14000, "speed": 2 } }
// result
{ "id": "sp-3d9e...", "startTime": 8000, "endTime": 14000, "speed": 2, "rampIn": 300, "rampOut": 300 }

delete_speed

Delete a speed segment by id, restoring that range to 1×. Get ids from get_timeline (timeline.speedSegments).

Returns: { id: string; deleted: true }. Errors: NOT_FOUND, INVALID_ARGS, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "delete_speed", "arguments": { "id": "sp-3d9e..." } }
// result
{ "id": "sp-3d9e...", "deleted": true }

Corrected guidance. This page previously said there was no delete_speed tool and told agents to "re-set_speed the range to 1.0" to remove a speed-up. Under the old push-only set_speed that advice was wrong: it stacked a second overlapping segment on the first and left the outcome to a last-wins scan of an unsorted array. delete_speed is now the correct way to undo a set_speed. Re-setting the range to 1.0 also works today — it replaces the old segment — but leaves an explicit 1× segment behind.

Masks

Time-bound regions over the source frame, in normalised 0–1 coordinates — always normalised, unlike sourceRect on the zoom tools, which can be pixels or normalised depending on a units field. Two effects:

• blur — hides sensitive content (credentials, tokens, notifications, faces) inside the region for the duration of the mask.
• highlight — the inverse: dims everything except the region, to spotlight it.

add_mask

Add a time-bound mask over a region of the video. Returns the created segment.

Times are in recording time — see Timebases.

Returns: MaskSegment (enabled: true). Errors: INVALID_ARGS (e.g. endTime <= startTime, rect out of bounds or not fitting the frame), NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request — blur a notification banner in the top-right for 4s
{ "name": "add_mask", "arguments": {
    "startTime": 12000, "endTime": 16000, "type": "blur",
    "rect": { "x": 0.7, "y": 0.05, "width": 0.28, "height": 0.12 } } }
// result
{ "id": "mask-4a1b...", "startTime": 12000, "endTime": 16000, "type": "blur",
  "rect": { "x": 0.7, "y": 0.05, "width": 0.28, "height": 0.12 },
  "enabled": true, "blurRadius": 20, "opacity": 1 }

update_mask

Update a mask segment by id (timing, rect, type, blurRadius, opacity, enabled). Get ids from get_timeline (timeline.maskSegments).

If both bounds are given, endTime must exceed startTime. An updates object with no recognised field is rejected rather than accepted as a no-op.

Returns: the updated MaskSegment. Errors: NOT_FOUND, INVALID_ARGS, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "update_mask", "arguments": { "id": "mask-4a1b...", "updates": { "enabled": false } } }
// result
{ "id": "mask-4a1b...", "startTime": 12000, "endTime": 16000, "type": "blur",
  "rect": { "x": 0.7, "y": 0.05, "width": 0.28, "height": 0.12 },
  "enabled": false, "blurRadius": 20, "opacity": 1 }

delete_mask

Delete a mask segment by id.

Returns: { id: string; deleted: true }. Errors: NOT_FOUND, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "delete_mask", "arguments": { "id": "mask-4a1b..." } }
// result
{ "id": "mask-4a1b...", "deleted": true }

Overlays

Text, SVG, native data-driven graphics, and rendered frame animations are drawn on the OUTPUT frame like captions — they do not pan or zoom with the content underneath, and position/width are normalised to the output frame, not the source. Four kinds are time-bound and return an OverlaySegment:

• add_text_overlay — a title, label, or annotation.
• add_svg_overlay — an arrow, badge, logo, or shape, from raw SVG markup.
• add_graphic_overlay — a native animated counter, bar chart, or progress ring.
• add_animation_overlay — a project-managed transparent PNG sequence imported from frames or video.

Both share position, width, rotationDeg, opacity, an animation spec (entrance, exit, continuous loop), and an optional keyframes track for motion the presets do not cover. Text overlays additionally take a textStyle. Native graphics and rendered animations have local recording-time motion; the shared pose animation and keyframes compose around that content.

Instead of a position you may pass an anchor — "beside the camera", "top-right of frame" — which is resolved to coordinates during the call. The two are mutually exclusive.

add_text_overlay

Add an animated text callout over the video.

Times are in recording time — see Timebases.

Returns: the created OverlaySegment (type: "text"). Verify it, then get_timeline shows it under timeline.overlaySegments.

Errors: INVALID_ARGS (e.g. endTime <= startTime, empty/oversized text, an out-of-range style or animation field), NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request — a title card that types itself in, top-center
{ "name": "add_text_overlay", "arguments": {
    "startTime": 0, "endTime": 2500, "text": "Deploying to production",
    "position": { "x": 0.5, "y": 0.12 }, "width": 0.7,
    "animation": { "enter": "typewriter", "enterDurationMs": 1800, "exit": "fade" } } }
// result
{ "id": "ov-7d2e...", "type": "text", "startTime": 0, "endTime": 2500,
  "position": { "x": 0.5, "y": 0.12 }, "width": 0.7, "rotationDeg": 0, "opacity": 1,
  "text": "Deploying to production",
  "textStyle": { "fontFamily": "-apple-system, BlinkMacSystemFont, SF Pro Display, Helvetica Neue, sans-serif",
                 "fontSizeFrac": 0.048, "fontWeight": 600,
                 "color": "#F5F5F7", "backgroundColor": "#1D1D1F", "backgroundOpacity": 0.78,
                 "paddingFrac": 0.42, "cornerRadiusFrac": 0.5, "align": "center" },
  "animation": { "enter": "typewriter", "enterDurationMs": 1800, "exit": "fade",
                 "exitDurationMs": 220, "loop": "none", "loopPeriodMs": 2000 } }

add_svg_overlay

Add an animated SVG sticker/graphic over the video.

Times are in recording time — see Timebases.

Sanitisation. The markup must parse as a standalone <svg> document and survive sanitisation with at least one drawable element remaining — an empty shell, a non-SVG string, or markup that is only scripts/forbidden elements is rejected before it reaches the timeline. This runs twice (main process, then again in the renderer as defence in depth), so a call that returns INVALID_ARGS never partially applies.

Returns: the created OverlaySegment (type: "svg", svg holding the sanitised markup, which may differ from what you sent).

Errors: INVALID_ARGS — endTime <= startTime; or the SVG is empty, over 200,000 characters, not a standalone <svg>…</svg> document, or contains nothing but stripped content after sanitisation.

// request — an arrow sticker pointing at a button, top-right
{ "name": "add_svg_overlay", "arguments": {
    "startTime": 4000, "endTime": 7000,
    "svg": "<svg viewBox=\"0 0 100 100\"><path d=\"M10 50 L80 50 L60 20 M80 50 L60 80\" stroke=\"#FF3B30\" stroke-width=\"8\" fill=\"none\"/></svg>",
    "position": { "x": 0.82, "y": 0.22 }, "width": 0.12 } }
// result
{ "id": "ov-9f0a...", "type": "svg", "startTime": 4000, "endTime": 7000,
  "position": { "x": 0.82, "y": 0.22 }, "width": 0.12, "rotationDeg": 0, "opacity": 1,
  "svg": "<svg viewBox=\"0 0 100 100\">...</svg>",
  "animation": { "enter": "fade", "enterDurationMs": 280, "exit": "fade",
                 "exitDurationMs": 220, "loop": "none", "loopPeriodMs": 2000 } }

add_graphic_overlay

Add a native data-driven counter, horizontal bar chart, or progress ring. These templates cover common editorial graphics; use add_animation_overlay for a custom rendered design. They share the deterministic Canvas2D overlay path used by preview and export.

The graphic's delayMs and durationMs use recording time local to the overlay: local 0 is startTime. Moving the block carries its motion. Shortening the block clips the motion without rescaling it, and the completed state holds until endTime. Cuts skip source animation time; speed edits change its rate in the output.

Common graphic defaults are title: "", subtitle: "", theme: "dark", accentColor: "#0A84FF", delayMs: 0, durationMs: 1200, and easing: "ease-out". Counter from defaults to 0; decimals and affixes default to zero/empty. Chart suffix defaults empty and an omitted maximum uses its largest bar. Ring decimals default to zero.

These requests use illustrative data; replace it with facts from the user's brief or recording:

{ "name": "add_graphic_overlay", "arguments": {
  "startTime": 4200, "endTime": 7200,
  "graphic": { "kind": "counter", "title": "Teams onboarded", "to": 128,
    "suffix": "+", "durationMs": 1400 },
  "position": { "x": 0.78, "y": 0.28 }, "width": 0.30 } }

{ "name": "add_graphic_overlay", "arguments": {
  "startTime": 10400, "endTime": 14800,
  "graphic": { "kind": "bar-chart", "title": "Illustrative adoption",
    "bars": [ { "label": "Core", "value": 72 }, { "label": "New", "value": 48 } ],
    "maxValue": 100, "suffix": "%", "theme": "light" },
  "anchor": { "target": "screen-region", "region": "top-right" } } }

{ "name": "add_graphic_overlay", "arguments": {
  "startTime": 18100, "endTime": 21600,
  "graphic": { "kind": "progress-ring", "title": "Migration", "value": 84.5,
    "decimals": 1, "accentColor": "#34C759" },
  "position": { "x": 0.76, "y": 0.32 }, "width": 0.24 } }

Returns: the canonical created OverlaySegment, with type: "graphic" and every default resolved. Errors: INVALID_ARGS, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

add_animation_overlay

Import a rendered animation as a transparent overlay. This is the general path for custom Canvas/SVG motion, Three.js, Blender, After Effects, or another renderer. Cadre does not execute or persist the source program. It converts the result to lossless RGBA PNG frames inside the project, which makes random-access scrubbing and export deterministic and keeps working if the original moves.

PNG filenames use natural numeric order (frame-2.png before frame-10.png) and are copied without resampling. Video audio is ignored. ProRes 4444 and VP8/VP9 WebM alpha decode into RGBA; opaque inputs also work. One import is bounded to 3,600 frames, 4,096 pixels per axis, and 1 GiB of encoded PNGs. It fails as a whole instead of truncating. Cancellation removes the temporary asset directory before any layer appears.

This returns a background-job receipt. Keep jobId and jobToken, poll get_agent_job_status, then read get_agent_job_result; the terminal result is the new type: "animation" overlay. The same project must stay open until commit.

{ "name": "add_animation_overlay", "arguments": {
  "path": "/Users/me/Renders/cube-assembly.webm", "fps": 30,
  "startTime": 24000, "endTime": 28000, "loop": false,
  "position": { "x": 0.75, "y": 0.38 }, "width": 0.34,
  "animation": { "enter": "fade", "exit": "fade" } } }
// immediate receipt
{ "jobId": "job-…", "jobToken": "secret-…", "type": "add_animation_overlay",
  "status": "running", "pollAfterMs": 500 }
// terminal result, abridged
{ "id": "ov-…", "type": "animation", "startTime": 24000, "endTime": 28000,
  "animationSource": { "assetId": "8f4c…", "frameCount": 120, "fps": 30,
    "width": 1080, "height": 1080, "loop": false } }

Inspect the timeline and convert user-quoted output times before mutation. Once the job completes, verify early, middle, and late recording-time frames with get_edited_frame — 24100, 26000, and 27900 for the range above. Check entry, motion, final state, alpha edges, placement, and caption collisions; export is the final parity check.

Older builds may not expose this tool. Discover it with MCP tools/list; if it is absent, use an available native graphic or text/SVG tool and do not send an unknown call.

update_overlay

Update an overlay segment by id (timing, content, position, size, rotation, opacity, textStyle, graphic, animationSource, or pose animation). Get ids from get_timeline (timeline.overlaySegments).

Nested objects merge field-by-field — { "textStyle": { "color": "#FF0000" } } changes only the colour and leaves every other textStyle/animation field as it was. This differs from set_style, whose nested objects must be sent whole.

keyframes, graphic, and animationSource are replacements. There is no per-item patch language — send the track you want, or [] to clear it. A track is one value that only reads end-to-end, so merging halves of two animations would produce motion neither caller asked for.

If both bounds are given, endTime must exceed startTime. An updates object with no recognised field is rejected rather than accepted as a no-op. Content fields cannot cross overlay types: text, svg, graphic, and animationSource are each rejected on the other three overlay types. Replacing a native graphic may switch among counter, bar chart, and progress ring because those are variants inside one graphic overlay.

Returns: { overlay, previous } — overlay is the updated OverlaySegment, previous is the full segment as it was before this call. Send previous straight back as updates to restore it exactly. That is the intended undo for agent edits: it is precise about which overlay it reverts, and it leaves the global undo stack — which you share with whatever the user is doing in the editor — alone.

Errors: NOT_FOUND, INVALID_ARGS (out-of-range field, cross-type content change, rejected media metadata, or a rejected SVG), NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request — only the pill colour changes; every other textStyle field is untouched
{ "name": "update_overlay", "arguments": {
    "id": "ov-7d2e...", "updates": { "textStyle": { "backgroundColor": "#1D4ED8" } } } }
// result (abridged) — `previous` carries the pre-call state, ready to send back
{ "overlay": { "id": "ov-7d2e...", "type": "text", "...": "...",
               "textStyle": { "...": "...", "backgroundColor": "#1D4ED8", "backgroundOpacity": 0.78 } },
  "previous": { "id": "ov-7d2e...", "type": "text", "...": "...",
                "textStyle": { "...": "...", "backgroundColor": "#1D1D1F", "backgroundOpacity": 0.78 } } }

delete_overlay

Delete an overlay segment by id.

Returns: { overlay: { id: string; deleted: true }; previous: OverlaySegment } — previous is the segment that was deleted, in full. Re-create it with add_text_overlay, add_svg_overlay, add_graphic_overlay, or add_animation_overlay if the deletion was a mistake; it comes back with a new id, and unlike undo it reverts nothing the user did in between.

Errors: NOT_FOUND, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

Animation fields

Shared by all four add-overlay tools and update_overlay's updates.animation. See OverlayAnimation for the return shape.

Keyframes

Shared by all four add-overlay tools and update_overlay's updates.keyframes. Use these when the entrance/exit/loop presets above cannot express the motion — a sticker that flies in and settles, a label that drifts, a badge that snaps between two spots.

Times are offsets, so moving the block moves the animation with it. Drag the overlay later on the timeline (or change startTime) and the motion rides along unchanged.

A keyframe claims only the properties it sets. position, scale, rotationDeg and opacity are independent tracks derived from the one list, so a track of two position keyframes animates position and leaves opacity on the segment's static value. Before the first and after the last keyframe of a property, that keyframe's value holds.

The later keyframe's easing governs its span. hold steps instead of interpolating: the earlier value stays put until the later keyframe's time. ease-out-back overshoots slightly before settling.

Entrance/exit/loop still compose on top — opacity multiplies, offsets add, scale multiplies. Keyframes replace only the base pose, so a keyframed overlay with enter: "fade" still fades in.

The track is sorted by atMs server-side, so you may send it in any order. At most 60 keyframes; more is rejected with INVALID_ARGS.

// request — a badge that drops in from above and settles with a slight overshoot
{ "name": "add_svg_overlay", "arguments": {
    "startTime": 3000, "endTime": 6000,
    "svg": "<svg viewBox=\"0 0 100 100\"><circle cx=\"50\" cy=\"50\" r=\"40\" fill=\"#0A84FF\"/></svg>",
    "position": { "x": 0.5, "y": 0.42 }, "width": 0.1,
    "keyframes": [
      { "atMs": 0,   "position": { "x": 0.5, "y": 0.15 } },
      { "atMs": 600, "position": { "x": 0.5, "y": 0.42 }, "easing": "ease-out-back" } ] } }

Anchors

Shared by all four add-overlay tools. Pass anchor instead of position — sending both is INVALID_ARGS, because two answers to "where does this go" is a caller confusion, not a precedence rule worth memorising.

Resolved once, at the call. The anchor becomes a plain position on the stored segment; the project file contains coordinates and nothing else. An overlay anchored to the camera therefore does not follow the camera if the user moves it afterwards — re-place it if you want the new geometry.

The exact mapping. Coordinates are normalised to the output frame (x right, y down) and an overlay's position is its centre. Writing m for marginFrac, hx for half the overlay's width (width / 2) and hy for half its height as a fraction of frame height:

• hy for an SVG is exact: width × frameAspect ÷ viewBoxAspect ÷ 2.
• hy for text is estimated from the pill geometry: fontSizeFrac × (1.25 × lines + 2 × paddingFrac) ÷ 2, where lines counts explicit \n breaks only — automatic word wrapping is not simulated, so a long wrapping string anchored to bottom sits slightly higher than it should. Pass an explicit position when you need it to the pixel.

target: "screen-region" maps the nine regions onto the frame:

So { "region": "top-left" } puts the overlay's top-left corner one margin in from the frame's top-left corner, not its centre.

target: "webcam" places the overlay one margin clear of the named side of the camera rectangle, centred on the camera's other axis:

The camera rectangle is the one the current style configures — position grid, size, offsetX/offsetY — read at call time from get_app_state's style.webcam. Under Camera Only the camera is the content, so the rectangle is the content rect it fills. INVALID_ARGS when the webcam is disabled or the recording has no camera track: there is no rectangle to anchor to, and guessing one would put the overlay somewhere the user never sees a camera.

Nothing lands off-frame. After the mapping the centre is clamped so the whole overlay fits with at least m / 2 of margin around it. An overlay too large to fit on an axis is centred on that axis rather than silently resized.

// request — a label beside the camera, wherever the camera currently is
{ "name": "add_text_overlay", "arguments": {
    "startTime": 8000, "endTime": 12000, "text": "Ask me anything",
    "anchor": { "target": "webcam", "placement": "left-of" }, "width": 0.3 } }
// result (abridged) — the anchor is gone; only the coordinates it produced remain
{ "id": "ov-3c81...", "type": "text", "position": { "x": 0.606, "y": 0.836 },
  "width": 0.3, "...": "..." }

Camera layout

What the camera is doing is a property of time, not just of the project. A LayoutSegment overrides the global style.webcam layout over a span of recording time; the stretches between segments are the global layout, unchanged. Four modes:

• fullscreen — the Camera Only layout: the camera becomes the frame's content, and the screen recording, cursor, click ripples, keyboard pills and zoom framing are all suppressed for that span. Not an overlay covering the screen.
• default — the ordinary corner overlay (the timeline labels it Corner), optionally overriding the global position and size.
• hide — no camera at all: screen only.
• split: screen and speaker share the frame, stacked for portrait output.

Three rules worth knowing before the first call:

1. A gap is how you go back. To open fullscreen and then run the demo in the corner you add one segment. The stretch after it is already the global layout. A second segment restating the global layout is redundant — and because two adjacent spans that resolve to the same thing are coalesced, it would not even produce a second move.
2. Segments may not overlap, and a disabled one still owns its span (or enabling it later would break the invariant silently). A conflicting range is rejected with the blocking segment's id and span, never adjusted for you.
3. The camera flies between layouts. Every boundary animates with a critically-damped ease, and the transition belongs to the later layout's opening moments. A segment starting at 0 opens already settled — the video opens fullscreen, it does not arrive there.

A project with no camera track cannot show a speaker. It can use a hide segment with layout settings to change its supporting section over time. Read get_timeline before making changes.

Composition settings

set_style with section frame accepts a complete layout object:

{ "mode": "split", "speakerSide": "left", "speakerSize": 0.28,
  "gap": 24, "supportHeight": 0.18 }

This is the whole-video arrangement. mode accepts default, fullscreen, hide, and split. speakerSide is left or right (top or bottom in portrait). speakerSize is 0.18–0.45 of the available divider space, gap is 0–80 pixels at the 1080px reference height, and supportHeight is 0–0.35 of the available height. Zero hides the supporting section. Screens are fitted without stretching; split camera panels crop to fill. Device frames are paused for split layouts or supporting sections. Frame margins, corners and inset appearance are shared.

add_camera_layout and update_camera_layout.updates also accept layout. The segment's mode takes precedence over layout.mode. Omitted settings follow the whole-video arrangement, and gaps return to that arrangement. Update with layout: null to clear a segment's arrangement override. position and customSize apply only to default, including inside a global split layout.

set_style with section webcam accepts reframe: { x: 0.5, y: 0.5, zoom: 1 }. x and y are 0–1 positions within the available crop; zoom is 1–3. This crop applies throughout the video. Camera position, size and mode still vary by segment.

add_text_overlay and update_overlay.updates accept region: "support". These text blocks use their own recording-time spans, centre inside the active supporting section and wrap to its width. They are clipped to that region and hidden while it is disabled. Canvas position and width are retained for restoration but do not place attached text; detach with region: null in an update. Use successive text blocks for changing section text. Existing text/SVG overlays remain on the canvas, including titles and logos.

Validation: invalid modes, missing layout fields, non-finite values and values outside these ranges are rejected before mutation. Supporting regions accept text only; combining a supporting region with an anchor is rejected. A previous snapshot includes an explicit null when needed to restore an absent arrangement override or text attachment. All spans use recording time; convert user timestamps with map_time after cuts or speed edits. Inspect get_timeline, apply the change, then verify with get_edited_frame in each target aspect ratio before export.

add_camera_layout

Change what the camera does over a span. Returns the created LayoutSegment.

Times are in recording time — see Timebases.

Omitting endTime is usually right: the default span is exactly what the editor's own "Camera: … from here" produces, and an agent-created segment is otherwise indistinguishable from a hand-made one.

Returns: the created LayoutSegment (enabled: true). Errors: INVALID_ARGS (range inverted or under 500 ms; the range overlaps an existing segment — the message names it and its span; position/customSize passed in a mode that has no corner; startTime with no room before the next segment or the end of the recording), NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request — the founder shape: open on the presenter, then demo in the corner
{ "name": "add_camera_layout", "arguments": {
    "startTime": 0, "endTime": 30000, "mode": "fullscreen" } }
// result
{ "id": "lay-7d20...", "startTime": 0, "endTime": 30000, "mode": "fullscreen",
  "position": null, "customSize": null, "enabled": true }

That single call is the whole edit. The first 30 s are the camera full-frame; from 30 s the project's normal corner pip returns on its own, because the gap after the segment is the global layout, and the camera flies between the two.

// request — a different corner and a bigger camera for the recap, to the end
{ "name": "add_camera_layout", "arguments": {
    "startTime": 92000, "mode": "default",
    "position": "bottom-left", "customSize": 0.28 } }
// result — endTime defaulted to the end of the recording
{ "id": "lay-b114...", "startTime": 92000, "endTime": 128400, "mode": "default",
  "position": "bottom-left", "customSize": 0.28, "enabled": true }

update_camera_layout

Update a camera layout segment by id. Get ids from get_timeline (timeline.layoutSegments).

A change of span goes through the same non-overlap check as creation, and the merged range is validated against the stored one — a one-sided endTime that would invert the segment is rejected rather than written. position / customSize are checked against the merged mode, so setting a corner on a segment that is (or becomes) fullscreen is refused rather than silently dropped. An updates object with no recognised field is rejected rather than accepted as a no-op.

Returns: { layout: LayoutSegment; previous: LayoutSegment } — previous is the full segment as it was before the call. Send it straight back as updates to restore it exactly; that is precise, and unlike undo it does not disturb the history you share with the user. Errors: NOT_FOUND, INVALID_ARGS, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request — the intro ran long; hold the presenter to 42s
{ "name": "update_camera_layout", "arguments": {
    "id": "lay-7d20...", "updates": { "endTime": 42000 } } }
// result
{ "layout": { "id": "lay-7d20...", "startTime": 0, "endTime": 42000,
              "mode": "fullscreen", "position": null, "customSize": null, "enabled": true },
  "previous": { "id": "lay-7d20...", "startTime": 0, "endTime": 30000,
                "mode": "fullscreen", "position": null, "customSize": null, "enabled": true } }

delete_camera_layout

Delete a camera layout segment by id. That span reverts to the project's global webcam style.

To silence a segment without losing it, prefer update_camera_layout with { "enabled": false } — the span stays reserved and the segment stays on the user's timeline.

Returns: { layout: { id: string; deleted: true }; previous: LayoutSegment } — previous is the segment that was deleted, in full. Re-create it with add_camera_layout if the deletion was a mistake; it comes back with a new id, and unlike undo it reverts nothing the user did in between. Errors: NOT_FOUND, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "delete_camera_layout", "arguments": { "id": "lay-7d20..." } }
// result
{ "layout": { "id": "lay-7d20...", "deleted": true },
  "previous": { "id": "lay-7d20...", "startTime": 0, "endTime": 42000,
                "mode": "fullscreen", "position": null, "customSize": null, "enabled": true } }

Worked recipe: intro fullscreen, then demo with a corner pip

The founder's own request, end to end. It is one mutating call.

// 1. Look first. Does this project even have a camera to lay out?
{ "name": "get_timeline", "arguments": {} }
//    → style.webcam.enabled === true, timeline.layoutSegments === []
//    → durations.recordingMs === 128400

// 2. The whole edit: the first 30 s are the presenter, full-frame.
{ "name": "add_camera_layout", "arguments": {
    "startTime": 0, "endTime": 30000, "mode": "fullscreen" } }
//    → { id: "lay-7d20…", startTime: 0, endTime: 30000, mode: "fullscreen",
//        position: null, customSize: null, enabled: true }
//    Nothing follows it. From 30 s the gap renders style.webcam — the corner
//    pip the project already had — and the camera flies out to it.

// 3. Confirm what you built, then ship it.
{ "name": "get_timeline", "arguments": {} }
{ "name": "save_project", "arguments": {} }
{ "name": "export_video", "arguments": { "outputPath": "/Users/me/Desktop/demo.mp4" } }

If the user wants a different corner for the demo than their global style uses, add a second segment for the demo span with mode: "default" and an explicit position — but only then. Restating the global layout as a segment buys nothing and coalesces away.

Audio

analyze_audio

Measure the open project's audio: an RMS loudness envelope in fixed buckets and detected silence ranges, per track (system audio, microphone) plus a combined view. Use this — not generate_captions — to find dead air; it takes seconds instead of minutes and it sees non-speech audio (typing, music, UI sounds) that a transcript cannot.

All times in the result are recording time, the same timebase every mutating tool takes, so combined.silenceRanges can be fed straight into add_cut with no conversion (only user-quoted preview timestamps need map_time).

Returns:

{
  hasAudio: boolean;               // false = no audio track at all, not "no silence found"
  reason?: 'NO_AUDIO_TRACK';       // present + explanatory only when hasAudio is false
  message?: string;
  recordingDurationMs: number | null;
  settings: { silenceThresholdDb: number; minSilenceDurationMs: number;
              requestedBucketMs: number; bucketMs: number };  // bucketMs may be wider than requested
  timebase: 'recording';
  tracks: Array<{
    track: 'system' | 'mic';
    filePath: string;
    durationMs: number;
    bucketMs: number;
    envelopeDb?: number[];         // omitted when includeEnvelope is false
    silenceRanges: Array<{ startMs: number; endMs: number; durationMs: number }>;
    silentMs: number;
    activeRatio: number;           // fraction NOT silent, 0-1
    peakDb: number | null;
    warnings: string[];
  }>;
  combined: {                      // null when hasAudio is false — never an empty silence list
    tracks: Array<'system' | 'mic'>;
    bucketMs: number;
    durationMs: number;
    envelopeDb?: number[];
    silenceRanges: Array<{ startMs: number; endMs: number; durationMs: number }>;  // silent on EVERY track
    silentMs: number;
    activeRatio: number;
    leadingSilenceMs: number;
    trailingSilenceMs: number;
  } | null;
  warnings: string[];
}

combined.silenceRanges is the safe default for trimming: a moment counts as dead air only when every track is silent, so a narrator pause over a playing video is correctly not treated as silence. combined.leadingSilenceMs / trailingSilenceMs answer "trim the dead air at the start and end" directly. If hasAudio is false, combined is null and reason/message explain why — do not trim against audio in that case, and do not read it as "no silence found".

Errors: NO_PROJECT_OPEN, INTERNAL (audio analysis failed, e.g. no decodable audio stream in a corrupt file).

// request
{ "name": "analyze_audio", "arguments": { "includeEnvelope": false } }
// result (abridged)
{ "hasAudio": true, "recordingDurationMs": 48200, "timebase": "recording",
  "settings": { "silenceThresholdDb": -40, "minSilenceDurationMs": 700, "requestedBucketMs": 100, "bucketMs": 100 },
  "tracks": [
    { "track": "system", "durationMs": 48200, "bucketMs": 100, "silenceRanges": [], "silentMs": 0, "activeRatio": 0.91, "peakDb": -6.2, "warnings": [] },
    { "track": "mic", "durationMs": 48200, "bucketMs": 100, "silenceRanges": [ { "startMs": 4200, "endMs": 5100, "durationMs": 900 } ], "silentMs": 900, "activeRatio": 0.74, "peakDb": -3.1, "warnings": [] }
  ],
  "combined": { "tracks": ["system", "mic"], "bucketMs": 100, "durationMs": 48200,
                "silenceRanges": [], "silentMs": 0, "activeRatio": 0.94,
                "leadingSilenceMs": 320, "trailingSilenceMs": 890 },
  "warnings": [] }

Speech context

get_recording_context

Read the metadata and compact, time-aligned speech context an agent needs to understand the recording. This is a read-only perception tool: it never adds a caption track and never changes preview or export output.

Returns:

{
  project: { id: string; name: string };
  recording: {
    startedAt: string | null; durationMs: number; fps: number;
    width: number | null; height: number | null; sourceType: string;
    hasSystemAudio: boolean; hasMicrophone: boolean;
  };
  transcript: {
    available: boolean;
    source: 'agent_transcript' | 'caption_track' | null;
    language: string | null; modelUsed: string | null; generatedAt: string | null;
    timebase: 'recording';
    totalSegmentCount: number; returnedSegmentCount: number;
    range: { startTime: number; endTime: number };
    segments: Array<{ startTime: number; endTime: number; text: string; confidence?: number }>;
    hint?: string;
  };
}

The stored private transcript is preferred. If none exists but the project has a visible caption track, that track is compacted and returned with source: "caption_track". If neither exists, available is false and the result points to generate_transcript. Use a time window for long recordings when only one passage matters; phrase-level compaction avoids the token overhead of Whisper's word-level caption representation.

{ "name": "get_recording_context", "arguments": { "startTime": 60000, "endTime": 90000 } }

generate_transcript

Run the local whisper.cpp pipeline over the open project's audio, compact the result into timestamped phrases, and atomically save it to analysis/transcript.json. This is intentionally separate from generate_captions: it does not attach any rendered caption layer or create an undoable edit. The initiating call returns a background-job receipt immediately so local Whisper inference cannot exceed the MCP client's call deadline.

Returns immediately: a background-job receipt. A completed get_agent_job_result contains result with the same shape as get_recording_context and the complete new transcript. If that result exceeds the 1 MiB retention bound, the job still reports completed with a compact resultOmitted.summary; call get_recording_context with a time window to read the saved transcript.

Errors: NO_PROJECT_OPEN, MODEL_NOT_DOWNLOADED, BUSY, INTERNAL, or INVALID_ARGS. Model management is shared with caption generation; use list_caption_models and download_caption_model first.

{ "name": "generate_transcript", "arguments": { "modelSize": "base", "language": "en" } }
// immediate result: { "jobId": "...", "jobToken": "...", "status": "running", ... }

Visual context

The visual ladder is text-first. Start with speech, then interactions, then local OCR. Request a frame only when those compact signals are insufficient. All timestamps use immutable recording time.

get_interaction_context

Returns compact click, scroll and shortcut moments. Cadre never records plain keystrokes — they are dropped at capture, so no typed text exists to return and typing is not a signal you can search for or act on. Only privacy-safe shortcut combinations are logged; a typing moment kind appears only when one group repeats a shortcut or exceeds the five-label cap. Cadre also discards raw cursor samples. New recordings may also provide the app name, accessible role/title/description, and normalized control bounds captured on click-down; text-field values are never read.

analyze_visual_context

Ranks moments using the interaction summary, visual changes measured on the already-generated 160 px timeline thumbnail strip, and uniform coverage. Only the selected frames are extracted at 960 px and sent to Apple Vision .fast OCR. Processing is local, lazy, cached, and uses no image-model tokens.

The result reports frameBudget, framesAnalyzed, cacheHits, imageTokensUsed: 0, and moments containing OCR text, confidence, normalized rectangles, interaction evidence, and OCR status. Obvious credential-like OCR strings are redacted before persistence or return.

get_video_frame

Returns one text metadata block followed by exactly one MCP image block. Use it only after text context is not enough: the connected model may charge visual tokens and the frame may contain sensitive screen content.

The metadata includes source/output dimensions, exact time, and normalized crop mapping. The normal whole-edit target is zero image calls, with three as the documented ceiling unless the user explicitly requests visual comparison.

get_edited_frame

Returns one text metadata block followed by exactly one JPEG image block from the live Cadre preview. Unlike get_video_frame, this includes the full edited composition: zoom/crop, background, device frame, cursor, webcam layout, masks, captions, and overlays. Use it after edits to verify the result, not to understand the untouched source.

Cadre temporarily moves the live editor playhead, waits for the newly decoded frame to render, captures only the preview bounds, hides editor-only mask handles, and restores the original playhead. The same privacy and token caution as get_video_frame applies; use it sparingly.

Captions

list_captions

List the current caption segments.

Parameters: none.

Returns: the full CaptionTrack or null if no captions exist yet:

{ language: string; modelUsed: string; generatedAt: string;
  segments: CaptionSegment[];
  style: { fontFamily; fontSize; fontWeight; color; backgroundColor;
           backgroundOpacity; position; marginBottom } } | null

Errors: NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "list_captions", "arguments": {} }
// result
{ "language": "en", "modelUsed": "manual", "generatedAt": "2026-07-19T21:10:00.000Z",
  "segments": [ { "id": "cap-1", "startTime": 1000, "endTime": 2000,
                  "text": "Let's get started.", "confidence": 1, "isUserEdited": true } ],
  "style": { "position": "bottom", "...": "..." } }

add_caption

Add a caption segment over a time range. Returns the created segment.

Returns: CaptionSegment (isUserEdited: true, confidence: 1). Errors: INVALID_ARGS, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "add_caption", "arguments": { "startTime": 1000, "endTime": 2000, "text": "Let's get started." } }
// result
{ "id": "cap-1", "startTime": 1000, "endTime": 2000, "text": "Let's get started.", "confidence": 1, "isUserEdited": true }

update_caption

Update a caption segment by id (text and/or timing).

If both bounds are given, endTime must exceed startTime. An updates object with no recognised field is rejected rather than accepted as a no-op.

Returns: the updated CaptionSegment. Errors: NOT_FOUND, INVALID_ARGS, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "update_caption", "arguments": { "id": "cap-1", "updates": { "text": "Let's dive in." } } }
// result
{ "id": "cap-1", "startTime": 1000, "endTime": 2000, "text": "Let's dive in.", "confidence": 1, "isUserEdited": true }

delete_caption

Delete a caption segment by id.

Returns: { id: string; deleted: true }. Errors: NOT_FOUND, NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

generate_captions

Transcribe the open project with whisper.cpp and attach the caption track. The call returns a background-job receipt immediately; local inference continues under the bounded job registry. The model must already be downloaded in Cadre — use list_caption_models and download_caption_model if it isn't.

Returns immediately: a background-job receipt. A completed get_agent_job_result contains the generated caption track (segments + metadata) in result. If an unusually large track exceeds the 1 MiB retention bound, resultOmitted.summary reports the segment count; call list_captions to inspect the attached track.

Errors:

• NO_PROJECT_OPEN — no project to transcribe.
• MODEL_NOT_DOWNLOADED — modelSize is not downloaded locally. The hint points at list_caption_models / download_caption_model; this replaces what used to be a bare INTERNAL failure.
• BUSY — a caption generation is already running (the caption:generate IPC handler and this tool share one guard, so an agent and the user cannot start two transcriptions that fight over the same temp audio file). Wait for it to finish and retry.
• INTERNAL — the whisper.cpp pipeline itself failed (audio extraction or transcription error).

// request
{ "name": "generate_captions", "arguments": { "modelSize": "base", "language": "en" } }
// immediate result — poll with the returned pair
{ "jobId": "...", "jobToken": "...", "type": "generate_captions", "status": "running", "pollAfterMs": 500, "statusTool": "get_agent_job_status", "resultTool": "get_agent_job_result", "cancelTool": "cancel_agent_job" }

// request — model not downloaded
{ "name": "generate_captions", "arguments": { "modelSize": "medium" } }
// immediate start succeeds; the terminal job result reports status:"failed"
// with error.code:"MODEL_NOT_DOWNLOADED".

set_caption_style

Update the caption track's visual style (font, size, colours, position). Requires an existing caption track — call generate_captions or add_caption first.

Returns: the resulting CaptionStyle:

{ fontFamily: string; fontSize: number; fontWeight: number;
  color: string; backgroundColor: string; backgroundOpacity: number;
  position: 'bottom' | 'top'; marginBottom: number }

Errors:

• NOT_FOUND — no caption track exists yet ("No caption track exists yet — generate or add captions before styling them.").
• INVALID_ARGS — updates has no recognised field, or a field is out of range / not a valid hex colour.
• NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request — bigger text for a small-screen export
{ "name": "set_caption_style", "arguments": { "updates": { "fontSize": 52, "fontWeight": 800 } } }
// result
{ "fontFamily": "-apple-system, BlinkMacSystemFont, SF Pro Display, Helvetica Neue, sans-serif",
  "fontSize": 52, "fontWeight": 800, "color": "#A1A1A6",
  "backgroundColor": "#1D1D1F", "backgroundOpacity": 0.72, "position": "bottom", "marginBottom": 48 }

list_caption_models

List the whisper.cpp models Cadre can transcribe with. Use before generate_captions to pick a modelSize that exists — or to decide whether to download_caption_model first.

Parameters: none.

Returns: array of

{ id: string; name: string; sizeMB: number; isDownloaded: boolean }

Catalogue as of this build: tiny (75 MB), base (142 MB), small (466 MB), medium (1536 MB).

Errors: none.

// request
{ "name": "list_caption_models", "arguments": {} }
// result
[
  { "id": "tiny", "name": "Tiny (Fastest)", "sizeMB": 75, "isDownloaded": true },
  { "id": "base", "name": "Base (Fast)", "sizeMB": 142, "isDownloaded": true },
  { "id": "small", "name": "Small (Balanced)", "sizeMB": 466, "isDownloaded": false },
  { "id": "medium", "name": "Medium (Accurate)", "sizeMB": 1536, "isDownloaded": false }
]

download_caption_model

Download a whisper.cpp model by id. The larger models are hundreds of MB to over a gigabyte, so the call returns a background-job receipt immediately. Already-downloaded models complete without re-downloading.

Returns immediately: a background-job receipt. A completed get_agent_job_result contains the refreshed model list in result, with the same shape as list_caption_models.

Errors:

• NOT_FOUND — modelId is not one of the known catalogue ids; the hint lists the valid ones.
• INVALID_ARGS — modelId fails the bare-identifier shape check.
• INTERNAL — the download itself failed (network error, integrity check failure).

// request
{ "name": "download_caption_model", "arguments": { "modelId": "small" } }
// immediate result: { "jobId": "...", "jobToken": "...", "status": "running", ... }
// completed get_agent_job_result.result is the refreshed model array.

Style, music & audio

set_style

Shallow-merge updates into one style section. Nested objects must be passed whole (e.g. to change a shadow, send the entire shadow object, not a single shadow field).

For an open-ended polish request, Cadre's shipped baseline is a quiet #F5F5F7 → #DDE7F4 gradient, 72px frame padding, 18px corners, a soft low-opacity shadow, minimal keyboard treatment, and blue used only as an accent. Preserve deliberate existing styling and let explicit user direction override this baseline.

updates is validated in two passes. Both run before the update crosses into the editor, so a rejected call changes nothing.

1. Structural — applied at every level of nesting:

• Must be a plain object (not an array, null, or a primitive), and non-empty.
• No key may be __proto__, constructor, or prototype, anywhere in the object — including nested objects.
• Nesting depth is capped at 6 levels.
• Any single object level is capped at 64 keys; any array is capped at 64 entries.
• String leaf values are capped at 5,000 characters.
• Numeric leaf values must be finite (no NaN/Infinity).

2. Field types — every key is checked against the target section's schema:

• Unknown field names are rejected, not ignored. The error lists the valid fields for that section, so a typo is correctable rather than a silent no-op.
• Numbers must be numbers within the field's range (a numeric string like "12" is rejected), enums must be one of the listed values, booleans must be booleans.
• Colours must be hex — #RGB, #RRGGBB, or #RRGGBBAA. Named CSS colours ("red") and rgb(...) are rejected: the compositor parses the hex digits directly and anything else renders as a transparent or black draw.
• Nested objects must be complete. The merge is one level deep, so a nested object replaces its predecessor entirely — sending {"shadow":{"enabled": false}} would drop the other shadow fields. Fields that are optional on the type (e.g. gradient.angle, the spring params of zoomAnimation) may be omitted; all others are required. The error names what is missing.

A violation of either pass is INVALID_ARGS, naming the offending path (e.g. updates.shadow.color), what was expected, and what was received.

Returns: { section: string; config: <full updated section object> }. Errors: INVALID_ARGS (unknown section, unknown field, wrong type, out-of-range number, bad enum or colour, incomplete nested object, or a structural limit above), NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

Section fields

Ranges are inclusive. Where the underlying type documents a range (cursor scale, webcam size, the 0–1 intensities) that range is used verbatim, so the API accepts exactly what the Style panel can express; other bounds are sanity ceilings. See src/shared/types/style.types.ts for the full types.

background

GradientStop is { stop: 0–1; color: hex }; MeshGradientPoint is { x: 0–1; y: 0–1; color: hex }.

Image backgrounds use project-owned assets, normally a relative path under style/assets/. fill centers and crops the image while preserving its aspect ratio; fit contains it with black margins; tile repeats it from the top left at the same density in preview and export. A missing or invalid selected image causes export to fail visibly. Transparent image pixels composite once over black in the opaque preview and MP4, including when blur is enabled.

frame

DeviceFrameType is one of macbook-pro-16, macbook-air-15, imac-24, iphone-16-pro, iphone-16, ipad-pro-13, browser-chrome, browser-safari, browser-arc.

cursor

returnToStart is a legacy project field. Existing values load unchanged; set_style rejects writes because cursor looping is no longer available.

keyboard

motion

webcam

// request
{ "name": "set_style", "arguments": { "section": "background", "updates": {
    "type": "gradient", "gradient": { "type": "linear", "angle": 135, "colors": [
      { "stop": 0, "color": "#F5F5F7" }, { "stop": 1, "color": "#DDE7F4" }
    ] } } } }
// result
{ "section": "background", "config": { "type": "gradient", "gradient": { "...": "..." } } }

set_music

Set or clear the background music track, from either the bundled Cadre Library or a caller-supplied file. Pass filePath: null to remove music.

* libraryTrackId and filePath are mutually exclusive and exactly one is required: passing both, or neither, is INVALID_ARGS. filePath: null clears music and must arrive alone.

Cadre Library. Five instrumental background tracks ship with the app, so a caller never has to supply a file:

A library track is copied into the project's music directory exactly as an imported file is — the returned filePath is the in-project copy, and the returned trackId is the library id above.

The path checks (and the name/fadeInMs/fadeOutMs checks) only run when a track is being set — set_music({ filePath: null }) clears music unconditionally and ignores the other fields.

Real duration. When a track is set, the tool probes it with ffprobe before returning — durationMs on the result is the track's actual length, not a placeholder. A path that ffprobe cannot read as audio (wrong format, corrupt file, a non-audio file with an audio-like extension) is rejected with INVALID_ARGS rather than silently stored with durationMs: 0, which used to render as a zero-width region on the music timeline and gave the loop math nothing to work with.

Returns: the resulting MusicTrack, or null when cleared. Errors: INVALID_ARGS (bad path, name, or fade duration; or filePath names a file ffprobe cannot read an audio duration from), NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request — a bundled Cadre Library track
{ "name": "set_music", "arguments": { "libraryTrackId": "library-the-steady-hand", "volume": 0.18 } }
// result
{ "trackId": "library-the-steady-hand",
  "filePath": "/Users/me/Movies/Demo.screencraft/recording/music/the-steady-hand.mp3",
  "name": "The Steady Hand", "volume": 0.18, "fadeInMs": 1500, "fadeOutMs": 1500,
  "loop": true, "durationMs": 102000 }

// request
{ "name": "set_music", "arguments": { "filePath": "/Users/me/Music/bed.mp3", "volume": 0.25 } }
// result
{ "trackId": "b6...", "filePath": "/Users/me/Music/bed.mp3", "name": "bed.mp3",
  "volume": 0.25, "fadeInMs": 2000, "fadeOutMs": 2000, "loop": true, "durationMs": 187340 }

// request — a file ffprobe can't decode
{ "name": "set_music", "arguments": { "filePath": "/Users/me/Music/corrupt.mp3" } }
// result
{ "error": { "code": "INVALID_ARGS",
             "message": "set_music: 'filePath' is not a readable audio file (ffprobe found no audio duration in /Users/me/Music/corrupt.mp3).",
             "hint": "Pass an absolute path to a decodable audio file (.mp3/.m4a/.wav/.aac)." } }

set_audio_gains

Set system and/or microphone gain multipliers. At least one must be provided.

*At least one of systemGain / micGain is required.

Returns: { systemAudioGain: number; micAudioGain: number } (the resulting gains — note the field names are the store's *AudioGain form). Errors: INVALID_ARGS (neither provided), NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "set_audio_gains", "arguments": { "micGain": 1.4 } }
// result
{ "systemAudioGain": 1, "micAudioGain": 1.4 }

History

undo

Undo the last undoable edit. Agent and human edits share one undo history.

Parameters: none. Returns: { canUndo: boolean; canRedo: boolean }. Errors: NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

// request
{ "name": "undo", "arguments": {} }
// result
{ "canUndo": false, "canRedo": true }

redo

Redo the last undone edit.

Parameters: none. Returns: { canUndo: boolean; canRedo: boolean }. Errors: NO_PROJECT_OPEN, EDITOR_NOT_AVAILABLE.

Background jobs

These three lifecycle tools make long imports, Whisper runs, and model downloads reliable with MCP clients that impose a 60-second call deadline. The registry is local to one Cadre app boot: active work is aborted and all job secrets disappear when the app exits or rotates its bearer token.

get_agent_job_status

Poll a job without returning its potentially larger result.

Returns:

{
  jobId: string;
  type: 'import_video' | 'add_animation_overlay' | 'generate_captions' |
        'generate_transcript' | 'download_caption_model';
  status: 'running' | 'cancelling' | 'completed' | 'failed' | 'cancelled';
  createdAt: string;
  finishedAt?: string;
  projectId?: string;     // project captured when transcription began
  progress?: unknown;     // bounded stage/percent/byte snapshot when available
  error?: { code: string; message: string; hint?: string };
  resultAvailable: boolean;
  resultOmitted?: {
    reason: 'size_limit'; bytes: number; maxBytes: number; summary?: unknown;
  };
}

failed is a terminal operation failure and carries the same structured error shape as other tools. resultAvailable becomes true only for a completed job whose full result fit the 1 MiB retention bound.

Errors: NOT_FOUND for an unknown, expired, wrong-token, or prior-boot job. Unknown ids and wrong tokens intentionally return the same response.

get_agent_job_result

Return the same snapshot as get_agent_job_status, plus result once the job is completed and its result is retained. Calling it while a job is running is safe: it returns the current status without a result field, so clients may use this as their only polling call.

Returns: the status shape above, plus result: unknown when status:"completed" and resultAvailable:true. Terminal results expire after 15 minutes; start-specific docs define the exact result shape.

Errors: NOT_FOUND under the same ownership/expiry rules as status.

cancel_agent_job

Request cancellation of one job. The registry aborts the owned runner, which propagates through the service's AbortSignal to Whisper, FFmpeg, streaming file copy, or HTTPS download cleanup. Poll status until cancelled; cancellation acknowledgement does not claim the child process has settled yet.

Returns: { jobId: string; accepted: boolean; status: AgentJobStatus }. accepted:false means the job is terminal or has crossed its irreversible commit point; inspect status/result rather than assuming work was discarded.

Errors: NOT_FOUND for an unknown, expired, or wrong-token pair.

// poll
{ "name": "get_agent_job_status", "arguments": { "jobId": "...", "jobToken": "..." } }
// result
{ "jobId": "...", "type": "import_video", "status": "running",
  "progress": { "stage": "Extracting audio...", "percent": 55 }, "resultAvailable": false }

// cancel only with the exact pair returned to this caller
{ "name": "cancel_agent_job", "arguments": { "jobId": "...", "jobToken": "..." } }
// result
{ "jobId": "...", "accepted": true, "status": "cancelling" }

Export

export_video

Render and encode the open project to outputPath, as .mp4 or .mov. Requires an active license — export is the paid action. Returns once the job has started; poll get_export_status for progress and completion.

Returns: { started: true; outputPath: string; status: ExportStatusSnapshot }.

Errors:

• INVALID_ARGS — outputPath fails a path check above, uses an extension other than .mp4/.mov, or resolves inside the project's recording/ directory. That directory holds the only copy of the captured video and audio and the encoder runs with -y (overwrite); exporting there would destroy the source recording, so it is refused outright rather than overwritten. Export to the Desktop, Movies, or the project's exports/ directory instead.
• NO_PROJECT_OPEN — no project to export.
• EXPORT_IN_PROGRESS — an export is already running.
• LICENSE_REQUIRED — no active subscription. This is the paywall; an agent cannot bypass it. Ask the user to activate a license in the Cadre app, then retry.

// request
{ "name": "export_video", "arguments": { "outputPath": "/Users/me/Desktop/demo.mp4", "resolutionMode": "1080p", "aspectRatio": "9:16" } }
// result
{ "started": true, "outputPath": "/Users/me/Desktop/demo.mp4",
  "status": { "phase": "exporting", "progress": null, "result": null, "error": null } }

get_export_status

Current export phase, latest progress, result, and any error. Poll this after export_video.

Parameters: none. Returns: ExportStatusSnapshot. Errors: none.

// request
{ "name": "get_export_status", "arguments": {} }
// result (mid-run)
{ "phase": "exporting",
  "progress": { "percent": 62, "eta": 8, "currentFrame": 1860, "totalFrames": 3000, "stage": "Encoding video" },
  "result": null, "error": null }
// result (done)
{ "phase": "completed", "progress": { "percent": 100, "...": "..." },
  "result": { "outputPath": "/Users/me/Desktop/demo.mp4", "fileSize": 8412345, "duration": 25000 }, "error": null }

cancel_export

Cancel the in-progress export, if any. No-op when nothing is running.

Parameters: none. Returns: { cancelling: boolean; status: ExportStatusSnapshot } (cancelling is true only if an export was actually running). Errors: none.

// request
{ "name": "cancel_export", "arguments": {} }
// result
{ "cancelling": true, "status": { "phase": "exporting", "...": "..." } }

Not in v1

For orientation, these are deliberately absent in v1:

• No webcam capture from an agent — the camera preview window is user-driven (see Recording). (Recording control itself was in this list; start_recording and its six siblings are now real tools.)
• No agent-settable recording duration or auto-stop: the agent sleeps and calls stop_recording itself.
• No camera-layout, no split-point, no playhead, no music.update as MCP tools — though these exist as bridge commands (split.add/remove, playhead.set, music.update) and may be surfaced in a later version. (delete_speed was in this list; it is now a real tool. Masks/highlights and overlays were too — add_mask / update_mask / delete_mask and add_text_overlay / add_svg_overlay / add_graphic_overlay / add_animation_overlay / update_overlay / delete_overlay are now real tools too.)
• Reads are constrained the same way writes are: export_video's outputPath (write) and set_music's filePath (read, to probe duration) are the only file paths most tools ever touch. import_video's path and add_animation_overlay's path are bounded import exceptions: they read caller-chosen local media only to copy or decode it into a Cadre-managed .screencraft project directory. There is still no general-purpose file read or write.