Benchmark: the guardrail, measured

Can a small local model actually edit video? Not by writing ffmpeg commands — a raw 3B model produced zero working commands out of 12 attempts. Given Kinocut's typed surface instead, the same 3B shipped 4 of 12, at lower latency than a 35B model writing raw commands. The deterministic layer does the heavy lifting exactly where the model cannot.

Results

Twelve representative edit tasks (trim, resize, audio extract, speed, rotate, thumbnail, WebM, crop, frame export, GIF, self-concat) on one deterministic fixture. Two arms per seat: alone (the model writes a raw ffmpeg command — what an unguarded agent does) and + Kinocut (the model drives the kino CLI; the engine builds and runs the real ffmpeg). One verifier, blind to arm: the artifact must exist and pass every ffprobe check.

Model (local)	Alone (raw ffmpeg)	+ Kinocut	Latency, median
3B VL	0 / 12	4 / 12	540 ms → 337 ms
2B	5 / 12	9 / 12 (+80% rel.)	5.6 s → 2.2 s
27B	10 / 12	11 / 12	2.2 s → 2.6 s
35B MoE	11 / 12	11 / 12 (parity)	18.3 s → 9.6 s

The curve is the story: an unbounded lift at the bottom, +80% at 2B, +10% at 27B, parity at 35B. The layer compensates precisely where the model is weakest — and on the tiny seats it is also cheaper, because synthesizing short typed parameters beats synthesizing long shell commands.

The trim trap

Every seat's raw-arm "canonical" trim was the same command an experienced human writes:

ffmpeg -i input.mp4 -ss 2 -to 5 -c copy out.mp4

It exits zero, reports the right duration — and silently drops the video stream (the stream-copy and output-seek interaction). A 27B model still writes it. Kinocut's engine path avoids the entire class by construction: parameters are typed, validated, and translated, never shell-echoed.

Method

Fixture: one deterministic source (testsrc2 + sine, 10 s, 1280×720, H.264 + AAC). No network, no camera.
Arms: identical prompts; the + Kinocut arm sees the real kino CLI help (as an MCP client sees the tool schema). Single-shot by design — no repair rounds in either arm.
Verifier: blind to arm; artifact must exist and pass duration, dimensions, codec, stream, and frame-count checks via ffprobe.
Execution: argv-only (no shell), sandboxed, 240 s cap. Fixed sampling parameters.
Honesty notes: n=12, one run per cell. Frontier hosted models were not run (no credential used; we do not fabricate reference rows). The bench also caught a real Kinocut bug — a trim end-time semantics flaw (K-1) that cost the Kinocut arm the trim task on capable seats; it is fixed in 1.15.2 (absolute end times now convert explicitly). ASR/transcription and GPU-heavy renders were out of scope for the headless loop.

Run it yourself

The bench uses the same CLI you install anyway:

pip install kinocut
kino doctor

Twelve tasks, one fixture, a blind verifier — the reproducible shape is above. Every operation carries a receipt; that is the point of the surface.