The Beat Engine and cartoon scenes

A shot maker buys clips from a video service and hopes. The Beat Engine is the other way to make animation: as data. A grammar file holds the timing of a show, a scene file holds what happens, and render.py derives every drawing from the two with a seeded random generator, so the same inputs give the same film and a change to timing, framing or a gesture is one line edited and a render of seconds. Nothing is keyframed by hand and nothing is generated by a model. It ships as the project's own skills aioum-beat-engine and aioum-cartoon-scene, with aioum-voice-clone for the voices and aioum-motion-compiler for a whole film from a screenplay (MIT, under Skills with aioum as their source), and as the Cartoon scene pipeline on Deploy Agents. Cartoon Studio sells the finished film: a prompt in, a cartoon out, at a price per length.

Animation as data

The grammar (scripts/grammar.json) is every animation constant with the file it was measured in: drawings on twos, because 55.6% of the reference's holds are exactly two frames; the hold lengths and their weights; the blink's closed lengths and gaps, 22.5 blinks a minute; the share of the speaker's changes that move the whole head and how far; a head snap at every line start and after a pause of three closed frames; a listener moving one part every 12 to 24 frames; the mouth gate, closed at 30 dB under the line's peak; and ranges, the reference's own spread on ten rows. A new show gets a new grammar: run the same detectors (scripts/detectors/) on its clips and write the constants with their sources. Swap --grammar and the same scene renders in the other show's timing.

The scene is what happens: fps and size, the background (quantised, outlines dilated, saturation set), four framings (zooms of the same background), models (one per character), placements (where each stands and how big), shots (start, end, framing, placement, characters), lines (speaker, start, end, file) and beats (a gesture held over a span, or a cycle), plus a viseme track: one mouth shape per frame per character, read from the audio alone. scene_from_script.py builds it from a voiced script: the lines laid out 0.3 to 0.7 s apart, a master mix, shots cut by speaker runs (a shot under 1.2 s joins the one before), beats from the gesture tags. visemes_local.py builds the track: closed under the gate, else a shape from the spectral centroid. At render, grammar.mouth gates and alternates the shapes: an open shape never repeats on the next drawing.

The rig. A character is a code model or layered art. A code model is a python module with one draw function, cartman(d, cx, gy, u, pose) or kid(d, cx, gy, u, pose, spec): d a PIL.ImageDraw on a transparent layer the size of the frame, cx the centre line, gy the ground line, u the head width in frame pixels, pose the state the grammar drives (view, head and body offsets, mouth, arm, brow, blink or pupil). Layered art is a folder of PNGs cut from one hero drawing, all the same size, with rig.json beside them (size, pivots, mouth_box, eye_boxes) and one mouth_<shape>.png per open viseme; the engine only moves, rotates and stacks the layers, never redraws them. A new character keeps the signature and the pose vocabulary. The example rig (Cartman, Kyle and Stan as code models, a winged brain as layers) and its five voiced lines are the owner's project art, included to learn the contract and check the chain, not for your own production.

The render. render.py walks each shot on twos with a global hold timeline; the forced boundaries are the shot start, a line start, a phrase start after a pause, a blink on and off and a scene beat. On a change the speaker moves its mouth plus one face part, or snaps its head on a phrase; a listener moves one part. Only drawings that differ are drawn; holds repeat the bytes. It prints one JSON line (frames, drawings drawn, seconds, the mouth-change share, head snaps and their gaps) and writes <out>.events.json beside the mp4. It is CPU work in Pillow: python3 with Pillow and numpy, ffmpeg and ffprobe on the PATH, scipy for check.py; no GPU, and the runner's GPU jobs on a compute listing do not run it. The shipped example, a 24-second scene at 1920×1080, renders in seconds on a laptop; a 5-second strip faster still.

Measurement is the verdict

check.py runs the detectors that set the grammar's ranges on the rendered piece (changes per second, snaps, median step, wide steps, blinks per minute, blink length, mouth-change share, head-snap gap) and prints each row inside or outside the reference's range. A row outside names the constant to change; a row inside is settled, and no judge is asked whether the timing is right. shot_sheet.py puts one frame from the middle of each shot on one image, stamped with its time and framing, beside reference frames when you give a folder of them: what an image judge scores and what you read in one look, framing, placement, pose, the mouth at that instant, the look against the reference. Timing and motion are never judged from the sheet.

The commands

From the workspace root, where an agent runs them with ws.run after load_skill put the folder under skills/aioum-beat-engine/:

S=skills/aioum-beat-engine/scripts
python3 $S/scene_from_script.py script/lines.json v1 --out-dir beat   # beat/scene_v1.json, beat/master_v1.wav
python3 $S/visemes_local.py beat/scene_v1.json beat/mouth_track_v1.json
python3 $S/render.py --scene beat/scene_v1.json --strip 0 5 --out beat/strip_0_5.mp4
python3 $S/render.py --scene beat/scene_v1.json --full --out beat/v1.mp4
python3 $S/shot_sheet.py beat/scene_v1.json beat/v1.mp4 beat/v1_sheet.jpg [--refs reference/frames]
python3 $S/check.py beat/v1.mp4 --t0 0 --t1 20

render.py takes --seed N for another take of the same scene, --no-audio, --grammar for another show's timing, --root (the project folder every relative path is read from; AIOUM_BEAT_ROOT) and --rig (another rig's modules; AIOUM_BEAT_RIG). A path not found under the project is read from the skill folder, which is how the base scene's art works from any project. Every file lands in the workspace with a URL under /workspace/.

From a script to a reviewed film

aioum-cartoon-scene is the chain around the engine: the voices, which are the skill aioum-voice-clone, and the loop that gets a scene to a confirmed pass in rounds of minutes.

The script. script/lines.json, a list of {id, speaker, text}; speaker a character key of the rig (cart, kyle, stan, brain in the example); a gesture tag in the text lands a beat on that line: [drag], [cough], [smile], [fists], [raise]. A line is one breath.

Casting from an openly licensed human library. Never a real actor's recordings, never a real person's voice without consent, never a voice from the show. One parquet shard of a CC0 or CC-BY speech set comes from the Hub, hf_hub_download("MushanW/GLOBE_V2", "data/<split>-00000-of-000NN.parquet", repo_type="dataset"): GLOBE_V2 is Common Voice derived, CC0 1.0, with age, gender and accent per speaker, hundreds of speakers a shard. measure.py shard.parquet globe voices/cand_globe.json voices/clips_globe measures every one: median pitch in Hz, brightness (the spectral centroid), age, gender, accent, and up to six clean clips of 3 to 12 s. One speaker per character, by the description: pitch first, then brightness, then age and gender, the top three heard. A boy is a high, bright male voice in his twenties pitched up; a smooth salesman a low, dark voice. The prompt set is one wav per character, voices/prompts/<character>.wav: the speaker's cleanest 10 to 13 s clip, pitched up 2 to 7 semitones where the character asks for it (more sounds like a chipmunk); the prompt's pitch becomes the character's. voices/prompts/SOURCE.json travels with the film: the set and its licence, and per character the speaker id, gender, age, natural pitch, the shift and the resulting pitch, the prompt's length.

Cloning, once, in one process. say.py clones every line from the prompt set with Chatterbox. The model loads once per process and a line takes about 12 s on Apple MPS (faster on CUDA, slower on a CPU), so the whole script goes in one --batch call, never one process per line:

python3 skills/aioum-voice-clone/scripts/say.py --batch voices/batch.json --prompts voices/prompts
python3 skills/aioum-voice-clone/scripts/level.py voices/batch.json

voices/batch.json is [{character, text, out}] with the gesture tags stripped from the text; level.py levels every take. Every take carries a sidecar, <out>.json, with its character, text, prompt and style, and --only-stale clones again only what changed. script/lines_voiced.json is the script with file added per line, tags kept: what the engine takes. normalize.py lines_voiced.json voices/prompts locks a character's lines to the prompt's median pitch when a take drifted; say_kokoro.py, in aioum-cartoon-scene, is the stand-in with preset voices (Kokoro-82M on the CPU, a second a line) for a first timing pass of a long script, never the cast. In the python that runs them: chatterbox-tts with torch and torchaudio, kokoro, librosa, soundfile, numpy, pyarrow, huggingface_hub. Then the engine: scene, viseme track, strip, full render, sheet, rows.

The review loop. One problem per round: name it, change one thing, render, measure or judge, decide. Timing, motion and shape are settled by check.py: a row outside its range is the round's problem, and the fix is a grammar constant. The look goes to the image judge paired with a position swap: the sheet and the reference frames sent twice, the candidate and the reference in swapped positions, the pair's agreement taken; a verdict that flips with the position is noise; act on the single biggest reason. One good score is a sample; two in a row with nothing changed but the seed is a pass. A round is minutes; when it runs long, the problem was two problems. Fixes that do not interact (a placement, a beat, a colour) go in one round; two timing constants one at a time. The voices are the slow step and are made once: re-clone a line only when its text changes.

The Motion Compiler

The Beat Engine animates dialogue over a rig that exists. The Motion Compiler makes the whole film from a screenplay: a fresh cast, fresh locations, bodies that cross a room, sit at a desk, hide in a locker, slip and get up. Nothing is drawn by hand per episode. It is the skill aioum-motion-compiler (MIT).

• Characters from a genome. Five families (kid, round, tall, block, critter), a head in five views from front to profile, a palette, headwear, hair, extras. The director writes the genome from the words that introduce a character, and the same name is always the same character. A rig takes about 1 ms.
• Locations from recipes. Sixteen kinds (bedroom, kitchen, street, bus stop, school hallway, classroom, cafeteria, playground, rooftop, bathroom, living room, store, park, alley, forest, car interior), a time of day, a seed, details, 108 props. The same recipe and seed give the same set. A set carries depth objects: what a character stands in front of, passes behind, sits on or hides in, with doors that open. A set takes 3 to 21 ms.
• Motion from verbs. Sixty, each a function of time with anticipation and settle: walking, running, sneaking, sitting, falling, recoiling, pointing with IK hands, grabbing, throwing; hovering, darting, orbiting and perching for a flyer. Idle, blinking and talking are automatic.
• A director. It parses the screenplay, casts it, times every line from its voice file, stages the bodies so nobody stands on anybody, hides them through doors, cuts the camera by the beat, and validates the scene: an error names the beat it comes from. Staging a three-minute screenplay takes about 3 s.
• A renderer. Skia at 1920x1080, the cast depth-sorted with the set's objects, props and effects, libx264 through ffmpeg. Three minutes of film render in about 16 s on an Apple M4.

The screenplay is plain text: the title, sluglines, action lines in the present tense, a name and its spoken words. A slugline's place carries a set's word, and a character is introduced in capitals with a description:

THE LATE PASS

INT. SCHOOL HALLWAY - MORNING

MINA, a small girl with pigtails and a yellow hoodie, runs in from the left. She slips, falls down, and gets up.

OTTO, a round boy with glasses and a green beanie, walks in from the right. He waves.

OTTO
You're late. Again.

MINA
(shrugs)
The bus left without me. I ran the whole way.

From the workspace root, in a python 3.12 with skia-python, numpy and pillow, ffmpeg on the PATH:

MC=skills/aioum-motion-compiler/scripts
# the staging before a voice exists: every line's length estimated from its words
PYTHONPATH=$MC python3 -m director.direct film/screenplay.txt --out film/scene.json --no-audio
# the voice batch, cloned with aioum-voice-clone, then the staging timed to the voices
PYTHONPATH=$MC python3 -m director.voices batch film/screenplay.txt --out film/voices
python3 skills/aioum-voice-clone/scripts/say.py --batch film/voices/batch.json --prompts voices/prompts
PYTHONPATH=$MC python3 -m director.direct film/screenplay.txt --out film/scene.json --batch film/voices/batch.json --target-seconds 120
PYTHONPATH=$MC python3 -m director.validate film/scene.json film/scene.notes.json
# a range first, the whole film when the range is right
python3 $MC/runtime/render.py film/scene.json film/first10.mp4 --start 0 --end 10 --sheet film/sheet.jpg
python3 $MC/runtime/render.py film/scene.json film/film.mp4

direct writes the scene, scene.notes.json (which beat every shot and event comes from, what it could not read) and timing.json (the seconds of dialogue and of action, scene by scene). --target-seconds tightens what is not dialogue until the film is within 10% of the length; it never cuts a line. render.py --sheet writes the frame sheet a judge or a person reads.

studio/server.py is a page on your machine, port 8575: a prompt becomes a screenplay through your own Claude plan, then voices, staging and a render, one job at a time.

Not yet: lip sync follows loudness, not phonemes; get_up is a stiff pivot; walking into a dumpster needs jump; profile mouths are small.

The Cartoon scene pipeline

Cartoon scene (Content). Voice caster casts from the library, measured, writes SOURCE.json and clones every line in one run of say.py --batch, then levels. Scene maker builds and renders with the engine it is told. With the Motion Compiler it writes the voice batch from the screenplay, stages to the voices, validates, renders a range and then the film with its sheet; a change is a changed sentence, a seed or a length. With the Beat Engine it runs the engine's commands from the workspace root and, given one change (a grammar constant, a placement, a beat, a seed), applies exactly that to the scene JSON or to a copy of the grammar under beat/ and renders again. Director, the lead, chooses the engine and writes the script, a screenplay for the Motion Compiler or tagged lines for the Beat Engine, with a voice description per character (age, gender, pitch, brightness), hires both, and runs the review rounds: the validation or the check.py rows first, never a judge for timing; then the sheet through the image judge twice with the positions swapped; a pass confirmed by a second round; a stop after the confirmed pass or six rounds, saying which. It delivers the mp4, the sheet, the script, SOURCE.json and a round log (problem, change, measurement or score, verdict). Stages: Script · Voices · Build and render · Review rounds · Deliver. Needs: the four skills (aioum-beat-engine, aioum-motion-compiler, aioum-cartoon-scene, aioum-voice-clone), from the runner's skills index; an image judge (four calls; $0.01 to $0.02 a call); ffmpeg and python3 beside the node; the workspace with ws.ffmpeg; optional a GPU on your node's own machine for the cloning (a rented listing cannot run your node's scripts; without one the voices clone on the CPU, and the render needs no GPU). One run: four judge calls, two hires at $0.20 and $0.30.

Cartoon Studio: a film from a prompt

You type what the cartoon is about and how long it runs, and you get the film: the studio writes the screenplay, draws the places and things it needs, casts and clones the voices, stages and renders. It is on the marketplace as Cartoon Studio, and you pay on delivery: a deposit of 0.02 USDC when you order, the rest when the finished film is handed over. A film that fails costs you the deposit and nothing more.

aioum tools --q "cartoon studio"              # the listings and their ids
aioum call <make> make --args '{"prompt": "Two kids both claim the last swing, and the teacher settles it.", "minutes": 1}'
# {"id": "20260929-…", "tier": 1, "collectService": "…", "priceOnDelivery": 0.0453, "deposit": 0.02}
aioum call <desk> status --args '{"id": "20260929-…"}'     # the step it is in, each step's seconds, the cost so far
aioum call <desk> result --args '{"id": "20260929-…"}'     # the preview: 480 wide, a bar across it
aioum call <collectService> collect --args '{"id": "20260929-…"}'   # the film's URL, an mp4, 1920x1080

make takes minutes (1 to 12), characters (simple, drawn from shapes, or illustrated, painted on an art GPU) and movement (recorded or hand-built). Its answer names the listing that hands the film over, one per length and kind of characters, and what is owed there. collect answers the film for a finished job of that length and kind; for any other, a film that is not finished, that failed, or that was collected already, it refuses and nothing is charged. The desk holds the rest at 0.001 USDC a call: status, result, cancel, retry (a stopped film again, from the step it stopped in, without a second deposit), around (the screenplay written again without a piece that could not be drawn; nothing is drawn in its place) and ready. The id sample is a finished film to try them on.

What you pay is the run's cost, the studio owner's surcharge and the protocol's fee, and every listing says which is which:

In USDC. The cost is measured on the films the studio sold, every call as their ledger recorded it: the voice GPU by the second it spoke at its rented rate, the writer's calls, the other planning calls and the judge's at the price of the listings a paid film asks; an illustrated film adds the art GPU and its judge for every character. Starting a GPU that was off is the studio owner's cost, as its idle minutes are. The surcharge is 10% of the cost, at least 0.02 USDC a film. The fee is the protocol's 5% at settlement. The studio records what each film really cost, the GPUs' dollars and the calls by model, and status, result and collect carry it beside the price.

Who makes a paid film. Models that are paid for: the studio owner's own keys first, then LLM and judge listings on the marketplace, chosen the way an agent chooses, by rating, by the model and by price, with the next in line asked when one is out of credit. Never the studio owner's Claude subscription. ready answers whether a writer and a judge are there; when they are not, make answers the studio has no funded model for paid runs: … and nothing is charged. A film written by a small model is made from the kit and the library: nothing new is drawn for it, and every character is a person.

The pipeline. Cartoon Studio (Content) on Deploy Agents runs the whole errand. Producer, the lead, orders the film with make, follows the job with status, takes up a stopped job with around or retry, once, looks at the preview, collects the film at the listing make named and fetches it into films/ in your workspace. Film reviewer samples four frames and has the image judge score them against your prompt. You get the film, its length, the score with its reasons, what was paid and the cost as the studio recorded it. The card shows the price of every length. Stages: Order the film · Watch the job · Review and deliver. Needs: the Cartoon Studio listings, an image judge, the workspace; optional ffmpeg beside the node for the frames. With the aioum-motion-compiler and aioum-voice-clone skills and a GPU, the Producer makes the film on your own node instead and no listing is paid. One run: the deposit, ten status calls and one result, one collect, one judge call, one hire at $0.10.

Reach. On the local development network the node answers on this machine only; on a public network the same listing is reachable by anyone.

Sell your own. Run the studio, name it to your node and list it:

# .env
CARTOON_STUDIO_URL=http://127.0.0.1:8575
CARTOON_STUDIO_SURCHARGE_PCT=10      # your surcharge, percent of the run's cost
CARTOON_STUDIO_SURCHARGE_MIN=0.02    # and its minimum, USDC a film

cd offchain
npm run cartoon-studio -- write --costs table.json   # the templates, from your studio's own cost table
npm run cartoon-studio -- list                       # every listing live on your node
npm run cartoon-studio -- prices

table.json is what the studio's costs.py table prints from its finished films. A changed surcharge is write and list again: a listing whose price changed is taken down and listed at the new one.

Next: Localize a service, distill a model — a paid listing your agents keep hiring, found again as an open-weight copy on your node and proved with the same judge in both orders.