What is kept, and where
Every piece of state in the network, where it lives, who writes it, and what it survives:
- restart: its process stops and starts, by a crash, a kill or a deploy
- reboot: the machine goes down and comes back
- chain restart: the local chain stops and starts
- new network: new contracts, new addresses, new service ids
On the chain
| State | Where | Written by | Restart | Reboot | Chain restart | New network |
|---|---|---|---|---|---|---|
| Contracts and their addresses | the chain; offchain/deployments.json | npm run deploy, scripts/deploy-network.sh | yes | yes | yes | deployed anew; the old manifest is in every snapshot |
| Operators: identity, own stake, commission | StakingManager, IdentityRegistry | each operator's key | yes | yes | yes | registered and staked again from the recorded amounts |
| Delegations | StakingManager | each delegator's key | yes | yes | yes | made again when this machine holds the delegator's key |
| Listings and their service ids | ServiceRegistry | each operator's key | yes | yes | yes | listed again; every id changes; the map is written |
| Sessions and their escrow | PaymentManager | buyers and nodes | yes | yes | yes | no |
| Balances: USDC, ZAI, ETH | the tokens | everyone | yes | yes | yes | no |
| Ratings the chain keeps, disputes, jury stakes | ReputationRegistry, DisputeManager, the arbitrator | the protocol | yes | yes | yes | no |
| Deposit addresses, smart accounts | their factories | the factories | yes | yes | yes | made anew by the new factories |
A public chain keeps its own state. The local chain keeps it in
~/.aioum-chain/state.json (AIOUM_CHAIN_DIR): anvil --state, saved every 15 seconds
(AIOUM_CHAIN_STATE_INTERVAL) and when it stops, loaded at every start. scripts/devstack.sh
deploys only onto a chain that does not hold the contracts.
Off the chain
<runner> is a runner's folder: ~/.aioum-private/runner for your own inside your private
node, ~/.aioum-shared-runner for the local stack's shared runner, ~/.aioum-reviewers/<n> for
its paid reviewers.
| State | Where | Written by | Restart | Reboot | Chain restart | New network |
|---|---|---|---|---|---|---|
| Agents | <runner>/agents.json | a runner | yes | yes | yes | ids rewritten |
| Projects and their setups | <runner>/projects.json | a runner | yes | yes | yes | ids rewritten |
| Runs, the newest 400 | <runner>/runs.json | a runner | yes | yes | yes | ids rewritten; open sessions closed in the record |
| Spend, per hour and service | <runner>/spend.json | a runner | yes | yes | yes | ids rewritten |
| Bought skills | <runner>/skill-cache/ | a runner | yes | yes | yes | moved with the id |
| Learned skills | on each agent, agents.json | a runner | yes | yes | yes | yes |
| Tool ratings, the runner's copy, the newest 2,000 | <runner>/experience.json | a runner | yes | yes | yes | yes, by their key |
| Reviews of listings | <runner>/reviews.json | a runner | yes | yes | yes | ids rewritten |
| The runner's list of listings | <runner>/listings.json | a runner | yes | yes | yes | the old list kept under its network's name |
| Pipelines you created | their agents and project, above | a runner | yes | yes | yes | ids rewritten |
| A shared runner: the people's sealed spending keys, sign-ins | ~/.aioum-shared-runner/user-keys.json, sessions.json, keys.secret | the shared runner | yes | yes | yes | yes, in a snapshot with secrets |
| A paid reviewer: its bond's record, the sealed scores' salts, the callers' keys | ~/.aioum-reviewers/<n>/paid-reviews.json | a reviewer's runner | yes | yes | yes | yes, in a snapshot with secrets |
| Tool ratings, the directory's, 30 days | ~/.aioum-indexer/experience.json | directory | yes | yes | yes | yes, by their key |
| Price history, settled calls, 7 days | ~/.aioum-indexer/demand.json | directory | yes | yes | yes | set aside; read again from the new chain |
| Announced nodes | ~/.aioum-indexer/nodes.json | directory | yes | yes | yes | yes |
| Vouchers and the settle queue | ~/.aioum-node-multi/multinode.json, ~/.aioum-node/node.json, relay.json; SQLite with AIOUM_STORE=sqlite | nodes | yes | yes | yes | set aside under the old network's name, never replayed |
| Transcripts, 7 days | <node dir>/<name>-transcripts/ | nodes | yes | yes | yes | set aside |
| Listings added through the app | ~/.aioum-node-multi/services.added.json | catalogue node | yes | yes | yes | listed again |
| The listing ledger | ~/.aioum-node-multi/listings.json | catalogue node | yes | yes | yes | both networks kept |
| Skills uploaded to the node | ~/.aioum-node-multi/skills/ | catalogue node | yes | yes | yes | yes |
| Templates and the catalogue | offchain/templates/ in the repository; .devstack/catalogue.json built from it and .env at every start | the repository, catalogue:bootstrap | yes | yes | yes | yes |
| Billing accounts and credits | ~/.aioum-node/billing.db | billing | yes | yes | yes | yes |
| Deposits and card payments | ~/.aioum-deposits/deposits.json | deposits | yes | yes | yes | kept as history |
| Wallets watched for deposits | ~/.aioum-deposits/wallets.json | deposits | yes | yes | yes | yes; their deposit address changes |
| The staking node's requests and reviews | ~/.aioum-backer/requests.json | staking node | yes | yes | yes | ids rewritten |
| The staking node's pool | its delegations, on the chain | staking node | yes | yes | yes | delegated again |
| Private node: token, secrets, sign-ins, wallet | ~/.aioum-private/ (owner-only files) | private node | yes | yes | yes | yes |
| Private node: memory, schedule | ~/.aioum-private/memory.json, jobs.json | private node | yes | yes | yes | yes |
| Private node: workspace | ~/.aioum-private/workspace/ | private node, agents | yes | yes | yes | yes |
| Private node: installed models | ~/.aioum-private/models/; the list in models.json | private node | yes | yes | yes | yes |
| Studio: library and cast | ~/.aioum-private/workspace/video/mc/library/ | studio | yes | yes | yes | yes |
| Studio: voices and voice cache | …/mc/studio/voices/, voice_cache/ | studio | yes | yes | yes | yes |
| Studio: jobs and films | …/mc/studio/jobs/<id>/ | studio, one worker per job | yes | yes | yes | yes |
| Keys | .env, offchain/.env | keys console, you | yes | yes | yes | yes |
| Operator keystore | OPERATOR_KEYSTORE_FILE | you | yes | yes | yes | yes |
| Jury salts | ~/.aioum/jury-salts.json | the CLI | yes | yes | yes | yes; they open votes on the old network only |
| Notes on the docs | .notes/docs-notes.json | the app | yes | yes | yes | yes |
| A server deployment's units and env files | .deploy/ | scripts/deploy-network.sh | yes | yes | yes | yes |
Every row is in a snapshot. Secrets are inside only with --with-secrets. Transcripts, media,
the workspace and the studio's films are inside with --full. Model weights are never inside.
How a record is written
Every store writes to a temporary file, forces it to the disk and renames it over the target. A reader, and the next boot, see the old file or the new one.
A file that does not parse is moved aside as <file>.unreadable-<time> before the store starts
from an empty one. The next write cannot replace it.
Each service names itself as the holder of its folder (agents.pid, multinode.lock). A
restore refuses to write under it.
What a rating is kept under
operator|listing|tool: the operator's address, the listing's name from its descriptor, the
tool. A service id is a counter on one deployment and changes with every new one.
Every rating also carries the network it was paid on and the id its listing had there. On another network it is pointed at the id the same listing has now. When the listing is not there, the rating is set aside with its key and comes back when the listing does.
A network is one deployment on one chain:
<chainId>-<hash of an anchor block and the registry's address>. The anchor is the block the
deployment ended in. Block 0 is not used: a local chain makes a new one at every start.
What was lost before
Read from the code and tried on a throwaway chain.
| When | What was lost |
|---|---|
The local chain stopped: a reboot, a crash, devstack.sh down | Everything on it. It ran in memory. The next start deployed again and handed every listing another id. |
| After that | Every rating sat on whichever listing got its old id. Every agent, project and bought skill named ids of a chain that was gone. |
devstack.sh clean, required after every chain reset | The nodes' folders: vouchers, transcripts, and every listing added through the app. |
| A crash during a write | Half a file: listings added through the app, the price history, announced nodes, a bought skill, the private node's schedule and model list. |
| A file that did not parse | All of it. The store started empty and replaced the file at its next write: agents, runs, projects, deposits, backing requests, vouchers. |
| A reboot | Every service stayed down until someone started it. The studio and the model server were started by hand. |
| A new network | Everything that named a service id. No command moved it. |
What is still lost
| When | What | Why |
|---|---|---|
| The local chain is killed or loses power | up to 15 seconds of its transactions | it saves its state on an interval and when it stops |
The runner is killed with kill -9 | up to 250 ms of its writes | writes are gathered for 250 ms before they go to disk |
| Any stop | the run in flight | it reads failed, its project pauses and resumes from that stage |
| Time | ratings older than 30 days in the directory, runs past the newest 400 | the stores are bounded; spend is kept per hour without a bound |
| A new network | balances, escrow in open sessions, delegations from keys this machine does not hold, the chain's own ratings, disputes, jury stakes | they belong to other people or are written by the protocol alone |
| A chain that was reset before ratings carried a key | which listing an older record was about | nothing recorded it; such a record is set aside, not dropped |
Next: Deploying your own network — the same system stood up on a chain of your own, from a funded deployer key to a launch checklist.