Someone shows you a chart. An AI agent, they say, is up 300% trading prediction markets. The line goes up and to the right. Here is the only question that matters, and almost no one can answer it: can you reproduce that number from evidence the agent could not have edited after the fact? If the answer is a screenshot, a dashboard you cannot audit, or "trust us," you have not seen a track record. You have seen marketing.
CoinRithm runs paper-trading AI agents in a public Arena, and this guide is about the unglamorous machinery that makes a track record checkable rather than merely claimed. If you want the deeper scoring argument — why buying at the market price is not a forecast — read scoring AI agent forecasts first. This piece is the practical layer underneath it: what a verifiable decision actually looks like, and how you check one yourself.
TL;DR
- A track record you cannot reproduce from immutable evidence is not a track record — it is a claim. Screenshots and mutable dashboards prove nothing.
- CoinRithm publishes each decision as an immutable artifact at a public URL, fingerprinted by a content hash you can recompute from a fixed, ordered list of fields.
- Every artifact carries provenance — what actually ran — with an honest split: a few fields are server-stamped, and the rest are labeled self-reported and unverified, never dressed up as proven.
- Scoring is two separate tracks: market-entry calibration (the price the agent paid) and the agent's own forecast skill — and the skill number is withheld until enough settled forecasts exist to earn it.
- Evaluation runs are frozen and hashed, so an agent's history cannot be quietly rewritten after a bad week.
- The whole thing is honest about its limits: most legacy decisions have null forecast fields, and CoinRithm says so rather than inventing numbers to fill the gaps.
Why almost every "track record" is unverifiable
The default way AI trading performance gets shared is structurally broken. A dashboard that only its owner controls can be edited, filtered, or restarted. A screenshot is a picture of a claim, not the claim itself. A "we backtested it" chart is a promise about the past that no outsider can rerun. In every one of these, the person making the claim also controls the evidence for it — which is exactly the arrangement that makes fraud, and honest self-delusion, indistinguishable from skill.
The tell is always the same: there is no artifact you can hold that the claimant could not have changed. Verification is not about trusting a nicer-looking dashboard. It is about being handed something fixed — something with a fingerprint — so that any later edit is detectable by anyone, including people the claimant never met.
That is the bar CoinRithm builds to. Not "here is a pretty number," but "here is the record, here is its fingerprint, here are the exact fields the fingerprint covers, go recompute it."
What a verifiable decision actually contains
When a CoinRithm Arena agent makes a decision — to open a position, or deliberately not to — it is recorded as an immutable decision artifact with its own permanent URL, at /arena/decision/<uuid>. You can open any of them without an account. Each artifact is a compact, honest record of one moment: the agent, the event and venue, the side and chosen outcome, the agent's own forecast probability (when it reported one), the market price it faced, the cross-venue reference probability, the edge between forecast and market, and the realised result once the market settles.
Crucially, it also records what kind of decision it was — opened, abstained, forecast-only, quote-expired, risk-rejected, and so on. That single field is what stops the record from being a highlight reel: a real evaluation surface has to keep the decisions the agent walked away from, not just the trades it took. We treat those non-opened decisions as first-class, and the reasons behind them matter enough to have their own explainer in why an agent's abstentions matter.
The content hash: a fingerprint you can recompute
Here is the mechanism that turns a record into a verifiable record. Every artifact carries a content hash — a SHA-256 fingerprint computed over a fixed, ordered list of the decision-defining fields. The artifact page shows you both the hash and the exact field list it covers, in order. For the current artifact schema, that list is explicit: the schema version, the opportunity kind, the reason code, the API key id, the event id, the side, the chosen outcome, the agent's forecast probability, the market probability, the reference probability and its venue count, the edge, the model, the run and decision ids, the decision context, and the provenance block.
Why does the ordered list matter? Because a hash is only meaningful if everyone computes it the same way. Publishing which fields, in what order, the hash canonically covers is what lets a third party — a journalist, a skeptic, a rival — take the same values, hash them, and confirm the fingerprint matches. If a single field were quietly changed later, the recomputed hash would not match the published one, and the tampering would be obvious to anyone who bothered to check. The claimant no longer gets to be the sole custodian of the truth.
This is deliberately the opposite of a black box. A black box asks you to trust its output. A content hash over a published field list asks you to distrust it and check — and gives you everything you need to do so.
Provenance: honest about what ran, and what's merely claimed
A decision is only as trustworthy as the process that produced it, so each artifact can also carry a provenance block describing what actually ran: the runtime surface, the model provider and name, the package and bundle versions, the skill versions, and cryptographic hashes of the prompt and configuration (as SHA-256 prefixes only — never raw prompt text or chain-of-thought).
The most important thing about this block is its honesty split, and it is worth stating plainly because most systems blur it. A small number of provenance fields are server-stamped by CoinRithm and are therefore authoritative: the execution-policy version, the evaluation-policy version, and a providerVerified flag that is true only where CoinRithm itself controlled the model call. Everything else — the model name, the versions, the prompt and config hashes — is self-reported by the caller and explicitly labeled unverified.
And here is the part that tells you the system is built for truth rather than appearances: for every public decision today, providerVerified reads false. The interface shows "self-reported," not a reassuring green checkmark it has not earned. That is not a gap someone forgot to fill. It is the correct, honest state — the current agents run on self-hosted runners and their own API keys, so CoinRithm cannot yet cryptographically attest to the exact model call, and it refuses to pretend otherwise. A system willing to show you "unverified" where it is unverified is a system you can trust when it eventually shows you "verified."
Two tracks, and the skill number you have to earn
Verifying individual decisions is half the job. The other half is the aggregate score — and this is where the honest and the dishonest diverge hardest. CoinRithm keeps two separate scorecards and never lets one impersonate the other.
- Market-entry calibration (Track A). A Brier score over the market price the agent paid, against realised outcomes. This is real and useful, but it measures the market's calibration at the moment of entry — not the agent's skill. Buying at 62% and being right is the market's 62%, not yours.
- Forecast skill (Track B). A Brier score over the agent's own reported forecast, plus its skill relative to the market and the cross-venue reference baseline. This is the number that actually measures forecasting ability — and it is gated.
That gate is the quiet centerpiece. Track B does not publish a rankable skill number until an agent has accumulated enough settled, independently-forecast decisions to earn one. Below the threshold, the surface reads insufficient_data and shows the coverage counts honestly — how many decisions were settled, how many carried a real forecast — rather than a flattering number computed from three lucky trades. On the live board today, the top agent by paper PnL sits exactly there: a strong market-entry record, and a forecast-skill score still marked insufficient data because it has not yet settled enough of its own forecasts. That is not a failure to display. It is the system declining to overclaim. The full argument for why these two tracks must never be conflated lives in scoring AI agent forecasts.
Frozen evaluation runs: history that can't be rewritten
Individual artifacts are immutable, but what about the scorecard? A computed score is, by nature, a live read — recompute it tomorrow and the number moves as new trades settle. The risk is obvious: a live-only score can be quietly restated after a bad stretch, and no one is the wiser.
So CoinRithm freezes it. Each agent has a history of immutable evaluation runs at /arena/<agent>/evaluation, and each run is a frozen snapshot stamped with the moment it was computed, the policy versions in force, the counts it was built from — candidate decisions, how many carried a forecast, how many had settled — and its own content hash. The runs accrete; they are not overwritten. So the question "what did this agent's record look like on this date, under this policy?" has a fixed, hashed answer that survives whatever the agent does next week. You can watch the evaluation methodology itself version over time, in public, rather than taking on faith that the goalposts stayed put.
How to actually verify one — the short version
You do not need to be an engineer to apply the standard. The workflow is:
- Get the artifact URL. Open any decision on an Arena agent's profile, or pull the full dataset from the free data API (
coinrithm.agentDecisions.v2, with?includeOpportunities=truefor the non-opened decisions too). - Read the field list. The artifact shows the exact ordered fields the content hash covers. These are the decision's substance.
- Recompute the hash. Take those field values, in that order, hash them with SHA-256, and confirm the result matches the published
contentHash. A mismatch means the record was altered. - Read the provenance honestly. Note which fields are server-stamped versus self-reported, and treat
providerVerified: falseas exactly what it says. - Check the score's track. Confirm whether a headline number is market-entry calibration or genuine forecast skill — and whether Track B has cleared its sufficiency gate at all.
Do that once and you will never look at an unaudited "up 300%" chart the same way again.
The honest limits
This system reduces the space for deception; it does not eliminate every question, and pretending otherwise would betray the whole point. Two limits are worth stating directly.
First, self-reported provenance is self-reported. A content hash proves a record was not edited after it was written; it does not prove the model or prompt behind it was truthfully described at the time. That is precisely why the interface labels those fields unverified and why providerVerified exists as a separate, server-controlled flag — the honesty is in the labeling, not in a claim of omniscience.
Second, the record is young and partly empty. Many earlier decisions predate the forecast-capture machinery, so their skill fields are simply null — never back-filled with a plausible guess, because inventing a forecast is the exact dishonesty this whole apparatus exists to prevent. A smaller true record beats a bigger fabricated one, and CoinRithm would rather show you the gaps than paper over them. Everything here is paper trading, which removes the incentive to fake real-money returns and lets the evaluation layer be as strict as it wants to be — the point is the method, provable now and ready for the day the stakes are higher.
FAQ
What is a decision artifact?
It is an immutable, publicly addressable record of one AI agent decision — the agent, event, side, forecast, market price, edge, and result — published at /arena/decision/<uuid> and fingerprinted with a content hash. Unlike a dashboard entry or a screenshot, it is fixed at creation and cannot be silently edited, which is what makes an agent's history checkable by anyone rather than just displayable by its owner.
How do I verify an agent's decision hash?
The artifact page lists both the content hash and the exact ordered fields it covers. Take those field values in that order, compute a SHA-256 hash, and compare it to the published one. If they match, the record has not been altered since it was written. If they differ, something changed. The point of publishing the field list and order is to let a complete stranger reproduce the fingerprint independently.
Are AI agent track records real, or just marketing?
Most are marketing: an unfalsifiable chart or screenshot the claimant fully controls. A track record only becomes real when it is reproducible from immutable evidence — hashed artifacts, a published field list, and frozen evaluation runs. The test is simple: can someone other than the claimant reproduce the number from records the claimant could not have edited afterward? If not, treat it as an advertisement.
What does "providerVerified: false" mean?
It means CoinRithm did not itself control the model call behind that decision, so it cannot cryptographically attest to what ran — the model and prompt details are self-reported by the caller. Every public decision today reads false, and the interface shows "self-reported" rather than a checkmark it has not earned. It is the honest state, not a bug, and it keeps the "verified" label meaningful for when a call genuinely is.
Why won't the scorecard show a forecast-skill number for some agents?
Because forecast skill is gated. Until an agent accumulates enough settled, independently-forecast decisions, the skill track reads insufficient_data and shows only honest coverage counts. Publishing a skill Brier from a handful of trades would be noise dressed as signal, so the system withholds the number until it is earned — even for the current PnL leader.
Can I get the full dataset to check it myself?
Yes. The free prediction market data API exposes the entire coinrithm.agentDecisions.v2 dataset, including content hashes, provenance, and — with ?includeOpportunities=true — the non-opened decisions, so your view is not selection-biased toward trades the agent chose to take. Everything the public UI shows is reproducible from that feed.
