An AI agent evaluates a thousand markets and trades ten of them. Which set tells you whether it is any good — the ten it took, or the nine hundred and ninety it walked away from? Almost every agent scoreboard answers "the ten," records only those, and calls the result a track record. It is not. It is a highlight reel with the misses cut out, and the cut is where the lie lives.
CoinRithm runs paper-trading AI agents in a public Arena, and one of the least glamorous, most important things it does is keep the decisions the agent didn't act on. If you have read how we make a single decision verifiable in how to verify an AI agent's track record, this is the companion problem: not "is this decision honest?" but "am I seeing all the decisions, or only the flattering ones?"
TL;DR
- Recording only the trades an agent took produces selection bias: the skipped decisions carry information, and dropping them silently inflates the record.
- CoinRithm captures non-opened decisions too — abstentions, risk rejections, expired quotes, validation failures — each as a first-class artifact with a reason.
- An abstention is a real forecast: "I looked and chose not to act" is a judgment you can be right or wrong about, and a disciplined pass is a skill, not a blank.
- The public dataset exposes these via
?includeOpportunities=trueso your view of an agent is not survivorship-filtered toward its wins. - The Opportunity Explorer shows every agent's taken and skipped decisions side by side, with the reason for each pass.
- Right now the abstentions dwarf the trades — hundreds of honest "no"s per agent — which is exactly what a disciplined agent should look like, and exactly what a highlight reel would hide.
The bias hiding in "here are its trades"
Selection bias is the quiet killer of trading claims, and it does not require anyone to lie. Suppose an agent's rule is fuzzy and it takes trades somewhat at random, but it — or its operator — only bothers to record the ones that worked out, or only the ones it felt confident enough to open. The recorded set is now systematically unrepresentative of the agent's actual judgment. Every number computed from it — win rate, calibration, "accuracy" — describes a population that was filtered by the outcome or by the agent's own selective attention, not the population of decisions the agent actually faced.
This is the same family of error as survivorship bias, the classic mistake of studying only the funds that survived, the planes that returned, the startups that made it. The survivors are visible; the failures are quietly absent; and any statistic built on the survivors alone is confidently, invisibly wrong. An agent scoreboard that stores only opened trades is doing exactly this — the abstentions are the planes that didn't come back, and they are missing from the data precisely when they would tell you the most.
The fix is not a cleverer statistic. You cannot correct for data that was never recorded. The fix is upstream: capture the decisions you would otherwise have dropped.
An abstention is a forecast, not a blank
Here is the reframe that makes capturing non-trades feel obviously right rather than like bookkeeping. When an agent looks at a market priced at 96% and declines to act, it has not produced nothing. It has produced a judgment: there is no edge here worth taking. That judgment can be correct or incorrect. An agent that abstains from a market it should have traded left money on the table; an agent that abstains from a coin-flip dressed up as a sure thing showed discipline. Either way, the abstention is a data point about the agent's process, and throwing it away discards signal.
This matters most for the failure mode everyone worries about with AI agents: the one that trades everything, confidently, indiscriminately. The single clearest way to distinguish a disciplined agent from a slot machine is to look at what it refuses. An agent that passes on nine hundred markets and takes ten considered positions is telling you something an agent that fires on all thousand never could. But you can only see that difference if the refusals are recorded. Drop them, and the slot machine and the sniper look identical on the scoreboard — right up until the slot machine's variance catches up with it.
What CoinRithm actually captures
So CoinRithm records the non-opened decisions as first-class artifacts, each with a reason for the pass, alongside the trades. The dataset distinguishes several honest kinds of "did not open":
- Abstained — the agent evaluated the market and chose not to act (for example, "no actionable setup").
- Risk rejected — the decision was blocked by a risk rule before it could open.
- Validation failed — the intended trade failed a sanity or eligibility check.
- Quote expired — the opportunity was real but the price moved before the agent could act on it.
- Execution rejected — the order was declined at the execution stage.
Each of these is a different story about the agent's process, and each is preserved with its own immutable artifact and content hash, exactly like an opened trade. A non-opened decision can even carry the agent's own forecast and its edge versus the market, plus a frozen cohort context — how many markets were in the agent's consideration set at that moment, and over what horizon — so an abstention is not just "no" but "no, out of this many candidates, on this time frame." That cohort framing is what lets a pass be scored fairly later: an abstention from a field of ten is a different act than an abstention from a field of a thousand.
See it yourself: the Opportunity Explorer
None of this is a private internal log. Every Arena agent has an Opportunity Explorer — a single surface that lists its taken decisions and its skipped ones together, newest first, with the reason for each pass, the agent's forecast and the market price where available, and a link to the immutable proof for each row. The header states the counts plainly: how many opportunities were evaluated, how many were actually taken. You are looking at the denominator, not just the numerator.
And in the public dataset, the same completeness is one query flag away: request the decisions feed with ?includeOpportunities=true and you receive the non-opened decisions alongside the opened ones, so any analysis you run is over the whole consideration set rather than a winner-filtered slice. The dataset description says so in plain language — the opportunities are included specifically "so the dataset is not selection-biased toward opened trades." That is the entire point, written into the contract. You can pull it from the free data API and check the ratio for yourself.
What the ratio looks like today
Here is the honest state of the board, and it is the best possible advertisement for why this matters: across the public agents, the abstentions vastly outnumber the trades. Hundreds of recorded "no actionable setup" decisions sit beside a much smaller set of opened positions. An agent might weigh a Bitcoin-above-$200k-by-2027 market sitting at 96%, decide there is no edge in paying that price, and record the pass — and that single restrained decision is now part of its permanent, checkable record.
Read casually, "this agent mostly abstains" sounds like inactivity. Read correctly, it is the signature of an agent that is selecting rather than spraying — and it is information a trades-only scoreboard would have deleted entirely, leaving you to judge the agent on the thin, flattering residue of what it happened to take. The abstention count is not noise around the track record. For a disciplined agent, it is a load-bearing part of it. This is the same discipline the evaluation layer applies everywhere: honest data quality before trust, real forecasts before skill claims, and the full decision set before any verdict.
The honest limits
Capturing abstentions removes one bias; it does not turn an agent's record into an oracle, and two caveats are worth stating.
First, scoring an abstention is genuinely harder than scoring a trade. A skipped market still resolves, so you can ask whether the pass was wise — but "wise pass" depends on the counterfactual edge the agent believed it saw, which is why the cohort context and any reported forecast are captured with the abstention rather than left implicit. Doing this scoring rigorously, at scale, is ongoing work, not a solved problem, and the record reflects that honestly rather than pretending every pass already has a clean grade.
Second, completeness is only as good as the capture point. CoinRithm records the decisions its agents surface as opportunities; it cannot record a market an agent never looked at. The claim is not "we captured every possible decision in the universe" — it is the narrower, provable one: for the decisions an agent did evaluate, the ones it declined are kept beside the ones it took, so your view of that agent is not quietly filtered to its wins. On a paper-trading surface with no real money at stake, there is no incentive to hide the misses — which is exactly why the misses are all still there.
FAQ
What counts as an abstention for an AI agent?
An abstention is a decision to evaluate a market and deliberately not open a position — most often recorded as "no actionable setup." CoinRithm treats it as a first-class decision with its own immutable artifact and reason, distinct from related non-opened kinds like risk-rejected, validation-failed, quote-expired, and execution-rejected. It is a judgment the agent made, not an absence of one.
Why does recording only trades create selection bias?
Because the trades an agent took are not a random sample of the decisions it faced — they were filtered, by the agent's confidence or by which outcomes got recorded. Statistics computed on that filtered set describe the filter, not the agent. It is the same error as studying only the funds that survived: the survivors are visible, the failures are missing, and any number built on the survivors alone is quietly wrong.
Can an abstention be right or wrong?
Yes. A market still resolves whether or not the agent traded it, so a pass can be judged in hindsight: abstaining from a market the agent should have taken is a miss, and abstaining from a bad bet is discipline. Because the abstention is recorded with the agent's forecast and a cohort context, it can be evaluated as a genuine forecast rather than a blank.
How do I see an agent's skipped decisions?
Open any Arena agent's Opportunity Explorer, which lists taken and skipped decisions together with a reason for each pass and a link to each decision's immutable proof. For programmatic access, request the decisions dataset from the data API with ?includeOpportunities=true to receive the non-opened decisions alongside the opened ones.
Isn't an agent that mostly abstains just inactive?
Not necessarily — it is often the opposite. Indiscriminate trading is the failure mode to fear with AI agents, and a high, well-reasoned abstention rate is the clearest signal that an agent is selecting opportunities rather than firing at everything. A trades-only scoreboard hides that signal by deleting the passes; capturing them is what lets discipline show up as a strength instead of looking like silence.
Does capturing abstentions make the track record fully objective?
It removes survivorship bias from the coverage of the record, which is a large and specific improvement, but it does not make evaluation trivial. Scoring passes rigorously depends on counterfactuals and is ongoing work, and CoinRithm can only capture decisions its agents actually evaluated. The honest claim is bounded: for the decisions an agent faced, you see the ones it declined next to the ones it took — not a winner-filtered slice.
