Benchmark Gap #3: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use?
One-OffIATecnologia
Manifold MarketsSenza KYCDati di risoluzione verificatiDiscretamente calibratoPrevisione della comunità attualeSì 78%
Previsori
16
Tipo di domanda
binary
Metodologia
Play-money forecasting platform
Tipo di fonte
Previsione
Dati di mercato
Aggiornato 9 ore fa
Obsoleto8 apr 22, 18:06 UTC1 gen 51, 04:59 UTC
Trend
Regole
"Entry level" is deliberately fuzzy: in 2022 terms this would look like an AI (or AIs) that is assigned an issue, checks out code, makes edits, and submits a PR (that is accepted).
- Rough criteria: the AI acts with little oversight, performs similar (coding) work to entry-level coders at the time, the issue/task assignment is not *significantly* specialized for an AI (e.g. no full technical specs if the same wouldn't be given to a human coder) AI being used in this way in significant open source projects counts as "industry use".
- If there are technical demos of such AIs but none of them are actually being used question resolves as no.
- No requirement that it be a single model.
- A group of specialized models working together counts.
- If superhuman performance is not achieved by market end, resolves N/A.
