In 2029, will any AI be able to construct "reasonably" bug-free code of >= 10k LOC from a natural language specification? (Gary Marcus benchmark #4)
IAOne-OffTecnologia3a
Manifold MarketsSenza KYCDati di risoluzione verificatiBen calibratoPrevisione della comunità attualeSì 92.2%
Previsori
116
Tipo di domanda
binary
Metodologia
Play-money forecasting platform
Tipo di fonte
Previsione
Dati di mercato
Aggiornato 1 ora fa
In ritardo16 set 22, 20:43 UTC1 gen 30, 08:00 UTC
Trend
Regole
The fourth question from this post: https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things
- The full text is: "In 2029, AI will not be able to reliably construct bug-free code of more than 10,000 lines from natural language specification or by interactions with a non-expert user. [Gluing together code from existing libraries doesn’t count.]"
- Judgment will be by me, not Gary Marcus.
- Ambiguous whether this means start or end of 2029, so I have set it for the end.
- For this question I am not using the exact text of the question, because I think "bug-free" is 1. silly 2. untestable.
- I will instead accept if it produces code of >=10k LOC with <= the number of bugs in an implementation by a human (many small bugs for some significant bugs will unfortunately be down to my subjective impression of whether it's "better")
