In 2029, will any AI be able to construct "reasonably" bug-free code of >= 10k LOC from a natural language specification? (Gary Marcus benchmark #4)
IAOne-OffTecnología3a
Manifold MarketsSin KYCDatos de resolución verificadosBien calibradoPronóstico comunitario actualSí 92.2%
Pronosticadores
116
Tipo de pregunta
binary
Metodología
Play-money forecasting platform
Tipo de fuente
Pronóstico
Datos de mercado
Actualizado hace 13 horas
Desactualizado16 sept 22, 20:43 UTC1 ene 30, 8:00 UTC
Tendencias
Reglas
The fourth question from this post: https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things
- The full text is: "In 2029, AI will not be able to reliably construct bug-free code of more than 10,000 lines from natural language specification or by interactions with a non-expert user. [Gluing together code from existing libraries doesn’t count.]"
- Judgment will be by me, not Gary Marcus.
- Ambiguous whether this means start or end of 2029, so I have set it for the end.
- For this question I am not using the exact text of the question, because I think "bug-free" is 1. silly 2. untestable.
- I will instead accept if it produces code of >=10k LOC with <= the number of bugs in an implementation by a human (many small bugs for some significant bugs will unfortunately be down to my subjective impression of whether it's "better")
