The development of GPT-4’s successor, codenamed “Orion”, is going through a difficult period at OpenAI. According to an in-depth investigation by the Wall Street Journal, the project is accumulating delays and astronomical costs without achieving the expected performance. A worrying situation for Sam Altman’s company, which had promised a revolution in the field of artificial intelligence.
Huge investments for disappointing results
The development of GPT-5 has been mobilizing teams for more than 18 months with staggering costs. Each training session can reach $500 million in computing power alone, requiring tens of thousands of Nvidia chips in data centers. Despite at least two major training phases carried out, the results remain below expectations. If the model outperforms its predecessors, the improvements do not yet justify the staggering costs of its operation.
Microsoft, OpenAI’s main investor, hoped to see the new model around mid-2024. Sam Altman even told Stanford students that OpenAI could say “with a high degree of scientific certainty” that GPT-5 would be significantly smarter than the current model. The goal was to achieve a model capable of making scientific discoveries and performing complex everyday tasks with the precision of a doctor in certain areas.
A new strategy in the face of technical challenges
Faced with these obstacles, OpenAI had to completely rethink its approach:
- Recruiting experts to create original training data
- Use of synthetic data generated by its o1 and o3 models
- Development of an approach based on reasoning rather than the quantity of data
- Collaboration with theoretical physicists to improve the resolution of complex problems
The company also faces internal challenges, with more than two dozen key executives and researchers leaving in 2024, including co-founder Ilya Sutskever and CTO Mira Murati. The recent departure of Alec Radford, a respected researcher and lead author of several OpenAI scientific publications, illustrates this instability.
The uncertain future of large language models
The OpenAI situation raises fundamental questions about the future of LLMs (Large Language Models). Ilya Sutskever, former chief scientist at OpenAI, recently said that “the era of peak data is over,” likening data to an “AI fossil fuel” whose reserves are running out.
Apple researchers have notably criticized current reasoning models, saying they only “mimic” training data rather than truly solving new problems. Their tests revealed “catastrophic performance drops” when irrelevant details were added to problems.
Last Friday, Sam Altman announced the development of a new reasoning model smarter than anything the company has produced so far, although he did not mention GPT-5. The company has already confirmed that no model named GPT-5 will be released in 2024, leaving doubt about the future of this long-awaited new generation of AI.
