December 26, 2024 - In a recent interview, Demis Hassabis, CEO of Google DeepMind, cautioned that the tech industry may face diminishing returns in artificial intelligence development due to a lack of available digital text. This observation follows discussions with 20 executives and researchers who believe that the majority of internet text has already been utilized.
Hassabis, who is set to accept a Nobel Prize for his contributions to AI, noted that the conventional method of enhancing large language models through vast amounts of data is nearing its limits. 'Everyone in the industry is seeing diminishing returns,' he stated.
Despite ongoing substantial investments in AI, including Databricks nearing $10 billion in funding, concerns about a slowdown are surfacing. Some industry leaders, such as OpenAI's Sam Altman and Nvidia's Jensen Huang, remain optimistic about continued progress.
The debate traces back to a 2020 paper by Jared Kaplan which established the 'Scaling Laws,' suggesting that AI systems improve with increased data intake. However, experts now warn that these laws may not hold indefinitely, with Ilya Sutskever emphasizing the need to adapt to the limited data available.
To counter this challenge, researchers are exploring new methodologies, including the use of 'synthetic data' generated by AI systems themselves. OpenAI's recent release of a system built on this principle demonstrates potential, although limitations remain in broader applications beyond mathematics and programming.
While Nvidia's Huang expressed confidence in ongoing demand for AI infrastructure, some major clients, like Meta, acknowledge the need to prepare for a potential plateau in AI advancement.