Debates on Artificial General Intelligence as a Milestone

Yesterday during our MIT meeting we spent a good part of the afternoon discussing the implications of AI’s scaling laws – the idea that making models larger, training on more data, and using more compute yields better performance, but only when grown together. This finding fueled our current race to build ever-larger models.However, the scale-everything approach that seemed to be the solution may now be reaching its limits.Exponential cost & need for power means that each small gain demands exponentially more compute and energy meaning that this path alone may become unsustainable. Another factor is the quality plateau where better perplexity doesn’t equal true understanding. Even as models get bigger and excel at benchmarks, they still hallucinate information and fail basic logic. Despite the hype, pure scaling hasn’t produced artificial general intelligence(AGI) yet – disproving the mantra that “scale is all you need”. Big models can display emergent skills, but crucial capabilities like commonsense reasoning remain absent until now. The next model may consume all high-quality text data by 2026–2032, and training the next giant might cost around $100B some sources say. So the future of AI will be truly defined by scale + innovation – combining big models with new strategies: -Hybrid systems: Combining large neural networks with other AI approaches (symbolic reasoning, external and private knowledge, etc.) to overcome the limits of pure scaling. -Architectural breakthroughs: New model designs (multimodal, modular, sparse, etc.) that get more out of fewer parameters making AI more efficient instead of just bigger. -New training paradigms: Models that learn continuously or interactively (via reinforcement learning, human feedback, etc.) instead of relying on one-off training runs. In the next 3–5 years, expect a shift from brute-force growth to more efficient methods. AI leaders will prioritize optimized models and smarter infrastructure over sheer scale looking for the opportunity to enable true AGI. #ai #artificialintelligence #digital

When we get to AGI, it will have come slowly, not overnight. A NeurIPS Outstanding Paper award recipient, Are Emergent Abilities of Large Language Models a Mirage? (by Rylan Schaeffer, Brando Miranda and Sanmi Koyejo) studies emergent properties of LLMs, and concludes: "... emergent abilities appear due the researcher’s choice of metric rather than due to fundamental changes in model behavior with scale. Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous, predictable changes in model performance." Public perception goes through discontinuities when lots of people suddenly become aware of a technology -- maybe one that's been developing for a long time -- leading to a surprise. But growth in AI capabilities is more continuous than one might think. That's why I expect the path to AGI to be one involving numerous steps forward, leading to step-by-step improvements in how intelligent our systems are.

As we navigate the path toward artificial general intelligence (AGI), this insightful fireside chat between David Luan (Adept) and Bryan Catanzaro (NVIDIA) at #GTC helped understand the future of AI and the path toward AGI. Here are the key highlights: - General intelligence is multi-dimensional; we need different types of specialized #intelligence rather than a single generalized model. - We are reaching the limits of what large language models (#LLMs) can achieve by just scaling up on more text data and thus need to fine-tune these models for specific tasks and domains. - Multimodal models that understand images, audio, video, etc., alongside text are the next frontier, enabling capabilities like #predictive world models. - Data quality and provenance will become crucial, including access to proprietary #data like private conversations, to unlock new capabilities. Safeguarding private data is paramount. - Hardware and architectural innovations like sparsity will drive improved intelligence per FLOP for more efficient #AI systems. What do you think?

New column by me in Science: "Debates on the nature of artificial general intelligence"   https://lnkd.in/g7KApxac   -- I ask "What the heck is this AGI thing everyone is talking about?"   -- I explore how the concept of AGI has changed over the history of AI   -- I discuss how many people in AI have a quite different view of how intelligence works than those who study biological intelligence   I hope you will read it!