Home / Blog / GPT-3 is getting old. Gopher was the new one, till yesterday when GLaM came up

GPT-3 is getting old. Gopher was the new one, till yesterday when GLaM came up

Roberto Saracco December 23, 2021 Blog 303 Views

The architecture of GLaM where each input token is dynamically routed to two selected expert networks out of 64 for prediction. Image credit: Google

Well, I just had the time to be amazed by GPT-3 than DeepMind, the AlphaBet’ AI research company released Gopher, a language processor that they claim has the comprehension capability of a high school student. I didn’t have time to digest this spectacular progress that Google released GLaM, supposedly an even better language processor (in certain application domains).

Reading about these two releases got me flabbergasted  by the huge numbers associated with them:

  • Gopher is based on 280 billion! parameters transformer language model (GPT-3 is based on 175 billion) (that is when processing a language, creating a sentence, providing an answer Gopher has access to 280 billion chunks of information)
  • GLaM is based on 1,200 billion parameters structured in 26 domains of expertise that can be merged on demand, however when processing a sentence it “only” refers to some 97 billion of then (8%). This makes GLaM more energy “conscious” than Gopher (remember, Gopher is the output of a research endeavour aiming at increasing the level of comprehension of what it is asked).

The evolution in the area of Natural Language Processing is exponential and the results achieved are impressive. Claiming that Gopher has the comprehension of a high school student means that it has achieved a level that is higher than the average population! (AGI – Artificial General Intelligence). GLaM, iin the other hand, by focussing (being trained) on more focussed domain can be more efficient yet delivering, in that domain, the same level of comprehension.

Why all this effort on Natural Language? Well, whatever we do is mediated by natural language and achieving a comprehension “au pair” of humans in natural language is pretty close to achieve an equivalent level of intelligence. Yes, mathematicians and physicists may not use natural language to communicate (and even to “think” about the world), artists use different forms of expressions (like music, paintings…) so having Gopher is not actually a mirror of human intelligence but, we have to admit, is frighteningly close!

Tags

About Roberto Saracco

Roberto Saracco fell in love with technology and its implications long time ago. His background is in math and computer science. Until April 2017 he led the EIT Digital Italian Node and then was head of the Industrial Doctoral School of EIT Digital up to September 2018. Previously, up to December 2011 he was the Director of the Telecom Italia Future Centre in Venice, looking at the interplay of technology evolution, economics and society. At the turn of the century he led a World Bank-Infodev project to stimulate entrepreneurship in Latin America. He is a senior member of IEEE where he leads the New Initiative Committee and co-chairs the Digital Reality Initiative. He is a member of the IEEE in 2050 Ad Hoc Committee. He teaches a Master course on Technology Forecasting and Market impact at the University of Trento. He has published over 100 papers in journals and magazines and 14 books.

Home | Contact & Support | Accessibility | Nondiscrimination Policy | IEEE Ethics Reporting | IEEE Privacy Policy | Terms
© Copyright 2022 IEEE – All rights reserved.
A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.