What's new with Gigamodels?

In each of their fields, Gigamodels establish the new state of the art. Such transitions occur more or less regularly in technology, and players generally adapt to new, more efficient approaches. What makes the transition to Gigamodels potentially different from others?

The race for size

One of the characteristics of Gigamodels is not only that they are large in size, but that more and more of them come out in bulk every year.

‍

‍

A small handful of players produce Gigamodels

As far as learning is concerned, the deep learning revolution has already led laboratories and research teams to equip themselves with GPUs(Graphical Processing Units) to launch the learning of the many parameters of deep neural networks. But learning a Gigamodel requires GPU farm sizes that only a few players in the world possess.

In fact, Gigamodels are almost all produced by a small handful of players: Google (Elmo, BERT, T5, Lambda, Imagen...), Microsoft+OpenAI (GPT2, GPT3, Dall-E, Whisper...), NVIDIA (Turing NLG, Megatron-Turing NLG with Microsoft, RIVA), Meta - ex-Facebook (XLM, Roberta, wav2vec...)

The emergence of specialized players

When it comes to using Gigamodels, whether for fine-tuning or efficient deployment in production, a number of players, such as HuggingFacea leading model distribution and tooling platform, as well as Microsoft with its ONNX format, and GPU manufacturer NVIDIA, with TensorRT, Triton, NEMO and RIVA..... Clearly, this know-how is key to the business models of these players, one of whom relies on revenues from model deployments in its Cloud, and the other on the sale of its GPUs.

The companies that produce Gigamodels, such as Google and Microsoft, generally also have a business model involving the sale or rental of processing capacity.

Impact on research and industry

What is the impact of these Gigamodels for research and industry in cognitive services and speech and language technologies?

It's probably too early to assess the precise impact of Gigamodels, but we can already point to the issues they raise and possible future developments.

Research: promise for poorly endowed languages; separation from learning?

Gigamodels hold out the promise of providing a better response to applications involving languages or domains with few or no written records, thanks to pre-trained models supplemented by supervised corpora of modest size, or directly thanks to self-supervised learning. But experiments are still needed to verify the effectiveness of this type of approach.

Another question is whether research will split between a few teams working on the design of the Gigamodels themselves, while the majority of others are interested in fine-tuning, producing lightweight models, or models with little data, and so on.

Industry: promise of speed; schism around data and the Cloud?

For the industry, the promise of Gigamodels is first and foremost that of lower barriers to entry and a reduced need for supervised learning data.

But Gigamodels also require specific infrastructures and environments for learning and production deployment, so we can see two types of user emerging in the industry: those who will rely on specialized players offering tool-based learning and deployment in their own cloud, and those who will develop their own capacity in terms of learning and production.

These different types of player can be represented in the diagram below, taking into account the size of the GPU infrastructure and the size of the data corpus being manipulated.

A schism is emerging between data and infrastructure, if we focus on the blue blocks. Whether or not this schism will be confirmed remains to be seen.

The development model of start-ups like HuggingFace seems to be banking on it. NVIDIA, which supplies HuggingFace as well as GAFAMs and specialized companies and players, seems ready to support other development models too, should the mole blocks that take over data and infrastructure become more important.

In conclusion: reservations, but a course of action

Having reached this conclusion, it is important to qualify the statement:

First of all, Gigamodels, as promising as they are, remain a recent phenomenon. They still largely coexist in the industry with hybrid models, from the previous generation, learned from data already collected and annotated, and which continue to guarantee high performance over their operating domain. The adoption and impact of Gigamodels on the industry is not yet a foregone conclusion , and will depend on the agility with which industry players seize on them and integrate them into business applications.

On the other hand, despite their growing prowess, the limitations of LLMs need to be taken into account: increasing the size of models does not mechanically resolve all the complexities of spoken communication, and each model has to find its place and operational relevance, which also largely depends on interactions with human users.

Finally, the legal issues surrounding the data used in these models cannot be ignored.

With all these nuances in mind, there's no denying that Gigamodels are revolutionizing the state of technology and opening up immense, unexplored application possibilities. If European players fail to seize this opportunity and leave the reins to the major US platforms that currently dominate production and deployment, they too will not have the chance to shape the new ecosystem currently being reconfigured.

The End!

‍

‍