4 Decades of AI Compute

Article Breakdown

Why do we care about this trend?

Understanding AI progress is crucial for strategy and policymaking but quantifying it compactly is difficult. Interestingly, whether we like it or not, computation^◥ has been an annoyingly reasonable proxy for AI progress. The reason is evident, yet also somewhat unpleasant: incorporating domain knowledge or rules into AI systems helps in the short run but breakthrough progress has been achieved only by methods that scale with computation^◥. Therefore, AI computation tells a lot about AI progress. Even though this might change in the future, I sure wouldn't bet my money on that. As succinctly explained in the famous Bitter Lesson^⧉ by Professor Richard Sutton:

Main enablers of this trend are:

1. Increasing hardware capabilities
2. Decreasing hardware costs
3. Increasing data availability
4. Algorithmic developments

↑ Increasing hardware capabilities

Capabilities of computer hardware relevant to AI, particularly that of Graphics Processing Units (GPUs), continue to increase mostly due to advancements in transistor size and number of cores. This progress is expected to continue until 2031 ± 4 years^{◥ [1]}.

This trend enables us to run AI trainings in a few days or weeks which would have taken couple of million years just a decade back. Nowadays, it is not unusual to encounter training durations of a few months but generally it is not worth to run trainings longer than 15 months^{◥ [2]}.

↓ Decreasing hardware costs

Tech is deflationary^◥. The cost of genome sequencing was around $100,000,000 per human genome in 2001. This figure decreased to $100,000 around 2009 and nowadays it is less than $1000 ^[3]. It is crucial to understand that this trend affects almost all genomics research activities. In other words, it "trickles down"^◥.

Computer hardware is no different. That's why the cheapest smartwatch today has several orders of magnitude more memory than the most expensive computer in 1969 that landed us on the moon. Historically, number of computations 1$ can afford doubled approximately every 2.5 years in the field of AI ^[4].

↑ Increasing data availability

Collecting, storing, and sharing data is ridiculously easy compared to 1980s (thanks Internet!). It is dirt cheap too. Even though labeling data is still costly, it gets cheaper with certain tricks^◥. Furthermore, most of the state-of-the-art AI models utilize unlabeled data by self-supervision^◥ anyway.

↑ Algorithmic developments

Most influential algorithmic advancements regarding AI focus heavily on trying to find methods that can scale. In other words, (a) ability to accommodate more training data (preferably unlabeled) (b) to reach higher performance (c) without needing proportionally more manual work or domain knowledge. Regarding these 3 criteria, traditional machine learning methods fall short^◥ and general-purpose methods such as deep representation learning or reinforcement learning don't. And larger models with a lot of parameters tend to scale better due to certain theoretical reasons^{◥ [5, 6]}. This creates an incentive for increasing the compute. Even though there are significant advancements that focus on data- and compute-efficiency of AI training such as importance sampling^◥ or curriculum learning^◥, they focus on the "bang for your buck" compute and have little effect on the limits of total compute in an absolute sense.

These factors mentioned above compounds, which renders the growth to be exponential^⧉. Or in "tech bro" words: it scales. Based on the reasonable assumption that there is nothing venture capital loves more than something that scales, investments most likely further amplifies this growth. Can we quantify it then?

The goal here is to have an approximate, yet data-driven estimate of rate of progress.

Analysis

Two trends are analyzed:

1. Training Compute
2. Number of Model Parameters

Data consists of influential^◥ AI models of last 40 years. It is curated from different sources ^[7-11]. I have corrected some of the calculations based on more up-to-date info and added more models either by retrieving from corresponding publications or by manual calculations. While most of the models in the data are neural networks, I am using the term "AI" somewhat loosely here.

1. Training Compute

The typical metric denoting the total compute to train an AI system is Floating Point Operations (FLOPs) which is essentially the total number of additions (+) and multiplications (×). This metric always serves as a pragmatic reminder that whenever someone talks about AI, what they really mean is a computer program that adds and multiplies numbers.

Reminder: whenever someone talks about AI, what they really mean is a computer program that adds and multiplies numbers in a useful manner.

For the interested, there are 2 ways to estimate the computational resources spent to train an AI system:

2. Number of Model Parameters

We have more samples for the parameter trend analysis as it is reported relatively more frequently and there is no need to estimate anything.

Is this a forecast?

Not in a strict sense.

While some previous work present similar analyses as a definitive forecast, I don't buy that. First, the moment this kind of analysis is presented as a forecast, it creates an urge to "improve" the trend fit. Then one ends up fitting higher order curves, removing outliers, dividing the timeline into arbitrary sub-timelines (e.g. before deep learning vs. after deep learning) or sub-categories (e.g. language vs. vision models) in order to "explain" the data better. That's an invitation to overfitting and calculating confidence intervals^⧉ does not change that fact. Second, there is bias^◥ in the data. Finally, the amount of data at hand is really not that much to reliably forecast such a complex phenomenon. That's why I present this as a descriptive analysis rather than a predictive one.

In fact, to make this point I created 4 different partitions of the same data:

1. Before & After Deep Learning (DL): this is already controversial because when did DL start really? Previously, 2010 ^[9] as well as 2012 ^[12] has been suggested as the start for similar analysis. I have always considered Hinton's Deep Belief Net paper coupled with the contrastive divergence algorithm ^[13] in 2006 to be the start of Deep Learning era.
2. Domain-specific: Treating Games, Language, Speech, and Vision as separate categories
3. Before & After Commoditization of AI: I take TensorFlow^⧉'s initial release date (end of 2015) as the start of the AI commoditization era. Before that, you had to write your own GPU kernel in Theano^⧉ like a total loser but since then you just spend double amount of time trying to make TensorFlow work with a GPU like a cool nerd.
4. With & Without Self-supervised Learning

As depicted below, all of these partitions of the same data result in rather different growth rates (mind the log-scale!):

One might as well construct all sorts of argumentations for why a certain partition totally makes sense:

• Before & during 202X global chip shortage → slowdown of compute steroid supply
• Low & high interest rate eras → affects AI investments
• And most importantly, before & after NVIDIA CEO, Jensen Huang^⧉, got obsessed with wearing a black leather jacket → well, it improves the fit

The point is, over-focusing on a single-number growth rate sounds like "We (as individuals, companies, societies) are ready only if AI capabilities double every 12 months instead of every 9 months next decade". This does not seem to match with the reality.

Implications

Quantifying AI compute trends, even only approximately, is useful simply because it tells a lot about the overall progress of AI as a field. One might of course ask isn't it bizarre that such a low-level detail as "number of additions and multiplications" is a reasonable indicator of a high-level concept as "AI progress" in the long run? Yes, it is! Does it have to be like this, meaning that do AI leaps have to come from leveraging of AI compute? Absolutely not! Yet, has it been like this for decades? Definitely yes! It is high time we face the music.