The cost and sustainability of generative AI

AI is resource intensive for any platform, including public clouds. Most AI technology requires numerous inference calculations that add up to higher processor, network, and storage requirements, and higher energy bills, infrastructure costs, and carbon footprints.

The rise of generative AI systems, such as ChatGPT, has brought this issue back to the fore. Given the popularity of this technology and the likely massive expansion of its use by businesses, governments and the public, we could see the growth curve of energy consumption take a worrying arc.

AI has been viable since the 1970s, but initially it didn’t have much of a commercial impact, given the amount of resources required to make a full-fledged AI system work. I remember designing AI-enabled systems in my early 20s that would have required over $40 million in hardware, software, and data center space to make it work. Spoiler alert: that project and many other AI projects never saw a release date. The business cases just didn’t work.

The cloud changed all that. What was previously inaccessible is now cost-effective enough to be possible with public clouds. In fact, the rise of the cloud, as you may have guessed, was more or less aligned with the rise of AI in the last 10-15 years. I would say they are now closely coupled.

Cost and Sustainability of Cloud Resources

You don’t really need to do much research to predict what’s going to happen here. Demand for AI services will skyrocket, such as the generative AI systems now generating interest, as well as other AI and machine learning systems. This surge will be led by companies looking for an innovative edge, like smart supply chains, or even thousands of college students wanting a generative AI system to write their term papers.

Higher demand for AI means higher demand on the resources these AI systems use, such as public clouds and the services they provide. This demand will most likely be met by more data centers housing power-hungry servers and network equipment.

Public cloud providers are like any other utility resource provider and will increase prices as demand increases, just as we see household energy bills increase seasonally (also based on demand). As a result, we typically cut back on usage, running the AC at 74 degrees instead of 68 in the summer.

However, higher cloud computing costs may not have the same effect on businesses. Companies may find that these AI systems are not optional and are necessary to power certain critical business processes. In many cases, they may try to save money within the company, perhaps by reducing the number of employees to offset the cost of AI systems. It’s no secret that generative AI systems will soon displace many information workers.

What can be done?

If the demand for resources to run AI systems will lead to higher computing costs and carbon emissions, what can we do? The answer may lie in finding more efficient ways for AI to use resources like processing, networking, and storage.

Sampling from a pipeline, for example, can speed up deep learning by reducing the amount of data processed. Research done at MIT and IBM shows that you can reduce the resources required to run a neural network on large data sets with this approach. However, it also limits accuracy, which might be acceptable for some business use cases, but not all.

Another approach that is already in use in other technology spaces is in-memory computing. This architecture can speed up AI processing by not moving data in and out of memory. Instead, the AI ​​calculations are executed directly within the memory module, which speeds things up significantly.

Other approaches are being developed, such as changes to physical processors (using coprocessors for AI computations to speed things up) or next-generation computing models, such as quantum. You can expect many announcements from the larger public cloud providers about the technology that will be able to solve many of these problems.

What should you do?

The message here is not to prevent AI from getting a lower cloud computing bill or to save the planet. AI is a fundamental approach to computing that most companies can leverage to gain a great deal of value.

I advise you to start a new AI system development or AI enablement project with a clear understanding of the costs and sustainability impact, which are directly related. You’re going to have to make a cost/benefit choice, and this really goes back to the value you can bring to the business for the cost and risk required. Nothing new here.

I think a lot of this problem will be solved by innovation, whether it’s in memory or quantum computing or something we haven’t seen yet. Both AI technology providers and cloud computing providers are interested in making AI more cost-effective and green. That’s the good news.