Taking on CUDA with ROCm: 'One Step After Another'

Challenger AMD’s ability to take data center GPU share from market leader Nvidia will certainly depend on the success or failure of its AI software stack, ROCm. Taking on Nvidia’s CUDA and its enormous installed base, seen by many as the most significant moat of the most valuable company in the world, would seem like an absolutely gargantuan task.

“It’s like climbing a mountain—one step in front of another,” AMD’s VP AI software, Anush Elangovan, told EE Times in an exclusive interview. “Get your direction, lock in, and the rest will follow.”

Elangovan joined AMD through the acquisition of his startup Nod.ai two and a half years ago. The startup’s team of 30 had been building AI compilers for five or six years and was well-known as a major contributor to some of the most important AI repositories, including Shark, Torch.MLIR, and IREE. Nod had been working with hyperscalers, enterprises, and startups that were using its compiler-based automation software.

EE Times last spoke to AMD about ROCm just before the Nod acquisition. During that interview, senior VP of AI Vamsi Boppana said that ROCm is AMD’s number one priority, and that the company was aiming to unify AI stacks across AMD’s different hardware types (CPUs, GPUs, and FPGAs).

Since then, ROCm has had two and a half years of consistent investment, Elangovan said.

“ROCm at that time was a collection of parts,” he said. “It grew up providing [firmware] to ASICs—like, here’s a firmware piece, here’s a firmware piece, let’s tie them together.”

Today’s ROCm team is striving to emulate the Google Chrome team Elangovan was on before founding Nod.

“If you’re a Chrome user, you probably don’t know what version you’re using—you don’t care because it just works,” Elangovan said. “We’re already there with ROCm. In the next few releases, we’ll get to a six-week release cadence. We’ll get to a point where it just works, and it becomes invisible.”

Having caught up with the deficit it started with, the ROCm team needs to continue to move as quickly as possible, Elangovan said.

“We’re shipping software like a software company now,” he said. “And we are developing software like a software company. But we also are super tuned into what we have to do for the next transition, which is AI-assisted engineering.”

Portability

The unification of AI stacks Boppana talked about two and a half years ago has materialized under Elangovan and the Nod team. Internally called OneROCm, some pieces are still hardware-specific, but all acceleration happens through the ROCm stack, Elangovan said. This enables portability between different types of AMD hardware.

Portability between Nvidia and AMD GPUs might have been a big deal two years ago, but this burden has eased as people generally work higher up the stack today, Elangovan said. This transition is in part enabled by OpenAI’s open-source AI framework, Triton.

“Back in the day, it was about converting CUDA kernels to HIP kernels,” Elangovan said. “But increasingly, people went to Triton, which became the great equalizer of GPU programming. This great equalizer allowed you to write a Triton kernel and run it on AMD or Nvidia. And we invested heavily.”

One of Nod’s key engineers is leading the Triton effort at AMD, working closely with OpenAI. AMD has also invested heavily in MLIR, a compiler infrastructure for accelerators, and the former Nod team continues to maintain Torch.MLIR, which allows retargeting code for different hardware types.

Converting from CUDA code is no longer a common request, Elangovan said, since most inference customers are using vLLM or SGLang, running one of a few LLM models, and are focused purely on achieving the maximum number of tokens per second.

“Our team has Triton kernels, so if there’s some new attention algorithm we didn’t anticipate, Triton would be the catch-all, but in a day or two we’ll have [built] an optimised version for speed,” he said. “Once we show the deployability is the same, they can do pip install vLLM, and everything else is inside.”

HIPify is still available for HPC customers, but in general, Elangovan said, he relies on AI tools like Claude to write and validate new AMD kernels.

“Claude is better than HIPify because it has web search built in.”

Open source

ROCm is 100% open source, up to but not including firmware. Elangovan said that while being open source puts ROCm under scrutiny from the developer community, it means ROCm can move at the speed of community innovation, rather than the speed of AMD.

“This way, you can pick it up and do whatever you want with it, and innovate,” he said. “Everyone can tap in at whatever spot they want, whether that’s the compiler or the runtime, and they’re limited by their abilities, not by how fast AMD can partner with them.”

AMD has aggressive targets for building its developer community and it’s actively working on developer outreach at the moment. Crucially, ROCm now runs on AMD Strix Halo-equipped laptops out of the box, which AMD is hoping will help bring developers to the platform. The company typically releases ROCm updates for Windows laptops on the same day it releases the version for Instinct data center hardware, Elangovan said.

Developer community

Engaging directly with the developer community is “very, very important,” for Elangovan.

Describing himself as a reluctant X (Twitter) user, Elangovan said he joined the platform to help provide a ground-level view of what the company is working on to developers.

“People started to follow, and that became one of my side jobs,” he laughed.

Elangovan personally monitors X keywords, including “ROCm,” “ROCm sucks,” “AMD software not working,” and responds to every single one.

“Mostly it’s just [a matter of] education,” he said, adding that he personally provides advice and support to anonymous developers wherever possible.

Last year, AMD ran a GitHub poll for ROCm complaints and received more than 1,000 responses. Many were around supporting older hardware, which is today supported either by AMD or by the community, and one year on, all 1,000 complaints have been addressed, Elangovan said. AMD has a team going through GitHub complaints, but Elangovan continues to encourage developers to reach out on X where he’s always happy to listen.

“That has really changed the mood, from AMD developers being so annoyed that whatever driver wasn’t supported, to believing their efforts are appreciated,” he said. “Typically, when we fix it, that has multiplicative effects, because then people [think], I like how that was resolved…I know that AMD works, and I’m happy to give it a try.”

Elangovan is “super-excited” for the forthcoming MI450 (due to ship in the second half of 2026), but beyond that, the team is also starting to look at features for ROCm that are differentiated from CUDA, beyond being a robust platform developers can build on.

“We want ROCm to be a platform you can build on for the next 10 years,” he said. “You shouldn’t have to worry about what happens when new hardware comes.”

In the meantime, Elangovan is relying on his startup experience—years of ups and downs that resulted in Nod-developed compiler technologies being used by almost every accelerator company.

“We need to have conviction on our path, and then it’s one step in front of the other,” he said.