AMD presents new method of handing indirect lighting off to an image generation model: 'frame-by-frame' solution could be part of AMD's answer to DLSS 5 someday
by https://www.pcgamer.com/uk/author/jacob-ridley/ · PC Gamer
Share this article
0
Join the conversation
Follow us
Add us as a preferred source on Google
Newsletter
Subscribe to our newsletter
AMD promised neural rendering is coming to Radeon cards and consoles with FSR Diamond, and we're starting to see signs of that new approach in the works from the red team. The company's technical staff just published a paper on how it's using image generation to replace global illumination in real-time.
The paper outlines a technique for generating more realistic-looking lighting on a frame-by-frame basis. It's titled "Temporally stable generative illumination with a one-step diffusion model". Diffusion models create images by working through a noisy image, often over many steps; however, AMD's researchers are using one that produces an image in one denoising step, using the details and lighting baked into a scene to generate a new one. That's important for being able to run it quickly.
The idea is to produce something that looks like global illumination—indirect light bouncing off objects and illuminating stuff nearby—through image generation, rather than computationally intensive techniques like path tracing.
Latest Videos FromPC Gamer
Watch full video here:
"In our work, we try to solve global illumination as an image generation problem," says SungYe Kim in a video presentation on the paper. Kim is one of the authors. "Global illumination is what makes a rendered scene look real, and it is also the most expensive part of rendering."
The proposed method takes a frame with direct lighting—light straight from a source to an object with no further bounces—as an initial lighting 'hint' to the diffusion model, Stable Diffusion 2.1 Turbo.
The model itself is trained on a synthetic dataset of 31,000 frames of interior scenes rendered in Blender Cycles, which is Blender's ray tracing production renderer. Each has 1-30 lights and objects that move around. That way, "the model sees a wide range of lighting conditions." The training comprises running through this dataset 50 times on AMD Instinct MI210 GPUs.
Sparse lighting, radiance, material, and geometry 'signals' are used to "keep generation aligned to geometry and lighting". The frame that is fed into the diffusion model is processed in such a way as to force the model to focus on illumination, rather than any other details. That's going to be a pretty important step should this technique ever make it out into the wild west of videogames; the public reaction to these sorts of AI tools can be pretty gnarly.
The biggest gaming news, reviews and hardware deals
Keep up to date with the most important stories and the best deals, as picked by the PC Gamer team.
Contact me with news and offers from other Future brandsReceive email from us on behalf of our trusted partners or sponsors
The researchers also use what's called a temporal VAE, or TVAE, to improve consistency. This uses motion vectors to align the current frame with previous ones. It stores a history of previous outputs to ensure that the lighting doesn't flicker or go all weird between generations. That can become an issue with these sorts of models: run the same task over and over again and they'll produce slightly different results every time, and a human will readily notice that when it's running in real-time on a frame-by-frame basis.
The final result is composited with the original frame using only direct lighting, which likely softens any errors a touch. The blog post says the method is "suitable for real-time rendering use cases with frame-by-frame generation."
Compared to other generative models, "our method recovers the soft shadows and the bounce lighting, while RGB↔X hallucinates content that is not present in the scene and DiffusionRenderer distorts the geometry," Kim says in the presentation.
The model is said to run at 0.29 seconds per frame, which is roughly 3.45 frames per second, at a resolution of 512 x 512. That's on an RTX 3090, too, which is deemed pretty quick compared to those other named techniques yet shows there's a long way to go for gaming. The whole process requires around 8–9 GB of VRAM, too. Remember, this is just that global illumination step, not the entire pipeline for rendering a frame.
So, it's not just Nvidia looking into neural rendering as a new frontier for computer graphics, though we knew that already. Earlier this year, AMD's Jack Huynh announced a new suite of FSR technologies that it called FSR Diamond. This includes "next-gen neural rendering", which this new research falls under; alongside improved ML-based upscaling, ML-based multi-frame generation, ray regeneration, and path tracing.
How AMD sells these sorts of technologies to us gamers once they're ready is going to be something to watch. I'm sure the red team has been looking closely at the firestorm erupting from Nvidia's slapdash announcement of DLSS 5—and subsequent backpedalling and rephrasing in order to make it seem less egregious to gamers and more a tool for developers—and thinking, 'eesh, how do we avoid that while still making use of all this research?'
Rumours are already circulating about an application of neural lighting from AMD, such as from leaker Kepler_L2 on the Anandtech forums (via Overclock3D). Whether that refers in part to this, or something further along in the development process, we can't be sure, but it's clear that neural rendering isn't exclusively being worked on by Nvidia.
Best graphics card 2026
1. Best overall: AMD Radeon RX 9070
2. Best value: AMD Radeon RX 9060 XT 16 GB
3. Best budget: Nvidia RTX 5050
4. Best mid-range: AMD Radeon RX 9070 XT
5. Best high-end: Nvidia GeForce RTX 5090