Thank you Tobias for your kind introduction.

What are the key ingredients that we need to do volume rendering? - First, we need a model or representation, essentially how do we define our volume. - Second, we need a way to obtain the volume, which can be procedural, artist generated or through some fluid simulation, for example. - And last, we need an algorithm that allows us to compute or approximate light transport in this volume, optionally a differentiable one, such that we can use it to also obtain the field from examples.

Focusing on physically based rendering, there are 3 common methods to describe volumes. - The first one is of course voxel grids. We are talking about dense grids, but also Open or nanovdb. They are simple to obtain and cheap to sample from. However, they scale very poorly with detail in memory. - On the other end, we have neural networks. They are very compact, scale better, but expensive to query and require more convolved ways to obtain the field. - A bit between them in terms of properties are primitives. They are explicit and compact, cheap to render or sample, but still require some gradient based optimization to regress the field.

Last year we presented don't splat your gaussians. By defining volumes as mixtures of primitives with a kernel inside, for example a Gaussian, we can do full path tracing with multiple scattering at a fraction of the memory cost of voxel grids. We are talking about going from gigabytes to a single megabyte. The takeaway for us was to stop searching for variance reducing techniques with complicated monte carlo methods using voxel grid volumes, to defining your field in such a way that integration becomes trivial. So now, the complexity lies entirely in how to ray trace these kernels as fast as possible.

Still, things that are simple to do with voxel grids, such as level of detail, are not so obvious with primitives.

Taking it from first principles, it should be easy. If we take a gaussian kernel, in the frequency domain, they are still Gaussians. Filtering in this domain can be understood as removing energy outside of an example band limit which I paint here as a circle.

But we cannot clip Gaussians like this and then go back to the spatial domain. What we can do is apply an analytic filter, in this case a gaussian. Convolution of a Gaussian with another Gaussian is still Gaussian, so it should work right?

In practice this is not so easy. We can apply these filters to produce lower levels of detail, but this makes primitives larger spatially. The increase in overlap, and the fact we still have the same number of gaussians, makes it very slow to render. We can of course think of ways to refit, prune, etc, but nothing that will not require extra memory, runtime compute or new regressions.

In computer vision, there are a number of Gaussian splatting papers proposing to simply heuristically prune the small primitives, as they hold higher frequencies. But as you can see, in a physically based renderer, this just destroys the integrity of the asset, as small gaussians also carry low frequencies. They span the entire spectral domain.

What we would like to have is something in the middle. We prune primitives, and by doing so we band limit the asset, obtaining cheap and continuous level of detail. Let's see how to do that.

What we want to do, is to basically move the energy of this Gaussian in the fourier domain to the sides, such that now it lives in specific frequency bands, ideal for filtering. Mathematically, we can do that applying a complex valued exponential. But how does this look like in the spatial domain. Well, we just need to look at Euler's equation. Which we can rewrite like this, leaving us with a single real valued cosine. With some small difference, this is very close to Gabor kernels, so we will call them like that for the rest of the talk. End of the math lecture.

A Gabor kernel is just the product of a Gaussian envelope with a harmonic carrier, in our case a cosine, wich we can also phase shift like here.

So now, pruning becomes a matter of removing kernels, which we can do during runtime essentially for free.

So this is Gabor fields. We compose volumes as a sum of a low pass asset, fitted with zero frequency Gabors (which is equivalent mathematically to Gaussians), plus a residual with arbitrary frequency gabor kernels.

This allows us to filter in a completely arbitrary manner, during runtime, at no cost, without precomputing pyramids or mipmaps, refitting the asset, etc. Just optimize once, and filter however you want. You can try this yourself in our released viewer.

On top, we propose an optimization approach that allows us to obtain better quality than Gaussians for asset regression, while rendering faster, and the mentioned LOD even for multiple scattering. On how we actually do this, derivations of closed form Gabor integrals and fourier transforms, I refer you to our paper.

We can use this for a number of applications, for example completely continuous foveated rendering.

Another interesting property of Gabors, is that they are not only selective in frequencies, but also in orientations. This means that many rays intersecting a gabor will see a null contribution from it, which allows us to preemptively render them to reduce runtime costs.

The analogy to pyramid representations would be moving from a laplacian, with gabors in each of this residual band, to a steerable pyramid, where we can filter also responses in specific orientations. The game here is as always, reduce the number of primitives intersected along each ray, which is the bottleneck of primitive volume rendering.

We can use this for example for efficient motion blur, since this is a 1D box filter that we can roughly approximate with a Gaussian. In specific orientations of this motion, Gabor kernels interfere destructively with themselves, reducing their contribution to the volume.

Modelling carefully this attenuation, we can quickly decide before tracing which gabors will not contribute, and remove them from the set, obtaining almost the same render but almost twice as fast.

My favorite is how we can connect this representation with variance reduction techniques in monte carlo rendering. Most of the density of this field is actually held in the low frequencies, the Gaussian low pass base. The gabor model residuals, but are marginal in terms of contribution. So how about we sample from these frequencies at different rates, depending on our needs? This is similar to control variate theory.

We can leverage this spectral decomposition in many different ways. For example, by always integrating gaussians, but randomly dropping some higher frequencies in the camera paths. But also, how about after the third bounce, we approximate light transport only using the gaussians, which are just 2% of the total primitives here? Multiple scattering is even faster And also, how about we choose a different strategy for secondary light rays, such as next event estimation? Here we just use the Gaussians for this. As you can see, we can pay a small price in bias, but accelerate rendering drastically.

For purely absorbing mediums, without scattering, this is actually still unbiased. remind you that we are still integrating all our kernels analytically, but choose to not intersect some of them.

Finally, we can connect our model with procedural generation. Artist have for many years layered natural looking noise on top of simple distributions to compose procedural assets, and we can do the same layering gabor noise over a gaussian base, to create things like this cloudscape

without any regression, optimization, and with assets ready to be path traced immediately.

There are a number of limitations to this yet, like of course performance when using very large numbers of primitives, but I wanted to highlight in particular that while it is currently slightly expensive to distill these assets through our gradient optimization, this is not an inherent limitation of the work, and in the future I envision we will connect this to wavelet transforms and feedforward point based reconstruction to obtain assets from examples immediately.

So my key takeways are - We propose using Gabor kernels as a volumetric medium representation - We develop the necessary math to allow physically based forward and inverse rendering - We achieve continuous level of detail by dropping kernels in runtime, which is almost free - And we can leverage the spectral properties of Gabor kernels for a quite wide range of applications, by understanding them from a pure Fourier perspective.

And that's all, we have released the code already in Mitsuba 3, and as a side note I will be graduating next year so if you are interested on this or my other works on neural scene reconstruction I would be happy to talk.