Hi, I’m Jorge Condor, and today I will be introducing our work on Neural Harmonic Textures or NHT for short.

In Novel View Synthesis, Gaussian Splatting is the method of choice. However, since each primitive has uniform color, complex appearance requires lots of primitives, which constitutes a substantial scalability problem.

We can see this clearly in 2D, where given the task for Gaussians to fit a planar texture, they are clearly limited by the entanglement between appearance and geometry.

In graphics, we solved this decades ago by introducing textures to our simpler geometry. This has also been done in a number of works in the context of Gaussian Splatting, but doing this explicitly complicates training, makes rendering slow and fails to scale upwards. We will talk in a second how our method solves these issues.

Our idea is to revisit Neural Fields. In novel view synthesis, we can separate between approaches using primitives, like gaussian splatting, and those that use Neural networks to model the scene, like NeRF.

Neural fields can model arbitrarily detailed scenes, but are slower to render and their spatial encoding still represents an scalability issue.

Work has already tried to go in both directions, either making NeRF more explicit, or introducing neural fields to primitives, but either speed or quality had to be sacrificied.

The naive way to texture a primitive is to store an explicit texture on it, weighted by its Gaussian. But even a tiny 16 by 16 texture means 256 parameters on every single primitive, which is far too heavy to store and render at scale.

The fist component is implicit texturing. Instead of storing a texture and fetching it during rasterization, we encapsulate Gaussians in a virtual scaffold and interpolate features on intersection points along virtual camera rays. This allows us to have very compact textures with effectively infinite resolution.

Secondly, we introduce harmonic textures. Instead of blending latent textures directly, we encode them with sinusoidal activations. Similarly to Fourier Analysis, this turns the problem from blending a signal, to blending the harmonics of the signal, providing much higher texturing capacity.

Finally, while NeRF or Gaussian splatting would evaluate appearance at each primitive and accumulate color directly, we accumulate these latent textures and decode them into color only in image space, in a single deferred shading pass with a tiny MLP.

And that’s it, the result is state of the art quality and very fast rendering, with just a few modifications over regular Gaussian splatting.

On average across standard benchmarks, we substantially improve quality for a small price in performance.

And this is across the board. Our method scales both upwards and downwards, and is particularly strong at small primitive counts, where texturing becomes crucial.

For example, here we have the Bonsai scene at 10 thousand primitives, where we can clearly see the texturing capacity of NHT.

Our method also interpolates well, as we can see in this video of the truck scene

Particularly in the windshield and in specular highlights in other scenes, NHT is capable of modelling high frequency texture to a higher degree than ever before.

We can also adapt our method to 2D image regression by fitting connected meshes of neural harmonic textures, where we also

Outperform neural field methods like instant NGP, with sharper images and reduced artifacts. Neural Harmonic Textures act here as a lagrangian alternative to hashgrid encoding

A key advantage of NHT is the fact that through the neural decoder, the dimensionality of the signal modelled is completely detached from the one stored at the Gaussian primitives, which enables real time reconstruction of much richer signals at a very compact size.

For example, we can achieve real time reconstruction of PBR stacks,

Which we can use for simple image based relighting, also real time.

Going even larger, we can even perform real time novel view synthesis of semantic features, like the 512 dimensional LSEG features.

Which we can use for example to perform open vocabulary masking from novel views.

To summarize, neural harmonic textures detach appearance from geometry in gaussian splatting, they do so in a very efficient manner, can be integrated with any primitive or method, allows for very compact interpolation of higher dimensional signals, and it is very scalable, maximizing quality and performance depending on your hardware needs by scaling the number of features, mlp size or number of gaussians used.

Thank you to all my collaborators in NVIDIA, and please check our project page and our implementation in gsplat.