Hello everyone!
My name is Merlin and I will present our work on unbiased inverse volume rendering.
This is joint work with Thomas Müller, Alexander Keller and Wenzel Jakob.
3D volumes are a simple yet powerful scene representation for inverse rendering.
For example, Neural Radiance Fields are a hugely successful volume-based representation.
NeRFs are essentially volumes that can absorb and emit light.
But a key limitation of such emissive volumes is that all information about the scene's
lighting ends up baked into the volume.
In this talk, we focus on physically based volumes, which can absorb but also scatter
light.
These are inherently relightable, but they're also harder to optimize.
In fact, did you know that taking derivatives w.r.t. to a volumetric path tracer produces
biased and high-variance gradients?!
On the right, we show what happens when trying to reconstruct a volume with stochastic gradient
descent and those noisy gradients.
Pretty terrible!
Figuring out a solution to this problem is worth it though,
because inverse volume rendering has many important applications, from relightable scene
capture to climate science and medical imaging.
Of course, we're not the first to look at differentiable volume rendering.
There has been strong interest on volume-based inverse rendering since at least 2014.
But to the best of our knowledge, all existing methods are
susceptible to the issue we will discuss in this talk, even the most recent method based
on delta tracking.
So let's jump in and find out what caused this inverse volume rendering to go so wrong.
We start with the volume rendering equation, and make a couple of assumptions:
we use a standard exponential transmittance model, and focus on non-emissive volumes.
We now have this simplified volume rendering equation, which tells us how much light reaches
a point x from direction omega, as a function of the medium's density sigma_t
and its albedo alpha.
T is the transmittance function, which accounts for how much light is absorbed by the volume
between two points.
L_e tells us how much light is emitted at the nearest surface,
and finally L_s represents all radiance scattered from somewhere else in the volume,
or the nearest surface, and reflected in our direction.
It's a recursive integral.
Let's look at the contribution of each term in more detail when rendering a scattering
volume such as this little sand storm.
If we zoom-in to such a volume, it is modelled as a collection of many small
and uniformly distributed particles that reflect or absorb light.
The second term models light coming from the nearest surface or emitter.
It gets attenuated due to the medium's absorption.
And the integral term models in-scattered radiance, which is radiance reflected from
somewhere else and into our direction.
The higher the medium density sigma_t at a given point t, the more light is in-scattered.
But then, some of that light also gets absorbed by the time it reaches us.
Of course, the volume rendering integral describes the aggregate statistics of many such particles,
and one key component of solving the integral is sampling so-called free-flight distances.
That is to say, how long it takes a typical ray, to hit a particle.
Looking at the volume rendering equation, sampling these free-flight distances proportional
to T times sigma_t looks like a really good choice.
Whether we're using delta tracking or ray marching, indeed this
is the density that ends up being used.
As a result, we'll place more samples where there are more
particles, very few samples where sigma_t is low,
and absolutely no samples in regions that have zero density.
And in this context of forward volume rendering, this is all perfectly fine!
So what's wrong then?
Well, in forward rendering we care about the parts of the volume that affect how much light
reaches us.
If the volume is empty, light goes straight through and we don't need to spend any samples
there.
But here we're interested in inverse volume rendering.
Even though a region is currently empty, it could still be required to add density in
order to recover the correct volume.
That's why it's important to sample all regions of space that are visible.
Not just those where a volume already exists.
The quantity we have to compute is not how much light there is, but rather how much it
is changing with respect to the volume's parameters.
So we differentiate the volume rendering equation with respect to those parameters.
We end up with a few terms: First, the transmittance gradients account
for the attenuation increasing with the density of the volume.
There are also two terms that account for changes later along the light path.
But most relevant to us are the in-scattering gradients, which indicate that, as the volume's
density or albedo increase, the amount of in-scattered radiance increases as well.
So what's the issue with those in-scattering gradients?
Well, if we keep the same sampling density we were using for the primal rendering,
the one over sigma_t factor no longer cancels out!
For samples placed in high-density regions that works out fine.
But when the density is very small, the 1 over sigma_t factor explodes, leading
to extreme variance.
Perhaps even worse, absolutely no samples are placed in empty regions.
It may be necessary to add density to those regions, but because no samples are placed
there, the in-scattering gradients will be completely missing.
This leads to bias.
The bias and extreme variance are what lead to these terrible inverse rendering
results.
Note that, this will happen whether you use delta tracking or ray marching, because they're
all sampling from the same underlying pdf.
Okay, so how do we fix the issue?
We discuss a few different options in the paper, but the best solution is simply
to use a better-suited sampling density.
The transmittance term in the pdf cancels out with the integrand, and we are left with
a much nicer-behaved sample contribution.
Of course, the next question is: how to even sample proportional to the transmittance instead
of the usual extinction-weighted transmittance?
This is where we introduce our method, Differential Ratio Tracking.
To achieve our objective of sampling proportional to transmittance only, we rely on the transmittance
estimators proposed by Novák and colleagues.
Each of these algorithms can estimate the transmittance between two points by stepping
through the medium and querying its density.
What we'll see is that they can easily be repurposed to sample a free-flight distance
proportional to transmittance.
In a heterogeneous medium, delta tracking works by repeatedly sampling steps from a
homogenized medium.
At each step, it randomly decides whether a real or virtual particle was encountered.
We can actually build a binary estimate of transmittance from this procedure.
The transmittance starts at 1 and remains constant with null interactions.
But it falls to zero when a real interaction is sampled.
As we repeat this procedure, averaging all of the individual trackings
converges to the true transmittance function.
What we show in the paper is that we can obtain a valid sample proportional to the true transmittance
function by simply sampling from any of the binary trackings.
And of course, sampling from a binary function is easy: we just need one random number to
resample uniformly in the nonzero part of the function.
One detail is that the transmittance function is not normalized.
But, conveniently, the area under our binary function, which is trivial to compute, is
also a valid estimator of the area under the true transmittance curve!
And so we achieve our goal by returning the resampled distance t_2,
which together with its normalization weight, is distributed proportional to the transmittance
function.
Of course, delta tracking is a pretty crude transmittance estimator.
Ratio tracking makes much better use of the available information and builds piecewise-constant
trackings instead of binary ones.
Luckily, it's not much more difficult to sample from this piecewise-constant function than
the binary one from delta tracking.
We just pick one of the constant segment according to its
relative area, and resample a distance along that segment.
We get an even better estimate of the area under the curve,
and return a weighted sample just as before.
Finally, the same idea can be applied to residual ratio tracking, which is covered in the paper.
One potentially problematic aspect of what I've just described is that we don't
know in advance how many of those segments there will be.
The number of segments grows with the density of the medium.
And we certainly don't want to have to store all of the segments during traversal, and
then only later sample from them.
Especially on the GPU, this kind of unbounded memory usage could be problematic.
Instead, we use reservoir sampling to pick a segment on the fly, as we are stepping
through the medium.
The first segment is always placed into the reservoir.
But at every subsequent step, we stochastically decide whether to replace the contents of
the reservoir with the newly encountered segment.
The probability is computed from the area under the segment relative to the sum of areas
seen so far.
Once we are done traversing the medium, our reservoir contains a valid sampled segment.
We just have to sample uniformly on it as before.
And as it turns out, the normalization weight we need to return together with our sample
also gets computed automatically as part of the
reservoir sampling procedure.
More details are in the paper about the correctness of this sampling technique,
how to combine it with with standard free-flight sampling using MIS, and finally
how to use it in the context of radiative backpropagation or path replay backpropagation
while preserving a complexity linear in the path length.
Now, let's have a look at some results.
We'll be evaluating our new sampling technique in a classic inverse rendering setting.
The object we're trying to reconstruct is lit by a known environment map.
We get observations from different viewpoints, and we know the camera positions.
So as you can tell it's not a full-blown reconstruction pipeline from real-life data.
But the goal here is to measure the effect of eliminating the bias and high variance
of gradients produced by the existing sampling technique.
As we saw before, if you try to optimize a volume with SGD to represent this object,
the extreme variance in the estimated gradients will prevent any meaningful convergence.
If we go to parameter space and inspect those gradients, the source of the issue is clearly
visible.
The false color image shows the density gradient estimated using the existing sampling technique.
The striped areas correspond to places where the density sigma_t is zero, which is biased
because the in-scattering gradients are missing, as we saw before.
At the bottom, we see the standard deviation of those gradients.
There are huge spikes in locations where the density is almost, but not quite zero.
They are due to the 1 over sigma_t term we've seen previously.
Using our method, both the bias and variance issues are eliminated.
The maximum standard deviation of the gradients is orders of magnitude lower with our method.
But this is not the first paper that tries to optimize physically based volumes, and
of course others didn't report such catastrophic failures.
Why is that?
It turns out that the choice of optimizer makes a huge difference in this context.
If we use a variance-aware optimizer such as Adam instead of SGD, the optimization works
significantly better.
Adam reacts to gradient outliers by drastically reducing the local step size for those specific
parameters.
Because gradient outliers occur only where the volume density is already close to zero,
it's not such a big problem if the step size is reduced at those locations.
So it seems there is a lucky coincidence that makes it possible for Adam to hide the issue
almost entirely.
It's probably why the problem was not formally identified until now.
Still, we can notice haze-like artifacts remaining in regions that should be completely empty.
Our method eliminates most of the haze artifacts.
In the paper, we also propose a new initialization scheme
that builds on a NeRF-style emissive volume, which significantly improves reconstruction
quality.
The effect is particularly visible in those error maps or by flipping back and forth between
the images.
Let's look at this sand storm volume again.
As long as we use Adam, the reconstruction goes pretty well.
The difference between the baseline and our result looks pretty subtle when we put the
results side by side.
But if we look at the recovered volume density parameters,
it's very clear in the baseline that all regions where the density should be zero,
are very noisy instead.
That's where the haze artifacts show up.
The artifacts become clear when we flip back-and-forth between the two
results.
Here's a more complex object that we try to approximate as a volume.
Here again, using the existing method, regions that should
be empty end up incorrectly filled with noisy density.
The haze artifacts are apparent when flipping back and forth.
One last thing I want to emphasize is that unlike NeRF-style emissive volumes, our method
produces physically based volumes that are natively relightable and editable.
The emissive volume looks good in its original lighting, but
if we change the lighting, it does not react at all.
The lighting is essentially baked in.
Whereas with our method, the volume correctly reacts to the new lighting.
Here's another example with a solid object.
The emissive volume's appearance is way off in the new lighting conditions.
We're reaching the end of the talk.
To recap, we've proposed a new sampling technique to estimate gradients
in the context of physically based inverse volume rendering.
Our method efficiently estimates unbiased and low-variance gradients.
In the paper, we also propose a new initialization scheme that builds on NeRF-style emissive
volumes to avoid local minima and significantly improve reconstruction quality.
Please feel free to visit the project page where you can find the paper, an interactive
results viewer, as well as our implementation.
Thank you for watching!