NAN: Noise-Aware NeRFs for Burst-Denoising

Naama Pearl1, Tali Treibitz1, Simon Korman2,
1Dept. of Marine Technologies,
School of Marine Sciences
University of Haifa, Israel 2Dept. of Computer Science,
University of Haifa, Israel

CVPR 2022

Paper Video Code

Abstract

Burst denoising is now more relevant than ever, as computational photography helps overcome sensitivity issues inherent in mobile phones and small cameras. A major challenge in burst-denoising is in coping with pixel misalignment, which was so far handled with rather simplistic assumptions of simple motion, or the ability to align in pre-processing. Such assumptions are not realistic in the presence of large motion and high levels of noise. We show that Neural Radiance Fields (NeRFs), originally suggested for physics-based novel-view rendering, can serve as a powerful framework for burst denoising. NeRFs have an inherent capability of handling noise as they integrate information from multiple images, but they are limited in doing so, mainly since they build on pixel-wise operations which are suitable to ideal imaging conditions. Our approach, termed NAN, leverages inter-view and spatial information in NeRFs to better deal with noise. It achieves state-of-the-art results in burst denoising and is especially successful in coping with large movement and occlusions, under very high levels of noise. With the rapid advances in accelerating NeRFs, it could provide a powerful platform for denoising in challenging environments.

Denoising of Low-Light Scene

  Download real world raw data

We took the images with a Google Pixel 4.

We used only 8 images per burst, due to BPN's architectural limitation.
The data contains more images per scene.


You can cycle through the images by pressing key 1 to 8 .

1. Dark frame

2. Pre-processed

3. Burst Average

4. HDR+ [6]*

5. BPN [1]

6. Deep-Rep [2]

7. IBRNet-N

8. NAN

*We use the implementation from [7]

Denoising of Synthetic Noise (Real-World Displacements)

LLFF-N Dataset - Gain 20

This is the validation set that was used in IBRNet [3], which is build on scenes from LLFF [4] and NeRF [5].
We describe in the paper the way we add noise to the clean images.

  Download results


You can cycle through the images by pressing key 1 to 7 .

1. Noisy frame

2. GT frame

3. Burst Average

4. BPN [1]

5. Deep-Rep [2]

6. IBRNet-N

7. NAN



Quantitative Evaluation



Comparison of Kernel Size and Bilateral in Post Processing

Novel View Synthesis Under Noise Conditions

LLFF-N Dataset - Gain 20

In the novel view task, the target frame is not available to the network, as opposed to the denoising task.


You can cycle through the images by pressing key 1 to 6 .

1. Noisy frame

2. GT frame

3. Burst Average

4. IBRNet [3]

5. IBRNet-N

6. NAN

BibTeX


      @inproceedings{pearl2022noiseaware,
        title={NAN: Noise-Aware NeRFs for Burst-Denoising},
        author={Pearl, Naama and Treibitz, Tali and Korman, Simon},
        booktitle=CVPR,
        year={2022}
      }

References

  1. Xia, Zhihao, et al. "Basis prediction networks for effective burst denoising with large kernels." In Proc. IEEE CVPR, 2020.
  2. Bhat, Goutam, et al. "Deep reparametrization of multi-frame super-resolution and denoising." In Proc. IEEE ICCV, 2021.
  3. Wang, Qianqian, et al. "Ibrnet: Learning multi-view image-based rendering." In Proc. IEEE CVPR, 2021.
  4. Mildenhall, Ben, et al. "Local light field fusion: Practical view synthesis with prescriptive sampling guidelines." ACM Transactions on Graphics, 2019.
  5. Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." In ECCV, 2020.
  6. Hasinoff, Samuel W., et al. "Burst photography for high dynamic range and low-light imaging on mobile cameras." ACM Transactions on Graphics, 2016.
  7. Monod, Antoine, et al. "An Analysis and Implementation of the HDR+ Burst Denoising Method." Image Processing On Line, 2021.