Self-Ensembling Depth Completion via Density-aware Consistency

Authors: Xuanmeng Zhang, Zhedong Zheng, Minyue Jiang, Xiaoqing Ye

Published in Pattern Recognition (PR), 2024

Recommended citation: Xuanmeng Zhang, Zhedong Zheng, Minyue Jiang, Xiaoqing Ye, "Self-Ensembling Depth Completion via Density-aware Consistency." Pattern Recognition, 2024. DOI: 10.1016/j.patcog.2024.110618
Download PDF: https://zdzheng.xyz/files/2024/PR_SEED.pdf

Abstract: Depth completion can predict a dense depth map by taking a sparse depth map and the aligned RGB image as input, but the acquisition of ground truth annotations is labor-intensive and non-scalable. Therefore, we resort to semi-supervised learn- ing, where we only need to annotate a few images and leverage massive unlabeled data without ground truth labels to facilitate model learning. In this paper, we propose SEED, a SElf-Ensembling Depth completion framework to enhance the generalization of the model on unlabeled data. Specifically, SEED contains a pair of the teacher and student models, which are given high-density and low-density sparse depth maps as input respectively. The main idea underpinning SEED is to enforce the density-aware consistency by encouraging consistent prediction across different-density input depth maps. One empirical challenge is that the pseudo- depth labels produced by the teacher model inevitably contain wrong depth values, which would mislead the convergence of the student model. To resist the noisy labels, we propose an automatic method to measure the reliability of the gener- ∗Zhedong Zheng is the corresponding author. (Email address: [email protected]) Preprint submitted to Pattern Recognition May 31, 2024 ated pseudo-depth labels adaptively. By leveraging the discrepancy of prediction distributions, we model the pixel-wise uncertainty map as the prediction variance and rectify the training process from noisy labels explicitly. To our knowledge, we are among the early semi-supervised attempts on the depth completion task. Ex- tensive experiments on both outdoor and indoor datasets demonstrate that SEED consistently improves the performance of the baseline model by a large margin and even is on par with several fully-supervised methods.

@article{zhang2024seed,
author = "Zhang, Xuanmeng and Zheng, Zhedong and Jiang, Minyue and Ye, Xiaoqing",
title = "Self-Ensembling Depth Completion via Density-aware Consistency",
abstract = "Depth completion can predict a dense depth map by taking a sparse depth map and the aligned RGB image as input, but the acquisition of ground truth annotations is labor-intensive and non-scalable. Therefore, we resort to semi-supervised learn- ing, where we only need to annotate a few images and leverage massive unlabeled data without ground truth labels to facilitate model learning. In this paper, we propose SEED, a SElf-Ensembling Depth completion framework to enhance the generalization of the model on unlabeled data. Specifically, SEED contains a pair of the teacher and student models, which are given high-density and low-density sparse depth maps as input respectively. The main idea underpinning SEED is to enforce the density-aware consistency by encouraging consistent prediction across different-density input depth maps. One empirical challenge is that the pseudo- depth labels produced by the teacher model inevitably contain wrong depth values, which would mislead the convergence of the student model. To resist the noisy labels, we propose an automatic method to measure the reliability of the gener- ∗Zhedong Zheng is the corresponding author. (Email address: [email protected]) Preprint submitted to Pattern Recognition May 31, 2024 ated pseudo-depth labels adaptively. By leveraging the discrepancy of prediction distributions, we model the pixel-wise uncertainty map as the prediction variance and rectify the training process from noisy labels explicitly. To our knowledge, we are among the early semi-supervised attempts on the depth completion task. Ex- tensive experiments on both outdoor and indoor datasets demonstrate that SEED consistently improves the performance of the baseline model by a large margin and even is on par with several fully-supervised methods.",
journal = "Pattern Recognition",
url = "https://zdzheng.xyz/files/2024/PR\_SEED.pdf",
doi = "10.1016/j.patcog.2024.110618",
year = "2024" }