Transferring to Real-World Layouts: A Depth-aware Framework for Scene Adaptation

Authors: Mu Chen, Zhedong Zheng, Yi Yang

Published in ACM International Conference on Multimedia (ACM MM), 2024

Recommended citation: Mu Chen, Zhedong Zheng, Yi Yang, "Transferring to Real-World Layouts: A Depth-aware Framework for Scene Adaptation." ACM Multimedia, 2024.
Download PDF: https://zdzheng.xyz/files/2024/ACMMM24-Layout.pdf

Code is available at: https://github.com/chen742/DCF

Abstract: Scene segmentation via unsupervised domain adaptation (UDA) enables the transfer of knowledge acquired from source synthetic data to real-world target data, which largely reduces the need for manual pixel-level annotations in the target domain. To facilitate domain-invariant feature learning, existing methods typically mix data from both the source domain and target domain by simply copying and pasting pixels. Such vanilla methods are usually sub-optimal since they do not take into account how well the mixed layouts correspond to real-world scenarios. Real-world scenarios are with an inherent layout. The model suffers from confusion in predicting the target domain due to the unrealistic mixing. For instance, it is not reasonable to directly paste the near pedestrian pixels into the remote sky area. Based on such observation, we propose a depth-aware framework to explicitly leverage depth estimation to mix categories and facilitate two complementary tasks, i.e., segmentation and depth learning in an end-to-end manner. Besides, several public datasets do not provide depth annotation. Therefore, we leverage the off-the-shelf depth estimation network to obtain the pseudo depth. Extensive experiments show that our methods, even with pseudo depth, achieve competitive performance, i.e., 77.7 mIoU on GTA to Cityscapes and 69.3 mIoU on Synthia to Cityscapes.

@inproceedings{chen2024transferring,
author = "Chen, Mu and Zheng, Zhedong and Yang, Yi",
title = "Transferring to Real-World Layouts: A Depth-aware Framework for Scene Adaptation",
abstract = "Scene segmentation via unsupervised domain adaptation (UDA) enables the transfer of knowledge acquired from source synthetic data to real-world target data, which largely reduces the need for manual pixel-level annotations in the target domain. To facilitate domain-invariant feature learning, existing methods typically mix data from both the source domain and target domain by simply copying and pasting pixels. Such vanilla methods are usually sub-optimal since they do not take into account how well the mixed layouts correspond to real-world scenarios. Real-world scenarios are with an inherent layout. The model suffers from confusion in predicting the target domain due to the unrealistic mixing. For instance, it is not reasonable to directly paste the near pedestrian pixels into the remote sky area. Based on such observation, we propose a depth-aware framework to explicitly leverage depth estimation to mix categories and facilitate two complementary tasks, i.e., segmentation and depth learning in an end-to-end manner. Besides, several public datasets do not provide depth annotation. Therefore, we leverage the off-the-shelf depth estimation network to obtain the pseudo depth. Extensive experiments show that our methods, even with pseudo depth, achieve competitive performance, i.e., 77.7 mIoU on GTA to Cityscapes and 69.3 mIoU on Synthia to Cityscapes.",
code = "https://github.com/chen742/DCF",
url = "https://zdzheng.xyz/files/2024/ACMMM24-Layout.pdf",
booktitle = "ACM Multimedia",
funding = "SRG2024-00002-FST",
note = "Oral Presentation",
year = "2024" }