Method Details

Details for method 'Hard Pixel Mining for Depth Privileged Semantic Segmentation'


Method overview

name Hard Pixel Mining for Depth Privileged Semantic Segmentation
challenge pixel-level semantic labeling
details Semantic segmentation has achieved remarkable progress but remains challenging due to the complex scene, object occlusion, and so on. Some research works have attempted to use extra information such as a depth map to help RGB based semantic segmentation because the depth map could provide complementary geometric cues. However, due to the inaccessibility of depth sensors, depth information is usually unavailable for the test images. In this paper, we leverage only the depth of training images as the privileged information to mine the hard pixels in semantic segmentation, in which depth information is only available for training images but not available for test images. Specifically, we propose a novel Loss Weight Module, which outputs a loss weight map by employing two depth-related measurements of hard pixels: Depth Prediction Error and Depthaware Segmentation Error. The loss weight map is then applied to segmentation loss, with the goal of learning a more robust model by paying more attention to the hard pixels. Besides, we also explore a curriculum learning strategy based on the loss weight map. Meanwhile, to fully mine the hard pixels on different scales, we apply our loss weight module to multi-scale side outputs. Our hard pixels mining method achieves the state-of-the-art results on three benchmark datasets, and even outperforms the methods which need depth input during testing.
publication Hard Pixel Mining for Depth Privileged Semantic Segmentation
Zhangxuan Gu, Li Niu, Haohua Zhao, and Liqing Zhang
project page / code
used Cityscapes data fine annotations, coarse annotations, stereo
used external data ImageNet
runtime n/a
subsampling no
submission date July, 2020
previous submissions


Average results

Metric Value
IoU Classes 83.3791
iIoU Classes 65.2467
IoU Categories 92.3387
iIoU Categories 82.6109


Class results

Class IoU iIoU
road 98.8128 -
sidewalk 87.8212 -
building 94.2875 -
wall 65.5797 -
fence 65.1612 -
pole 72.8538 -
traffic light 79.4759 -
traffic sign 82.3188 -
vegetation 94.2532 -
terrain 74.4412 -
sky 96.0928 -
person 88.6146 73.9742
rider 75.8121 57.7537
car 96.5669 91.8871
truck 77.8893 49.4111
bus 93.1813 63.8062
train 88.793 60.1939
motorcycle 73.0497 56.3989
bicycle 79.1971 68.5485


Category results

Category IoU iIoU
flat 98.8274 -
nature 93.9424 -
object 78.193 -
sky 96.0928 -
construction 94.4359 -
human 88.6235 74.8795
vehicle 96.2561 90.3423



Download results as .csv file

Benchmark page