Haochen Zhang

Haochen Zhang

Ph.D. student in ECE at UCSD. Master degree received from USTC in 2020.

Two-Stream Action Recognition-Oriented Video Super-Resolution

Published in International Conference on Computer Vision (ICCV), 2019

In this paper, we study the video super-resolution (SR) problem for facilitating video analytics tasks, e.g. action recognition, instead of for visual quality. Tailored for two-stream action recognition networks, we propose two video SR methods for the spatial and temporal streams respectively.

The pipeline of performing video SR prior to action recognition for LR video. Note that our work focuses on the SR network, we directly adopt the well-trained two-stream action recognition network without any tuning.
The pipeline of performing video SR prior to action recognition for LR video. Note that our work focuses on the SR network, we directly adopt the well-trained two-stream action recognition network without any tuning.

On the one hand, we observe that regions with action are more important to recognition, and we propose an optical-flow guided weighted mean-squared-error loss for spatial-oriented SR network to emphasize the reconstruction of moving objects. On the other hand, we observe that existing video SR methods incur temporal discontinuity between frames, which also worsens the recognition accuracy, and we propose a siamese network for our temporal-oriented SR training that emphasizes the temporal continuity between consecutive frames.

We jointly consider two consecutive frames and design four loss terms to ensure the quality of individual frames as well as the temporal consistency between them.
We jointly consider two consecutive frames and design four loss terms to ensure the quality of individual frames as well as the temporal consistency between them.

Paper | Slides | Poster

Citation:

@inproceedings{zhang2019two,
  title={Two-stream action recognition-oriented video super-resolution},
  author={Zhang, Haochen and Liu, Dong and Xiong, Zhiwei},
  booktitle={IEEE/CVF International Conference on Computer Vision (ICCV)},
  pages={8799--8808},
  year={2019}
}