Makoto Okabe[Japanese]
Interactive Video Completion
Makoto Okabe 1 Keita Noda 1 Yoshinori Dobashi 2 Ken Anjyo 3,4
1 Shizuoka University 2 Hokkaido University
3 OLM Digital, Inc. 4 CMIC, Victoria University of Wellington

It took 2.5 minutes in total to remove the bike from the 80-frame video with a resolution of 854x480 pixels. The top row shows the user-drawn masks and the buttom row shows the frames of the completed video.

 Abstract

We propose an interactive video completion method aiming for practical use in a digital production workplace. The results of earlier automatic solutions often require considerable amount of manual modifications to make them usable in practice. To reduce such a laborious task, our method offers an efficient editing tool. Our iterative algorithm estimates the flow fields and colors in space-time holes in the video. As in earlier approaches, our algorithm uses an L1 data term to estimate flow fields. However, we employ a novel L2 data term to estimate temporally coherent color transitions. Our GPU implementation enables the user to interactively complete a video by drawing holes and immediately removes objects from the video. In addition, our method successfully interpolates sparse modifications initialized by the designer. According to our subjective evaluation, the videos completed with our method look significantly better than those with other state-of-the-art approaches.

 Paper

IEEE Computer Graphics and Applications (CG&A) - preprint (PDF, 4.6MB)
 Supplementary materials
Results and comparisons (website)
Results and comparisons (ZIP, 138MB)

Supplementary video (MP4, 193MB)
 Acknowledgements

We would like to thank the anonymous reviewers for their insightful and constructive comments. Many thanks also go to So Hashimoto, and Ayumi Kimura for discussions and encouragements. This work was partially supported by JSPS KAKENHI Grant Numbers JP15H05924 and JP18K11352.

 Citation

Makoto Okabe, Keita Noda, Yoshinori Dobashi, Ken Anjyo, "Interactive Video Completion", to appear at IEEE Computer Graphics and Applications (CG&A).

 BibTeX citation
@article{ONDA19,
  author = {Makoto Okabe and Keita Noda and Yoshinori Dobashi and Ken Anjyo},
  title = {Interactive Video Completion},
  journal = {IEEE Computer Graphics and Applications (CG&A)},
  volume = {?},
  number = {?},
  year = {2019},
  pages = {1--9},
  doi = {10.1109/MCG.2019.2950176},
}