Current location: Home -> Research -> Interactive Multiview Video


Interactive Multiview Video and Parallelism over Multi-core

Interactive Multiview Video:

  The next generation multimedia applications will be interactive and realistic based on network. In one hand, people want to be more active and participant to meet the requirement of personality, in the other hand, they hope to get realistic experience while viewing the scene or interacting with other people. This project is proposed to resolve the represent, interaction, codec and transmission of 3D interactive video, and implement the interactive network multimedia application in the near future with the stereo video and multi-view video technology putting into practicability.

Objects:
  Viewpoint selecting and multi-view 3D image synthesizing
  High efficiency multi-view video codec algorithms
  Streaming technology for Multi-view video
  A prototype of multi-view video streaming system

Parallelism over Multi-core:

  By using the Intel multicore architectures and the emerging multiview video coding standard, we introduce a framework for performing analysis, simulation, and evaluation of heuristics scheduling algorithms for implementing computationally intensive algorithms on multicore processors. The framework allows for accurate and quantitative characterization of the performance of dynamic scheduling algorithms for multimedia applications on different multicore processors without actual implementation of the scheduling algorithm and application on the actual platform. Experimental results demonstrate the effectiveness and scalability of our framework.

Objects:
  Model the dynamic nature of the dependencies between tasks at different granularities
  Come up with different heuristics for designing efficient scheduling algorithms
  Predicting the performance of scheduling algorithms on multicore platforms

Content of Interactive Multiview Video

  • Multiview video synthesis

    Motion vector based video objects segmentation and tracking
    Feature based incomplete 3D structure video synthesizing
    Layered depth images based multiview video rendering



    Semi-immersive MultiView display environment


  • High efficiency Multiview Video Coding

    Disparity based multiview video coding



    View dependent multiview video predicting codec


  • Interactive multiview video streaming





    Interactive Functions
     View switch
     Time frozen moment
     Time continuous sweep

Content of Interactive Multiview Video

  • DAG-based framework

    The DAG-based framework consists of three modules: the encoding module, the heuristics scheduling module and the multicore module.

    Encoder Module: Video encoding algorithm is abstracted as a hierarchical multisource multisink DAG extended over the time axis.

    Heuristic Scheduling Module: The heuristics scheduling module synthesizes the states of encoder graph and multicore processor, and makes task scheduling decisions.

    Multicore Module: The module records and updates the states of the multicore processor, and calculates the set V(i) of tasks which have been finished on the ith step.



    Structure diagram of framework.

  • Application to MVC on Symmetric Multicore Architectures



    GOP scheduling in MVC.


    Working flowchart of framework.
  • Speedup ratio



    Speedup ratio as a function of the number of concurrent GOPs, estimated by the proposed framework.

Projects with sponsor:

国家高科技计划863项目: 面向交互应用的多视点视频处理技术研究 No.2002AA119040
国家自然科学基金:交互多视点视频编码技术研究 No.60503063
下一代互联网(CNGI)应用示范项目:基于IPv6的高性能视频传输和共享虚拟现实-多视点交互视频

People:

Professor Shiqiang Yang,Lingfeng Sun and Pin Tao;

Yi Pang,Lu Liu;

Publications:

已发表论文:
1. Yi Pang, Lifeng Sun, Jiangtao (Gene) Wen, Fengyan Zhang, Weidong Hu, Wei Feng and Shiqiang Yang, A Framework for Heuristic Scheduling for Parallel Processing on Multi-Core Architecture - A Case Study with Multi-view Video Coding (MVC), IEEE Trans. On CSVT, accepted.
2.Li Fang, Yang Shiqiang, “Virtual Viewpoint Synthesis in Multi-view video system”, SPIE VCIP 2005. 2005, 9
3.Zhang Jianning, Sun Lifeng, Zhong Yuzhuo, “All-zero block detection for H.264 intra prediction”, SPIE VCIP2005, 2005,9
4.Zhang Jianning, Sun Lifeng, Zhong Yuzhuo, “JOINT INTER AND INTRA SHOT MODELING FOR SPECTRAL VIDEO SHOT CLUSTERING”, IEEE ICME2005,2005,7
5.Zhang Jianning, Sun Lifeng, Zhong Yuzhuo, “POSITION PREDICTION MOTION-COMPENSATED INTERPOLATION FOR FRAME RATE UP-CONVERSION USING TEMPORAL MODELING”, IEEE ICIP2005,2005,9
6.Changjie Guo, Guobin Shen, Shipeng Li, Yuzhuo Zhong, “PCS-DisC: Peer-Assisted Cooperative Streaming Framework with Distributed Caching and Associated Protocol”, Journal of Tsinghua Science and Technology, Accept, to appear.
7.Baitao Long,Lifeng Sun,Wenping Chen,Yuzhuo Zhong, “Improving the Stability of Spanning Trees for Application-layer Multicast”, Electronic Journal (English Version), accepted, to appear

MPEG proposal:
1.Results for EE3 on evaluation of MAC for stereoscopic video coding,Zhang Hu, Sun Lifeng, Yang shiqiang, Cheng Xiaoyu,ISO/IEC JTC1/SC29/WG11/m10943, July 2004
2.Consideration on Quality Measures for Multi-view Video Scenario, Cheng Xiaoyu, Li Fang, Sun Lifeng, Yang Shiqiang, ISO/IEC JTC1/SC29/WG11/m10952, July 2004
3.Test Incomplete 3D Synthesis Method for Multi-view Video, Li Fang, Cheng Xiaoyu, Sun lifeng, Yang shiqiang, ISO/IEC JTC1/SC29/WG11/m10970, July 2004
4.Object based Rate Allocation Test Results for EE3 on evaluation of MAC for stereoscopic video coding, Zhang Hu, Sun Lifeng, Yang shiqiang, Cheng Xiaoyu, Li Fang, ISO/IEC JTC1/SC29/WG11/m11607, January, 2005
5.A system of interactive multiple view video, Cheng Xiaoyu, Li Fang, Sun Lifeng, Yang shiqiang, ISO/IEC JTC1/SC29/WG11/m11578, January, 2005

Demos and Resources:

 


Links: Tsinghua - DCST - HCI&MI


Any questions or comments about our web site please Contact Us
Copyrights (C) 2005 Multimedia Group, Dept. of CS, Tsinhua University.
All rights reserved.