Interactive Multiview Video and Parallelism over Multi-core
Interactive Multiview Video:
The next generation multimedia applications will be interactive and realistic based on network. In one hand, people want to be more active and participant to meet the requirement of personality, in the other hand, they hope to get realistic experience while viewing the scene or interacting with other people. This project is proposed to resolve the represent, interaction, codec and transmission of 3D interactive video, and implement the interactive network multimedia application in the near future with the stereo video and multi-view video technology putting into practicability.
Viewpoint selecting and multi-view 3D image synthesizing
High efficiency multi-view video codec algorithms
Streaming technology for Multi-view video
A prototype of multi-view video streaming system
Parallelism over Multi-core:
By using the Intel multicore architectures
and the emerging multiview video coding standard, we introduce
a framework for performing analysis, simulation, and evaluation
of heuristics scheduling algorithms for implementing computationally
intensive algorithms on multicore processors. The framework
allows for accurate and quantitative characterization of the
performance of dynamic scheduling algorithms for multimedia
applications on different multicore processors without actual
implementation of the scheduling algorithm and application
on the actual platform. Experimental results demonstrate the
effectiveness and scalability of our framework.
Model the dynamic nature of the dependencies between tasks at different granularities
Come up with different heuristics for designing efficient scheduling algorithms
Predicting the performance of scheduling algorithms on multicore platforms
Content of Interactive Multiview Video
- Multiview video synthesis
Motion vector based video objects segmentation and tracking
Feature based incomplete 3D structure video synthesizing
Layered depth images based multiview video rendering
Semi-immersive MultiView display environment
- High efficiency Multiview Video Coding
Disparity based multiview video coding
View dependent multiview video predicting codec
- Interactive multiview video streaming
Time frozen moment
Time continuous sweep
Content of Interactive Multiview Video
- DAG-based framework
The DAG-based framework consists of three modules: the encoding module, the heuristics scheduling module
and the multicore module.
Encoder Module: Video encoding algorithm is abstracted as
a hierarchical multisource multisink DAG extended over the
Heuristic Scheduling Module: The heuristics scheduling
module synthesizes the states of encoder graph and multicore
processor, and makes task scheduling decisions.
Multicore Module: The module records and updates the
states of the multicore processor, and calculates the set V(i)
of tasks which have been finished on the ith step.
Structure diagram of framework.
- Application to MVC on Symmetric Multicore
GOP scheduling in MVC.
Working flowchart of framework.
- Speedup ratio
Speedup ratio as a function of the number of concurrent GOPs,
estimated by the proposed framework.
Projects with sponsor:
国家高科技计划863项目： 面向交互应用的多视点视频处理技术研究 No.2002AA119040
Professor Shiqiang Yang,Lingfeng Sun and Pin Tao;
Yi Pang,Lu Liu;
1. Yi Pang, Lifeng Sun, Jiangtao (Gene) Wen, Fengyan Zhang, Weidong Hu, Wei Feng and Shiqiang Yang, A Framework for Heuristic Scheduling for Parallel Processing on Multi-Core Architecture - A Case Study with Multi-view Video Coding (MVC), IEEE Trans. On CSVT, accepted.
2．Li Fang, Yang Shiqiang, “Virtual Viewpoint Synthesis in Multi-view video system”, SPIE VCIP 2005. 2005, 9
3．Zhang Jianning, Sun Lifeng, Zhong Yuzhuo, “All-zero block detection for H.264 intra prediction”, SPIE VCIP2005, 2005,9
4．Zhang Jianning, Sun Lifeng, Zhong Yuzhuo, “JOINT INTER AND INTRA SHOT MODELING FOR SPECTRAL VIDEO SHOT CLUSTERING”, IEEE ICME2005，2005，7
5．Zhang Jianning, Sun Lifeng, Zhong Yuzhuo, “POSITION PREDICTION MOTION-COMPENSATED INTERPOLATION FOR FRAME RATE UP-CONVERSION USING TEMPORAL MODELING”, IEEE ICIP2005，2005，9
6．Changjie Guo, Guobin Shen, Shipeng Li, Yuzhuo Zhong, “PCS-DisC: Peer-Assisted Cooperative Streaming Framework with Distributed Caching and Associated Protocol”, Journal of Tsinghua Science and Technology, Accept, to appear.
7．Baitao Long，Lifeng Sun，Wenping Chen，Yuzhuo Zhong, “Improving the Stability of Spanning Trees for Application-layer Multicast”, Electronic Journal (English Version), accepted, to appear
1．Results for EE3 on evaluation of MAC for stereoscopic video coding，Zhang Hu, Sun Lifeng, Yang shiqiang, Cheng Xiaoyu，ISO/IEC JTC1/SC29/WG11/m10943, July 2004
2．Consideration on Quality Measures for Multi-view Video Scenario, Cheng Xiaoyu, Li Fang, Sun Lifeng, Yang Shiqiang, ISO/IEC JTC1/SC29/WG11/m10952, July 2004
3．Test Incomplete 3D Synthesis Method for Multi-view Video, Li Fang, Cheng Xiaoyu, Sun lifeng, Yang shiqiang, ISO/IEC JTC1/SC29/WG11/m10970, July 2004
4．Object based Rate Allocation Test Results for EE3 on evaluation of MAC for stereoscopic video coding, Zhang Hu, Sun Lifeng, Yang shiqiang, Cheng Xiaoyu, Li Fang, ISO/IEC JTC1/SC29/WG11/m11607, January, 2005
5．A system of interactive multiple view video, Cheng Xiaoyu, Li Fang, Sun Lifeng, Yang shiqiang, ISO/IEC JTC1/SC29/WG11/m11578, January, 2005
Demos and Resources: