MPEG의 3D 비디오 압축기술 표준화 동향

2010.5.11.

실감방송시스템연구팀
Contents
 Depth cue in human visual system
 MPEG standard overview
 MPEG-2 MVP
 MPEG-4 MVC
 MPEG-C Part 3
 3DV
 Overall Summary

1 방통미디어연구본부
3차원 인지 요인 - Depth Cue in the human visual system (1/2)

심리적인 요인(psychological cues)

원근감(Linear Perspective) 가려짐(Overlapping)

그늘과 그림자(Shades and Shadows) 결의 경사(Texture Gradient)

2
3차원 인지 요인(2/2)

생리적인 요인(physiological cue)

원근감 조절(Accommodation) 양안 시차(Binocular Parallax)

수렴(Convergence) 운동 시차(Motion Parallax)

3
Stereoscopic Viewing Display
 양안 시차(Binocular Parallax)
 동일 장면을 양안으로 볼 때 좌/우 눈 간격에 의해 발생하는 영상간 불일치
(Disparity)로 물체의 가깝고 먼 것을 인지할 수 있음.
 3차원 인지 요인중 가장 중요한 요소임.
 Stereoscopic Viewing Display이란,
 시청자에게 양안 시차를 갖는 영상을 좌/우 눈에 각각 보여줌으로써 입체
감을 주는 디스플레이.

흠, 영상간 차
이가 별로 없
군. 먼거리에
있는 물체네!

4 방통미디어연구본부
Consideration of redundancy in video compression

2D Video Compression
Spectral redundancy: Color Sub-sampling
Spatial redundancy: Frequency domain(DCT+Quantization)
Temporal redundancy: Inter-frame
Statistical redundancy: Entropy coding
3D Video Compression??
Spectral redundancy: Color Sub-sampling
Spatial redundancy: Frequency domain
Temporal redundancy: Inter-frame, Inter-view
Statistical redundancy: Entropy coding

5 방통미디어연구본부
MPEG Standard Overview
ISO/IEC JTC1 SC29/WG11 = MPEG

Media System
(Video, Audio) (Transport, Storage, Representation)

MPEG-1: the standard for storage and retrieval of moving pictures and audio on storage media (approved Nov.
1992)
MPEG-2: the standard for digital television (approved Nov. 1994)
MPEG-4: the standard for multimedia applications
MPEG-7 :the content representation standard for multimedia information search and filtering
MPEG-21: the multimedia framework
MPEG-A: the collection of standards for Application Formats
MPEG-B: the collection of Systems-related standards
MPEG-C: the collection of Video-related standards
MPEG-D: the collection of Audio-related standards
MPEG-E: the Multimedia Terminal standard
MPEG-M: the standard for packaging and reusability of MPEG technologies
MPEG-U: the standard for rich-media user interfaces

MPEG-4: in its different components (Systems, Video, Audio, 3D Graphics, Composition, Fonts etc.)
MPEG-7: particularly MPEG Query Format
MPEG-C: particularly Reconfigurable Video Coding
MPEG-D: particularly Universal Speech and Audio Coding

MPEG-V: the standard for real and virtual worlds, and for their interactions
3D Video Coding: the standard for coding 3D Visual information
Advanced IPTV Terminal: the standard for digital media ecosystems
High Efficiency Video Coding: the standard for a new frontier in video coding.
출처: http://mpeg.chiariglione.org/who_we_are.htm

6 방통미디어연구본부
Video Coding Standard of MPEG
ITU-T
VCEG
H.261
H.263 H.263 H.263 H.26L H.264 ?
v1

H.262
Video phone: PSTN, B-ISDN
Low quality: 64kbps~1.5Mbps
JVT JCT-VC
H.264/AVC, SVC, MVC HVEC

Digital Broadcasting,
ISO/IEC DVD, Digital Camcorder
High quality:1.5~80 Mbps
MPEG
MPEG-4
MPEG-1 MPEG-2 MPEG-4 MPEG-4
AVC
?

1992 1996 2000 2004 2010

Video CD, Multimedia Authoring
Internet, Mobile communication
VHS quality: Internet streaming
< 1.5Mbps Various quality: 64kbps~2Gbps

7 방통미디어연구본부
3D Video Coding Standard of MPEG

ITU-T H.262 H.264

VCEG

JVT
MVC(Multiview Video Coding)

ISO/IEC MPEG-2
MPEG-2
MVP(Multi-View
MPEG-4 MPEG-C 3DV
(3D Video
MPEG Profile AVC Part 3 Coding) cont.

1992 1996 2000 2004 2007 2010

Representation of
auxiliary video and
supplemental
information

8 방통미디어연구본부
Summary: Stereo Video
L/R simulcast possible with any MPEG standard
MPEG-2 Multi-view profile is essentially stereo with
temporal L/R interleaving
Stereoscopic MAF ISO/IEC 23000-11 based on MPEG-4
part 2 video (L/R packing, for handhelds)
MPEG-4 part 10 AVC Stereo SEI message and Frame
Packing Arrangement SEI message (the latter in 14496-
10/5e Amd.1, to be finalized by July 2009) allow various
methods of L/R packing
Temporal, spatial row/column, spatial side-by-side/up-and-bottom,
checkerboard (quincunx)
MPEG-4 AVC Stereo High Profile (new in Study 14496-
10/5e Amd.1, to be finalized by July 2009)
Subset of MVC, restricted to 2 views, allows progressive and interlaced
stereo
9 방통미디어연구본부
MPEG-2 Multi-view Profile(MVP)
Two-Layer Video Coding Scheme
Base layer
Assigned to Left View Video
MPEG-2 Main Profile(MP) Video Coding
Enhancement Layer
Temporal Scalability Video Coding(Disparity estimation+Motion estimation)
Assigned to Right View Video
Same spatial resolution in both layers
Forward compatibility and Backward compatibility
Decoder for MVP can process MP(Forward compatibility)
Decoder for MP can process Base layer of MVP(Backward compatibility)
Support a bit stream syntax
including camera position for generating new scene from any other angle

10 방통미디어연구본부
Enhancement layer predict mode

For P picture, Disparity

estimation

For B picture,

11 방통미디어연구본부
MPEG-2 MVP Performance
단위: PSNR(dB)

• Up to 1.6dB over the simulcast approach

• Consideration of enhancement techniques
(1) Brightness balancing of two views for disparity
estimation and compensation;
(2) Horizontal view offset for disparity estimation and
compensation;
(3) Rate control for stereoscopic video encoding

12 방통미디어연구본부
Multi-view Video Coding(MVC)

• Standard was approve in July 2008

1. Specified as an amendment of H.264/AVC
2. Integrated into 5th Edition of ISO/IEC 14496-10 (Annex H)

13 방통미디어연구본부
MVC scheme
Extension of the H.264/AVC for multiple view video
Temporal/Inter-view prediction video coding scheme
Temporal prediction video coding scheme
Hierarchical B picture structure
Inter-view prediction video coding scheme
Key-picture/Nonkey-picture prediction structure

Example of multiview video data with linear camera arrangement

14 방통미디어연구본부
 Prediction mode evaluation result
Temporal/Inter-view prediction mode
Inter-view prediction axis

Temporal prediction axis

Probability of temporal mode

Probability of Inter-view mode

Probability of chosen predictor when minimizing a Lagrangian cost function in

motion estimation for sequences "Uli" and "Breakdancers".
15 방통미디어연구본부
The structure of MVC

Temporal prediction using Hierarchical B pictures

Inter-view prediction for key pictures

MVC basic structure: Inter-view prediction for key/nonkey picture

16 방통미디어연구본부
Coding order of MVC Structure

출처: http://ip.hhi.de/imagecom_G1/cod_pattern.htm

17 방통미디어연구본부
MVC performance & summary

~25%

Ballroom sequence Race-1 sequence

• Up to 3.2 dB better than anchor coding(MVC)

• Half of coding gain when using Hierarchical B pictures(Simulcast)
• High Level Syntax
1. Improved random access
2. Low delay
3. Memory optimization
• Limitation/Issues
1. Acquision/production with large camera array is not common
2. Although more efficient than simulcast, rate of MVC is still proportional to the
number of views: Varies with scene, camera arrangement etc.

18 방통미디어연구본부
Depth Based Rendering

original Depth map

3D warp

Virtual view
19 방통미디어연구본부
MPEG-C Part 3:ISO/IEC 23002-3

• Video+Depth as data representation for 3DTV

• Initiative driven by Philips, FhG-HHI and other partners as a result of
ATTEST project
• Define a simple container format
• Not specify transport and compression techniques
• Finalize at January 2007 FDIS
• ISO/IEC 23002-3 Representation of Auxiliary Video and Supplemental
Information
• ISO/IEC 13818-1: 2003 Carriage of Auxiliary Data

20 방통미디어연구본부
Extension to 3DV: Current
MVC
Usage of N views
No continuum
For large N very inefficient
MPEG-C part 3
Disocclusion artifacts increase with distance of virtual
view from available original view
Does not support wide range multi-view 3D displays
Very limited free viewpoint navigation

21 방통미디어연구본부
3DV Data Format

22 방통미디어연구본부
Bit rate vs 3D Rendering Capabilities

23 방통미디어연구본부
3D Video Framework

24 방통미디어연구본부
3DV summary
Main Objectives
Support auto-stereoscopic displays from a limited
number of input views and also variable baseline for
stereo processing
Inclusion of depth: decouple number of transmitted
views with number of required views for display
MPEG exploration underway
In the process of establishing suitable reference
Gathering available Test multi depth and video sequence
To subjective quality testing, Depth Estimation/View Synthesis
Anchor Coding Experiment
Expecting to issue Call for Proposals at Oct. 2010

25 방통미디어연구본부
Overall summary

MEPG has actively contributed compression

technology for stereo and multi-view video and is
considering to take the next steps towards 3D
and free-viewpoint video
In 3D video part, we are always trying to define
generic formats that support to high-fidelity and
compatibility with other standard(not easy!)
ETRI, Samsung, LG and GIST are very actively
participating in 3DV group

26 방통미디어연구본부
Reference
Chen, Xuemin; Luthra, Ajay, “MPEG-2 Multi-View Profile and its application in 3DTV”,
Proc. SPIE Vol. 3021, p. 212-223, Multimedia Hardware Architectures 1997.
Philipp Merkle, Aljoscha Smolic´, Karsten Müller, T.Wiegand, “Efficient Prediction
Structures for Multiview Video Coding”, IEEE TRANSACTIONS ON CIRCUITS AND
SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 17, NO. 11, NOVEMBER 2007
P. Merkle, K. Müller, A. Smolic, and T. Wiegand, “EFFICIENT COMPRESSION OF MULTI-
VIEW VIDEO EXPLOITING INTER-VIEW DEPENDENCIES BASED ON H.264/MPEG4-AVC”,
IEEE International Conference on Multimedia and Expo (ICME'06), Toronto, Ontario,
Canada, July 2006.
Video and Requirements, “Applications & Requirements on 3D video coding,” ISO/IEC
JTC1/SC29/WG11 Doc. N11061, Xian, China, October 2009.

27 방통미디어연구본부