PRENTICE HALL SIGNAL PROCESSING SERIES Alan V. Oppenheim, Series Editor ANDREWS & HUNT Digital Image Restoration BRACEWELL Two Dimensional Imaging BRIGHAM The Fast Fourier Transform and Its Applications BURDIC Underwater Acoustic System Analysis 2/E CASTLEMAN Digital Image Processing CROCHIERE & RABINER Multirate Digital Signal Processing DUDGEON & MERSEREAU Multidimensional Digital Signal Processing HAYKIN Advances in Spectrum Analysis and Array Processing. Vols. I, II & III HAYKIN, ED. Array Signal Processing JOHNSON & DUDGEON Array Signal Processing Kay Fundamentals of Statistical Signal Processing KAY Modern Spectral Estimation KINO = Acoustic Waves: Devices, Imaging, and Analog Signal Processing LIM Two-Dimensional Signal and Image Processing LIM, ED. Speech Enhancement LIM & OPPENHEIM, EDS. Advanced Topics in Signal Processing MARPLE Digital Spectral Analysis with Applications MCCLELLAN & RADER Number Theory in Digital Signal Processing MENDEL Lessons in Estimation Theory for Signal Processing Communications and Control 2/E NIKIAS_ Higher Order Spectra Analysis OPPENHEIM & NAWAB_ Symbolic and Knowledge-Based Signal Processing OPPENHEIM, WILLSKY, WITH YOUNG Signals and Systems OPPENHEIM & SCHAFER Digital Signal Processing OPPENHEIM & SCHAFER Discrete-Time Signal Processing PHILLIPS & NAGLE Digital Control Systems Analysis and Design, 3/E PICINBONO Random Signals and Systems RABINER & GOLD Theory and Applications of Digital Signal Processing RABINER & SCHAFER Digital Processing of Speech Signals RABINER,.G&IJUANG | Fundamentals of Speech Recognition ROBINSON & TREITEL “Geophysical Signal Analysis STEARNS & DAVID Signal Processing Algorithms in Fortran and C TEKALP _ Digital Video Processing THERRIEN Discrete Random Signals and Statistical Signal Processing TRIBOLET Seismic Applications of Homomorphic Signal Processing VIADYANATHAN Multirate Systems and Filter Banks WIDROW & STEARNS Adaptive Signal ProcessingDigital Video Processing A. Murat Tekalp University of Rochester For book and bookstore information httpy/www.prenhall.com the Prentice Hall PTR Upper Saddle River, NJ 07458Tekalp, A. Murat. Digital video processing / A. Murat Tekalp. p. om, ~~ (Prentice-Hall signal processing series) ISBN 0-13-190075-7 (alk. paper) 1. Digital video. I. Title. II. Series. TK6680.5.T45 1995 621.388'33--dc20 95-16650 CIP Editorial/production supervision: Ann Sullivan Cover design: Design Source ‘Manufacturing manager: Alexis R. Heydt Acquisitions editor: Karen Gettman Editorial assistant: Barbara Alfieri & Printed on Recycled Paper === ©1995 by Prentice Hall PTR SAGE Prentice-Hall, Inc. 58 4 Simon and Schuster Company Upper Saddle River, NJ 07458 The publisher offers discounts on this book when ordered in bulk quantities. For more information, contact: Corporate Sales Department Prentice Hall PTR One Lake Street Upper Saddle River, NJ 07458 Phone: 800-382-3419 Fax: 201-236-7141 email: corpsales @prenhall.com All rights reserved. No part of this book may be reproduced, in any form or by any means without permission in writing from the publisher. Printed in the United States of America 10987654321 “ISBN: 0-13-190075-7 Prentice-Hall International (UK) Limited, London Prentice-Hall of Australia Pty. Limited, Sydney Prentice-Hall Canada Inc., Toronto Prentice-Hall Hispanoamericana, S.A., Mexico Prentice-Hall of India Private Limited, New Delhi Prentice-Hall of Japan, Inc., Tokyo Simon & Schuster Asia Pte. Ltd., Singapore Editora Prentice-Hall do Brasil, Ltda., Rio de JaneiroTo Sevim and Kaya Tekalp, my mom and dad and to Ozge, my beloved wifeContents Pretace Hes tete toda ele rete eat eee parte be eesti Ca act eee ie taste eth xvii About the Author... 2.6... ee xix About the Notation... 2.2... 20... 0.0.00 eee eee eee xxi I REPRESENTATION OF DIGITAL VIDEO 1 BASICS OF VIDEO 1 1.1 Amalog Video... 1.6... 6. ce eee 1 1.1.1 Analog Video Signal . . 2 1.1.2 Analog Video Standards 4 1.1.38 Analog Video Equipment 8 1.2 Digital Video............00. 000005 9 1.2.1 Digital Video Signal . . 9 1.2.2 Digital Video Standards ll 1.2.3 Why Digital Video? Lee 14 1.3 Digital Video Processing.................-000- 16 2 TIME-VARYING IMAGE FORMATION MODELS 19 2.1. Three-Dimensional Motion Models ...............- 20 2.1.1 Rigid Motion in the Cartesian Coordinates 20 2.1.2 Rigid Motion in the Homogeneous Coordinates ........ 26 2.1.38 Deformable Motion..............05000+0055 27 2.2 Geometric Image Formation ..... 28 2.2.1 Perspective Projection... ..........0000005 wo 28 2.2.2 Orthographic Projection... 1... 2.0... 0s eee eee 30 2.3 Photometric Image Formation .. . . cee 32 2.3.1 Lambertian Reflectance Model .............---. 32 2.3.2 Photometric Effects of 3-D Motion .............-5 33 2.4 Observation Noise Teas 33 2,0 Mammen cases ee Peco ete ae Hace Set er eee eiceaHicte Hae See StS HSet ea etast ieee 34 viiviii CONTENTS 3 SPATIO-TEMPORAL SAMPLING 36 3.1 Sampling for Analog and Digital Video 37 3.1.1 Sampling Structures for Analog Video . . 37 3.1.2 Sampling Structures for Digital Video . . 38 3.2. Two-Dimensional Rectangular Sampling. . 40 3.2.1 2-D Fourier Transform Relations 41 3.2.2 Spectrum of the Sampled Signal .... . 42 3.3 Two-Dimensional Periodic Sampling . . . . eee 43 3.3.1 SamplingGeometry .............0 00005 44 3.3.2 2-D Fourier Transform Relations in Vector Form .. . . 44 3.3.3 Spectrum of the Sampled Signal 46 3.4 Sampling on 3-D Structures 46 3.4.1. Sampling on a Lattice 47 3.4.2 Fourier Transform on a Lattice - 47 3.4.3 Spectrum of Signals Sampled on a Lattice... 2.2... 49 3.4.4 Other Sampling Structures 2.2.2.0 ........00-- 51 3.5 Reconstruction from Samples.......... 53 3.5.1 Reconstruction from Rectangular Samples... ......-- 53 3.5.2 Reconstruction from Samples on a Lattice... .. 2... - 55 3.6 Exercises... 0... eee 56 4 SAMPLING STRUCTURE CONVERSION 57 4.1 Sampling Rate Change for 1-D Signals 58 4.1.1 Interpolation of 1-D Signals 58 4.1.2 Decimation of 1-D Signals 62 4.1.3 Sampling Rate Change by a Rational Factor Se 64 4.2 Sampling Lattice Conversion .........-.0-00 00005 66 4.3 Exercises... 0... ee 70 II TWO-DIMENSIONAL MOTION ESTIMATION 5 OPTICAL FLOW METHODS 72 5.1 2-D Motion vs. Apparent Motion..............-.- 72 5.1.1 2-DMotion...............00- 73 5.1.2 Correspondence and Optical Flow ..... 74 5.2 2-D Motion Estimation. ..... aaa 76 5.2.1 The Occlusion Problem 78 5.2.2 The Aperture Problem 78 5.2.3 Two-Dimensional Motion Field Models. . . 79 5.3 Methods Using the Optical Flow Equation . 81 5.3.1 The Optical Flow Equation 81 5.3.2 Second-Order Differential Methods 82 5.3.3 Block Motion Model... 2.2.2.2. 2000000 e eee 83CONTENTS ix 5.3.4 Horn and Schunck Method 84 5.3.5 Estimation of the Gradients . 85 5.3.6 Adaptive Methods ...... feb eee - 86 54 Examples... 00.0 .000000 0000 cee cece eee eee 88 DEC Exercises iste tetas te ede eaeb atte Het eSteeiaeeeabte tect ate atte Pette ea 93 6 BLOCK-BASED METHODS 95 6.1 Block-Motion Models . . : 95 6.1.1 Translational Block Motion wae . 96 6.1.2 Generalized/Deformable Block Motion + 97 6.2 Phase-Correlation Method eit eth 99 6.2.1 The Phase-Correlation Function .... . 99 6.2.2 Implementation Issues . . . . 100 6.3 Block-Matching Method... . 101 6.3.1 Matching Criteria ...... 102 6.3.2 Search Procedures ..... . 104 6.4 Hierarchical Motion Estimation ...... . 106 6.5 Generalized Block-Motion Estimation .. . . ie 109 6.5.1 Postprocessing for Improved Motion Compensation EE 109 6.5.2 Deformable Block Matching . 109 6.6 Examples .............. Berea 112 (Fi dael O>,<=s 11 1-1: Beeman ae BECERTCEE Tran seran sures SarsSTeCeesereesirenseeataraetaceryeceryeceraic 115 7 PEL-RECURSIVE METHODS 117 7.1 Displaced Frame Difference ..................0-0- 118 7.2 Gradient-Based Optimization...................4 119 7.3 8 BAYESIAN METHODS 8.1 7.2.1 Steepest-Descent Method 7.2.2. Newton-Raphson Method . . . 7.2.3 Local vs. Global Minima Steepest-Descent-Based Algorithms. ..............- 7.3.1 Netravali=Robbins Algorithm . peta Pees 7.8.2 Walker-Rao Algorithm. ... 0.0... 00 00000 e eee 7.3.3 Extension to the Block Motion Model Wiener-Estimation-Based Algorithms ... . He § OF ocT0y 1) (cr RED NSPa ERED NS HENESGSNG Bees EDEN Ena) EDREAsPESNeSENEPEE erSUSEPEONSreEt=PESteD Exercises Optimization Methods EREEEeY 8.1.1 Simulated Annealing... ... 8.1.2 Iterated Conditional Modes . . 8.1.3 Mean Field Annealing aE peesecee 8.1.4 Highest Confidence First... 2.6... 0-0-0000 eeex CONTENTS 8.2 Basics of MAP Motion Estimation ................ 8.2.1 The Likelihood Model 8.2.2 The Prior Model ........ SEE eee 8.3 MAP Motion Estimation Algorithms... . . 8.3.1 Formulation with Discontinuity Models 8.3.2 Estimation with Local Outlier Rejection . . 8.3.3 Estimation with Region Labeling ..... . 8.4 Examples ee Hee a 8.5 Exercises III THREE-DIMENSIONAL MOTION ESTIMATION AND SEGMENTATION 9 METHODS USING POINT CORRESPONDENCES 9.1 Modeling the Projected Displacement Field .......... 9.1.1 Orthographic Displacement Field Model ..........-.. 9.1.2 Perspective Displacement Field Model . . . . 9.2 Methods Based on the Orthographic Model 9.2.1 Two-Step Iteration Method from Two Views 9.2.2 An Improved Iterative Method ........ 9.3 Methods Based on the Perspective Model eat 9.3.1 The Epipolar Constraint and Essential Parameters eee 9.3.2 Estimation of the Essential Parameters 9.3.3 Decomposition of the E-Matrix 9.3.4 Algorithm............ 9.4 The Case of 3-D Planar Surfaces 9.4.1 The Pure Parameters eee 9.4.2 Estimation of the Pure Parameters. ............ 9.4.3 Estimation of the Motion and Structure Parameters . . . 9.5 Examples ... 2... 0.02... c eee eee eee 9.5.1 Numerical Simulations... 2.2.2... 2.0.2... eee 9.5.2 Experiments with Two Frames of Miss America... . . . 9.6 Exercises... 6... ee 10 OPTICAL FLOW AND DIRECT METHODS 10.1 Modeling the Projected Velocity Field.............. 10.1.1 Orthographic Velocity Field Model . 10.1.2 Perspective Velocity Field Model ....... 10.1.3 Perspective Velocity vs. Displacement Models 10.2 Focus of Expansion .... 2.0.2... 0. eee eee ee 10.3 Algebraic Methods Using Optical Flow Reet 10.3.1 Uniqueness of the Solution ©... .....-....- 00005 10.3.2 Affine Flow... 2... eeCONTENTS xi 10.3.3 Quadratic Flow... 6 6. ee 183 10.3.4 Arbitrary Flow at 184 10.4 Optimization Methods Using Optical Flow ........... 186 10.5 Direct Methods ............. 0.0.00. 0 ee ee eeee 187 10.5.1 Extension of Optical Flow-Based Methods . . Te 187 10.5.2 Tsai-Huang Method ........... 0.000005 bees 188 10.6 Examples ..... 2... 20.0 0c eee 190 10.6.1 Numerical Simulations............. 191 10.6.2 Experiments with Two Frames of Miss America... . . . 194 10.7 Exercises... 2... ee 196 11 MOTION SEGMENTATION 198 11.1 Direct Methods ...... 0.0.0... e eee eee 200 11.1.1 Thresholding for Change Detection... ..........4- 200 11.1.2 An Algorithm Using Mapping Parameters . . ae 201 11.1.3 Estimation of Model Parameters .............-4- 203 11.2 Optical Flow Segmentation ..............--0.005 204 11.2.1 Modified Hough Transform Method .............- 205 11.2.2 Segmentation for Layered Video Representation 11.2.3 Bayesian Segmentation............. eee 11.3 Simultaneous Estimation and Segmentation .......... 209 11.3.1 Motion Field Model 11.3.2 Problem Formulation 11.3.3 The Algorithm ............ 11.3.4 Relationship to Other Algorithms eee ae 11.4 Examples .. 2.2... ee eee 214 11.5 Exercises. 2. eee 217 12 STEREO AND MOTION TRACKING 219 12.1 Motion and Structure from Stereo 219 12.1.1 Still-Frame Stereo Imaging .... . 220 12.1.2 3-D Feature Matching for Motion Estimation . 222 12.1.3 Stereo-Motion Fusion .............000005 - 224 12.1.4 Extension to Multiple Motion . 227 12.2 Motion Tracking aH cece cas 229 12.2.1 Basic Principles... 2... 1... ee ee es 229 12.2.2 2-D Motion Tracking... . 2... 2... ee eee ee ee 232 235 . 12.2.3 3-D Rigid Motion ees 12.3 Examples .. 12.4 Exercisesxii CONTENTS IV VIDEO FILTERING 13 MOTION COMPENSATED FILTERING 13.1 Spatio-Temporal Fourier Spectrum ................ 13.1.1 Global Motion with Constant Velocity ............. 13.1.2 Global Motion with Acceleration ................ 13.2 Sub-Nyquist Spatio-Temporal Sampling . . . eee 13.2.1 Sampling in the Temporal Direction Only ........... 13.2.2 Sampling on a Spatio-Temporal Lattice 13.2.3 Critical Velocities coat 13.3 Filtering Along Motion Trajectories ............... 13.3.1 Arbitrary Motion Trajectories... .. 0.0.0.0... 000% 13.3.2 Global Motion with Constant Velocity . . . otek 13.3.3 Accelerated Motion ... 2... . 0.0.0.0 eee eee 13.4 Applications... 2-2... ee 13.4.1 Motion-Compensated Noise Filtering . . . . pete 13.4.2 Motion-Compensated Reconstruction Filtering ....... . 258 13.5 Exercises... ee 260 14 NOISE FILTERING 14.1 Intraframe Filtering... . 2.2.2... 0.0.0.0 00 00005 14.1.1 LMMSE Filtering Cee seaseraes 14.1.2 Adaptive (Local) LMMSE Filtering .............. 14.1.3 Directional Filtering... 6... ee 14.1.4 Median and Weighted Median Filtering eeeHeey 14.2 Motion-Adaptive Filtering ................--00- 14-211 Direct Piltermge oe eee eee eee eee 14.2.2 Motion-Detection Based Filtering 14.3 Motion-Compensated Filtering. . . 14.3.1 Spatio-Temporal Adaptive LMMSE Filtering 14.3.2 Adaptive Weighted Averaging Filter... . . 14.4 Examples eee 14.5 Exercises 15 RESTORATION Tb Modeling so 2st Se Saeed Lactate Ecce ater acta att 15.1.1 Shift-Invariant Spatial Blurring . 15.1.2 Shift-Varying Spatial Blurring . . 15.2 Intraframe Shift-Invariant Restoration.............. 286 15.2.1 Pseudo Inverse Filtering ............ 15.2.2 Constrained Least Squares and Wiener Filtering 15.3 Intraframe Shift-Varying Restoration 15.3.1 Overview of the POCS Method . petsesee Seee eee 15.3.2 Restoration Using POCS ...........-...---45CONTENTS xiii 15.4 Multiframe Restoration peaches 292 15.4.1 Cross-Correlated Multiframe Filter . . . . 294 15.4.2 Motion-Compensated Multiframe Filter . 295 15.5 Examples ................ aa peat . 295 15.6 Exercises... 2... ee ee 296 16 STANDARDS CONVERSION 302 16.1 Down-Conversion . . 304 16.1.1 Down-Conversion with Anti- Alias Filtering : 305 16.1.2 Down-Conversion without Anti-Alias Filtering 305 16.2 Practical Up-Conversion Methods . 308 16.2.1 Intraframe Filtering .... . 309 16.2.2 Motion-Adaptive Filtering . 314 16.3 Motion-Compensated Up-Conversion . 317 16.3.1 Basic Principles... .........545 317 322 16.3.2 Global-Motion-Compensated De-interlacing . 16.4 Examples cee eee Pee 16.5 Exercises 17 SUPERRESOLUTION 331 If Modeling . bcc tc tee ee yet cee estes at atta ea 332 17.1.1 Continuous-Discrete Model Pete eae . 332 17.1.2 Discrete-Discrete Model ..........---00000005 335 17.1.3 Problem Interrelations ..... 2.2... ee eee ee 336 17.2 Interpolation-Restoration Methods see - 336 17.2.1 Intraframe Methods ............- 00000000 ee 337 17.2.2 Multiframe Methods .... 2... 2.000000 0 00-0 ue 337 17.3 A Frequency Domain Method . See tt . 338 17.4 A Unifying POCS Method .................0.5. 341 17.5 Examples 343 17.6 Exercises .. . 346 V_ STILL IMAGE COMPRESSION 18 LOSSLESS COMPRESSION 18.1 Basics of Image Compression. ..............-- : 18.1.1 Elements of an Image Compression System eae 18.1.2 Information Theoretic Concepts .... 2.0.0.0 .0005 18-24S ymboli Coding fe ere eee See Sree eee eee 18.2.1 Fixed-Length Coding See Perea 1822.24 Mutiman: Codmg-Ha pete tse eta are eee eee saeea 18.2.3 Arithmetic Coding... 1... eee eeexiv 18.3 18.4 CONTENTS Lossless Compression Methods. .............5+505 360 18.3.1 Lossless Predictive Coding a 18.3.2 Run-Length Coding of Bit-Planes.............--- 363 18.3.3 Ziv-Lempel Coding Exercises... ........... 19 DPCM AND TRANSFORM CODING 368 19.1 19.2 19.3 19.4 Quantization ©... 2 ee 19.1.1 Nonuniform Quantization . 19.1.2 Uniform Quantization . . . Differential Pulse Code Modulation . 19.2.1 Optimal Prediction 19.2.2 Quantization of the Prediction Error . 19.2.3 Adaptive Quantization. ........ 19.2.4 Delta Modulation....... Transform Coding......... 19.3.1 Discrete Cosine Transform . . 19.3.2 Quantization/Bit Allocation . aera Rees 19.3.3 Coding ©... -- eee ee eee 19.3.4 Blocking Artifacts in Transform Coding ............ 385 TEXerel sess ee tsetse He tette-tata-ee et ee te ast att ate ct ebe-ee 385 20 STILL IMAGE COMPRESSION STANDARDS 388 20.1 20.3 Bilevel Image Compression Standards 20.1.1 One-Dimensional RLC ......... 20.1.2 Two-Dimensional RLC. . . . 20.1.3 The JBIG Standard ..... The JPEG Standard . 20.2.1 Baseline Algorithm 20.2.2 JPEG Progressive 20.2.3 JPEG Lossless . . 20.2.4 JPEG Hierarchical 20.2.5 Implementations of JPEG . . Exercises 21 VECTOR QUANTIZATION, SUBBAND CODING AND OTHER METHODS 404 21.1 21.2 Vector Quantization... 6... 6... eee eee 404 21.1.1 Structure of a Vector Quantizer..... ee SEP eeee 21.1.2 VQ Codebook Design ..... 2.0.0... 2000000 00- 408 21.1.3 Practical VQ Implementations ...............-- 408 Fractal Compression... 2.00.00 ee 409