A Fast Spatial Patch Blending Algorithm For Artefact Reduction in Pattern-Based Image Inpainting

A Fast Spatial Patch Blending Algorithm for Artefact Reduction in Pattern-based Image Inpainting1
Olivier Lezoray Maxime Daisy, David Tschumperle,

GREYC - UMR 6072 CNRS, ENSICAEN, University of Caen Image team
SIGGRAPH Asia
November 21, 2013
1
This project is funded by Action 3DS National Project.
Outline
Inpainting : Context and State of the Art Contribution : Fast Multiscale Patch Blending for Inpainting Results and Comparisons with the State of the Art
M. Daisy (GREYC, Caen - FRANCE)
SIGGRAPH Asia 2013
2 / 43
Outline
SIGGRAPH Asia 2013
3 / 43
Image Inpainting : Overview
Original
M. Daisy (GREYC, Caen - FRANCE) SIGGRAPH Asia 2013 4 / 43
Masked image
Inpainted image
Image Inpainting : a little history

Masnou and Morel : Level lines based disocclusion, ICIP 1998 Efros and Leung : Texture synthesis by non-parametric
sampling, ICCV 1999

Bertalm o et al. : Image Inpainting, SIGGRAPH 2000 Criminisi et al. : Object Removal by Exemplar-based
Inpainting, CVPR 2003

Wexler et al. : Space-time video completion, IEEE PAMI 2007 Sun et al. : Image Inpainting by Patch Propagation Using
Patch Sparsity IEEE 2010

Darabi et al. : Image Melding : Combining Inconsistent
Images using Patch-based Synthesis, SIGGRAPH 2012

...
Criminisi et al. 2003 : overview
SIGGRAPH Asia 2013
8 / 43
Good texture reconstruction
SIGGRAPH Asia 2013
9 / 43
Good texture reconstruction
SIGGRAPH Asia 2013
10 / 43
Local geometric inconsistencies
SIGGRAPH Asia 2013
11 / 43
Since Criminisi et al. 2003 inpainting algorithm
Content-aware Filling, Adobe 2010

Wexler et al. 2007 : Space-time completion of videos Barnes et al. 2009 : PatchMatch
some blurry textures Le Meur et al. 2011

local geometry aware : structure tensor analysis K-nearest neighbours averaging
blurry textures, geometric inconsistencies
SIGGRAPH Asia 2013
12 / 43
Since Criminisi et al. 2003 inpainting algorithm
Content-aware lling
Le Meur et al. 2011
SIGGRAPH Asia 2013
13 / 43
The question
How to keep better textures and geometry consistency ?
SIGGRAPH Asia 2013
14 / 43
Outline
SIGGRAPH Asia 2013
15 / 43
Spatial patch blending - Extreme case
Masked image
Inpainted image Criminisi et al. 2003

Diffusion PDE inpainting result 2006 Tschumperle

Content-aware lling algorithm Adobe 2010

Effect of our spatial patch blending Our Contribution

(p) : an artefact-aware blending amplitude
Break eld R (p)

Blending amplitude eld (p)

SIGGRAPH Asia 2013 21 / 43
Spatial Patch Blending : principle
SIGGRAPH Asia 2013
22 / 43
Mathematical expression
Usage of reconstruction points and patch locations,
saved during any patch-based inpainting process w (q , p) q (p q ) + w (q , p )

q p
J (p) =
q p
p : set of patches used for the reconstruction around p q (p q ) : pixel value at ((N /2, N /2)T + (p q )) in q (p ) : blending amplitude at p w (q , p ) gaussian weight of variance (p )2
Fast algorithm - Blend scales
SIGGRAPH Asia 2013
24 / 43
Outline
SIGGRAPH Asia 2013
25 / 43
Processing Times
SIGGRAPH Asia 2013
26 / 43
Result - Nature
Original Image
Result - Nature
Content-aware Filling
Result - Nature
Criminisi + our spatial patch blending

Result - Fake Owl
Original Image
Result - Fake Owl
Criminisi et al. 2003

Result - Fake Owl
Result - Fake Owl
Criminisi et al. 2003 + our spatial patch blending

Result - Fake Owl

Result - Savanna
Original Image
Result - Savanna

Result - Savanna
Result - Savanna
Criminisi et al. 2003 + our spatial patch blending

GIMP Software Integration : GMIC plugin
SIGGRAPH Asia 2013
39 / 43
SIGGRAPH Asia 2013
40 / 43
SIGGRAPH Asia 2013
41 / 43
SIGGRAPH Asia 2013
42 / 43
maxime.daisy@ensicaen.fr http ://gmic.sourceforge.net/gimp.shtml https ://daisy.users.greyc.fr/daisy-etal-siggraph-asia-2013


A Fast Spatial Patch Blending Algorithm For Artefact Reduction in Pattern-Based Image Inpainting

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

A Fast Spatial Patch Blending Algorithm For Artefact Reduction in Pattern-Based Image Inpainting

Enviado por

Direitos autorais:

Formatos disponíveis

A Fast Spatial Patch Blending Algorithm for Artefact Reduction in Pattern-based Image Inpainting1

Olivier Lezoray Maxime Daisy, David Tschumperle,

This project is funded by Action 3DS National Project.

M. Daisy (GREYC, Caen - FRANCE)

SIGGRAPH Asia 2013

M. Daisy (GREYC, Caen - FRANCE)

SIGGRAPH Asia 2013

Image Inpainting : Overview

Image Inpainting : Overview

Image Inpainting : Overview

Image Inpainting : a little history

sampling, ICCV 1999

Inpainting, CVPR 2003

Patch Sparsity IEEE 2010

Images using Patch-based Synthesis, SIGGRAPH 2012

Criminisi et al. 2003 : overview

M. Daisy (GREYC, Caen - FRANCE)

SIGGRAPH Asia 2013

Good texture reconstruction

M. Daisy (GREYC, Caen - FRANCE)

SIGGRAPH Asia 2013

Good texture reconstruction

M. Daisy (GREYC, Caen - FRANCE)

SIGGRAPH Asia 2013

Local geometric inconsistencies

M. Daisy (GREYC, Caen - FRANCE)

SIGGRAPH Asia 2013

Since Criminisi et al. 2003 inpainting algorithm

Content-aware Filling, Adobe 2010

some blurry textures Le Meur et al. 2011

blurry textures, geometric inconsistencies

M. Daisy (GREYC, Caen - FRANCE)

SIGGRAPH Asia 2013

Since Criminisi et al. 2003 inpainting algorithm

Le Meur et al. 2011

M. Daisy (GREYC, Caen - FRANCE)

SIGGRAPH Asia 2013

How to keep better textures and geometry consistency ?

M. Daisy (GREYC, Caen - FRANCE)

SIGGRAPH Asia 2013

M. Daisy (GREYC, Caen - FRANCE)

SIGGRAPH Asia 2013

Spatial patch blending - Extreme case

Spatial patch blending - Extreme case

Inpainted image Criminisi et al. 2003

Spatial patch blending - Extreme case

Diffusion PDE inpainting result 2006 Tschumperle

Spatial patch blending - Extreme case

Content-aware lling algorithm Adobe 2010

Spatial patch blending - Extreme case

Effect of our spatial patch blending Our Contribution

(p) : an artefact-aware blending amplitude

Break eld R (p)

Blending amplitude eld (p)

Spatial Patch Blending : principle

M. Daisy (GREYC, Caen - FRANCE)

SIGGRAPH Asia 2013

Usage of reconstruction points and patch locations,

saved during any patch-based inpainting process w (q , p) q (p q ) + w (q , p )

Fast algorithm - Blend scales

M. Daisy (GREYC, Caen - FRANCE)

SIGGRAPH Asia 2013

M. Daisy (GREYC, Caen - FRANCE)