GUIDE TO VISUALIZING VIDEO AND IMAGE SEQUENCES

author: Lev Manovich version: 3/30/2012. email: manovich@ucsd.edu lab: softwarestudies.com

Software used in the guide: ImageJ ImageMontage and ImageSlice plug-ins for ImageJ (Lev Manovich and Matias Giachino)

Intended audiences: - researchers in digital humanities, media studies, game studies, film and TV studies, art history, archeology, anthropology, and other academic disciplines which work with images and video; - designers, artists, and video makers interested to visualize patterns in their works; - everybody else works with image sequences or video.

Examples of the visualizations created with the techniques and tools described in this guide: Visualizing image and video collections: examples

CONTENTS:
INTRODUCTION Image sequences and video Theory Examples IMAGEJ SOFTWARE Install ImageJ Change imageJ default memory setting WORKING WITH IMAGE SEQUENCES AND VIDEO IN IMAGEJ Prepare video files for visualization

Automatically detect shots in a video How to scale all images in a folder and/or change their format IMAGEJ STACK FORMAT ImageJ uses stack format to work with image sequences Import a sequence of same size images into ImageJ VISUALIZING IMAGE SEQUENCES WITH IMAGEJ STACK COMMANDS Create montage visualization Create slice visualization Create image average visualization MORE VISUALIZATION OPTIONS - IMAGEJ MACROS AND PLUGINS Visualize sets of images which do not have the same size or located in multiple folders Image_Montage and Image_Slice macros Organize a visualization according to image attributes Generate a list of image file paths using Unix commands Additional ImageJ commands and plugins for creating media visualizations

INTRODUCTION Image Sequences Many cultural artifacts and processes exists as, or produce sequences of images. For example, films and video consists from shots and frames; comics and manga consists from sequential pages; printed newspapers consist from pages. The software in your digital camera or phone displays the images you captured as a sequence sorted by dates and times. Other examples of image sequences may include all paintings created by an artist during her career, or all covers of a magazine published over many decades. (Here is a concrete example: 4535 covers of Time magazine from 1923 to 2009 visualized using our tools described below.) The standard information visualization techniques such as line graphs, sparklines, and 1D timelines can visualize metadata about such image sequences (for example, publication or capture dates, tags describing their content, and so on.) The problems with this familiar method is that such metadata is not available, and adding metadata

takes considerable time. More importantly, the metadata which uses language cant adequately capture the nuances of the visual attributes of images and video. Editing, animation and compositing applications such as Final Cut, Premiere and After Effects use a different method for representing image sequences. As the user zooms in the timeline, the display of video tracks as bars is replaced by the sequences of small keyframes. Instead of seeing only metadata about the video (the names of different video clips, their duration, etc.), we see actual sampled frames from this video. This interface is an example of a more general method for visualizing image sequences which I call media visualization. My article Media Visualization: Visual Techniques for Exploring Large Media Collections describes a few media visualization techniques we found to be particularly useful for working with image sequences in our lab softwarestudies.com. You can see examples of application of these techniques to various types of media in our Flickr galleries and projects descriptions on softwarestudies.com. The examples of image sequences we visualized include sets of paintings, photographs, comics, pages of manga titles, magazine covers and pages, newspaper pages, and sets of frames of feature films, animation, motion graphics, and TV programs.

This guide will teach you how to create media visualizations using the same tools we employ in our lab - popular free open source application ImageJ and the macros we develop to supplement its existing commands. (In addition to the techniques described below, we also developed another technique which we call image plot. We offer a free ImagePlot macro for ImageJ. Because ImagePlot has a separate detailed documentation, we will not describe it further in this guide.)

Theory Lev Manovich (2010). What is Visualization? Media visualization vs. information visualization; important examples of media visualizations done by artists. Lev Manovich (2011). Media Visualization: Visual Techniques for Exploring Large Media Collections. Methods and techniques for media visualization; analysis of our labs digital humanities projects that use media visualization.

Examples

Some of the projects created by our lab researchers, visitors and students that use visualizations of image sequences and video: Science and Popular Science magazines (magazine pages) Time magazine covers 1923-2009 (magazine covers) Motion studies (films, cartoons, TV commercials, motion graphics) Dziga Vertovs films comparisons (films) Kingdom Hears videogame play (videogames) YouTube remix (user-generated video remixes) Digging Into Global News (political online video) The N^3 Report (TV news)

IMAGEJ SOFTWARE

Install ImageJ: 1. Download ImageJ software from its web site. 2. Spend a few minutes to familiarize yourself with ImageJ interface by checking the following entries in the Basic Concepts section of ImageJs online documentation: Windows, Toolbar, Status Bar, Progress Bar and Images. 3. Because media visualizations can take lots of computer memory (RAM), you should change the default memory setting of the ImageJ program before proceeding with this tutorial. Follow the instructions in Change ImageJ Default Memory Setting below. If you using the 32-bit version of ImageJ, the maximum amount of RAM it can use is 1.7 GB. If you are using 64bit version of the program, you can allocate more than 1.7 GB. We recommend that you use 64-bit version, unless it does not run on your computer. For example, if your computer has 4 GB, allocate 3 GB to ImageJ.

(The donwnload page for the ImageJ software contains detailed instructions on using different versions and trouble shooting.)

Change imageJ Default Memory Setting: Follow these steps to change ImageJ default memory setting: 1. Select Edit > Options > Memory & Threads.... in ImageJ top menu. if this menu is greyed out, click on ImageJ toolbox (it is the narrow horizontal window with icons for various tools.) 2. Type the new memory setting into Maximum Memory box. Memory is specified in megabytes. For example, to allow ImageJ to use 3 GB of RAM, type 3000. ImageJ documentation recommends setting the amount of memory for the program at maximum %70 of the physical RAM. Thus, if your computer has 4 GB RAM, change the settings to 3000; if it has 8 GB of RAM, you can change it to 5600, and so on. For more details, see ImageJ documentation. 3. Restart ImageJ. ImageJ will save the new setting and will use it every time from now when you run it, so you dont need to change it again. If you want to use ImagePlot on a different computer, and you download a new copy of ImageJ program from its web site, you will need to change its memory setting before creating high resolution visualizations using the same steps.

WORKING WITH IMAGE SEQUENCES AND VIDEO IN IMAGEJ In this section, we will use a term image sequence to refer to number of images which are: 1) Located in a single directory on your computer. 2) Have sequential names, e.g. 0001.jpg, 0002.jpg, etc. 3) Have the same size (i.e., same width and height). (Later we will learn how to use images which may be located in different directories, and also have different sizes.)

Preparing Video Files for Visualization:

if you want to create media visualizations of a video file, you need to first export it as a sequence of single frames. You can do it using QuickTime Player, or any video editing or animation program. When you donload ImageJ, the package includes 32-bit and 64-bit versions of the program. You can read video and export it as an image sequence in ImageJ 32-bit version - but not in 64-bit version. Therefore, First use a 32-bit program version to do the exporting, and then switch to a 64-bit version to work with an exported image sequence. To open a video in ImageJ 32: File > Import > Using QuickTime... Once you loaded video, you can save it as an image sequence: save as image sequence (ImageJ documentation section) Important an image sequence in ImageJ 64: save as image sequence (ImageJ documentation section)

The rest of the guide assumes that you exported your videos as image sequences. Since the input to ImageJ commands and macros is always a sequence of separate images (which may represent video frames, artists paintings, magazine covers, newspaper pages, etc.), we will not be mentioning video explicitly in the rest of the guide. All techniques described below will work on any image sequence, regardless of whether it represents video frames or comes from another source.

Automatically detecting shots in a video: You can use free shotdetect software to automatically detect shot boundaries in your video. (Note: this software works on a video file, and not an image sequence). The software outputs a list of shots which can be then used in visualizing video. For instance, you can use this list to generate a data file containing only first frame of each shot, or a middle frame, or a set of frames. This file can be then used with ImageMontage or ImageSlice macros to visualize the selected frames. ( Generating a list of image filepaths with Unix commands section below provides details on the format of such data files.) The accuracy of the shot detection depends on your choice of program settings and the kinds of video you are analyzing. You should not expect %100 correct results. If you want to make sure that your file contains every shot in your video file, change shotdetect settings so it finds some false positives (i.e. it identifies as new shot the moments of

visual change in a video which are technically not new shots), and then go though the results manually and delete these false results.

How to scale all images in a folder and/or change their format: If you working with large image sequences, you often may want to convert files to a different format . For example, you can convert TIFF images into JPEG to save space. You may also want to scale images to a smaller size to speed up subsequent visualization rendering. To accomplish this, use this ImageJ command: Process > Batch > Convert Note: for higher quality results, use Interpollation:Bicubic option.

You can also use Photoshop: File > Scripts > Image Processor... (Photoshop has more options for scaling individual images.)

Free ImageMagic software offers many additional options and controls for working with image sequences and video, so if you cant figure out how to accomplish what you need with ImageJ or Photoshop, try it next.

IMAGEJ STACK FORMAT

ImageJ uses stack format to work with image sequences: Documentation for ImageJ Stacks menu commands (When you open a video file in Image 32, it is represented as a stack.)

To import a sequence of images into ImageJ: File > Import > Image Sequence.. Note 1: You can only use this command if all images have the same width and height.

Note 2: Depending on the size of an image sequence, you may have to use virtual stack option to import it. Virtual stack option allows you to work with very long image sequences - for instance you can import a whole feature film. However, when you run various ImageJ commands on a virtual stack, they would run more slowly. The reason: With a virtual stack, the actual images remain on the computer drive, and only pointers to these frames are loaded. Therefore, unless the size of image sequence prevents this, we recommend that you import your image sequence normally. However, with big image sequences, virtual stack is the only option. Note 3: When you use this command, you can set its options to sample your image sequences, importing only some of the images. This is very useful especially for the initial exploration of a really long sequence. You can for instance only load every 10th image, every 50th image, etc. This would allow you to load a very long sequence without using virtual stack option. After you imported an image sequence into ImageJ as a stack, you can use a number of ImageJ commands to create media visualizations.

VISUALIZING IMAGE SEQUENCES WITH IMAGEJ STACK COMMANDS

To create montage visualization: Image > Stacks > Make Montage.. visualization example: montage of 535 Time magzine covers, 1923-2009

To create slice visualization: Image > Stacks > Orthogonal Views.. visualization example: slice of 4535 Time magazine covers, 1823-2009

To create image average visualization: Image > Stacks > Z-project..

visualization example: average of Man with a Movie Camera frames

MORE VISUALIZATION OPTIONS - IMAGEJ MACROS AND PLUGINS Visualize sets of images which do not have the same size or located in multiple folders: ImageJ build-in commands which operate on image sets using Stack format have limitations. While video frames or magazine pages have the same size, many other types of image sequences do not: think, for example, of all paintings by an artist such as van Gogh, or user-generated images on deviantArt. If the images in you set do not have the same size, you cant directly import them in imageJ as a stack to make a montage and a slice visualizations.

Image_Montage and Image_Slice macros: (Note: The operations described below can be also carried out using various free software tools - Unix scripting, ffmpeg, ImageMagic, files rename utilities, Photoshop scripting, Applescript, etc.. Here we show how to perform them using ImageJ built-in commands and our custom ImageJ macros.)

To deal with these limitations, we wrote two custom ImageJ macros: ImageMontage ImageSlice These macros do not require the input images to be the same size. The images also do not have to be inside a single folder - they can be in multiple folders across your hard drive, and as long as you specify their file paths in a data file, you can show all of them in a single visualization. And because the macros do not need to load images into a stack first, they can create visualizations from very image sets. The macros also offer a number of options for customizing your visualizations. See the documentation of the macros for details.

Generating a list of image filepaths with Unix commands

Both ImageMontage and ImageSlice macros allow you to specify the input using Image Source option: Folder - uses all the images in a specified folder. Folder and Subfolders - uses all the images in both the specified folder and all its subfolders. Data File - lets you specify a tab-delimited text file with a column of image filenames. Data File as Filepaths - works like the Data File option but expects absolute paths to the files, instead of just filenames.

To use Data File and Data File as Filepaths options, you need to know how to create the appropriate data files containing image filenames or absolute file paths. To create the list of image filenames located in a single folder, open this directory, select all files, and then paste this list into Excel or a text editor. Save the new file using tabdelimited format (.txt). If you need to add absolute file paths, you can also do this in Excel (or another spreadsheet program.) If your images are located across multiple folders, you can move them all inside a single master folder, and then use Folder and Subfolders option.

Alternatively, you can use basic Unix commands to create a list containing full file paths for each image. (Unix is built-in into Mac OS). For example, lets say that I have a number of images which are digitized pages of a popular manga title Vampire Knight. The images for each chapter are in its own folder. To generate a data file containing a list of file paths to all images in all chapters, I would do the following:

1) Open Terminal application which comes with your Mac. 2) Use the following UNIX commands to navigate to a folder containing images in the first chapter folder, and display the full file path to these images (see any of UNIX documentation guides on the web for description of these commands):

cd directory - change current directory. ls - lists your files. ls -l - lists your files in 'long format' pwd - tells you where you currently are in the hierarchy of directories on your computer. 3) Lets assume that pwd (print working directory) shows the following path to the file 0001.jpg inside the first chapter folder is the following: /Users/manovich/Documents/Vampire_Knight_chapters/Vampire_K night_ch_1/0001.jpg Navigate back to the Vampire_Knight_chapters folder, and use this command sequence: ls */*/* > list.txt This command sequence generates list.txt which contains the file paths for all images in all chapter folders: /Users/manovich/Documents/Vampire_Knight_chapters/Vampire_K night_ch_1/0001.jpg /Users/manovich/Documents/Vampire_Knight_chapters/Vampire_K night_ch_1/0002.jpg ... /Users/manovich/Documents/Vampire_Knight_chapters//Vampire_ Knight_ch_2/0001.jpg Now you can use this data file as the input to the Image_Montage or Image_Slice macro to visualize all images in all chapter folders together.

Organizing media visualization according to image attributes: When you visualize an image sequence using ImageJ built-in commands, the order of images is determined by their filenames. If you saved a video as an image sequence, their file names automatically reflect the order of frames in the video. In other cases, you can rename the files according to a sequence provided in metadata (such as newspaper front pages publication dates).

However, reordering images in different ways is also very useful in exploring the patterns in a sequence. One of the ways to do this is to automatically measure various visual properties of images, and then sort the images using the values of these properties. For example, images assembled in a montage or a stack visualization can be organized by their gray scale mean, gray scale standard deviation, number of shapes, color saturation, number of edges, etc. ImageJ has a built-in command which allows you to automatically measure a number of properties of all images in a folder: 1) Select the features you want to measure (such as mean and standard deviation) using Analyze > Set Measurements... Make sure to check display label option, so image filanames will be included in the output. 2) Run the command below, and select the folder containing all images you want to measure: Process > Batch > Measure.. The measurements will be printed in Results window. After the command is finished, you can save the contents of this window into a text file using File > Save as...

ImageJ visualization commands which use Stack format cant directly sort images since ImageJ loads images in a stack according to their file names. However, if you use Image_Montage or Image_Slice, you can specify the exact order of filenames yourself. This allows you to experiment with different orders (without having to rename the actual image files.) Open the text file created by batch measurement command in Excel, and sort it according to the column containing the measurement you want. This will also sort the filenames. Save the new file, and use it as the input to the macros. See ImageJ manual for the description of features which can be measured with Measure command. Bach measurements can also be used to visually explore image sets using image plot visualization technique available in our ImagePlot macro. ImagePlot package includes two additional macros which allow you to measure basic properties of color images, and also automatically find and measure separate shapes in an image.

Additional ImageJ commands and plugins for working with stacks and creating media visualizations if you explore ImageJ buit-in commands and plugins, you will more possibilities for measuring images, working with stacks, and creating media visualizations. Here are a few of our favorites: 3D color inspector/color histogram slice remover substack select concatenate images or stacks stack normalizer Custom montage