Animation merger

Animmerger converts a 2D animation from local-frame-of-reference into global-frame-of-reference. That is, for a movie that follows an actor around (and the background scrolls to follow them), it creates a movie that has a fixed background, and the camera moves around in the scene.

It does this with a motion detection algorithm, a set of different pixel methods, and a simulated infinite 2D canvas — a 2D canvas that extends infinitely to all four directions (well, as infinite as 32-bit integers can get…)

As a sample, here is the original animation (712100 bytes). The animation was created literally by taking a screenshot from the NES emulator every frame. (I hacked the emulator to automatically produce a screenshot after every frame.) What follows below, is a list of the pixel methods supported by animmerger, along with example images demonstrating what that pixel method does. The -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF parameter was given to animmerger to remove the HUD that is 256x16 wide and begins at 0,8. The hexadecimal numbers listed are these colors: 020202 A64010 D09030 006E84 511800 FFFFFF This removes the text (white) as well as the blinking coin.

The graphics material comes from Super Mario Bros.
Mario, Super Mario Bros., and The Nintendo Entertainment System (NES) are registered trademarks of Nintendo of America Inc. But you knew that, right?

The "average" method produces a "motion blur" effect of the entire input, reducing it into a single frame.

You can see a faint trace of all animated actors that appeared in the animation. Mario moved very fast so his trace is quite difficult to spot.

Produced with commandline:
# animmerger -pa snaps/*.png -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF
# mv tile-0000.png demo/method-a.png

An alternative implementation of "average" is also provided: "tinyaverage" (option -A). It requires less memory to store, but is less accurate to calculate.

If you want the color averages to be calculated through the YUV colorspace rather than the RGB colorspace, add the --yuv option (not supported by tinyaverage).

The "actionavg" method attempts to fix the faintness problem with "average" method by keeping track separately of the background (using the "mostused" method) and adding it only once to the average of moving actors.

Produced with commandline:
# animmerger -pt snaps/*.png -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF
# mv tile-0000.png demo/method-t.png

If you want the color averages to be calculated through the YUV colorspace rather than the RGB colorspace, add the --yuv option.

The "most used" method produces what might be the background image for the entire animation.

Note: If there is an actor that sits in a certain location for a long time, it is also recorded.
In this example, there were none though.
This mode does not thus remove all actors, but it does remove anything that wanders around.

Produced with commandline:
# animmerger -pm snaps/*.png -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF
# mv tile-0000.png demo/method-m.png

The "last" method is a simpler implementation of the MostUsed method, simply recording the last pixel value in any location.

Produced with commandline:
# animmerger -pl snaps/*.png -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF
# mv tile-0000.png demo/method-l.png

The "first" method is analogous to "last". It shows whatever first appeared in a particular pixel location.

The turtles are distorted, because they moved while the screen scrolled.
It is the same effect as if you move the paper in a desktop scanner during the scanning.

Produced with commandline:
# animmerger -pf snaps/*.png -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF
# mv tile-0000.png demo/method-f.png

The "solid" method is an experimental light-weight replacement to the "mostused" method. It simply ignores anything that moves and retains whatever stays still for the longest time.
Unlike "mostused", it does not sum separate appearances together; it only finds the maximum length of consecutive sameness.

As seen here, it has shortcomings, too.

Produced with commandline:
# animmerger -pO snaps/*.png -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF
# mv tile-0000.png demo/method-O.png

The "firstnmost" method is analogous to "first" and "mostused"; it chooses the most common pixel of first N pixel values. Set N with the --firstlast (-f) option.
If N is 0, instead gets last uncommon pixel.
If N is negative, using least common values rather than most common.

Most common of first 4:

Most common of first 10:

Most common of first 16:

First uncommon:

Least common of first 10:

Produced with commandline:
# for f in 4 10 -10 16 0; do
#   animmerger -pF -f$f snaps/*.png -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF
#   mv tile-0000.png demo/method-Ff$f.png
# done

The "lastnmost" method is analogous to "last" and "mostused"; it chooses the most common pixel of last N pixel values. Set N with the --firstlast (-f) option.
If N is 0, instead gets last uncommon pixel.
If N is negative, using least common values rather than most common.

Most common of last 10:

Last uncommon:

Least common of last 10:

Produced with commandline:
# for f in 4 10 -10 16 0; do
#   animmerger -pL -f$f snaps/*.png -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF
#   mv tile-0000.png demo/method-Lf$f.png
# done

The "least used" method is analogous to "most used".
It can be used to find graphical artifacts and teleporting actors, but for the most part, the output is not very useful.

Produced with commandline:
# animmerger -pe snaps/*.png -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF
# mv tile-0000.png demo/method-e.png

The "changelog" method records the entire animation (121995 bytes in this example).
It takes considerably less disk space (and is faster to load) than the original animation, because now the background does not scroll.

You see some artifacts in the turtle and in Mario when they appear near the top of the screen. This is because they were behind the HUD (the text "WORLD 8-2" for instance), which was removed.
In the case of the turtle, the turtle's white pixels were also removed, because the HUD removal was based on color as well as coordinates.
Horizontal disappearance of the actors is because of the viewport scrolling past them. They do not exist outside those parameters in the original animation either.

Here is how the animation looks like, if the HUD is not removed. (246643 bytes)

Exteriors, i.e. content outside the "current" viewport of the animation are colored as in the MostUsed pixel method.
This is evident in the trails left by the HUD as it scrolls by at different speeds.

Produced with commandline:
# rm tile-*.png tile-*.gif
# animmerger --gif -pc snaps/*.png -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF
# gifsicle -O2 -o demo/method-c.gif -l0 -d3 tile-*.gif
The version with HUD intact was created with the same commandline, except with the -m option removed.

The background for ChangeLog is normally generated with the MostUsed method, but it can be explicitly controlled with the --bgmethod0 and --bgmethod1 options.
Here is how the above animation (HUD-less) looks like with --bgmethod0 first --bgmethod1 last:

Note that the --bgmethod0 and --bgmethod1 options only affect the ChangeLog method, and only when motion blur is not used.

The changelog method also supports motion blur. Use the --motionblur (-B) option to set it. Value 0 disables motion blur (default: 0).

Blur length 1:

Blur length 4:

Blur length 20:

Produced with commandline:
# for b in 1 4 20;do
#   rm tile-*.png tile-*.gif
#   animmerger --gif -B$b -pc snaps/*.png -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF
#   gifsicle -O2 -o demo/method-cB"$b".gif -l0 -d3 tile-*.gif
# done

The "loopinglog" method records the entire animation, but reuses existing frames. Use the -l option to set the loop length in frames.
The smaller value you use, the shorter the animation is in the number of frames, but the more pronounced is the "lemmings" effect.

30 frames (94895 bytes):

10 frames (66738 bytes):

4 frames (40372 bytes):

Produced with commandline:
# for l in 4 10 30; do
#   rm tile-*.png tile-*.gif
#   animmerger --gif -l$l -po snaps/*.png -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF
#   gifsicle -O2 -o demo/method-sl"$l".gif -l0 -d3 tile-*.gif
# done

It is also called "loopinglast" mode (option -s) to differentiate from "loopingavg".

The loopinglog method also supports motion blur. Use the --motionblur (-B) option to set it. Value 0 disables motion blur (default: 0).

The "loopingavg" method combines the "loopinglog" and "actionavg" methods. Use the -l option to set the loop length in frames.
The most important difference to "loopinglog" is that overlapping action is averaged rather than explicitly choosing one of the acting pixels.
It looks slightly better, but may require GIF palette reduction.
In comparison, "loopinglog" only uses pixel colors present in the original images.

30 frames (file size depends on selected palette size):

10 frames:

4 frames:

Produced with commandline:
# for l in 4 10 30 80; do
#   rm tile-*.png tile-*.gif
#   animmerger --gif -l$l -pv snaps/*.png -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF
#   gifsicle -O2 -k128 -o demo/method-ov"$l".gif -l0 -d3 tile-*.gif
# done

If you want the color averages to be calculated through the YUV colorspace rather than the RGB colorspace, add the --yuv option.
In most cases, the difference is neglible though.

The loopingavg method also supports motion blur. Use the --motionblur (-B) option to set it. Value 0 disables motion blur (default: 0).

Loop length 30 frames, blur length 20:

Loop length 30 frames, blur length 20, with YUV calculations:

Loop length 30 frames, blur length 20, with YUV calculations, and diversity-quantized palette of 16 colors:

Loop length 10 frames, blur length 4:

Produced with commandline:
# for b in 4 8 20;do
#   for l in 10 30;do
#     rm tile-*.png tile-*.gif
#     animmerger --gif -B$b -l$l -pl snaps/*.png -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF
#     gifsicle -O2 -o demo/method-vl"$l"B"$b".gif -l0 -d3 tile-*.gif
#   done
# done

Method name	Static or animated	Composes new colors	Obeys YUV option	Memory size per pixel*	Primary use
First	Static	No	No	4	Maps
Last	Static	No	No	4
FirstNMost	Static	No	No	As ChangeLog
· FirstUncommon	Static	No	No	As ChangeLog
· FirstNLeast	Static	No	No	As ChangeLog
LastNMost	Static	No	No	As ChangeLog
· LastUncommon	Static	No	No	As ChangeLog
· LastNLeast	Static	No	No	As ChangeLog
MostUsed	Static	No	No	12…16 + 6*number of unique colors	Maps
LeastUsed	Static	No	No	As MostUsed
Average	Static	Yes	Yes	16
Tinyaverage	Static	Yes	No	8
ActionAvg	Static	Yes	Yes	As MostUsed
ChangeLog	Animated (movie)	If blur	For blur	12…16 + 8*number of content changes	Animations
LoopingLog	Animated (loop)	If blur	For blur	As ChangeLog
LoopingAvg	Animated (loop)	Yes	Yes	As ChangeLog	Fun

*) These numbers are estimates. Actual memory size per pixel depends on the exact selection of pixel methods requested and the memory allocation overhead. Animmerger strives to always select the smallest combination of pixel methods (memoryconsumptionwise) that can implement all the requested methods.

Masked areas can be removed with a number of different methods. To best demonstrate them, I added an extra huge mask in the middle of the image.
It is best seen in the "black" masking, below.

These images were produced with this commandline:
# for method in censor hole interpolate extrapolate; do
#   rm *-*.gif *-*.png
#   ./animmerger -r4,4 --mvrange 0,0,4,0 --bgmethod0=first --bgmethod1=last \
#   -u$method -p* pano3/*.png \
#   -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF \
#   -m3,128,250,72 -m0,73,256,2
#   gifsicle -O2 -o demo/mask-$method.gif -l0 -d3 ChangeLog-*.gif
#   cp -p Average-0000.png demo/mask-$method.png
# done

This method shows clearly which areas were affected by the mask. Specifically, the HUD, and a huge rectangle, and a narrower line extending from the very left edge to the very right edge of the screen at all times, effectively blocking the contents of the entire scanline from ever being seen.

Animation:

Averaged:

This method is what animmerger does by default. The transparent regions are simply treated as holes; there is no content on the affected areas. If the hidden content becomes available when the camera moves, then those pixels are recorded.

Animation:

Averaged:

This method removes the content with a circular blur pattern. The method is almost identical to the delogo filter that can be used in MPlayer to remove a tv station logo from video. Content that coindices with the removed part is replaced with interpolated surrounding pixels; original pixels of the affected area are not sampled.

Animation (palette-reduced and dithered with -Qd,16 in order to make the 1.5 MB GIF file smaller):

Averaged:

The extrapolate filter tries to extrapolate the content of the masked areas by detecting repeating tile patterns outside the masked area, and extrapolating those patterns over the masked area. The results of this method vary a lot from frame to frame, so it is not very suitable to be used over large unknown areas. For small areas, it works nicely.

Note that this algorithm is rather slow on large areas like this.

Animation:

Averaged:

Animations are always created as GIF files.
GIF files however are limited to a palette of 256 colors, while it is possible that animmerger creates images with more colors than 256. Therefore, the colormap must be reduced before the GIF image can be generated. animmerger supports a number of different color reduction methods, which are listed below.
If no method is chosen, whatever is libGD default will be used.

The images in this section were generated by making a 30-frame LoopingAvg animation with blur length of 20, rendering it with different palettization parameters and picking the 11th frame.

The exact commandline to produce the images was:
# for m in m d b o q; do
#  for q in 2 4 8 16 32 64 128; do
#   rm tile-*.png tile-*.gif
#   ./animmerger --gif --yuv -Q"$m",$q -B20 -l30 -pv pano3/*.png -m0,8,256,16,020202,A64010,D09030,006E84,511800,FFFFFF
#   gifsicle -O2 -k128 -o demo/method-vl30yB20Q"$m"$q.gif -l0 -d3 tile-*.gif
#   convert tile-0010.gif -quality 100 demo/dither-"$c"$q.png
#  done
# done

Palette reduction methods can be chained in order to take benefits of the differently-appearing strengths of the different methods, but in this test set, each method was used alone.

When palette reduction methods have been explicitly selected, animmerger always uses an ordered-dithering method (crosshatch artifacts) to optimize the rendering. This is better for animation than other methods such as Floyd-Steinberg are, because the dithering patterns do not jitter between frames. Currently it is not possible to disable this dithering.

Heckbert median-cut quantization method repeatedly splits the palette into two roughly equal-proportion sections ("low" and "high" part in any of red/green/blue channels) until the desired number of sections have been generated; the palette is generated by averaging the values in each section together.
It is good at generating relevant palettes, but at smallest palette sizes, it suffers from a blurring problem.

4 colors:

8 colors:

16 colors:

32 colors:

Diversity quantization method alternates between choosing the most popular remaining color in the image for a "seed" and choosing of the remaining colors the one that is most distant to any colors selected so far.
The result is generally a very good representation of the original image's colors.
At the smallest palette sizes, the colors are of course off, but the contrast is still sharp.

4 colors:

8 colors:

16 colors:

32 colors:

The blend-diversity method is a variation to the diversity method; after the colors have been chosen, they are merged together with the remaining colors that are most similar to the chosen one.

4 colors:

8 colors:

16 colors:

32 colors:

The NeuQuant method, developed by Anthony Dekker in 1994, uses a Kohonen self-balancing neural network to quickly come up with an optimized palette. It is especially powerful with optimizing smooth gradients, such as the motion-blur trails in this pictureset.

4 colors:

8 colors:

16 colors:

32 colors:

Parallax motion is bad. When animating video game content, please ensure that all background layers are synchronized. Note that this will likely require you to hack the emulator that is used to produce the video frames.

If different background layers are moving at different speeds with respect to the camera, animmerger will sync into one of them (probably the one that occupies the largest screen area), and the rest will appear to be moving with respect to the chosen background.

Example:

This scene is from Super Mario World. The HUD layer is disabled, but otherwise it is an intact animation. The palette was reduced and fps halved to make the file slightly smaller for web distribution.
You can easily see the problem with parallax motion: Sometimes the image alignment syncs to the platforms, sometimes it syncs to the stalactite background. When it syncs to the platforms, the other background can be seeing moving as a distinct layer.	In this version, the background layers move in unison with respect to the camera. As such, the image alignment is perfect. This was achieved with the following patch to snes9x: --- ppu.cpp~ 2010-08-17 23:46:11.022843689 +0300 +++ ppu.cpp 2010-08-17 22:34:52.000000000 +0300 @@ -416,2 +416,3 @@ PPU.BG[0].HOffset = (Byte<<8) | PPU.BGnxOFSbyte; + PPU.BG[1].HOffset = (Byte<<8) | PPU.BGnxOFSbyte; PPU.BGnxOFSbyte = Byte; @@ -423,2 +424,3 @@ PPU.BG[0].VOffset = (Byte<<8) | PPU.BGnxOFSbyte; + PPU.BG[1].VOffset = (Byte<<8) | PPU.BGnxOFSbyte; PPU.BGnxOFSbyte = Byte; @@ -429,3 +431,3 @@ case 0x210F: - PPU.BG[1].HOffset = (Byte<<8) | PPU.BGnxOFSbyte; + //PPU.BG[1].HOffset = (Byte<<8) | PPU.BGnxOFSbyte; PPU.BGnxOFSbyte = Byte; @@ -436,3 +438,3 @@ case 0x2110: - PPU.BG[1].VOffset = (Byte<<8) | PPU.BGnxOFSbyte; + //PPU.BG[1].VOffset = (Byte<<8) | PPU.BGnxOFSbyte; PPU.BGnxOFSbyte = Byte;

The commands used to produce these animations were:
# rm tile-*.gif; animmerger -v -r12x12 -bl -pc -a -4,-3,6,9 pano5/*.png
# rm tile-???[13579].gif # Delete 50% of frames to reduce fps in half
# gifsicle -O2 -o demo/pano5-cl.gif --crop 8,8+480x400 --use-colormap smwpalette.gif -l0 -d6 tile-????.gif
# rm tile-*.gif; animmerger -v -r12x12 -bl -pc -a -4,-3,6,9 pano5b/*.png
# rm tile-???[13579].gif # Delete 50% of frames to reduce fps in half
# gifsicle -O2 -o demo/pano5-fl.gif --crop 8,8+480x400 --use-colormap smwpalette.gif -l0 -d6 tile-????.gif

The palette file was customized by hand, by taking a representative snapshot of the movie, and then progressively merging near-identical entries in the colormap in GIMP manually until only the minimal set of unique colors/tones remain.

The image aligning engine is confused by anything that globally changes the screen brightness. This includes any pain-red-tinting, white-explosion flashes, fog clouds, etc. Please try to avoid them.

Example:
TODO: Add successful Super Metroid animation

TODO: Add example of how image alignment suffers when using the power bomb in Super Metroid

animmerger v1.4.1 - Copyright (C) 2010 Joel Yliluoma (http://iki.fi/bisqwit/)

Usage: animmerger [<options>] <imagefile> [<...>]

Merges animation frames together with motion shifting.

 --help, -h             This help
 --mask, -m <defs>      Define a mask, see instructions below
 --method, -p <mode>    Select pixel type, see below
 --bgmethod, -b <mode>  Select pixel type for alignment tests
 --looplength, -l <int> Set loop length for the LOOPINGxx modes
 --motionblur, -B <int> Set motion blur length for animated modes
 --firstlast, -f <int>  Set threshold for xxNMOST modes
 --version, -V          Displays version information
 --yuv, -y              Sets YUV mode for average-color calculations
                        Affects AVERAGE, ACTIONAVG and LOOPINGAVG.
 --refscale, -r <x>,<y>
     Change the grid size that controls
     how many samples are taken from the background image
     for comparing with the input image, for image alignment.
     Smaller grid = more accurate but slower aligning.
     Default: -r32,32
     Set to e.g. -r8,8 if you experience misalignment problems.
 --mvrange, -a <xmin>,<ymin>,<xmax>,<ymax>
     Change the limits of motion vectors.
     Default: -9999,-9999,9999,9999
     Example: --mvrange -4,0,4,0 specifies that the screen may
     only scroll horizontally and by 4 pixels at most per frame.
 --gif, -g              Save GIF frames instead of PNG frames
 --verbose, -v          Increase verbosity

animmerger will always output PNG files into the current
working directory, with the filename pattern tile-####.png
where #### is a sequential number beginning from 0000.

AVAILABLE PIXEL TYPES

  AVERAGE, long option: --method=average , short option: -pa
     Produces a single image. Each pixel
     is the average of all frames addressing that pixel.
  TINYAVERAGE, long option: --method=tinyaverage , short option: -pA
     A less accurate but more space-efficient version of "average".
  LAST, long option: --method=last , short option: -pl
     Produces a single image. Each pixel
     records the latest color addressing that pixel.
  FIRST, long option: --method=first , short option: -pf
     Produces a single image. Each pixel
     records whatever first appeared in that spot.
  MOSTUSED, long option: --method=mostused, short option: -pm
     Produces a single image. Each pixel
     records the color that most often occured in that location.
     Use this option for making maps!
  LEASTUSED, long option: --method=leastused, short option: -pe
     Produces a single image. Each pixel
     records the color that least commonly occured in that location.
  LASTNMOST, long option: --method=lastnmost, short option: -pL
     Combines "mostused" and "last". Set threshold using
     the -f option. Example: -f16 -pL = most used of last 16 pixels.
     If -f0, then selects the last not-common pixel value.
     If -f value is negative, uses leastused instead of mostused.
  FIRSTNMOST, long option: --method=firstnmost, short option: -pF
     Combines "mostused" and "first". Set threshold using
     the -f option. Example: -f16 -pF = most used of first 16 pixels.
     If -f0, then selects the first not-common pixel value.
     If -f value is negative, uses leastused instead of mostused.
  ACTIONAVG, long option: --method=actionavg, short option: -pt
     Similar to average, except that blurring of actors
     over the background is avoided.
  CHANGELOG, long option: --method=changelog, short option: -pc
     Produces an animation. Also supports motion blur.
  LOOPINGLOG, long option: --methods=loopinglog, short option: -po
     Produces a time-restricted animation.
     Also called, "lemmings mode".
     Use the -l option to set loop length in frames. Supports motion blur.
  LOOPINGAVG, long option: --methods=loopingavg, short option: -pv
     A combination of loopinglog and actionavg, also supports motion blur.

DEFINING MASKS

  You can use masks to block out HUD / splitscreens
  so that it will not intervene with the animation.
  To define mask, use the --mask option, or -m for short.
  Mask syntax: x1,y1,width,height,colors
  Examples:
    -m0,0,256,32
       Mask out a 256x32 wide section at the top of screen
    -m0,0,256,32,FFFFFF
       From the 256x32 wide section at the top of screen,
       mask out those pixels whose color is white (#FFFFFF)
    -m16,16,8,40,000000,483D8B
       From the 8x40 wide section at coordinates 16x16,
       mask out those pixels whose color is either
       black (#000000) or dark slate blue (#483D8B)

TIPS

Converting a GIF animation into individual frame files:
   gifsicle -U -E animation.gif
   animmerger <...> animation.gif.*

To create images with multiple methods in succession,
you can use the multimode option. For example,
    --method average,last,mostused, or -pa,l,m
creates three images, corresponding to that if
you ran animmerger with -pa, -pl, -pm options
in succession. Note that all modes share the same
other parameters (firstlast, looplength).
The benefit in doing this is that the image alignment
phase needs only be done once.

Different combinations of pixel methods require different
amounts of memory. Use the -v option to see how much memory
is required per pixel when using different options.
animmerger always strives to choose the smallest pixel
implementation that provides all of the requested features.

When creating animations of video game content, please take
all necessary steps to ensure that background stays immobile
while characters move. Parallax animation is bad; If possible,
please fix all background layers so that they scroll at even
rate.

animmerger has been written by Joel Yliluoma, a.k.a. Bisqwit,
and is distributed under the terms of the General Public License (GPL).

GNU make and C++ compiler is required to recompile the source code.
libgd is also required.

• pngout for making PNG files considerably smaller.
• gifsicle for creating and optimizing GIF animations

The official home page of animmerger is at http://iki.fi/bisqwit/source/animmerger.html.
Check there for new versions.

Additionally, the most recent source code (bleeding edge) for animmerger can also be downloaded by cloning the Git repository by:

• git clone git://bisqwit.iki.fi/animmerger.git
• git checkout origin/release -b release
• git checkout origin/master -b master