Feb 23: Movies
Getting going
Video is supported by functions in Processing libraries, instead of by "built-in" functions. To be able to use the functions in these libraries, the line import processing.video.*; must be at the top of the sketch. You can either just type that in, or select "video" from the "Sketch | Import Library ..." menu to have Processing insert it.
Processing uses QuickTime format for video; files end in ".mov". That means that Mac people shouldn't have to do anything to get going, but Windows people might. In particular, Windows people should make sure that QuickTime is installed, either on its own or via iTunes (which includes it).
Sketches using video don't run as applets, so you'll have to download the pde files and run them within the Processing environment.
As with an image, in order to give Processing access to a movie, it must be placed within the data folder, either manually or with the menu option Sketch | Add File. The examples here use the "station.mov" movie from the Processing example Libraries | Video (Movie) | Loop. To use a different movie file, make sure it's in QuickTime format, with extension ".mov". In either case, be sure to put the movie in the data folder.
Unfortunately, the use of QuickTime in Processing via a Java library has in recent versions become quite unstable. Sometimes a sketch will work fine, while other times strange errors will show up (at unpredictable points) and eventually kill off the sketch. The Processing discussion boards have various suggestions that sometimes help, such as including the additional parameter P3D (and I've seen P2D in the built-in examples) when setting the size of the window, as well as setting the window to be a bit larger than needed. Neither of those consistently helps me, so I just put up with having to re-start the sketch.
Reading
In Processing, there's a strong analogy between moving from static to dynamic sketches, and moving from images to movies. In particular, a movie is treated as nothing more than one image after another, presented each frame (at the movie's rate). So the only real question is how to read in the images; after that, we're just using them exactly as we did all last week.
To deal with a movie, we declare a global variable of type Movie at the top of the sketch. We create a new instance within setup(), since we only want to create it once. The first parameter to the constructor is "this" (i.e., the Processing applet itself) and the second is the filename.
Once a movie object is created, we start it playing either with the Movie.play() method (which plays it once) or the Movie.loop() method (which repeats ad infinitum, or until the Movie.noLoop() method is invoked).
There are two ways to deal with displaying the frames that come in from a playing movie, somewhat analogous to the two ways we deal with mouse presses. The first approach explicitly tests, within the draw() function, for the availability of another movie frame. If one is available, it reads it (using the Movie.read() method), at which point the movie variable can be treated exactly like a PImage. In the following, we simply display it, via image().
import processing.video.*; Movie station; void setup() { size(160,120); textFont(loadFont("CourierNewPSMT-24.vlw")); station = new Movie(this, "station.mov"); station.loop(); } void draw() { if (station.available()) { // a movie frame is available for reading station.read(); // read it image(station,0,0); // now treat the movie frame like an image text(station.time(),0,height); } }
[pde]
This sketch also uses the Movie.time() method to get the time (in seconds, as a float) of the current frame.
The second approach uses a separate function that we define, movieEvent(); this function gets called whenever a frame is available for reading. The movieEvent function takes a parameter that says which movie it is for which a frame is available; we then ask it to read a frame.
import processing.video.*; Movie station; void setup() { size(160,120,P2D); textFont(loadFont("CourierNewPSMT-24.vlw")); station = new Movie(this, "station.mov"); station.loop(); } void draw() { image(station,0,0); text(station.time(),0,height); } void movieEvent(Movie m) { m.read(); }
The Movie reference summarizes all the available methods, e.g., to pause or jump to a particular time.
Processing
That's all there is to the machinery. Once we've read in a movie frame, it's treated just like any PImage, and we can do all the stuff we've done with images, combined with all the stuff we've done with sketches in general. For example, the Processing example Libraries | Video (Movie) | Loop "draws" with a movie. That is, it puts the movie image wherever the mouse is.
The example Libraries | Video (Movie) | Pixelate renders each frame with rectangles; I devised a simpler version shown below. It's just like what we'd do with an image, but the images come in frame-by-frame as the movie is read. I added one twist: the mouse allows panning across the image. We use our usual formula for converting mouse position to fraction. We then scale up by how far over we could be (subtracting nx or ny, so that there are enough pixels left to cover the whole window), and add that to the x,y of the loop.
import processing.video.*; Movie station; int blockSize = 10; // how big the pixel rectangles are int nx, ny; // how many pixels to draw (calculated in setup() void setup() { size(640,480); nx = width / blockSize; ny = height / blockSize; station = new Movie(this, "station.mov"); station.loop(); frameRate(10); } void draw() { if (station.available()) { station.read(); station.loadPixels(); for (int y=0; y<ny; y++) { for (int x=0; x<nx; x++) { // use mouse to pan across underlying image int getx = x + (station.width-nx)*mouseX/width; int gety = y + (station.height-ny)*mouseY/height; fill(station.pixels[getx+gety*station.width]); rect(x*blockSize, y*blockSize, blockSize-1, blockSize-1); } } } }
[pde]
One thing to be careful of when manipulating pixels from a movie: the movie frame must be read in before the pixel array can be loaded. This is easy in if-available approach (which is why I used that). It can lead to a problem in the movieEvent approach if the draw() tries to access the first frame before it's been read.
Another example uses our pixel-level image manipulation to create a mirror image. We define a function that takes a PImage as a parameter (we call it from draw() with the current movie frame). It creates a new blank PImage with the function createImage(), specifying the size and color mode. It then sets the pixels of this new image as the mirror image of the original, and returns the new image to the caller of the function. The draw() function then checkerboards the original and the mirror image.
import processing.video.*; Movie station; void setup() { size(480,360); station = new Movie(this, "station.mov"); station.loop(); } void draw() { if (station.available()) { // a movie frame is available for reading station.read(); // read it station.loadPixels(); // Create a mirror image PImage mirr = mirror(station); // Checkerboard regular and mirror image for (int j=0; j<3; j++) for (int i=0; i<3; i++) if ((i+j)%2 == 0) image(station,i*station.width,j*station.height); else image(mirr,i*mirr.width,j*mirr.height); } } // Returns a new image that is the mirror image of img PImage mirror(PImage img) { PImage m = createImage(img.width, img.height, RGB); for (int y=0; y<img.height; y++) for (int x=0; x<img.width; x++) m.pixels[x+y*img.width] = img.pixels[(img.width-1)-x + y*img.width]; return m; }
[pde]
Since this is a movie, we might want to change the image according to when in the movie it is. For example, we can fade to black simply by adding the line tint(255*(1-station.time()/station.duration())); to the first sketch. (I found by trial and error that this doesn't work if the "P2D" parameter is included in the size() call.)
Shiffman 16-5 illustrates using the mouse position to jump.
The following sketch "unrolls" movie frames into separate thumbnail images on a grid. We keep track of where we are on the grid with variables x and y, for each frame incrementing x until it rolls over, at which point we increment y and roll it over as needed.
import processing.video.*; Movie station; int szx=40,szy=30; // how big to draw the frames int nx,ny; // number of frames across and down (computed in setup) int x=0,y=0; // current frame index on frame grid void setup() { size(800,600); nx = width/szx; ny = height/szy; station = new Movie(this, "station.mov"); station.loop(); } void draw() { if (station.available()) { // a movie frame is available for reading station.read(); // read it // draw it at the current position image(station,x*szx,y*szy,szx,szy); // advance the position, wrapping around in x and y x++; if (x==nx) { x=0; y++; if (y==ny) y=0; } } }
[pde]
Writing
Just as we can save a snapshot of the current window, we can save a movie whose frames are made from snapshots of the window. The MovieMaker class supports this functionality. There's not much to it. First we declare a global MovieMaker variable and create a new object in setup(). The constructor takes "this" again, along with the width, height, name of the output file, and some optional parameters about the movie encoding. Then any time we want to add a frame containing the current window, we invoke the MovieMaker.addFrame() method. When we're finished, we call the MovieMaker.finish() method to make sure the movie is cleanly saved.
One illustration is provided by Processing example Libraries | Video (Movie Maker) | Drawing Movie; another is on the MovieMaker documentation page. These just do standard mouse-based drawing, saving out a snapshot every frame. Warning: the default parameters (high quality) can produce pretty large files.