Feb 18: Video
Setup
Video is supported by functions in Processing libraries, instead of by "built-in" functions. To be able to use the functions in these libraries, the line "import processing.video.*;" must be at the top of the sketch. You can either just type that in, or select "video" from the "Sketch | Import Library ..." menu to have Processing insert it.
Processing uses QuickTime format for video. That means that Mac people shouldn't have to do anything to get going, but Windows people might for reading and writing, and will for capture. Windows people: first make sure that QuickTime is installed on your machine. Either install it on its own, or install iTunes, which includes QuickTime. Now comes the tricky part, which only matters if you're going to do video capture with your own webcam. You need to install a video digitizer that will convert the webcam video into QuickTime. The Processing documentation points to WinVDIG; the original site has apparently gone away, but a nice person has kept a WinVDIG mirror. Download and install version 1.0.1 (apparently the other versions have some issues). If this becomes too painful, it might be a good opportunity to instead visit the Mac lab in Sudikoff.
More details, and some troubleshooting tips, are available in the Video library reference.
Reading
In this section, for simplicity I'll stick with the "station.mov" movie from the Processing example Libraries | Video (Movie) | Loop. To use a different movie file, make sure it's in QuickTime format, with extension ".mov". The examples here don't run as applets, so you'll have to download the pde files and run them within Processing.
Loading a movie is quite analogous to loading an image. The movie file must be placed within the data folder, either manually or with the menu option Sketch | Add File. Then declare a global variable of type Movie at the top of the sketch, and create a new instance within setup(). The first parameter to the constructor is "this" (i.e., the Processing applet itself) and the second is the filename.
Once the movie is loaded, start it playing either with the play() method (which plays it once) or the Movie.loop() method (which repeats ad infinitum, or until the Movie.noLoop() method is invoked). There are two ways to deal with displaying the frames that come in from a playing movie. The first approach explicitly tests, within the draw() function, for the availability of another movie frame. If one is available, it reads it (using the Movie.read() method), at which point the movie variable can be treated exactly like a PImage, and displayed (e.g., via image()).
import processing.video.*; Movie station; void setup() { size(160,120); textFont(loadFont("CourierNewPSMT-24.vlw")); station = new Movie(this, "station.mov"); station.loop(); } void draw() { if (station.available()) { // a movie frame is available for reading station.read(); // read it image(station,0,0); // now treat the movie frame like an image text(station.time(),0,height); } }
[pde]
This sketch also uses the Movie.time() method to get the time (in seconds, as a float) of the current frame.
The second approach uses a separate function that we define, movieEvent(); this function gets called whenever a frame is available for reading. (These two approaches are somewhat analaogous to our two ways of dealing with mouse presses.) The movieEvent function takes a parameter that says which movie it is for which a frame is available; we then ask it to read a frame.
import processing.video.*; Movie station; void setup() { size(160,120); textFont(loadFont("CourierNewPSMT-24.vlw")); station = new Movie(this, "station.mov"); station.loop(); } void draw() { image(station,0,0); text(station.time(),0,height); } void movieEvent(Movie m) { m.read(); }
The Movie reference summarizes other available methods, e.g., to pause or jump to a particular time.
That's all there is to the machinery. Once we've read in a movie frame, it's treated just like any PImage, and we can do all the stuff we've done with images, combined with all the stuff we've done with sketches in general. For example, try adding the line tint(255*(1-station.time()/station.duration())); to the above sketch.
The Processing example Libraries | Video (Movie) | Loop draws with a movie. The example Libraries | Video (Movie) | Pixelate renders each frame with rectangles; I devised a simpler version shown below. It's just like what we'd do with an image, but the images come in frame-by-frame as the movie is read.
import processing.video.*; Movie station; int blockSize = 10; // how big the pixel rectangles are int nx, ny; // how many pixels to draw (calculated in setup() void setup() { size(640, 480); nx = width / blockSize; ny = height / blockSize; station = new Movie(this, "station.mov"); station.loop(); } void draw() { if (station.available()) { station.read(); station.loadPixels(); for (int y=0; y<ny; y++) { for (int x=0; x<nx; x++) { fill(station.pixels[x+y*station.width]); rect(x*blockSize, y*blockSize, blockSize-1, blockSize-1); } } } }
[pde]
One thing to be careful of when manipulating pixels from a movie: the movie frame must be read in before the pixel array can be loaded. This is easy in if-available approach (which is why I used that). It can lead to a problem in the movieEvent approach if the draw() tries to access the first frame before it's been read.
The following sketch "unrolls" movie frames into separate thumbnail images on a grid. We keep track of where we are on the grid with variables x and y, for each frame incrementing x until it rolls over, at which point we increment y and roll it over as needed.
import processing.video.*; Movie station; int szx=40,szy=30; // how big to draw the frames int nx,ny; // number of frames across and down (computed in setup) int x=0,y=0; // current frame index on frame grid void setup() { size(800,600); nx = width/szx; ny = height/szy; station = new Movie(this, "station.mov"); station.loop(); } void draw() { if (station.available()) { // a movie frame is available for reading station.read(); // read it // draw it at the current position image(station,x*szx,y*szy,szx,szy); // advance the position, wrapping around in x and y x++; if (x==nx) { x=0; y++; if (y==ny) y=0; } } }
[pde]
One other example uses our pixel-level image manipulation to create a mirror image. We define a function that takes a PImage as a parameter (we call it from draw() with the current movie frame). It creates a new PImage with createImage() (given the size and color mode). It then sets the pixels as the mirror image, and returns the resulting PImage to the caller of the function. The draw() function then checkerboards the original and the mirror image.
import processing.video.*; Movie station; void setup() { size(480,360); station = new Movie(this, "station.mov"); station.loop(); } void draw() { if (station.available()) { // a movie frame is available for reading station.read(); // read it station.loadPixels(); // Create a mirror image PImage mirr = mirror(station); for (int j=0; j<3; j++) for (int i=0; i<3; i++) if ((i+j)%2 == 0) image(station,i*station.width,j*station.height); else image(mirr,i*mirr.width,j*mirr.height); } } // Returns a new image that is the mirror image of img PImage mirror(PImage img) { PImage m = createImage(img.width, img.height, RGB); for (int y=0; y<img.height; y++) for (int x=0; x<img.width; x++) m.pixels[x+y*img.width] = img.pixels[(img.width-1)-x + y*img.width]; return m; }
[pde]
Writing
Just as we can save a snapshot of the current window, we can save a movie whose frames are made from snapshots of the window. The MovieMaker class supports this functionality. There's not much to it. First we declare a global MovieMaker variable and create a new object in setup(). The constructor takes "this" again, along with the width, height, name of the output file, and some optional parameters about the movie encoding. Then any time we want to add a frame containing the current window, we invoke the MovieMaker.addFrame() method. When we're finished, we call the MovieMaker.finish() method to make sure the movie is cleanly saved.
One illustration is provided by Processing example Libraries | Video (Movie Maker) | Drawing Movie; another is on the MovieMaker documentation page. Warning: the default parameters (high quality) can produce pretty large files.
Capturing
Capturing video from a webcam works pretty much the same way as reading it from a file. The Capture class provides the basic functionality. We declare a variable, create an instance, and read from it whenever a frame is available. There are again two ways to handle the availability of a capture frame, either by testing within the draw() body the result of the Capture.available() method, or else by providing a separate captureEvent() function.
The following example is mostly just a basic image capture, with one addition: after capturing an image, we process it with our edge kernel from the Image processing lecture. As illustrated in the mirror image example above, we package up the image processing into a function that takes a PImage parameter and returns a processed PImage.
import processing.video.*; Capture cam; void setup() { size(320,240); cam = new Capture(this,320,240); } void draw() { if (cam.available()) { cam.read(); PImage edgeImg = edges(cam); image(edgeImg,0,0); } } // Based on code from Image Processing lecture // Here, takes an image as a parameter, and returns a new image computed by the edge kernel PImage edges(PImage img) { float[][] kernel = { {-1, -1, -1}, {-1, 8, -1}, {-1, -1, -1} }; PImage edges = createImage(img.width, img.height, RGB); for (int y=1; y<img.height-1; y++) { for (int x=1; x<img.width-1; x++) { // add together the values for all the neighbors, // weighted by the corresponding kernel value float r=0, g=0, b=0; for (int dy=-1; dy<=1; dy++) { for (int dx=-1; dx<=1; dx++) { color p=img.pixels[x+dx + (y+dy)*img.width]; r += red(p)*kernel[dy+1][dx+1]; g += green(p)*kernel[dy+1][dx+1]; b += blue(p)*kernel[dy+1][dx+1]; } } // set the color edges.pixels[x+y*img.width] = color(r,g,b); } } return edges; }
[pde]
The magic of Processing again comes into play -- each frame of image capture is just another PImage. The following example lets us take a bunch of thumbnail snapshots, using the same type of thumbnail grid as with the movie sketch above.
import processing.video.*; Capture cam; int camW=320,camH=240; // size of the capture window int szx=80,szy=60; // size of the thumbnails int nx,ny; // how many thumbnails (computed in setup()) int x=0,y=0; // which thumbnail is next void setup() { size(640,240); cam = new Capture(this,camW,camH); nx = (width-camW)/szx; ny = height/szy; } void draw() { if (cam.available()) { cam.read(); image(cam,0,0); } } void mousePressed() { // put up a thumbnail image(cam,camW+x*szx,y*szy,szx,szy); // and advance to the next position, wrapping around if needed x++; if (x==nx) { x=0; y++; if (y==ny) y=0; } }
[pde]