Video processing


Processing video brings together the concepts we've seen so far. Video is just a stream of images, and we know how to represent and manipulate them. So we'll look at a couple of examples of just doing image processing periodically. But then we'll also consider what we can do once we start recognizing that the images are related to each other.

Outline

All the code files for today: Webcam.java; WebcamBg.java; WebcamColorTracking.java; WebcamDiff.java; WebcamLoop.java; WebcamProcessing.java; WebcamRendering.java

Slides from class

Core code

There's not a standard machinery for processing Webcam in Java. We'll be using JavaCV, which is a wrapper over the popular OpenCV image processing library. See the software page for installation details. I've wrapped up the details as best I could in a new Webcam class, analogous to DrawingGUI; see Webcam.java. Unfortunately, I haven't been able to make a one-size-fits-all version, and depending on whether you're a Mac or Windows person (or Linux), you'll need to set the mac variable to true or false, as with the WebcamTest program you tried when installing JavaCV. That seemed to address all the issues I saw with the webcam test, but post on Piazza if there are still others. In general, the mechanism for dealing with a webcam can be flakey. Worst case: just use the lab machines in Sudikoff for this bit. One note: when a program "freezes", Eclipse has a little red "stop" button in the panel with the console, to kill the program without having to resort to your operating system's halt mechanism.

You should be able to run that file on its own and see video. Note that there are two set up parameters: scale, to down-size the image (useful for intensive processing things), and mirror, to flip the image left-right (feels more natural). The constructor prints out the native camera size, so you can decide what a good scale factor is to yield a sufficiently small image for your machine.

You don't really need to be familiar with the Webcam code. It's built off DrawingGUI, so the same methods apply for handling events and drawing. One new method is processImage, which we define in our own subclass to do something with each separate image as it comes off the webcam. The image itself is stored in an instance variable called image, to which processImage has access, and which it can modify.

Webcam processing and rendering

Since the result of a grab is just our usual BufferedImage, we could apply any of our image processing methods to it after it's grabbed and before displaying it. Try subclassing and giving something for the processImage method, as I did in WebcamProcessing.java, taking advantage of our nice image processing class. We can also change the rendering by overriding draw, e.g., by rendering the pixels of the image as big rectangles or randomly distributed and sized circles in WebcamRendering.java.

Color tracking

We can use webcam as a form of user input. For example, given a color, we can identify the pixel whose color is closest to that. This could then serve as kind of a mouse.

WebcamColorTracking.java demonstrates how to do such color tracking. Upon getting mouse press, we remember (in trackColor) the color of the pixel at the mouse position. Then when drawing a frame, we call a helper method track to find the Point whose color is closest to that. We draw an oval there (after drawing the image). Not particularly robust (in practice, we'd assume that the thing we're tracking moves smoothly), but kind of works!

Frame differencing

To detect movement, we can subtract one frame from the next: WebcamDiff.java. Each frame, we make a copy of the current image (with the big Java incantation). We then simply subtract RGB values for corresponding pixels from the current one and the previous one, and compute the absolute value (since we can't have negative color values, and the direction of the different doesn't matter anyway for this purpose).

Recording a loop

If we can keep track of one frame, we can keep track of many. Let's record a list of images and set up a loop to play them back (backwards, just for fun). To focus on the concept, I've made a not-so-user-friendly version that starts recording as soon as the program starts, and just toggles between recording and playback upon mouse press. See WebcamLoop.java. We add instance variables, to keep track of the recording vs. playback state, the list of frames, and the current frame. When recording, we add the current frame to the list. We modify the draw to either draw the live action or the current stored frame (and step backwards). And we have the mouse press toggle between recording and playback, re-setting the recording variables accordingly.

Background subtraction

How can we make a webcam video, shot from the comfort of our room, look like we are standing in front of Baker? (For best results, one of the same size as our webcam; for me baker-640-480.jpg works.) Filmmakers use the trick of filming in front of a green screen, and then replacing the green parts with the desired background footage. Since we don't happen to have a green screen here, we'll have to go a step further. What we'll do instead is first to take a snapshot of the background without us in it. Then for each frame, we'll get rid of that part of the image, replacing it with the corresponding part of the desired scenery.

WebcamBg.java performs background subtraction in the processImage method, and provides standard code to read in the scenery and capture the "background" upon mouse press. The idea is to compare the image pixel's values with those of the corresponding background pixel. If they are similar, according to Euclidean distance and a specified threshold, then we replace the image pixel with the corresponding one from scenery. This works best under controlled lighting and if your webcam isn't trying to be too fancy itself by adjusting brightness, etc. You'll need to be sure to scale the webcam image down to the size of the background scenery (the setup scale, stored in a static final variable).