Feb 27: Video processing
Background
How can we make a webcam video, shot from the comfort of our room, look like we are standing in front of Baker? Filmmakers use the trick of filming in front of a green screen, and then replacing the green parts with the desired background footage. Since we don't happen to have a green screen here, we'll have to go a step further. What we'll do instead is first to take a snapshot of the background without us in it. Then for each frame, we'll get rid of that part of the image, replacing it with the corresponding part of the desired background. We know how to take a snapshot, so the only issue is getting rid of it. The approach in Shiffman Sec. 16.16 (with code in example 16-12) is to compare a webcam pixel's r,g,b values with those of the corresponding background pixel, and see if they are too similar (using the dist() function). Shiffman sets a too-similar pixel to green; can you modify the sketch to set it to a color from a desired background?
The same basic idea can be used to detect movement, by subtracting one frame from the next. Shiffman Sec. 16.7 discusses this approach and presents sketch 16-13.
Brightness tracking
We can use video as a form of user input. The Processing example Libraries | Video (Capture) | BrightnessTracking demonstrates how to find a bright spot (e.g., from a flashlight). To find the brightest pixel in a frame, it goes pixel by pixel, checking the current pixel's brightness (extracted with the brightness() function) against the brightness of the brightest pixel before it. If the current pixel is brighter, it is remembered as the new brightest. Shiffman Sec. 16.5 discusses this in more detail, extending the basic idea to allow tracking an arbitrary color (example 16-11).
This sketch simply plots an ellipse at the brightest pixel. But that's entirely equivalent to plotting an ellipse at the current mouse position. So we could turn the brightness tracker into a new form of input. For example, can you do a "mouse over"?
Globs
The brightness (or color) tracker just looks for the single brightest (or most similarly colored) pixel, but what if we want to find a whole region of such pixels? This would, for example, allow us to track movement of a whole hand. We call a contiguous region a "glob"; the problem of finding globs has been studied for years, and there are many approaches. For example, one could do a "paint fill" — start growing a glob from one pixel, and expand to its neighbors if they are similar enough in color, and then spread to their neighbors, etc. There's also a clear connection between edges and globs — edges capture sharp contrasts; globs capture the stuff between those sharp contrasts.
Rather than implementing a glob finder, we'll simply utilize a contributed library called JMyron that provides this ability (among others). As a contributed library, it doesn't come pre-installed with Processing; however, it's very straightforward to install, requiring some files to be put in the proper folders. We must also include the appropriate import statement at the start of our sketch.
JMyron is used much like Capture: declare and create an instance, and do something with it each frame. (The JMyron web page documents the various methods.) Rather than passing the capture size to the constructor, we call a separate method, start(), after creating a new instance. We then call some methods that control how globs are found. The minDensity() method sets how many pixels must make up a glob, while the trackColor() method sets what color globs to look for. In trackColor(), we give both the r-g-b values and a "sensitivity" (exactly what that means isn't documented, but clearly larger numbers allow the color to be more different from the desired value).
A single method, update(), checks availability of a frame and reads it. We can make a copy of the image with imageCopy() method (copying it into the pixels array). We can then get information about the globs that have been found. The following sketch gets the outlines of the globs and draws them as shapes. The way JMyron represents the outlines is as an array of arrays of arrays [!]. The main array is over the various globs; index 0 represents the first glob, 1 the second, and so forth. The array for a glob is over its vertices; index 0 represents the first vertex, 1 the second, and so forth. And finally the array for a vertex is over its coordinates; index 0 has the x and 1 the y.
import JMyron.*; JMyron video; // Video capture void setup() { size(640,480); smooth(); video = new JMyron(); video.start(width,height); video.minDensity(1000); // decent-sized globs video.trackColor(0,0,0,200); // fairly dark globs } void draw() { video.update(); // Camera image loadPixels(); video.imageCopy(pixels); updatePixels(); // Outlines of globs stroke(0,255,0); noFill(); // an array (over globs) of arrays (over edge points) of 2 elements (0=x, 1=y) int[][][] edgePointSets = video.globEdgePoints(30); for (int eps=0; eps<edgePointSets.length; eps++) { // an array (over edge points) of 2 elements (0=x, 1=y) int[][] edgePoints = edgePointSets[eps]; if (edgePoints != null) { // following one of the examples beginShape(); for (int p=0; p<edgePoints.length; p++) { // an array of 0=x, 1=y int[] edgePoint = edgePoints[p]; vertex(edgePoint[0], edgePoint[1]); } endShape(CLOSE); } } }
[pde]
In addition to extracting a fairly tight-fitting boundary of a glob, we can also extract a bounding box — a rectangle completely containing the glob. The following sketch uses the bounding box to set the size of an image (guess which one?) that shows through where the glob is. This gives kind of a custom "fabric" feel. To only show through where a glob is, we draw the shape as in the previous sketch, but this time fill it in black on a white background. We then use the PImage.mask() method to set the transparency of the camera image. Where the mask is black (grayscale=0), the camera image is totally transparent (alpha=0), while where it is white (grayscale=255), the camer image is totally opaque (alpha=255). The transparent parts let the chosen image (drawn within the bounding box) show through.
import JMyron.*; JMyron video; // Video capture PImage baker; // Custom "clothing" void setup() { size(640,480); smooth(); baker = loadImage("baker-small.jpg"); video = new JMyron(); video.start(width,height); video.minDensity(1000); // decent-sized globs video.trackColor(0,0,0,200); // fairly dark globs } void draw() { video.update(); // Use window temporarily, drawing globs in black on a white background background(255); noStroke(); fill(0); int[][][] edgePointSets = video.globEdgePoints(30); // an array (over globs) of arrays (over edge points) of 2 elements (0=x, 1=y) for (int eps=0; eps<edgePointSets.length; eps++) { int[][] edgePoints = edgePointSets[eps]; if (edgePoints != null) { // following one of the examples beginShape(); for (int p=0; p<edgePoints.length; p++) vertex(edgePoints[p][0], edgePoints[p][1]); endShape(CLOSE); } } // Capture the camera image and mask it according to glob image, // which is filled black (=> transparent as mask) where globs are PImage cam = createImage(width,height,ARGB); cam.loadPixels(); video.imageCopy(cam.pixels); PImage msk = get(); cam.mask(msk); // Put Baker in bounding boxes of globs background(0,255,0); int[][] bboxes = video.globBoxes(); // an array (over globs) of corners and sizes of rectangles for (int bb=0; bb<bboxes.length; bb++) { int[] bbox = bboxes[bb]; image(baker, bbox[0], bbox[1], bbox[2], bbox[3]); } // Show camera image, masked so that Baker shows through image(cam,0,0); }
[pde]
Finally, inspired by the Shadow Monsters sketch in the Processing exhibition, the following sketch puts "hair" around the globs. We have a loop that, for each glob, iterates over the vertices, drawing hair along the line from one vertex to the next. To draw hair, we start (x,y) at the first vertex and repeatedly take a step towards the other. Note that we don't want the same number of steps along each segment, since some segments may be shorter than others (and thus would have denser hair). Instead, we want (roughly) equally sized steps. Thus we do the same thing as in the crawling worm sketch in the springs lecture, computing the overall direction and the fraction of it along x and along y, multiplying that by the constant "speed". However, to be more hair-like, we don't take equal-sized steps, but instead choose a random size up to the maximum step size. We then draw hair (a random length line) so that it points away from (is perpendicular to) the segment. If a line is going in direction (dx,dy), then its perpendicular is going in direction (-dy,dx) or (dy,-dx). To see this, think about the difference between walking north (dx=0,dy=-1) vs. east (dx=1,dy=0), and so forth. In this case, it appears that the vertices are ordered so that the (-dy,dx) perpendicular points inward and the (dy,-dx) one outward.
import JMyron.*; JMyron video; // Video capture // maximum spacing between hairs; maximum length of hairs float hairStep=2, hairLength=30; void setup() { size(640,480); video = new JMyron(); video.start(width,height); video.minDensity(1000); // decent-sized globs video.trackColor(0,0,0,200); // fairly dark globs stroke(221,30,232); // magenta hair smooth(); } void draw() { video.update(); // Show the camera image loadPixels(); video.imageCopy(pixels); updatePixels(); // Follow outlines int[][][] edgePointSets = video.globEdgePoints(30); // an array (over globs) of arrays (over edge points) of 2 elements (0=x, 1=y) for (int eps=0; eps<edgePointSets.length; eps++) { int[][] edgePoints = edgePointSets[eps]; if (edgePoints != null) { // following one of the examples for (int p=0; p<edgePoints.length; p++) { int x1=edgePoints[p][0], y1=edgePoints[p][1]; int p2 = (p+1)%edgePoints.length; // wrap around so close shape int x2=edgePoints[p2][0], y2=edgePoints[p2][1]; hairify(x1,y1,x2,y2); } } } } // Draw hair between (x1,y1) and (x2,y2) void hairify(int x1, int y1, int x2, int y2) { float l = dist(x1,y1,x2,y2); if (l>0) { // Take equal-sized steps from x1 to x2 float dx=(x2-x1)/l, dy=(y2-y1)/l; float x=x1, y=y1; boolean done=false; while (!done) { // Choose random step and length up to maximum values float hs=random(0.1,hairStep), hl=random(1,hairLength); // Draw line perpendicular to (x1,y1)-(x2,y2) direction line(x,y,x+hl*dy,y-hl*dx); // Take a step x+=hs*dx; y+=hs*dy; // Test depends on whether moving left or right if (x1<x2) done = x>=x2; else done = x<=x2; } } }
[pde]