Feb 22: Video processing
If you don't have a webcam, the sketches covered today can be modified to use QuickTime movies instead -- declare and create a Movie object instead of a Capture object. In fact, that's a good way to test and debug your own sketches...
More displaying
There are a bunch of great video processing examples that come with Processing, in the Libraries | Video (Capture) section. Mirror and Mirror 2 are similar -- they both provide a different rendering of the pixels, mirror imaged left-to-right as we did in the image processing lecture. Mirror displays the pixels as equal-sized rectangles, rotated according to brightness, while Mirror 2 displays them as rectangles whose size depends on brightness. The brightness() function returns the brightness component of a color (under the hue-saturation-brightness color model, as compared to red-green-blue). The structure of both skecthes is the same as we've been using, including the declaration and initialization of the capture, the reading of an available frame, and the use of nested for-loops to extract pixel colors for analysis and display.
The SlitScan example unrolls time into space. A vertical column (at videoSliceX) is extracted from each frame. The columns are displayed from right to left (at drawPositionX), wrapping back around when the window is full.
Background
In order to detect something new or different about an image, we can subtract a reference image from it. Where the images are the same, the difference image will have low pixel values (0 where identical); where they are quite different, the pixel values will be high. The Processing examples (Libraries | Video (Capture)) BackgroundSubtraction and FrameDifferencing use this technique. BackgroundSubtraction sets the reference when a key is pressed, it then subtracts that reference from every subsequent image. This allows us to detect what's there now that wasn't there when we started. FrameDifferencing, on the other hand, subtracts the previous frame from the current one. This allows us to detect motion -- what has just changed in the last fraction of a second. In both cases, a total of the differences is printed out -- if the total is large, something is going on (but notice that even when nothing is going on, the small random variations add up to a not-so-small total).
There are a couple of things to clarify about these sketches. The first is that the reference image is stored just as an array of ints. It turns out that colors can be treated as ints. This point is taken one step further in the rather odd-looking expressions for extracting red, green, and blue components. The sketches could have just used the red(), green(), and blue() functions, but these expressions are a bit faster, and those bits add up with video processing. To understand these expressions would require diving into binary and hexademical representations of colors, which is a bit far afield from our current focus. Ask me (or do a little web search) if you are interested.
Interaction via brightness
We can use video as a form of input, in conjunction with or as a replacement for the mouse. Two Processing example sketches, BrightnessThresholding and BrightnessTracking (same place) demonstraate how to use brightness that way. BrightnessThresholding first thresholds the image into black and white (with pixel operations rather than the filter() function; not sure why), and then tests whether the mouse is in a black region or white region. This approach thus allows interplay between what's coming in via these two input mechanisms (I'm imagining a game of "Operation" where the mouse has to stay in the right region or else). BrightnessTracking looks for the brightest pixel in each frame and essentially uses that as the mouse position. It sets up the usual nested loop, checking each pixel's brightness against the current brightest value (which starts off at 0), changing which it thinks is brightest as appropriate. This really is then equivalent to a mouse.
Globs
A "glob" is a contiguous region of similarly colored pixels. We saw how edges capture sharp contrasts; globs capture the stuff between those sharp contrasts. The problem of finding globs has been studied for years, and there are many approaches. For example, one could do a "paint fill" -- start growing a glob from one pixel, and expand to its neighbors if they are simiar enough in color, and then spread to their neighbors, etc. Rather than implementing a glob finder, we'll simply utilize a contributed library called JMyron that provides this ability (among others). As a contributed library, it doesn't come pre-installed with Processing; however, it's very straightforward to install -- just put some files in the proper folders. It has been installed on the machines in the Mac lab.
JMyron is used much like Capture -- declare and create an instance, and do something with it each frame. (The JMyron web page documents the various methods.) Rather than passing the capture size to the constructor, we call a separate method, start(), after creating a new instance. We then call some methods that control how globs are found. The minDensity() method sets how many pixels must make up a glob, while the trackColor() method sets what color globs to look for. In trackColor(), we give both the r-g-b values and a "sensitivity" (exactly what that means isn't documented, but clearly larger numbers allow the color to be more different from the desired value).
A single method, update(), checks availability of a frame and reads it. We can make a copy the image with imageCopy() method (copying it into the pixels array). We can then get information about the globs that have been found. The following sketch gets the centers and outlines of the globs. Each center has an x and y coordinate, stored as elements 0 and 1 of an array. The centers are stuck together into another array; thus we have an array (over centers) of arrays (x=0 and y=1) of the coordinates of the centers. The outline of each glob is an array of vertex coordinates, stored the same way (each as a two-element array). Thus over all the globs we have an array of these arrays. We draw the outline with the shape functions.
import JMyron.*; JMyron video; // Video capture void setup() { size(320,240); smooth(); video = new JMyron(); video.start(width,height); video.minDensity(1000); // decent-sized globs video.trackColor(0,0,0,200); // fairly dark globs } void draw() { video.update(); // Camera image loadPixels(); video.imageCopy(pixels); updatePixels(); // Centers of globs noStroke(); fill(255,0,0); // an array (over globs) of arrays of 2 elements (0=x, 1=y) int[][] centers = video.globCenters(); for (int c=0; c<centers.length; c++) { int[] pt = centers[c]; if (pt[0] != 0 || pt[1] != 0) // it seems to put spurious centers in the corner ellipse(pt[0],pt[1],10,10); } // Outlines of globs stroke(0,255,0); noFill(); // an array (over globs) of arrays (over edge points) of 2 elements (0=x, 1=y) int[][][] edgePointSets = video.globEdgePoints(30); for (int eps=0; eps<edgePointSets.length; eps++) { int[][] edgePoints = edgePointSets[eps]; if (edgePoints != null) { // following one of the examples beginShape(); for (int p=0; p<edgePoints.length; p++) vertex(edgePoints[p][0], edgePoints[p][1]); endShape(CLOSE); } } }
[pde]
In addition to extracting a fairly tight-fitting boundary of a glob, we can also extract a "bounding box" -- a rectangle completely containing the glob. The following sketch uses the bounding box to set the size of an image (guess which one?) that shows through where the glob is. This gives kind of a custom "fabric" feel. To only show through where a glob is, we draw the shape as in the previous sketch, but this time fill it in black on a white background, thereby setting up a mask that's completely transparent where the glob is and completely opaque elsewhere. We apply this mask to the camera image, and display the camera image over a background with the chosen image placed wherever the bounding boxes are.
import JMyron.*; JMyron video; // Video capture PImage baker; // Custom "fabric" void setup() { size(320,240); smooth(); baker = loadImage("baker-small.jpg"); video = new JMyron(); video.start(width,height); video.minDensity(1000); // decent-sized globs video.trackColor(0,0,0,200); // fairly dark globs } void draw() { video.update(); background(255); // Filled globs noStroke(); fill(0); int[][][] edgePointSets = video.globEdgePoints(30); // an array (over globs) of arrays (over edge points) of 2 elements (0=x, 1=y) for (int eps=0; eps<edgePointSets.length; eps++) { int[][] edgePoints = edgePointSets[eps]; if (edgePoints != null) { // following one of the examples beginShape(); for (int p=0; p<edgePoints.length; p++) vertex(edgePoints[p][0], edgePoints[p][1]); endShape(CLOSE); } } // Capture the camera image and mask it according to globs PImage cam = createImage(width,height,ARGB); cam.loadPixels(); video.imageCopy(cam.pixels); PImage msk = get(); cam.mask(msk); // Put Baker in bounding boxes of globs background(0,255,0); int[][] bboxes = video.globBoxes(); // an array (over globs) of corners and sizes of rectangles for (int bb=0; bb<bboxes.length; bb++) { int[] bbox = bboxes[bb]; image(baker, bbox[0], bbox[1], bbox[2], bbox[3]); } // Show camera image, masked so that Baker shows through image(cam,0,0); }
[pde]
Finally, inspired by the Shadow Monsters sketch in the Processing exhibition, the following sketch puts "hair" around the globs. We have a loop that, for each glob, iterates over the vertices, drawing hair along the line from one vertex to the next. To draw hair, we start (x,y) at the first vertex and repeatedly take a step towards the other. Note that we don't want the same number of steps along each segment, since some segments may be shorter than others (and thus would have denser hair). Instead, we want (roughly) equally sized steps. Thus we do the same thing as in the crawling worm sketch in the springs lecture, computing the overall direction and the fraction of it along x and along y, multiplying that by the constant "speed". However, to be more hair-like, we don't take all steps to be equal, so choose a random size up to the maximum step size. We then draw hair (a random length line) so that it points away from (is perpendicular to) the segment. If a line is going in direction (dx,dy), then its perpendicular is going in direction (-dy,dx) or (dy,-dx). In this case, it appears that the vertices are ordered so that the (-dy,dx) perpendicular points inward and the (dy,-dx) one outward.
import JMyron.*; JMyron video; // Video capture // maximum spacing between and length of hairs float hairStep=2, hairLength=20; void setup() { size(320,240); video = new JMyron(); video.start(width,height); video.minDensity(1000); // decent-sized globs video.trackColor(0,0,0,200); // fairly dark globs stroke(221,30,232); smooth(); } void draw() { video.update(); // Show the camera image loadPixels(); video.imageCopy(pixels); updatePixels(); // Follow outlines int[][][] edgePointSets = video.globEdgePoints(30); // an array (over globs) of arrays (over edge points) of 2 elements (0=x, 1=y) for (int eps=0; eps<edgePointSets.length; eps++) { int[][] edgePoints = edgePointSets[eps]; if (edgePoints != null) { // following one of the examples for (int p=0; p<edgePoints.length; p++) { int x1=edgePoints[p][0], y1=edgePoints[p][1]; int p2 = (p+1)%edgePoints.length; // wrap around so close shape int x2=edgePoints[p2][0], y2=edgePoints[p2][1]; hairify(x1,y1,x2,y2); } } } } // Draw hair between (x1,y1) and (x2,y2) void hairify(int x1, int y1, int x2, int y2) { float l = dist(x1,y1,x2,y2); if (l>0) { // Take equal-sized steps from x1 to x2 float dx=(x2-x1)/l, dy=(y2-y1)/l; float x=x1, y=y1; boolean done=false; while (!done) { // Choose random step and length up to maximum values float hs=random(0.1,hairStep), hl=random(1,hairLength); // Draw line perpendicular to (x1,y1)-(x2,y2) direction line(x,y,x+hl*dy,y-hl*dx); // Take a step, and see if have reached the end (test whether moving left or right) x+=hs*dx; y+=hs*dy; if (x1<x2) done = x>=x2; else done = x<=x2; } } }
[pde]