Feb 25: Webcam
Getting going
Mac people are all set. Windows people: you need to install a video digitizer that will convert the webcam video into QuickTime. The Processing documentation points to WinVDIG; the original site has apparently gone away, but a nice person has kept a WinVDIG mirror. Download and install version 1.0.1 (apparently the other versions have some issues). If this becomes too painful, it might be a good opportunity to instead visit the Mac lab in Sudikoff. More details, and some troubleshooting tips, are available in the Video library reference.
Capturing
Capturing video from a webcam works pretty much the same way as reading it from a file. The Capture class provides the basic functionality. We declare a variable, create an instance, and read from it whenever a frame is available. There are again two ways to handle the availability of a capture frame, either by testing within the draw() body the result of the Capture.available() method, or else by providing a separate captureEvent() function.
The following example is mostly just a basic image capture, with one addition: after capturing an image, we process it with the edge kernel from the Image processing lecture. As illustrated in the mirror image example above, we package up the image processing into a function that takes a PImage parameter and returns a processed PImage.
import processing.video.*; Capture cam; void setup() { size(640,480); cam = new Capture(this,640,480); } void draw() { if (cam.available()) { cam.read(); PImage edgeImg = edges(cam); image(edgeImg,0,0); } } // Based on code from Image Processing lecture // Here, takes an image as a parameter, and returns a new image computed by the edge kernel PImage edges(PImage img) { float[][] kernel = { {-1, -1, -1}, {-1, 8, -1}, {-1, -1, -1} }; PImage edges = createImage(img.width, img.height, RGB); for (int y=1; y<img.height-1; y++) { for (int x=1; x<img.width-1; x++) { // add together the values for all the neighbors, // weighted by the corresponding kernel value float r=0, g=0, b=0; for (int dy=-1; dy<=1; dy++) { for (int dx=-1; dx<=1; dx++) { color p=img.pixels[x+dx + (y+dy)*img.width]; r += red(p)*kernel[dy+1][dx+1]; g += green(p)*kernel[dy+1][dx+1]; b += blue(p)*kernel[dy+1][dx+1]; } } // set the color edges.pixels[x+y*img.width] = color(r,g,b); } } return edges; }
[pde]
The magic of Processing again comes into play — each frame of image capture is just another PImage, and we can do all kinds of image-y things. For example Shiffman 16-2 sets tint and size from the mouse.
Processing
The following example lets us take a bunch of thumbnail snapshots, using the same type of thumbnail grid as when we unrolled a movie last time.
import processing.video.*; Capture cam; int camW=320,camH=240; // size of the capture window int szx=80,szy=60; // size of the thumbnails int nx,ny; // how many thumbnails (computed in setup()) int x=0,y=0; // which thumbnail is next void setup() { size(640,240); cam = new Capture(this,camW,camH); nx = (width-camW)/szx; ny = height/szy; } void draw() { if (cam.available()) { cam.read(); image(cam,0,0); } } void mousePressed() { // put up a thumbnail image(cam, camW+x*szx, y*szy, szx, sszy); // and advance to the next position, wrapping around if needed x++; if (x==nx) { x=0; y++; if (y==ny) y=0; } }
[pde]
The Libraries | Video (Capture) | SlitScan example unrolls time into space. A vertical column (at videoSliceX) is extracted from each frame. The columns are displayed from right to left (at drawPositionX), wrapping back around when the window is full.
Two other examples in the Libraries | Video (Capture) section, Mirror and Mirror 2, provide different renderings of webcam images. These are "mirrors", in that they allow us to see ourselves (albeit in a somewhat modified reflection). As part of mirroring, they also mirror image the pixels left-to-right as we did in the image processing lecture. Mirror displays the pixels as equal-sized rectangles, rotated according to brightness, while Mirror 2 displays them as rectangles whose size depends on brightness. The brightness() function returns the brightness component of a color (under the hue-saturation-brightness color model, as compared to red-green-blue). The structure of both skecthes is the same as we've been using, including the declaration and initialization of the capture, the reading of an available frame, and the use of nested for-loops to extract pixel colors for analysis and display.
Finally, let's get a little discombobulated, making a puzzle from the webcam input. The sketch works much like our earlier image-based puzzle (in fact, the mousePressed function is a direct copy), but we have to be able to create the same layout of rectangles each time we read another frame. Thus rather than dividing up the static image into little images and swapping them, we keep the identities of the rectangles (e.g., the rectangle that is 5 over and 2 down) and swap where they will be drawn (e.g., draw the (5,2) rectangle at (3,0) and vice-versa). Then to display the current frame, we copy rectangles (that is, copy their pixels) from the webcam to the window, according to the rectangle swapping.
The actual code is a little tricky. We identify webcam rectangles by x and y indices (in the above example, webcam rectangle (5,2)). We identify window rectangles by their linearized indices, as with the pixels array (in the example, window rectangle (3,0) becomes window rectangle 3). We use the xs and ys arrays keep track of which webcame rectangle is displayed at which window rectangle. In our example, xs[3] = 5 and ys[3] = 2, so that the (5,2) rectangle in the webcam is copied to the 3rd rectangle in the window. We copy all the rectangles with one nested loop; for each rectangle, we copy all its pixels with another nested loop.
import processing.video.*; Capture cam; int[] xs, ys; // x and y coordinates of rectangle sources // x[i],y[i] says which camera rectangle goes to window rectangle #i int nx=3, ny=3; // number of rectangles across and down int dx, dy; // rectangle sizes, computed from width,height and nx,ny int selX=-1,selY=-1; // if a piece has been selected, its x and y indices (-1 if none) void setup() { size(640,480); cam = new Capture(this,640,480); dx = width/nx; dy = height/ny; // Initially, have our usual mapping, where piece at x+y*nx corresponds to (x,y) xs = new int[nx*ny]; ys = new int[nx*ny]; int p=0; for (int y=0; y<ny; y++) { for (int x=0; x<nx; x++) { xs[p] = x; ys[p] = y; p++; } } // Mix up the rectangles shuffle(); stroke(255,0,0); noFill(); } void draw() { if (cam.available()) { loadPixels(); cam.read(); cam.loadPixels(); int p=0; // which window piece we're currently drawing // y and x loop over the rectangle positions for (int y=0; y<ny; y++) { for (int x=0; x<nx; x++) { // i and j loop over the pixels within a rectangle for (int i=0; i<dy; i++) { for (int j=0; j<dx; j++) { // the window pixel is based on x,y and i,j // the camera pixel is based on xs[p],ys[p] and i,j pixels[(y*dy+i)*width + x*dx+j] = cam.pixels[(ys[p]*dy+i)*width + xs[p]*dx+j]; } } p++; } } updatePixels(); } if (selX >= 0 && selY >= 0) { rect(selX*dx, selY*dy, dx-1, dy-1); } } // Mix up all the pieces void shuffle() { // Swap each piece with some piece later in the array for (int i=0; i<xs.length; i++) swapPieces(i, int(random(i,xs.length))); } // Swap the pieces (both x and y values) at the two indices void swapPieces(int i1, int i2) { int x = xs[i1], y = ys[i1]; xs[i1] = xs[i2]; ys[i1] = ys[i2]; xs[i2] = x; ys[i2] = y; } void mousePressed() { // Find which piece was clicked on int px = mouseX/dx, py = mouseY/dy; // Select two pieces to swap; same piece twice to cancel if (selX >= 0 && selY >= 0) { if (selX == px && selY == py) { // same one twice -- cancel selX = -1; selY = -1; } else { // second selection -- swap swapPieces(px+py*nx, selX+selY*nx); selX = -1; selY = -1; } } else { // first selection selX = px; selY = py; } } void keyPressed() { if (key == 's') shuffle(); }
[pde]