Feb 20: Image processing

Setting pixel values
Pixel-level image manipulation
Neighborhoods

Images used here are from Wikimedia commons (Category: Dartmouth College), and are in the public domain. Thanks go primarily to user Kane5187.

Setting pixel values

There are two equivalent ways to directly set pixel values. The first is the set() function and PImage.set() method. These are the counterparts to get() and PImage.get() — we give coordinates and a color.

void draw()
{
  for (int y=0; y<height; y++)
    for (int x=0; x<width; x++)
      set(x,y,color(random(255),random(255),random(255)));
}

[applet]

The other way to set pixel values is to directly manipulate an array of the pixel values. There is a global array pixels[] for the pixels of the window, and each PImage has a field pixels[] with its own pixels. Before using one of these, we must call the loadPixels() function or the PImage.loadPixels() method to fill in the array, else we'll get a null pointer exception. After we're finished, we must call the updatePixels() function or PImage.updatePixels() to store the array else our changes don't take. The pixels[] array is indexed the same way we saw last time with the pieces of the puzzle — rows are put into the array from top to bottom. Thus if the image is 800x600, the array has 480000 elements, with 0-799 from the first row, 800-1599 from the second, etc. To get pixel (x,y), we access array element x+y*width, since we skip width pixels for each row.

Here's the same random colors sketch using the direct pixel approach.

void draw()
{
  loadPixels();
  for (int y=0; y<height; y++)
    for (int x=0; x<width; x++)
      pixels[x+y*width] = color(random(255),random(255),random(255));
  updatePixels();
}

[applet]

The direct pixel approach is faster, and is more commonly used, so I'll stick with it. Here's another example, with a color gradient (red in x, green in y, and blue in mouse x).

void draw()
{
  loadPixels();
  for (int y=0; y<height; y++)
    for (int x=0; x<width; x++)
      pixels[x+y*width] = color(x*255.0/width, y*255.0/height, mouseX*255.0/width);
  updatePixels();
}

[applet]

Pixel-level image manipulation

Processing provides some pretty powerful functions to modify images. If you're interested, read the reference for filter() and PImage.filter() (convert from color to black and white, grayscale, or reduced color; blur; or invert); blend() and PImage.blend() (combine pixels from of two images); and PImage.mask() (set the transparency of one image based on the grayscale values of another). Because they're like what you'd find in photo editing software (though they can be integrated in a sketch), I won't spend any time on them. Instead I'll focus on lower-level pixel operations where we have a lot more control and can come up with new and different effects. Following are a bunch of short examples.

Shiffman 15-8 illustrates adjusting the brightness. We just multiply the red, green, and blue components by a scale factor set by the mouse x.

Using conditionals, we can be much more selective, e.g., only "reddening" pixels that are moderately red, and "blueing" those that are moderately blue.

PImage baker;

void setup()
{
  size(800,600);
  baker = loadImage("baker.jpg");
}

void draw()
{
  background(baker);
  loadPixels();
  for (int i=0; i<pixels.length; i++) {
    // if it's already red-ish or blue-ish, allow the mouse to control it
    float r = red(pixels[i]), b = blue(pixels[i]);
    if (r > 96 && r < 192) r = mouseX*255.0/width;
    if (b > 96 && b < 192) b = mouseY*255.0/height;
    pixels[i] = color(r, green(pixels[i]), b);
  }
  updatePixels();
}

[applet]

For a contrast enhancement, we can accentuate the redness/greenness/blueness or the lack thereof. Greenberg (10-34) uses a formula that compares the difference between the current value and the middle value (127.5), and scales that difference by a specified factor. For example, if the current value is 130 and the factor is 2, then the new value is 130 + 2*2.5 = 135. Thus the value gets pushed out to one extreme (255) or the other (0), depending on which side of the middle it's on.

PImage baker;

void setup()
{
  size(800,600);
  baker = loadImage("baker.jpg");
}

void draw()
{
  float contrast = .01*mouseX/width;  // factor by which to push toward either 0 or 255
  loadPixels();
  baker.loadPixels();
  for (int i=0; i<pixels.length; i++) {
    color p=baker.pixels[i];
    float r=red(p), g=green(p), b=blue(p);
    r = constrain(r+r*contrast*(r-127.5), 0,255);
    g = constrain(g+g*contrast*(g-127.5), 0,255);
    b = constrain(b+b*contrast*(b-127.5), 0,255);
    pixels[i] = color(r,g,b);
  }
  updatePixels();
}

[applet]

To invert an image, we subtract pixel values from 255. While a standard invert filter works the same for every pixel, once we move to setting pixels ourselves, we can modify it to invert more or less for different pixels. Greenberg has a sketch (10-41) in which the inversion varies row by row; I've done a variation on that here. We subtract from 0 for the first row, then 1/height for the second row, then 2/height, etc., taking the absolute value after subtracting. Thus subtracting from 0 does nothing, subtracting from 1 only slightly changes it, ..., till it's completely inverted.

size(800,600);
PImage baker = loadImage("baker.jpg");

loadPixels();
float invert = 0;  // subtract pixel value from this, which varies from top to bottom
for (int y=0; y<height; y++) {
  for (int x=0; x<width; x++) {
    color p=baker.pixels[x+y*baker.width];
    float r=red(p), g=green(p), b=blue(p);
    pixels[x+y*width] = color(abs(invert-r), abs(invert-g), abs(invert-b));
  }
  invert += 255.0/height;
}
updatePixels();

[applet]

Before looking at the following sketch code, look at the image and see if you can tell what it does.

size(800,463);
background(loadImage("green.jpg"));

loadPixels();
for (int y=0; y<height; y++) {
  for (int x=0; x<width/2; x++) {
    // swap (x,y) with (width-1-x,y)  (width-1 is the rightmost)
    color p = pixels[x+y*width];
    pixels[x+y*width] = pixels[(width-1)-x+y*width];
    pixels[(width-1)-x+y*width] = p;
  }
}
updatePixels();

[applet]

Notice the "swap" idiom that we discussed last time.

We can adjust the pixel values based on how close they are to the mouse. Processing example Topics | Image Processing | Brightness does just that, as does Shiffman 15-9.

We can also define our own approaches to blending images, as illustrated in the following two sketches. The first does a checkerboard, alternating pixels from Baker and Dartmouth Hall, using the modulus operator to switch. The second takes the red value from Baker, the blue value from Dartmouth Hall, and the green value as a fraction (accoring to the mouse position) from each.

size(800,600);
PImage baker = loadImage("baker.jpg");
PImage dart = loadImage("dartmouth-hall.jpg");

loadPixels();
for (int y=0; y<height; y++) {
  for (int x=0; x<width; x++) {
    if ((x+y)%2 == 0) pixels[x+width*y] = baker.pixels[x+width*y];
    else pixels[x+width*y] = dart.pixels[x+width*y];
  }
}
updatePixels();

[applet]

PImage baker, dart;

void setup()
{
  size(800,600);
  baker = loadImage("baker.jpg");
  dart = loadImage("dartmouth-hall.jpg");
}

void draw()
{
  float frac = ((float)mouseX)/width;  // how much green from baker?
  loadPixels();
  for (int i=0; i<pixels.length; i++) {
    float r = red(baker.pixels[i]);
    float gb = green(baker.pixels[i]);
    float gd = green(dart.pixels[i]);
    float b = blue(dart.pixels[i]);
    pixels[i] = color(r, frac*gb + (1-frac)*gd, b);
  }
  updatePixels();
}

[applet]

Neighborhoods

The examples so far have treated each pixel independently of each other pixel (except for swapping a pair). A whole other class of image processing algorithms deals with the "neighborhood" around a pixel. The neighborhood is often defined as those pixels immediately adjacent to a given pixel (N, NE, E, ..., NW).

Let's consider the values in the neigbhorhood around one pixel.

10	12	13
12	34	11
10	13	11

The one pixel's value is quite a bit higher than the others; this might be just an artifact that should be smoothed out. To do so (a "blur" filter), we can replace the value in the center with the average of all the values in its neighborhood (including itself). Thus instead of 34 we'd have (10+12+13+12+34+11+10+13+11)/9 = 14, which is much more like its neighbors. The blurring filter does this for each pixel in the image. In the sketch below, the neighbors are indexed as offsets, from -1 to 1, in x and in y. We add up all the values, and then divide by 9.

size(800,600);
PImage baker=loadImage("baker.jpg");
loadPixels();
baker.loadPixels();

for (int y=1; y<height-1; y++) {
  for (int x=1; x<width/2; x++) {
    // add together the values for all the neighbors
    float r=0, g=0, b=0;
    for (int dy=-1; dy<=1; dy++) {
      for (int dx=-1; dx<=1; dx++) {
        color p=baker.pixels[x+dx + (y+dy)*baker.width];
        r += red(p); g += green(p); b += blue(p);
      }
    }
    // set the color as the average
    pixels[x+y*width] = color(r/9,g/9,b/9);
  }
  // show after/before
  for (int x=0; x<width/2; x++)
    pixels[x+width/2+y*width] = baker.pixels[x+y*width];
}
updatePixels();

[applet]

The idea of adding up values from a neighborhood is very powerful, and by varying how much each neighbor contributes, we can get quite different results. To make clear the contributions, we use a kernel that gives the weight to each neighbor. In our blurring filter, it's 1/9 everywhere:

1/9	1/9	1/9
1/9	1/9	1/9
1/9	1/9	1/9

Using this kernel, we can re-write the sketch so that it multiplies pixel values by the corresponding kernel values before adding them to the running total.

size(800,600);
PImage baker=loadImage("baker.jpg");
loadPixels();
baker.loadPixels();

float[][] kernel = { {1/9.0, 1/9.0, 1/9.0},
                     {1/9.0, 1/9.0, 1/9.0},
                     {1/9.0, 1/9.0, 1/9.0} };

for (int y=1; y<height-1; y++) {
  for (int x=1; x<width/2; x++) {
    // add together the values for all the neighbors,
    // weighted by the corresponding kernel value
    float r=0, g=0, b=0;
    for (int dy=-1; dy<=1; dy++) {
      for (int dx=-1; dx<=1; dx++) {
        color p=baker.pixels[x+dx + (y+dy)*baker.width];
        r += red(p)*kernel[dy+1][dx+1]; 
        g += green(p)*kernel[dy+1][dx+1];
        b += blue(p)*kernel[dy+1][dx+1];
      }
    }
    // set the color
    pixels[x+y*width] = color(r,g,b);
  }
  // show after/before
  for (int x=0; x<width/2; x++)
    pixels[x+width/2+y*width] = baker.pixels[x+y*width];
}
updatePixels();

[applet]

This sketch makes use of a two-dimensional array (rather than squashing it down into one dimension, as we did with the pixels). It is accessed with two indices. Futhermore, we give the values for the array elements are given when we declare the array, with each row in curly braces and the elements separated by commas.

Another image processing technique is to identify edges in a picture. Where there's an edge, we see a sharp transition in pixel value, e.g.:

10	12	13
12	200	237
10	225	219

To pick up on these transitions, we want to compare the center pixel against its neighbors, and see how similar/different they are. So rather than averaging, we subtract each neighbor from the center, giving us a kernel:

-1	-1	-1
-1	8	-1
-1	-1	-1

The factor of 8 for the center pixel is because each of the neighbors is subtracted from it. We make just a single-line change to our previous sketch to include this kernel:

float[][] kernel = { {-1, -1, -1},
                     {-1, 8, -1},
                     {-1, -1, -1} };

[applet]

Simply changing the center value from 8 to 9 yields a "sharpen" filter, instead of the edge detector. With this kernel, we take the average difference, as in the edge detector, and then add it to the original value. Shiffman 15-12 illustrates.

Yet other kernels pick up on other properties of a neighborhood.