This website will shut down soon. We Recommend Completing The Lessons On Our New Website CodeSciLab.Org

Image processing in Python

Why is image processing useful in science?

You may already have experience with some form of image processing in your daily life. When you crop photos, apply filters on Instagram or Snapchat, or use effects like Blur or Sharpen in Photoshop, you are using algorithms that help highlight information that you want your images to convey. In scientific computing, we use image processing to make it easier to make certain measurements or to find something specific of interest within the image. Image processing is important for analyzing information from a variety of instruments, such as MRI machines, microscopes, telescopes, and of course, regular cameras. Here are some examples of things we might want to do:

• Identifying similarities between images (useful for tasks like Google’s reverse image search)

• Make it easier to detect specific objects in an image, such as small, faint exoplanets orbiting a very bright star or a tumor in an MRI.

• Analyzing changes between images. This might be used to track how something like a bacterium is moving in a video (which is essentially a series of images), or to look for new objects that might have appeared in the sky.

• Make measurements or count objects. Perhaps you’re interested in how large a galaxy is, or how many people are in an image.

How are images stored in Python?

The picture of flowers above is what you’re used to thinking of as an image. But to a computer, an image is a series of numbers that are organized in a specific way. We demonstrate this in the code block below:

• The filename of the photo is “Flowers.jpg”. This filename is a string in Python.

• To turn this image into a form that Python can read, we will use the free and publicly available scikit-image library (also sometimes called a package). Lesson 2 taught us that a library is a collection of code that other people can use as part of their own software projects. Note that scikit-image is imported as “skimage” because the name used inside Python is not necessarily the same as the name that we call a package in writing.

• Within the scikit-image library is a function called io.imread. When you pass your filename to the io.imread function, the function will return a NumPy array (notice that the scikit-image library itself uses the NumPy library). For a refresher on NumPy arrays, see Lesson 2. In order to do more with the NumPy array holding the image information, we need to store it in the variable that we choose to call “imagefile.” (Note that running the code will be a little slow because the scikit-image package has to be downloaded by replit; however, if you're working with a Python installation on your own computer, you only have to download and install new packages the first time you use them). 

• Since “imagefile” is a NumPy array, we can print out its shape, which is (1464, 1687, 3). 1464 represents the number of pixels in the image in the vertical direction. 1687 represents the number of pixels in the horizontal direction. Finally, 3 represents the number of colors that we can break this image down into: red, green, and blue (in that order). It is very common for computers to represent colors as some combination of these three colors. This is known as RGB for short.

• We replot the image with the matplotlib library (see Lesson 2 for a refresher on matplotlib). You should get a plot that looks like the image below. The numbers along the axes of the photo correspond to pixel numbers, so we can think of a photo as a two-dimensional plot, where each (x,y) coordinate is associated with some value. In this case, the value is simply the color of the pixel. Note that the origin of the plot is in the upper left corner, which is probably different from the typical xy-plane that you're used to, which has the origin in the lower left corner. Remember that Python is zero-indexed, so the first pixel is pixel 0, not pixel 1. Each of these values should be an integer from 0 to 255. The larger the number, the more the individual color contributes to the overall color. If all three values are 0, the pixel is black. If they are all 255, the pixel is white.

We can see what the array looks like with the statement “print(imagefile).” It will print with ellipses because the array is so large, so this isn’t a good strategy for looking up what values correspond to what pixels in the image. However, since an image in Python is just a NumPy array, we can use the same functions that we would for other NumPy arrays. For example, we can select part of the image by using normal indexing. We can figure out the color of the pixel at a vertical position of 1200 and a horizontal position of 1491with the command “imagefile[1200,1491].” (Notice that the vertical coordinate comes first here, which is the opposite of how you’re probably used to writing coordinates in math class. The value that gets returned by Python should be a three-element array, [118,169,50]. This tells you that the pixel is dominated by the color green. This makes sense if you look at the image above, since this pixel is in a part of the picture with a lot of leaves.

Oftentimes, the types of images you'll analyze for a scientific purpose won't be in color. This might be because color is not important to the task at hand (e.g., if you’re tracking the motion of something under a microscope) or because the image is mapping some other quantity entirely (like an MRI, which maps signals coming from your tissues after a magnetic field is applied). Thus, image pixels do not necessarily have RGB values associated with each pixel. Instead, each pixel will be associated with a single value that denotes the amount of signal being measured (i.e., grayscale). In the case of photographs, the signal is light.

We will read the flowers image into Python again, but this time we will convert to grayscale. The code is nearly identical to what was shown previously, but we add an "as_gray=True" argument to the imread function call. 

Note that when we print the shape of the image now, we get (1464, 1687) instead of (1464, 1687,3) because each pixel now corresponds to a single intensity value rather than an RGB color. The value is also normalized so that the maximum value corresponds to 1, since the computer does not have information about your units.

You should also now get a plot of the image in grayscale like the one below:

Again, since the image is stored in Python as a NumPy array, we can select part of the image by using normal indexing. In the code above, the code "subimage = imagefile[1000:, :400]" will select all the pixels with y-positions above 1000 and x-positions up to 400 (i.e., the lower-left corner of the image. This is like cropping an image. The result is shown below. 

We can also identify the maximum (largest) and minimum (smallest) values of the image using np.max() and np.min(), respectively, each of which takes an array as an argument. We see that the maximum and minimum values are 1.0 and 0.00196, respectively. In this particular case, 1.0 corresponds to completely white, 0.0 corresponds to completely back, and values in-between correspond to shades of gray (larger values are ligther).

What is more useful is finding out where in the image these maximum and minimum values occur using the np.where() function, which returns all the values that meet a certain condition. For example, the command "max_yvals, max_xvals = np.where(imagefile==maxvalue)" will return the y and x coordinates of all the pixels in the image that are equal to the maximum value of the image. The locations of the pixels matching the minimum value can be found in a similar way. 

Using matplotlib, we take our grayscale image and make a scatterplot of the locations of the brightest pixels (in blue) and the darkest pixels (in red). We see that the maximum pixel values can be found in the flowers, while the darkest pixels (see the red dot in the lower right corner) can be found in a shadow.  

The results above suggest that if we want to pick out the locations of the flowers, we can try selecting the brightest pixels through a command such as "upper_yvals, upper_xvals = np.where(imagefile>=0.9maxvalue)," which returns the coordinates of all the pixels with values that are at least 90% of the maximum value. We make a scatterplot of these coordinates in pink below. We see that this simple step actually does a surprisingly decent job of picking out the pixels belonging to the flowers, but it's not perfect - we don't get all of the pixels belonging to the flowers, and we also selected some of the pixels belonging to leaves. Nevertheless, this demonstrates something important: we can search for features in an image if we know roughly what values they have, and we can figure out what kind of feature a pixel belongs to by measuring its value. The next part of the lesson will explore some more sophisticated image processing techniques. 

Now it's your turn!

Click on the Edit in Repl.it button in the upper right corner of the terminal below. This will take you to the repl.it website, where you can create a free account (recommended, if you haven't already). Then, you can write, run, and save your own code to answer the questions below (also provided as comments on the repl.it site):

If you have any trouble navigating the repl.it site, please see our Student Guide to Repl.it.