An algorithm that looks at and analyzes electron microscope images of steel surfaces is able to detect defects more than 80 times faster than human experts. The automated image analysis could bring this technique up to speed with a high rate of image collection from high-speed detectors in electron microscopes.

Electron microscopy images provide crucial information on microstructures of materials, especially defects and grain boundaries that can affect a material’s fundamental properties. Experts with years of experience examining electron micrographs can easily identify relevant microstructures. But such identification can involve judgement calls, which naturally introduces some variability into the data. Electron microscopes today also produce images at a faster rate than a researcher could ever analyze them: High-speed detectors can produce hundreds to thousands of electron micrographs in a second. “If you want to get value from all the data you can extract from a machine, you need a machine,” says Dane Morgan, co-leader of the Computational Materials Group at the University of Wisconsin, Madison, who published this work in a recent issue of npj Computational Materials.

Computer science techniques developed over the past 10 to 20 years are beginning to help automate electron micrograph analysis. A machine learning algorithm, given a set of electron micrographs with microstructures labeled by experts, can identify unique features that a computer vision algorithm could use to identify microstructures in other images. “There are a lot of [software] tools available, and these tools are not applied to microstructural analysis as much as they should have, given how important it is to materials engineering,” says Daniel Lewis, at Rensselaer Polytechnic Institute, who was not involved with the current work.

Researchers have used computer vision and machine learning previously to automatically classify microstructures in electron microscope images. However, few algorithms can detect the location of and quantitative information about the microstructures.

Morgan and his colleagues developed software to identify ellipse-shaped dislocation loops that are produced in steel surfaces following irradiation. To teach a computer what the loops look like, the researchers collected 270 transmission electron micrographs of ferritic steels containing 8,424 loops already identified by an expert. They also had a large set of images that did not contain defects, so that the computer also had graphical references for non-loop features.

The researchers fed the labeled images to a machine learning algorithm powered by a convolutional neural network. This type of algorithm quickly analyzes the intensity of pixels in an image and learns intensity patterns that correlate to recognizable objects. From the set of training data, this algorithm learned which patterns signified dislocation loops and it labeled those loops with a box. The researchers then used computer vision algorithms, different from the previous machine learning processing, to analyze the intensity of the pixels inside the box. In the electron micrographs, the loops appear as dark rings on a lighter background, so by analyzing pixel intensity, the computer vision algorithm could determine the diameter of the loop.

The automated workflow of machine learning processing followed by computer vision analysis identified previously labeled loops with about 80% precision, a metric that reflects how well the algorithm avoids incorrectly identifying a feature as a dislocation loop. About 80% of loops the program found were actually present, indicating that the algorithm detected most of the loops that humans had already identified. Performance was similar to that of experts reading the images. But the program was much faster: for a set of six images, the algorithm detected loops in an average of 27 seconds, more than 80 times faster than five human experts. Morgan imagines eventually using multiple processors and faster algorithms so that micrographs are analyzed as they are collected.

Using computer vision and machine learning tools to advance electron micrograph analysis—an area of research that has not changed much in the past century—is a good thing, Lewis says.

Read the abstract in npj Computational Materials.