Environmental Microorganism Classification Using CRFs and Deep CNNs

Environmental Microorganism Classification Using Conditional Random Fields and Deep Convolutional Neural Networks

Sergey Kosov, Kimiaki Shirahama, Chen Li and Marcin Grzegorzek

University of Siegen (Germany), Northeastern University (China) and University of Economics in Katowice (Poland)

Pattern Recognition 2018

Abstract

The labeling of Environmental Microorganisms (EM) which help decomposing pollutants, plays a fundamental role for establishing sustainable ecosystem. We propose an environmental microorganism classification engine that can automatically analyze microscopic images using Conditional Random Fields (CRF) and Deep Convolutional Neural Networks (DCNN). First, to effectively represent scarce training images, a DCNN pre-trained for image classification using a large amount of data is re-purposed to our feature extractor that distils pixel level features in microscopic images. In addition, pixel-level classification results by such features can be refined using global features that describe the whole image in toto. Finally, our CRF model localizes and classifies EMs by considering the spatial relations among DCNN-based features, and their relations to global features. The experimental results have shown 94.2% of overall segmentation accuracy and up to 91.4% mean average precision of the results.

An overview of our CRF-based EM classification and segmentation framework.

We re-purpose one of the most successful DCNNs, called VGG-16, which has been trained on 1.3 million images in ImageNet dataset, and fine-tune it using training EM images. For an image, feature maps output by the second last layer of the re-purposed VGG-16 are used to generate pixel-level features. We also extract global features from the image. Then extracted features together with the ground truth data are used to train Random Forest (RF) classifiers by analyzing pixel-level and global features in training images. The trained RF classifiers are used as the unary potentials by the CRF model. Finally, together with the pairwise potentials, the CRF model is applied to localize and label into the classes the objects of interest in the test EM images.

Downloads

A more general implementation of the project is available for download. The code is under BSD license, feel free to use and modify it. We’re happy to hear about applications of this research, please send us an email if you use the code. If you use it for a publication, please cite our paper. If you think you found a bug in the code, please let us know as well. We can also provide assistance with installing, understanding, or running the code.

Paper

BibTex

EM Dataset

Code

Environmental Microorganism Classification Using CRFs and Deep CNNs

About The Authors

Sergey Kosov

received his Diploma in Applied Mathematics from the Kirgiz-Russian Slavic University, Kirgyzstan in 2004, and M.Sc. degree in Computer Science from the Saarland University, Germany in 2008. From 2008 to 2013, he worked as a researcher in the Max Plank Institute for Informatics and Leibniz University, Germany. Currently he is an external Ph.D. student at Pattern Recognition Group in University of Siegen, Germany. His research interests include classification with conditional random fields and deep neural networks, motion estimation with optical flow, 3-D reconstruction as well as movie industry.

Kimiaki Shirahama

received his B.E., M.E. and D.E degrees in Engineering from Kobe University, Japan in 2003, 2005 and 2011, respectively. After working as an assistant professor in Muroran Institute of Technology, Japan, since 2013, he is working as a postdoctoral researcher at Pattern Recognition Group in University of Siegen, Germany. From 2013 to 2015, his research activity was supported by the Postdoctoral Fellowship of Japan Society for the Promotion of Science (JSPS), and is now supported within a project of German Federal Ministry of Education and Research (BMBF). His research interests include multimedia data processing, machine learning, data mining and sensor-based human activity recognition. He is a member of ACM SIGKDD, ACM SIGMM, the Institute of Image Information and Television Engineers in Japan (ITE), Information Processing Society of Japan (IPSJ) and the Institute of Electronics, Information and Communication Engineering in Japan (IEICE).

Chen Li

received his B.E. degree in the University of Science and Technology Beijing, China in 2008, M.Sc. degree in the Northeast Normal University, China in 2011, and Dr.-Ing. degree in the University of Siegen, Germany in 2016. From 2016 to 2017, he worked as a postdoctoral researcher in the Johannes Gutenberg University of Mainz, Germany. Currently, he is working as an associate professor in the Northeastern University, China. His research interests are machine learning, pattern recognition, microscopic and biomedical image analysis.

Marcin Grzegorzek

is Head of the Research Group for Pattern Recognition at the University of Siegen, Professor at the Department of Knowledge Engineering at the University of Economics in Katowice and Chairman of the Board of Data Understanding Lab Ltd. He studied Computer Science at the Silesian University of Technology, did his Ph.D. at the Pattern Recognition Lab at the University of Erlangen-Nuremberg, worked scientifically as Postdoc in the Multimedia and Vision Research Group at the Queen Mary University of London and at the Institute for Web Science and Technologies at the University of Koblenz-Landau, did his habilitation at the AGH University of Science and Technology in Kraków. He published more than 100 papers in pattern recognition, image processing, machine learning, and multimedia analysis and acted as examiner in 18 finalised doctoral procedures. For the time being, he runs eight externally funded research projects. For instance, Marcin coordinates the project CogAge (www.cognitivevillage.de) aiming at developing a user-friendly support system for elderly that applies machine learning algorithms for sensor-based health assessment.