Research story · Statistical image analysis
How can we get reliable information from images?
Images of seeds are both data for classifying plants and a test case for image segmentation. This line of work has followed three connected threads: separating seeds from the background, classifying seeds from their images and validating segmentations with classifiers, up to PARSEG, which needs only a small sample of pixels.
At a glance
- Period
- 2011 to 2024
- Threads
- Segmentation by background subtraction; classification of seeds; validation of segmentation
- Selected collaborators
- Jaromír Antoch, Claudio Conversano, Francesco Mola, Gianluigi Bacchetta. All authors are listed with each publication.
- Data
- Images of botanical seeds
- Key paper
- Scientific Reports 2024 (open access)
- Code
- R implementation on GitHub (opens in a new tab)
The question
Image segmentation separates the objects of interest, such as seeds, from the background, and many algorithms can do it. Images of seeds are also used to measure their size, texture and shape and to classify them. But how do we know that a segmentation is right, and how can we check it when an image has millions of pixels?
Where it started 2011–2016
The first works of this line date from the years of my joint PhD, in Probability and Mathematical Statistics at Charles University in Prague and in Environmental and Applied Botany at the University of Cagliari, which I completed in 2016 with a thesis on morpho-colorimetric and non-parametric analysis in the statistical classification of vascular flora.
In October 2011 I was a visiting fellow at the Department of Probability and Mathematical Statistics of Charles University, to work on classification methods based on Fourier analysis, a technique that also appears in the study of Prunus seeds described below.
Separating the seed from the background 2015–2016
Segmentation algorithms separate the foreground from the background using the intensity of the pixels, so it is good practice to choose a background colour as different as possible from that of the objects. When many images are digitized, it is useful to take this choice away from the user and automate the process; moreover, high-resolution images slow down the algorithms based on local thresholding.
We adapted background subtraction, a method used to detect moving objects in videos, to image segmentation. It combines local and global thresholding, gives good segmentations, automates the process when the colour of the objects is not constant and speeds it up considerably; on images of botanical seeds, its results were compared statistically with those of standard segmentation methods (Acta Universitatis Palackianae Olomucensis. Facultas Rerum Naturalium. Mathematica, 2016 (opens in a new tab)). Two conference contributions had preceded the article (ISI 2015; ODAM 2015).
Classifying seeds from their images 2018–2020
In taxonomic studies, digital image analysis is increasingly used to compare and discriminate groups of seeds. A conference contribution compared non-parametric approaches to the classification of seeds (ISBIS 2018).
The many variables measured on seeds, on size, texture and shape, are generally used in classification algorithms without any pre-processing, which brings noise and makes the process less consistent for new samples. In a study of 19 Sardinian cultivars of Prunus domestica and four cultivars of other Prunus species, the variables of each kind were pre-processed separately, with principal component analysis and Fourier analysis.
The accuracy of each kind of data was then measured by comparing k-nearest neighbours, linear discriminant analysis, naive Bayes, support vector machines and random forests. The combination of size, texture and shape discriminated the seeds well, and image analysis with pre-processed data proved useful down to the cultivar level (Plant Biosystems, 2020 (opens in a new tab)).
Is the segmentation right? 2016–2021
A statistical approach to validation uses the labels that a segmentation method assigns to the pixels, foreground or background, as the response of a classifier that predicts them from the colour of each pixel: the better the two agree, the better the segmentation method is ranked.
Conference contributions of those years dealt with evaluating an image segmentation method (ISBIS 2016) and validating the segmentation of images of botanical seeds (Statistische Woche 2016; CLADAG 2017); one proposed a classification-based approach to validating segmentation algorithms (y-BIS 2019). In December 2017 I visited Charles University again, to work on computationally efficient non-parametric methods for validating statistical image segmentation.
We proposed MaLIV, a machine learning approach that ranks the outputs of different gray-level thresholding segmentation algorithms, taking into account both the computational complexity of the validation and the robustness of its results. It uses subsampling to find Fisher-consistent estimates of validity measures from a sample of pixels of very small size, combined with one of three alternative approaches: learning curves, asymptotic regression and convergence in probability. The article presents experiments on the validation of five images segmented with thirteen algorithms (Statistical Papers, 2021 (opens in a new tab)).
How many pixels are enough? 2024
When an image has millions of pixels, validation becomes very demanding, both in practice and in computing time. PARSEG, which we developed, rests on an empirical finding: the results of validation obtained by training a classifier on all the pixels of an image are consistent with those obtained from much smaller random samples of pixels of a suitable size (Scientific Reports, 2024 (opens in a new tab)). The procedure:
- partitions the image into subsets of pixels of approximately equal size;
- on one subset, finds the minimum size of a random sample of pixels that preserves the information needed for validation, by studying how the performance of the classifier changes with the sample size;
- validates the remaining subsets using only samples of that size, and combines the results.
1 · Image
2 · Partition and random sample
3 · Smallest sufficient sample
On images of seeds of about 13 million pixels, PARSEG gives roughly the same precision as validation on the entire image.
about 4% of the pixels are used
about 90% less computing time
Values reported in the abstract of the article, for the validation of a binary segmented image.
Code
The R implementation of PARSEG (opens in a new tab), with a reproducible example, is available on GitHub under the MIT license.
Publications in this line
2024
Scientific Reports, 14(1): 6052
Validating a binary image segmentation from a small, optimally chosen sample of pixels, which cuts the computational cost of validation.
doi:10.1038/s41598-024-56228-6 (opens in a new tab) Code (opens in a new tab)Cite
@article{frigau2024parseg, author = {Frigau, L. and Conversano, C. and Antoch, J.}, title = {PARSEG: A computationally efficient approach for statistical validation of botanical seeds’ images}, journal = {Scientific Reports}, year = {2024}, volume = {14}, number = {1}, pages = {6052}, doi = {10.1038/s41598-024-56228-6}, issn = {2045-2322}, }
2021
Statistical Papers, 62(3): 1363–1386
doi:10.1007/s00362-019-01138-3 (opens in a new tab)Cite
@article{frigau2021consistent, author = {Frigau, L. and Conversano, C. and Mola, F.}, title = {Consistent validation of gray-level thresholding image segmentation algorithms based on machine learning classifiers}, journal = {Statistical Papers}, year = {2021}, volume = {62}, number = {3}, pages = {1363--1386}, doi = {10.1007/s00362-019-01138-3}, issn = {1613-9798}, }
2020
A statistical approach to the morphological classification of Prunus sp. seeds (opens in a new tab)
Plant Biosystems, 154(6): 877–886
doi:10.1080/11263504.2019.1701126 (opens in a new tab)Cite
@article{frigau2020statistical, author = {Frigau, L. and Antoch, J. and Bacchetta, G. and Sarigu, M. and Ucchesu, M. and Zaratin Alves, C. and Mola, F.}, title = {A statistical approach to the morphological classification of Prunus sp. seeds}, journal = {Plant Biosystems}, year = {2020}, volume = {154}, number = {6}, pages = {877--886}, doi = {10.1080/11263504.2019.1701126}, issn = {1126-3504}, }
2019
Classification-based Approach for Validating Image Segmentation Algorithms
Proceedings of y-BIS 2019, p. 42. Mimar Sinan Fine Arts University Publications
Cite
@inproceedings{frigau2019classification, author = {Frigau, L. and Mola, F. and Contu, G.}, title = {Classification-based Approach for Validating Image Segmentation Algorithms}, booktitle = {Proceedings of y-BIS 2019}, publisher = {Mimar Sinan Fine Arts University Publications}, address = {Istanbul}, year = {2019}, pages = {42}, isbn = {978-605-5005-95-5}, }
2018
Comparison of Non-parametric Approaches in Classification of Seeds
Book of Abstracts of ISBIS 2018
Cite
@inproceedings{frigau2018comparison, author = {Frigau, L.}, title = {Comparison of Non-parametric Approaches in Classification of Seeds}, booktitle = {Book of Abstracts of ISBIS 2018}, year = {2018}, }
2017
Validation of Experiments Involving Image Segmentation of Botanic Seeds
Proceedings of CLADAG 2017. Universitas Studiorum
Cite
@inproceedings{antoch2017validation, author = {Antoch, J. and Conversano, C. and Frigau, L. and Mola, F.}, title = {Validation of Experiments Involving Image Segmentation of Botanic Seeds}, booktitle = {Proceedings of CLADAG 2017}, publisher = {Universitas Studiorum}, address = {Mantua}, year = {2017}, isbn = {978-88-99459-71-0}, }
2016
Classification of Images Background Subtraction in Image Segmentation (opens in a new tab)
Acta Universitatis Palackianae Olomucensis. Facultas Rerum Naturalium. Mathematica, 55(1): 73–86
Cite
@article{mola2016classification, author = {Mola, F. and Antoch, J. and Frigau, L. and Conversano, C.}, title = {Classification of Images Background Subtraction in Image Segmentation}, journal = {Acta Universitatis Palackianae Olomucensis. Facultas Rerum Naturalium. Mathematica}, year = {2016}, volume = {55}, number = {1}, pages = {73--86}, url = {https://dml.cz/handle/10338.dmlcz/145819}, }Validating image segmentation of botanical seeds
Book of Abstracts of Statistische-Woche 2016
Cite
@inproceedings{frigau2016validating, author = {Frigau, L. and Mola, F. and Conversano, C.}, title = {Validating image segmentation of botanical seeds}, booktitle = {Book of Abstracts of Statistische-Woche 2016}, year = {2016}, }Evaluating the Effectiveness of an Image Segmentation method
Book of Abstracts of ISBIS 2016, p. 43
Cite
@inproceedings{frigau2016evaluating, author = {Frigau, L. and Conversano, C. and Mola, F.}, title = {Evaluating the Effectiveness of an Image Segmentation method}, booktitle = {Book of Abstracts of ISBIS 2016}, year = {2016}, pages = {43}, isbn = {978-972-674-795-6}, }
2015
Background Subtraction in Image Segmentation
Book of Abstracts of ODAM 2015
Cite
@inproceedings{mola2015background, author = {Mola, F. and Antoch, J. and Frigau, L. and Conversano, C.}, title = {Background Subtraction in Image Segmentation}, booktitle = {Book of Abstracts of ODAM 2015}, year = {2015}, }Improving the performance of image segmentation methods through background subtraction
Proceedings of ISI 2015, pp. 56–61. International Statistical Institute
Cite
@inproceedings{frigau2015improving, author = {Frigau, L. and Conversano, C. and Mola, F.}, title = {Improving the performance of image segmentation methods through background subtraction}, booktitle = {Proceedings of ISI 2015}, publisher = {International Statistical Institute}, address = {The Hague}, year = {2015}, pages = {56--61}, isbn = {978-90-73592-35-3}, }