Image-Based Profiling Enters Its Deep Learning Decade

Microscopy has quietly become one of the most information-rich data types in biology. A single image of stained cells contains millions of pixels that encode size, shape, texture, intensity, and spatial organization, and researchers have spent the past decade learning how to compress that visual richness into high-dimensional numerical profiles. This approach, known as image-based profiling, converts cell microscopy data into high-dimensional representations that enable the systematic analysis of cell responses to genetic or chemical perturbations at single-cell resolution. A comprehensive new review published in Molecular Systems Biology by Erik Serrano, John Peters, and colleagues from the CytoData Society traces the field’s rapid evolution and identifies the methodological and computational infrastructure that now underpins modern image-based profiling, along with the challenges that remain on the horizon.

The power of image-based profiling lies in its scalability and its agnosticism. Rather than measuring a predefined molecular target, profiles capture the phenotypic consequences of perturbations, allowing researchers to group compounds by mechanism of action, detect off-target effects, and identify subtle morphological changes that molecular assays might overlook. This has made the approach central to phenotypic drug discovery, functional genomics, and disease modeling. In functional genomics, image-based profiles quantify the phenotypic impact of CRISPR knockouts or RNAi perturbations, helping map gene function and pathway relationships at single-cell resolution. In disease modeling, morphological features derived from patient cells can distinguish disease subtypes, identify biomarkers, and screen for phenotype-correcting compounds. The landmark 2017 review by Caicedo and colleagues established the foundational standards for the field, outlining critical steps such as illumination correction, cell segmentation, feature quantification, feature processing, and profile generation. Since then, the field has advanced rapidly, driven by progress in experimental protocols, complementary profiling modalities, and rapidly advancing computational tools.

The experimental side of the workflow begins with careful design. Negative controls such as untreated or vehicle-treated samples establish baselines for normalization, while biological replicates capture natural variability and improve statistical power. Plate layout constrains replicate number; in a standard 96-well plate using the inner 60 wells, six replicates per treatment with flanking positive controls is common. The authors recommend including as many replicates as feasible, typically a minimum of four for treatments and at least nine for controls. Maintaining consistent seeding density and media conditions is equally critical, as inconsistent conditions can introduce artifacts like overcrowding or edge effects. Higher-density formats such as 1536-well plates minimize data loss from discarding edge wells affected by spatial biases. The Cell Painting assay, which stains multiple subcellular compartments across five to six channels, has become the predominant fluorescence-based approach, providing a rich and unbiased representation of cellular morphology. Label-free techniques including brightfield and phase-contrast imaging are increasingly explored for their non-invasive nature and compatibility with live-cell experiments.

Once images are acquired, the computational core of the pipeline takes over. Whole-image quality control removes out-of-focus images or those containing large smudges and debris that would decrease segmentation quality and introduce noise. Illumination correction follows, learning and correcting patterns of uneven background lighting and signal variations from optical artifacts so that downstream measurements reflect biological differences rather than technical noise. Cell segmentation, the delineation of individual cells from the background, has witnessed a decisive shift toward deep learning. Classical approaches implemented in tools like CellProfiler and Ilastik use intensity thresholding and watershed algorithms, but foundational architectures like U-Net catalyzed a transition to convolutional neural networks that achieve superior segmentation accuracy. This paved the way for tools such as StarDist and Cellpose that offer models capable of identifying cell borders with minimal parameter tuning. More recently, transformer-based foundation models like CellSAM and CellViT leverage these architectures to further improve performance and generalizability across diverse imaging conditions.

Feature extraction represents the most consequential transformation in the field. Classical workflows extract hundreds to thousands of hand-crafted features from segmented objects, quantifying aspects like size, shape, texture, intensity, and spatial relationships. These features provide a mathematically well-defined representation of cell morphology, and their transparency supports hypothesis-driven research. However, the feature set might miss particular phenotypic changes and can be prone to redundancy and noise. Deep learning methods learn relevant morphological representations directly from raw pixel data, producing embeddings that are typically less redundant and frequently outperform classical features in downstream tasks. The review highlights striking examples of both directions: the count of cells or mitochondria outperformed convolutional neural networks to predict assay outcomes, and mechanism of action prediction using K-nearest neighbors on CellProfiler features achieved an F1 score of 0.41, comparable to DeepProfiler’s 0.42. Yet self-supervised DINO frameworks increased mean average precision by 29 percent when classifying chemical perturbations compared to hand-crafted features, and zero-shot image-to-image mechanism of action classification using contrastive learning achieved 61.3 percent in top-10 accuracy compared to 24.5 percent with CellProfiler.

Two major neural network architectural families dominate deep learning-based feature extraction. Convolutional neural networks process images through stacked convolutional filters that perform localized transformations to extract features hierarchically, learning increasingly descriptive representations while maintaining relatively low computational demands. Vision transformers, adapted from natural language processing, partition an input image into non-overlapping patches that are flattened and linearly embedded, then processed in parallel through self-attention blocks that capture global relationships among all image patches at once. The scaling properties of transformers have made the architecture especially useful with large datasets. Training strategies have evolved in parallel. Supervised learning requires ground truth labels that are frequently unavailable in profiling experiments. Weakly supervised learning uses easily obtained annotations like perturbation labels, which are biologically meaningful but do not directly reveal phenotypic effects, as auxiliary tasks. Self-supervised learning relies entirely on unannotated image data, defining objective functions derived from the data itself, such as reconstructing masked patches or distinguishing between augmented views of the same image.

Post-processing deep learning features requires careful attention to order of operations. Applying a sphering transform to deep learning features followed by robust z-scoring against plate-level statistics yields the most effective post-processing strategy for self-supervised features, and performing z-scoring before sphering was found to substantially degrade performance in most cases. Feature selection, which is essential for classical features due to their high correlation, might be unnecessary for deep learning embeddings since only weak correlations are generally observed. Batch effect correction remains a persistent challenge regardless of feature type. A recent study evaluated ten batch correction techniques from single-cell RNA sequencing on image-based profiling data and concluded that none could adequately remove batch effects in the most complex cases involving multiple microscope types and a large number of compounds, underscoring the ongoing need for new methods.

The field is now moving toward foundation models, large models trained on massive and diverse datasets that can be adapted to solve a wide range of tasks. Early work repurposed natural-image convolutional networks by pseudo-RGB channel mapping, while more recent efforts have trained self-supervised vision transformers on millions of microscopy images. One masked autoencoder-based strategy scaled vision transformer models to approximately 1.9 billion parameters and datasets with roughly 16 million images, demonstrating that performance improves at that scale. Channel adaptation remains a significant hurdle, as most models have been trained using the five-channel Cell Painting assay while imaging panels vary with experimental goals. Several strategies have emerged, including models that process one channel at a time, channel-agnostic masked autoencoders that simultaneously process all channels adaptively, and novel transformer architectures incorporating channel-aware attention mechanisms.

Emerging frontiers are expanding the dimensions of profiling beyond static two-dimensional snapshots. Temporal imaging captures dynamic cell processes that static methods cannot, circumventing the survivorship bias of fixed time point profiles. Live-cell painting using minimally toxic dyes enables time-lapse imaging, and advances in spinning disk confocal and light sheet microscopy make acquisition of large temporal datasets possible. The third spatial dimension remains underexplored but holds promise for revealing more nuanced cellular signatures, particularly for complex cell shapes like neurons or macrophages with extended protrusions that may appear disconnected in a two-dimensional slice. Virtual staining, which predicts fluorescence signals from label-free modalities such as brightfield imaging using machine learning, offers lower cost and reduced phototoxicity. However, virtual staining has not yet been used in any published studies for image-based profiling applications, and its generalizability across unseen cell types and phenotypes remains highly variable and dependent on the biological context of training data.

Single-cell image-based profiling offers an effective, affordable, high-content alternative to single-cell RNA sequencing. While population-level aggregation creates robust representations that minimize technical noise, averaging leads to loss of information about cell heterogeneity and the diversity of cell states. Methods that capture heterogeneity, subpopulations, and cell state transitions have been applied to identify drug toxicity, cell types, cell cycle stages, and cell health alterations in rare subpopulations. Emerging evidence suggests that single-cell analysis may improve mechanism of action prediction compared to population-level analysis. Optical pooled screening has emerged as a transformative technology, enabling high-throughput single-cell resolution mapping of genotype-to-phenotype relationships at unprecedented scale by integrating high-content imaging with in situ genotyping methods. The review identifies six core challenges for the next decade: developing end-to-end workflows, fortifying methodological infrastructure for temporal, three-dimensional, and virtual staining modalities, bridging profiles to mechanisms through interpretable features, establishing rigorous quality control standards, building systematic benchmarking frameworks, and reaping the rewards of artificial intelligence and foundation models while accounting for their high computational costs. The authors emphasize that these challenges form a cohesive foundation for the next generation of image-based profiling, and that realizing the field’s promise depends on collective resolve to confront and overcome the obstacles that remain.

Subject of Research: Image-based profiling methods and computational challenges in high-content microscopy

Article Title: Progress and new challenges in image-based profiling

Article References: Serrano, E., Peters, J., Wagner, J., Graham, R. E., Chen, Z., Feng, B. Y., Miranda, G., Kalinin, A. A., Vulliard, L., Tomkinson, J., Mattson, C., Lippincott, M. J., Kang, Z., Sitani, D., Bunten, D., Seal, S., Carragher, N. O., Carpenter, A. E., Singh, S., … Way, G. P. (2026). Progress and new challenges in image-based profiling. Molecular Systems Biology, 22(5), 624-658. https://doi.org/10.1038/s44320-026-00197-7

Image Credits: AI Generated

DOI: 10.1038/s44320-026-00197-7

Keywords: image-based profiling, Cell Painting, deep learning, high-content screening, cell segmentation, foundation models, Progress, challenges, image-based, profiling, scientific research