Deep Reinforcement Learning Framework Uncovers Cellular Migration Mechanics in Living Embryos

A newly published computational framework named CellMigrationGym standardizes and expedites the use of deep reinforcement learning to investigate the mechanisms governing cell migration through three-dimensional live-cell imaging data. Cell migration is a fundamental biological phenomenon that underpins normal embryonic development as well as cancer, directing critical morphogenetic events in developing embryos and facilitating vital tissue renewal and wound repair in adult organisms. Understanding how migrating cells coordinate with neighboring cells and respond to mechanical and chemical cues within complex microenvironments remains a major challenge in developmental biology. By formulating cell migration as a sequential Markov decision process, CellMigrationGym enables researchers to integrate biological hypotheses as high-level regulatory rules and constraints within an agent-based simulation environment. Developed to provide an open, standardized infrastructure, the framework bridges computational learning libraries with digitized time-lapse microscopic observations, allowing scientists to infer the dynamic biological rules of cellular movement directly from data without prior knowledge.

To place this work in context, computational biologists have long sought mathematical and simulation tools capable of capturing cellular movement. Early physical models relied on differential equations to describe biomechanical principles in cell protrusion, adhesion, and retraction. While these equation-based models successfully explained cell migration in two-dimensional scenarios, they generally failed to capture the complexities of three-dimensional tissue environments. Consequently, agent-based modeling emerged as a suitable simulation approach where each cell is represented as an autonomous agent interacting with neighbors and its surroundings. Notable frameworks such as the Cellular Potts Model effectively simulated specific cell behaviors, while hybrid systems combining agent-based modeling with statecharts reproduced developmental dynamics, including ligand interaction distances in the Caenorhabditis elegans germ line and pancreatic organogenesis in mice. Nevertheless, conventional agent-based systems depend on predefined biological rules, which fundamentally limits their ability to discover novel, uncharacterized regulatory mechanisms when complete developmental pathways remain scientifically unknown.

Deep reinforcement learning overcomes these historical limitations by treating the migrating cell as an autonomous learning agent operating within an evolving embryonic tissue environment. In this formulation, the embryo serves as the environment, while cellular translocation is structured as a sequential Markov decision process. Hypothesized migration mechanisms—whether driven by external directional chemical gradients or collective multicellular behavior—are implemented as objective reward functions and environmental constraints. By comparing reinforcement learning outcomes directly with observational data, researchers can evaluate the biological validity of proposed regulatory rules. An earlier study applied hierarchical deep reinforcement learning to live-cell images of Caenorhabditis elegans embryogenesis, utilizing ubiquitous nuclear labels and simple empirical statistical rules to reveal a modular organization of cell movement confirmed by imaging experiments. Building on that foundation, CellMigrationGym transforms those pioneering methods into an open, unified framework that standardizes data input, hypothesis testing, and agent training across dynamic biological processes.

The architecture of CellMigrationGym comprises three primary components: the Tissue Environment, the Migrating Cell, and the standardized OpenAI Gym interface. The Tissue Environment operates as an agent-based model constructed upon the PyBullet physics engine. It ingests digitized microscopic images, builds a three-dimensional spatial tissue representation, tracks cell identities and coordinates across frames, and extracts names, locations, sizes, and lineage progeny. Within this digital environment, each migrating cell functions as a reinforcement learning agent selecting directional movements based on sensory state inputs and reward feedback. The OpenAI Gym toolkit coordinates interactions between the tissue environment and migrating agents through dedicated functions: Create establishes environmental connections, Get_state monitors tissue states, step coordinates stepwise interactions, render visualizes the simulation, and reset reinitializes parameters. Furthermore, interactive graphical tools enable users to adjust camera positions, angles, and distances via mouse and keyboard, generating three-dimensional perspectives, two-dimensional top views, and depth images.

To support robust computational training, the framework incorporates rigorous data preparation and standardization protocols for microscopic observations. In an experimental group of fifty wild-type Caenorhabditis elegans embryos, live-cell time-lapse imaging was performed using an inverted Zeiss AxioObserver Z1 microscope equipped with a 40x objective and MetaMorph software. Optical recordings attained resolutions of 0.254 micrometers along the X and Y axes and 0.1 micrometers across thirty focal planes in the Z axis. Lineage tracing and manual corrections were executed using Starrynite II and AceTree, before data was stored in AceTree format and converted into NumPy arrays. Because biological embryos display natural developmental disparities in migration timing, the authors devised a standardization method: they computed distance arrays between migrating and target cells, aligning them across six validation embryos using a calculated Midpoint. Furthermore, original sixty-second imaging intervals were linearly interpolated ten-fold to six-second intervals, providing fine-grained temporal resolution for training.

Within the Tissue Environment, the reward system formulates three essential biological relationships: interactions with neighboring cells, neighbor spatial distances, and destination proximity. The overall reward sums feedback from three models: the Neighbor Relationship Model, the Neighbor Distance Model, and the Destination Model. The Neighbor Relationship Model uses a random forest classifier trained on wild-type embryos with Voronoi ground truth, evaluating coordinates, sizes, and cell counts to identify neighbor groups. The Neighbor Distance Model evaluates the ratio between cell distance and the sum of estimated radii. Calibrated from fifty embryos, acceptable ratios span 0.3 to 0.8, where a linear penalty from negative one to zero is applied; ratios below 0.3 incur a collision penalty of negative one thousand, while ratios exceeding 0.8 receive no penalty. Finally, the Destination Model awards one hundred points when the migrating cell reaches its target within ninety percent of observed migration time and remains stable for five timesteps.

To demonstrate framework capabilities, the authors conducted a case study of Cpaaa intercalation during early Caenorhabditis elegans embryogenesis. The Cpaaa cell is born on the dorsal posterior embryo side and intercalates between two rows of ABarp cells, starting fifteen minutes post-birth and concluding when reaching target cell ABarpaapp. The researchers first tested an influence-by-gradient hypothesis, assuming migration is directed by a chemical gradient toward the target. In this configuration, the Destination Model provided continuous dense feedback proportional to target proximity alongside the final arrival reward. The Cpaaa agent utilized a deep Q-learning model whose policy network included two convolutional layers and a fully connected layer. Operating with eight directional actions and step sizes scaled to average speed, the model was trained using Adam across nine hundred epochs on an Nvidia DGX station in five hours using three embryos. While simulated distance matched observational data, no physical or chemical gradient has been biologically verified.

Next, the investigators explored a contrasting hypothesis proposing that Cpaaa migration is driven by collective cell behavior. Under this setup, the team deployed a two-level Hierarchical Deep Q-Network that introduced subgoals—defined as environmental cells along plausible migration paths—to shrink the search space. Subgoals were chosen as pairs of secondary neighbors identified by the Neighbor Relationship Model. The higher-level module used a fully connected network with three hidden layers of 512, 1024, and 1024 nodes to select subgoals based on Cpaaa position, coloring them white in visual representations. The lower-level module learned motor actions to reach each subgoal. The Destination Model supplied delayed sparse rewards: ten points for achieving a subgoal and one hundred for the final destination. Trained for nine hundred epochs across seven hours on the DGX station with three embryos, the model successfully guided Cpaaa to the destination, closely reproducing observational distance curves across ensemble simulations.

Crucially, unlike the gradient hypothesis, the collective cell behavior hypothesis was directly substantiated by biological evidence captured in time-lapse imaging. Microscopic observations revealed that Cpaaa coordinates with neighboring cells to mediate migration through the sequential formation of multicellular rosettes. Rosettes are defined as five or more cells joined at a common contact point. Live imaging demonstrated that Cpaaa and six neighboring ABarp cells formed three successive rosettes at approximately minute five, minute twelve, and minute seventeen after migration commenced, with minute zero defined as eighteen minutes after Cpaaa birth. Each rosette underwent sequential edge contraction and resolution, with structural transitions directly correlated with forward anterior movement of Cpaaa toward the target cell ABarpaapp. The authors identified this recurring structural arrangement, termed sequential rosettes, as a novel cellular movement mechanism guiding migration through developmental tissues, confirming that collective multicellular rosette dynamics provide the underlying mechanism driving intercalary cell navigation.

To evaluate the specific influence of surrounding cells, the researchers conducted in silico ablation experiments in CellMigrationGym, removing identified neighbor cells from input images before retraining the agent. When neighbor cells were present in embryo number one, the trained Cpaaa trajectory matched observational data in start and end regions, overall shape, and key turning points. In stark contrast, when neighbor cells were removed, Cpaaa accelerated abnormally, reaching the destination five minutes earlier than observed, along an erratic path with substantially higher variance across ensemble runs. These computational findings proved that neighboring cells play an indispensable regulatory role in governing cell migration paths and velocities. By standardizing the integration of deep reinforcement learning with empirical time-lapse microscopy, CellMigrationGym delivers a versatile open framework. The authors anticipate that this open platform will inspire broader applications of reinforcement learning to investigate other dynamic biological processes, including cell proliferation, gene expression dynamics, and embryonic neuronal activities.

Subject of Research: A computational framework using deep reinforcement learning and live-cell imaging to model cell migration mechanisms

Article Title: CellMigrationGym: a data-driven framework utilizing deep reinforcement learning to unravel mechanisms of cell migration

Article References: Wang, D., Yang, J., Wang, Z., Xu, Y., & Bao, Z. (2026). CellMigrationGym: a data-driven framework utilizing deep reinforcement learning to unravel mechanisms of cell migration. BMC Proceedings, 20(S10), Article 9. https://doi.org/10.1186/s12919-026-00365-5

Image Credits: AI Generated

DOI: 10.1186/s12919-026-00365-5

Keywords: Cell Migration, Deep Reinforcement Learning, CellMigrationGym, Computational Biology, Embryogenesis, Agent-Based Modeling, Live-Cell Imaging, Caenorhabditis elegans, data-driven, framework, utilizing, deep