When surgeons remove a tumor, the question that haunts the operating room is deceptively simple: did they get it all? For dogs with soft tissue sarcomas, the answer traditionally arrives days later, when a pathologist has sliced, stained, and studied the excised tissue under a microscope. If cancerous cells turn out to touch the surgical edge, the margin is called positive, the local recurrence risk climbs, and the dog may need a second operation. A new study published in Veterinary Oncology by researchers at The Ohio State University offers a striking alternative: a deep learning system that reads polarization-sensitive optical coherence tomography images of excised tumor margins and flags residual cancer with an area under the receiver operating characteristic curve of 0.989 and 91 percent accuracy, all within minutes of surgery.
Soft tissue sarcomas are malignant, locally invasive tumors arising from mesenchymal cells, and they are among the most common cancers in dogs. Sarcomas account for roughly 10 to 15 percent of malignant tumors in dogs, and about 80 percent of those are soft tissue sarcomas rather than bone tumors. The primary treatment is surgical removal, and the success of local control hinges on whether histological assessment of the margins confirms complete excision. Positive margins leave cancerous cells behind, increasing the risk of local recurrence and the morbidity that comes with it. The stakes are therefore high for getting margin information quickly, ideally while the patient is still on the table.
Optical coherence tomography, or OCT, is the imaging technology at the heart of this effort. It uses near-infrared light to generate real-time, high-resolution images of tissue microstructure, in much the same way ultrasound uses sound waves but with micrometer-scale resolution. Traditional spectral-domain OCT builds images from the intensity of reflected light, revealing depth-resolved structural detail. The technology has been tested for margin assessment in human breast cancer, where it achieved sensitivities of 92 to 100 percent for detecting positive margins, and human clinical trials are underway. The Ohio State team, led by Yuanlong Wang, Laura E. Selmic, and Ping Zhang, has now extended this approach to companion animals with an important upgrade: polarization sensitivity.
Polarization-sensitive OCT, or PS-OCT, is a set of hardware and software extensions that track the polarization state of the light reflected from tissue. This adds contrast mechanisms that ordinary OCT cannot provide. The key property is birefringence, an optical signature that arises from the arrangement of subcellular collagen within tissue and reflects how organized that tissue is. When tissue is damaged, degenerated, or necrotic, its structure breaks down and its birefringence drops. Cancerous tissue, with its disrupted architecture, therefore looks measurably different from healthy fat or muscle under polarization contrast. In prior studies of human breast tissue, PS-OCT demonstrated both qualitative and quantitative differences between cancerous and normal tissue. The system used in the study, a Thorlabs Telesto PS-OCT with a 1300 nanometer central wavelength, 3.5 millimeter imaging depth, and 5.5 micrometer axial resolution in air, synchronously captures traditional OCT images and three polarization metrics: retardation, optic axis, and degree of polarization uniformity, known as DOPU.
The physics behind these metrics is elegant. Two cameras record the reflected light as complex numbers, from which total intensity is computed pixel-wise for the standard OCT image. Retardation measures the difference in optical path experienced by two orthogonal linearly polarized states, calculated as an angle that reflects the ratio of irradiances in each polarization channel. The optic axis describes the orientation of the tissue’s birefringent axis within the plane transverse to the beam, derived from the Stokes parameters that fully characterize the polarization of light. DOPU quantifies the uniformity of polarization within a pixel neighborhood, ranging from 0 to 1, and serves as a regularized measure of how orderly the tissue’s polarization response is. Together with the intensity image, these four channels provide a far richer description of tissue than intensity alone, and the study set out to prove that a neural network could exploit that richness.
The researchers enrolled 48 canine soft tissue sarcoma specimens under an Institutional Animal Care and Use Committee approved protocol, ultimately analyzing 40 after excluding eight tumors that turned out not to be sarcomas. Board-certified veterinary surgeons excised the tumors with margins chosen purely on clinical grounds, and each specimen was wrapped in saline-soaked gauze to prevent drying before imaging. The team scanned the entire surgical margin in B-mode, continuously sweeping the tissue, and captured paired OCT and PS-OCT frames. An expert reviewed image quality, and a pathologist’s evaluation of corresponding histopathology sections provided the gold standard tissue labels. From 140 image pairs, the team cropped 1,553 patches using a sliding window with a 50-pixel stride, resizing each to 224 by 224 pixels to fit the ResNet50 backbone, a convolutional neural network architecture widely used in medical imaging.
The heart of the technical contribution lies in how the four image channels are combined. The team tested two fusion strategies. Early fusion simply concatenates all four images along the channel dimension and feeds them into a single ResNet50 backbone. Joint fusion instead runs four separate backbones, one per metric, and merges their learned feature vectors through a learnable weighted sum before a linear classifier makes the final cancer-versus-normal call. The joint fusion model won decisively, though at the cost of four times the parameters, longer training, and greater vulnerability to overfitting, a trade-off the authors describe explicitly between performance and computational complexity. Training used a 70-15-15 split of train, validation, and test data, with dogs kept whole within a single split to prevent patient overlap, five-fold cross-validation for hyperparameter tuning, random horizontal flips for augmentation, and early stopping against the validation set. Performance was assessed with AUROC, area under the precision-recall curve, F1 score, precision, recall, and accuracy, with uncertainty estimated by bootstrapping the test set 1,000 times.
The results were unambiguous. Both fusion strategies substantially outperformed a baseline model trained solely on traditional OCT images, which the authors attribute to the complementary polarization information. Adding PS-OCT metrics improved AUROC by up to 0.15 for cancerous image classification, and the more polarization metrics included, the greater the gain. The final joint fusion model reached 0.989 AUROC and 91 percent accuracy in detecting positive margins. Notably, the OCT-only baseline achieved the highest recall but at a punishing cost to precision, illustrating why intensity alone is insufficient for reliable intraoperative decisions. Ablation experiments with partial inputs, pairing OCT with just one PS-OCT metric at a time, confirmed that each polarization channel contributes, and that the full multimodal model is the strongest performer on the threshold-agnostic metrics that best reflect underlying discriminative power.
Perhaps the most clinically compelling feature is the diagnostic curve. Rather than simply reporting whether an image contains cancer, the model slides a fine window with a 5-pixel stride across the original image, computes the cancer probability for each patch, and aggregates overlapping predictions into a one-dimensional curve showing the probability of cancer at every horizontal position. In case studies, the curve stayed flat for pure cancerous and pure normal images, and for a mixed image containing tumor in roughly one third of the frame, it rose sharply over the cancerous region and tapered gradually across the margin into fat. This effectively performs a one-dimensional segmentation of the tumor, giving surgeons a map of where cancerous tissue lies rather than a bare yes-or-no verdict, and the window size and aggregation method remain adjustable to clinical preference.
The authors are candid about limitations. The training set is relatively small, and convolutional networks depend heavily on the comprehensiveness of their data, so broader validation is needed before generalizability can be trusted. Raw OCT images carry optical artifacts, motion blur, and noise whose effects on the model remain unquantified, and the fusion strategies explored were deliberately simple, leaving higher-order interactions between polarization metrics unexploited. The model is also far from clinical deployment; explainability, integration into surgical workflows, and careful validation of its effect on surgical outcomes all remain ahead. Still, the trajectory is clear. By teaching a neural network to read the polarization fingerprints that cancer leaves in collagen, this work moves real-time, AI-assisted margin assessment from a promising concept toward a practical tool, one that could spare dogs a second surgery and, if the companion-animal findings translate, may one day help human surgeons answer that oldest of operating room questions with far greater confidence.
Subject of Research: Deep learning-based intraoperative surgical margin assessment for canine soft tissue sarcoma using polarization-sensitive optical coherence tomography
Article Title: Deep learning-assisted surgical margin assessment for canine soft tissue sarcoma based on polarization-sensitive optical coherence tomography
Article References: Wang, Y., Selmic, L. E., & Zhang, P. (2025). Deep learning-assisted surgical margin assessment for canine soft tissue sarcoma based on polarization-sensitive optical coherence tomography. Veterinary Oncology, 2(1), Article 17. https://doi.org/10.1186/s44356-025-00032-5
Image Credits: AI Generated
DOI: 10.1186/s44356-025-00032-5
Keywords: deep learning, polarization-sensitive OCT, optical coherence tomography, soft tissue sarcoma, surgical margins, veterinary oncology, canine cancer, convolutional neural networks, birefringence, cancer imaging, intraoperative diagnosis, ResNet50
Cite Scienmag News
APA MLA Chicago
Nathaniel Bowman. (October 2, 2026). AI Reads Light’s Polarization to Spot Cancer Left Behind in Dog Tumors. Scienmag. https://scienmag.com/ai-reads-lights-polarization-to-spot-cancer-left-behind-in-dog-tumors/
Nathaniel Bowman. “AI Reads Light’s Polarization to Spot Cancer Left Behind in Dog Tumors.” Scienmag, 2 October 2026, https://scienmag.com/ai-reads-lights-polarization-to-spot-cancer-left-behind-in-dog-tumors/. Accessed 2 October 2026.
Nathaniel Bowman. “AI Reads Light’s Polarization to Spot Cancer Left Behind in Dog Tumors.” Scienmag. October 2, 2026. https://scienmag.com/ai-reads-lights-polarization-to-spot-cancer-left-behind-in-dog-tumors/
Copy citation Download RIS
Tags: AI-assisted tumor margin assessmentAI-based histological assessment in veterinary surgerybirefringencecancer imagingcanine cancerconvolutional neural networksdeep learningdeep learning for cancer detection in surgical marginsimproving surgical outcomes in canine soft tissue sarcomasintraoperative diagnosismachine learning for cancer detection in veterinary medicineoptical coherence tomographyoptical coherence tomography for tumor imagingpolarization-sensitive OCTpolarization-sensitive optical coherence tomography in veterinary oncologyrapid intraoperative cancer detection in dog tumorsreal-time tumor margin assessment in dogsresidual cancer detection using AI imagingResNet50soft tissue sarcoma surgical margin analysissoft-tissue sarcomasurgical marginsveterinary oncologyveterinary tumor margin evaluation technology
