Open-source pipeline pairs human judgment with AI to label tricky industrial images

Labeling images of industrial equipment sounds straightforward until two buttons look exactly alike. A new open-source software tool called HILIPS tackles this problem directly, combining the semantic reasoning of a multimodal large language model with human review to build training data for object detection in settings where appearance alone is not enough.

The tool, described in the journal SoftwareX, was developed by Taehyung Noh, Beejin Son, Junghyun Kim, Haein Yeo and Kyungsik Han, who define the core challenge as visual-semantic ambiguity: objects that appear visually similar but serve distinct functions. In such cases, the researchers argue, classification requires contextual cues such as embedded text or spatial relationships rather than fine-grained visual features alone. This differs qualitatively from standard fine-grained recognition benchmarks such as CUB-200 or Stanford Cars, where the difficulty lies in detecting subtle visual differences like beak shape. On a CNC control panel, buttons may be physically identical and distinguishable only by their printed labels or coordinates.

The problem matters because collaborative robots are increasingly common in industrial environments, taking on tasks such as control panel operation, equipment inspection and maintenance support. Reliable object recognition is essential for these tasks, yet building high-quality training data that reflects the diverse environmental conditions and object variety of industrial settings remains a significant challenge. Image labeling is often the primary bottleneck, requiring substantial time and manual effort. The authors also point to a mismatch in expertise: domain experts understand the target objects but are unfamiliar with structured annotation workflows, while annotation workers are proficient with labeling tools but lack domain-specific knowledge. Workers without a domain background must repeatedly consult reference materials, and extended sessions lead to decision fatigue that degrades labeling consistency.

Existing annotation tools offer only partial help. Traditional platforms such as LabelMe and VGG Image Annotator mainly support manual annotation, while newer systems such as CVAT and Label Studio can integrate the Segment Anything Model (SAM) for boundary extraction, but semantic label assignment still requires user input. Grounded SAM-2 combines text-guided grounding with SAM2, although the HILIPS team notes that their study does not directly compare HILIPS with Grounded SAM-2 in the target industrial setting. Multimodal large language models, which jointly process images and text, offer a promising direction for interpreting complex visual scenes from natural-language descriptions, particularly in cold-start scenarios where no training data exist. However, the high inference cost and latency of these models make their direct deployment in real-time industrial environments impractical.

HILIPS resolves this tension with a three-stage workflow. In the first stage, Cold-start Labeling, the system pairs the Google Gemini 2.5 Pro API with SAM2. Given an image and a user prompt, the language model returns bounding boxes and class labels for the described objects, and those boxes are passed to SAM2 as prompts to produce precise segmentation masks. SAM2 runs locally, removing the dependency on an external segmentation API. The second stage, Label Transfer, uses the user-reviewed annotations as ground truth to train a lightweight YOLOv8 detection model through what the authors call supervised label transfer, chosen for its favorable accuracy-speed trade-off and its ability to run on edge devices. In the third stage, Iterative Refinement, the trained model is deployed back into the labeling pipeline: detections at or above an auto-approval threshold, set to 0.80 by default, are automatically approved and shown in green, while lower-confidence detections are shown in yellow and routed to the user for review. Corrected annotations are appended to the training set, and with each retraining cycle the fraction of auto-approved detections increases.

The system is implemented as a web application with a Next.js 15 and React 19 frontend built in TypeScript and a FastAPI backend managing data storage and machine learning pipelines. Users can mix three annotation methods within a single image: manual polygon drawing for complex or irregular shapes, point-based labeling in which a single click triggers SAM2 to generate a segmentation mask, and prompt-based auto-labeling in which the user describes target objects in natural language. During training, users configure hyperparameters such as epochs, batch size, image size and learning rate through the interface, and the system reports mean average precision, precision and recall on a held-out validation set. Each trained model is saved with a version tag so users can compare runs. The MLLM backend, segmentation model and detection model are each designed as independently swappable modules, so upgrading one component does not require changes to the rest of the pipeline.

To test whether the workflow actually reduces labeling effort, the team ran a within-subjects user study with ten participants who had varied levels of experience with object-detection or labeling tools but were not image-annotation experts. After a tutorial and a five-minute practice session, each participant labeled ten images per method in a randomized order: fully manual polygon drawing and class assignment, SAM2-assisted labeling in which masks came from point clicks but class labels were assigned manually, and YOLO auto-labeling in which the trained model proposed labels for review. After each method, participants completed the NASA-TLX workload questionnaire and the System Usability Scale, and the system logged objective metrics including time per image and interaction count. Because the study involved repeated measures with a small sample, the data were analyzed with Friedman tests followed by post-hoc Wilcoxon signed-rank tests.

The results showed consistent improvement with increasing automation. Perceived workload fell across the three methods (Friedman chi-square 16.22, p<.001, W=.901), with all pairwise differences reaching statistical significance. Mental demand dropped from a mean of 6.00 on a seven-point scale for manual labeling to 2.33 for SAM2-assisted labeling and 1.33 for auto-labeling. Usability scores followed the same pattern: manual labeling was rated “Not Acceptable” on the adjective scale with a mean of 46.7, while both assisted methods were rated “Excellent,” at 86.7 and 91.7. Time spent per image fell from 37.5 seconds under manual labeling to 15.4 seconds with SAM2 assistance and 5.1 seconds with auto-labeling, and interaction counts fell from 60.6 to 16.0 to 4.4, with every participant showing a consistent decrease. In the auto-labeling condition, no participant found it necessary to add objects manually, and the total number of detected objects remained comparable across methods, suggesting that increased automation did not compromise detection completeness.

Because effort and user experience do not establish that the resulting annotations are correct, the team also evaluated the transferred YOLOv8 detector on a held-out test set of 100 images excluded from training, using human-reviewed ground truth that the detector did not contribute to. The detector achieved 99.12 percent precision and 94.03 percent recall overall, with mAP@0.50 of 97.34 percent and mAP@0.50:0.95 of 84.66 percent. The team also examined how the auto-approval threshold affects automation: applying thresholds from 0.50 to 0.95 to the confidence scores logged for the 30 detections in the auto-labeling condition, the system would have auto-approved 29 of 30 detections at the default threshold of 0.80, with the rate falling to 83.3 percent at 0.90 and 23.3 percent at 0.95. Participants accepted all 30 detections, so labeling quality was 100 percent at every threshold, and in this task a higher threshold would have increased the review workload without improving quality. In semi-structured interviews, participants attributed lower cognitive effort to the reduced need for class judgment, with one noting that with the assisted method they no longer had to figure out what to label from scratch, while others raised concerns about precise interaction and about reliability in high-stakes contexts without oversight.

The authors are explicit about the limitations. The participant pool was small, and a broader study is planned. The evaluation is limited to industrial control panels, and applying the pipeline to other domains that involve visual-semantic ambiguity remains to be validated. The work does not include a direct component-level comparison between the MLLM-assisted cold-start annotations and end-to-end methods such as Grounded SAM-2; the evaluation instead reports downstream detection performance after supervised label transfer. Extending the evaluation to other industrial domains, benchmarking the cold-start stage directly, and integrating active learning are left for future work, and adding image-preprocessing modules could further improve robustness. HILIPS is released as open-source software under an MIT license, with the code repository, documentation and release packages publicly available, so that research teams without machine learning staff or dedicated annotation workers can use and extend the full path from cold-start annotation to a deployable detector.

Subject of Research: A human-in-the-loop image labeling pipeline for industrial images with visual-semantic ambiguity

Article Title: HILIPS: A human-in-the-loop image labeling pipeline for visual-semantic ambiguity

Article References: Noh, T., Son, B., Kim, J., Yeo, H., & Han, K. (2026). HILIPS: A human-in-the-loop image labeling pipeline for visual-semantic ambiguity. SoftwareX, 36, Article 103173. https://doi.org/10.1016/j.softx.2026.103173

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103173

Keywords: HILIPS, image labeling, human-in-the-loop, multimodal large language model, SAM2, YOLOv8, object detection, industrial automation, annotation tools, visual-semantic ambiguity, user study, open-source software