LLaMA-CLIP fusion improves multimodal digital-human interaction benchmarks

A digital-human interaction system combining LLaMA and CLIP through token-level cross-attention achieved 90.3% image-text matching accuracy, 87.6% intention-recognition accuracy and 82.5% personalized-recommendation accuracy in a computational evaluation. The study also reported user-satisfaction and dialogue-naturalness ratings of 4.6 out of 5 and 4.5 out of 5, respectively, alongside batch-size-one inference latency of 180 milliseconds on an A100 GPU. The experiments used the MSVD and MSR-VTT video-description datasets and 1,000 manually annotated interaction samples; they did not test the system in a public-facing service or a clinical setting.

The researchers designed the system around three stated weaknesses in existing digital humans: limited integration of semantic information across speech, text and images; insufficient adaptation to individual users during multi-turn interactions; and delays distributed across recognition, reasoning, generation and rendering. Its architecture consists of four connected layers. A multimodal perception layer handles voice, text, images and video through functions such as speech noise reduction, endpoint detection, automatic speech recognition, text normalization and video-frame extraction.

A core processing layer uses CLIP to encode visual material and LLaMA to represent language and dialogue history. A personalized-interaction layer maintains user state and dialogue context, while a rendering and output layer converts text into speech and drives facial expressions, body movements and lip synchronization. At the center of the approach is a cross-attention fusion module that projects CLIP visual features and LLaMA language states into a shared space and aligns them at the token level.

The fusion creates a combined representation containing the current user input, dialogue history and visual context. In the reported comparisons, LLaMA alone and CLIP alone reached image-text matching accuracies of 76.5% and 81.2%, respectively, whereas the cross-attention configuration reached 90.3%. For intention recognition, LLaMA alone scored 74.2%, CLIP alone 70.5% and the fused model 87.6%.

These results compare the complete fusion configuration with the two separate-model baselines on the stated metrics, rather than showing that every aspect of the fused system was faster than either baseline. Personalization is handled through an interest-decay long short-term memory network, or LSTM. Historical behaviors are ordered by timestamp and divided into consecutive intervals.

An influence coefficient combines annotated interest scores with an interval-dependent decay rate, while a square-root transformation limits the effect of unusually large values. Division by the square root of the number of records in an interval prevents intervals containing more events from dominating the calculation. The decay rate can vary between intervals, and the paper states that prolonged non-access increases the inter-interval decay increment.

Unlike a standard LSTM, the modified network introduces the time-dependent interest terms into the input and forget gates while retaining the standard candidate-state, output-gate and state-update equations. After each interaction, the modified LSTM’s hidden state replaces the preceding user-state vector. A two-layer mapper converts that state into a personalized prompt, and the final LLaMA input combines the prompt, dialogue history and current user input.

The amount of personalization is also adjusted through a context-aware threshold: the threshold equals a basic value plus an adjustment coefficient multiplied by the logarithm of one plus the user’s interaction frequency during the previous 24 hours. New users are initialized with the average state of an assigned user cluster. The paper identifies weaker performance in cold-start situations, when little or no interaction history is available, as a limitation of the current system.

The personalization evaluation used 1,000 manually annotated samples divided independently into training, validation and test sets at an 8:1:1 ratio. The collection contained 340 customer-service question-and-answer samples, 330 product-explanation samples and 330 casual-chat or companionship samples, with visual contexts drawn from MSVD and MSR-VTT. Sixty participants were assigned to three background groups: 20 with study or work experience in artificial intelligence or human-computer interaction, 20 frequent users of digital services or conversational systems, and 20 general users without specialist technical backgrounds.

Three annotators from different groups evaluated each sample. They rated recommendation relevance, emotional appropriateness, dialogue naturalness and consistency with historical preferences on five-point scales, and majority voting determined categorical recommendation labels. Each experiment was independently repeated five times, with the mean and 95% confidence interval reported, and paired t-tests were used for comparisons, with p<0.05 treated as statistically significant.

The personalization module was optimized with AdamW, a learning rate of 2e−5 and a batch size of 32. Its objective combined categorical cross-entropy for recommendation labels with mean-squared error between predicted and annotated normalized interest vectors. MSVD contains 1,970 short videos divided into 1,200 training, 100 validation and 670 test videos, while MSR-VTT contains 10,000 multi-domain videos divided into 6,573 training, 497 validation and 2,990 test videos.

The official partitions were retained, with the held-out test portions used for final multimodal and generation-quality evaluation. Component ablations showed that removing CLIP reduced image-text matching accuracy by 13.8 percentage points, while replacing cross-attention with feature concatenation reduced it by 3.6 points. Removing the personalization module lowered user satisfaction by 0.7 point on the five-point scale, and replacing the interest-decay LSTM with a conventional LSTM lowered personalized-recommendation accuracy by 4.2 percentage points.

These comparisons isolate the reported effects of the main components under the study’s evaluation conditions. They do not quantify every possible architectural choice, and the supplied paper does not provide a separate count of users in the subjective evaluation beyond the 60-member annotation panel, so the satisfaction ratings should not be treated as population-level estimates. Generation quality was assessed with BLEU-4, METEOR and CIDEr alongside subjective ratings.

The paper treats BLEU-4 as a supplementary lexical-overlap measure because open-ended digital-human conversations can have several semantically valid answers. METEOR is described as more sensitive to semantic correspondence but more computationally complex, while CIDEr depends on the quality of reference descriptions. The authors also distinguish three timing measurements.

The 2.1-second average dialogue-turn response time covers multimodal encoding, fusion and autoregressive text generation but excludes speech synthesis and rendering. The 2.4-second end-to-end response time includes input preprocessing, model processing, speech synthesis and rendering, compared with 1.8 seconds for the text-driven system. Separate batch-size-one hardware benchmarks measured preprocessing, model inference and real-time post-processing without dialogue interaction overhead, speech synthesis or digital-human rendering.

Latency was 180 milliseconds on an A100 GPU, 260 milliseconds on an RTX 4060 and 420 milliseconds on a Jetson AGX. Reported GPU-memory use was 9.8 GB, 6.4 GB and 3.1 GB, respectively, while the Jetson AGX benchmark recorded average CPU utilization of 73%. In scenario-specific evaluations, the highest emotional-interaction success rate was 86.7% in casual-chat or companionship exchanges, lip-shape driving accuracy reached 94.5% during product explanations, and gesture-interaction success reached 90.1% in that product-explanation category.

The paper identifies limited audio-modality support, weaker cold-start performance and increased latency for long videos and complex scenes as limitations. It also reports that user data were anonymized and user profiles were stored locally, but the evidence does not include a comparative audit of privacy controls, bias or safety performance in deployment. Overall, the study provides computational evidence that token-level fusion of CLIP visual features and LLaMA language states can improve several measured aspects of digital-human interaction while preserving low batch-size-one inference times on three hardware platforms.

The ablation results further associate the matching performance with both the visual encoder and cross-attention design, and the personalization results associate recommendation accuracy with the modified recurrent model. The findings remain bounded by benchmark videos, manually annotated interactions and controlled subjective evaluations. They do not establish human-like understanding, reliable generalization beyond the sampled data or safe performance in unrestricted open-domain conversations.

Subject of Research: Real-time multimodal digital-human interaction using LLaMA-CLIP cross-attention and interest-decay user modeling

Article Title: A real-time multimodal digital human interaction system via cross-attention fusion of LLaMA and CLIP models

Article References: Qiu, Y., & Feng, P. (2026). A real-time multimodal digital human interaction system via cross-attention fusion of LLaMA and CLIP models. Discover Artificial Intelligence, 6(1), Article 1075. https://doi.org/10.1007/s44163-026-02107-0

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02107-0

Keywords: digital humans, multimodal AI, LLaMA, CLIP, cross-attention, personalization, user modeling, real-time inference, vision-language models, real-time, multimodal, digital