DSO National Laboratories · Feb 2024 – May 2025
MAESTRO
Multimodal Video Understanding
with Low Li Ying Amy
Recognised by Singapore's Minister of Defence
overview
MAESTRO (Multimodal Agentic model with Efficient, Synergistic Text-Reasoning Optimisation) is a multimodal video-language framework that reasons over visual information, speech, and non-speech audio together.
It was first developed for hateful-video understanding, where meaning often lives in the interplay between what is shown, what is said, and what is heard, and was then evaluated on general video question-answering benchmarks.
motivation
Platforms such as YouTube receive over 500 hours of video every minute, and TikTok and Bilibili have reached billions of users. Moderation still leans heavily on manual review, which is slow, hard to scale, and exposes moderators to mental distress.
Vision-language models could help, but video is hard for them: meaning frequently emerges only when frames, speech, and sound are considered together. Many models ignore audio, append a transcript as plain text, process audio and video in separate streams, or pay a heavy cost to look at every frame.
problem
Hateful video classification: given a video and its audio, decide whether it meets two criteria. (1) It targets an individual or group based on a characteristic. (2) It discriminates against, blames, or encourages harm, fear, or hatred toward them, or threatens societal peace.
The same framework is also evaluated on general video question answering, to test whether the design carries beyond a single use case.
technical approach
Semantic segmentation
The speech transcript is split into semantic chunks by a modified BERT-based chunker (boundary threshold 0.65). Each chunk is paired with the video frames in its time span, forming an "idea-frame pair", so reasoning follows ideas rather than fixed-length windows.
Unified modality alignment: speech
Voice activity detection finds speech segments, which are transcribed with Whisper and aligned to the timeline with forced phoneme alignment.
Unified modality alignment: non-speech audio
MAESTRO-CLAP, inspired by LAION-CLAP, maps a sound event to a descriptive label (what it is) and a contextual label (what it is doing there, e.g. marking a punchline). It pairs PANN and HTS-AT audio encoders with CLIP, BERT and RoBERTa text encoders in a shared 512-d space, trained contrastively on SocialSound-11k.
Unified modality alignment: visual
LLaVA-OneVision describes each chunk's frames. Descriptions, transcript and sound labels are concatenated into a unified text representation of the video.
Global-local reasoning loop
A control agent (GPT-3.5-Turbo in the paper) first reads a global representation built from keyframes per chunk. Each round it proposes a hypothesis, scores its own confidence from 1 to 3, and either answers or calls a tool on a chosen chunk: LLaVA-OneVision for actions, YOLO for objects and symbols, and videoDeepFace, an extension of DeepFace that tracks identities across frames.
main contributions
- 01
Semantic Segmentation Mechanism
Partitions videos into meaningful temporal chunks, precisely aligning visual, speech-audio, and non-speech audio signals without inefficient frame-by-frame processing.
- 02
Unified Modality Alignment
Recognised the underused role of audio in then-dominant video-language systems, jointly aligning visual, speech, and non-speech audio representations for deeper multimodal reasoning.
- 03
Global-Local Reasoning Loop
Forms a global semantic understanding before iteratively refining local analysis, reducing quadratic fusion costs and improving robustness on long-form videos.
- 04
SocialSound-11k
Curated an 11,028-pair audio-language dataset for non-speech social-sound understanding.
experimental setup
- Hateful video detection: the English YouTube subset of MultiHateClip, as a binary task. Metrics are accuracy, macro-F1, and F1, recall and precision on the offensive class, written F1(O), R(O), P(O).
- MAESTRO is evaluated without dataset-specific training on MultiHateClip, whereas several comparison models were fine-tuned on it. MAESTRO-CLAP itself is trained separately on SocialSound-11k: 8,822 of its 11,028 examples, 20 epochs, about 153.5M trainable parameters.
- General video QA: MSRVTT-QA and MSVD-QA, reported as accuracy.
results
- F1(O) · MultiHateClip
- 93%
- macro-F1 · MultiHateClip
- 95%
- MSRVTT-QA · accuracy
- 80%
- MSVD-QA · accuracy
- 86%
Tables are reproduced from the report. Best results are highlighted.
- On MultiHateClip, MAESTRO scores 0.96 accuracy and 0.95 macro-F1, ahead of GPT-4V (0.81 / 0.79).
- The framework is model-agnostic: swapping the control agent to Qwen2 (7B) raises accuracy from 0.43 (Qwen-VL alone) to 0.54, and ChatGPT 3.5 from 0.70 (text prompt only) to 0.96.
- MAESTRO-CLAP reaches an F1 of 0.84 zero-shot on the SocialSound-11k test set, ahead of the other non-speech audio classifiers compared in the report.
- In the report's experimental trials MAESTRO was 2.6× faster than human moderators.
Hateful video detection · MultiHateClip (English YouTube, binary)
| Model | Acc | M-F1 | F1(O) | R(O) | P(O) |
|---|---|---|---|---|---|
| mBERT | 0.57 | 0.57 | 0.52 | 0.42 | 0.68 |
| GPT-4 | 0.81 | 0.79 | 0.73 | 0.69 | 0.78 |
| Qwen | 0.72 | 0.71 | 0.65 | 0.57 | 0.75 |
| MFCC | 0.54 | 0.50 | 0.36 | 0.33 | 0.40 |
| Wav2Vec | 0.53 | 0.48 | 0.64 | 0.50 | 0.90 |
| ViViT | 0.73 | 0.73 | 0.68 | 0.57 | 0.86 |
| ViT | 0.63 | 0.58 | 0.44 | 0.46 | 0.45 |
| VLM | 0.70 | 0.64 | 0.48 | 0.59 | 0.41 |
| GPT-4V | 0.81 | 0.79 | 0.73 | 0.72 | 0.73 |
| Qwen-VL | 0.62 | 0.61 | 0.56 | 0.46 | 0.72 |
| T1 ⊙ A1 ⊙ V1 (multimodal) | 0.75 | 0.74 | 0.67 | 0.61 | 0.77 |
| MAESTRO (ours) | 0.96 | 0.95 | 0.93 | 0.87 | 1.0 |
T = text, A = audio, V = video, VL = vision-language, M = multimodal. F1/R/P are for the offensive class.
General video QA · accuracy
| Model | Modality | MSRVTT-QA | MSVD-QA |
|---|---|---|---|
| Video LLaMA (7B) | V, A | 0.30 | 0.52 |
| Video-ChatGPT (7B) | V | 0.49 | 0.65 |
| PLLaVA (34B) | V | 0.69 | 0.79 |
| Flash-VStream | V, A | 0.72 | 0.80 |
| Tarsier (34B) | V | 0.66 | 0.80 |
| MAESTRO (ours) | V, A | 0.80 | 0.86 |
Ablations · MultiHateClip
| Model | Acc | M-F1 | F1(O) | R(O) | P(O) |
|---|---|---|---|---|---|
| MAESTRO w/o semantic segmentation | 0.60 | 0.62 | 0.61 | 0.63 | 0.62 |
| MAESTRO w/o global-local reasoning loop | 0.70 | 0.69 | 0.71 | 0.68 | 0.70 |
| Qwen-VL (7B) | 0.43 | 0.46 | 0.40 | 0.39 | 0.41 |
| MAESTRO with Qwen2 (7B) | 0.54 | 0.52 | 0.51 | 0.50 | 0.48 |
| ChatGPT 3.5 with text prompt | 0.70 | 0.68 | 0.69 | 0.67 | 0.70 |
| MAESTRO with ChatGPT 3.5 | 0.96 | 0.95 | 0.93 | 0.87 | 1.0 |
Removing either the semantic segmentation or the reasoning loop hurts; swapping the control agent shows the gain is not tied to one model.
figures
poster & report
Great Minds Think Alike, Great Models Think All At Once: MAESTRO, A Unified Framework for Video-Audio Understanding & Reasoning
recognition
- Regeneron ISEF 2025: 2nd Place Grand Award, Robotics and Intelligent Machines
- Singapore Science and Engineering Fair 2025: Gold Award
- Recognised by Singapore's Minister of Defence
limitations & future directions
limitations noted in the report
- Occasional hallucinations.
- Misclassification when context is incomplete, especially for content that depends on cultural context.
- Tone in speech is not interpreted (sarcasm, for example).
future work
- Segment Anything and Set-of-Marks prompting to feed the agent key visual segments and reduce hallucinations.
- A tone-detection tool, and a Self-RAG mechanism to retrieve missing information.
- Swapping the control agent and transcription model to cover other languages and cultures.