Multi-modal transformer-based model that fuses by Shreya SinghMulti-modal transformer-based model that fuses by Shreya Singh

Multi-modal transformer-based model that fuses

Portfolio project
Shreya Singh
Multi-modal transformer-based model that fuses audio spectrograms, video frame embeddings, and contextual text transcripts to autonomously generate highlight clips from long-form videos. The architecture employs a Combiner module for joint spatio-temporal feature extraction from aligned audio-video chunks, followed by an autoregressive Transformer that models temporal dependencies across partitioned video segments, enabling context-aware detection of engaging moments like speech peaks, visual action shifts, and semantic relevance. Achieves 72.2% F1-score for highlight detection—outperforming unimodal baselines by 7-10%—while reducing clip generation time by 10x and boosting viewer engagement metrics by 25% through precise 15-60 second excerpt selection

0 likes · 0 views · Feb 18, 2026

Multi-modal transformer-based model that fuses audio spectrograms, video frame embeddings, and contextual text transcripts to autonomously generate highlight...

Tags