Semantic Video RAG
Ask questions about a video on an ordinary laptop, and get answers drawn only from what the video shows.
- Year
- 2025
- Built with
- Python, CLIP, k-means, BLIP, FAISS, LangChain, Flan-T5, Streamlit
- Try it
- Live demo
The problem
Of a clip’s frames dropped before anything is captioned: 10 kept out of 62, with the story intact.
Video question answering usually needs large GPUs and end-to-end training. Most frames of a clip repeat each other, like the 1878 gallop above, so SV-RAG works out which frames are different and spends its effort only on those. It runs on Apple Metal or a plain CPU, with no NVIDIA GPU and no training.
How it works
- SampleThe video is read at one frame per second.
- ClusterEach frame is embedded with CLIP (ViT-B/32) and k-means picks the frames nearest each cluster centre: the distinct scenes.
- CaptionBLIP-Large describes each kept frame, with a confidence score.
- IndexCaptions are embedded with MiniLM and stored in FAISS, with their timestamps.
- AnswerA question retrieves the right moments, and Flan-T5-Large answers under a strict prompt that allows only visual evidence.
- ConnectA spatio-temporal graph shows how objects and ideas link across the video.
Scrub the gallop
Drag through the 31 frames of the 1878 clip, and set how many to keep, the way SV-RAG’s slider does. Every skipped frame has a kept one that already shows its pose.
Frame 1 of 31.
Grouped here by k-means on the frames’ pixels, a stand-in for the CLIP embeddings SV-RAG uses. The kept frame is the one nearest the centre of its group.
Results
| Measure | Every second | SV-RAG | Change |
|---|---|---|---|
| Frames processed | 62 | 10 | 83.8% fewer |
| Processing time | about 45 s | about 8 s | about 5× faster |
| What is kept | many near-duplicates | one frame per distinct scene |
In the app, a slider sets the number of clusters, so you choose how much detail to keep before asking questions such as “What colour is the car?” or “Describe the sequence of events.”