VIGiA: Instructional Video Guidance via Dialogue Reasoning and Retrieval

Research Summary

Contribution Summary

• We introduce VIGiA, a novel multimodal dialogue model designed to understand and reason over complex, multi-step instructional video action plans.
• Unlike prior work which focuses mainly on text-only guidance, or treats vision and language in isolation, VIGiA supports grounded, plan-aware dialogue that requires reasoning over visual inputs, instructional plans, and interleaved user int
• To this end, VIGiA incorporates two key capabilities: (1) multimodal plan reasoning, enabling the model to align uni- and multimodal queries with the current task plan and respond accurately; and (2) plan-based retrieval, allowing it to ret

Why It Matters For Eval

• Our evaluation shows that VIGiA outperforms existing state-of-the-art models on all tasks in a conversational plan guidance setting, reaching over 90\% accuracy on plan-aware VQA.

Related Papers

• When Stereotypes GTG: The Impact of Predictive Text Suggestions on Gender Bias in Human-AI Co-Writing Embedding Neighbor
• A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding Embedding Neighbor
• Refusal Direction is Universal Across Safety-Aligned Languages Embedding Neighbor
• From Medical Records to Diagnostic Dialogues: A Clinical-Grounded Approach and Dataset for Psychiatric Comorbidity Embedding Neighbor
• GLM-5: from Vibe Coding to Agentic Engineering Embedding Neighbor
• Context-Aware Mapping of 2D Drawing Annotations to 3D CAD Features Using LLM-Assisted Reasoning for Manufacturing Automation Embedding Neighbor
• Do Large Language Models Grasp The Grammar? Evidence from Grammar-Book-Guided Probing in Luxembourgish Embedding Neighbor
• Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference Embedding Neighbor
• Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-training Embedding Neighbor
• Uncovering Autoregressive LLM Knowledge of Thematic Fit in Event Representation Tag Overlap
• BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learning Tag Overlap
• Provably Safe Generative Sampling with Constricting Barrier Functions Tag Overlap