Vision as Unified Multimodal Generation
Abstract
Domain fit: AI-core · Core AI workload signals detected from paper context and implementation/artifact evidence.
We formulate computer vision as unified multimodal generation, where heterogeneous visual tasks are expressed in the native text and image generation spaces of a unified multimodal model, without task-specific architectures. Under this formulation, SenseNova-Vision uses natural-language instructions and optional visual prompts to specify tasks, target regions or views, and decoding conventions, and generates responses as text for symbolic outputs, images for dense spatial predictions, or mixed text-and-image outputs for compositional tasks. To support large-scale training, we convert diverse computer vision annotations into instruction-response examples compatible with these generation spaces, resulting in the SenseNova-Vision Corpus, a computer-vision instruction-response corpus spanning text, image, and mixed targets. Starting from an off-the-shelf pretrained unified multimodal model, SenseNova-Vision is trained primarily on this corpus, with auxiliary multimodal data used as a capability-preserving mixture, and requires no task-specific prediction heads or architectural modifications. The resulting model covers a broad range of vision tasks, including detection, OCR, keypoint estimation, segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation, while supporting language-defined variants that combine category, color, region, and other visual cues. Experiments show that a single unified model can match leading task-specialized systems across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. These results suggest unified multimodal generation as a scalable route for integrating computer vision capabilities into general-purpose foundation models. The model and corpus are publicly available.
Results and benchmarks
We formulate computer vision as unified multimodal generation, where heterogeneous visual tasks are expressed in the native text and image generation spaces of a unified multimodal model, without task-specific architectures.
| Task | Dataset | Metric | Value | Source |
|---|---|---|---|---|
| Detection | COCO | Sem. | 70.4 | paper-derived |
Audit each benchmark finding before selecting an implementation path. Evidence refs map to the disclosure below.
Evidence graph: 4 refs, 4 links.
Utility signals: depth 90/100, grounding 95/100, status high.
Implementation
Best maintained implementation now
Vision as Unified Multimodal Generation
647 stars · 36 forks · Last push Jul 22, 2026 · Apache-2.0 license
- License
- CI
- Dependencies
- Docker
Matched via arXiv identifier search · Strong overlap with paper title keywords · Community adoption signal (647 stars)
OpenSenseNova/SenseNova-Vision is the best available implementation candidate based on ranking signals, but recommendation confidence is not yet high. License is declared (Apache-2.0). Dependency/environment manifests are present.
Open OpenSenseNova/SenseNova-Vision- No CI workflows detected
- Selected OpenSenseNova/SenseNova-Vision as the strongest maintained implementation for new work.
- Includes dependency/environment manifest signals.
- Repository activity is within the last 24 months.
Compare implementation paths
Compare maintenance quality, reproducibility coverage, and evidence confidence before choosing a reproduction baseline.
- Maintenance
- Recently updated
- Confidence
- Medium
- Reproducibility
- Moderate
- Stars
- 647
- Last push
- Jul 22, 2026 (35d)
Matched via arXiv identifier search · Strong overlap with paper title keywords
- No CI pipeline detected
- No tagged releases
- No Docker setup
- Maintenance
- Recently updated
- Confidence
- Low
- Reproducibility
- Moderate
- Stars
- 1
- Last push
- Jul 19, 2026 (38d)
Matched via arXiv identifier search · Partial overlap with paper title keywords
- No CI pipeline detected
- No tagged releases
- No Docker setup
- Maintenance
- Active
- Confidence
- Low
- Reproducibility
- Moderate
- Stars
- 0
- Last push
- Aug 17, 2026 (9d)
Matched via arXiv identifier search · Partial overlap with paper title keywords
- No tagged releases
- No Docker setup
- Dependency manifest missing
Reproduction readiness
Setup required
Dependencies pinned, manual setup needed
- OpenSenseNova/SenseNova-Vision has environment.yml but requires manual environment setup.
- No Dockerfile, so you will set up the environment manually.
- No CI pipeline, so test coverage is unknown.
Quick start
git clone https://github.com/OpenSenseNova/SenseNova-Vision.git
conda env create -f environment.yml && conda activate <env-name> Repositories and ecosystem
No additional verified repositories beyond the primary recommendation.
These repositories had low-confidence matching signals and are hidden by default.
- csyifan/sense_med_vision
Confidence: Low · 1 stars
- Deep-Learning-101/Computer-Vision-Paper
Confidence: Low · 0 stars
- brandondev24/Chat
Confidence: Low · 0 stars
Hugging Face artifacts
No direct paper-linked artifacts were found. Showing strongest curated related artifacts for faster exploration.
Models
- sensenova/SenseNova-Vision-7B-MoT
621 downloads · 107 likes
Broaden model search
Datasets
No trustworthy datasets matches right now.
Search datasets on Hugging FaceSpaces
No trustworthy spaces matches right now.
Search spaces on Hugging FaceResearch context
Tasks
Instruction tuning
Methods
Transformer
Domains
Computer vision, Large Language Models
Open this paper in HFEPX to review benchmark signals, evaluation modes, and human-feedback protocol context.
Open in HFEPXJump to Paper2Code search queries derived from this paper's research context.