SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
Results and benchmarks
SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension is the primary contribution described in this paper.
| Task | Dataset | Metric | Value | Source |
|---|---|---|---|---|
| Language modeling | Vicuna | Temporal | 29.46 | paper-derived |
Audit each benchmark finding before selecting an implementation path. Evidence refs map to the disclosure below.
Evidence graph: 3 refs, 3 links.
Utility signals: depth 100/100, grounding 85/100, status high.
Implementation
Best maintained implementation now
(CVPR2024)A benchmark for evaluating Multimodal LLMs using multiple-choice questions.
367 stars · 13 forks · Last push Jan 14, 2025 · NOASSERTION license
- License
- CI
- Dependencies
- Docker
Official implementation from Papers with Code · Repository link is mentioned in the paper metadata · Strong overlap with paper title keywords
ailab-cvc/seed-bench is the strongest maintained implementation based on ranking signals. License is declared (NOASSERTION).
Open ailab-cvc/seed-bench- No CI workflows detected
- Dependency manifest is missing
- Selected ailab-cvc/seed-bench as the strongest maintained implementation for new work.
- Repository activity is within the last 24 months.
Compare implementation paths
Compare maintenance quality, reproducibility coverage, and evidence confidence before choosing a reproduction baseline.
- Maintenance
- Stale
- Confidence
- High
- Reproducibility
- Limited
- Stars
- 367
- Last push
- Jan 14, 2025 (590d)
Official implementation from Papers with Code · Repository link is mentioned in the paper metadata
- No push in 12+ months
- No CI pipeline detected
- No tagged releases
- Maintenance
- Stale
- Confidence
- Low
- Reproducibility
- Moderate
- Stars
- 470
- Last push
- May 10, 2024 (839d)
Community adoption signal (470 stars) · Repository appears stale (>24 months since last push)
- No push in 12+ months
- No CI pipeline detected
- No tagged releases
- Maintenance
- Stale
- Confidence
- Low
- Reproducibility
- Limited
- Stars
- 68
- Last push
- Feb 21, 2025 (552d)
Matched via arXiv identifier search · Community adoption signal (68 stars)
- No push in 12+ months
- No CI pipeline detected
- No tagged releases
Reproduction readiness
Major work
No dependency manifest, manual reconstruction required
- ailab-cvc/seed-bench has no requirements.txt, environment.yml, pyproject.toml, or Dockerfile.
- You will need to reverse-engineer dependencies from import statements in the source code.
- Last push was 590 days ago.
Hardware requirements
- Expect multi-day setup/compute for meaningful reproduction based on current guidance.
Repositories and ecosystem
No additional verified repositories beyond the primary recommendation.
These repositories had low-confidence matching signals and are hidden by default.
- kakaobrain/honeybee
Confidence: Low · 470 stars
- facebookresearch/multimodal_rewardbench
Confidence: Low · 68 stars
- VyetGokyra/awaresome_LLM_eval_benchmark
Confidence: Low · 8 stars
- DUN-official/seed-bench-vlm-pipeline-public
Confidence: Medium · 0 stars
Hugging Face artifacts
No direct paper-linked artifacts were found. Showing strongest curated related artifacts for faster exploration.
Models
No trustworthy models matches right now.
Search models on Hugging FaceDatasets
- meituan-longcat/UNO-Bench
1,092 downloads · 23 likes · Updated Dec 4, 2025
Broaden dataset search
Spaces
No trustworthy spaces matches right now.
Search spaces on Hugging FaceResearch context
Tasks
Language modeling
Methods
None detected
Domains
None detected
Open this paper in HFEPX to review benchmark signals, evaluation modes, and human-feedback protocol context.
Open in HFEPXJump to Paper2Code search queries derived from this paper's research context.
Data includes links from Papers with Code ( CC-BY-SA-4.0 ).