LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representation
Weiquan Huang, Aoqi Wu, Yifan Yang, Xufang Luo, Yuqing Yang +9 more
Abstract
CLIP is a seminal multimodal model that maps images and text into a shared representation space through contrastive learning on billions of image-caption pairs. Inspired by the rapid progress of large language models (LLMs), we investigate how the superior linguistic understanding and broad world knowledge of LLMs can further strengthen CLIP, particularly in handling long and complex captions. We introduce an efficie...
Results & Benchmarks
| Task | Dataset | Metric | Value |
|---|---|---|---|
| Retrieval / indexing | COCO | AP | 12.9 |
| Retrieval / indexing | PASCAL VOC | AP | 11.5 |
Hardware Requirements
- Expect multi-day setup/compute for meaningful reproduction based on current guidance.
Best Implementation
LLM2CLIP significantly improves already state-of-the-art CLIP models.
- Selected microsoft/LLM2CLIP as the strongest maintained implementation for new work.
- Includes CI workflow signals.
- Repository activity is within the last 24 months.
Reproduction Path
- 1
Start with microsoft/LLM2CLIP and validate setup instructions in README.
- 2
Reproduce the baseline result with the provided defaults before modifying hyperparameters.
- 3
Log exact dependency versions and runtime environment for reproducibility.
Additional Implementations
No additional verified repositories beyond the primary recommendation.
Hugging Face Artifacts
No direct paper-linked artifacts were found. Showing strongest curated related artifacts.