Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats

Research Summary

Contribution Summary

• As LLMs scale, low-bit floating-point formats like MXFP and NVFP4 offer new opportunities for precision and efficiency.
• In this work, we evaluate HiFloat (HiF8 and HiF4), a family of formats tailored for Ascend NPUs.
• Through rigorous comparison across weight-activation and KV-cache tasks, we provide three key insights: (1) INT8 suits narrow-range data, while floating-point formats excel with high-variance data; (2) in 4-bit regimes, HiF4's hierarchical

Related Papers

• Evaluating Proactive Risk Awareness of Large Language Models Embedding Neighbor
• Revisiting RAG Retrievers: An Information Theoretic Benchmark Embedding Neighbor
• A Survey on the Optimization of Large Language Model-based Agents Embedding Neighbor
• DLT-Corpus: A Large-Scale Text Collection for the Distributed Ledger Technology Domain Embedding Neighbor
• MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping Embedding Neighbor
• Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training Embedding Neighbor
• Measuring Social Integration Through Participation: Categorizing Organizations and Leisure Activities in the Displaced Karelians Interview Archive using LLMs Embedding Neighbor
• Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback Embedding Neighbor
• Disentangling Geometry, Performance, and Training in Language Models Embedding Neighbor
• The Headless Firm: How AI Reshapes Enterprise Boundaries Tag Overlap
• Measuring AI Ability to Complete Long Software Tasks Tag Overlap
• Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers Tag Overlap