LLMStructBench: Benchmarking Large Language Model Structured Data Extraction

Research Summary

Contribution Summary

• We present LLMStructBench, a novel benchmark for evaluating Large Language Models (LLMs) on extracting structured data and generating valid JavaScript Object Notation (JSON) outputs from natural-language text.
• Our open dataset comprises diverse, manually verified parsing scenarios of varying complexity and enables systematic testing across 22 models and five prompting strategies.
• We further introduce complementary performance metrics that capture both token-level accuracy and document-level validity, facilitating rigorous comparison of model, size, and prompting effects on parsing reliability.

Why It Matters For Eval

• We present LLMStructBench, a novel benchmark for evaluating Large Language Models (LLMs) on extracting structured data and generating valid JavaScript Object Notation (JSON) outputs from natural-language text.
• Our benchmark suite is an step towards future research in the area of LLM applied to parsing or Extract, Transform and Load (ETL) applications.

Related Papers

• When Stereotypes GTG: The Impact of Predictive Text Suggestions on Gender Bias in Human-AI Co-Writing Embedding Neighbor
• OR-Agent: Bridging Evolutionary Search and Structured Research for Automated Algorithm Discovery Embedding Neighbor
• General Exploratory Bonus for Optimistic Exploration in RLHF Embedding Neighbor
• Revela: Dense Retriever Learning via Language Modeling Embedding Neighbor
• Chain of Summaries: Summarization Through Iterative Questioning Embedding Neighbor
• CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference Embedding Neighbor
• DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning Embedding Neighbor
• Is Information Density Uniform when Utterances are Grounded on Perception and Discourse? Embedding Neighbor
• LUMI: Unsupervised Intent Clustering with Multiple Pseudo-Labels Embedding Neighbor
• The Headless Firm: How AI Reshapes Enterprise Boundaries Tag Overlap
• Measuring AI Ability to Complete Long Software Tasks Tag Overlap
• Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers Tag Overlap