SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation

Should You Rely On This Paper?

This paper is adjacent to HFEPX scope and is best used for background context, not as a primary protocol reference.

What We Could Verify

These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.

Protocol And Measurement Signals

Despite strong performance on Text-to-SQL benchmarks, it remains unclear whether LLM-generated SQL programs are structurally reliable. HFEPX signals include Automatic Metrics with confidence 0.45. Updated from current HFEPX corpus.

Generated Apr 13, 2026, 9:54 AM · Grounded in abstract + metadata only

Research Summary

Contribution Summary

• Despite strong performance on Text-to-SQL benchmarks, it remains unclear whether LLM-generated SQL programs are structurally reliable.
• Our experiments on the Spider benchmark show that modern LLMs often produce structurally diverse queries for the same input, even when execution results are correct, and that such variance is frequently triggered by surface-level input…
• We further show that generating queries in a structured space via a compile-style pipeline can improve both execution accuracy and structural consistency.

Why It Matters For Eval

• Despite strong performance on Text-to-SQL benchmarks, it remains unclear whether LLM-generated SQL programs are structurally reliable.
• Our experiments on the Spider benchmark show that modern LLMs often produce structurally diverse queries for the same input, even when execution results are correct, and that such variance is frequently triggered by surface-level input…

Researcher Checklist

• Gap: Human feedback protocol is explicit

  No explicit human feedback protocol detected.

• Pass: Evaluation mode is explicit

  Detected: Automatic Metrics

• Gap: Quality control reporting appears

  No calibration/adjudication/IAA control explicitly detected.

• Pass: Benchmark or dataset anchors are present

  Detected: Spider, Sqlstructeval, Structeval

• Pass: Metric reporting is present

  Detected: accuracy

Category-Adjacent Papers (Broader Context)

These papers are nearby in arXiv category and useful for broader context, but not necessarily protocol-matched to this paper.

• Guaranteeing Knowledge Integration with Joint Decoding for Retrieval-Augmented Generation

  Benchmark-aligned Protocol Overlap
  Citations: 0 Relevance: 3.30
  • Shared arXiv category (cs.CL)
  • Shared metric mentions
• HyperMem: Hypergraph Memory for Long-Term Conversations

  Benchmark-aligned Protocol Overlap
  Citations: 0 Relevance: 3.30
  • Shared arXiv category (cs.CL)
  • Shared metric mentions
• AfriVoices-KE: A Multilingual Speech Dataset for Kenyan Languages

  Benchmark-aligned Protocol Overlap
  Citations: 0 Relevance: 2.85
  • Shared arXiv category (cs.CL)
  • Shared metric mentions
• Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts

  Benchmark-aligned Protocol Overlap
  Citations: 0 Relevance: 2.85
  • Shared arXiv category (cs.CL)
  • Shared metric mentions
• Don't Overthink It: Inter-Rollout Action Agreement as a Free Adaptive-Compute Signal for LLM Agents

  Benchmark-aligned Protocol Overlap
  Citations: 0 Relevance: 2.85
  • Shared arXiv category (cs.CL)
  • Shared metric mentions
• Kathleen: Oscillator-Based Byte-Level Text Classification Without Tokenization or Attention

  Benchmark-aligned Protocol Overlap
  Citations: 0 Relevance: 2.85
  • Shared arXiv category (cs.CL)
  • Shared metric mentions
• KV Cache Offloading for Context-Intensive Tasks

  Benchmark-aligned Protocol Overlap
  Citations: 0 Relevance: 2.85
  • Shared arXiv category (cs.CL)
  • Shared metric mentions
• Quantum Vision Theory Applied to Audio Classification for Deepfake Speech Detection

  Benchmark-aligned Protocol Overlap
  Citations: 0 Relevance: 2.85
  • Shared arXiv category (cs.CL)
  • Shared metric mentions