HFEPX · Eval paper review

WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics

Chenxu Liu, Yingjie Fu, Wei Yang, Ying Zhang +1 more

Published

Jan 5, 2026

Citations

0

Trust level

High

Usefulness score

67/100 (Medium)

Extraction confidence

75% (High)

Derived from extracted protocol signals and abstract evidence.

Rater population

Not reported

Signals refreshed

Mar 16, 2026

Open arXiv Find implementation Shortlist (0) RSS feed

Tags

Coding Llm As Judge Pairwise Preference

Should you rely on this paper?

This paper has useful evaluation signal, but protocol completeness is partial; pair it with related papers before deciding implementation strategy.

Use this as a practical starting point for protocol research, then validate against the original paper.

Best use

Secondary protocol comparison source

Use if you need

A secondary eval reference to pair with stronger protocol papers.

What to verify

Validate the evaluation procedure and quality controls in the full paper before operational use.

Main weakness

No major weakness surfaced.

Human feedback signal

Detected

From extracted signals

Evaluation signal

Detected

Eval setup described

Usefulness for eval research

67/100

Moderate-confidence candidate

Useful as a secondary reference; validate protocol details against neighboring papers.

If you are doing eval pipeline work, start here

Human Eval Papers LLM-as-Judge Papers Pairwise Preference Papers Tool-Use Evaluation Search

Abstract

Web applications (web apps) have become a key arena for large language models (LLMs) to demonstrate their code generation capabilities and commercial potential. However, building a benchmark for LLM-generated web apps remains challenging due to the need for real-world user requirements, generalizable evaluation metrics without relying on ground-truth implementations or test cases, and interpretable evaluation results. To address these challenges, we introduce WebCoderBench, the first real-world-collected, generalizable, and interpretable benchmark for web app generation. WebCoderBench comprises 1,572 real user requirements, covering diverse modalities and expression styles that reflect realistic user intentions. WebCoderBench provides 24 fine-grained evaluation metrics across 9 perspectives, combining rule-based and LLM-as-a-judge paradigm for fully automated, objective, and general evaluation. Moreover, WebCoderBench adopts human-preference-aligned weights over metrics to yield interpretable overall scores. Experiments across 12 representative LLMs and 2 LLM-based agents show that there exists no dominant model across all evaluation metrics, offering an opportunity for LLM developers to optimize their models in a targeted manner for a more powerful version.

What we could verify

These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.

Human Feedback Types

strong

Pairwise Preference

Directly usable for protocol triage.

"Web applications (web apps) have become a key arena for large language models (LLMs) to demonstrate their code generation capabilities and commercial potential."

Evaluation Modes

strong

Llm As Judge

Includes extracted eval setup.

"Web applications (web apps) have become a key arena for large language models (LLMs) to demonstrate their code generation capabilities and commercial potential."

Quality Controls

missing

Not reported

No explicit QC controls found.

"Web applications (web apps) have become a key arena for large language models (LLMs) to demonstrate their code generation capabilities and commercial potential."

Benchmarks / Datasets

strong

LMSYS Chatbot Arena, APPS, Webcoderbench

Useful for quick benchmark comparison.

"Web applications (web apps) have become a key arena for large language models (LLMs) to demonstrate their code generation capabilities and commercial potential."

Reported Metrics

missing

Not extracted

No metric anchors detected.

"Web applications (web apps) have become a key arena for large language models (LLMs) to demonstrate their code generation capabilities and commercial potential."

Benchmarks and datasets

LMSYS Chatbot ArenaAPPSWebcoderbench

Reported metrics

No metric terms were extracted from the available abstract.

Human feedback details

Uses human feedback: Yes
Feedback types: Pairwise Preference
Rater population: Not reported
Expertise required: Coding

Evaluation details

Evaluation modes: Llm As Judge
Agentic eval: None
Quality controls: Not reported
Evidence quality: High
Use this page as: Secondary protocol comparison source

Research brief

Metadata summary

Web applications (web apps) have become a key arena for large language models (LLMs) to demonstrate their code generation capabilities and commercial potential.

Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.

Key takeaways

Web applications (web apps) have become a key arena for large language models (LLMs) to demonstrate their code generation capabilities and commercial potential.
However, building a benchmark for LLM-generated web apps remains challenging due to the need for real-world user requirements, generalizable evaluation metrics without relying on ground-truth implementations or test cases, and interpretable evaluation results.
To address these challenges, we introduce WebCoderBench, the first real-world-collected, generalizable, and interpretable benchmark for web app generation.

Researcher actions

Compare this paper against nearby papers in the same arXiv category before using it for protocol decisions.
Check the full text for explicit evaluation design choices (raters, protocol, and metrics).
Use related-paper links to find stronger protocol-specific references.

Caveats

Generated from abstract + metadata only; no PDF parsing.
Signals below are heuristic and may miss details reported outside the abstract.

Recommended queries

Pairwise preference evaluation

Contribution summary

However, building a benchmark for LLM-generated web apps remains challenging due to the need for real-world user requirements, generalizable evaluation metrics without relying on ground-truth implementations or test cases, and interpretable…
To address these challenges, we introduce WebCoderBench, the first real-world-collected, generalizable, and interpretable benchmark for web app generation.
WebCoderBench provides 24 fine-grained evaluation metrics across 9 perspectives, combining rule-based and LLM-as-a-judge paradigm for fully automated, objective, and general evaluation.

Why it matters for eval

However, building a benchmark for LLM-generated web apps remains challenging due to the need for real-world user requirements, generalizable evaluation metrics without relying on ground-truth implementations or test cases, and interpretable…
To address these challenges, we introduce WebCoderBench, the first real-world-collected, generalizable, and interpretable benchmark for web app generation.

Researcher checklist

Human feedback protocol is explicit

Detected: Pairwise Preference
Evaluation mode is explicit

Detected: Llm As Judge
Quality control reporting appears

No calibration/adjudication/IAA control explicitly detected.
Benchmark or dataset anchors are present

Detected: LMSYS Chatbot Arena, APPS, Webcoderbench
Metric reporting is present

No metric terms extracted.