Jingguang Li, Yebo Wu, Zuyi Guo, Kailang Ma, Xianjie Dai, Han Zheng · Sep 29, 2026 · Citations: 0
Automatic Metrics Coding
← Back to explorer CoEM: Empowering Long-Context Reasoning with Commit-on-Evidence Memory Can Language Models Learn to Forecast Stock Prices ELF-REG: Scaling Continuous Diffusion Language Models to Reasoning Tasks JEV-as-a-Judge: Accept When Confident, Escalate When Unsure Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection Preference Shapes Relevance: Cross-component Hierarchical Semantic Alignment for Personalized Generative Retrieval WebWorld: The Browser as a World Model for Self-Improving Web Code Learning Where Outcomes Change:Credit-Addressable Reasoning for Multimodal Geometry From Metaheuristics to Exact Methods: A CP-SAT Approach for Multi-Objective Healthcare Workforce Scheduling The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agents Rubric-to-Code Credit Assignment for Reinforcement Learning Plans You Can Check: Verifier-Grounded Learning of an Open-Weight Planner for Executable Video-Editing OenoBench: A Wine-Domain Benchmark for Knowledge-Grounded Evaluation of Large Language Models Stopping and Routing LLM Judge Panels One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows Decomposing Wrong-Consensus Agreement in LLM Self-Consistency AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design MARC v1: An Open-Source Multi-Agent Framework for Clinical AI Reasoning and Coordination Spark-to-Paper: End-to-End Research Paper Generation as a Composable Skill The Sleeping Agent: What Gist-Based Context Compression Loses and Why
Tag: Coding
Coding evaluation papers that call for domain expertise or specialist review (373 papers).
Papers in tag: 373
Need Coding evaluators for your project?
Post a Job →Research Utility Snapshot
Evaluation Modes
- Automatic Metrics (11)
- Llm As Judge (2)
- Human Eval (1)
Human Feedback Types
- Pairwise Preference (8)
- Rubric Rating (3)
- Critique Edit (2)
Required Expertise
- Coding (20)
- Math (4)
- Law (1)
Jiacheng Guo, Suozhi Huang, Shuzhen Li, Yunlong Gao, Zerui Cheng, Jason Ge · Sep 29, 2026 · Citations: 0
Demonstrations Automatic Metrics MathCoding
- The resulting AURA-4B more than doubles the starting direction--magnitude score, from 20.94 to 43.31, and is comparable to frontier language models on this benchmark.
- These results show that post-training can substantially improve financial forecasting performance, together with changes in how the model investigates the market, on this outcome-selected benchmark.
Zeyu Michael Li, William Xingxu Chen, Bingshuo Qian, Jiayin Liu, Xiang Cheng · Sep 24, 2026 · Citations: 0
Automatic Metrics MathCoding
- ELF-REG-L achieves 55.96% pass@1 on GSM8K at 64 network function evaluations (NFE), and 13.39% on MATH-500 and 22.56% on HumanEval at 128 NFE.
- At 16 NFE, ELF-REG-L reaches 41.21% HumanEval pass@10, outperforming recent continuous dLMs of comparable scale.
Yubo Li, Yidi Miao, Ramayya Krishnan, Rema Padman · Sep 22, 2026 · Citations: 0
Llm As JudgeAutomatic Metrics MathCoding
- LLM-as-a-judge scales evaluation, but reasoning judges are slow and costly.
- Against sixteen generative and reward-model judges, with blinded human adjudication, JEV comes within three points of GPT-6 wherever a verdict can be read off the text, at 0.36% of its fee and a 0.15-second median latency, and falls behind…
Girish A. Koushik, Diptesh Kanojia, Helen Treharne · Sep 16, 2026 · Citations: 0
Pairwise Preference Automatic Metrics Coding
- We distinguish these cases in Gemma-3 and Qwen3.5 using sparse autoencoders, role-conditioned probes, causal interventions, and recovery experiments across six harmful content benchmarks, with additional Spanish and Hindi-English code-mixed…
Gaoming Zhang, Angqing Jiang, Jianchun Song, Kena Qi, Dayao Chen, Wei Lin · Aug 31, 2026 · Citations: 0
Pairwise Preference Coding
Jiajun Wu, Jian Yang, Yaxin Du, Wei Zhang, Haowen Wang, Junhang Cheng · Aug 31, 2026 · Citations: 0
Critique Edit Simulation Env LawCoding
- VLM-driven self-improvement of web code has a structural flaw: the model that proposes the repair is the model that judges it, and visual plausibility under that judge is a poor proxy for whether the page actually works.
Jiani Guo, Junjie Wang, Jie Wu, Pengxiang Zhao, Dongdong Zhang, Shaohan Huang · Aug 31, 2026 · Citations: 0
Automatic Metrics Coding
- Across nine geometry benchmarks, CE-GRPO achieves an average accuracy of 76.04, outperforming Qwen3-VL-8B and trajectory-level GRPO by 8.09 and 3.43 points, respectively.
Vipul Patel, Anirudh Deodhar, Dagnachew Birru · Aug 31, 2026 · Citations: 0
Pairwise Preference Automatic Metrics Coding
- Healthcare workforce scheduling is an NP-hard optimization problem requiring simultaneous satisfaction of labor regulations, coverage requirements, employee preferences and cost objectives.
- CP-SAT is evaluated on 18 instances: five synthetic hospital units (10-33 nurses), 10 INRC-II benchmarks (5-80 nurses, up to 8-week horizons) and 3 NRP-23 compatible instances (10-25 nurses) with cross-midnight Night shifts.
Achint Mehta · Aug 28, 2026 · Citations: 0
Rubric Rating Coding
- Modern artificial-intelligence coding agents can be equipped with tools for checking their own work e.g.
- We call this set the agent's verification surface.
Rui Jin, Jikai Chen, Yihan Chen, Hao Zhou, Demin Zhu, Kaichen Yang · Aug 28, 2026 · Citations: 0
Rubric Rating Coding
Haoyu Wang, Cheng Feng, Liuyang Bian, Ruiyang Huang, Lei Wei, Yafei Wen · Aug 26, 2026 · Citations: 0
Pairwise PreferenceRubric Rating Coding
- A second stage, RefineCut-Evo, lets the student score its own repairs with the verifier and a task rubric and trains on high-margin preference pairs, so the final 8B planner runs in a closed verifier loop with no teacher calls at inference.
Nikita Khudov · Aug 20, 2026 · Citations: 0
Pairwise Preference Automatic Metrics Coding
- We introduce OenoBench, a wine-domain knowledge benchmark of 3,266 multiple-choice questions across six pillars (regions, grape varieties, viticulture, winemaking, producers, business) and four difficulty tiers.
- Evaluating sixteen frontier configurations, we find: (i) overall accuracy spans 53%-84%, led by o3 at 83.6%; (ii) reasoning-mode lift concentrates in DeepSeek R1 (+6.8pp) and is absent in Claude Opus and Gemini Pro; (iii) Anthropic shows…
Bin Zhu, Yi Xie, Yanghui Rao · Aug 20, 2026 · Citations: 0
Pairwise Preference Llm As Judge MathCoding
- LLM evaluation pipelines often have many candidate judges: general LLM-as-a-judge prompts, reward models, safety classifiers, confidence variants, and task-specific verifiers.
- The deployment question is not only which judge is best, but which judges should be called, on which examples, and when panel construction should stop.
Zhuochun Li, Youngmin Ko, Ali Keramati, Tuhin Kundu, Liang-Chun Tsai, Nicola Ferri · Aug 20, 2026 · Citations: 0
Automatic Metrics Coding
- Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling.
- In this paper, we introduce Thinkingbox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state.
Lizhuo Zhang, Mengmeng Tang, Chenfeng Long, Xiaoyong Tang, Xiang Luo · Aug 19, 2026 · Citations: 0
Pairwise Preference Automatic Metrics Coding
- The mechanical reference is leak-free: each case's preference and accuracy are estimated from its other runs only.
- A cross-system contrast at comparable aggregate accuracy contrasts near-complete mechanical agreement in the open-weights models against a larger preference-unexplained residual in the frontier family.
Yaxin Luo, Haobin Jiang, Jialv Zou, Xu Huang, Wenhao Yan, Haodong Li · Aug 13, 2026 · Citations: 0
Pairwise Preference Human Eval Coding
- In this paper, we present AutoDesign, a framework that aligns with human design priors, where a meta-harness optimizer guides a code agent to recursively improve harness based on rollout feedback.
- Across seven controlled code-agent-model configurations, integrating the learned DesignHarness consistently improves performance, increasing the average PosterBench Score from 54.99 to 67.39 (+12.4%).
Saisha Shetty, Satvik Tripathi, Austin Lin, Colin Zhao, Theodore Kim, Don Enwerem · Aug 13, 2026 · Citations: 0
Expert Verification MedicineCoding
- We present Multi-Agent Reasoning and Coordination (MARC), an open-source framework that replaces monolithic LLM prompting with deterministic multi-agent orchestration for clinical reasoning.
- MARC coordinates role-specialized agents for extraction, reasoning, answer generation, and evaluation, with explicit context passing and traceable intermediate outputs, enabling stage-wise failure attribution.
Zhuoyang Qian, Biao Wu, Yiran Wang, Chris D Yan, Desan Dai, Liangwei Zheng · Aug 12, 2026 · Citations: 0
Critique Edit Coding
- We present Spark-to-Paper, an end-to-end research paper generation system implemented as thirteen composable skills inside an existing coding assistant, without requiring a separate agent platform or orchestration service.
Nicholas E. Kyrkewood · Aug 12, 2026 · Citations: 0
Automatic Metrics Coding
- Gist-based context compression---summarising older conversation history into compact representations---is a common approach in long-horizon language model agents, yet its effect on different types of memory retrieval is poorly understood.
- The prompt modification recovers +0.314 [0.254, 0.375] judge accuracy on category-2 (temporal) questions in the matched set.