Skip to content
OpenTrain AIFor AI Companies
← Back to explorer

Tag: Law

Law evaluation papers that call for domain expertise or specialist review (67 papers).

Papers in tag: 67

Global RSS

Need Law evaluators for your project?

Post a Job →

Research Utility Snapshot

Evaluation Modes

  • Automatic Metrics (11)
  • Llm As Judge (3)
  • Simulation Env (2)

Human Feedback Types

  • Pairwise Preference (5)
  • Demonstrations (4)
  • Expert Verification (3)

Required Expertise

  • Law (20)
  • Coding (4)
  • Math (4)
ContractScrub: A benchmark for final review of legal contracts

Yejin Bang, Kirsty Fielding, Brandan Oliver, Brian Birke, Nabeel Seedat, Andrew M. Bean · Aug 20, 2026 · Citations: 0

Expert Verification Automatic Metrics Law
  • Despite the economic value and potential for automation, no formal evaluations of LLMs performing contract scrubbing have been conducted.
  • We introduce ContractScrub, the first benchmark designed to evaluate contract scrubbing capabilities, comprising contracts hand-crafted by experienced lawyers over diverse error categories such as misuse of defined terms, incorrect…
OpenVisTool: An Open Recipe for Synthesizing Instructive Visual Tool-Use Trajectories

Changhao Xiang, Shilin Zhang, Zheng Ma, Kanzhi Cheng, Ruize Ma, Yi Feng · Aug 9, 2026 · Citations: 0

Demonstrations Law
  • Visual tool use has emerged as a fundamental capability for multimodal agents to actively acquire evidence beyond a fixed image encoding.
  • Using this framework, we construct OpenVisTool-42K, a dataset spanning five visual reasoning domains, together with OpenVisTool-Bench, a benchmark covering the same domains.
Will Scaling Improve Social Simulation with LLMs?

Caleb Ziems, William Held, Su Doga Karaca, David Grusky, Tatsunori Hashimoto, Diyi Yang · Jul 2, 2026 · Citations: 0

Automatic MetricsSimulation Env Law
  • We use scaling laws to study the relationship between LLMs' compute scale, general capability benchmarks, and the fidelity of social simulation in three representative sub-domains: opinion modeling, behavioral simulation, and longitudinal…
  • Longitudinal forecasting and underrepresented opinions scale more slowly, especially when they are less correlated with general knowledge and reasoning benchmarks like MMLU.
EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures

Buğra Alperen Uluırmak, Rifat Kurban · Jun 29, 2026 · Citations: 0

Red Team Llm As Judge Law
  • LLM evaluation and AI safety face a shared measurement problem: benchmark scores, reward-model signals, and reported safety metrics can improve while the latent properties they are meant to represent remain difficult to verify.
  • We introduce EvalSafetyGap as an organizing hypothesis for comparing evaluation-side and alignment-side proxy failures under optimization pressure, using Goodhart's Law together with two constructs we develop here - an Instability…
PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models

Chang Wu, Junfeng Fang, Houcheng Jiang, Kai Tang, Pengyu Cheng, Xiaoxi Jiang · Jun 24, 2026 · Citations: 0

Pairwise PreferenceDemonstrations LawMedicine
  • Safety alignment of large language models (LLMs) typically depends on high-quality supervision data, such as safe demonstrations or preference pairs.
  • To address this, we propose PolicyAlign, a simple yet effective framework for directly aligning LLMs with safety policies.
Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment

Guruprakash J, Krithika L. B · Jun 23, 2026 · Citations: 0

Pairwise Preference Automatic Metrics Law
  • We then extend the discussion to post-2023 developments that changed the picture in practice: instruction tuning, reinforcement learning from human feedback, direct preference optimisation, mixture-of-experts scaling, retrieval augmentation…
  • We also discuss how alignment methods, data provenance and benchmark saturation change what it means to call a model "state of the art".
Self-Preference Is Weak or Absent in Verifiable Instruction-Following Revision: A Four-Model Test Under Genuine Authorship

William Guey, Pierrick Bougault · Jun 18, 2026 · Citations: 0

Pairwise PreferenceCritique Edit Law
  • Across four mid-tier model families and 85 author-versus-fresh comparisons, we find no detectable self-preference: authors reject verified-good fixes to their own drafts at essentially the same rate as fresh models judging the same drafts…
  • The one robust observation is qualitative: when authors do reject a verified-good fix, 97% of their stated reasons are flaw-catching rather than preference, that is, about the character of rejections, not an elevated rate.
ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

Wanghan Xu, Shuo Li, Tianlin Ye, Qinglong Cao, Yixin Chen, Hengjian Gao · May 28, 2026 · Citations: 0

Rubric Rating LawCoding
  • We present ResearchClawBench, a benchmark for evaluating autonomous scientific research across 40 tasks from 10 scientific domains.
  • We evaluate seven autonomous research (auto-research) agents under a unified protocol and seventeen native LLMs through the lightweight ResearchHarness.
Scaling Laws for Agent Harnesses via Effective Feedback Compute

Xuanliang Zhang, Dingzirui Wang, Keyan Xu, Qingfu Zhu, Wanxiang Che · May 28, 2026 · Citations: 0

Automatic Metrics MathLaw
  • Agent harnesses shape language-model performance by controlling tool use, feedback, verification, memory, and repair.
  • Across synthetic, real, held-out, and prospective evaluations, EFC-based coordinates outperform raw-compute baselines and SAS.
Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents

Bin Wu, Guanyun Zou, Bingbing Wang, Huan Zhao, Chuan Shi · May 27, 2026 · Citations: 0

Pairwise Preference Automatic Metrics Law
  • A long-lived LLM agent, such as OpenClaw, earns its value by acting on a user's preferences and constraints across sessions, not just the current request.
  • Yet today's agents keep what a user volunteers but rarely ask for what stays unspoken, leaving a proactivity gap in long-lived LLM agents: an agent cannot act on a preference it never obtained.
Orchard: An Open-Source Agentic Modeling Framework

Baolin Peng, Wenlin Yao, Qianhui Wu, Hao Cheng, Xiao Yu, Rui Yang · May 14, 2026 · Citations: 0

Automatic Metrics LawCoding
  • Agentic modeling aims to transform LLMs into autonomous agents capable of solving complex tasks through planning, reasoning, tool use, and multi-turn interaction with environments.
  • We present Orchard, an open-source framework for scalable agentic modeling.
Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study

Volodymyr Ovcharov · May 14, 2026 · Citations: 0

Demonstrations Automatic Metrics Law
  • We benchmark seven models from five providers on 273 validated court decisions from Ukraine's state registry (EDRSR), measuring tokenizer fertility and zero-shot performance on three tasks.
  • To support reproducibility and address the absence of Ukrainian from legal NLP benchmarks, we release a public dataset of 14,452 court decisions spanning 2008-2026, annotated with seven outcome labels across three temporal epochs that…
RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization

Qiming Bao, Juho Leinonen, Paul Denny, Michael J. Witbrock · May 6, 2026 · Citations: 0

Pairwise Preference Llm As Judge Law
  • We propose RLearner-LLM with Hybrid-DPO: an automated preference pipeline that fuses a DeBERTa-v3 NLI signal with a verifier LLM score, removing human annotation while overcoming the "alignment tax" of single-signal optimization.
  • Our Qwen3-8B RLearner-LLM wins 95% of pairwise comparisons against its own SFT baseline; GPT-4o-mini in turn wins 95% against our concise output -- alongside the 69% win the same judge gives a verbose SFT over our DPO model, this replicates…
Safety and accuracy follow different scaling laws in clinical large language models

Sebastian Wind, Tri-Thien Nguyen, Jeta Sopa, Mahshad Lotfinia, Sebastian Bickelhaup, Michael Uder · May 5, 2026 · Citations: 0

Expert Verification Automatic Metrics LawMedicine
  • We introduce SaFE-Scale, a framework for measuring how clinical LLM safety changes across model scale, evidence quality, retrieval strategy, context exposure, and inference-time compute.
  • To instantiate this framework, we introduce RadSaFE-200, a Radiology Safety-Focused Evaluation benchmark of 200 multiple-choice questions with clinician-defined clean evidence, conflict evidence, and option-level labels for high-risk error,…
Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards

Tianyang Han, Hengyu Shi, Junjie Hu, Xu Yang, Zhiling Wang, Junhao Su · May 5, 2026 · Citations: 0

Rubric Rating Automatic Metrics MathLaw
  • Extensive experiments on code and math benchmarks show that this executor-grounded reasoning reward improves the two-stage planner-executor system over execution-only training, suggesting that reasoning supervision should evaluate not only…
Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond

Meng Chu, Xuan Billy Zhang, Kevin Qinghong Lin, Lingdong Kong, Jize Zhang, Teng Tu · Apr 24, 2026 · Citations: 0

Simulation Env Law
  • Agents that manipulate objects, navigate software, coordinate with others, or design experiments require predictive environment models, yet the term world model carries different meanings across research communities.
  • Using this framework, we synthesize over 400 works and summarize more than 100 representative systems spanning model-based reinforcement learning, video generation, web and GUI agents, multi-agent social simulation, and AI-driven scientific…