Human Label Variation in Implicit Discourse Relation Recognition

Research Summary

Contribution Summary

• There is growing recognition that many NLP tasks lack a single ground truth, as human judgments reflect diverse perspectives.
• To capture this variation, models have been developed to predict full annotation distributions rather than majority labels, while perspectivist models aim to reproduce the interpretations of individual annotators.
• In this work, we compare these approaches on Implicit Discourse Relation Recognition (IDRR), a highly ambiguous task where disagreement often arises from cognitive complexity rather than ideological bias.

Why It Matters For Eval

• There is growing recognition that many NLP tasks lack a single ground truth, as human judgments reflect diverse perspectives.
• Further analysis indicates that frequent cognitively demanding cases drive inconsistency in human interpretation, posing challenges for perspectivist modeling in IDRR.

Related Papers

• GDGB: A Benchmark for Generative Dynamic Text-Attributed Graph Learning Embedding Neighbor
• Resisting Contextual Interference in RAG via Parametric-Knowledge Reinforcement Embedding Neighbor
• GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL Embedding Neighbor
• How Well Can LLM Agents Simulate End-User Security and Privacy Attitudes and Behaviors? Embedding Neighbor
• vCache: Verified Semantic Prompt Caching Embedding Neighbor
• A Survey of Data Agents: Emerging Paradigm or Overstated Hype? Embedding Neighbor
• Revisiting Northrop Frye's Four Myths Theory with Large Language Models Embedding Neighbor
• RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models Embedding Neighbor
• Language and Experience: A Computational Model of Social Learning in Complex Tasks Embedding Neighbor
• PoeTone: A Framework for Constrained Generation of Structured Chinese Songci with LLMs Tag Overlap
• Yor-Sarc: A gold-standard dataset for sarcasm detection in a low-resource African language Tag Overlap
• Measuring the Measurers: Quality Evaluation of Hallucination Benchmarks for Large Vision-Language Models Tag Overlap