Equitable Evaluation via Elicitation

Research Summary

Contribution Summary

• Individuals with similar qualifications and skills may vary in their demeanor, or outward manner: some tend toward self-promotion while others are modest to the point of omitting crucial information.
• Comparing the self-descriptions of equally qualified job-seekers with different self-presentation styles is therefore problematic.
• We build an interactive AI for skill elicitation that provides accurate determination of skills while simultaneously allowing individuals to speak in their own voice.

Why It Matters For Eval

• To obtain sufficient training data, we train an LLM to act as synthetic humans.
• To address systematic model bias we enforce a mathematically rigorous notion of equitability ensuring that the covariance between self-presentation manner and skill evaluation error is small.

Related Papers

• InterviewSim: A Scalable Framework for Interview-Grounded Personality Simulation Embedding Neighbor
• STAR: Similarity-guided Teacher-Assisted Refinement for Super-Tiny Function Calling Models Embedding Neighbor
• Virtual Biopsy for Intracranial Tumors Diagnosis on MRI Embedding Neighbor
• Precise Attribute Intensity Control in Large Language Models via Targeted Representation Editing Embedding Neighbor
• Cold-Start Personalization via Training-Free Priors from Structured World Models Embedding Neighbor
• Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attacks Embedding Neighbor
• Mixture-of-Experts as Soft Clustering: A Dual Jacobian-PCA Spectral Geometry Perspective Embedding Neighbor
• Towards Expectation Detection in Language: A Case Study on Treatment Expectations in Reddit Embedding Neighbor
• SoK: Agentic Skills -- Beyond Tool Use in LLM Agents Embedding Neighbor
• GATES: Self-Distillation under Privileged Context with Consensus Gating Tag Overlap
• Proof-RM: A Scalable and Generalizable Reward Model for Math Proof Tag Overlap
• Esoteric Language Models: Bridging Autoregressive and Masked Diffusion LLMs Tag Overlap