Human Feedback Types
strongRubric Rating
Directly usable for protocol triage.
"Background: Crisis helplines are essential to mental health systems but are often understaffed and underfunded."
HFEPX · Eval paper review
Akshay Swaminathan, Filip Ramazan, Sharang Phadke, Kevina Wang +14 more
Published
Nov 18, 2026
Citations
0
Trust level
High
Usefulness score
67/100 (Medium)
Extraction confidence
75% (High)
Derived from extracted protocol signals and abstract evidence.
Rater population
Not reported
Signals refreshed
Nov 18, 2026
This paper has useful evaluation signal, but protocol completeness is partial; pair it with related papers before deciding implementation strategy.
Use this as a practical starting point for protocol research, then validate against the original paper.
Use this page for context, then validate protocol choices against stronger HFEPX references before implementation decisions.
Best use
Secondary protocol comparison source
Use if you need
A concrete protocol example with enough signal to inform rater workflow design.
What to verify
Read the full paper before copying any benchmark, metric, or protocol choices.
Main weakness
The abstract does not clearly name benchmarks or metrics.
Useful as a secondary reference; validate protocol details against neighboring papers.
If you are doing eval pipeline work, start here
Background: Crisis helplines are essential to mental health systems but are often understaffed and underfunded. LLMs could support counselors, but existing benchmarks do not capture the escalation, risk assessment, and de-escalation central to crisis conversations, while privacy restrictions limit use of real transcripts. Methods: We present CrisisBench, a privacy-preserving suite for single- and multi-turn crisis counseling evaluation. It includes 1,085 single-turn tasks from 200 de-identified conversations and 1,185 patient LLMs derived from hotline metadata. Patient models and a 23-item LLM judge were calibrated against expert counselors. Results: We evaluated five GPT models plus draft-then-edit and retrieval-augmented generation. All systems underperformed human counselors on excessive empathy and questioning in single-turn evaluation. The rubric achieved human-human Krippendorff's $\alpha$ = 0.51 and LLM-human $\alpha$ = 0.54 (Pearson r = 0.80). In multi-turn evaluation, retrieval-augmented generation scored highest overall (0.961), with all systems within 0.058. Microskills and style/tone were near ceiling, while productivity and safety/risk accounted for most between-system variation. Counselors rated patient LLMs realistic (86%) and useful (92%). Conclusion: CrisisBench enables scalable, privacy-preserving evaluation of crisis counseling. Current systems perform strongly on core structural competencies but still diverge from experts on important response-level failure modes. We release the rubrics, judge prompts, and evaluation tasks.
These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.
Rubric Rating
Directly usable for protocol triage.
"Background: Crisis helplines are essential to mental health systems but are often understaffed and underfunded."
Llm As Judge
Includes extracted eval setup.
"Background: Crisis helplines are essential to mental health systems but are often understaffed and underfunded."
Inter Annotator Agreement Reported
Calibration/adjudication style controls detected.
"Background: Crisis helplines are essential to mental health systems but are often understaffed and underfunded."
Not extracted
No benchmark anchors detected.
"Background: Crisis helplines are essential to mental health systems but are often understaffed and underfunded."
Not extracted
No metric anchors detected.
"Background: Crisis helplines are essential to mental health systems but are often understaffed and underfunded."
No benchmark or dataset names were extracted from the available abstract.
No metric terms were extracted from the available abstract.
Background: Crisis helplines are essential to mental health systems but are often understaffed and underfunded.
Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.
Human feedback protocol is explicit
Detected: Rubric Rating
Evaluation mode is explicit
Detected: Llm As Judge
Quality control reporting appears
Detected: Inter Annotator Agreement Reported
Benchmark or dataset anchors are present
No benchmark/dataset anchor extracted from abstract.
Metric reporting is present
No metric terms extracted.