Human Feedback Types
missingNone explicit
No explicit feedback protocol extracted.
"Before an AI agent takes on a paid job, someone has to judge whether it is likely to succeed."
HFEPX · Eval paper review
PAYWORTHY: Can LLMs Predict Which Freelance Jobs an AI Agent Will Complete?
Mohammad Reza Rezaei, Dingxian Wang, Brett Levert, Celeste Fejzo +2 more
Published
Nov 3, 2026
Citations
0
Trust level
Low
Usefulness score
49/100 (Medium)
Extraction confidence
45% (Low)
Derived from extracted protocol signals and abstract evidence.
Rater population
Not reported
Signals refreshed
Nov 3, 2026
Should you rely on this paper?
This paper has useful evaluation signal, but protocol completeness is partial; pair it with related papers before deciding implementation strategy.
Use this as background context only. Do not make protocol decisions from this page alone.
Low-signal caution for protocol decisions
Use this page for context, then validate protocol choices against stronger HFEPX references before implementation decisions.
- The available metadata is too thin to trust this as a primary source.
Best use
Secondary protocol comparison source
Use if you need
A secondary eval reference to pair with stronger protocol papers.
What to verify
Validate the evaluation procedure and quality controls in the full paper before operational use.
Main weakness
The available metadata is too thin to trust this as a primary source.
Human feedback signal
Not explicit
Not explicit in abstract metadata
Evaluation signal
Detected
Eval setup described
Usefulness for eval research
49/100
Moderate-confidence candidate
Useful as a secondary reference; validate protocol details against neighboring papers.
If you are doing eval pipeline work, start here
Abstract
Before an AI agent takes on a paid job, someone has to judge whether it is likely to succeed. We test whether frontier language models can make this call before any attempt. We propose PAYWORTHY which contains 1{,}494 real freelance job postings, sampled from 18.8M postings made between 2020 and 2026 on a freelancing platform. An agent attempts each job from the posting alone in a single run, and an LLM judge acting as the client decides whether to accept the work. We ask 18 models to forecast how likely the agent is to succeed and measure how well each model's forecasts rank the accepted jobs above the rejected ones (AUC, where 0.5 is a random ranking). The answer depends on how the agent is described. Told what the agent cannot do, every model ranks the jobs well (AUC 0.66--0.86). Told instead that it is a generic agent with a browser and code execution, the same models rank the same jobs about as well as a random ordering (0.45--0.56). These results hold under judges from four other model families (Claude, GLM, Kimi and Qwen), and the informed forecasts also rank the verdicts of two human raters. Code is available at https://github.com/upwork/Payworthy
What we could verify
These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.
Evaluation Modes
partialHuman Eval, Llm As Judge, Automatic Metrics
Includes extracted eval setup.
"Before an AI agent takes on a paid job, someone has to judge whether it is likely to succeed."
Quality Controls
missingNot reported
No explicit QC controls found.
"Before an AI agent takes on a paid job, someone has to judge whether it is likely to succeed."
Benchmarks / Datasets
missingNot extracted
No benchmark anchors detected.
"Before an AI agent takes on a paid job, someone has to judge whether it is likely to succeed."
Reported Metrics
partialAuroc
Useful for evaluation criteria comparison.
"Before an AI agent takes on a paid job, someone has to judge whether it is likely to succeed."
Benchmarks and datasets
No benchmark or dataset names were extracted from the available abstract.
Reported metrics
auroc
Human feedback details
- Uses human feedback
- No
- Feedback types
- None
- Rater population
- Not reported
- Expertise required
- Coding
Evaluation details
- Evaluation modes
- Human Eval, Llm As Judge, Automatic Metrics
- Agentic eval
- None
- Quality controls
- Not reported
- Evidence quality
- Low
- Use this page as
- Secondary protocol comparison source
Research brief
Metadata summary
Before an AI agent takes on a paid job, someone has to judge whether it is likely to succeed.
Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.
Key takeaways
- Before an AI agent takes on a paid job, someone has to judge whether it is likely to succeed.
- We test whether frontier language models can make this call before any attempt.
- We propose PAYWORTHY which contains 1{,}494 real freelance job postings, sampled from 18.8M postings made between 2020 and 2026 on a freelancing platform.
Researcher actions
- Compare this paper against nearby papers in the same arXiv category before using it for protocol decisions.
- Check the full text for explicit evaluation design choices (raters, protocol, and metrics).
- Use related-paper links to find stronger protocol-specific references.
Caveats
- Generated from abstract + metadata only; no PDF parsing.
- Signals below are heuristic and may miss details reported outside the abstract.
Recommended queries
Contribution summary
- Before an AI agent takes on a paid job, someone has to judge whether it is likely to succeed.
- We test whether frontier language models can make this call before any attempt.
Why it matters for eval
- An agent attempts each job from the posting alone in a single run, and an LLM judge acting as the client decides whether to accept the work.
Researcher checklist
-
Human feedback protocol is explicit
No explicit human feedback protocol detected.
-
Evaluation mode is explicit
Detected: Human Eval, Llm As Judge, Automatic Metrics
-
Quality control reporting appears
No calibration/adjudication/IAA control explicitly detected.
-
Benchmark or dataset anchors are present
No benchmark/dataset anchor extracted from abstract.
-
Metric reporting is present
Detected: auroc