CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?

Xiangsen Chen, Xuan Feng, Shuo Chen, Matthieu Maitre, Sudipto Rakshit, Diana Duvieilh, Ashley Picone, Nan Tang · Mar 10, 2026 · Citations: 0

Automatic Metrics

Open arXiv Find Implementation RSS feed Shortlist (0)

Data freshness

Extraction: Fresh

Check recency before relying on this page for active eval decisions. Use stale pages as context and verify against current hub results.

Metadata refreshed

Mar 10, 2026, 10:04 AM

Recent

Extraction refreshed

Mar 13, 2026, 10:00 PM

Fresh

Extraction source

Persisted extraction

Confidence 0.45

Abstract

Analyzing Open Source Intelligence (OSINT) from large volumes of data is critical for drafting and publishing comprehensive CTI reports. This process usually follows a three-stage workflow -- triage, deep search and TI drafting. While Large Language Models (LLMs) offer a promising route toward automation, existing benchmarks still have limitations. These benchmarks often consist of tasks that do not reflect real-world analyst workflows. For example, human analysts rarely receive tasks in the form of multiple-choice questions. Also, existing benchmarks often rely on model-centric metrics that emphasize lexical overlap rather than actionable, detailed insights essential for security analysts. Moreover, they typically fail to cover the complete three-stage workflow. To address these issues, we introduce CyberThreat-Eval, which is collected from the daily CTI workflow of a world-leading company. This expert-annotated benchmark assesses LLMs on practical tasks across all three stages as mentioned above. It utilizes analyst-centric metrics that measure factual accuracy, content quality, and operational costs. Our evaluation using this benchmark reveals important insights into the limitations of current LLMs. For example, LLMs often lack the nuanced expertise required to handle complex details and struggle to distinguish between correct and incorrect information. To address these challenges, the CTI workflow incorporates both external ground-truth databases and human expert knowledge. TRA allows human experts to iteratively provide feedback for continuous improvement. The code is available at \href{https://github.com/xschen-beb/CyberThreat-Eval}{\texttt{GitHub}} and \href{https://huggingface.co/datasets/xse/CyberThreat-Eval}{\texttt{HuggingFace}}.

Low-signal caution for protocol decisions

Use this page for context, then validate protocol choices against stronger HFEPX references before implementation decisions.

Extraction flags indicate low-signal or possible false-positive protocol mapping.
Extraction confidence is 0.45 (below strong-reference threshold).

Human Eval Hub LLM-as-Judge Hub Pairwise Preference Hub

HFEPX Relevance Assessment

This paper is adjacent to HFEPX scope and is best used for background context, not as a primary protocol reference.

Best use

Background context only

Use if you need

A benchmark-and-metrics comparison anchor.

Main weakness

Extraction flags indicate low-signal or possible false-positive protocol mapping.

Trust level

Low

Eval-Fit Score

5/100 • Low

Treat as adjacent context, not a core eval-method reference.

Human Feedback Signal

Not explicit in abstract metadata

Evaluation Signal

Detected

HFEPX Fit

Adjacent candidate

Extraction confidence: Low

If you are doing eval pipeline work, start here:

Human Eval Hub LLM-as-Judge Hub Pairwise Preference Hub Tool-Use Eval Hub

Field Provenance & Confidence

Each key protocol field shows extraction state, confidence band, and data source so you can decide whether to trust it directly or validate from full text.

Human Feedback Types

missing

None explicit

Confidence: Low Source: Persisted extraction missing

No explicit feedback protocol extracted.

Evidence snippet: Analyzing Open Source Intelligence (OSINT) from large volumes of data is critical for drafting and publishing comprehensive CTI reports.

Evaluation Modes

partial

Automatic Metrics

Confidence: Low Source: Persisted extraction evidenced

Includes extracted eval setup.

Evidence snippet: Analyzing Open Source Intelligence (OSINT) from large volumes of data is critical for drafting and publishing comprehensive CTI reports.

Quality Controls

missing

Not reported

Confidence: Low Source: Persisted extraction missing

No explicit QC controls found.

Evidence snippet: Analyzing Open Source Intelligence (OSINT) from large volumes of data is critical for drafting and publishing comprehensive CTI reports.

Benchmarks / Datasets

partial

Cyberthreat Eval

Confidence: Low Source: Persisted extraction evidenced

Useful for quick benchmark comparison.

Evidence snippet: To address these issues, we introduce CyberThreat-Eval, which is collected from the daily CTI workflow of a world-leading company.

Reported Metrics

partial

Accuracy

Confidence: Low Source: Persisted extraction evidenced

Useful for evaluation criteria comparison.

Evidence snippet: It utilizes analyst-centric metrics that measure factual accuracy, content quality, and operational costs.

Rater Population

partial

Domain Experts

Confidence: Low Source: Persisted extraction evidenced

Helpful for staffing comparability.

Evidence snippet: This expert-annotated benchmark assesses LLMs on practical tasks across all three stages as mentioned above.

Human Data Lens

Uses human feedback: No
Feedback types: None
Rater population: Domain Experts
Unit of annotation: Unknown
Expertise required: Coding
Extraction source: Persisted extraction

Evaluation Lens

Evaluation modes: Automatic Metrics
Agentic eval: None
Quality controls: Not reported
Confidence: 0.45
Flags: low_signal, possible_false_positive

Protocol And Measurement Signals

Benchmarks / Datasets

Cyberthreat-Eval

Reported Metrics

accuracy

Research Brief

Deterministic synthesis

While Large Language Models (LLMs) offer a promising route toward automation, existing benchmarks still have limitations. HFEPX signals include Automatic Metrics with confidence 0.45. Updated from current HFEPX corpus.

Generated Mar 13, 2026, 10:00 PM · Grounded in abstract + metadata only

Key Takeaways

While Large Language Models (LLMs) offer a promising route toward automation, existing benchmarks still have limitations.
These benchmarks often consist of tasks that do not reflect real-world analyst workflows.

Researcher Actions

Treat this as method context, then pivot to protocol-specific HFEPX hubs.
Cross-check benchmark overlap: Cyberthreat-Eval.
Validate metric comparability (accuracy).

Caveats

Generated from title, abstract, and extracted metadata only; full-paper implementation details are not parsed.
Low-signal flag detected: protocol relevance may be indirect.

Recommended Queries

human-eval protocol design pairwise preference data quality inter-rater agreement adjudication

Research Summary

Contribution Summary

While Large Language Models (LLMs) offer a promising route toward automation, existing benchmarks still have limitations.
These benchmarks often consist of tasks that do not reflect real-world analyst workflows.
To address these issues, we introduce CyberThreat-Eval, which is collected from the daily CTI workflow of a world-leading company.

Why It Matters For Eval

While Large Language Models (LLMs) offer a promising route toward automation, existing benchmarks still have limitations.
These benchmarks often consist of tasks that do not reflect real-world analyst workflows.

Researcher Checklist

Gap: Human feedback protocol is explicit

No explicit human feedback protocol detected.
Pass: Evaluation mode is explicit

Detected: Automatic Metrics
Gap: Quality control reporting appears

No calibration/adjudication/IAA control explicitly detected.
Pass: Benchmark or dataset anchors are present

Detected: Cyberthreat-Eval
Pass: Metric reporting is present

Detected: accuracy

Category-Adjacent Papers (Broader Context)

These papers are nearby in arXiv category and useful for broader context, but not necessarily protocol-matched to this paper.

IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs
Category Neighbor Protocol Overlap

Citations: 0 Relevance: 3.65
- Shared arXiv category (cs.CR, cs.CL)
- Shared terminology (research)
Emulating Clinician Cognition via Self-Evolving Deep Clinical Research
Benchmark-aligned Protocol Overlap

Citations: 0 Relevance: 3.30
- Shared arXiv category (cs.CL)
- Shared metric mentions
Human-AI Co-reasoning for Clinical Diagnosis with Evidence-Integrated Language Agent
Benchmark-aligned Protocol Overlap

Citations: 0 Relevance: 3.30
- Shared arXiv category (cs.CL)
- Shared metric mentions
Adaptive Activation Cancellation for Hallucination Mitigation in Large Language Models
Benchmark-aligned Protocol Overlap

Citations: 0 Relevance: 2.85
- Shared arXiv category (cs.CL)
- Shared metric mentions
Dynamic Knowledge Fusion for Multi-Domain Dialogue State Tracking
Benchmark-aligned Protocol Overlap

Citations: 0 Relevance: 2.85
- Shared arXiv category (cs.CL)
- Shared metric mentions
HeartAgent: An Autonomous Agent System for Explainable Differential Diagnosis in Cardiology
Benchmark-aligned Protocol Overlap

Citations: 0 Relevance: 2.85
- Shared arXiv category (cs.CL)
- Shared metric mentions
Large language models can disambiguate opioid slang on social media
Benchmark-aligned Protocol Overlap

Citations: 0 Relevance: 2.85
- Shared arXiv category (cs.CL)
- Shared metric mentions
PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses
Benchmark-aligned Protocol Overlap

Citations: 0 Relevance: 2.85
- Shared arXiv category (cs.CL)
- Shared metric mentions

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote