Skip to content
OpenTrain AIFor AI Companies
← Back to explorer

DialogPII: A multilingual dataset of synthetic dialog transcripts to detect personal information

Roland Roller, Vera Czehmann, Derya Erman, Luke Flanagan, Ibrahim Baroud, Frédéric Blain, Viviana Cotik, Eletta Giusto, Akhil Juneja, Mariana Neves, Maria Słowińska, Christine Hovhannisyan, Aaron Louis Eidt, Lisa Raithel, Sebastian Möller, Maija Poikela · Jun 29, 2026 · Citations: 0

Automatic Metrics
Open arXiv Find Implementation RSS feed Shortlist (0)

How to use this page

Low trust

Use this as background context only. Do not make protocol decisions from this page alone.

Best use

Background context only

What to verify

Validate the exact study setup in the full paper before operational use.

Evidence quality

Low

Derived from extracted protocol signals and abstract evidence.

Abstract

Conversational data collected in domains such as healthcare or social sciences is a valuable resource for research and automated analysis. However, responsible data sharing requires the detection and removal of personally identifiable and sensitive information to protect individual privacy. To support the development and evaluation of automatic de-identification systems, we present DialogPII, a multilingual dataset of synthetic dialogs and speech-derived transcripts for personal information detection. DialogPII covers eight interaction scenarios (emergency calls, medical anamnesis interviews, therapy sessions, insurance communication, customer support, clinical interviews regarding an AI-supported dashboard, police reports, and group therapy discussions), 19 entity types, and 11 languages (English, Arabic, Finnish, French, German, Hindi, Italian, Polish, Portuguese, Spanish, and Turkish). Dialogs were generated semi-automatically using large language models, manually curated for plausibility and diversity, and localized to country- and city-specific contexts. All dialogs were additionally converted to speech via text-to-speech synthesis, transcribed with Whisper, and annotated through automatic projection and manual correction, yielding aligned written and speech-derived resources across all languages. We further release baseline multilingual named entity recognition models and provide technical validation through inter-annotator agreement analysis, translation quality evaluation, annotation projection assessment, and benchmark experiments with transformer-based sequence labeling models.

Abstract-only analysis — low confidence

All signals on this page are inferred from the abstract only and may be inaccurate. Do not use this page as a primary protocol reference.

  • This paper looks adjacent to evaluation work, but not like a strong protocol reference.
  • The available metadata is too thin to trust this as a primary source.
Human Eval Hub LLM-as-Judge Hub Pairwise Preference Hub

Should You Rely On This Paper?

This paper is adjacent to HFEPX scope and is best used for background context, not as a primary protocol reference.

Best use

Background context only

Use if you need

A secondary eval reference to pair with stronger protocol papers.

Main weakness

This paper looks adjacent to evaluation work, but not like a strong protocol reference.

Trust level

Low

Usefulness score

15/100 • Low

Treat as adjacent context, not a core eval-method reference.

Human Feedback Signal

Not explicit in abstract metadata

Evaluation Signal

Detected

Usefulness for eval research

Adjacent candidate

Extraction confidence 45%

If you are doing eval pipeline work, start here:

Human Eval Hub LLM-as-Judge Hub Pairwise Preference Hub Tool-Use Eval Hub

What We Could Verify

These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.

Human Feedback Types

missing

None explicit

No explicit feedback protocol extracted.

"Conversational data collected in domains such as healthcare or social sciences is a valuable resource for research and automated analysis."

Evaluation Modes

partial

Automatic Metrics

Includes extracted eval setup.

"Conversational data collected in domains such as healthcare or social sciences is a valuable resource for research and automated analysis."

Quality Controls

partial

Inter Annotator Agreement Reported

Calibration/adjudication style controls detected.

"Conversational data collected in domains such as healthcare or social sciences is a valuable resource for research and automated analysis."

Benchmarks / Datasets

missing

Not extracted

No benchmark anchors detected.

"Conversational data collected in domains such as healthcare or social sciences is a valuable resource for research and automated analysis."

Reported Metrics

partial

Agreement

Useful for evaluation criteria comparison.

"We further release baseline multilingual named entity recognition models and provide technical validation through inter-annotator agreement analysis, translation quality evaluation, annotation projection assessment, and benchmark experiments with transformer-based sequence labeling models."

Human Feedback Details

  • Uses human feedback: No
  • Feedback types: None
  • Rater population: Not reported
  • Expertise required: Medicine, Multilingual

Evaluation Details

  • Evaluation modes: Automatic Metrics
  • Agentic eval: None
  • Quality controls: Inter Annotator Agreement Reported
  • Evidence quality: Low
  • Use this page as: Background context only

Protocol And Measurement Signals

Benchmarks / Datasets

No benchmark or dataset names were extracted from the available abstract.

Reported Metrics

agreement

Research Brief

Metadata summary

Conversational data collected in domains such as healthcare or social sciences is a valuable resource for research and automated analysis.

Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.

Key Takeaways

  • Conversational data collected in domains such as healthcare or social sciences is a valuable resource for research and automated analysis.
  • However, responsible data sharing requires the detection and removal of personally identifiable and sensitive information to protect individual privacy.
  • To support the development and evaluation of automatic de-identification systems, we present DialogPII, a multilingual dataset of synthetic dialogs and speech-derived transcripts for personal information detection.

Researcher Actions

  • Compare this paper against nearby papers in the same arXiv category before using it for protocol decisions.
  • Check the full text for explicit evaluation design choices (raters, protocol, and metrics).
  • Use related-paper links to find stronger protocol-specific references.

Caveats

  • Generated from abstract + metadata only; no PDF parsing.
  • Signals below are heuristic and may miss details reported outside the abstract.

Research Summary

Contribution Summary

  • To support the development and evaluation of automatic de-identification systems, we present DialogPII, a multilingual dataset of synthetic dialogs and speech-derived transcripts for personal information detection.
  • We further release baseline multilingual named entity recognition models and provide technical validation through inter-annotator agreement analysis, translation quality evaluation, annotation projection assessment, and benchmark…

Why It Matters For Eval

  • To support the development and evaluation of automatic de-identification systems, we present DialogPII, a multilingual dataset of synthetic dialogs and speech-derived transcripts for personal information detection.
  • We further release baseline multilingual named entity recognition models and provide technical validation through inter-annotator agreement analysis, translation quality evaluation, annotation projection assessment, and benchmark…

Researcher Checklist

  • Gap: Human feedback protocol is explicit

    No explicit human feedback protocol detected.

  • Pass: Evaluation mode is explicit

    Detected: Automatic Metrics

  • Pass: Quality control reporting appears

    Detected: Inter Annotator Agreement Reported

  • Gap: Benchmark or dataset anchors are present

    No benchmark/dataset anchor extracted from abstract.

  • Pass: Metric reporting is present

    Detected: agreement

Related Papers

Papers are ranked by protocol overlap, extraction signal alignment, and semantic proximity.