Skip to content
OpenTrain AIFor AI Companies

HFEPX · Eval paper review

Bridging the Language Gap in Scholarly Data II: Enhancing Gender Inference for Chinese Names

Lisette Espín-Noboa, Mingrong She, Ana Maria Jaramillo, Liuhuaying Yang

Published

Oct 16, 2026

Citations

0

Trust level

Low

Usefulness score

0/100 (Low)

Extraction confidence

35% (Low)

Derived from extracted protocol signals and abstract evidence.

Rater population

Not reported

Signals refreshed

Oct 16, 2026

Should you rely on this paper?

This paper is adjacent to HFEPX scope and is best used for background context, not as a primary protocol reference.

Use this as background context only. Do not make protocol decisions from this page alone.

Best use

Background context only

Use if you need

A secondary eval reference to pair with stronger protocol papers.

What to verify

Validate the evaluation procedure and quality controls in the full paper before operational use.

Main weakness

This paper looks adjacent to evaluation work, but not like a strong protocol reference.

Human feedback signal
Not explicit
Not explicit in abstract metadata
Evaluation signal
Detected
Eval setup described
Usefulness for eval research
0/100
Adjacent candidate

Treat as adjacent context, not a core eval-method reference.

Abstract

Perceived gender inference from names is important for inclusive scientometrics research, yet existing methods perform poorly on non-Western scholars. We address this gap for Chinese names, which appear in two forms, Hanzi and Pinyin, its Romanized form (e.g., Zhang Wei, Li Na). In international bibliographic records, names are often available only in Pinyin. However, a single pinyin name can correspond to multiple distinct hanzi characters with different gender associations, so gender inference from Pinyin alone is intrinsically ambiguous. To address this, we first develop two hanzi models, a regression model that produces a continuous gender score from 0 (male) to 1 (female), and a classification model that outputs one of three labels (\em{male}, \em{neutral}, \em{female}). We train them on a sample of hanzi names from Chinese-language physics publications, rated by 11 native Chinese speakers. On this sample, the models achieve an RMSE of $0.09$ and an R$^2$ of $0.85$ (regression), and macro-AUC of $0.97$ and a Cohen's $κ$ of $0.81$ (classification), demonstrating high performance that is robust to class imbalance. For pinyin names, we propose a pipeline that aggregates hanzi predictions via transliteration and frequency weighting, together with an ambiguity score that flags names whose gender cannot be resolved from Pinyin alone. Benchmarked against three off-the-shelf tools (Namsor, GPT, and Claude), our models achieve the highest agreement with human annotations for both gender scores and gender labels, in both scripts. Applied to an independent sample of Chinese-named authors in APS journals, our models assign a gender to every name, whereas existing Pinyin-based methods leave between 20\% and 100\% of names unclassified. These findings demonstrate the importance of hanzi-level information for reliable gender inference of Chinese names. Code and data are publicly available.

What we could verify

These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.

Human Feedback Types

missing

None explicit

No explicit feedback protocol extracted.

"Perceived gender inference from names is important for inclusive scientometrics research, yet existing methods perform poorly on non-Western scholars."

Evaluation Modes

partial

Automatic Metrics

Includes extracted eval setup.

"Perceived gender inference from names is important for inclusive scientometrics research, yet existing methods perform poorly on non-Western scholars."

Quality Controls

missing

Not reported

No explicit QC controls found.

"Perceived gender inference from names is important for inclusive scientometrics research, yet existing methods perform poorly on non-Western scholars."

Benchmarks / Datasets

missing

Not extracted

No benchmark anchors detected.

"Perceived gender inference from names is important for inclusive scientometrics research, yet existing methods perform poorly on non-Western scholars."

Reported Metrics

partial

Agreement, Rmse

Useful for evaluation criteria comparison.

"On this sample, the models achieve an RMSE of $0.09$ and an R$^2$ of $0.85$ (regression), and macro-AUC of $0.97$ and a Cohen's $κ$ of $0.81$ (classification), demonstrating high performance that is robust to class imbalance."

Benchmarks and datasets

No benchmark or dataset names were extracted from the available abstract.

Reported metrics

agreementrmse
Human feedback details
Uses human feedback
No
Feedback types
None
Rater population
Not reported
Unit of annotation
Scalar (inferred)
Expertise required
Coding
Evaluation details
Evaluation modes
Automatic Metrics
Agentic eval
None
Quality controls
Not reported
Evidence quality
Low
Use this page as
Background context only

Research brief

Metadata summary

Perceived gender inference from names is important for inclusive scientometrics research, yet existing methods perform poorly on non-Western scholars.

Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.

Key takeaways

  • Perceived gender inference from names is important for inclusive scientometrics research, yet existing methods perform poorly on non-Western scholars.
  • We address this gap for Chinese names, which appear in two forms, Hanzi and Pinyin, its Romanized form (e.g., Zhang Wei, Li Na).
  • In international bibliographic records, names are often available only in Pinyin.

Researcher actions

  • Compare this paper against nearby papers in the same arXiv category before using it for protocol decisions.
  • Check the full text for explicit evaluation design choices (raters, protocol, and metrics).
  • Use related-paper links to find stronger protocol-specific references.

Caveats

  • Generated from abstract + metadata only; no PDF parsing.
  • Signals below are heuristic and may miss details reported outside the abstract.

Recommended queries

Contribution summary

  • For pinyin names, we propose a pipeline that aggregates hanzi predictions via transliteration and frequency weighting, together with an ambiguity score that flags names whose gender cannot be resolved from Pinyin alone.
  • Benchmarked against three off-the-shelf tools (Namsor, GPT, and Claude), our models achieve the highest agreement with human annotations for both gender scores and gender labels, in both scripts.

Why it matters for eval

  • Benchmarked against three off-the-shelf tools (Namsor, GPT, and Claude), our models achieve the highest agreement with human annotations for both gender scores and gender labels, in both scripts.

Researcher checklist

  • Human feedback protocol is explicit

    No explicit human feedback protocol detected.

  • Evaluation mode is explicit

    Detected: Automatic Metrics

  • Quality control reporting appears

    No calibration/adjudication/IAA control explicitly detected.

  • Benchmark or dataset anchors are present

    No benchmark/dataset anchor extracted from abstract.

  • Metric reporting is present

    Detected: agreement, rmse