Thierry Bossy, Julien Vignoud, Tahseen Rabbani, Juan R. Troncoso Pastoriza, Martin Jaggi
Core AI workload signals detected from paper context and implementation/artifact evidence.
Federated learning (FL) is a popular paradigm for collaborative training which avoids direct data exposure between clients. However, data privacy issues still remain: FL-trained large language models are capable of memorizing and completing phrases and sentences contained in training data when given their prefixes. Thus, it is possible for adversarial and honest- but-curious clients to recover training data of other ...
participants simply through targeted prompting. In this work, we demonstrate that a popular and simple fine-tuning strategy, low-rank adaptation (LoRA), reduces memorization during FL by a factor of up to 10 without significant performance cost. We study this effect by performing fine-tuning tasks in high-risk domains such as medicine, law, and finance. We observe a reduction in memorization for a wide variety of model families, from 1B to 70B parameters. We find that LoRA can reduce memorization in centralized learning as well, and we compare how the memorization patterns differ. Furthermore, we study the effect of hyperparameters and show that LoRA can be combined with other privacy-preserving techniques such as gradient clipping and Gaussian noise, secure aggregation, and Goldfish loss to further improve record-level privacy while maintaining performance.
Audit each benchmark finding before selecting an implementation path. Evidence refs map to the disclosure section below.
| Task | Dataset | Metric | Value | Source | Evidence refs |
|---|---|---|---|---|---|
| LoRA | MMLU | Accuracy | 0.308 | paper-derived | No explicit refs |
| LoRA | Llama 3.2 1B | Accuracy per round. | 0.425 | paper-derived | No explicit refs |
| LoRA | LoRA | Accuracy per round. | 0.422 | paper-derived | No explicit refs |
| LoRA | 0.430 | Accuracy per round. | 0.434 | paper-derived | No explicit refs |
| LoRA | Llama 3.1 70B | BLEU | 0.55 | paper-derived | No explicit refs |
| LoRA | Llama 3.2 3B | BLEU | 0.29 | paper-derived | No explicit refs |
Federated learning (FL) is a popular paradigm for collaborative training which avoids direct data exposure between clients.
Only historical official repository was found (tuneinsight/federated-llms).
Open tuneinsight/federated-llmsHardware Notes
Expect multi-day setup/compute for meaningful reproduction based on current guidance.
Evidence graph: 2 refs, 1 links.
Utility signals: depth 95/100, grounding 68/100, status medium.
Compare maintenance quality, reproducibility coverage, and evidence confidence before choosing a reproduction baseline.
Official implementation from Papers with Code · Repository link is mentioned in the paper metadata
Risk flags
Only a historical official implementation is available.
Use with caution for new projects; verify against current tooling and maintained community alternatives.
Hardware requirements
No dependency manifest — manual reconstruction required
No trustworthy direct or curated related Hugging Face artifacts were found yet.
Continue with targeted Hugging Face searches derived from the paper title and method context:
Models
Tip: start with models, then check datasets/spaces if you need evaluation data or demos.
Direct artifact matches are currently sparse. Use targeted Hugging Face searches to quickly locate candidate models, datasets, and demos.
Tasks
LoRA
Methods
LoRA / Parameter-efficient tuning
Domains
Natural Language Processing
Evaluation & Human Feedback Data
Open this paper in HFEPX to review benchmark signals, evaluation modes, and human-feedback protocol context.
Open in HFEPXExplore Similar Papers
Jump to Paper2Code search queries derived from this paper's research context.
Need human evaluators for your AI research? Scale annotation with expert AI Trainers.