Ismail Khalfaoui-Hassani, Stefan Kesselheim
Which functions can be used as activations in deep neural networks? This article explores families of functions based on orthonormal bases, including the Hermite polynomial basis and the Fourier trigonometric basis, as well as a basis resulting from the tropicalization of a polynomial basis. Our study shows that, through simple variance-preserving initialization and without additional clamping mechanisms, these activ ...
ations can successfully be used to train deep models, such as GPT-2 for next-token prediction on OpenWebText and ConvNeXt for image classification on ImageNet. Our work addresses the issue of exploding and vanishing activations and gradients, particularly prevalent with polynomial activations, and opens the door for improving the efficiency of large-scale learning tasks. Furthermore, our approach provides insight into the structure of neural networks, revealing that networks with polynomial activations can be interpreted as multivariate polynomial mappings. Finally, using Hermite interpolation, we show that our activations can closely approximate classical ones in pre-trained models by matching both the function and its derivative, making them especially useful for fine-tuning tasks. These activations are available in the torchortho library via: https://github.com/K-H-Ismail/torchortho.
Audit each benchmark finding before selecting an implementation path. Evidence refs map to the disclosure section below.
| Task | Dataset | Metric | Value | Source | Evidence refs |
|---|---|---|---|---|---|
| Image classification | ImageNet-1K | top-1 accuracy | improves over GELU baseline | llm-grounded | No explicit refs |
Which functions can be used as activations in deep neural networks?
K-H-Ismail/torchortho is the strongest maintained implementation based on ranking signals. CI workflows are present. License is declared (GPL-3.0).
Open K-H-Ismail/torchorthoLLM evidence refs: paper.abstract, evidencePack.repoSources[repoUrl=https://github.com/K-H-Ismail/torchortho].sections[id=readme_3], evidencePack.repoSources[repoUrl=https://github.com/K-H-Ismail/torchortho].sections[id=readme_13], guidance.riskFlags[0], researcherSummary.reproductionRisks[0], repos[0].fullName, paper.title, summary.hasReliableImplementation
Evidence graph: 3 refs, 3 links.
Utility signals: depth 35/100, grounding 75/100, status low.
Compare maintenance quality, reproducibility coverage, and evidence confidence before choosing a reproduction baseline.
Official implementation from Papers with Code · Repository link is mentioned in the paper metadata
Risk flags
[ICLR 2026] Polynomial, trigonometric, and tropical activations
Ready to reproduce
Quick start
git clone https://github.com/K-H-Ismail/torchortho.git
pip install -e .No trustworthy direct or curated related Hugging Face artifacts were found yet.
Continue with targeted Hugging Face searches derived from the paper title and method context:
Tip: start with models, then check datasets/spaces if you need evaluation data or demos.
Direct artifact matches are currently sparse. Use targeted Hugging Face searches to quickly locate candidate models, datasets, and demos.
Tasks
Image classification
Methods
None detected
Domains
Computer vision
Evaluation & Human Feedback Data
Open this paper in HFEPX to review benchmark signals, evaluation modes, and human-feedback protocol context.
Open in HFEPXExplore Similar Papers
Jump to Paper2Code search queries derived from this paper's research context.
Need human evaluators for your AI research? Scale annotation with expert AI Trainers.