High-Performance LLM Training Pipeline (Helium-Nano)
Built and trained a 45M parameter Transformer training pipeline from scratch on a 560M token dataset to support model learning. Developed mixed-precision (bfloat16) training and performance optimizations to improve throughput and stability while reducing GPU memory consumption. Integrated RAG and multi-agent orchestration components to enable retrieval-augmented workflows during experimentation. • Trained a 45M parameter Transformer on a 560M token dataset using PyTorch mixed-precision • Optimized training throughput from 25K to 409K tokens/sec using PyTorch Inductor and CUDA Graphs • Integrated Weights & Biases for experiment tracking, artifacts, and visualization • Experimented with Retrieval-Augmented Generation (LangChain) and multi-agent orchestration (LangGraph)