I annotated and categorized intent and prompt-response pairs to train and validate our custom Q/A models
I annotated and categorized intent and prompt-response pairs to train and validate our custom Q/A models. I also developed structured metadata tagging systems for text chunks to optimize our Retrieval-Augmented Generation (RAG) system, ensuring the retrieval mechanism matched queries with the correct context.I utilized these labeled datasets to train and optimize a transformer-based Q/A model using Sentence Transformers, fine-tuning the model's retrieval and ranking layers to bring response latency down to 250 milliseconds. Additionally, I handled data preprocessingextracting, cleaning, and filtering out noise from large educational datasets using Pythonto prepare high-quality pipelines for model ingestion and OpenAI API alignment.