LlamaIndex Developer — Code Review & Evaluation

About OpenTrain and the AI-training Industry

OpenTrain is the #1 platform for finding and building careers in AI training and data labeling. Contributors use the platform to discover projects, build profiles, and apply quickly — all remote and often with flexible schedules.

AI training (data labeling, annotation, and human feedback) is the human side of building modern AI. Teams rely on experienced annotators and evaluators to prepare, review, and improve examples that teach models how to index, retrieve, and generate accurate outputs.

The Role

We’re hiring an experienced LlamaIndex developer to evaluate AI-generated prompts, responses, and code related to retrieval-augmented generation (RAG), document indexing, and structured data retrieval. This is a part-time contractor role: less than 20 hours per week, paid per hour at $25 USD.

You will both perform technical code and explanation reviews and run structured, AI-driven interviews that assess other candidates’ hands-on LlamaIndex expertise and English communication skills.

• Review and label AI-generated LlamaIndex prompts, code, and explanations for accuracy, relevance, and best practices.
• Provide detailed, structured feedback and corrections for code snippets, retrieval strategies, and indexing approaches.
• Conduct technical interviews that verify candidates’ practical LlamaIndex experience and query optimization skills.

What You’ll Do — Review Tasks & Interview Workflow

This role combines annotation/evaluation with interviewing. You will follow a set of technical checks and interview guidelines to assess candidate responses and AI outputs. Your assessments should be structured, evidence-based, and written clearly in English.

• Label and categorize AI outputs: correctness, relevance, completeness, and adherence to LlamaIndex best practices.
• Identify code errors, inefficiencies, and missing safeguards in LlamaIndex integrations and provide corrected snippets or suggested fixes.
• Run an AI-driven interview that probes for 5+ years of hands-on LlamaIndex work, including RAG, document indexing, and structured retrieval experience.
• Present a short LlamaIndex-based code snippet with a deliberate issue, ask the candidate to find/fix it, and evaluate their solution.
• Assess experience with vector DBs (FAISS, Pinecone, ChromaDB), embeddings, document chunking, query routing, and integration with LangChain or standalone LLMs.

Technical Areas We’ll Test

Your evaluations should cover practical engineering choices and RAG design patterns. Focus on retrieval accuracy, indexing strategies, performance trade-offs, and integration correctness.

• Document indexing strategies and chunking approaches for long documents.
• Embedding choices and vector store trade-offs (FAISS, Pinecone, ChromaDB, etc.).
• Query optimization, reranking, and routing in RAG pipelines.
• Integration patterns between LlamaIndex and LLM frameworks (LangChain or standalone).
• Code hygiene, error handling, and reproducible examples for indexing and retrieval.

Requirements

You must meet the role’s technical and communication requirements. The job requires hands-on, practical LlamaIndex experience and strong English writing ability to produce clear, structured feedback.

• Minimum of 5+ years hands-on experience working with LlamaIndex and RAG/document indexing systems.
• Proven experience with vector databases and embeddings (examples such as FAISS, Pinecone, ChromaDB are relevant).
• Deep familiarity with query optimization, document chunking, retrieval routing, and best practices for LlamaIndex integrations.
• Ability to read, analyze, and correct code snippets in real-world LlamaIndex/LLM integrations.
• Strong English writing skills for producing detailed, structured evaluation reports and interview feedback.
• Available to work less than 20 hours per week as a contractor (part-time).
• Worldwide candidates accepted; role is remote and contractor-based.

Compensation, Tools, and Logistics

This is a contract, part-time role paid hourly at $25 USD per hour (PAY_PER_HOUR). Work is billed and tracked as agreed with the project manager. Labeling software is listed as OTHER and the primary data type is computer code/programming.

You will receive example AI outputs, code snippets, and a structured rubric to label and score responses. Deliverables include labeled evaluations, written feedback, and interview notes for candidate screening.

• Pay rate: $25 USD per hour, contractor, part-time.
• Estimated time commitment: under 20 hours/week.
• Primary tasks involve COMPUTER_CODE_PROGRAMMING labels and technical evaluation.
• Tooling: custom/other labeling software provided by the project.

Who Should Apply

Apply if you are a practical LlamaIndex engineer who enjoys reading and fixing code, explaining complex concepts in simple terms, and judging technical answers with a critical eye. This role favors demonstrable, hands-on experience over theoretical familiarity.

Do not apply if your LlamaIndex experience is purely theoretical or minimal — we will prioritize candidates who can show concrete project examples and code-level competence.

• Ideal candidates can point to real-world LlamaIndex projects and clearly explain their role and technical decisions.
• Prior experience reviewing AI-generated code or performing code-quality evaluations is strongly preferred.
• Only proceed with candidates who provide concrete, practical answers and examples — vague or purely theoretical responses will be rejected.

How to Apply

Create or sign in to your OpenTrain account, complete your profile, and submit your application for this project. Include a brief resume and links or snippets showing your LlamaIndex work (repositories, code samples, or detailed project descriptions).

In your application, state your weekly availability, confirm you meet the 5+ year LlamaIndex experience requirement, and note any prior experience evaluating AI-generated code. Clear, structured examples will speed review of your application.