Search And Product Relevance Evaluator (English)

About OpenTrain AI

OpenTrain AI is the hiring and contracting organization for this role. OpenTrain helps people start and build careers in AI training and data labeling, connecting contributors with paid projects that teach machines using human expertise.

Creating an OpenTrain account is free; through our platform you can discover projects, build a profile, and apply quickly for remote, flexible work that directly shapes how AI systems behave.

Why AI Training Matters

AI training (data labeling/annotation) is the human side of building intelligent systems. Annotators evaluate search queries, rate product relevancy, compare items, and provide rationale so models return accurate results and better shopping experiences.

This line of work is highly flexible, often remote and part-time, and places contributors on the cutting edge of how search and e-commerce systems are improved.

• Remote, flexible work you can fit around other commitments
• Work that directly improves e-commerce search relevance and user experience
• Accessible to many backgrounds—specialist projects reward domain knowledge

The Role — What This Project Is

You will evaluate and annotate search queries and product comparisons to improve search results and product relevancy. The project asks you to analyze queries, decide whether corrections are needed, select the most relevant products, and categorize relevance according to provided guidelines.

This listing is for the Test Phase: a paid, 1-hour evaluation. Successful testers may be invited to the full project expected to last 1–2 months. The role is contractor, part-time, and worldwide.

• Format: Paid 1-hour test, then performance-based selection for 1–2 month assignment
• Employment type: Contractor, part-time
• Workload: Less than 20 hours/week

What You'll Do Day-to-Day

Follow detailed annotation guidelines to assess how well product results match user intent and search queries. Work is image-based but also requires interpreting textual queries and justifying decisions in writing.

• Analyze search queries and determine intent and possible corrections
• Select and rank the most relevant products for a query
• Categorize product relevance and attributes based on set criteria
• Write concise rationales explaining each decision
• Use search engines and e-commerce sites to validate assumptions when instructed
• Complete tasks that include classification, evaluation/rating, question answering, and data collection of image-based results

Requirements

You must meet the mandatory qualifications listed below; they are essential for accurate, consistent labeling and for advancement beyond the test phase.

• Previous experience in search engine evaluation projects (assessing relevancy, query corrections, product categorization) is required
• Strong English comprehension and writing skills for clear justifications
• Analytical and critical thinking with high attention to detail
• Familiarity with e-commerce platforms, product categories, brands, and consumer search behavior is highly advantageous
• Comfort navigating and using search engines (e.g., Google, Amazon) to validate labels
• Ability to follow detailed instructions and adhere to strict annotation protocols
• Technical comfort with annotation tools or similar platforms used for labeling

Test Phase & Interview

The first step is a paid 1-hour test. Performance in the test determines placement into the full project. During the live chat interview you will be asked to complete a short writing test and must answer before the interview ends.

Writing test (you will be asked this in live chat): Evaluate the query "red sports shoes." Explain how you would assess the relevance of a pair of red running sneakers from a well-known sports brand — what factors you would consider and how you would justify your decision.

• Test length: 1 hour, paid at listed hourly rate
• The live chat includes the writing test above and requires a response to be considered
• Advancement to the full project is based on accuracy, consistency, and adherence to guidelines during the test

Compensation, Schedule, And How To Apply

Compensation is hourly at USD $4. This is a part-time contractor role with less than 20 hours/week expected; the initial test is one hour and the full project runs about 1–2 months if you are selected.

This project uses image-based tasks and includes classification, evaluation/rating, question answering, and data-collection style labeling. OpenTrain will provide the annotation guidelines and access to the labeling interface.

• Pay type: Pay per hour — USD $4/hour
• Data type: Image tasks; label types include classification, data collection, evaluation/rating, and question answering
• Worldwide applicants accepted; you must meet the listed experience and language requirements
• To apply: create a free OpenTrain account, complete the test, and participate in the live chat interview