Generalist
Text
Evaluation/RatingThe project is basically a task where you make a difficult research question for an AI model to answer. The question has to be clear, self-contained, and have only one correct answer, but it should not be easy to find with one quick Google search. You start by finding a hard-to-reach fact first, usually from something like a government PDF, local report, database, school page, or official document, and then you build a prompt around that fact using clues that require multiple sources. After writing the prompt, you test model responses and at least one model has to fail, otherwise the question is too easy. Then you provide the correct answer, list all the source URLs, explain exactly how each source supports the answer, check the fanout claims, and write a golden trajectory showing the ideal search path. The main goal is to create a question where the answer is short and verifiable, but the path to finding it requires careful searching, source-checking, and reasoning across at least five distinct sources.