You will develop problems within your area of expertise that test advanced reasoning rather than simple factual recall. You will identify weaknesses in frontier AI models, including incorrect assumptions, incomplete reasoning, mathematical or scientific errors, missed edge cases, and answers that appear plausible but fail under expert scrutiny.

A major focus of this role is rubric design and writing. You will translate complex technical judgment into clear, specific, and measurable evaluation criteria, including required concepts, reasoning steps, acceptable alternative approaches, critical errors, and partial-credit considerations. You may also critique and improve rubrics developed by other experts.

This role is with SME Careers, a fast-growing AI Data Services company and subsidiary of SuperAnnotate, delivering training data for many of the world’s largest AI companies and foundation-model labs. Your technical expertise directly helps improve the world’s premier AI models by making their reasoning more accurate, rigorous, and reliable.

• Demonstrated advanced expertise in a clearly defined technical field or specialization.
• Ability to create challenging technical questions that require multi-step reasoning, domain expertise, or professional judgment.
• Strong ability to design detailed evaluation rubrics defining required reasoning, correct methodology, acceptable alternatives, partial-credit criteria, and critical errors.
• Ability to distinguish between a correct final answer and an answer supported by valid versus flawed reasoning.
• Comfortable reviewing and critiquing other experts’ rubrics for ambiguity, missing criteria, redundancy, technical inaccuracies, or poor scoring design.
• High attention to detail when evaluating formulas, assumptions, units, methodology, edge cases, logical consistency, and technical terminology.
• Reliable, self-directed, and able to deliver consistent quality in an hourly, remote contractor workflow.

Skills & technologies

• STEM
• Mathematics
• Physics
• Chemistry
• Engineering
• Computer Science.
• Technical Reasoning.
• AI Evaluation.
• Rubric Design.
• Rubric Writing.
• Benchmarking
• Model Evaluation.
• Question Writing.
• Expert Review.
• English

Why Work with SME Careers?

• Be a part of a forward-thinking team and help us train AI to communicate more effectively.

Shape the Future of AI

• Your work directly contributes to how AI systems learn and communicate.

Your Schedule, Your Rules

• Set your own hours and work around your life.

Weekly Pay

• Receive fast, reliable payments once work is approved.

Community Rewards

• Get rewarded for referrals with ongoing bonus income.

• Create expert-level technical questions: Develop difficult questions that test genuine technical reasoning and expose weaknesses in advanced AI systems.
• Design evaluation rubrics: Write structured criteria defining what a correct, rigorous, and complete response must contain.
• Define partial-credit standards: Identify essential reasoning steps, acceptable alternatives, minor errors, and critical failures.
• Evaluate AI-generated solutions: Assess outputs for correctness, reasoning quality, completeness, methodology, and technical precision.
• Identify hidden reasoning errors: Detect cases where an answer reaches the correct conclusion using invalid assumptions or flawed logic.
• Critique peer rubrics: Review other experts’ criteria for technical accuracy, clarity, completeness, and scoring consistency.
• Develop reference content: Produce expert answers, explanations, critiques, and other gold-standard content.
• Support AI training: Create questions, rubrics, evaluations, preference judgments, and reference-answer pairs suitable for RL and SFT workflows.
• Maintain technical rigor: Ensure content reflects appropriate professional or academic standards within the relevant discipline.

• Bachelor’s degree or higher in Mathematics, Physics, Chemistry, Engineering, Computer Science, Statistics, Applied Science, or another technical discipline.
• 3+ years of professional, academic, research, or industry experience in your stated technical domain.
• Experience with exam writing, academic grading, peer review, research review, technical QA, standards development, or assessment design is strongly preferred.
• Prior experience with AI evaluation, RLHF, SFT, benchmarking, data annotation, prompt design, or LLM evaluation is preferred.

INTERVIEW

Required

Work arrangement: Fully remote

Eligible countries: United States