Repo-based Code Annotator
Build reproducible, standardized test environments using Docker images to accurately replicate known issues or generate expected outputs according to specified procedures.
Hiring companyRemoExperts
About the role
Review and enhance the coverage and effectiveness of existing unit tests to assess the correctness and stability of target code.
Validate the completeness and rationality of test sets, ensuring that workflows for tasks related to SWE-Bench and Terminal-Bench are precisely aligned.
Write high-quality task documentation (task.yaml/README), emphasizing reproducibility and standardized processes.
Required skills
- Proficient in Linux command line and Shell scripting, skilled with tools like grep, sed, awk, curl, jq.
- Expert in Python programming for writing task harnesses, test cases, and automation tools.
- Proficient with Docker, including writing Dockerfiles and building reproducible environments.
- Familiar with testing frameworks like pytest, capable of writing structured unit tests and using techniques for mocking data and controlling randomness.
- Familiar with Git/GitHub workflows, able to submit high-quality, reproducible pull requests.
Preferred skills
- Proficiency in high-performance languages like Go or Rust.
- Familiarity with other sandbox technologies like Docker Compose or Podman.
- Ability to design datasets/tasks that prevent "task cheating."
- Understanding of scientific benchmark design principles (fairness, repeatability, scalability).
- Experience with automated testing systems or CI/CD, and a cross-disciplinary perspective.
Experience and education
- Background in Computer Science, Software Engineering, Artificial Intelligence, or related fields; or experience in roles such as Software Development, Test Engineering, DevOps, or Data Engineering.
- Contributors to open-source projects (especially in automated testing, CI/CD, containerization) are preferred.
Eligibility
Work arrangement: Fully remote
Eligible locations: Worldwide
Compensation details
$80–$120 per day.
- USD $80 - 120/Day, dependent on actual skills and experience.
More about the hiring companyRemoExperts
RemoExpertsFunction-based Code Annotator
Design and develop high-quality, function-level algorithm problems covering a wide range of algorithms and data structures, from basic to advanced.
TuringAgentic Life Science Expert
We are seeking Computational Life Sciences experts with strong scientific programming skills to develop complex, realistic tasks for Agentic Life Sciences.