Start date: TBC.

End Date: TBC.

This is an operational, hands-on role, you will own day-to-day execution for one or more annotation queues on a live AI training/eval program: managing throughput, running quality calibrations, tracking contributor performance, and building out reporting systems that keep stakeholders informed in real time.

You will have experience running queues and quality on AI data programs. The selected candidate will be able to speak to specific projects, calibration processes they’ve built, and contributor performance systems they’ve run.

What "good" looks like for us:

• Scale up to running a full AI end-to-end data program, owning pipeline, throughput, calibration, and contributor performance at program level.
• Operates as an AI training consultant, you are comfortable advising on processes and methodology.

Success looks like

Predictable SLA delivery, a stable backlog, improving quality and reducing rework, and proactive dashboards reporting that give stakeholders real-time visibility.

Why Prolific is a great platform to join as an expert participant

• Joining our platform as a participant will give you the chance to influence the AI models of the future. Once you pass our assessment, you can join Prolific in just 15 minutes and start enjoying competitive pay rates, flexible hours, and the ability to work from home. 

Queue Operations & Task Management

• Own work queues: intake, triage, prioritization, task assignment, and SLA delivery.
• Monitor task queue and backlog control.
• Identify and troubleshoot bottlenecks (unclear policy, tooling issues, supply gaps).

Quality Assurance & Performance Management

• Monitor quality at project and individual annotator or contributor level.
• Perform QA tasks such as making corrections, providing annotator feedback, flagging systematic confusion and suggestions to team lead.
• Manage annotator or contributor performance by performing tasks like unassigning underperformers, identifying and retaining high-performers.

Reporting & Dashboards

• Build and maintain dashboards to track throughput, disagreement rates, defect or rework rates, and contributor performance.
• Turn raw queue data into clear, actionable reporting for stakeholders.
• Provide feedback for process improvements based on data trends.

Training & Support

• Oversee dedicated Slack channels by answering participant questions, providing support, and escalating complex issues to the Client Delivery Lead.
• Train new annotators or contributors on project guidelines.
• Conduct live or recorded webinars as new participants join projects.
• Host "office hours" for participants.
• Oversee dedicated Slack channels by answering questions, providing support, escalating complex issues to Client Delivery Lead.

• Proven, direct experience running queues and quality on AI data programs, ideally at a competitor.
• 3+ years in high-throughput ops (data labeling, trust & safety, BPO, content ops, CX ops), with recent, specific AI data ops experience.
• Able to identify shallow, low-effort, AI-generated, or phoned-in responses.
• Proven ability to design and run calibration processes from scratch.
• Ability to track record by building or maintaining dashboards or reporting systems (spreadsheets is required, SQL/BI/Looker/Tableau a strong plus).
• Experience managing contributor performance at scale including making retention or removal decisions.
• Comfortable with ambiguity and fast-changing priorities on live programs.

Work arrangement: Fully remote

Applicant eligibility: 35 eligible countries · 1 eligible region

At this time, we can only onboard successful participants who are currently based in, and able to verify their identification within, the specified regions.

If you cannot find your current country of residence provided, please select: "REST OF WORLD."

Hourly Rate: Include the rate you are comfortable with on your CV.