Home
Mercor

AI Evaluation Rubric Auditor

  • Pay: $60 to $85 / hour
  • Contract: Contract
  • Who can apply: Canada, United States, Argentina +10
  • Posted Nov 16, 2026

In short

Get paid 60 to 85 USD per hour to check AI work from home. This freelance gig on Mercor lets you judge how AI agents handle shopping and delivery tasks. It's made for you if you love details and spotting mistakes.

What you will do

  • Audit complete tasks, each with its prompt and grading rubric.
  • Decide whether rubric failures are fair, and check if the information was available to the agent.
  • Spot the cases where a valid answer got marked wrong by the rubric.
  • Confirm that each rubric's structure follows the guidelines.
  • Send clear feedback to reviewers about prompt and rubric problems.

What you need to apply

  • Substantial experience writing or reviewing rubrics for LLM evals, with your projects listed in your application.
  • Experience giving feedback to writers or reviewers.
  • Consistent availability.
  • Can read a long agent transcript and pinpoint where it went wrong.
  • Precise writing, and you verify the data instead of trusting the reference answer.

Who can apply

Open to residents of CA, US, AR, BO, BR, CL, CO, EC, GY, PY, PE, UY and GB.

A tip from Romowork

Name each rubric project you've worked on in your application, since listing your projects is a must. We suggest showing a real example of a rubric you caught being unfair.

Originally published on Mercor

Related jobs

SME Careers by SuperAnnotate

Document Annotation Specialist (PDF)

Pay: $3 to $10 / hour

  • Contract: Contract
  • Who can apply: Indonesia, Philippines, Pakistan +3

Posted Nov 10, 2026

micro1

AI Trainer (ML Engineering)

Pay: $60 to $120 / hour

  • Contract: Contract
  • Who can apply: Open worldwide

Posted Nov 9, 2026

micro1

Open Source Contributor

Pay: $60 to $120 / hour

  • Contract: Contract
  • Who can apply: Open worldwide

Posted Nov 8, 2026