Home
Mercor

SWE-Bench Task Auditor

  • Pay: $70 to $90 / hour
  • Contract: Contract
  • Who can apply: United States
  • Posted Aug 28, 2026

In short

Want remote work on a freelance contract that pays 70 to 90 USD per hour? This mission puts you in charge of checking coding tasks used to train and test frontier AI models. If you know real software projects inside out and love spotting mistakes, this one is built for you.

What you will do

  • Judge whether software engineering benchmark tasks are high quality, correct and reproducible.
  • Review repository-level tasks, reference patches and test harnesses.
  • Check grading integrity, looking for answer leaks and ways models could game the scoring.
  • Write clear, rubric-based feedback on every task you review.

What you need to apply

  • 3+ years of professional software engineering experience.
  • Real open-source work, such as merged pull requests or maintainer or committer roles.
  • Skill at auditing reference patches, test runners and Docker isolation.
  • Fluency in Python plus at least one of Java, Go, TypeScript or C++.

Who can apply

This job is open to residents of the US only. It is a remote freelance contract with 3 openings.

A tip from Romowork

We suggest putting your merged open-source pull requests and any maintainer history at the top of your application, since those are core requirements for this role.

Originally published on Mercor

Related jobs