Software Engineers: Paid Code Review for AI Agent Evaluation
Company shown to members 🔒 · Remote · US only
Apply free: 3 jobs are on us✨ Tailor my resume for this job🎤 Practise the interview question, free
✓ Straight from the company's own careers page. No middlemen, no “we'll send you a check for equipment”. How we check jobs
✓ Sent 100 applications and heard nothing? Make this one count: tailor your resume to this ad in one tap, free.
✨ Not sure this is the one? Get your remote job matches in 30 seconds →
First recorded by Psst: 12 October 2026. Your first 3 jobs are free to apply to (the first right away, 2 more with your email). After that, the company and the apply link are for Psst members, from $4.99 for 7 days.
Practise the first interview question for this job
If the company replies, the first call usually starts with a question like this. Vague answers are the easiest to reject. Write yours and get an AI score in under a minute.
The 20-second read
- Experience
- Entry-level (years not stated)
- Where you can live
- US only
- Skills and tools
- machine learning
- Job type
- Contract or freelance
Read from the ad by Psst's rules. Check the full ad below before applying.
Job details
- Where you can work
- United States
- Employment type
- Contract
- Advertised pay
- $65/hr
- Experience
- Entry-level (years not stated)
- Employer publication date
- 2 October 2026
- First recorded by Psst
- 12 October 2026
- Last checked at the source
- 11 November 2026
About the job
WHAT WE'RE RESEARCHING
We're running a paid study on the coding environments and programming tasks used to benchmark artificial intelligence agents. Creating robust evaluation harnesses ensures that AI models are tested against realistic software engineering scenarios. This work directly feeds into improving how autonomous agents handle complex coding objectives.
HOW IT WORKS
During this remote session, you will review a series of coding tasks and their corresponding evaluation harnesses. You will assess whether the programming challenges accurately reflect real-world software engineering problems. We will ask you to verify the logic, test cases, and overall structure of the environments provided. Your technical feedback will be captured through a guided conversation and screen-sharing exercises.
WHO THIS IS FOR
We are looking for practicing software engineers with strong backgrounds in building and testing complex systems. Candidates should have direct experience writing test suites, evaluation harnesses, or comprehensive code reviews. We welcome backend engineers, full-stack developers, machine learning engineers, and software architects.
WHAT YOU'LL DO
- Review realistic programming tasks for accuracy and difficulty
- Verify the logic and test cases within provided evaluation harnesses
- Assess whether coding environments effectively measure software engineering skills
- Walk us through your thought process while analyzing complex code structures
WHO SHOULD APPLY
- Active professional experience as a software engineer
- Hands-on experience building or maintaining test suites and evaluation harnesses
- Comfortable reviewing code and explaining technical concepts aloud
- Familiarity with complex real-world software architecture
COMPENSATION
$65 per hour
READY TO PARTICIPATE?
Start your paid interview now [link for members]
ABOUT the company
The company is building the world's largest pool of vetted human experts for AI. Researchers, AI labs, and product teams use the company to recruit, screen, and pay study participants across industries, languages, and skill sets.
Learn more at the company [link for members] or on YouTube at @jointerac [link for members]
Apply free: 3 jobs are on us✨ Tailor my resume for this jobPrepare with the Apply kitMore remote software engineer jobs →
Posted recently with nothing unusual.
Signals, not proof. Check the company's page before you spend time on it.Similar remote jobs
- Principal Software Engineer - Agent AI PlatformUS only
- Senior Lua Developer (Roblox) – AI Code EvaluationUS only
- Staff Software Engineer, AI Foundations (AI Agent Optimization)US, Canada only
- Pharmacists: Paid Expert Review for Clinical AI BenchmarksUS only
- AI Voice Evaluation SpecialistUS only
Explore more jobs
A resume that gets past the first 6 seconds
Recruiters skim. The Psst resume coach fixes what makes them skip you, line by line, and one tap tailors it to the exact job you're applying to.
- 1Every line checkedWeak starts, clichés and AI-sounding phrases flagged, with a 3-question fix.
- 2Tailored to the jobSee which skills and tools from the ad your resume is missing. Try it free on one of your matches.
- 3Ready for remoteTime zone, tools and proof you work on your own, the things remote teams look for.
- 4Looks the partOne page, 6 designs, 8 colours and a matching cover letter, free to build. Premium downloads the PDF and Word files.
Example with made-up details. The score is the builder's own check, not an ATS rating or a promise of an interview.