AI Evaluation Engineer
Raydar · Remote · US only
Apply on Raydar's sitePrepare with the Apply kit
First recorded by Psst: 2 November 2026. Sign in with Google to continue to the employer. A free account is enough.
The 20-second read
- Experience
- Experience not stated
- Where you can live
- US only
- Job type
- Full-time
Read from the ad by Psst's rules. Check the full ad below before applying.
Job details
- Where you can work
- United States
- Employment type
- Full-time
- Advertised pay
- $200K–300K/yr
- Experience
- Experience not stated
- Employer publication date
- 15 October 2026
- First recorded by Psst
- 2 November 2026
- Last checked at the source
- 2 November 2026
About the job
About the company
Our client is a software company.
The role
Raydar is recruiting for this role on behalf of our client. Lead the creation of a measurement approach for an AI system that carries out technical design work. Draw on both hands-on engineering judgment and evaluation know-how, and collaborate with domain experts and other teams to convert findings into clear improvement priorities.
What you'll do
- Establish a new assessment framework that defines quality for design work where several answers can be valid.
- Collaborate with domain experts, external users and internal teams to see how seasoned engineers operate and where the system struggles.
- Reproduce realistic design tasks and typical failure cases in repeatable test setups that developers can run again.
- Create and tune automated graders alongside human review processes, and confirm that the graders agree with expert opinion.
- Convert assessment results into a ranked list of improvements to the system's reasoning.
Requirements
What we're looking for
- About five or more years of professional experience in hardware engineering, AI, or a blend of both, with depth in one and practical familiarity with the other.
- Substantial hands-on background assessing AI systems, including authoring test tasks, building automated graders and benchmarking them against human review.
- Solid grasp of how autonomous, agent-style AI systems behave.
- Grounding in a hardware engineering field, enough to tell sound design from weak design.
- Comfort working directly with external users, specialists, partners and technical colleagues across functions.
- Self-assurance to guide work among many stakeholders in a role with high ownership.
- Based in the US or Canada, and able to travel to an office for one week every 6 weeks.
Bonus points
- Prior time at a robotics or physical AI company.
- Experience building things from zero in an early-stage startup.
Benefits
Compensation and benefits
- Base salary: USD 200,000 to 300,000 per year
- Equity
Location and work model
- Remote (US or Canada)
- Full-time
Posted recently with nothing unusual.
Signals, not proof. Check the company's page before you spend time on it.A resume that gets past the first 6 seconds
Recruiters skim. The Psst resume coach fixes what makes them skip you, line by line. Premium then shapes it for the exact job you're applying to.
- 1Every line checkedWeak starts, clichés and AI-sounding phrases flagged, with a 3-question fix.
- 2Tailored to the jobSee which skills and tools from the ad your resume is missing. Try it free on one of your matches.
- 3Ready for remoteTime zone, tools and proof you work on your own, the things remote teams look for.
- 4Looks the partOne page and an ATS-safe PDF, free. Premium adds 6 designs, 8 colours and a matching cover letter.
Example with made-up details. The score is the builder's own check, not an ATS rating or a promise of an interview.