Software Engineer – AI Evaluation & Automation

1 day ago

To Apply for this Job Click Here

IDR is seeking a Software Engineer – AI Evaluation & Automation to join one of our top clients for a remote opportunity. This role is within the technology sector, focusing on developing tools to measure the performance of AI-powered software development solutions. The position offers the chance to work on innovative projects that ensure the reliability and reproducibility of AI evaluations.

Position Overview for the Software Engineer – AI Evaluation & Automation:

Help build and scale tooling used to measure the performance of AI-powered software development tools.
Develop evaluation harnesses, automate benchmark runs, and ensure results are reproducible and trustworthy.
Focus on validating measurement approaches against human judgment and analyzing benchmarking results.
Collaborate with engineering and data teams to improve tooling and document evaluation processes.
Support execution-based benchmarking across quality, productivity, and efficiency measures, including cost and latency.

Requirements for the Software Engineer – AI Evaluation & Automation:

Strong software engineering background, with experience building automation, developer tooling, or validation systems.
Proficient in Python, and comfortable with Java, JavaScript, or similar languages.
Solid working knowledge of Git, including branch management, history, and worktrees, and containerization with Docker.
Experience with APIs, development environments, CI/CD pipelines, and engineering workflows.
Some familiarity with AI, LLM, or agent evaluation processes, including understanding common pitfalls.

What's in it for you?

Competitive compensation package
Full Benefits; Medical, Vision, Dental, and more!
Opportunity to get in with an industry leading organization.

Why IDR?

25+ Years of Proven Industry Experience in 4 major markets
Employee Stock Ownership Program
Dedicated Engagement Manager who is committed to you and your success.
Medical, Dental, Vision, and Life Insurance
ClearlyRated's Best of Staffing® Client and Talent Award winner 12 years in a row.

#LI-Remote IDR is an equal opportunity employer. We provide reasonable accommodations for disabled employees and do not discriminate based on race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, disability, veteran status, marital status, criminal history, or any other protected category under applicable federal, state, or local laws. We consider all qualified applicants, including those with criminal histories, in compliance with applicable Fair Chance laws. To comply with civil rights laws, we invite applicants to voluntarily self-identify their race, ethnicity, gender, and veteran status. This information is confidential, optional, and will not affect your application. You may access this form here. Pursuant to pay transparency laws, salary ranges and benefits will be disclosed where required by law. Compensation is based on factors such as experience, skills, and qualifications. If you require a reasonable accommodation during the application or interview process, please contact Human Resources at hr@idr-inc.com