QA/Test Engineer
- Salary
- $60 - $90 per hour
- Level
- Junior
- Work type
- Remote · US only
Not stated in the posting: visa.
Open. First seen 2 October 2026.
About the role
This role is for one of our clients
Compensation: $60-$90 per hour
A leading AI research organization is developing the next generation of agentic evaluation benchmarks for advanced AI models. These complex, multi-step tasks need to be reliable, unambiguous, accurately graded, and resistant to shortcuts. We are seeking experienced QA/Test Engineers to help establish the quality standards and testing processes that ensure every benchmark task measures what it is intended to measure.
Each task may represent one to two days of expert development and can span multiple technical skills. As a QA/Test Engineer, you will deeply evaluate these tasks by executing them, testing edge cases, debugging environments, and identifying potential weaknesses before they reach production evaluation. You will work closely with researchers and task authors in an iterative feedback environment.
Work Arrangement: Fully Remote — United States
Commitment: Approximately 35 hours per week
Employment Type: Full-Time
Requirements
Key Responsibilities
- Design Test Cases: Develop comprehensive test scenarios to verify that benchmark tasks function correctly, including edge cases, failure conditions, and unexpected inputs.
- Review Task Quality: Thoroughly review task descriptions, requirements, expected outcomes, and reference solutions to identify ambiguity, inconsistencies, missing requirements, and potential quality issues.
- Debug & Troubleshoot: Use Python and other development tools to investigate and resolve issues within task environments, validation logic, and automated checks.
- Develop QA Processes: Create practical, repeatable testing procedures, quality checklists, and review frameworks that can be consistently applied across benchmark tasks.
- Identify Evaluation Gaps: Examine grading logic and AI-agent execution results to detect loopholes, shortcuts, inconsistent scoring, or other factors that could compromise benchmark reliability.
- Collaborate with Researchers: Work closely with researchers and task authors to communicate issues clearly, recommend improvements, and ensure fixes are properly validated.
- Maintain Quality Standards: Help establish and enforce rigorous quality standards across complex technical evaluation datasets.
Core Qualifications
- Education: MSc or PhD in a STEM discipline, or equivalent practical experience in a research-intensive or engineering-focused environment.
- Experience: 1+ years of experience in QA, test engineering, software engineering, research engineering, or a comparable role involving significant ownership of quality.
- Proven ability to design effective test cases, develop QA processes, and investigate complex technical systems end-to-end.
- Working proficiency in Python and Git, with the ability to understand unfamiliar codebases, environments, and technical workflows.
- Strong debugging and analytical skills with an ability to systematically isolate and resolve issues.
- Exceptional attention to detail and a strong habit of maintaining clear, structured technical documentation.
- Ability to identify subtle failures, inconsistencies, edge cases, and unintended behaviors that may be overlooked by others.
- Previous experience with AI evaluation, AI training, model testing, benchmark development, or reviewing AI-generated outputs is highly desirable.
- Comfortable working independently on ambiguous and open-ended technical problems.
- Strong written communication skills and the ability to provide actionable feedback to technical stakeholders.
- A quality-focused mindset with the creativity and persistence to uncover problems others may miss.
- Ability to commit reliably to approximately 35 hours per week.
Ideal Candidate
The ideal candidate combines strong software testing and debugging capabilities with exceptional analytical judgment. You should enjoy breaking complex systems, investigating unexpected behavior, and asking whether a test genuinely proves what it claims to prove.
Experience working with AI systems, evaluation frameworks, automated grading, or complex technical benchmarks will be particularly valuable.
What this role asks for
Particular to this role
Git
Rare in Weekday AI's other 299 open postings.
Standard for QA / testing roles
Python
Common in QA / testing postings at any company.
Compared with Weekday AI's other open jobs.
The team: AI Training HIRES STEADILY
This role is one of 172 open in AI Training. 51 were posted in the last 30 days and the oldest in April: a team that hires all year rather than one being built.
WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS
Python (52 of 172), Machine Learning (41 of 172), Generative AI (31 of 172), LLM (31 of 172), Git (18 of 172), C++ (15 of 172), Excel (15 of 172)
OTHER ROLES ON THIS TEAM
- Hardware Engineering Expert - AI Training & Evaluation posted Nov 18
- DevOps / SRE Engineer (Kubernetes, Terraform), AI Expert posted Nov 18
- Film Colorist / Finishing Editor (Resolve, Baselight), AI Expert posted Nov 18
- Senior Character Animator (Film, VFX, Games), AI Expert posted Nov 18
- Software Engineer (Python, Java, C++, Go, Rust), AI Expert posted Nov 18
- VFX Supervisor / Lead Compositor (Nuke, Flame), AI Expert posted Nov 18
- Aerodynamics Expert - AI Training & Evaluation posted Nov 18
- Competitive Data Scientist (Kaggle) - AI Training & Evaluation posted Nov 18
- Mechanical Engineering Expert - AI Training & Evaluation posted Nov 18
- 3D Artist - Modeling, Rigging, Animation & AI Training posted Nov 17
- AI Safety, Responsible AI & Cybersecurity Specialist (AI Evaluation) posted Nov 17
- Brand & Advertising Art Director - Brand Identity, Campaigns & AI Evaluation posted Nov 17
- Commercial Music Production & Audio Post-Production Expert (AI Training) posted Nov 17
- Machine Learning & Scientific Computing Engineer (AI Evaluation) posted Nov 17
- Presentation, Editorial & Document Design Specialist (AI Evaluation) posted Nov 17
- Senior Luxury Automotive Strategy Leader, AI Expert posted Nov 17
- Senior SWE - Python / TypeScript posted Nov 17
- Software Engineer - Core Contributor posted Nov 17
- AI Engineer posted Nov 17
- Architectural Design & 3D Visualization Expert (AI Training) posted Nov 17
- Board-Certified Anaesthesiologist, AI Expert posted Nov 17
- Board-Certified Cardiologist, AI Expert posted Nov 17
- Board-Certified Dermatologist, AI Expert posted Nov 17
- Board-Certified Radiologist, AI Expert posted Nov 17
- Formal Mathematics & Theorem Proving Specialist (AI Evaluation) posted Nov 17
- Economists - Economic Analysis for AI Training posted Nov 16
- P&C Insurance Professionals: Paid AI Training Project posted Nov 16
- Legal Research Attorney - AI Training posted Nov 16
- Customer Support & Success Domain Expert - AI Model Training posted Nov 14
- Subject Matter Experts: ML Model Evaluation (PhD) posted Nov 13
- Senior Motion Designer (After Effects), AI Expert (Remote, US) posted Nov 12
- Commercial Video Editor (Premiere, Resolve), AI Expert (Remote, US) posted Nov 12
- ML Infrastructure and Performance Engineer, AI Expert (Remote) posted Nov 12
- Board-Certified Neurologist, Medical AI Expert (Remote, US) posted Nov 12
- Board-Certified Orthopedic Surgeon, Medical AI Expert (Remote, US) posted Nov 12
- Electrical Engineer, AI Expert (Full-Time, US) posted Nov 12
- Embedded Systems and Firmware Engineer, AI Expert (Remote, US) posted Nov 12
- Licensed Clinician (MD, NP, PA, RN, PharmD), Healthcare AI Expert (Remote, US) posted Nov 12
- Mechanical Engineer, AI Expert (Full-Time, US) posted Nov 12
- PhD Economist, AI Research Expert (Remote) posted Nov 12
- PhD Genomics and Cell Biology Scientist, AI Expert (Remote, UK or US East Coast) posted Nov 12
- Software Engineer - Backend posted Nov 12
- PhD Mathematician or Statistician, AI Expert (Remote, US) posted Nov 12
- PhD Pharmacologist or Toxicologist (Drug Discovery), AI Expert (Remote UK or US) posted Nov 12
- Game Developer (Godot Expert) posted Nov 12
- Finance Professional from a Top-Tier Firm, AI Expert (Remote, US) posted Nov 12
- PhD Physicist, AI Expert (Remote, US) posted Nov 12
- Electronics Engineer (DesignSpark PCB Specialist) posted Nov 12
- PhD Biologist (Life Sciences), AI Expert (Remote, US) posted Nov 12
- PhD Chemist, AI Expert (Remote, US) posted Nov 12
- PhD Synthetic and Analytical Chemist, AI Expert (Remote, UK or US East Coast) posted Nov 12
- Pod lead - Scientific Computing / Research Engineering Expert - Earth Sciences posted Oct 19
- Senior Software Engineer posted Oct 19
- Senior SWE - Python / TypeScript posted Oct 19
- Agentic Life Science Expert posted Oct 19
- Backend Security Engineer posted Oct 19
- Engineering Expert posted Oct 19
- Sr. Full-Stack Software Engineer posted Oct 19
- Software Engineer (Open Source Contributor) posted Oct 19
- Scientific Computing / Research Engineering Expert - Earth Sciences posted Oct 19
- Business Analyst: US Raters posted Oct 19
- Competitive Data Scientist (Kaggle) posted Oct 19
- Junior Data Scientist posted Oct 19
- Junior Software Engineer posted Oct 19
- Business Analyst - US - Educational Raters - Gemini posted Oct 19
- Business Analyst: Non-US Raters posted Oct 19
- Insurance Expert - AI Training & Evaluation posted Oct 8
- Biology / environmental science Evaluator posted Sep 17
- Biotech Investment Analyst - Drug Asset & Company Assessment posted Sep 17
- Data Scientist Talent Network posted Sep 17
- Equity Research Expert - Single-Name Analyst Publications posted Sep 17
- Engineering & Software Domain Expert posted Sep 17
- Management Consultant Talent Network posted Sep 17
- Multilingual Inpatient Hospitalist (MD) - Clinical Documentation & AI Evaluation posted Sep 17
- Finance Domain Expert - AI Training & Evaluation posted Sep 17
- Analyst / Insights Experts posted Sep 17
- MLOps Engineer, LLM Systems (Serving, GPU Kernels, Profiling) posted Sep 17
- Software Engineer, Go - Codebase Q&A posted Sep 16
- Biology / environmental science Evaluator posted Sep 16
- Applied History & Political Science Benchmark Specialist posted Sep 16
- Software Engineer, Python - Codebase Q&A posted Sep 16
- Software Engineer, C - Codebase Q&A posted Sep 16
- AI Safety Experts - English & Finnish posted Sep 16
- Biology PhD Coding Experts posted Sep 14
- STEM PhDs and Technical Domain Experts posted Sep 14
- Physics PhD Coding Experts posted Sep 14
- Material Science PhD Coding Experts posted Sep 14
- Chemistry PhD Coding Experts posted Sep 14
- Data quality / CRM operations Evaluator posted Sep 14
- Mathematics PhD Coding Experts posted Sep 14
- FP&A & Treasury Analyst posted Sep 14
- Electrical Engineering Expert posted Sep 8
- Technical Accounting & SEC Reporting Specialist posted Sep 8
- Incident management / reliability / SRE Evaluator posted Sep 8
- Engineering / manufacturing / technical operations Evaluator posted Sep 8
- Mechanical Engineering Expert posted Sep 8
- Financial Crime, AML & KYC Analyst posted Sep 8
- Generalist (AI Training Specialist) posted Sep 7
- Compliance / regulatory response with financial-services AI Evaluator posted Sep 2
- Cloud / DevOps Engineer (Infra & IaC) posted Sep 2
- Materials Science Domain Expert posted Sep 2
- Product / Platform Experts posted Sep 2
- Engineering / Platform Professionals posted Sep 1
- Product / Artifacts Experts posted Sep 1
- Applied Computer Science Benchmark Specialist posted Aug 17
- Sales Engineering Expert posted Aug 17
- UI / UX Design Expert posted Aug 17
- Professional Design Experts posted Aug 15
- Project Coordinator - AI & Data Projects posted Aug 15
- Senior Software Engineer, Full Stack (Python, Java, Rust, C#, C++) posted Aug 14
- Legal Domain Expert - AI Training & Evaluation posted Aug 14
- Software Engineer, Full Stack (Python, Java, Rust, C#, C++) posted Aug 14
- Senior Design Expert - Paid AI Design Research Study posted Aug 14
- Biology Expert (PhD) - AI Safety posted Jul 30
- Electrical Engineering Expert posted Jul 30
- Software Engineering Expert posted Jul 30
- Mechanical Engineering Expert posted Jul 30
- Chemical Engineering Expert posted Jul 30
- Machine Learning Engineer - Model Evaluation & Experimentation posted Jul 30
- ASIC/SoC Design & Verification Engineer posted Jul 29
- Data Engineer - Finance posted Jul 29
- MLOps Engineer (JAX, PyTorch, Pallas/Triton) posted Jul 29
- Performance Engineer (C++, Python, Rust) posted Jul 29
- Computer Vision Expert posted Jul 29
- Machine Learning & NLP Expert posted Jul 29
- AI Rater Guidelines Writer (Linguist / Instructional Designer) posted Jul 29
- Data Science & Quantitative Analysis Expert posted Jul 29
- QA/Test Engineer posted Jul 29
- Data Science Expert posted Jul 28
- Architect (CAD & Construction Documentation) posted Jul 14
- Frontend Software Engineer posted Jul 10
- AI Evaluation Specialist posted Jul 9
- Technology AI Evaluation Expert posted Jul 9
- Southern American English Voice Actor (AI Speech & Voice Modeling) posted Jul 9
- Legal AI Evaluation Expert posted Jul 9
- Software Engineering Expert (AI Training) posted Jul 9
- Equity Research AI Specialist posted Jul 8
- Internal Medicine AI Evaluation Specialist posted Jul 8
- Medical Expert AI Evaluation Specialist posted Jul 8
- Game Tools & Modding Engineer posted Jul 3
- Biology Subject Matter Expert (AI Training & Scientific Content) posted Jul 3
- Chemistry Subject Matter Expert - AI Training (Remote) posted Jul 3
- AI Safety & Red Teaming Specialist posted Jul 3
- Medical Subject Matter Expert (AI Training & Documentation) posted Jul 3
- Engineering Subject Matter Expert (AI Training) posted Jul 3
- Computer Science Subject Matter Expert (AI Training) posted Jul 3
- Financial Documentation Expert (AI Training) posted Jul 3
- Physics Subject Matter Expert (AI Training) posted Jul 3
- Mathematics Subject Matter Expert (AI Training) posted Jul 3
- Network Engineer - Data for Autonomous Systems annotation posted Jun 24
- Management & Strategy Consultants (MBB/Big 5) AI Trainer posted Jun 24
- CUDA Engineering Expert posted Jun 24
- Cybersecurity SWE - AI Safety posted Jun 24
- Mechanical Engineer posted Jun 5
- US Tax Law AI Specialist (CPA / EA / CA) posted May 14
- AI Tax Compliance & Legal Interpretation Expert posted May 14
- Physics AI Specialist posted May 5
- Clinical AI Specialist posted May 5
- Investment Banking AI Specialist posted May 5
- Tax AI Expert posted Apr 21
- Capital allocation AI Expert posted Apr 21
- Insurance Underwriting AI Expert posted Apr 21
- Financial modeling AI Expert posted Apr 21
- Compliance and controls AI Expert posted Apr 21
- Insurance Policy Administration AI Expert posted Apr 21
- Risk & Actuarial AI Expert posted Apr 21
- Financial reporting AI Expert posted Apr 21
- Strategic finance AI Expert posted Apr 21
- Financial planning & analysis (FP&A) AI Expert posted Apr 21
- Insurance Claims Management AI Expert posted Apr 21
- Ledger and reconciliation AI Expert posted Apr 21
From the team and dates on Weekday AI's own job board.
Other openings for this role
2 open postings for this role in United States, first posted 2 October 2026. The company is likely hiring for more than one position. None has been taken down and posted again.
- 2 October 2026 – open (this posting)
- 2 October 2026 – open
Similar jobs
- Site Reliability Engineer, Kubernetes at SpaceX
Shares 2 skills: Git, Python · Junior level · same country - Software and Process Engineer at STR
Shares 2 skills: Git, Python · same country - AI/ML Software Engineer, Platform at Falcomm
Shares 2 skills: Git, Python · same country - Contract Student Worker- Software Engineer Mobile Developer Experience at Zoox
Shares 2 skills: Git, Python · same country - Software Engineer II at Rocket Lab Corporation
Shares 2 skills: Git, Python · same country - VFX Supervisor / Lead Compositor (Nuke, Flame), AI Expert at Weekday AI
Also at this company
More jobs like this
Is this your posting? Email [email protected] to have it removed.