QA/Test Engineer
- Salary
- $60 - $90 per hour
- Level
- Junior
- Work type
- Remote · US only
Not stated in the posting: visa.
Open. First seen 2 October 2026.
About the role
This role is for one of our clients
Compensation: $60-$90 per hour
Join a pioneering AI initiative focused on building the next generation of evaluation benchmarks for frontier AI models. We are seeking experienced QA and Test Engineers to ensure every benchmark is reliable, reproducible, and accurately measures real AI capabilities.
In this role, you will review complex, multi-step evaluation tasks, validate their correctness, identify edge cases, and strengthen testing methodologies before benchmarks are deployed. You'll collaborate closely with AI researchers and task authors to improve evaluation quality, eliminate ambiguity, and ensure benchmark integrity.
This is a fully remote, full-time engagement requiring approximately 35 hours per week.
Requirements
Key Responsibilities
- Design comprehensive test cases that validate evaluation tasks, including complex edge cases and unexpected scenarios.
- Review benchmark tasks and reference solutions to identify ambiguity, inconsistencies, missing requirements, and grading gaps.
- Debug task environments and Python-based evaluation scripts to ensure reliable execution and accurate results.
- Develop repeatable quality assurance processes, validation checklists, and testing frameworks for benchmark creation.
- Identify potential shortcuts, exploits, or weaknesses that could compromise evaluation accuracy or benchmark integrity.
- Collaborate with AI researchers, engineers, and task authors to improve task quality, reproducibility, and technical rigor.
Required Qualifications
- Master's degree, PhD, or equivalent practical experience in a STEM discipline involving software engineering, research, or advanced technical problem solving.
- Minimum 1 year of professional experience in Quality Assurance, Test Engineering, Software Engineering, Research Engineering, or a related technical field with strong quality ownership.
- Proven experience designing test cases, validating complex software systems, and debugging end-to-end workflows.
- Strong proficiency in Python and Git, with the ability to troubleshoot unfamiliar codebases and technical environments.
- Excellent analytical thinking, problem-solving skills, and exceptional attention to detail.
- Experience documenting bugs, test strategies, and technical findings with clear written communication.
- Experience evaluating AI systems, machine learning models, or AI-generated outputs is preferred.
- Ability to work independently while managing multiple complex tasks with minimal supervision.
- Ability to commit approximately 35 hours per week on a consistent basis.
Preferred Qualifications
- Experience with AI evaluation, benchmark development, or quality assurance for machine learning systems.
- Background in automation testing, validation frameworks, or software quality engineering.
- Familiarity with large language models, AI agent workflows, or evaluation pipelines.
- Experience creating repeatable QA processes for research or engineering projects.
Why Join
- Play a key role in improving the quality and reliability of next-generation AI evaluation benchmarks.
- Collaborate with leading AI researchers on cutting-edge evaluation methodologies.
- Help ensure AI systems are tested against rigorous, real-world scenarios.
- Apply your software testing and quality engineering expertise to advance AI reliability.
- Enjoy the flexibility of a fully remote engagement while contributing to impactful AI research.
Equal Opportunity
We are committed to creating an inclusive workplace where all qualified applicants receive equal consideration. Reasonable accommodations are available throughout the application and engagement process.
Contract & Engagement Details
- Independent contractor engagement.
- Fully remote with flexible working hours.
- Expected commitment of approximately 35 hours per week.
- Project duration may be extended, shortened, or concluded based on project requirements and individual performance.
- Work does not require access to confidential or proprietary information from any current or former employer.
- Payments are issued weekly based on approved work completed.
- At this time, we are unable to support H1-B or STEM OPT candidates.
What this role asks for
Particular to this role
Git · LLM
Rare in Weekday AI's other 299 open postings.
Standard for QA / testing roles
Python · Test Automation
Common in QA / testing postings at any company.
Compared with Weekday AI's other open jobs.
The team: AI Training HIRES STEADILY
This role is one of 172 open in AI Training. 51 were posted in the last 30 days and the oldest in April: a team that hires all year rather than one being built.
WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS
Python (52 of 172), Machine Learning (41 of 172), Generative AI (31 of 172), LLM (31 of 172), Git (18 of 172), C++ (15 of 172), Excel (15 of 172)
OTHER ROLES ON THIS TEAM
- Hardware Engineering Expert - AI Training & Evaluation posted Nov 18
- DevOps / SRE Engineer (Kubernetes, Terraform), AI Expert posted Nov 18
- Film Colorist / Finishing Editor (Resolve, Baselight), AI Expert posted Nov 18
- Senior Character Animator (Film, VFX, Games), AI Expert posted Nov 18
- Software Engineer (Python, Java, C++, Go, Rust), AI Expert posted Nov 18
- VFX Supervisor / Lead Compositor (Nuke, Flame), AI Expert posted Nov 18
- Aerodynamics Expert - AI Training & Evaluation posted Nov 18
- Competitive Data Scientist (Kaggle) - AI Training & Evaluation posted Nov 18
- Mechanical Engineering Expert - AI Training & Evaluation posted Nov 18
- 3D Artist - Modeling, Rigging, Animation & AI Training posted Nov 17
- AI Safety, Responsible AI & Cybersecurity Specialist (AI Evaluation) posted Nov 17
- Brand & Advertising Art Director - Brand Identity, Campaigns & AI Evaluation posted Nov 17
- Commercial Music Production & Audio Post-Production Expert (AI Training) posted Nov 17
- Machine Learning & Scientific Computing Engineer (AI Evaluation) posted Nov 17
- Presentation, Editorial & Document Design Specialist (AI Evaluation) posted Nov 17
- Senior Luxury Automotive Strategy Leader, AI Expert posted Nov 17
- Senior SWE - Python / TypeScript posted Nov 17
- Software Engineer - Core Contributor posted Nov 17
- AI Engineer posted Nov 17
- Architectural Design & 3D Visualization Expert (AI Training) posted Nov 17
- Board-Certified Anaesthesiologist, AI Expert posted Nov 17
- Board-Certified Cardiologist, AI Expert posted Nov 17
- Board-Certified Dermatologist, AI Expert posted Nov 17
- Board-Certified Radiologist, AI Expert posted Nov 17
- Formal Mathematics & Theorem Proving Specialist (AI Evaluation) posted Nov 17
- Economists - Economic Analysis for AI Training posted Nov 16
- P&C Insurance Professionals: Paid AI Training Project posted Nov 16
- Legal Research Attorney - AI Training posted Nov 16
- Customer Support & Success Domain Expert - AI Model Training posted Nov 14
- Subject Matter Experts: ML Model Evaluation (PhD) posted Nov 13
- Senior Motion Designer (After Effects), AI Expert (Remote, US) posted Nov 12
- Commercial Video Editor (Premiere, Resolve), AI Expert (Remote, US) posted Nov 12
- ML Infrastructure and Performance Engineer, AI Expert (Remote) posted Nov 12
- Board-Certified Neurologist, Medical AI Expert (Remote, US) posted Nov 12
- Board-Certified Orthopedic Surgeon, Medical AI Expert (Remote, US) posted Nov 12
- Electrical Engineer, AI Expert (Full-Time, US) posted Nov 12
- Embedded Systems and Firmware Engineer, AI Expert (Remote, US) posted Nov 12
- Licensed Clinician (MD, NP, PA, RN, PharmD), Healthcare AI Expert (Remote, US) posted Nov 12
- Mechanical Engineer, AI Expert (Full-Time, US) posted Nov 12
- PhD Economist, AI Research Expert (Remote) posted Nov 12
- PhD Genomics and Cell Biology Scientist, AI Expert (Remote, UK or US East Coast) posted Nov 12
- Software Engineer - Backend posted Nov 12
- PhD Mathematician or Statistician, AI Expert (Remote, US) posted Nov 12
- PhD Pharmacologist or Toxicologist (Drug Discovery), AI Expert (Remote UK or US) posted Nov 12
- Game Developer (Godot Expert) posted Nov 12
- Finance Professional from a Top-Tier Firm, AI Expert (Remote, US) posted Nov 12
- PhD Physicist, AI Expert (Remote, US) posted Nov 12
- Electronics Engineer (DesignSpark PCB Specialist) posted Nov 12
- PhD Biologist (Life Sciences), AI Expert (Remote, US) posted Nov 12
- PhD Chemist, AI Expert (Remote, US) posted Nov 12
- PhD Synthetic and Analytical Chemist, AI Expert (Remote, UK or US East Coast) posted Nov 12
- Pod lead - Scientific Computing / Research Engineering Expert - Earth Sciences posted Oct 19
- Senior Software Engineer posted Oct 19
- Senior SWE - Python / TypeScript posted Oct 19
- Agentic Life Science Expert posted Oct 19
- Backend Security Engineer posted Oct 19
- Engineering Expert posted Oct 19
- Sr. Full-Stack Software Engineer posted Oct 19
- Software Engineer (Open Source Contributor) posted Oct 19
- Scientific Computing / Research Engineering Expert - Earth Sciences posted Oct 19
- Business Analyst: US Raters posted Oct 19
- Competitive Data Scientist (Kaggle) posted Oct 19
- Junior Data Scientist posted Oct 19
- Junior Software Engineer posted Oct 19
- Business Analyst - US - Educational Raters - Gemini posted Oct 19
- Business Analyst: Non-US Raters posted Oct 19
- Insurance Expert - AI Training & Evaluation posted Oct 8
- Biology / environmental science Evaluator posted Sep 17
- Biotech Investment Analyst - Drug Asset & Company Assessment posted Sep 17
- Data Scientist Talent Network posted Sep 17
- Equity Research Expert - Single-Name Analyst Publications posted Sep 17
- Engineering & Software Domain Expert posted Sep 17
- Management Consultant Talent Network posted Sep 17
- Multilingual Inpatient Hospitalist (MD) - Clinical Documentation & AI Evaluation posted Sep 17
- Finance Domain Expert - AI Training & Evaluation posted Sep 17
- Analyst / Insights Experts posted Sep 17
- MLOps Engineer, LLM Systems (Serving, GPU Kernels, Profiling) posted Sep 17
- Software Engineer, Go - Codebase Q&A posted Sep 16
- Biology / environmental science Evaluator posted Sep 16
- Applied History & Political Science Benchmark Specialist posted Sep 16
- Software Engineer, Python - Codebase Q&A posted Sep 16
- Software Engineer, C - Codebase Q&A posted Sep 16
- AI Safety Experts - English & Finnish posted Sep 16
- Biology PhD Coding Experts posted Sep 14
- STEM PhDs and Technical Domain Experts posted Sep 14
- Physics PhD Coding Experts posted Sep 14
- Material Science PhD Coding Experts posted Sep 14
- Chemistry PhD Coding Experts posted Sep 14
- Data quality / CRM operations Evaluator posted Sep 14
- Mathematics PhD Coding Experts posted Sep 14
- FP&A & Treasury Analyst posted Sep 14
- Electrical Engineering Expert posted Sep 8
- Technical Accounting & SEC Reporting Specialist posted Sep 8
- Incident management / reliability / SRE Evaluator posted Sep 8
- Engineering / manufacturing / technical operations Evaluator posted Sep 8
- Mechanical Engineering Expert posted Sep 8
- Financial Crime, AML & KYC Analyst posted Sep 8
- Generalist (AI Training Specialist) posted Sep 7
- Compliance / regulatory response with financial-services AI Evaluator posted Sep 2
- Cloud / DevOps Engineer (Infra & IaC) posted Sep 2
- Materials Science Domain Expert posted Sep 2
- Product / Platform Experts posted Sep 2
- Engineering / Platform Professionals posted Sep 1
- Product / Artifacts Experts posted Sep 1
- Applied Computer Science Benchmark Specialist posted Aug 17
- Sales Engineering Expert posted Aug 17
- UI / UX Design Expert posted Aug 17
- Professional Design Experts posted Aug 15
- Project Coordinator - AI & Data Projects posted Aug 15
- Senior Software Engineer, Full Stack (Python, Java, Rust, C#, C++) posted Aug 14
- Legal Domain Expert - AI Training & Evaluation posted Aug 14
- Software Engineer, Full Stack (Python, Java, Rust, C#, C++) posted Aug 14
- Senior Design Expert - Paid AI Design Research Study posted Aug 14
- QA/Test Engineer posted Aug 14
- Biology Expert (PhD) - AI Safety posted Jul 30
- Electrical Engineering Expert posted Jul 30
- Software Engineering Expert posted Jul 30
- Mechanical Engineering Expert posted Jul 30
- Chemical Engineering Expert posted Jul 30
- Machine Learning Engineer - Model Evaluation & Experimentation posted Jul 30
- ASIC/SoC Design & Verification Engineer posted Jul 29
- Data Engineer - Finance posted Jul 29
- MLOps Engineer (JAX, PyTorch, Pallas/Triton) posted Jul 29
- Performance Engineer (C++, Python, Rust) posted Jul 29
- Computer Vision Expert posted Jul 29
- Machine Learning & NLP Expert posted Jul 29
- AI Rater Guidelines Writer (Linguist / Instructional Designer) posted Jul 29
- Data Science & Quantitative Analysis Expert posted Jul 29
- Data Science Expert posted Jul 28
- Architect (CAD & Construction Documentation) posted Jul 14
- Frontend Software Engineer posted Jul 10
- AI Evaluation Specialist posted Jul 9
- Technology AI Evaluation Expert posted Jul 9
- Southern American English Voice Actor (AI Speech & Voice Modeling) posted Jul 9
- Legal AI Evaluation Expert posted Jul 9
- Software Engineering Expert (AI Training) posted Jul 9
- Equity Research AI Specialist posted Jul 8
- Internal Medicine AI Evaluation Specialist posted Jul 8
- Medical Expert AI Evaluation Specialist posted Jul 8
- Game Tools & Modding Engineer posted Jul 3
- Biology Subject Matter Expert (AI Training & Scientific Content) posted Jul 3
- Chemistry Subject Matter Expert - AI Training (Remote) posted Jul 3
- AI Safety & Red Teaming Specialist posted Jul 3
- Medical Subject Matter Expert (AI Training & Documentation) posted Jul 3
- Engineering Subject Matter Expert (AI Training) posted Jul 3
- Computer Science Subject Matter Expert (AI Training) posted Jul 3
- Financial Documentation Expert (AI Training) posted Jul 3
- Physics Subject Matter Expert (AI Training) posted Jul 3
- Mathematics Subject Matter Expert (AI Training) posted Jul 3
- Network Engineer - Data for Autonomous Systems annotation posted Jun 24
- Management & Strategy Consultants (MBB/Big 5) AI Trainer posted Jun 24
- CUDA Engineering Expert posted Jun 24
- Cybersecurity SWE - AI Safety posted Jun 24
- Mechanical Engineer posted Jun 5
- US Tax Law AI Specialist (CPA / EA / CA) posted May 14
- AI Tax Compliance & Legal Interpretation Expert posted May 14
- Physics AI Specialist posted May 5
- Clinical AI Specialist posted May 5
- Investment Banking AI Specialist posted May 5
- Tax AI Expert posted Apr 21
- Capital allocation AI Expert posted Apr 21
- Insurance Underwriting AI Expert posted Apr 21
- Financial modeling AI Expert posted Apr 21
- Compliance and controls AI Expert posted Apr 21
- Insurance Policy Administration AI Expert posted Apr 21
- Risk & Actuarial AI Expert posted Apr 21
- Financial reporting AI Expert posted Apr 21
- Strategic finance AI Expert posted Apr 21
- Financial planning & analysis (FP&A) AI Expert posted Apr 21
- Insurance Claims Management AI Expert posted Apr 21
- Ledger and reconciliation AI Expert posted Apr 21
From the team and dates on Weekday AI's own job board.
Other openings for this role
2 open postings for this role in United States, first posted 2 October 2026. The company is likely hiring for more than one position. None has been taken down and posted again.
- 2 October 2026 – open
- 2 October 2026 – open (this posting)
Similar jobs
- AI/ML Software Engineer, Platform at Falcomm
Shares 5 skills: Test Automation, Git, Machine Learning… · same country - Product & Test Engineering, Intern Summer 2027 at Lightmatter
Shares 3 skills: Test Automation, Git, Python · same country - Data Engineer at Benchling
Shares 4 skills: Test Automation, Machine Learning, LLM… · same country - Site Reliability Engineer, Kubernetes at SpaceX
Shares 3 skills: Test Automation, Git, Python · Junior level · same country - Automation & Controls Engineer, New Grads at General Matter
Shares 3 skills: Test Automation, Git, Python · Junior level · same country - VFX Supervisor / Lead Compositor (Nuke, Flame), AI Expert at Weekday AI
Also at this company
More jobs like this
Is this your posting? Email [email protected] to have it removed.