Lead QA automation for AI/LLM systems: define validation criteria for non-deterministic models, build automated and performance tests, validate RAG systems, perform API and integration testing, ensure release readiness and trace defects to data/models/requirements.
N-iX is looking for a Senior QA Automation Engineer to lead the validation and verification strategies for EDGE Group’s AI transformation. You will be responsible for defining "what good looks like" for non-deterministic AI systems, ensuring that Large Language Models (LLMs) and predictive engines meet the strict reliability standards of the defense sector.
Qualifications
- 5+ years in QA Automation
- Experience testing data-driven systems, AI, or ML models
- Strong analytical skills for defining test criteria in non-deterministic systems
- Ability to collaborate with engineers, data scientists, and product teams
- Experience in regulated industries (defense, aerospace) will be a plus
- English level - at least Upper-Intermediate
Technical Requirements
- Strong experience in Python and test automation frameworks (Pytest, Selenium/Playwright, API testing)
- Experience with performance testing tools (JMeter, Locust, K6)
- Familiarity with LLM evaluation tools (e.g., DeepEval, TruLens)
- Knowledge of SQL and data validation tools
- Experience with CI/CD pipelines (GitLab) and version control (Git)
- Exposure to test reporting tools (Allure, TestRail)
Key Responsibilities1. AI & LLM Testing
- Build automated tests to evaluate AI outputs for accuracy, consistency, and reliability
- Validate Retrieval-Augmented Generation (RAG) systems using metrics like relevance and correctness
- Create regression tests to detect changes in AI behavior (prompt drift)
- Test integrations between AI systems and enterprise platforms (e.g., SAP)
- Perform API testing, including security and access control validation
- Design performance tests for latency, scalability, and system throughput
- Align test cases with system requirements and user needs
- Support release readiness through structured testing reviews
- Track and manage defects, ensuring traceability to data, models, or requirements
We offer*:
- Flexible working format - remote, office-based or flexible
- A competitive salary and good compensation package
- Personalized career growth
- Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
- Active tech communities with regular knowledge sharing
- Education reimbursement
- Memorable anniversary presents
- Corporate events and team buildings
- Other location-specific benefits
*not applicable for freelancers
Similar Jobs
Artificial Intelligence • Productivity • Software • Automation
Lead Okta administration and SSO, build automations with Okta Workflows and Zapier, optimize SaaS spend and licensing, automate provisioning/deprovisioning via APIs, support and improve the macOS employee experience, triage IT issues, and manage SaaS tooling and reporting across the organization.
Top Skills:
1PasswordAPIsGoogle WorkspaceJIRAmacOSOktaOkta WorkflowsSlackSsoZapierZoom
Artificial Intelligence • Productivity • Software • Automation
As a Staff Engineer for Revenue, you'll shape technical vision and architecture for billing and pricing systems, ensuring correctness while enhancing cross-team collaboration.
Top Skills:
APIsBilling SystemsPerformance OptimizationSubscription Management
Artificial Intelligence • Cybersecurity
As a Senior SRE, ensure reliability and performance of cloud infrastructure, manage incident response, implement monitoring, and drive continuous improvements.
Top Skills:
ArgocdAws EksElk StackGithub ActionsGrafanaKubernetesOpsgeniePagerdutyPrometheusTerraform
What you need to know about the Colorado Tech Scene
With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.
Key Facts About Colorado Tech
- Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3
- Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech
- Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures
- Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute
