Senior Automation Tester in Python (for Gen AI powered applications)

Job description

We are looking for a Senior Automation Tester in Python to join Production Pods within the program. This is not a manual/functional QA role. You will build and maintain automated test frameworks for AI/GenAI-powered clinical applications, validate non-deterministic AI outputs, and ensure GxP compliance through automated IQ/OQ/PQ qualification. You will work across multiple use cases in Safety, Regulatory, Clinical Operations, and SAGE platform tooling.

Responsibilities

- Build automated test frameworks from scratch for AI-powered clinical applications using Playwright, Selenium, and pytest

- Design and execute test strategies for non-deterministic AI/GenAI outputs, including accuracy, RAG relevance, hallucination checks, and prompt regression

- Integrate automated tests into CI/CD pipelines with quality gates such as block-on-failure and coverage thresholds

- Execute GxP-compliant IQ/OQ/PQ scripts and generate qualification evidence packages

- Manage AI-specific test data, including edge cases, adversarial inputs, and golden datasets

- Monitor model drift, regression, and performance degradation over time

- Produce test strategy documents, test plans, and compliance-ready test evidence

- Collaborate with Data Scientists and Backend Developers to define testable acceptance criteria for AI features

Requirements

- 5+ years of hands-on QA/test automation experience

- Expertise in building and maintaining automated test suites from scratch using Playwright, Selenium, Cypress, and pytest

- Skills in API and backend test automation, including REST APIs, microservices, and serverless architectures with Postman/Newman and pytest + requests

- Proficiency in Python as the primary automation language, with the ability to write harnesses, fixtures, and utilities independently

- Knowledge of CI/CD test integration, embedding automated tests into pipelines such as Jenkins and GitHub Actions with quality gates

- Background in testing AI/GenAI applications, validating non-deterministic AI/LLM outputs for accuracy, RAG relevance, hallucination, and prompt regression

- Familiarity with AI/ML model testing, including accuracy, bias, and performance validation using RAGAS, LLM-as-judge, and golden datasets

- English proficiency at B2 level or higher

Nice to have

- Background in pharmaceutical / life sciences domain, with GxP awareness and IQ/OQ/PQ qualification

- Experience with Jira/Xray, TestRail, or Zephyr test management

- Skills in performance/load testing using JMeter and Locust

- Understanding of security testing basics such as OWASP

- Familiarity with AWS services including Lambda, API Gateway, S3, and CloudWatch