At Pfizer, technology accelerates every facet of our mission. Within the AI Center of Excellence, the AI Quality Engineering team embeds trust, quality, and responsibility into every AI solution we deliver.

As a Staff QA Engineer, you will lead the design and operation of enterprise‑grade evaluation frameworks, guardrails, and continuous quality pipelines that keep our models performant and safe. Your work enables teams across the company to ship AI solutions with confidence—meeting rigorous standards for accuracy, fairness, security, and compliance while driving measurable impact for patients and colleagues alike.

description

• Define & lead enterprise AI evaluation frameworks for model performance, robustness, fairness, bias, and safety across LLMs, traditional ML, and agentic AI systems.
• Architect and operate pre‑release & continuous evaluation (batch + online) integrated with CI/CD, including dataset versioning, synthetic test generation, and production quality gates.
• Design, implement, and maintain guardrail services for hallucinations, prompt injection, PII leakage, unsafe content, and policy violations; instrument telemetry for detection, alerting, and drift tracking.
• Run structured red‑teaming programs (jailbreaks, data exfiltration, adversarial prompts) and drive measurable mitigation with engineering teams; publish risk findings and remediation plans.
• Encode quality & compliance policies as automated checks, partnering with Enterprise Architecture and Compliance to meet internal standards and emerging regulations (e.g., EU AI Act).

For a full list of responsibilities, please click apply. 

profile

• Advanced degree in Computer Science, Engineering, Data Science, or related field.
• 7+ years in Software QA/AI QA/Quality Engineering, including ownership of test strategy for distributed or data‑intensive systems
• Demonstrated experience building evaluation pipelines for ML/LLM systems: design of datasets/benchmarks, metrics for accuracy/robustness, and failure‑mode analysis (e.g., hallucination, bias, toxicity).
• Strong fluency in software engineering best practices (automated testing, version control, CI/CD) and observability (logging, metrics, tracing) for model‑centric applications.

For the full list of requirements, please click apply.

Breakthroughs that change patients' lives... At Pfizer we are a patient centric company, guided by our four values: courage, joy, equity and excellence. Our breakthrough culture lends itself to our dedication to transforming millions of lives.

Work Location Assignment: Hybrid