AI Test Architect

Job Description

What You’ll Be Doing

1.AI-Driven Quality Strategy & Architecture 

Architect a comprehensive "Quality Intelligence" platform using generative AI to predict defect hotspots, intelligently optimize regression suites, auto-generate tests, and enable self-healing automation. 

Define enterprise-wide AI-first testing strategy, including non-deterministic evaluation paradigms, continuous monitoring for drift/hallucination, and integration across the full SDLC. 

Establish governance for ethical AI testing, aligning with emerging standards 

2.LLM & Agent Evaluation Frameworks 

Design and implement advanced benchmarks, red teaming protocols, and adversarial testing for internal AI agents and generative features—focusing on hallucination rates, bias/fairness, prompt injection, jailbreaks, and goal misalignment. 

Build evaluation pipelines with statistical rigor (e.g., multi-trial runs, LLM-as-judge, human-in-the-loop) using tools like LangFuse, LangSmith, DeepEval, RAGAS, or Arize Phoenix for metrics such as faithfulness, context precision, and safety compliance. 

Architect harnesses for agentic workflows, tool-calling, planning, multi-agent simulations, and post-deployment observability. 

3.Infrastructure & Automation Architecture 

Partner with DevOps to embed AI-based testing into GitHub-based CI/CD pipelines (e.g., AI-generated tests, predictive flakiness detection, automated gating with quality signals). 

Lead design of self-healing test frameworks (integrating AI plugins with Playwright/Cypress or similar) that adapt to UI/model changes with minimal maintenance. 

Architect synthetic data generation, maintain golden data-sets and AI-powered data masking solutions to enable privacy-compliant, high-fidelity testing at scale. 

4.Cross-Functional Leadership & Evangelism 

Collaborate with product, data science, ML engineering, and security teams to influence AI feature design with quality guardrails from day one. 

Evangelize and mentor: Upskill traditional QA engineers into AI-augmented testers through workshops, playbooks, and communities of practice. 

Drive adoption of AI quality best practices organization-wide, including metrics dashboards for DORA + AI-specific indicators (e.g., hallucination rate, red team success rate, self-healing coverage). 

5.Observability, Metrics & Continuous Evolution 

Define and implement AI-specific quality telemetry (e.g., drift detection, faithfulness scoring, compliance incidents) integrated with tools like Langfuse. 

Establish feedback loops for model iteration, A/B testing guardrails, and proactive risk mitigation in production. 

Challenges You'll Architect Solutions For

Building reliable evaluation for non-deterministic, agentic AI in a fast-moving SaaS landscape. 

Scaling self-healing and generative test automation without introducing new flakiness or security debt. 

Balancing innovation speed with rigorous red teaming and ethical safeguards for customer-facing AI. 

Success in the First 6-12 Months

Launch the "Quality Intelligence" platform foundation with AI-augmented pipelines covering > 70%+ of critical paths. 

Establish red teaming/red-teaming-as-code processes that reduce high-severity AI risks by > 40%+. 

Upskill > 50%+ of QA/engineering teams on AI testing fundamentals and deliver measurable velocity/safety gains. 

Accuracy Baseline: Establish a baseline 90%+ Faithfulness score for all RAG-powered features. 

What You Will Bring

8+ years in Quality Engineering/Test Architecture within cloud-native SaaS environments, with 2+ years focused on AI/ML/LLM testing and validation. 

Deep expertise in AWS (serverless, microservices, IaC with Terraform/CloudFormation) and GitHub CI/CD ecosystems. 

Proficiency architecting LLM-based applications and testing frameworks (LangChain/LangGraph/LangSmith strongly preferred; equivalents acceptable). 

Mastery of modern automation (Playwright, Cypress) with hands-on experience integrating self-healing AI plugins or generative test tools. 

Strong programming skills in JavaScript/TypeScript and/or Python; solid understanding of foundational AI concepts (transformers, embeddings, RAG, evaluation trade-offs). 

Experience with LLM evaluation tools like Bedrock Evaluations, Prompt Management, Guardrails, DeepEval, RAGAS, Arize Phoenix, Langfuse. 

Experience with Red teaming frameworks/tools (Cobalt Strike, Sliver, Nmap) and knowledge of adversarial testing methodologies is a bonus. 

Proven leadership: Mentoring teams, defining standards, and driving cross-functional change in ambiguous, high-growth settings. 

Bachelor's/Master's in Computer Science, AI/ML, or equivalent; relevant certifications a strong plus. 

Strong English language communication and collaboration skills

Perks & Benefits

Contrato a termino Indefinido with all the legal benefits

Prepaid Medicine

Life insurance and funeral assistance

Internet allowance

Home office stipend

Competitive compensation — above the market average

100% remote work environment and an excellent work-life balance

Opportunity to work for a growing global SaaS leader company

A culture that promotes independence, innovation, trust, and accountability

Open space to be creative, innovative, and strategize for the future

Mentorship by a highly experienced professional 

Budget for training, we want you to grow

5 Personal Time Off days per year

Sick Leave Top up to total 100% of salary paid by the employer from Day 3 to 90. 

Recognition Award, additional paid time off in recognition of the corresponding year of service

Upgrade vacation starting at 5 years of service