As our Senior AI Quality Engineer, your job is to ensure that the solutions we ship are high quality by methodically testing them and working with engineering to implement improvements. This requires a test and verify mindset; you don’t assume things are working, you understand the shape of the system and test it from every angle to ensure quality. During planning phases, you explore system architecture, write test plans, and align with the team on what “quality” looks like. As systems are built, you red-team them, code scripts to automate and scale testing processes, implement logging, and methodically analyze the results. You think like an AI agent, stress-testing the behavior of AI systems under a wide range of conditions beyond the happy-path.

This is a new role at Addition, created because the complexity of the AI systems we ship has outgrown traditional QA practices and ad-hoc evaluation. We need someone who turns quality into a new discipline for the AI systems era — evolving fuzzy notions of “quality” into measurable signals, and bringing the rigor of a structured adversarial mindset. This role sits within our engineering team, working closely with engineers, PMs, cross-functional project teams, and clients.

Primary Responsibilities

• Test Planning: During project planning phases, translate product and technical requirements into structured test plans — covering functional behavior, edge cases, failure modes, and AI-specific risks like hallucination, prompt injection, and output drift.
• Agentic Testing: Working with coding agents like Claude Code and local or online versions of AI System builds to write test scripts, setup logging frameworks, analyze agent behavior, user behavior, to identify points of failure and opportunities for improvement. This is automated and streamlined for scale, but always done with intentionality, human supervision, and verification along the way.
• AI Red Teaming: When systems are built, systematically red-team them before launch. Probe LLM behaviors, agent decision chains, brand voice outputs, and multi-model pipelines for failure modes that functional testing won't surface. Assess the agent’s overall trajectory, tool selection and reasoning steps. Design prompts that attempt to manipulate behavior, discover conflicting instructions and surface failure cases before they reach production.
• Launch Gating: Act as the quality gate on launches — raising blockers, documenting failure cases, and holding the line on ship criteria. You are the friction layer between "built" and "live."
• Regression & Evaluation Frameworks: Build and maintain lightweight eval frameworks and test harnesses for recurring system types (e.g., brand voice agents, agentic creative tools, generative UIs) to systematize and improve efficiency, speed and cost of QA and red teaming practices over time.
• Cross-Project Quality Standards: Work across our project portfolio to develop shared quality standards for AI systems — what "good enough to ship" actually means for agentic products.
• Tracing and Documentation: Document failure patterns, red team findings, and test results in a form that engineers and PMs can act on. When something fails, trace tool interactions across the reasoning steps to pinpoint how the input propagated through the agent's logic — so a finding points at a root cause, not just a symptom.

The Ideal Person

• Problem Solver & Adaptable: Thrives in fast-paced, high-ambiguity environments and turns complex challenges into structured, elegant solutions. Brings a genuinely adversarial mindset—finding failure interesting, not frustrating.
• High EQ & Collaborative: A cross-functional team player who builds strong, trusting relationships with internal teams, C-suite executives, and enterprise clients.
• Authentic Communicator: Exceptional written and verbal communication skills, able to translate complex technical/AI concepts into clear, actionable, and persuasive ideas. Provides strong written communication so findings are actionable for engineers and legible to PMs and clients.
• Curious & Forward-Thinking: Genuinely passionate about frontier AI, continuous learning, and exploring applied innovation.

• Experience: 4+ years of software QA, test engineering, or quality engineering experience, with at least 1–2 years working on AI-powered or LLM-based systems.
• Technical & Industry Literacy:
• Deep familiarity with failure modes specific to generative AI systems (hallucination, prompt injection, context window degradation, inconsistent persona adherence, retrieval errors, agent loop failures, non-determinism, temperature, retrieval mechanics, context windows, grounding, and context adherence).
• Ability to read code and understand system architecture well enough to write meaningful test plans and probe the right seams without needing to redesign the system.
• Experience writing structured test plans and evaluation criteria, including trajectory-level assessment of multi-step reasoning and judgment (understanding when to evaluate per span vs per trace vs per session).
• Execution & Track Record: Hands-on fluency with AI development tools, agent frameworks, and prompt engineering, with the ability to speak the same language as the engineers whose work is being tested.
• Tooling Proficiency: Comfort operating with modern collaboration tools (e.g., Notion, Google Workspace, Figma/FigJam, Slack) and AI capabilities.

• Background working in an agency, studio, or consultancy where projects run fast and quality standards need to travel across diverse client work.
• Experience red-teaming production AI agents (not just evaluating model outputs in isolation) and familiarity with agentic frameworks (Claude Agent SDK, LangGraph, AutoGen, or similar) and multi-step agent pipelines.
• Experience building lightweight eval harnesses or automated regression pipelines for LLM outputs, including agent-as-evaluator setups, tracing/observability for agent runs, and debugging tool/MCP integrations.
• Prior work in brand safety, content moderation, or AI safety adjacent to production systems.

Why Join Addition?

• Shape the Future: Lead and innovate at an AI-powered studio at the forefront of applied AI innovation.
• High-Impact Work: Build and drive technological and creative solutions that redefine how global brands operate.
• Direct Leadership Access: Collaborate closely with our CEO, leadership team, and R/GA network to help define how we scale.

NYC / Los Angeles / San Francisco Area

This role is based in the NYC / Tri-State area, Greater Los Angeles area, or San Francisco Bay Area. Candidates must be located within commuting distance in the Tri-State area (NY, NJ, CT), Southern California, or the Bay Area, or willing to relocate. This position offers a hybrid work model, with regular in-person collaboration and team / client workshops.

The hiring range for this position is $135,000 to $150,000. Where an employee or prospective employee is paid within this range will depend on, among other factors, actual ranges for current/former employees in the subject position; budgetary considerations; tenure and standing with the company (applicable to current employees); as well as the employee’s/applicant’s applicable skills, pertinent experience, and qualifications.