• Completes assigned features and bug fixes independently, with limited need for day-to-day guidance
• Designs components within a well-defined scope; escalates questions on complex system design rather than guessing
• Participates in code review and provides constructive feedback
• Surfaces blockers proactively rather than waiting for check-ins

Craft & Professionalism

• Writes tests that cover the functionality they ship
• Monitors and responds to issues with their own work
• Documents decisions and implementation details that others will need later

Agent Tracing & Explainability

• Understands how to instrument an agent so a single conversation can be followed from the first user message through every model call, tool call, retrieval, agent handoff, and decision to the final response
• Builds the correlation that ties actions across multiple agents in one workflow into a single readable trace
• Turns raw trace data into an explanation a human can follow: what the agent did, what it relied on, and why it chose that path

Customer-Facing Trust & Explanation

• Understands that the explanation an engineer needs and the explanation a borrower or loan officer needs are different, and builds for both
• Works with product engineers on what an agent should disclose at the product surface: what it did, what data it used, how confident it was, and what a human should verify
• Applies judgment about the right level of explanation for a regulated lending and account-opening context

Trust & Explainability Frameworks

• Familiar with the open-source landscape for LLM observability, tracing, and evaluation, and can evaluate a framework against the platform's needs
• Integrates and extends existing frameworks rather than rebuilding them, and builds what does not exist yet
• Keeps the platform's instrumentation aligned to emerging standards so traces stay portable across tools

Agent Evaluation & Quality

• Understands how to measure the quality of non-deterministic output: golden datasets, rubric and LLM-as-judge scoring, and regression baselines
• Builds evaluation checks that run in CI so a model swap, prompt change, or tool change is tested before it reaches customers
• Treats evaluation data as versioned, reviewed code

Safety & Tenant Isolation

• Knows the common attack patterns against LLM applications (prompt injection, jailbreaks, tool misuse, data exfiltration) and how to write tests for them
• Understands tenant isolation as a platform guarantee and builds tests that prove one customer's agent can never see another customer's data
• Contributes to the shared guardrail layer so every agent inherits protection without re-implementing it

What Success Looks Like

In the first few months, a successful hire has shipped tracing that follows a full agent workflow end to end, readable by an engineer who did not write it, and has contributed working components to the platform's evaluation harness. Over the first year, success looks like this: a first version of the customer-facing explanation primitive is live in at least one product, evaluation and isolation checks run automatically on every agent release, and when an engineer asks why an agent did something, or a customer asks whether to trust it, the platform answers. Engineers who thrive here like the intersection of AI and rigor, want to become the team's go-to on agent tracing and explainability, and are motivated by making AI something people can see into and trust.

Key Responsibilities

Multi-Agent Tracing & Explainability

• Build tracing across the platform's gateway, orchestration, memory, and tool layers, following defined designs from the team lead and architects
• Build the correlation that links actions across agents in a single multi-agent workflow, including handoffs, parallel branches, and retries
• Build the developer-facing trace view that makes a full agent conversation readable to someone who did not write the agent
• Build the explanation layer that turns trace data into a human-readable account of why an agent did what it did

Customer-Facing Trust

• Build the platform primitives product teams surface to end users: explanation records, confidence and provenance metadata, and "what the agent relied on" summaries
• Partner with product engineers on the Document Request Agent and the MLM agents to land those primitives in real products
• Iterate on the explanation format based on feedback from product teams and customer-facing staff

Trust & Explainability Frameworks

• Evaluate and integrate open-source observability, tracing, and evaluation frameworks into the platform runtime
• Extend those frameworks where our agent workloads need more than they offer, and contribute fixes and extensions back upstream where it makes sense
• Build the gaps: the pieces of trust and explainability tooling the ecosystem does not provide yet

Evaluation Tooling

• Build and maintain components of the platform's evaluation framework: golden dataset management, test runners, scoring pipelines, and regression reporting
• Build the tooling that helps teams create and version golden datasets from de-identified real traffic and synthetic cases
• Run model and prompt comparisons for the platform's shared components and report what changed

Safety & Isolation Testing

• Build red-team and adversarial test suites covering prompt injection, jailbreak attempts, tool misuse, and data exfiltration, and run them in the platform's release process
• Build the automated test suite that proves agents on the shared runtime cannot cross tenant boundaries through memory, retrieval, tool calls, or model context
• Share red-team findings and new attack patterns with Security Operations and pull their threat models into the platform's tests

Collaboration & Growing Others

• Participate in design discussions and code reviews; give and receive feedback constructively
• Support onboarding of L1 AI Engineer teammates; share context and help them get unblocked
• Contribute to documentation that reduces tribal knowledge on the team

Required Experience

• 3+ years of professional software engineering experience, delivering features independently in a production environment
• Solid understanding of algorithms, data structures, and software design fundamentals
• Proficiency with standard development tooling: Git, Docker, automated testing, and modern scripting languages
• Active daily use of AI-assisted development tools
• Bachelor's degree in Computer Science, Software Engineering, or equivalent experience
• Hands-on experience building software that integrates large language models (LLM APIs, agent frameworks, RAG pipelines, or similar), in production or in substantial personal or open-source projects
• Proficiency in Python with production experience; TypeScript a plus
• Hands-on experience in at least one of the following, with real interest in growing into the others: LLM observability and tracing, LLM evaluation and testing, agent frameworks and multi-agent orchestration, or application security testing
• Experience with distributed tracing or observability tooling in a production system
• Strong automated testing instincts, including an interest in how to test systems that do not return the same answer twice
• Experience running workloads on Azure or AWS, including the basics of identity and access management, networking, and secrets management

Preferred Qualifications

• Experience with OpenTelemetry, including the GenAI semantic conventions, or OpenLLMetry
• Experience with LLM observability and evaluation tools (Langfuse, Arize Phoenix, LangSmith, Braintrust, promptfoo, DeepEval, or equivalent)
• Contributions to open-source AI observability, evaluation, or agent framework projects
• Experience with cloud-managed model services such as AWS Bedrock or Azure OpenAI
• Experience building or operating multi-tenant SaaS systems where tenant isolation was a hard requirement
• Prior experience in financial services, fintech, or another regulated industry where explainability shaped technical decisions
• Experience building developer-facing debugging or visualization tools