Agent Harness

• Design the core agent loop, including planning, reasoning, tool selection and calling, memory, and error recovery, as reusable patterns that generalize across many business workflows.
• Develop context engineering strategies: prompt architecture, context window management, summarization and compaction, memory, and retrieval (RAG) integration.
• Define how agents connect to tools and enterprise systems through the Model Context Protocol (MCP), including tool design, server patterns, and the conventions that make tools reliable for agents to use.
• Design how agents discover, load, and apply skills (packaged instructions, scripts, and resources for specific tasks) so capabilities built once can be reused across many workflows.
• Explore sub-agent and multi-agent patterns, and decide with evidence what belongs in the platform.
• Evaluate new frontier and open-weight models as they are released, and understand their tradeoffs in capability, latency, and cost for platform use.

Evaluation Harness

• Design evaluation methodology: task-level metrics, rubrics, golden datasets, LLM-as-judge graders, and human review, with automated graders calibrated against human judgment.
• Build the eval harness that lets any team define, run, and compare evaluations for their agents, spanning offline benchmarks, simulated users and environments, and online quality monitoring.
• Make eval results trustworthy by accounting for variance, statistical significance, dataset contamination, and grader bias.
• Diagnose failure modes by analyzing traces and eval results, and turn qualitative observations into measurable hypotheses.

Experiment and Iterate

• Run fast, well-designed experiments on prompts, architectures, models, and tools, owning the loop from hypothesis to result to decision.
• Iterate until agent quality meets the bar for production business workflows, and help define what that bar should be.
• Track the research frontier and industry practice, bring the best ideas into GEICO quickly, and filter out the hype.
• Share what you learn through write-ups, internal talks, reusable patterns, and playbooks that raise the level of GenAI practice across the company.

Lead

• Set technical direction for agent design and evaluation across the AI Agent Platform, in partnership with senior AI and engineering leaders.
• Mentor engineers and data scientists on GenAI engineering practice and experimental rigor.
• Partner with platform engineers, product managers, and business teams to turn workflow needs into measurable quality targets and generalized platform capabilities.

How You Work

• Intellectual curiosity. You want to understand why something works, not just whether it does.
• Speed to learn. A new model, paper, protocol, or framework drops, and you're productive with it within days.
• Adaptability. You change direction when the evidence says so, and hold strong opinions loosely.
• Empirical rigor. You trust data over intuition, and your intuition is well calibrated because of it.
• Bias to action. You'd rather build a prototype and measure it than debate it in a meeting.

Minimum Qualifications

• 10+ years of experience in software engineering, ML engineering, or applied science, with strong Python proficiency.
• Tech-lead experience: owned the design and delivery of significant systems involving multiple engineers or teams.
• Significant hands-on experience building GenAI applications and agentic systems with LLMs such as GPT, Claude, Llama, or Qwen, taken beyond prototype into real use.
• Demonstrated experience designing and running experiments to improve LLM or agent performance, including defining metrics, building evaluation sets, analyzing results, and iterating.
• Deep practical understanding of prompt and context engineering, tool calling, RAG, and agent architectures, including hands-on experience with MCP.
• Solid distributed systems fundamentals, including concurrency, fault tolerance, latency and throughput tradeoffs, and API design, with experience building or working closely with production services at scale.
• A track record of quickly learning and applying new techniques in a fast-moving field.
• Demonstrated technical leadership influencing direction across teams.

Preferred Qualifications

• Experience building evaluation frameworks or benchmarks for LLMs or agents, including LLM-as-judge methods and human annotation pipelines, using tools such as Inspect AI, Braintrust, DeepEval, or RAG.
• Experience building MCP servers and tools, or authoring skills using the Agent Skills standard (or similar packaged agent capabilities) used across teams.
• Experience with agent frameworks such as LangGraph, Microsoft Agent Framework, OpenAI Agents SDK, Claude Agent SDK, or Google ADK, and eval and observability tools such as Langfuse, LangSmith, Braintrust, or Arize Phoenix.
• Background in ML or applied science, such as statistics and experimental design, fine-tuning, reinforcement fine-tuning (e.g., RLHF, RLVR), or model evaluation.
• Experience with cloud AI platforms such as Microsoft Foundry (including Azure OpenAI and Foundry Agent Service) or Amazon Bedrock (including AgentCore).
• Publications, open-source contributions, or public writing on GenAI or agents.
• Understanding of AI security (e.g., prompt injection, tool and skill supply chain risks), safety, and responsible AI practices.

Why This Role Is Unique

• Define how agents behave, and how their quality is measured, across one of the largest insurers in the United States.
• Work hands-on with the newest models and techniques as they're released, not years later.
• See your experiments turn directly into scalable platform capabilities used by teams across the enterprise.
• Build AI capabilities that impact 30,000+ associates and help shape AI experiences used by millions of customers.
• Influence AI strategy and architecture across the enterprise, partnering with senior AI and engineering leaders on the next generation of AI at GEICO.

Annual Salary

$115,000.00 - $260,000.00 The above annual salary range is a general guideline. Multiple factors are taken into consideration to arrive at the final hourly rate/ annual salary to be offered to the selected candidate. Factors include, but are not limited to, the scope and responsibilities of the role, the selected candidate’s work experience, education and training, the work location as well as market and business considerations.

GEICO will consider sponsoring a new qualified applicant for employment authorization for this position.

Great Company: Protecting customers through life’s twists and turns with innovation and integrity.

Great Careers: Personalized development programs, mentorship, and certification assistance.

Great Culture: Inclusive and collaborative culture rooted in shared success.

Great Rewards: Competitive pay, benefits, and flexibility to support your well-being and future.

The equal employment opportunity policy of the GEICO Companies provides for a fair and equal employment opportunity for all associates and job applicants regardless of race, color, religious creed, national origin, ancestry, age, gender, pregnancy, sexual orientation, gender identity, marital status, familial status, disability or genetic information, in compliance with applicable federal, state and local law. GEICO hires and promotes individuals solely on the basis of their qualifications for the job to be filled.

GEICO reasonably accommodates qualified individuals with disabilities to enable them to receive equal employment opportunity and/or perform the essential functions of the job, unless the accommodation would impose an undue hardship to the Company. This applies to all applicants and associates. GEICO also provides a work environment in which each associate is able to be productive and work to the best of their ability. We do not condone or tolerate an atmosphere of intimidation or harassment. We expect and require the cooperation of all associates in maintaining an atmosphere free from discrimination and harassment with mutual respect by and for all associates and applicants.