45+ Interview Questions and Answers for AI Engineers (2026 Guide)

Introduction

A complete AI engineer interview guide: LLM concepts, inference infrastructure, RAG, agent design, model evaluation, hands-on coding, and strategy. All questions are collected from real interviews at OpenAI, Anthropic, Scale AI, Sierra, xAI, Databricks, Perplexity, and other companies.

Top 10 most common questions (with company)

1. Design a batching system for single-GPU inference that handles up to 100 requests per batch while users wait synchronously. (Anthropic)
2. Design an end-to-end batching system for LLM requests. (Anthropic)
3. Design an insurance-claims agent that accepts applications and decides approval using RAG, with token-budget and cost controls. (Scale AI)
4. Explain how RAG works. (Sierra)
5. Build a support agent for a hypothetical company that sells outdoor gear. (Sierra)
6. Investigate and fix a model that gives confident but factually wrong answers in high-risk scenarios. (Anthropic, ML Engineer)
7. Implement a GPU credit management system. (OpenAI)
8. Write an image-processing pipeline that applies transforms specified in per-image instruction files. (Anthropic)
9. Design a strategy for OpenAI’s fine-tuning service. (OpenAI)
10. How do you approach generative AI safety in consumer products? (OpenAI, Anthropic, Google)

LLM and generative AI concepts

These questions check whether you understand what happens under the hood of an API call.

Real-world example: Sierra often asks candidates to explain RAG from scratch.

Questions

• Explain how RAG works. (Sierra)
• Which generative AI skills will be critical next year? (Anthropic)
• What is a tokenizer, and why does token count affect cost and context limits?
• What is an embedding, and what does “semantic similarity” mean in a retrieval system?
• What is a context window, and what breaks when you exceed it?
• What does the temperature parameter control, and when would you set it near zero?
• What is quantization, and what do you pay for when you use it?
• Prompting, RAG, or fine-tuning: how do you choose?

Serving infrastructure and LLM inference

This is the most distinctive category for AI engineers. The questions are surprisingly consistent. You are expected to batch requests for maximum GPU utilization without destroying latency.

Quirk: Anthropic asks these questions in interviews for SWEs, ML engineers, research engineers, and even managers.

Questions

• Design a single-GPU batching system (up to 100 inputs/batch, synchronous users). (Anthropic)
• Design an end-to-end batching system for LLM requests. (Anthropic)
• Design an API for efficient sampling from large models, with solid batching and request orchestration. (Anthropic)
• Design a system to support ML models in production. (Scale AI)
• Implement a GPU credit management system. (OpenAI)
• Design OpenAI Playground. (OpenAI)
• Design a document-processing pipeline that indexes large volumes of heterogeneous documents for LLM use. (Databricks)
• Design a platform for tracking and analyzing ML experiments. (Google)

RAG systems and agent design

Agent design is now a standard question, especially at companies whose product is an agent (Sierra, Scale, Glean). Interviewers value guardrails, cost control, and human-in-the-loop.

Questions

• Design an insurance-claims agent that uses RAG and controls token cost. (Scale AI)
• Build a support agent for an outdoor gear company. (Sierra)
• Design an agentic AI system for Spotify support. (Sierra)
• Design an AI agent for a streaming service. (Sierra)
• Design an AI system that integrates with a third-party company’s data and business processes. (Sierra)
• Have you built end-to-end agentic systems? (Glean)
• Design a file uploader for an AI chat that handles large multimodal files and places them in the model context. (xAI)
• Design a real-time voice AI application.

Model behavior, data, and evaluation

In ML engineering loops they check whether you can diagnose model problems and work with messy data — not just call an API.

Questions

• A deployed conversational model gives confident but wrong answers in risky scenarios. How do you investigate and fix it? (Anthropic, ML Engineer)
• Extract and clean a dataset from the company’s demo database using only SQL and Python. (Anthropic, ML Engineer)
• How do you evaluate an LLM feature when there is no single correct answer?
• What is LLM-as-a-judge, and what are its weaknesses?
• How do you catch regressions when you change a prompt or model?

Hands-on programming (AI-assisted coding)

Coding rounds at these companies look like mini-projects: ship a working feature, fix someone else’s code, often with an AI assistant allowed.

Questions

• Write an image-processing pipeline that reads instructions from files and applies transforms to a large image set. (Anthropic)
• Debug and extend Python code by adding async retries with exponential backoff and timeouts. (Sierra)
• Traverse cell dependencies in an Excel-like spreadsheet and detect circular references. (Sierra)
• Scan a filesystem and find duplicate files. (Anthropic)
• Implement an in-memory key-value store with set, begin, commit, and abort (transactions). (OpenAI, xAI)
• Given credit grant and spend events with expiry, compute the remaining balance. (OpenAI)
• Implement encoding and decoding of a list of strings. (OpenAI)

Strategy, safety, and behavioral questions

Even in engineering loops, AI companies probe judgment — especially ethics and responsibility. Anthropic is particularly known for these rounds.

Questions

• Design a strategy for OpenAI’s fine-tuning service using public documentation. (OpenAI)
• How would you define success metrics for an AI feature or product? (Perplexity)
• How do you approach generative AI safety in consumer products? (OpenAI, Anthropic, Google)
• Tell me about a time you were confident in a decision and later realized you were wrong. (Anthropic)
• Tell me about a time you had to do something that conflicted with your values. (Anthropic)
• Why do you want to work at Anthropic? (Anthropic)
• Which recent AI advances interest you? (Anthropic)

Answer frameworks (how to structure)

1. Framework for RAG / agent design

2. Framework for LLM serving / infrastructure

3. STAR (for behavioral)

Frequently asked questions

How to prepare (3-week plan)

• Week 1: Foundations (RAG, embeddings, transformer architecture, batching).
• Week 2: System design (agent design, cost control, load balancing).
• Week 3: Hands-on coding + STAR stories and behavioral answers.

Glossary (key interview terms):

• Batching: Grouping requests for efficient GPU utilization.
• Continuous Batching: Dynamically adding/removing requests from a batch during generation.
• KV Cache: Caching keys and values to speed up generation (avoid recomputing them for already processed tokens).
• Guardrails: Rules and filters that control model output (safety, format, escalation).
• Eval Set: A gold-standard dataset for measuring model quality.
• LLM-as-a-Judge: Using another LLM to score the primary model’s answers.
• Structured Output: Generating answers in a strict format (e.g., JSON) for system integration.