AI Engineer Skills Checklist: What to Learn and in What Order

The core AI engineer skills in 2026, in learning order, are Python and software engineering, how LLMs behave, prompting and structured output, retrieval (RAG), tool calling and agents, evaluations, and production work: security, observability and cost control. Each step below has a checklist, a measurable "done when" test, and a self-score, so you can see exactly where you stand.

This page is the audit tool for our step-by-step guide to becoming an AI engineer. The guide explains the path and the projects; this checklist sets the bar for calling each skill done.

Updated November 2026. Exam outlines and survey figures below are dated and linked to their sources; check them again before you rely on them.

The AI engineer skills checklist at a glance#

The table below is the whole checklist in one view. Work top to bottom, and don't move on until the "done when" test passes for the step you're on.

Step	Skill area	Priority	Done when…
1	Python and software engineering	Required	A teammate can clone, test and deploy your service from the README alone
2	How LLMs behave	Required	You predicted failures on 10 inputs before running them, and at least 7 were right
3	Prompting, structured output, context	Required	20 of 20 test inputs return valid JSON, and your retry path has fired at least once
4	Retrieval (RAG)	Required	You can report how often the right passage is retrieved, for keyword, vector and hybrid search
5	Tool calling and agents	Required	Your agent stops safely when a tool fails or a step limit is hit
6	Evaluations	Required	Evals run in CI and block a change that makes quality worse
7	Security, observability, cost	Required for production	A prompt-injection test passes, and you can show a trace and cost for one request
8	Fine-tuning and ML depth	Optional	You can show, with evals, whether a fine-tune beat your prompt-and-RAG baseline

How this checklist was built#

The skills come from what working AI engineers report and what certification bodies list. The order is our editorial sequence: each step uses what the one before it taught. We don't sell training, and our methodology explains how we stay neutral.

Three kinds of evidence shaped the list:

• Profile data. LinkedIn's Jobs on the Rise 2026 list for the US, published January 2026, draws on the profiles of members who hold the title and names LangChain, Retrieval-Augmented Generation (RAG) and PyTorch as the most common skills for AI engineers, with a median of 3.7 years of prior experience. The same entry notes the role "can also be known as" machine learning engineer, which helps explain PyTorch's place on the list, and why model training is optional Step 8 here.
• Certification outlines. Two current exams publish how much each skill area counts, though each is tied to its vendor's cloud.
• Developer survey data. The Stack Overflow 2025 Developer Survey shows which tools developers building with AI agents use.

The AWS Certified Generative AI Developer – Professional (AIP-C01) exam guide spreads its weight across five domains. Integrating foundation models and their data is the largest; safety, security and testing together make up almost a third.

Source: AWS AIP-C01 exam guide, share of scored content, as of November 2026.

Microsoft's exam points the same way. The AI-103 study guide (skills measured as of April 16, 2026) weights "Implement generative AI and agentic solutions" at 30–35% of the exam. The AI-102 study guide states that exam was retired on June 30, 2026; its final outline split the same work into generative AI (15–20%) and agentic solutions (5–10%).

How to score yourself#

Give each step a score from 0 to 3, and be strict:

Score	Meaning
0	Haven't started
1	Followed a tutorial or course module
2	Built it myself on my own data or problem
3	Passed the "done when" test and can show the artifact

Only a 3 counts as done. A row of 1s means you've watched a lot and built little; go back to the lowest step and build. For how your background changes where to start, see who the AI engineer roadmap is for.

Before you start: prerequisites#

You need a working development setup and the habit of building things, not a degree. Check these first:

• You can use a terminal, Git and a code editor.
• You can read API documentation and make an authenticated HTTP request.
• You have an API key for at least one hosted model, with a spending limit set.
• You have a small, real problem to build toward: your own documents, a team's support questions, a process you repeat weekly.

If you're not sure the role is the right fit, read what an AI engineer does day to day before you commit study time.

Step 1: Python and software engineering#

Python and ordinary software engineering are the foundation every other AI engineer skill sits on. The AI-103 study guide says candidates "should have experience developing apps by using Python," and the AWS AIP-C01 guide expects "2 or more years of experience building production-grade applications."

Python's usage is still growing. The Stack Overflow 2025 survey reports a 7 percentage point increase in Python use from 2024 to 2025, to 57.9% of respondents to its language question.

• Write Python with functions, classes, type hints and virtual environments.
• Build a small HTTP API (FastAPI, Flask or similar) with input validation.
• Write unit tests and run them in CI.
• Store and query data with SQL.
• Deploy a service with Docker or a managed platform, and read its logs.

Done when: someone else can clone your repo, run the tests and deploy the service using only the README.

Step 2: How LLMs behave#

This step is about intuition for model behavior, not the math behind training. Can you look at a task and say in advance where a model is likely to go wrong?

• Explain tokens, context windows and why long inputs cost more.
• Explain what temperature and other sampling settings change.
• Explain embeddings: text turned into vectors so similar meanings sit close together.
• Describe common failure modes: confident wrong answers, ignored instructions, stale knowledge.
• Measure two models on the same 20 inputs for quality, speed and price.

Done when: you wrote down which of 10 inputs a model would get wrong, ran them, and at least 7 predictions held.

Step 3: Prompting, structured output and context#

Prompting is now part of a broader skill: deciding exactly what goes into the model's context. Anthropic's engineering team describes context engineering as "the natural progression of prompt engineering."

• Write system prompts with clear instructions, examples and output formats.
• Get structured output (JSON matching a schema) and validate it in code.
• Handle errors and retries when output fails validation.
• Stream responses to a user interface.
• Start a small test set now: 20 real inputs with expected results.

Done when: all 20 inputs in your test set return valid, schema-checked JSON, and your logs show the retry path firing at least once.

Step 4: Retrieval (RAG)#

Retrieval-augmented generation is how AI engineers make a model answer from company data. Microsoft's Azure AI Search documentation advises: "Use hybrid queries that combine keyword (nonvector) and vector search for maximum recall." Measuring that claim on your own data is a good RAG exercise.

• Split documents into chunks and explain why your chunk size works.
• Generate embeddings and store them in a vector store (pgvector, Chroma or a managed service).
• Run keyword, vector and hybrid search, and compare the results.
• Add reranking and return citations with each answer.
• Handle the case where nothing relevant is found, instead of letting the model guess.

Done when: for 20 real questions with known answers, you can report how often the right passage appears in the top results for keyword, vector and hybrid search.

Step 5: Tool calling and agents#

Tool calling lets a model take actions such as searching, querying a database or calling an API, and agents chain those actions together. Anthropic's guide to building effective agents (December 2024) separates workflows, where tools run "through predefined code paths," from agents, which "dynamically direct their own processes and tool usage." Reach for a workflow first.

Learn frameworks second; LangChain tops LinkedIn's skills list for the role, so they matter. Among developers who use AI agents, 32.9% used LangChain and 16.2% used LangGraph in the past year, according to the Stack Overflow 2025 survey, which also found that "AI agents are not yet mainstream."

• Define tools with clear schemas and handle tool errors.
• Write one tool-calling loop without a framework, then rebuild it with one and note what changed.
• Implement at least two workflow patterns, such as prompt chaining and routing.
• Add limits: maximum steps, timeouts and a human approval step for risky actions.
• Log every step so you can replay a failed run.

Done when: you can break a tool on purpose, or hit the step limit, and the agent stops cleanly with a log that shows why.

Step 6: Evaluations#

Evals turn "it seems better" into a number you can defend. OpenAI's evaluation best practices advise: "Adopt eval-driven development: Evaluate early and often." The AI-103 study guide includes evaluating apps for "fabrications, relevance, quality, and safety," and the AIP-C01 guide gives testing and validation 11% of its exam.

The problem evals catch is common. In the Stack Overflow 2025 survey, 66% of developers named "AI solutions that are almost right, but not quite" as their top frustration with AI tools.

• Version the Step 3 test set, and add every failure you've seen since.
• Spot-check model-graded scores by hand to see whether the grader agrees with you.
• Score retrieval and generation separately, so you know which one broke.
• Wire the eval run into CI.

Favor coverage over polish. Anthropic's guide to developing tests says "more questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals."

Done when: your evals run automatically in CI, and a pull request that lowers quality fails the build.

Step 7: Security, observability and cost#

Production work means knowing how your system can be attacked, what it is doing, and what it costs. The AIP-C01 exam guide gives "AI Safety, Security, and Governance" 20% of the exam. Use the OWASP Top 10 for LLM applications (2025) as your security checklist. It starts with prompt injection and sensitive information disclosure, and includes excessive agency and unbounded consumption.

On observability, the AI-103 study guide asks candidates to "set up observability by implementing tracing, token analytics, safety signals, and latency breakdowns." Familiar tools carry over. Among agent users in the Stack Overflow 2025 survey, 43% used Grafana and Prometheus for agent observability; the survey says developers are "adapting their existing, traditional monitoring tools for this new task."

• Test your app against prompt injection, including instructions hidden in retrieved documents.
• Keep secrets and personal data out of prompts and logs.
• Give agents the minimum permissions they need.
• Trace each request from user input through retrieval, tools and model calls.
• Set per-user rate limits and a budget alert, and report cost per request.

Done when: a document with hidden instructions fails to hijack your app, and you can show the trace and the cost of that request.

Step 8 (optional): Fine-tuning and ML depth#

Fine-tuning and model training are optional for most AI engineer roles, and belong after Step 7, not before. The AWS AIP-C01 guide lists "Model development and training" and "Advanced ML techniques" among job tasks "the target candidate is not expected to be able to perform."

Some ML knowledge still helps. Chip Huyen's AI Engineering companion repository notes that the field involves "parameter-efficient finetuning," and that "since foundation models are ML models, some concepts are relevant to working with both." If the job posts you're targeting ask you to train models at scale, they may be ML engineering roles; our AI engineer vs. ML engineer comparison shows how to tell.

• Explain when fine-tuning is worth it compared with better prompts or retrieval.
• Run one parameter-efficient fine-tune (such as LoRA) on a small open model.
• Compare it against your prompt-and-RAG baseline using your Step 6 evals.

Done when: you can show an eval comparison where the fine-tune beat, or failed to beat, your baseline, and explain why.

How to prove each AI engineer skill#

Employers can't see your checklist, so turn each step into something they can open. One project can cover several steps.

Step	Proof an employer can check
1. Software engineering	A public repo with tests, CI and a README someone else can deploy from
2. LLM behavior	A short write-up: predicted failures vs. what happened
3. Prompting and structured output	The schema, the validation code and the test set
4. RAG	Retrieval hit rates for keyword, vector and hybrid search on your questions
5. Agents	A log or video of a run that hits a failure and stops cleanly
6. Evals	A CI run that failed a regression, and the fix that passed it
7. Security, observability, cost	An injection test, a trace and a cost-per-request figure
8. Fine-tuning	An eval table comparing the fine-tune with your baseline

Common mistakes with this checklist#

• Scoring yourself on effort. A finished course module is a 1, not a 3. Only a passed "done when" test counts.
• Learning the framework instead of the pattern. If you can't describe what LangChain does for you in one tool-calling loop, go back to Step 5.
• Writing evals last. A test set started at Step 3 makes every later step measurable.
• Treating security as a later problem. Prompt injection is first on the OWASP list, and any app that reads untrusted documents is exposed to it.
• Mistaking ML depth for AI engineering depth. For application roles, Step 8 is a specialization, not a gate.

Frequently asked questions#

Do AI engineers need to know math?#

Most application-focused AI engineers need working intuition rather than advanced math. Understanding embeddings, probabilities and evaluation metrics usually covers what Steps 1–7 need. The AWS AIP-C01 guide lists advanced ML techniques as out of scope. Research and model-training roles are different; see our AI engineer vs. ML engineer comparison.

Do AI engineers need to learn PyTorch?#

Not first. PyTorch appears among the most common AI engineer skills in LinkedIn's Jobs on the Rise 2026 data, but that entry also covers machine learning engineer titles. Learn it at Step 8 if your target roles involve fine-tuning or training models; Steps 1–7 don't depend on it.

Is prompt engineering still a skill worth learning?#

Yes, as part of a broader skill. Anthropic's engineering team calls context engineering, which covers everything that goes into the model's input, "the natural progression of prompt engineering." Clear instructions, examples and output formats still matter; retrieval results, tool output and memory now matter too.

Next steps#

Score yourself on all eight steps, then build the "done when" artifact for your lowest-scoring required step this week. When you're ready for the projects around this checklist, return to the AI engineering path, step by step.

If you want structured help, our guide to choosing AI engineer training compares program types and prices, and you can compare AI engineering programs by curriculum, price and format, and read graduate reviews checked under our review guidelines before you enroll. Check whether a program's syllabus covers evals, security and deployment, not only prompting and RAG.