Machine learning engineer screening questions: production models, evaluation and LLM work
On this page
- ML engineer, data scientist, research scientist or AI engineer: place the candidate
- One model in production, end to end
- Evaluation and experiments: where depth shows
- LLM and generative AI work: separating the layers
- How ML resumes overstate the work
- Certifications, publications and portfolios
- Logistics, knockout checklist and scorecard
- Questions people ask
Machine learning engineer screening questions should find out whether the candidate's models reached production and stayed useful there, not how many algorithms they can name. Ask for one model end to end: the business problem, the metric, how they evaluated it before launch, how it was served, how they monitored it, and what happened when performance dropped. For large language model work, ask whether they trained or fine-tuned anything or built on hosted models, and how they measured quality. Engineers with real experience talk about leakage they caught, latency budgets and a model rolled back. Inflated resumes list PyTorch, TensorFlow, LLMs and "built AI solutions" with no metric anywhere.
This bank is narrower than software engineer phone screen questions, which already has one row on AI claims, and different from data engineer phone screen questions, which covers the pipelines that feed models. Here the focus is modeling, evaluation, serving and the newer LLM application work that now shares the title.
ML engineer, data scientist, research scientist or AI engineer: place the candidate
| Role in practice | What the work produces | Question that places them | Common mismatch |
|---|---|---|---|
| Machine learning engineer | Training pipelines, model serving, monitoring, retraining, feature pipelines | "Which of your models ran in production, and who deployed them?" | Notebook modeling handed to someone else to ship |
| Data scientist | Analysis, experiments, prototype models, stakeholder recommendations | "When a model worked, what happened next, and whose job was that?" | Analyst with a few regressions, titled data scientist |
| Research scientist | New methods, papers, benchmarks, large training runs | "What did you publish or ship, and what was your contribution?" | Coursework or a thesis described as research experience |
| AI or LLM application engineer | Features built on hosted models: retrieval, prompts, tool use, evaluation, guardrails | "Did you train or fine-tune models, or build on an API?" | A single API call presented as ML engineering |
| MLOps or ML platform engineer | Tooling others use: feature stores, model registry, training infrastructure | "Did you build the platform, or train models on it?" | Model builder described as platform owner |
One model in production, end to end
| Prompt | What a strong answer sounds like | Red flags |
|---|---|---|
| What problem did the model solve, and for whom? | A business outcome: fraud flagged before payout, delivery time estimates, search ranking, support ticket routing. | Starts with the algorithm and never reaches a user. |
| What metric did you optimize, and why that one? | A metric tied to the problem, such as precision at a fixed alert volume, with a trade-off explained. | "Accuracy," on a problem where one class is rare. |
| What did the model replace, and how much better was it? | A baseline (rules, a previous model, human process) and the improvement measured against it. | No baseline. |
| Where did the training data come from, and what was wrong with it? | Named sources, labeling method, and a real data problem: leakage, missing labels, a shift over time. | Clean data from the start. |
| How was it served: batch, real time, on device? | The method, a latency or throughput figure, and the infrastructure (SageMaker, Vertex AI, a Kubernetes service, a batch job). | Does not know how predictions reached users. |
| How did you know it kept working after launch? | Monitoring of inputs and outcomes, alerts, a retraining schedule or trigger, and a time it degraded. | Deployed and never checked again. |
Evaluation and experiments: where depth shows
Good offline numbers are easy to produce and often wrong. The best ML engineers are suspicious of their own results. These questions show whether the candidate is.
- "Tell me about a model that looked great offline and disappointed in production." Strong: a specific cause such as leakage, a training period unlike live traffic, or a feedback loop, and what they changed. Red flag: has never had this happen.
- "How did you split data for evaluation, and why?" Strong: a reason tied to the problem, such as a time-based split for forecasting or grouping by customer to avoid the same customer in both sets. Red flag: always a random split, never questioned.
- "Did you run an online test before full launch?" Strong: an A/B test or staged rollout, the metric watched and the decision made. Red flag: shipped to everyone on offline numbers alone.
- "What did you do when stakeholders wanted a result the data did not support?" Strong: a calm story of showing the evidence and proposing an alternative. Red flag: adjusted the analysis until it agreed.
- "How did you check the model for unfair outcomes across groups, if that applied?" Strong: a method they used and what they found, for models affecting people (credit, hiring, pricing). Red flag: never considered it for a model making decisions about people.
LLM and generative AI work: separating the layers
Nearly every ML resume written since 2023 mentions large language models. The claims range from pre-training models to adding a chat box that calls a hosted API. Both can be valuable; they are different jobs. Place the candidate on this list and check it against the role.
| Layer | What it sounds like | Follow-up question |
|---|---|---|
| Prompting a hosted model | "Built an assistant with the OpenAI or Anthropic API" | "How did you measure whether answers were good, and what changed when they were not?" |
| Retrieval-augmented generation (RAG) | "Built RAG over our documentation" | "How did you chunk and index documents, and how did you test retrieval quality separately from the answers?" |
| Agents and tool use | "Built agentic workflows" | "What tools could it call, what went wrong in production, and what limits did you add?" |
| Fine-tuning | "Fine-tuned an open model" | "Which base model, how many examples, what method, and what did it beat?" |
| Training or pre-training | "Trained large models" | "How many GPUs, how long a run, and what was your part of the training stack?" |
Strong LLM candidates, at any layer, can answer these:
- "What did a request cost, and what was the latency?" A rough figure and something they did to reduce either.
- "How did you build your evaluation set?" Real examples, graded by people or by rules, rerun on every change.
- "What was the worst output in production, and what did you change?" A specific failure: a wrong answer stated confidently, a leaked instruction, an unsafe action.
- "How did you handle sensitive data sent to the model?" Redaction, contracts with the provider, or self-hosting, with a reason.
How ML resumes overstate the work
| Pattern | What it looks like | How to check |
|---|---|---|
| Notebook as production | "Developed models improving retention 20%" | "Did it run in the product? Who deployed it, and how was the 20 percent measured?" |
| Framework list | PyTorch, TensorFlow, JAX, scikit-learn, XGBoost, Hugging Face | "Which did you use every week last year, and for what?" |
| API call as AI engineering | "Built generative AI platform" | "Which parts did you build beyond calling the model?" |
| Competition as work | Kaggle rankings listed with job experience | "Was that at work or in a competition?" Competitions are good; labeling matters. |
| Team paper as personal | Middle author on a paper, described as leading the research | "What was your contribution to that paper?" |
| Uplift with no baseline | "Improved accuracy 35%" | "From what, to what, on which data, compared with what?" |
Certifications, publications and portfolios
| Certification | Issuer | Notes, as of September 2026 |
|---|---|---|
| AWS Certified Machine Learning Engineer – Associate | Amazon Web Services | Building and operating ML on AWS. AWS is replacing the MLA-C01 exam with MLA-C02, with the last English sitting of MLA-C01 on September 28, 2026, per AWS. Certifications earned on either version stay valid for their full term. |
| Professional Machine Learning Engineer | Google Cloud | Framing, building, deploying and monitoring ML on Google Cloud; Google recommends three or more years of industry experience, per Google Cloud. |
For research-heavy roles, publications matter more than certifications; ask for the candidate's part in each paper. For early-career candidates, a public repository or project write-up helps, as long as you ask what they built themselves versus followed from a tutorial. For experienced engineers, most production work is confidential, so rely on the end-to-end story rather than asking for code.
Logistics, knockout checklist and scorecard
| Question | What a strong answer sounds like | Red flags |
|---|---|---|
| The process includes an ML system design round and a coding round. Are you comfortable with both? | Yes, or an objection you can raise with the client early. | Declines any technical assessment for a client that requires one. |
| What total compensation do you need, and how do you weigh base against equity? | A base number and a view on equity. See salary expectation questions. | Only a figure from a salary website. |
| Where else are you interviewing, and how far along? | Stages and dates. | "Nowhere," from a strong candidate in a busy market. |
| Are you legally authorized to work in the US, and will you now or in the future need visa sponsorship? | A direct answer to both. | None. Ask every candidate the same two questions. |
Knock out, or flag to the client before submitting, if:
- The role ships models to production and the candidate has never had a model in production.
- The candidate cannot name the metric, baseline and evaluation method for their main project.
- The role requires training or fine-tuning and their LLM work is API-only, or the reverse where the client needs product engineering.
- A client must-have (a cloud ML platform, a serving stack, a domain such as ranking or forecasting) is missing entirely.
| Area | 1 | 2 | 3 | 4 |
|---|---|---|---|---|
| Production ownership | Notebooks only | Handed models to others | Deployed and monitored models | Designed the serving and retraining approach |
| Evaluation rigor | Accuracy only | Sensible metrics, no baseline | Baseline, correct splits, online test | Caught leakage or failures, changed practice |
| LLM work (if required) | Buzzwords | API prototypes | Production features with evaluation sets | Fine-tuning or systems with cost and safety controls |
| Communication | Jargon, no outcomes | Clear with prompting | Explains trade-offs to a non-specialist | Clear, concise, and ties work to business results |
| Logistics fit | Deal-breaker | Two open questions | One open question | All aligned |
Record the metric and the failure story in the candidate's words: "fraud model, precision at 500 alerts a day; 'it fell apart when we launched in a new country, so we added a region feature and retrained monthly'" tells a hiring manager exactly where the candidate sits. Interview Signal attaches quotes like that to each score from the call.
Questions people ask
What is the difference between a machine learning engineer and a data scientist?
Titles overlap heavily. A data scientist usually explores data, builds models and explains results, often in notebooks. A machine learning engineer usually gets models into production and keeps them working: pipelines, serving, monitoring and retraining. Ask whether the candidate's models ran in a live product and who deployed them.
Is 'AI engineer' the same as machine learning engineer?
Not always. Many AI engineer roles build applications on top of large language models through APIs, with retrieval, prompts and evaluation, and do little model training. Others are ML engineering under a new name. Ask whether the candidate trained or fine-tuned models, or built on hosted models, and screen for what the client's role actually needs.
How can a recruiter check machine learning depth without an ML background?
Ask for one model that reached production: the business problem, the metric they optimized, how they evaluated it before launch, how it was served, and what happened when its performance dropped. People with real experience have a specific story about data leakage, drift or a model that looked good offline and failed live.
Do machine learning certifications matter?
They matter less than production work, but they show structured knowledge of a cloud platform. AWS and Google Cloud both offer machine learning engineer certifications. Check the issuer's badge for the expiry date, and note that AWS is moving its associate exam to a new version in 2026.