QA engineer phone screen questions: manual testing, automation frameworks and flaky tests
On this page
- Manual tester, automation engineer or SDET: place the candidate
- Test design and manual testing questions
- Automation framework ownership: built it, extended it or ran it
- Flaky tests, CI and release gates
- How QA resumes overstate the work
- Certifications and technical exercises
- Logistics and knockout checklist
- Scorecard
- Questions people ask
A QA engineer phone screen should establish which kind of tester the candidate is: someone who tests by hand, someone who adds automated tests to an existing framework, or someone who builds and owns the framework and its place in the delivery pipeline. Ask what share of their week was writing code, how many automated tests they maintained, how long the suite took to run, and what they did about the last flaky test. Strong candidates answer with numbers and a failure story. Inflated resumes list Selenium, Cypress and Playwright with no test they can describe.
This is different from software engineer phone screen questions, which tests engineers who build the product. Here the product is confidence that the release works: test design, automation ownership and how bugs get found and reported. For contract QA roles, add the rate and availability questions from IT contractor screening questions.
Manual tester, automation engineer or SDET: place the candidate
"QA engineer" covers three jobs with different pay, different interview loops and different deal-breakers. Place the candidate in the first five minutes so the rest of the screen asks the right questions.
| Role in practice | What the week produces | Question that places them | Common mismatch |
|---|---|---|---|
| Manual or functional tester | Test cases, exploratory sessions, bug reports, regression passes | "How many test cases did you run by hand in a release, and where did they live?" | Recorded a few scripts once, listed as automation |
| Automation engineer | New automated tests inside an existing framework, fixes to broken tests | "Who built the framework, and what did you add to it?" | Maintained tests, described as "designed the framework" |
| SDET | Framework code, test tooling, test data services, pipeline stages | "What share of your week was writing code, and in which language?" | Title SDET, work mostly manual regression |
| QA lead or test manager | Test strategy, release sign-off, a team of testers | "How many testers, and who decided whether a release shipped?" | Senior tester who coordinated, described as managing |
| Performance or security tester | Load tests, results analysis, vulnerability scans | "Which tool, what load, and what did you find?" | Ran one load test script written by someone else |
Test design and manual testing questions
Even automation-heavy roles need someone who can decide what to test. These questions show whether the candidate thinks about risk or just follows a script.
| Question | What a strong answer sounds like | Red flags |
|---|---|---|
| A new feature lands on Friday for a Monday release. How do you decide what to test first? | Starts from risk: what changed, what users touch most, what broke last time, what cannot be rolled back (payments, data migrations). | "I test everything." Or waits for someone to hand them test cases. |
| Tell me about a bug you found that nobody expected. | A specific defect and how they got there: an edge case in dates or time zones, a permission combination, an interrupted upload. | Only obvious bugs a user would hit on the first click. |
| What goes into a good bug report? | Steps to reproduce, expected versus actual result, environment and build, evidence (logs, screenshots), severity and why. | "A title and a screenshot." |
| Tell me about a bug a developer rejected. What happened? | Calm disagreement resolved with evidence or by checking the requirement with the product owner. | Frames developers as the enemy, or dropped it without checking. |
| Where did your test cases live, and how did you keep them current? | Names the tool (TestRail, Xray or Zephyr in Jira, a spreadsheet) and a habit for retiring stale cases. | Hundreds of cases nobody had reviewed in years. |
| What did you test that was not in the requirements? | Exploratory testing with a note of what they covered: accessibility, slow networks, browser back button, concurrent edits. | Nothing outside the written acceptance criteria. |
Automation framework ownership: built it, extended it or ran it
You do not need to know Playwright from Cypress. You need to know where the candidate sits on this scale for the framework the client uses, and whether they can talk about it in their own words.
- "Who set up the framework, and what did it look like when you joined?" Strong: an honest answer with detail, such as "it existed, but I moved it from Selenium to Playwright and rewrote the page objects." Red flag: claims to have built it but cannot say what came before.
- "Roughly how many automated tests did you maintain, at which levels?" Strong: numbers by layer, for example a few hundred API tests and a smaller set of browser tests. Red flag: "a lot", or only browser tests with no view on why.
- "How long did the full suite take, and what did you do to speed it up?" Strong: a before and after time and the change: running in parallel, moving checks from the UI to the API, better test data. Red flag: does not know how long it ran.
- "Which language did you write tests in, and did developers review your code?" Strong: a language (Java, TypeScript, Python, C#) and a code review habit. Red flag: no review, no version control, tests edited on a shared machine.
- "How did tests get their data: seeded, created through the API, or shared accounts?" Strong: a deliberate approach and a problem it caused. Red flag: every test used the same login, and they never questioned it.
- "What did you choose not to automate, and why?" Strong: a reasoned list, such as one-off checks, visual layout judged by eye, or features about to change. Red flag: aims for 100 percent automation of everything.
Flaky tests, CI and release gates
Flaky tests, the ones that pass and fail without any code change, are where automation experience shows. Anyone who has maintained a real suite has fought them.
| Question | What a strong answer sounds like | Red flags |
|---|---|---|
| Tell me about the last flaky test you fixed. | The cause and the fix: a fixed wait replaced with waiting for an element, shared test data, test order dependence, a slow third-party service mocked out. | "We just re-run the pipeline until it goes green." |
| Where did the tests run, and what triggered them? | Names the CI tool (GitHub Actions, GitLab CI, Jenkins, Azure DevOps) and when suites ran: every pull request, nightly, before release. | Ran tests on their own laptop and emailed results. |
| Could a failing test block a release? Who could override it? | A clear gate and a clear owner, with an example of an override and what followed. | Tests ran but nobody looked at failures. |
| How did you report test results to the team? | A report or dashboard people used, and a habit of triaging failures the same day. | Results in a folder no one opened. |
| Did you test APIs directly? With what? | Postman, REST Assured, Playwright's request context, pytest with requests, and what the tests checked beyond status codes. | Only checked that a call returned 200. |
How QA resumes overstate the work
| Pattern | What it looks like | How to check on the call |
|---|---|---|
| Tool list as experience | Selenium, Cypress, Playwright, Appium, JMeter, Postman, all at once | "Which one did you use every day in your last role?" Ask only about that one. |
| Recorder as automation | "Automated regression" using a record-and-playback tool | "Did you write the test code, or record steps?" |
| Framework tenant as author | "Designed a hybrid framework" | "What existed before you, and what exactly did you add?" |
| Coverage claims | "Achieved 90% automation coverage" | "90 percent of what: test cases, code lines, user journeys? Measured how?" |
| Team numbers as personal | "Reduced regression time from 5 days to 4 hours" | "What was your part, and who else worked on it?" |
| Mobile claim without devices | "Mobile testing" with no device story | "Real devices, emulators or a device cloud? Which OS versions?" |
| Performance testing by association | "JMeter" in a skills list | "What load did you simulate, and what did the results change?" |
Certifications and technical exercises
QA certifications are asked for more often in Europe, government contracts and consulting firms than in US product companies. Know the common ones so you can answer a client's question.
| Certification | Issuer | What it shows |
|---|---|---|
| Certified Tester Foundation Level (CTFL) v4.0 | ISTQB, delivered through national boards and exam providers | Testing vocabulary and techniques. A 40-question, 60-minute multiple-choice exam, per ISTQB. It is the entry point to the other ISTQB certifications. |
| Certified Tester Advanced Level Test Automation Engineering (CTAL-TAE) v2.0 | ISTQB | Designing and running test automation, including in CI/CD, per ISTQB. A knowledge exam, not a coding test. |
Neither certification tests whether someone can write maintainable automation. Treat them as a plus for roles that ask for them, and rely on the framework and flaky-test questions for everything else. If the client uses a technical exercise, find out which kind before you submit:
- Test design exercise: write test cases or a test plan for a small feature. Fits manual and lead roles.
- Automation exercise: automate a few scenarios against a sample app in the client's framework. Fits automation and SDET roles. Ask the candidate whether they have written tests in that framework, not just the language.
- Live pairing: debug a failing test with an engineer. Tell the candidate it is coming; some strong testers perform poorly when surprised by it.
How to evaluate a take-home assignment covers keeping exercises short and scoring them consistently.
Logistics and knockout checklist
| Question | What a strong answer sounds like | Red flags |
|---|---|---|
| Releases go out Thursday evening and QA signs off. Does that schedule work? | A clear yes or a stated limit. | Surprised by any release-night work in a role that includes it. |
| The team is two testers for twelve developers. How have you handled a ratio like that? | Describes prioritizing by risk and getting developers to own unit and API tests. | Expects a large QA team to share the load. |
| What salary or rate do you need, and on what basis? | A number and whether it is salary, W2 hourly or corp-to-corp. See salary expectation questions. | Only a number from a salary website. |
| Are you legally authorized to work in the US, and will you now or in the future need visa sponsorship? | A direct answer to both. | None. Ask every candidate the same two questions. |
Must-ask on every QA screen
- Which role in practice: manual, automation, SDET, lead, performance.
- The share of the week spent writing code, and the language.
- Framework history: built, extended or used.
- Suite size, run time and where it ran.
- The last flaky test and how they fixed it.
- A bug they found that nobody expected.
- Release schedule, compensation basis, notice, other processes, work authorization.
Knock out, or flag to the client before submitting, if:
- The role is automation or SDET and the candidate has never written test code under version control.
- The client's framework is a must-have and the candidate has only recorded tests with a playback tool.
- The role owns release sign-off and the candidate has never had authority to block a release.
- The candidate cannot describe one bug report they wrote in detail.
Scorecard
| Area | 1 | 2 | 3 | 4 |
|---|---|---|---|---|
| Test design | Follows given scripts | Writes cases from requirements | Prioritizes by risk, explores beyond requirements | Sets test strategy others follow |
| Automation | None or recorded only | Adds tests to an existing framework | Maintains the framework, fixes flakiness | Built or restructured it, with measured results |
| Pipeline and release | Runs tests locally | Tests run in CI | Owns a release gate and triage | Designed the gates and reporting |
| Bug reporting and collaboration | Vague reports | Reproducible reports | Resolves disagreements with evidence | Shifts testing earlier with developers |
| Logistics fit | Deal-breaker | Two open questions | One open question | All aligned |
Write the evidence beside each score in the candidate's words: "moved 400 browser tests to API level; 'suite went from two hours to 25 minutes'" tells an engineering manager more than "strong automation skills". Interview Signal attaches quotes like that to each score from the call, so the evidence is written when you hang up. Ask about the schedule itself rather than family or caregiving when you discuss release nights; the EEOC notes that pre-employment inquiries pointing at protected characteristics can be used as evidence of discrimination.
Questions people ask
What is the difference between a QA engineer and an SDET?
Companies use the titles loosely. A QA engineer may test mostly by hand, write automation, or both. An SDET (software development engineer in test) is usually expected to write production-quality code: frameworks, test tooling and pipeline integration. Ask what share of the candidate's week was writing code.
Is an ISTQB certification required for QA jobs?
Rarely in the US, more often in Europe and in consulting or regulated work. The Foundation Level shows the candidate knows testing vocabulary and techniques. It does not show they can build or maintain automation, so ask about the framework they worked in either way.
How can a non-technical recruiter check automation experience?
Ask whether the candidate built the framework or added tests to one someone else built, how many tests ran and how long the run took, and what they did about the last flaky test. People who maintain automation daily have a specific flaky-test story.
Should QA candidates get a take-home test?
Many clients use one: write test cases for a small feature, or automate a few scenarios against a sample app. Keep it short and tell the candidate early, because experienced testers drop out of long unpaid exercises.