The job description is a spec written before the job

6. How do you measure the ability to ask?

"We are looking for people who can ask questions" is easy to say, but you can't put that sentence in an ad, because it can't be measured. What can't be measured is ignored by a hiring process. So this is the most practical section of the essay: making Layer 3 as concrete as Layer 1.

Layer 3 had two halves: asking and understanding. Understanding itself splits in two: understanding something unfamiliar and noticing something wrong. The three formats below measure those three parts in turn:

These three situations make up almost the whole day of an engineer who works with AI. The request arrives incomplete; the generated code often uses a library or a pattern they don't know; and some of that code is fluent but wrong. A skill list measures none of these, because all three happen outside what the candidate already knows.

6.1 The common skeleton

Before the details, four principles tie all three formats together.

You measure the trace, not the answer. Where the candidate ends up is secondary. What is measured is the trace they leave on the way: which questions they asked and in which order, what they assumed, and whether they said what they assumed. So the candidate is invited to think aloud from the start, and the interviewer notes the process, not the result.

The hidden context card. In every format the interviewer holds a card the candidate doesn't see: the answers to the incomplete spec, the code's real behaviour, the location of the error in the AI answer. The card reveals only what the candidate asks for. This is a model of real life: in organizations the information usually exists, but it lives in someone's head and only comes out if asked for.

The interviewer is an information source, not a jury. The interviewer's job is not to pressure the candidate but to answer every question honestly and only as far as it was asked. Extra hints corrupt the measurement; dodging a question punishes the candidate. The role itself is a skill that has to be taught; the formats don't work without interviewer training.

The rubric is written in advance and anchored to behaviour. "Asked good questions" is not a score. "Asked whether the deletion is reversible" is a score. Each format's rubric is tied to the items on the hidden card and to the three-rung questioning ladder I describe in section 8: accept → question → choose what is worth asking.

6.2 The incomplete spec interview

Setup. The candidate gets a one-sentence task that looks ordinary. The sentence is deliberately incomplete, but the gap isn't obvious. The test of a good incomplete spec: on first reading it should make you think "what's hard about this?", and on second reading at least five separate mines should surface. The answers to those mines are on the interviewer's card.

Example.

On the surface, this is a cron job and a single DELETE. The weak candidate writes exactly that:

DELETE FROM users
WHERE deleted_at < now() - interval '30 days';

The strong candidate asks before writing code. Their first question is usually: "What does 'delete' mean here? Can it be undone?" That single question reveals that this is not a DELETE but a data lifecycle problem. The questions that follow open the card item by item.

Here is the striking part: opening the whole card requires no special knowledge. You don't need to know Article 17 of the GDPR by heart, including its exception for data that must be kept to comply with a legal obligation. What you need is curiosity about how many different things "the user's data" might refer to. In this format Layer 1 is almost useless; Layer 3 is everything.

What is measured?

• How many mines were opened by a question? Every mine passed over by assumption is a future production incident.
• The order of the questions. The strong candidate asks about meaning first ("what does delete mean?"), then scope ("which data?"), and implementation last ("what time should it run?"). The order shows how they grasp the problem.
• Assumptions spoken aloud. Under time pressure you can't ask everything. The strong candidate states what they didn't ask: "I'm assuming backups are out of scope, but that needs its own conversation." A spoken assumption is a decision; an unspoken one is a defect.
• Knowing when to stop. A candidate who asks questions forever also fails. Telling that enough is now clear and the rest can be settled during the work is the top rung of the ladder: choosing what is worth asking.

6.3 Reading unfamiliar code

Setup. The candidate gets 10–15 lines of code in a language that isn't on their CV and that they have probably never seen. If they do know it, the code is swapped. The question is simple: "What does this do?" Layer 1 has been deliberately reset to zero. What remains is the ability to make sense of something unfamiliar by likening it to familiar things, and to see where the likeness breaks.

Example. A candidate with a JavaScript, Java or Python background is given this Erlang code:

loop(Balance) ->
    receive
        {deposit, Amount, From} ->
            From ! {ok, Balance + Amount},
            loop(Balance + Amount);
        {withdraw, Amount, From} when Amount =< Balance ->
            From ! {ok, Balance - Amount},
            loop(Balance - Amount);
        {withdraw, _Amount, From} ->
            From ! {error, insufficient_funds},
            loop(Balance)
    end.

The first round is the reading round. A strong candidate leaves a trace like this:

Three things stand out in that trace. The candidate builds every inference on an analogy (switch, send, loop). They mark the inference they aren't sure of as not sure. And they note the place where the analogy breaks, the absence of mutation, as an oddity. Noticing that oddity is the most valuable moment of the reading.

The second round is the striking one. The interviewer asks:

For a candidate with a Java or Python background, the reflex answer is a race condition: two threads read the balance at once, both see enough, the account goes negative; so you need a lock. There is no lock in the code. The candidate either proposes adding one, or stalls.

The real answer on the card: there is no race condition. This function is the body of an Erlang process. The process takes messages from its mailbox one at a time; even if two withdraw messages arrive at the same moment, they are handled in turn. There is no lock because there is no shared variable. Safety comes from the architecture, not from a lock.

The candidate can't know this; their Layer 1 has been reset. What is measured is not whether they know it but whether they can get there by asking. The strong candidate suspends the reflex and asks: "Can this code run on more than one thread at once? Does receive take messages one at a time?" When that question is asked, the card opens, and the candidate sees that their reflex was an assumption from another world.

The third round is for the top of the ladder:

The card has two answers. First: an unrecognised message, say {balance, From}, matches no branch and stays in the mailbox forever. The mailbox grows silently. Second: nothing checks that Amount is positive for a withdraw. Withdrawing a negative amount increases the balance. Finding the second answer needs no Erlang at all; it only needs asking about the gap between what the code intends and what it does.

What is measured?

• The quality of the analogy. Can the candidate tie the unknown to the known? While doing so, can they see where the tie is weak?
• Marking uncertainty. Not every inference is stated with the same confidence. A candidate who can separate "this is certain", "this is a guess" and "I need to check this" will make the same separation when working with AI output.
• Suspending the reflex. Can they question an assumption from their own world (you need a lock) before carrying it into a new one? This is the format's most discriminating measure.
• Knowing what to look up. A variation: tell the candidate "you can now ask an LLM or the documentation exactly one question; what would it be?" The quality of that question summarises the quality of the reading. "How does receive work?" is a middling question. "Does an Erlang process handle the messages in its mailbox one at a time or concurrently?" is a much better one, because it goes like a needle to the single critical unknown.

6.4 Auditing AI output

Setup. The candidate is shown a question and the answer an LLM gave to it. The answer is fluent and confident, the code runs and passes the tests; but it contains an error that will blow up silently in production. The framing: "A teammate sent you this and is about to merge it. What do you say?"

The choice of error is the heart of the format. It mustn't be a syntax error; the compiler finds those anyway. It should be the kind of error where the code's correctness depends on something outside the code: the data, the schema, time, scale.

Example. The question and the answer:

SELECT id, email
FROM customers
WHERE id NOT IN (SELECT customer_id FROM orders);

Everything in the answer looks right. The index advice is sensible, the query is readable, and it returns the correct result on the test database.

The error on the card: orders.customer_id is nullable, because six months ago the company launched guest checkout. Guest orders have no customer_id. In SQL's three-valued logic, x NOT IN (1, 2, NULL) is never true; at best it is unknown. If the subquery returns a single NULL, the query returns no rows at all.

No error. No warning. An empty list. The marketing team uses the query to send a campaign to "customers who have never ordered"; the list is empty; nobody notices anything went wrong; the campaign reaches no one. It is a small-scale rehearsal of the silent failure I describe in section 7.

The answer also contains a second, shallower problem: the claim that it is "generally more performant" is unfounded. PostgreSQL can turn NOT EXISTS into an anti-join, and in most cases it does at least as well. This problem is a deliberately planted decoy: a shallow audit stops here, corrects the performance claim, and approves the answer. The real error stays underneath.

This example separates the three rungs of the ladder in section 8 almost exactly:

• The candidate who accepts: "Looks clean, the index advice is good too. Approved." The answer's confidence becomes the candidate's confidence.
• The candidate who questions: catches the performance claim, or remembers that NOT IN misbehaves with NULLs and proposes NOT EXISTS. They have found the error; but they found it as a rule.
• The candidate who chooses what is worth asking: asks first, "Is customer_id nullable? Is there any case where an order has no customer?" When the answer comes, they find the error, but they don't stop there: "This query's correctness depends on the schema. The schema changed six months ago and nobody noticed. Who is going to notice the next change?" Besides fixing the query, they propose a test containing an order with a NULL customer. They protect not the query but the condition under which it can be wrong.

Notice the difference between these three responses. The second candidate used Layer 1: they knew a rule. The third used Layer 3: even without the rule they would have reached the same place with the right question, and once there they drew the lesson not from the error itself but from how the error was possible.

What is measured?

• Where did they start looking? In the syntax, the logic, or the assumptions outside the code? The strong candidate starts with "what does this code assume?"
• Resistance to confidence. Fluent, self-assured text pushes the reader towards approval. How well the candidate holds out against that pressure is the best indicator of how reliable an auditor they will be when working with AI.
• Did they stop at the decoy? A candidate who ends the audit at the first problem found will stop at the first problem found in real life too.
• Fix, or protect? Fixing the error is one rung; making its recurrence impossible or visible is the rung above.

6.5 Design risks

These formats have weaknesses too; applying them without knowing those weaknesses can do worse than a skill list.

Leakage. Once the formats are known, candidates prepare. That is not a problem but the point: a candidate who prepares by building the habit of asking first in an incomplete spec has already acquired the skill being sought. The real problem is leakage of content. The NOT IN example burned the moment it was published, as it has in this essay. So the format stays fixed and the content rotates. A good team extracts a new example from its own production incidents every quarter. There is no better interview material than your own mistakes.

Interviewer load. An ATS can check a skill list. Only a trained human can run these formats. An interviewer who doesn't know the card well, gives too many hints, or can't tolerate silence corrupts the measurement. This is the formats' cost, and a genuine source of the resistance I describe in section 7.

Anxiety and language. Thinking aloud is not equally easy for everyone. A candidate whose first language differs from the interview's, or who freezes under pressure, can look weak in Layer 3 while being strong. The mitigation is to prepare the candidate for the format in advance and to start with a warm-up question. The format doesn't need to be a surprise; only the content does.

Calibration. Whether different interviewers using the same card give the same candidate the same score has to be measured. If they don't, the problem is not the candidate but the rubric.

All three formats are structured, and that is no accident. Asked without structure, "does this person ask good questions?" turns into the interviewer choosing the candidate they like. That is why structured interviews came out on top in both 1998 and 2022, and why unstructured interviews fell to .19 in the revision. Measuring Layer 3 takes more discipline than measuring Layer 1, not less.