How a Series-B should evaluate a Fractional Chief AI Officer

A few months ago a Series-B founder I had met once at a North Park dinner sent me a Friday-evening LinkedIn message that read, in full, board says I need an AI executive, can we talk Monday. On the call, she told me she had nine introductions to fractional CAIOs lined up over the next two weeks and no idea what to ask any of them. Could I write her a short list?

What follows is approximately what I wrote her, lightly edited for an audience wider than one. It is not a buyer’s guide in the sense of a checklist with weighted scoring. It is closer to what a director gives an actor when the actor is about to audition someone — a short list of moments to watch for, and what those moments tell you about whether this person can be in the room with you for the next eighteen months.

The basic frame

A fractional Chief AI Officer is a senior executive who you rent for half a day to two days a week, for somewhere between six and eighteen months. They are the person responsible for the AI work that nobody internal yet has the seniority to be responsible for. The problem they solve is not “we don’t know how to use ChatGPT.” The problem they solve is “the board is asking quarterly questions about AI that nobody on the leadership team can answer with the kind of confidence that closes the conversation.”

That framing matters because it disqualifies a category of candidates immediately. A fractional CAIO is not a fractional ML engineer, not a fractional prompt engineer, and not a fractional data scientist. Those are useful hires; they are not the hire you are doing. If a candidate’s strongest answers are about model architectures and their weakest answers are about board decks, you are interviewing the wrong layer of the stack.

The four questions

If the call is going to be one hour, here are four questions to spend it on. They are not trick questions. The point is to listen for the register of the answer, not the answer itself.

1. Tell me about the most recent time you killed an AI project.

Watch for whether they have actually killed one. The answer should be specific — a vendor, a use case, a deadline, a number. If the answer is shaped like “we always validate ROI before greenlighting,” you are listening to a methodology. If the answer is shaped like “we shut down the support-ticket-classifier in week three because the eval set was contaminated and the gain was inside the noise band,” you are listening to an operator. The first answer can be rehearsed. The second is reported.

This question is also the only reliable disqualifier for the candidate who has only ever been on the vendor side of AI procurement. A career on the vendor side teaches you to sell projects, not to kill them, and the fractional CAIO who cannot kill is the fractional CAIO who will burn your budget on the wrong thing for two quarters before the board finally asks the question that ends the engagement.

2. What is the smallest evaluation suite you have ever shipped?

The right answer is some version of “a Google Sheet with 40 prompts and a column for human grade.” The wrong answer is some version of “we use a comprehensive evaluation framework integrating LLM-as-judge with human review at scale.” The first is what a real evaluation suite looks like in week one of a real engagement. The second is what an evaluation suite looks like in a deck.

Real evaluation work is unglamorous. The candidate who treats it as glamorous has not done enough of it. The candidate who can describe the smallest, most embarrassingly low-tech eval suite they have shipped — and tell you what it caught — has done enough of it.

3. Walk me through how you would structure your first ninety days here.

Listen for whether the answer treats the first ninety days as discovery or as deployment. The right answer is a discovery-heavy sequence: stakeholder interviews, an inventory of every model and vendor currently in production (almost always more than the company thinks), a data lineage audit, and a one-page memo at day sixty that names the three things that should be killed, kept, or built. The wrong answer is a deployment-heavy sequence: we’ll launch the customer support copilot in week six.

A fractional CAIO who promises a launch in their first ninety days is promising a launch they have not yet done the work to scope. That promise is reliable evidence that the candidate is selling rather than scoping.

4. When this engagement ends, what does my full-time hire look like?

This question is the one most candidates handle worst, and it is also the one that tells you the most. The right answer names the type of hire — a head of AI engineering vs. a head of AI strategy vs. a head of data; an internal promotion vs. an external hire — and is honest about which they cannot yet tell. The wrong answer is some variant of “oh, I think I’d love to convert to full-time eventually.”

The fractional CAIO whose end-state vision is “convert to full-time” has misread the assignment. The end-state of a good fractional engagement is the company having clarity on what permanent role to hire and the runway to hire it. If the fractional cannot describe that hand-off in concrete terms in the first hour, they are not the fractional you want for an eighteen-month engagement.

What the candidate should be doing

A point of symmetry: the candidate, if they are any good, will be evaluating you on similar terms during the same hour. They will ask whether your board has signed off on AI investment in writing, or only in conversation. They will ask whether your data infrastructure is owned by an internal team or an offshore vendor with a renewing contract. They will ask whether your CFO has a number she expects AI to move, and whether anyone has tested whether AI actually moves it. If the candidate does not ask anything in this register, they are pricing you the same way they price every other client, and the price is wrong for both of you.

This is, again, a director’s note rather than a checklist. A casting decision is made on a hundred small signals, most of which are below the level of language. The four questions above are not the only questions; they are the four that have, in my own engagements, most reliably distinguished the senior practitioner from the senior storyteller.

The senior practitioner is the one who, six months in, has killed at least one of your AI projects, shipped a working version of another, and given you the language to defend both decisions to a board member who reads the Wall Street Journal on Saturday mornings. That is the engagement worth eighteen months. Anything less is, in the end, a slide deck with a salary attached.