How a Series-B should evaluate a Fractional Chief AI Officer

A few months ago a Series-B founder I had met once, at a North Park dinner, sent me a Friday-evening LinkedIn message that read, in full: board says I need an AI executive, can we talk Monday. On the call she told me she had nine introductions to fractional CAIOs lined up over the next two weeks and no idea what to ask any of them. Could I write her a short list?

What follows is roughly what I wrote her, lightly edited for an audience wider than one. It is not a buyer's guide in the checklist-with-weighted-scoring sense. It is closer to what a director hands an actor before the actor auditions someone — a few moments to watch for, and what each moment tells you about whether this person can sit in the room with you for the next eighteen months.

The basic frame

A fractional Chief AI Officer is a senior executive you rent for half a day to two days a week, somewhere between six and eighteen months. They own the AI work that nobody internal yet has the seniority to own. The problem they solve is not "we don't know how to use ChatGPT." The problem they solve is "the board asks quarterly questions about AI that nobody on the leadership team can answer with the kind of confidence that closes the conversation."

That framing disqualifies a whole category of candidate on contact. A fractional CAIO is not a fractional ML engineer, not a fractional prompt engineer, not a fractional data scientist. Those are useful hires. They are not the hire you are doing. If a candidate's strongest answers are about model architectures and their weakest answers are about board decks, you are interviewing the wrong layer of the stack.

The four questions

If the call is one hour, here are four questions to spend it on. None of them is a trick. Listen for the register of the answer, not the answer itself.

1. Tell me about the most recent time you killed an AI project.

Watch for whether they have actually killed one. The answer should be specific — a vendor, a use case, a deadline, a number. If it sounds like "we always validate ROI before greenlighting," you are listening to a methodology. If it sounds like "we shut down the support-ticket classifier in week three because the eval set was contaminated and the gain sat inside the noise band," you are listening to an operator. The first answer can be rehearsed. The second is reported.

This is also the only reliable disqualifier for the candidate who has only ever stood on the vendor side of AI procurement. The vendor side teaches you to sell projects, not to kill them. The fractional CAIO who cannot kill is the one who will burn your budget on the wrong thing for two quarters, until the board finally asks the question that ends the engagement.

2. What is the smallest evaluation suite you have ever shipped?

The right answer is some version of "a Google Sheet with 40 prompts and a column for human grade." The wrong answer is some version of "we use a comprehensive evaluation framework integrating LLM-as-judge with human review at scale." The first is what an evaluation suite looks like in week one of a real engagement. The second is what it looks like in a deck.

Real evaluation work is unglamorous. The candidate who treats it as glamorous has not done enough of it. The candidate who can describe the smallest, most embarrassingly low-tech eval suite they ever shipped — and tell you what it caught — has.

3. Walk me through how you would structure your first ninety days here.

Listen for whether they treat the first ninety days as discovery or as deployment. The right answer is discovery-heavy: stakeholder interviews, an inventory of every model and vendor currently in production (almost always more than the company thinks), a data-lineage audit, and a one-page memo at day sixty naming the three things to kill, keep, or build. The wrong answer is deployment-heavy: we'll launch the customer-support copilot in week six.

A fractional CAIO who promises a launch in the first ninety days is promising a launch they have not yet done the work to scope. The promise is reliable evidence that the candidate is selling rather than scoping.

4. When this engagement ends, what does my full-time hire look like?

Most candidates handle this one worst, and it tells you the most. The right answer names the type of hire — head of AI engineering versus head of AI strategy versus head of data; internal promotion versus external search — and is honest about what they cannot yet tell. The wrong answer is some variant of "oh, I think I'd love to convert to full-time eventually."

The fractional CAIO whose end-state is "convert to full-time" has misread the assignment. The end-state of a good fractional engagement is a company with clarity on which permanent role to hire and the runway to hire it. If the candidate cannot describe that hand-off in concrete terms inside the first hour, they are not the fractional you want for eighteen months.

What the candidate should be doing

A point of symmetry: a good candidate is evaluating you on the same terms during the same hour. They will ask whether your board has signed off on AI investment in writing, or only in conversation. Whether your data infrastructure is owned by an internal team or by an offshore vendor on a renewing contract. Whether your CFO has a number she expects AI to move, and whether anyone has tested whether AI actually moves it. If the candidate asks nothing in this register, they are pricing you the way they price every other client, and the price is wrong for both of you.

This is, again, a director's note rather than a checklist. A casting decision rests on a hundred small signals, most of them below the level of language. The four questions above are not the only ones. They are the four that, in my own engagements, have most reliably separated the senior practitioner from the senior storyteller.

The senior practitioner is the one who, six months in, has killed at least one of your AI projects, shipped a working version of another, and handed you the language to defend both decisions to a board member who reads the Wall Street Journal on Saturday mornings. That is the engagement worth eighteen months. Anything less is a slide deck with a salary attached.