An AI pilot can look successful from one desk and feel exhausting from another. Drafts arrive faster. Someone still has to find the missing context, correct the details and explain why the suggested next step cannot happen.
For a growing organisation, this is a practical measurement problem. If the aim is to release capacity, measure the work across the whole process. Otherwise, a reduction in founder time can hide an increase in everyone else's workload.
Count completed work
Generating a reply is one step. Resolving the customer's question includes checking the reply, sending it, handling any follow-up and correcting mistakes. A useful pilot follows that complete journey.
There is a real example worth looking at. In their November 2023 working paper, Generative AI at Work, Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied an AI assistant used by 5,179 customer support agents. They reported a 14% average increase in issues resolved per hour. Gains were larger for novice and lower-skilled workers, with minimal impact on experienced, highly skilled workers.
This was a particular setting, not a forecast for every business. But the measurement is useful: issues resolved per hour connects the tool to completed work. And the different results across workers show why a team average deserves a closer look. The same tool can change different people's work in different ways.
Follow the checking and repair
Before starting a pilot, record a small baseline. How many cases are completed? How much active time do they take? How long do they wait? How often do they come back? Include the founder's interruptions and the team's preparation and correction time.
During the pilot, use the same measures. Add time spent maintaining instructions, updating source material and resolving tool problems. Keep those costs visible even if they fall to someone enthusiastic who quietly absorbs them.
Human review also needs a defined purpose. Ask the reviewer to check something specific: whether the source is current, whether the proposed action fits the policy, or whether a commitment needs approval. Give that person the evidence and authority to act. An instruction to “check everything” makes the workload difficult to estimate and the responsibility difficult to carry.
Decide what the result means
Agree the decision before looking at the results. Who will decide whether to continue, adapt or stop? What improvement would justify keeping the tool? What increase in errors, waiting or team workload would require a change?
A short case record helps: source, AI draft, reviewer, final action and any correction. Keep it proportionate to the work and its consequences. When a result looks unusually good or bad, you can then reconstruct what happened.
This is something I would establish in a Blueprint before a wider Build: the baseline, the review responsibilities and the criteria for continuing. Stewardship can then check whether the benefit holds as the workload, team and tools change.
Source: November 2023 working paper linked above. Used as an established research example, not a new development.