Remember the famous scene from “When Harry Met Sally,’’ when the older woman in the deli proclaims, “I’ll have what she’s having”? As AI continues to steamroll its way into organizations, leaders are clamoring for bite-sized tastes of AI in the form of pilots to see what they can automate. The goal, of course, is for pilots to move to production and translate into cost savings to help businesses remain competitive.
Partners have certainly whetted clients’ appetites, having “aggressively marketed AI” over the past 18 months, according to a recent Techaisle report of 4,500 channel partners. However, the results thus far have been as underwhelming as a sandwich on stale bread—and difficulty monetizing AI investments is hampering partner profitability, leaving clients stuck in what Techaisle calls “pilot purgatory.”
“Partners have successfully sold the promise of AI, but they are struggling to translate technology demonstrations into recurring, high-margin revenue streams,’’ the report states. “The industry is caught in the middle: carrying the cost of AI innovation without yet realizing the financial return.”
What Causes AI Pilots To Fail
One main culprit of AI pilot purgatory is the “wow demo” trap,’’ observes Anurag Agrawal, founder and chief global analyst of Techaisle. “Partners have become very good at running impressive demonstrations using vendor-supplied sample data, but the pilot breaks down the moment it hits the client’s actual messy, siloed, ungoverned data,’’ he says. “The gap between demo and production is fundamentally a data quality gap.”
Speed without forethought is another. “What we’ve seen is usually from someone higher up … will say, ‘I saw something we need to start using because it did something fast or it replaced this,’” notes Bryan Antepara, service delivery lead at eMazzanti Technologies.
Another issue is that customers allow the proliferation of AI tools in their business, and “there is a lack of clarity over who manages the solutions and defines business benefits,’’ says Martin Summerhayes, head of managed and support services at Northdoor, a London-based IT consultancy and MSP.
Perhaps the biggest problem is data governance debt, Agrawal says. “Many pilots stall simply because the prerequisite data work was never scoped into the engagement.”
What Clients Are Clamoring For
There are three common types of AI pilots partners are deploying for clients, Agrawal says. The first are copilot-style tools that sit on top of a client’s internal data to automate search and summarization, as well as perform compliance checks.
Partners are also building customer-facing automation, notably, AI-driven chatbots that conduct ticket triage, and contact center copilots, he says. The third category, which Agrawal says is quickly becoming the most exciting, is operational workflow automation using agentic AI. These are instances “where the AI doesn’t just assist a human but actually acts autonomously across systems,’’ he says, to perform multi-step tasks, such as invoice reconciliation, procurement approvals or IT incident remediation.
This is the year that “pilots are shifting from single-task copilots to these multistep agentic workflows,” Agrawal says, “and that shift is exactly where the monetization opportunity lives—but also where the complexity explodes.”
The Partner Proposition
Partners understand all too well why they need to help clients regroup and rework pilots to address issues including lack of leadership sponsors and success criteria, as well as complexity and a lack of governance. Part of the problem is that clients sometime confuse AI with automation.
“The first wave of AI was demo-led [and] clients bought the dream and the broad transformational idea,’’ says Brad Lassiter, CEO of managed IT services company LastTech. “The problem is that this isn’t where AI was and the companies creating it, buying it or selling it did not know how to bring about this transformation.”
Still, the pilot is impressive and necessary to show its value to the client, Lassiter adds. “But at the end of the day, clients really want to buy reliability of business outcomes … The problem is that this can’t be done just by selling a tool. You have to have significant manhours devoted to understanding the business, the process, the workflow, and the technology.”
Then, the workflow needs to be built and tested prior to allowing the business to rely on it. Only then can partners charge for the outcome, Lassiter says. “And a single outcome is not worth a lot. This must be done dozens of times before significant margins develop,’’ he stresses. “People try to jump to the last step and skip the work. The work is the value. And that’s what clients are buying.”
Pilots In ‘AI Purgatory’
Northdoor has been deep in the process of developing its own AI services over the past 18 months. This has given Summerhayes time to reflect on why pilots fail.
“Pilot purgatory is real, and it’s largely self-inflicted,’’ he says. “Almost every AI-related client conversation we have starts the same way: They’ve run a proof of concept, usually something around document summarization, a co-pilot tool, or an internal chatbot, it’s technically worked, people liked the demo, and then … nothing. It sits.”
Although clients rarely admit this, Summerhayes believes they are keenly aware that no one defined what “success” looked like before the pilot started. “There was no commercial case, no change management plan, no owner on the business side,’’ he says. “The technology somewhat proved itself; the business case didn’t.”
Agrawal agrees with this scenario, stressing that the root causes of unsuccessful pilots are “structural and commercial, not technical.”
Summerhayes also faults partners who lead with technology. “Every time an MSP goes into a conversation with ‘Here’s what this AI tool can do,’ they end up in a feature demonstration that doesn’t connect to a budget holder’s priorities,’’ he says. “The pilots that die were almost always sold to the wrong person in the wrong language.”
Sometimes, AI agents aren’t the answer. Antepara recalls a situation his team ran into with a manufacturing customer that wanted to automate processes and increase collaboration among departments. Antepara’s team deployed Copilot to test whether employees could communicate tasks that they wanted automated. However, it soon became apparent that an AI agent wasn’t the panacea the company was hoping for.
“We realized It wasn’t an AI agent we needed to roll out; people just needed automation,’’ Antepara says. “People keep thinking AI is thing that does it all, but it’s one vital component of automation.”
The client had someone manually collecting information from documents, spreadsheets, and PDFs, and giving it to someone else to review and make decisions. That’s where the AI worked well — reducing the time it took to do that.
“The issue that cropped up was that [the AI] wasn’t adopted because of the trust factor,’’ Antepara says. Someone had been reviewing that data on spreadsheets for years and understood the context behind it and felt Copilot didn’t.
The feeling was, if a human was still jumping in to review its work, the client didn’t see the point of automating it. What the company didn’t understand was that “it’s not going to be perfect [the] first time around,” Antepara says, adding that the more you work with AI agents, the better they will function and provide better outcomes.
“There’s always someone who isn’t seeing the vision, so there has to be a champion group” to help others become open to change, Antepara says. “These AI pilots fail when they’re treated like a tech project, but they succeed when there are behavioral changes.”
Lassiter had a client who trialed Copilot as a secure, embedded AI platform within their organization. “But the additional security added additional restrictions and greatly reduced its value,’’ he says. “It was still useful, but as a low-level assistant, parsing emails and meetings to surface the most valuable information. That reduces the payoff of making the transition in the first place.”
The real value comes in AI-assisted coding, file management, and decision making, Lassiter notes, “but to get that, you have to give these systems significantly more data.”
The Elements Of A Successful Pilot
Northdoor’s Summerhayes points to an AI pilot they worked on for a financial services insurance organization that wanted to start with a small Copilot deployment, since leadership was concerned about allowing the tools to be used across the business.
Summerhayes’ team started by conducting a small assessment of the firm’s AI readiness. “For these types of assessments, anything lower than a score of 2.5, means that the organization does not have either the maturity of governance [or] some of the underlying compliance from a data and file access, management and visibility perspective,’’ he explains.
The Northdoor team made several recommendations to the internal IT group, which decided to complete the remediation work themselves. Now, Northdoor and the IT group will jointly plan the pilot, “including clear business-focused use cases with measurable ROI, especially to the processing of insurance claims and risk,’’ Summerhayes says.
From Pilot To Production
To ensure a greater chance of success, Antepara says they ask clients a lot of questions and then prepare an AI roadmap before beginning a pilot deployment. Agrawal echoes that, saying it is critical to start with the data, not the model.
“Partners who are successfully converting pilots to production are leading with a data readiness assessment before any AI is deployed,” Agrawal says, “packaging data governance and data fabric services as Phase 0 of every AI engagement.”
The clients Northwood sees moving to production “are the ones being guided away from building bespoke models toward deploying AI within existing workflows and platforms they already pay for,’’ says Summerhayes. “The quick wins that build internal confidence and internal advocacy … come from augmenting what already exists—rather than replacing it.”
It’s important to remember that AI pilot purgatory is not a technology failure but a business model failure, says Agrawal. “Partners who treat AI pilots as structured, outcome-defined, data-ready engagements with production economics built in from the start are the ones converting pilots into recurring, high-margin revenue streams.”
5 Critical Elements Needed For AI Pilot Success
- Start small and operationalize one key task that proves its merit. Make sure it is bulletproof before adding others.
- Define success criteria before the pilot begins to help ensure it will earn a production budget.
- Identify an executive sponsor who is accountable for outcomes—not open-ended, unmeasurable deliverables—and isn’t interested in just the technology.
- Shift from selling tools to packaging autonomous outcomes.
- Engineer time-to-value from a pilot in 30 days, rather than quarters.
To learn more about AI trends for MSPs, check out 5 AI Trends Defining Success For MSPs.
