The race to an AI workforce faces one important trust gap: What happens when an agent goes rogue?

At Fortune’s recent Brainstorm AI event in San Francisco, an expert roundtable grappled with that question as insiders shared how their companies are approaching security and governance—an issue that is leapfrogging even more practical challenges such as data and compute power. Companies are in an arm’s race to parachute AI agents into their workflows that can tackle tasks autonomously and with little human supervision. But many are facing a fundamental paradox that is slowing adoption to a crawl: Moving fast requires trust, and yet building trust takes a lot of time. 

Dev Rishi, general manager for AI at Rubrik, joined the security company last summer following its acquisition of his deep learning AI startup Predibase. Afterward, he spent the next four months meeting with executives from 180 companies. He used those insights to divide agentic AI adoption into four phases, he told the Brainstorm AI audience. (To level set, agentic adoption refers to businesses implementing AI systems that work autonomously, rather than responding to prompts.) 

According to Rishi’s learnings, the four phases he unearthed include the early experimentation phase where companies are hard at work on prototyping their agents and mapping goals they think could be integrated into their workflows. The second phase, said Rishi, is the trickiest. That’s when companies shift their agents from prototypes and into formal work production. The third phase involves scaling those autonomous agents across the entire company. The fourth and final stage—which no one Rishi spoke with had achieved—is autonomous AI. 

Roughly half of the 180 companies were in the experimentation and prototyping phase, Rishi found, while 25% were hard at work formalizing their prototypes. Another 13% were scaling, and the remaining 12% hadn’t started any AI projects. However, Rishi projects a dramatic change ahead: In the next two years, those in the 50% bucket are anticipating that they will move into phase two, according to their roadmaps. 

“I think we’re going to see a lot of adoption very quickly,” Rishi told the audience. 

However, there’s a major risk holding companies back from going “fast and hard,” when it comes to speeding up the implementation of AI agents in the workforce, he noted. That risk—and the No.1 blocker to broader deployment of agents— is security and governance, he said. And because of that, companies are struggling to shift from agents being used for knowledge retrieval to being action oriented.

“Our focus actually is to accelerate the AI transformation,” said Rishi. “I think the number one risk factor, the number one bottleneck to that, is risk [itself].”

Integrating agents into the workforce

Kathleen Peters, chief innovation office at Experian who leads product strategy, said the slowing is due to not fully understanding the risks when AI agents overstep the guardrails that companies have put into place and the failsafes needed for when that happens.

“If something goes wrong, if there’s a hallucination, if there’s a power outage, what can we fall back to,” she questioned. “It’s one of those things where some executives, depending on the industry, are wanting to understand ‘How do we feel safe?’”

Figuring out that piece will be different for every company and is likely to be particularly thorny for companies in highly regulated industries, she noted. Chandhu Nair, senior vice president in data, AI, and innovation at home improvement retailer Lowe’s, noted that it’s “fairly easy” to build agents, but people don’t understand what they are: Are they a digital employee? Is it a workforce? How will it be incorporated into the organizational fabric? 

“It’s almost like hiring a whole bunch of people without an HR function,” said Nair. “So we have a lot of agents, with no kind of ways to properly map them, and that’s been the focus.”

The company has been working through some of these questions, including who might be responsible if something goes wrong. “It’s hard to trace that back,” said Nair. 

Experian’s Peters predicted that the next few years will see a lot of those very questions hashed out in public even as conversations take place simultaneously behind closed doors in boardrooms and among senior compliance and strategy committees. 

“I actually think something bad is going to happen,” Peters said. “There are going to be breaches. There are going to be agents that go rogue in unexpected ways. And those are going to make for a very interesting headlines in the news.”

Big blowups will generate a lot of attention, Peters continued, and reputational risk will be on the line. That will force the issue of uncomfortable conversations about where liabilities reside regarding software and agents, and it will all likely add up to increased regulation, she said. 

“I think that’s going to be part of our societal overall change management in thinking about these new ways of working,” Peters said.

Still, there are concrete examples as to how AI can benefit companies when it is implemented in ways that resonate with employees and customers. 

Nair said Lowe’s has seen strong adoption and “tangible” return on investment from the AI it has embedded into the company’s operations thus far. For instance, among its 250,000 store associates, each has an agent companion with extensive product knowledge across its 100,000 square foot stores that sell anything from electrical equipment, to paints, to plumbing supplies. A lot of the newer entrants to the Lowe’s workforce aren’t tradespeople, said Nair, and the agent companions have become the “fastest-adopted technology” so far.

“It was important to get the use cases right that really resonate back with the customer,” he said. In terms of driving change management in stores, “if the product is good and can add value, the adoption just goes through the roof.”

Who’s watching the agent?

But for those who work at headquarters, the change management techniques have to be different, he added, which piles on the complexity. 

And many enterprises are stuck at another early-stage question, which is whether they should build their own agents or rely on the AI capabilities developed by major software vendors. 

Rakesh Jain, executive director for cloud and AI engineering at healthcare system Mass General Brigham, said his organization is taking a wait-and-see approach. With major platforms like Salesforce, Workday, and ServiceNow building their own agents, it could create redundancies if his organization builds its own agents at the same time. 

“If there are gaps, then we want to build our own agents,” said Jain. “Otherwise, we would rely on buying the agents that the product vendors are building.”

In healthcare, Jain said there’s a critical need for human oversight given the high stakes. 

“The patient complexity cannot be determined through algorithms,” he said. “There has to be a human involved in it.” In his experience, agents can accelerate decision making, but humans have to make the final judgment, with doctors validating everything before any action is taken. 

Still, Jain also sees enormous potential upside as the technology matures. In radiology, for example, an agent trained on the expertise of multiple doctors could catch tumors in dense tissue that a single radiologist might miss. But even with agents trained on multiple doctors, “you still have to have a human judgment in there,” said Jain. 

And the threat of overreach by an agent that is supposed to be a trusted entity is ever present. He compared a rogue agent to an autoimmune disease, which is one of the most difficult conditions for doctors to diagnose and treat because the threat is internal. If an agent inside a system “becomes corrupt,” he said, “it’s going to cause massive damages which people have not been able to really quantify.”

Despite the open questions and looming challenges, Rishi said there’s a path forward. He identified two requirements for building trust in agents. First, companies need systems that provide confidence that agents are operating within policy guardrails. Second, they need clear policies and procedures for when things will inevitably go wrong—a policy with teeth. Nair, additionally, added three factors for building trust and moving forward smartly: identity and accountability and knowing who the agent is; evaluating how consistent the quality of each agent’s output is; and, reviewing the post-mortem trail that can explain why and when mistakes have occurred. 

“Systems can make mistakes, just like humans can as well,” said Nair. “ But to be able to explain and recover is equally important.”