Researchers Just Unlocked AI’s Black Box

Just last month, I wrote about how today’s AI models are essentially black boxes.

We know what goes in, and we know what comes out. But what happens in between has remained one of the biggest mysteries in artificial intelligence.

But that could finally be starting to change.

According to new research from Anthropic, scientists are beginning to peer inside some of the world’s most advanced AI models as they reason through problems.

And what they’ve uncovered could alter the way we think about artificial intelligence forever.

A Window Into AI’s Mind

Engineers don’t program ChatGPT or Claude the way they program a normal app.

Instead, they train them on huge amounts of information. Then they test them, adjust them and watch how they behave.

That means today’s AI models often know how to do things that no one directly taught them to do.

It also means that no one fully understands what happens inside them.

But Anthropic’s new research is an attempt to change that.

The company developed a tool called the Jacobian lens, or J-lens. It lets researchers look inside an AI model while it’s working and watch its reasoning take shape before it produces an answer.

And some of the results are astonishing.

In one test, Anthropic gave Claude this sentence: “The number of legs on the animal that spins webs is…”

To answer correctly, Claude first had to recognize the answer was a spider. Then it had to remember that spiders have eight legs.

But here’s what I find utterly fascinating.

The word “spider” never appeared in the prompt. And Claude’s answer was simply “eight.” Yet inside the model, researchers could see the concept of “spider” appear before the answer came out.

Then they tried something even stranger. They swapped that internal “spider” concept for “ant.”

And Claude’s answer changed from eight to six.

Image: Anthropic

In other words, when researchers changed the model’s hidden reasoning, the final answer changed with it.

That’s a huge breakthrough.

Researchers aren’t just peering inside AI’s black box. They’re beginning to understand what they’re seeing well enough that they can test it, change it and eventually make it more reliable.

And Anthropic found examples like this again and again.

In another test, the model was tasked with writing a rhyming couplet.

You might assume it would simply write one word at a time, the way autocomplete predicts your next word. But that’s not what researchers found.

Instead, Claude appeared to plan the rhyme before it reached the end of the line.

Given the line, “The soldier marched into the night,” the model internally planned to end the next line with “fight.” But when researchers swapped that hidden plan from “fight” to “light,” the entire sentence changed.

Instead of writing “Prepared to face the coming fight,” the model shifted toward “morning light.”

Image: Anthropic

That means the model wasn’t merely predicting the next word. It was carrying a future word in mind, then shaping the words before it to make the rhyme work.

That’s not how most people think AI works.

Critics often call AI models “stochastic parrots,” implying that they’re mostly repeating patterns from their training data. But this research suggests something more complicated is happening.

The model appears to build temporary ideas, use them, revise them and sometimes act on them before we ever see the final answer.

It even happened with math.

Researchers asked the model to copy a sentence word for word. At the same time, they secretly instructed it to calculate 3² minus 2.

To anyone watching the output, Claude appeared to be doing nothing more than copying text.

But inside the model, researchers watched the model’s internal reasoning move from the idea of arithmetic to the number nine and finally to the answer seven.

In other words, Claude was quietly solving the math problem even though nothing about its visible response suggested it was doing any math at all.

This tells us there’s an entire layer of hidden activity taking place inside these models.

And sometimes that hidden activity can be more interesting than the answer itself.

In one example, Claude was shown fake search results designed to trick it. This is called a prompt injection, which is basically an attempt to sneak bad instructions into the information an AI is reading.

Claude ignored the malicious instructions instead of following them.

But inside the model, Anthropic’s tool showed words like “fake,” “fraud” and “secret.”

Image: Anthropic

So the model appears to have recognized that the search results were suspicious before deciding not to use them.

That could prove to be extremely important.

Because AI models are increasingly being targeted by prompt injection attacks that try to manipulate their behavior.

If researchers can detect those attacks while they’re happening inside the model, they might eventually be able to stop them before the AI ever produces a response.

Here’s My Take

Your brain processes huge amounts of information all the time, yet most of it never enters your awareness.

Different parts of the brain process different kinds of information before sharing it in a temporary mental workspace where decisions are made.

Anthropic argues that language models have something that plays a similar functional role.

To be clear, the company isn’t claiming that its AI is conscious.

The researchers are simply saying that some of the same organizational principles may also appear inside large language models.

And that’s a big deal.

Because understanding how AI reaches its conclusions could ultimately prove just as important as making it smarter.

Regards,

Ian KingChief Strategist, Banyan Hill Publishing

Editor’s Note: We’d love to hear from you!

If you want to share your thoughts or suggestions about the Daily Disruptor, or if there are any specific topics you’d like us to cover, just send an email to [email protected].

Don’t worry, we won’t reveal your full name in the event we publish a response. So feel free to comment away!