Despite Progress, AI Inner Workings Remain a Black Box
Author
: Copyright © 2026 Todd A. Jacobs
Posted
: 2026-10-19
Large language models are currently promoted to the public as forms of artificial intelligence. No one doubts that #AI and #LLM tools can be useful, but treating these technologies as "thinking machines" is a marketing ploy. Even the scientists and engineers who build today's frontier models will admit that they know how to build systems that simulate thinking, but that many key elements of how they work remain unexplained outcomes rather than the result of deliberate design choices.
Scientists and engineers call observable but unexplained behaviors of a system "emergent properties." A useful comparison is seen in ant colonies: ants search for resources in individual and arguably inefficient random patterns, but then act collectively to gather any resources found to efficiently carry them back to the anthill once found. The apparent efficiency of ants' resource gathering is thus an emergent property that results from merging random search patterns with scent trails and an instinct that drives each ant to follow the most-used and therefore strongest path left by the rest of the colony.
Ben Patterson of PCWorld explains Anthropic's own theories on this issue in his July, 2026 article.
We still don’t know much about how AI 'thinks.' Give ChatGPT, Claude, or Gemini a prompt, and they’ll spit out an answer. Much of what happens in between remains shrouded in mystery.
However, Patterson explains that researchers at Anthropic have discovered an emergent property in their Claude models: an unplanned feature that Anthropic calls a Jacobian workspace ("J-space").
What’s weird about this workspace...is that Anthropic didn’t actually design it. Instead, the J-space appears to have 'emerged on its own' during Claude’s training[.]
Like our ant colony, #J_space wasn't something deliberately designed into the system by Anthropic. It's something they discovered while researching how their models process information. To their credit, Anthropic's researchers are actively theorizing and exploring this process, as the only sure thing is that LLMs and other AI systems process information differently than people do.
Even models that provide chain-of-thought (#CoT) traces don't technically meet the definition of #Explainable_AI (#XAI). Instead, they transform their internal black-box processes into words using a language model. Despite being promoted as a visible thought process that accurately describes what the model is doing, AI chains of thought are best thought of as proxies that can't yet be audited end-to-end. The statistical nature of LLM conversion to natural language makes these proxied chains of thought inherently unreliable for evaluating the underlying system processes, no matter how convincing the user-visible chains of thought appear or how well-reasoned they sound.
NicFab Blog — Privacy, GDPR & Artificial IntelligenceNewsletter #43 - 20 October 2026NicFab Newsletter #43 — 20 October 2026. Weekly review on privacy, data protection, AI regulation, cybersecurity and digital rights.
