first of all ... should we even bother?
Yes — it is possible to argue this, and some researchers do argue it.
But it is not the majority view, and the argument has limits.
“Explainability is not a worthy exercise because we are not meant to know the internal logic.”
This argument usually comes in three forms:
1. “Opaque complexity is natural and unavoidable.”
Just like:
the human brain
evolution
weather
even some quantum systems
…AI systems may have internal structures too large or high-dimensional for humans to intuitively understand.
So the argument goes:
If the system’s internal logic is inherently incomprehensible, then trying to “explain” it in simple human terms is fundamentally misguided.
2. “AI is not built for human-legible reasoning.”
Neural networks don’t learn logic rules. They learn:
distributed embeddings
nonlinear transformations
emergent features
As a result:
The model doesn’t have human-like reasoning inside it, so there is nothing to "explain" in the way people expect.
This argument says explainability forces humans’ expectations onto an alien computational structure.
3. “Performance matters more than interpretability.”
Some argue that:
aircraft designers don’t fully track airflow around every molecule
we trust antibiotics without understanding all biological processes
we use GPS without knowing general relativity
By analogy:
If the model is reliable, safe, and tested, we don’t need to know why it works.
Explainability is seen as a luxury, not a requirement.
Counterarguments (why explainability is considered important)
Even if the internal logic is inherently non-human, explainability is still critical for:
1. Safety
Without interpretability, we cannot detect:
hidden dangerous behaviors
deceptive alignment
bias
catastrophic failure modes
2. Trust
Regulators, auditors, and users need transparency for high-stakes applications:
medical diagnosis
credit scoring
criminal justice
autonomous vehicles
3. Debugging
Engineers need interpretability to fix:
hallucinations
brittle features
failures under distribution shift
4. Moral/Legal Responsibility
Governments increasingly require explainability:
EU AI Act
US FTC guidelines
Right to Explanation laws
So even if internal logic is alien, external explanations (counterfactuals, feature relevance, causal tests) are still practically necessary.
Philosophical Question: “Are we not meant to know the internal logic?”
This is a deeper, almost metaphysical point.
Some argue:
neural networks evolve internal structures through optimization
these structures are not designed to be interpretable
therefore, pursuing “full understanding” may be like trying to understand every neuron in a human brain:
possible in principle, but not what the system is for
This is similar to arguments in:
evolutionary biology
cognitive science
complex systems theory
The system’s function is primary; transparency is a human desire layered on top.
For me:
Yes, it is logically and philosophically valid to argue that explainability research is not inherently meaningful if:
the system’s internal logic is fundamentally distant from human understanding,
and the system can be validated purely through external behavior.
However, in engineering, law, safety, and ethics, explainability remains critical even if the internal mechanisms can never be fully understood.
So I operate with the logic that:
We may never understand the internal logic fully, but we still need explainability tools to understand the consequences of that logic.