When people call something a blackbox problem, they usually mean it as a complaint about the present. A system is deployed, nobody can explain it, and now it needs to be audited. The framing suggests the opacity arrived recently and could, with enough effort, be fixed.
It didn't. It was decided earlier, in the architecture.
Neural networks at scale trade interpretability for performance. That trade is made once, at the build stage, before any deployment decision, before any governance question, before any auditor enters the picture. What the audit encounters are downstream effects of that original decision, not the decision itself.
Three places the inheritance shows up
Blackbox is not one problem. The most obvious version is the architecture: the model's internal logic is inaccessible, either because it is proprietary or because a large neural network makes tracing any individual decision practically impossible. Nobody in the audit can change this.
The less visible version is documentation. A model might be explainable in principle, but the organisation deploying it has no record of how it was trained, tested, or validated. The weights exist. The documentation does not.
Then there is output definition. Nobody has said what a correct output looks like. Without that, there is no basis for evaluating whether the system is doing what it is supposed to do. This one is almost never architectural. It is a decision that was not made.
What the EU AI Act asks for
The Act imposes transparency and traceability obligations on systems that were, in many cases, built before those obligations existed. [1] For high-risk systems, including recruitment AI under Annex III, deployers must maintain logs, support human oversight, and explain individual decisions to affected people on request. [1]
None of that is full model explainability. But the obligation lands on organisations operating systems they did not build, trained on data they may not have reviewed, producing outputs they never formally defined. The architecture was inherited. The compliance requirement is new. The gap between those two facts is where the audit work starts.
What is actually testable
Was there a risk assessment before deployment? Is the scope formally defined and monitored? Was there pre-deployment testing, and against what criteria? Is anyone watching outputs after the system goes live?
None of this requires access to the weights. In the context of the Act, none of it is optional either.
Audit methodology here is still catching up to deployment reality. The frameworks exist. [2][3] But the field is producing judgment calls faster than settled procedure, and for now being explicit about what you can and cannot conclude is part of the work, not a gap in it.