Can we trust AI if we don't know how it works?

The better question is, can you trust the programmers not to embed "features" that violate individual human rights.

In a word... bullshit. We *DO* know how it works... the same way we know Statistics work, or Calculus Works, or Differential Equations... or Matrix Algebra.

An Expert System is created (organized? programmed? designed?) with a specific knowledge domain in mind. Domain Experts are used in the build & testing phases to validate the knowledge base. Knowledge engineers monitor and maintain the knowledge base to prune out spurious knowledge (superstitions) that may inadvertently arise at run time due to conflictng logic or faulty modification. Ultimately, it comes down to Backward Chaining Logic (Diagnostic AI) or Forward Chaining Logic (Predictive AI).

At any point a well designed Expert System can print a trace showing *EXACTLY* how it reached its conclusion... chapter & verse.

As to Genetic Algorithms (GAs) or Artificial Neural Networks (ANNs) what you are asking for is ridiculous! The input nodes HAVE an assigned meaning with a given ‘strength value’ ... the nodes on the output side also have known meanings with calculated cetainty factors. The nodes in the intermediate layer have no inherent meaning, they are pure numbers. In Multivariate Statistics you get the same conceptual problem. The cells of the inbound matrix have inherent meaning and the results have meaning (subject to interpretation) with a certain confidence value. But during the calculations if you grabbed a random value from a location in the intermediate matrix you’d be hard pressed to give an interpretation.

Do the math... :-)