Skip to content

Fannar

Representation spaces of trained models from Gram matrices of several sources: one positive-definite kernel per source of information, admissible transformations, Hadamard composition, and the geometry and spectral energy of the resulting matrix.

Fannar is the name of the method, and this library is its implementation. Both are described in the paper Building Spaces via Similarity Tensors in Reproducing Kernel Hilbert Spaces: A Mechanistic Interpretability Method.

pip install fannar            # core: numpy + matplotlib
pip install "fannar[all]"     # + PyTorch, scikit-learn and pandas

The idea in one line

\[ G \;=\; T\big(\,T_1(K_1) \circ T_2(K_2) \circ \cdots \circ T_d(K_d)\,\big) \]

\(K_l\) is the raw Gram matrix of source \(l\) (from a kernel, or one you supply), \(T_l\) the component transform, \(\circ\) the Hadamard product and \(T\) the final transform. By the Schur product theorem, \(G\) is positive semidefinite for any such choice, so it defines a geometry (distances, neighbourhoods, level sets) and a spectral decomposition of energy. Everything Fannar reports is read from \(G\).

The library keeps apart, as the paper does:

What it shows

The same cat photo with a small mark in the corner, seen by a clean model and by a model that learned a shortcut, according to Fannar, SHAP and LIME
A ResNet-18 trained with a small mark on 95% of the dog photos learns the shortcut and calls 40% of the marked cats dogs. In that model, half of Fannar's energy map falls on the mark, which covers 2% of the image; in the clean model, 91% falls on the animal.
Errors of a churn model made with more than 90% confidence, and how many of them the geometry flags
Confident errors. 113 of the 136 errors of a churn network come with more than 90% confidence; the geometric margin flags 55 of them.
The features that most separate a rejected applicant from an approved one according to Fannar, SHAP and LIME
Pair explanations (XGBoost). What separates a rejected applicant from a similar approved one; where Fannar, SHAP and LIME agree, the conclusion is firmer.

Note

These examples come from models trained to illustrate the method, one run each; the systematic evaluation is in the paper. More case studies, and the interpretability service, at fannar.fernandocsdm.com.

Where to go next

License

Fannar is released under the Creative Commons Attribution-NonCommercial 4.0 International License. For commercial use of the method, see the interpretability service at fannar.fernandocsdm.com.