Explanations¶
Beyond the readings on whole spaces, Fannar explains individual decisions by contrast: why
does the model place object \(i\) apart from object \(j\)? Pair explanations need vector inputs,
X of shape (n, F).
Pairs¶
For a pair \((i, j)\) and each feature \(f\), let \(x_i^{(f)}\) be \(x_i\) with feature \(f\) replaced by its value in \(x_j\). The score of \(f\) in a layer is
measured in that layer's representation space. A positive \(\Delta d_f\) means the swap brings \(i\) closer to \(j\): the feature is part of what separates them for the model. Only the \(F + 2\) rows \(x_j, x_i, x_i^{(1)}, \dots, x_i^{(F)}\) go through the model, so the cost is linear in the number of features.
expl = interp.explain_pair(i) # j = nearest object (in input space) with another prediction
expl = interp.explain_pair(i, j) # or an explicit partner
expl.j, expl.pred_i, expl.pred_j
expl.top(5) # [(feature, x_i, x_j, share), ...], last layer
expl.top(5, layer="layer2")
expl.ranking(), expl.shares() # indices, and positive scores normalised to sum 1
expl.scores["layer2"] # raw Δd per feature
expl.to_frame() # pandas
expl.plot(k=8) # share per layer (heatmap) and the values of i and j
First order
\(\Delta d_f\) swaps one feature at a time. Features that only matter together with others get no credit.
Errors¶
explain_errors aggregates pair explanations over the model's mistakes. Each misclassified
object \(i\) is paired with the nearest object the model gets right in the true class of \(i\),
so the shares say which features keep the error away from where it belongs.
errs = interp.explain_errors() # against="correct" (default) or "unlike"
errs = interp.explain_errors(max_errors=200, reference=100, seed=0)
errs.top(5) # [(feature, mean share, fraction of errors where it is first)]
errs.mean_shares() # per feature, over the errors
errs.reference_shares() # the same over correctly predicted objects: the general importance
errs.direction() # +1: errors have lower values than their partners; −1: higher
errs.confusions() # [(true, predicted, count)], most frequent first
errs.by_class(), errs.by_confusion()
errs.pairs # [(error, partner)], e.g. for swap_test
errs.to_frame()
errs.plot(rows="class") # or rows="confusion"
Compare mean_shares() with reference_shares(): a feature with a high share among the errors
but a low one in general is specific to the mistakes.
Validating with the swap test¶
The pair ranking is validated outside the geometry. Copy the top-\(k\) features of \(j\) into \(i\), for \(k = 0, \dots, F\), and check whether the model's prediction becomes the prediction of \(j\); the area under that curve is higher when the ranking finds the decisive features sooner. This is the procedure of nearest-instance counterfactuals such as NICE.
interp.swap_test(50, layer="layer3")
# {"method": <area>, "absdiff": <area>, "random": <area>, "pairs": 50, "layer": "layer3"}
interp.swap_test(errs.pairs[:30]) # explicit pairs
method ranks by \(\Delta d\), absdiff by \(|x_j - x_i|\), and random shuffles the features that
differ. The method's ranking is useful where it beats both. With an integer, the pairs are random
correctly predicted objects, each with its nearest object of another prediction.
Energy maps¶
For a single input with spatial structure (an image, a series, a sequence of tokens), the
positions of that input can be the objects. energy_map builds the space of the positions of
one layer and returns the retained energy \(E_\parallel\) of each one:
emap = interp.energy_map(x, "layer2", max_side=32) # (h, w) for images, (L,) for sequences
interp.plot_energy_map(x, "layer2", image=rgb) # rgb: (H, W[, 3]) in [0, 1]
The components use linear kernels without the angular transform, which would give every
position the same energy (fa.positions_pipeline()). max_side caps the grid, since the Gram
is \(P \times P\) for \(P\) positions. Energy maps need a PyTorch model.