A prediction is scientifically useful only when it can be understood, interrogated, and trusted. In the physical sciences, the goal of explanation extends beyond verifying that a model performs well on held-out data: researchers seek to extract governing mechanisms, identify the physical features that drive predictions, and determine whether a model has learned physically meaningful relationships or merely exploited statistical cor- relations in the training set.

This workshop surveys the landscape of explainability methods, distinguishing between inherently interpretable models and post-hoc explanation of opaque architectures. It examines SHAP values and their use for feature importance analysis in molecular and materials models, attention mechanisms and what they do (and do not) reveal about learned representations, gradient-based saliency maps for spectral and image data, and symbolic regression as a route to recovering closed-form physical laws from trained models.

The session develops scientific standards for explanation, addressing the question of when an explanation is physically meaningful as opposed to merely statistically informative. Participants will gain practical tools for interrogating their own models and a critical framework for evaluating explainability claims in the literature.

Extended Key Takeaways & Outputs

• Clear distinction between interpretable models and post-hoc explanation methods

• Practical knowledge of SHAP, attention analysis, saliency maps, and symbolic regression

• Scientific standards for evaluating when an explanation is physically meaningful

• Strategies for interrogating model predictions in molecular and materials contexts

• Critical framework for assessing explainability claims in the scientific ML literature