Large Language Models (LLMs) are transforming healthcare NLP applications — including clinical summarization, diagnosis assistance, adverse event detection, and risk prediction — but their black-box nature remains a barrier to trust, safety, and regulatory adoption. This talk presents a comparative analysis of leading interpretability techniques for LLMs, spanning:

• Token-level attribution
• Model-internal reasoning
• Behavior-level safety evaluation

We examine methods such as LIME, SHAP, Integrated Gradients, and Attention Rollout for explaining which clinical tokens influence model predictions, along with probing classifiers and neuron activation analysis to reveal how LLMs encode:

• Medical terminology
• Negation
• Temporality
• Clinical relations

Additionally, we evaluate counterfactual explanations, bias and fairness assessments, and hallucination detection to measure the real-world reliability of LLM behavior.

Through a unified comparison framework, we highlight the strengths, limitations, and clinical relevance of each technique and offer practical recommendations for deploying interpretable and trustworthy LLMs in healthcare settings. This session provides clinicians, researchers, and AI practitioners with actionable insights to enhance transparency, mitigate risk, and improve the safety of AI-driven clinical decision support systems.