Hardening container images for LLM inference: separating vulnerability count from real risk
Container images that serve large language models carry unusually large, heavily transitive dependency trees spanning system libraries and machine learning frameworks, and publicly distributed images routinely register vulnerability counts that block production deployment outright; prior work found that publicly distributed container images contain more than one hundred and eighty vulnerabilities on average. This talk shows that vulnerability count and actual risk are distributed very differently, and conflating them misdirects remediation effort. In a widely used public inference image, operating system packages accounted for roughly ninety-three percent of all findings, yet only about one percent of those were rated critical or high severity; the Python dependency tree, by contrast, represented only about six percent of findings but carried roughly two-thirds of all critical and high severity findings, with more than a third of Python findings rated severe. A single family of nine multimedia codec packages, never invoked by the workload, contributed roughly a quarter of the entire report. We present a practical framework built on this asymmetry: minimal base images and multi-stage builds to collapse the count, paired with targeted Python-tree remediation, including vendored copies and auxiliary manifests that inflate findings without exploitable risk. Applied to a production inference image, this framework reduced findings from over four thousand to under forty, with zero critical or high severity, in three weeks, faster than a specialized security vendor’s six-week estimate. Attendees leave with a repeatable process for separating genuine exposure from scanner artifacts in AI/LLM infrastructure.
About the speaker
Devavrat Khanolkar
Machine Learning Engineering Technical Leader at Cisco Systems
Bio coming soon