Securing LLM Gateways: Threats, Hardening, and Compliance Guide (2026)

The Attack Surface: Why LLM Gateways Are High-Value Targets

An LLM gateway is a reverse proxy that sits between your application code and one or more LLM provider APIs. Your application sends requests to one internal endpoint; the gateway handles authentication, routing, rate limiting, caching, and logging across all the providers it manages. From a security posture standpoint, an LLM gateway is a credential broker. Compromise that proxy and you don’t get one set of credentials. You get all of them simultaneously. That concentration of access puts gateways in a category alongside identity providers and secrets vaults: infrastructure where a single breach has disproportionate downstream consequences.

The attack surface has three layers. The gateway’s own application layer carries all the risks of any web service: authentication flaws, injection vulnerabilities, privilege escalation, supply-chain tampering. The data plane carries sensitive content: user prompts that may include PII, internal documents, and system instructions routed to external providers whose security posture you don’t control. And the control plane governs how traffic is routed, logged, rate-limited, and billed, making it a target for cost-manipulation and data exfiltration attacks distinct from traditional application security.

The 2026 incident record for LiteLLM, the most widely deployed open-source LLM gateway, validates the theoretical risk with documented specifics: 8 official GitHub Security Advisories published in a single calendar year, including three rated Critical, and a supply-chain incident involving published versions that contained credential-harvesting malware, catalogued in full in our writeup of the LiteLLM security incident. The gateway category is not hypothetically risky. It is actively targeted.

The OWASP LLM Top 10 at the Gateway Layer

The OWASP Top 10 for LLM Applications v1.1 (2025) defines the canonical threat model for AI systems. The gateway layer doesn’t eliminate these threats, but it’s the right place to implement controls for LLM01 through LLM05 and LLM10 (Unbounded Consumption). Understanding what the gateway can and cannot protect against is the starting point for a realistic security architecture.

What the Incident Record Shows: LiteLLM’s 2026 CVE History

LiteLLM is the dominant open-source LLM gateway. Its security advisory history in 2026 is the most documented example of what gateway vulnerability looks like in practice. The maintainers publish advisories, ship patches, and demonstrate responsible disclosure hygiene; this isn’t a project indictment. It’s a data point about the complexity of securing a component that sits between applications and all their LLM providers.

The 8 official GitHub Security Advisories break down into two categories. Five address application-layer vulnerabilities in the proxy code itself: command execution via MCP stdio test endpoints (High), sandbox escape in the custom-code guardrail (High), SSTI (Server-Side Template Injection) in the /prompts/test endpoint (High), password hash exposure and pass-the-hash bypass (High), and privilege escalation via an unrestricted proxy configuration endpoint (High). Three are rated Critical: authentication bypass via host header injection, SQL injection in proxy API key verification, and OIDC (OpenID Connect) userinfo cache key collision.

The supply-chain incident is structurally distinct. GHSA-5mg7-485q-xm76 (Critical, published March 2026) documents that two LiteLLM versions were published to PyPI containing credential-harvesting malware. This isn’t a code vulnerability in the gateway logic; it’s a packaging compromise where the published artifact itself was malicious. Patching doesn’t help because the malicious code arrived with the package. Prevention requires hash verification before install and cosign image signing for container deployments.

Across 2024–2026, the GitHub Advisory Database records 16 documented CVEs for LiteLLM, spanning credential logging exposure, denial of service, remote code execution, SQL injection, improper authorization, and API key information disclosure. Fixed versions referenced in advisory metadata include 1.83.10, 1.83.14, and 1.65.4 for specific issues. Running below these versions with internet-exposed management endpoints carries documented, not theoretical, risk.

Infrastructure Hardening: From Container to Network

The gap between a default LiteLLM deployment and a hardened one is significant. Defaults are optimized for getting started. Production deployments facing real threat models need a different configuration profile, and the hardening reference material is available; it just isn’t applied by default.

Container hardening

LiteLLM ships a docker-compose.hardened.yml reference configuration. The critical changes from the default are a non-root user, a read-only filesystem with explicit tmpfs mounts for writable paths, dropped Linux capabilities to a minimal set, and resource limits on CPU and memory. Before deploying any version, verify the container image against published cosign signatures: the supply-chain incident demonstrates the published image can be compromised at the packaging stage.

Network isolation

The LiteLLM management interface, the UI, health endpoints, and admin API, should never be internet-exposed. This is the design intent: the management plane is a separate attack surface from the proxy plane. Run the proxy in a private subnet. If the management UI is needed externally, put it behind an authenticated reverse proxy with MFA. GHSA-53mr-6c8q-9789 (privilege escalation via unrestricted proxy configuration endpoint) is only exploitable when the management endpoint is reachable by unauthorized principals. Network isolation eliminates the exploitability of an entire class of advisories.

TLS and transport security

All client-to-gateway connections require TLS 1.3 minimum. For service-to-service calls within a zero trust architecture, implement mutual TLS (mTLS) so both parties authenticate at the transport layer. Cloudflare AI Gateway provides TLS termination at the edge by design. Self-hosted gateways, LiteLLM and Kong, require explicit TLS configuration via your load balancer or ingress controller.

Immutable deployments

Gateway containers should be immutable: no runtime package installs, no writable application directories outside of explicitly mounted paths. Immutability eliminates post-compromise persistence techniques where attackers modify the running container to survive restarts. Combine with image digest pinning (not tag pinning) to prevent tag-to-digest swaps from silently changing what’s running in production.

IAM and Key Management at the Gateway

Gateway IAM has two distinct layers: the provider credentials stored inside the gateway (how the gateway authenticates to providers), and the client credentials issued by the gateway to consuming applications and users. Getting either layer wrong creates different but equally serious exposure paths.

Provider credential storage

Provider API keys must never live in environment variables, config files, or container image layers that reach version control or a container registry. The correct pattern is integration with a secrets manager: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager. LiteLLM supports direct integration with all four. The gateway retrieves credentials at startup and on rotation; key material never touches the filesystem in plaintext.

Kong’s Konnect Config Store provides an analogous pattern: provider credentials are encrypted at rest and referenced by ID in config files, never stored in plaintext. For Portkey VPC deployments, provider credentials stay within the customer’s cloud environment; Portkey never holds them.

Virtual key architecture

LiteLLM’s virtual key system lets you issue scoped keys to applications and teams without exposing real provider credentials. Each virtual key carries constraints: which models it can access, its rate limits (RPM, tokens per minute), a spend budget, a maximum parallel request count, and an optional TTL. A compromised virtual key exposes only what that key was permitted to do, nothing more.

Operator access controls

The gateway admin interface, where virtual keys are managed, budgets configured, and logs accessed, needs its own access control stack. LiteLLM Enterprise supports SSO/SAML. Portkey uses OIDC SSO with RBAC. Kong integrates with enterprise identity providers via OAuth 2.0 and OpenID Connect natively. For any production gateway, the admin interface requires MFA and all access should be logged and reviewed.

Guardrails as Security Controls

Guardrails are often framed as content moderation. For a security practitioner, they’re better understood as a request-and-response inspection layer with policy enforcement capability: input sanitization, PII detection, output validation, and secondary model-based judgment all happen at the gateway before the LLM call and before the response reaches the client.

Input filtering: exact match and semantic layers

Kong AI Gateway ships two complementary prompt inspection plugins. AI Prompt Guard operates on exact string matching: administrators define allow and deny lists and the gateway accepts or blocks based on content match. This catches known attack patterns with low false-positive rates. AI Semantic Prompt Guard adds a topic-level layer, classifying request intent and blocking prompts that fall into restricted topic areas even when the attacker paraphrases. The two layers together cover the narrow-exact and broad-semantic dimensions of the injection surface.

For LiteLLM deployments, guardrail integration is handled via callbacks to third-party providers including Lakera Guard and Aim Security. These provide semantic filtering as external services rather than native plugins. The architectural tradeoff is latency (external API call per request) versus operational simplicity (no additional gateway component to operate).

PII sanitization on both paths

Kong AI PII Sanitizer covers 20 PII categories: SSN, payment card numbers, email addresses, phone numbers, passport numbers, national IDs, and more, across 9 languages. It operates on both the request path (stripping PII before it reaches the model) and the response path (filtering any PII that appears in model outputs before delivery to the client). Portkey’s PII Redaction module provides similar ingress-side functionality with the option to suppress log storage entirely via privacy mode: a compliance-friendly configuration for deployments where prompt content logging creates regulatory risk.

System prompt integrity

One threat to the system prompt is tampering: a middleware component, a misconfigured proxy layer, or a vulnerable gateway instance changes the instructions the developer wrote before they reach the model. One mitigation is HMAC (Hash-based Message Authentication Code) signing: sign the system prompt at configuration time and verify the signature at each LLM call. If the system prompt content has been altered in transit, the verification fails and the call is blocked. This defends against prompt tampering, not against indirect prompt injection (where the injection arrives via user input or retrieved documents while the system prompt stays intact). This pattern requires application-layer implementation coordinated with the gateway; no current platform implements it natively.

Output validation

Kong’s AI Semantic Response Guard and AI LLM as Judge plugins operate after the model responds. The former classifies response content and blocks off-policy outputs. The latter routes the primary model’s output to a secondary model for quality and compliance evaluation before delivery. Both add latency; AI LLM as Judge adds a full additional inference round, making them appropriate for high-risk output paths rather than all-traffic inspection. JSON schema validation is the lowest-overhead output control: if an application expects structured output, enforce the schema at the gateway before the response reaches application code. This eliminates the LLM05 class of vulnerabilities at minimal latency cost.

Observability and Security Monitoring

A gateway that blocks attacks generates no useful security signal unless those blocks are logged, aggregated, and monitored. Security observability for LLM gateways has a different profile than traditional web service observability. Anomalous spend, unexpected model selections, guardrail trigger rates, and geographic access pattern shifts are all security-relevant signals that don’t map to traditional APM metrics like HTTP 5xx rates.

What to log

At minimum, every gateway deployment should produce logs covering: authentication events (successful and failed, with client key IDs not key values), guardrail trigger events (which policy matched, what action was taken), provider routing decisions (which model was called, with which parameters), spend and token consumption per key per interval, rate limit hits, and any admin plane operations (key issuance, budget changes, config modifications). For regulated environments under HIPAA or GDPR, logging must distinguish between metadata (latency, model called, key used) and content (prompt text, response text); both categories carry different retention and access control requirements.

SIEM integration per platform

LiteLLM ships native integrations with Langfuse and Helicone for LLM-specific observability and supports custom callback functions that can route events to any SIEM (Security Information and Event Management system). Kong Konnect provides native Splunk, Elasticsearch, and Datadog log forwarding. Cloudflare AI Gateway exposes logs via Logpush with delivery to Sumo Logic, Splunk, Datadog, Elasticsearch, and S3/GCS/R2 for custom SIEM ingestion. Portkey’s built-in logging exports to enterprise SIEM with configurable retention policies.

Anomaly detection patterns

A sudden increase in guardrail triggers from a specific client key warrants investigation: it may indicate a compromised key being used for injection attempts, or an application bug generating malformed prompts. Spend acceleration above normal baseline by more than 2–3 standard deviations (adjust thresholds to your traffic volume and acceptable false-positive rate) suggests either an Unbounded Consumption attack (LLM10) or a compromised key driving unauthorized model usage. Geographic access shifts (a key that normally calls from a single region appearing in multiple regions simultaneously) are a classic credential compromise indicator. Build these detections explicitly; they won’t surface from generic APM alerting rules.

Platform Security Posture: LiteLLM, Portkey, Kong, Cloudflare

The four major LLM gateway platforms, surveyed across our LLM gateways hub, take meaningfully different approaches to security. The right choice depends on your threat model, compliance requirements, and operational preferences.

For enterprise compliance (HIPAA, SOC2, ISO27001): Portkey is the fastest path to a certified managed deployment. Its VPC private cloud option delivers the compliance controls without requiring data to leave a customer-controlled environment.

For maximum guardrail depth: Kong AI Gateway’s native plugin stack is the most complete inspection layer available in any current gateway. The tradeoff is operational complexity: these plugins require configuration, tuning, and maintenance.

For self-hosted deployments: LiteLLM with the hardened configuration and an aggressive patch cadence is the most flexible option. Its active CVE history is a maintenance commitment, not a disqualifier, but organizations deploying it need a process for tracking advisories and applying patches within a short window of publication.

For edge-layer protection with minimal operational overhead: Cloudflare handles DDoS, rate limiting, and TLS termination without infrastructure to operate. Key security features remain in beta as of mid-2026, so evaluate current release status before planning around them for production compliance use cases.

Compliance Mapping: NIST AI RMF, GDPR, and HIPAA

LLM gateways sit at the intersection of three compliance frameworks that each impose distinct requirements. Organizations operating in regulated industries need to map gateway capabilities to specific controls: not at the product marketing level, but at the control level.

NIST AI Risk Management Framework

The NIST AI RMF (NIST AI 100-1) organizes AI risk management across four functions: Govern, Map, Measure, and Manage. Three subcategories apply directly to gateway deployments.

MEASURE 2.7 (NIST AI 100-1, Table 3) requires that AI system security and resilience, as identified in the MAP function, are evaluated and documented. For a gateway deployment, this maps to: documented threat modeling against the OWASP LLM Top 10 and the gateway’s specific attack surface; penetration testing against the management plane and proxy endpoints; ongoing CVE monitoring with a documented patch process; and periodic security advisory reviews.

MEASURE 2.10 (NIST AI 100-1, Table 3) requires examination and documentation of privacy risk. PII flows through the gateway must be inventoried: which client applications send PII in prompts, which providers receive it, what the logging configuration captures, and what the data subject rights implications are. PII sanitization controls (Portkey PII Redaction, Kong AI PII Sanitizer) should be mapped to specific data categories in the inventory.

GOVERN 1.2 (NIST AI 100-1) requires that the characteristics of trustworthy AI are integrated into organizational policies, processes, procedures, and practices. For a gateway this means: documented API key lifecycle policies, guardrail configuration change management, incident response runbooks covering credential compromise and supply-chain events, and audit logging with defined retention periods and access controls.

GDPR and CCPA

Personal data in prompts must have a legal basis for processing under GDPR. Data subjects have rights to erasure that may require selective log purging. Cross-border transfers, such as prompts routed to US providers from EU-based users, require appropriate safeguards such as Standard Contractual Clauses with providers. Your gateway configuration doesn’t satisfy this alone; provider Data Processing Agreements must be in place separately. For gateways with logging enabled, GDPR’s storage limitation principle requires defined retention periods proportionate to purpose. Portkey’s privacy mode (no content storage) is a compliant configuration for use cases where the legal basis for logging prompts is weak.

HIPAA for healthcare deployments

Protected Health Information in LLM prompts is subject to HIPAA Security Rule technical safeguard requirements: encryption in transit satisfied by TLS 1.3; encryption at rest by AES-256 for any stored PHI-containing logs; access controls satisfied by virtual key RBAC and audit logging; and audit controls by immutable gateway logs with defined access controls. Business Associate Agreements are required with any gateway provider that handles PHI. Portkey holds HIPAA compliance certification for its cloud service; confirm current BAA availability directly with any provider before deployment.

EU AI Act

By July 2026, the EU AI Act is enforcing obligations for providers and deployers of General Purpose AI (GPAI) models and high-risk AI systems. LLM gateways routing to GPAI models (GPT-4, Claude, Gemini) on behalf of EU-based users put the gateway operator in scope as a deployer. Key obligations for deployers include: implementing usage monitoring and logging, maintaining an AI literacy program for employees using AI, conducting fundamental rights impact assessments for high-risk use cases, and establishing a post-market monitoring system for incidents. Article 13 (transparency) and Article 16 (human oversight) are the most directly applicable provisions. Organizations deploying LLM gateways with EU users should log conversations sufficient to support impact assessment reviews and maintain records of which models are in production use and why.