What does a private LLM cost to run, beyond the model?

Running a private LLM beyond the model itself means paying for GPU or managed-service compute, inference tokens, storage, monitoring, and governance overhead. GPU hours on RunPod range from $0.58 per hour for a 16 GB class worker to $8.99 per hour for a B300, according to RunPod's GPU Cloud Pricing page read in October 2026. Token fees and the often-invisible costs of governance and change management add further layers that most teams underestimate.

What do the published compute numbers actually tell you?

The table above shows a wide spread even within a single GPU tier. RunPod lists three H100 pod configurations between $2.89 and $3.99 per hour as of October 2026, while CoreWeave lists an HGX H100 node of eight GPUs at $49.24 per hour on demand in North America, about $6.16 per GPU (our arithmetic). Lambda's reserved H100 pricing runs from $5.54 to $6.16 per GPU per hour depending on cluster size, with 256-GPU clusters priced lower per GPU than 16-GPU clusters.

Token costs on managed inference services add a separate line to the bill. On Together AI, MiniMax M3 input tokens cost $0.30 per million while Kimi K3 output tokens cost $15.00 per million. On AWS in US East and US West regions, Gemma 4 31B input tokens are listed at $0.14 per million and output at $0.40 per million, while DeepSeek v3.2 output tokens are listed at $1.85 per million. Model choice matters as much as provider choice.

These are list prices read on a single date and they change. The arithmetic below is Brand Media Works' own worked example using published figures, not a vendor claim. A team running a single RunPod H100 PCIe pod for 8 hours per working day, 22 days per month, would spend roughly 176 compute hours at $2.89 per hour, totalling approximately $509 in GPU time alone before tokens, storage, or any monitoring overhead. That is a floor, not a ceiling.

What costs sit beyond compute and tokens?

Storage, monitoring, and governance are the layers that rarely appear in a vendor's headline rate. For private AI for business deployments, data residency rules create real storage obligations. AWS states that when cross-region inference is enabled, retained inputs and outputs land in destination regions, not necessarily the originating region.

If cross-region inference is enabled for these models, retained inputs and outputs are stored in destination regions (i.e., the region where your inference request is processed).

AWS also specifies retention windows that add compliance overhead. For models requiring internal review, the retention period is up to 30 days within the AWS boundary. A brand handling customer creative briefs, product descriptions, or pricing data needs to account for where those prompts sit and for how long. That accounting is a governance cost, not a compute cost.

Access control is another line item that surfaces only at implementation: single sign-on, roles by user and sign-off from IT and security take engineering time whatever platform a brand chooses. Seeing, controlling, and accounting for AI token spend at scale requires tooling that does not ship in the base model price.

Why do so many teams underestimate the total private LLM bill?

Most founders and growth leaders focus on token cost because it is the most visible number in a vendor dashboard. EY's September 2026 analysis of agentic AI puts that number in proportion:

By our estimate, the token accounts for roughly a third of what an agent actually costs to run.

The same EY analysis puts the full enterprise cost of AI at roughly three times the token invoice once infrastructure, software, governance, organizational change, expected failure and regulatory costs are included. Adoption work and change management belong in the budget from the start.

A practical budget treats total cost of ownership, technical expertise and data governance as separate lines from the start, rather than as afterthoughts surfaced by the first quarterly invoice. Most savings come from architecture decisions made before the first prompt runs, such as model size and when to use a managed endpoint, not from negotiating a better rate afterwards.

Teams considering private LLMs on open-weight models need a compute budget, a token budget, a storage and compliance budget, and a governance budget. When self-hosting clears the cost bar against a managed service is the question that self-hosted LLM vs ChatGPT Enterprise for a marketing team works through in detail.

Brand Media Works, an AI consulting and marketing data engineering firm for ecommerce brands, works through this cost structure with brands before any infrastructure is provisioned.

Book a discovery call

Frequently asked questions

Is GPU compute always the largest cost in a private LLM deployment?

Not necessarily. EY puts the full enterprise cost of AI at roughly three times the token invoice once infrastructure, governance, organizational change and regulatory costs are counted, so access controls, monitoring and retention policies need their own budget lines. Token fees on high-output models are a further variable outside the GPU bill.

Does data leave my environment when I use a managed inference service?

It depends on the provider's terms and the model. AWS states that retained inputs and outputs are stored and processed by AWS and are not shared with third-party model providers, but the destination region may differ from your originating region when cross-region inference is active. Always confirm the specific model's retention rules before processing sensitive data.

Can a smaller GPU tier handle most DTC brand AI workloads?

Sometimes. Smaller open models can run on 16 GB class GPUs, which RunPod lists at $0.58 per hour as of October 2026, but whether they are good enough depends on the task, so test on your own catalog and copy before sizing. Larger context windows, multi-modal inputs and concurrent high-traffic jobs typically need 80 GB or larger GPUs.