Self-Hosted LLM Running Your Own Models on Your Own Hardware

What Is a Self-Hosted LLM?

The plain definition, the vocabulary that surrounds it, and what "self-hosted" does and does not promise.

A self-hosted LLM is a large language model whose weights you have downloaded and whose inference runs on computers you control. When a user submits a prompt, that prompt travels to your server, is processed by a model file sitting on your storage, and the response comes back without any part of the exchange crossing into a vendor's infrastructure. The alternative, and still the default for most organizations, is calling a hosted API where the model runs on the provider's hardware and your text is transmitted to them for processing.

The vocabulary in this space overlaps heavily, which makes research confusing. To self host an LLM and to run an on prem LLM usually mean the same thing when the hardware is in your building. LLM hosting is a broader phrase that covers both self-managed deployments and rented dedicated capacity. A local LLM typically refers to a model running on a single workstation or laptop for one person, while a self-hosted LLM implies a shared service that multiple users or applications call. LLM as a service describes the opposite model, where a provider handles everything and you consume an endpoint. The distinction that actually matters for compliance is not where the box physically sits, but who has technical access to the data in transit and at rest.

It is worth being precise about what self-hosting does not give you. It does not make a model more accurate, and it does not remove the possibility of a model producing confident nonsense. It does not automatically satisfy a compliance framework, because a self-hosted LLM sitting on an unpatched server with no access logging is a finding waiting to happen. What it does give you is control over the data path, control over model versions so behavior does not change underneath your applications without warning, and freedom from per-token pricing. Those three properties are the entire case for self-hosting, and organizations that need none of them are usually better served by an API.

Petronella Technology Group has been deploying private AI solutions since launching its AI division in 2023, and the pattern that recurs across engagements is that the decision is rarely driven by cost at the start. It is driven by a contract clause, a regulator, or a client who asked a hard question about where their documents go. Cost becomes the argument later, once the workload is large enough that the per-token bill starts to look like a lease payment.

Why Regulated Organizations Self-Host

The four reasons that actually appear in scoping conversations, in the order they usually appear.

For defense contractors specifically, the calculation is not subtle. If a workflow involves controlled unclassified information, the language model processing that information falls inside the assessment boundary, and the security requirements of NIST SP 800-171 apply to it the way they apply to any other system component. Petronella Technology Group is a CyberAB Registered Provider Organization, RPO #1449, and Craig Petronella is a CMMC Registered Practitioner, so private AI deployments for contractors are scoped against the same control set as the rest of the environment rather than treated as a separate side project. The CMMC compliance guide covers how that boundary is drawn.

Self-Hosted, Cloud API, or Managed Private Deployment

Three delivery models, compared on the dimensions that decide the choice.

The middle column is not a trap, and it is worth saying so plainly. For an organization doing marketing drafts and meeting summaries on public information, a cloud API is faster, cheaper, and better. Self-hosting earns its complexity when the data is sensitive, the volume is high, or the version stability matters. When only the first of those is true and you have no appetite to run GPU infrastructure, a managed private deployment usually lands in the right place: the data properties of self-hosting without the requirement to hire for it.

What Actually Serves a Self-Hosted LLM

The inference engine, the model server, and the layers that turn a model file into a service.

A downloaded model is a large file of numbers. The software that loads those numbers onto a GPU and turns prompts into tokens is the inference engine, and the choice of engine has more effect on real-world throughput than most teams expect. Engines differ in how they batch concurrent requests, how they manage the memory used to hold conversation context, and which quantization formats they support. A model that feels slow on one engine can serve several times the concurrent users on another with no change to the hardware.

In practice the field sorts into two categories. High-throughput serving engines such as vLLM, SGLang, and Text Generation Inference are built for many simultaneous users, use continuous batching so that requests do not wait for a batch to fill, and manage context memory efficiently enough to keep GPUs busy. They are what belongs behind a shared internal endpoint. Single-user runtimes such as Ollama and llama.cpp are built for convenience on one machine, run comfortably on consumer hardware, and are ideal for evaluation, developer workstations, and small pilots. Deploying a single-user runtime as a shared inference server is one of the most common self-hosting mistakes, because the deployment works fine in testing with one person and falls over when the tenth user connects.

Above the engine sit the parts that make the service usable. Nearly every serious engine exposes an OpenAI-compatible interface, which matters more than it sounds: it means existing application code, libraries, and tools point at your internal endpoint by changing a base URL, and it means you are not rewriting integrations if you later change engines. In front of that you need a gateway handling authentication against your identity provider, per-user rate limits, request logging for audit and incident response, and routing if you run more than one model. Then a chat interface for staff who are not calling the endpoint from code, and the retrieval layer that connects the model to your documents.

One detail causes more silent failures than any other: tool calling and structured output depend on the engine being configured with the correct parser for the specific model you are running. A model that answers ordinary chat questions perfectly can go completely silent when an application asks it to call a function, and the cause is almost always a parser mismatch in the serving configuration rather than a weak model. Petronella Technology Group runs its own production agents on self-hosted infrastructure, and this class of problem is why deployments include a functional test of the actual application path rather than a "the model responded" check.

What It Takes to Run an LLM Server

Why video memory is the binding constraint, and how quantization changes the arithmetic.

The first question every hardware conversation reduces to is whether the model fits in GPU video memory. If the weights fit, the model runs at a speed people will tolerate. If they do not, the system spills onto system memory or disk and generation slows to the point where users abandon the tool. Video memory, not raw compute, is the constraint that decides which models are available to you.

The arithmetic is approachable. A model's parameter count multiplied by the bytes used per parameter gives a floor for the memory needed to hold the weights. At full sixteen-bit precision each parameter costs roughly two bytes, so a model with eight billion parameters needs somewhere near sixteen gigabytes for weights alone. Quantization reduces the precision of each parameter, and eight-bit or four-bit quantization cuts that requirement substantially, which is why quantized versions of large models can run on hardware that could never hold them at full precision. Quantization costs some quality, and how much depends on the model and the task, so the honest answer is that it must be tested against your own workload rather than assumed.

Weights are only part of the budget. Every concurrent conversation consumes additional memory to hold its context, and that consumption grows with both the number of simultaneous users and the length of the conversations. A configuration sized precisely to the weights will serve one user and then refuse connections. Long-context work, such as analyzing lengthy contracts or full case files, is particularly memory hungry, and it is a workload where teams routinely underestimate the hardware needed. This is the single most common reason a self-hosted pilot succeeds and the production rollout disappoints.

There is no universal recommendation, because the right configuration falls out of the four workload numbers described earlier rather than from a specification sheet. What is consistent is the shape of the tiers. A single professional GPU in a workstation supports evaluation, development, and a small team. Multiple GPUs in a server support a department with real concurrency. Multi-node deployments enter the picture for the largest open-weight models or for organizations serving the whole company. Petronella Technology Group builds custom AI servers and AI workstations specified against a measured workload, and offers GPU server hosting for organizations that want dedicated capacity without a capital purchase or a lead time.

Choosing the Best Local LLM for Your Work

Why the leaderboard is the wrong starting point, and what to evaluate instead.

The open-weight ecosystem now includes several strong families, and the practical shortlist for business deployments generally draws from Meta's Llama models, Alibaba's Qwen models, Mistral's releases, DeepSeek's models, Google's Gemma family, and Microsoft's Phi models. Any question of the form "what is the best local LLM" has a short shelf life, because a new release reshuffles the ranking every few months. This is an argument for building an evaluation process rather than memorizing a recommendation.

Public benchmark scores are a weak signal for business use. They measure performance on academic tasks that rarely resemble summarizing a client intake form, drafting a policy section in your organization's voice, or extracting structured data from a scanned invoice. A model that ranks lower on a leaderboard often performs better on your specific documents, and the only way to discover that is to test candidates on a set of your own real tasks with a known correct answer. Twenty representative examples, scored consistently, will tell you more than any benchmark table.

Four practical criteria tend to decide the choice. The first is licensing, because open weights are not automatically permissive and some licenses restrict commercial use or impose conditions worth reading before you build a product on them. The second is the size-to-hardware fit, since the best model you cannot serve is worse than the good model you can. The third is instruction following and tool-calling reliability, which matters far more than eloquence once the model is embedded in a workflow rather than a chat window. The fourth is context length, which needs to comfortably exceed the documents you intend to feed it.

A useful pattern is to run more than one model. A small, fast model handles classification, routing, and short extraction tasks cheaply, while a larger model handles the work that genuinely needs reasoning. Because both sit behind the same internal endpoint, applications choose per request, and the overall system stays responsive without paying for the large model on every trivial call.

Retrieval, Fine-Tuning, and Where the Value Actually Comes From

A self-hosted model that has never seen your data is a slower version of a public chatbot.

The disappointment that follows many self-hosting projects has a predictable cause. The team stands up an inference server, opens a chat window, asks a general knowledge question, gets a mediocre answer, and concludes the open-weight model is not good enough. The model was never the point. Business value comes from connecting the model to information it could not otherwise have: your policies, your contracts, your ticket history, your procedures, your client files.

Retrieval augmented generation is the mechanism for that. Documents are indexed, the system finds the passages relevant to a question, and those passages are supplied to the model alongside the question so the answer is grounded in your material and can cite where it came from. This is what turns a general model into something that answers questions about your organization accurately, and it is why a well-built retrieval layer over a mid-sized model outperforms a much larger model working from memory alone. Retrieval also keeps information current without retraining, since updating a document updates the answers.

Fine-tuning solves a different problem. It adjusts the model's weights to shift its behavior, which is the right tool for teaching a consistent output format, a house style, or a specialized vocabulary the base model handles poorly. It is the wrong tool for teaching facts, because fine-tuned knowledge is difficult to update and impossible to cite. The sequence that works is retrieval first, measure, then fine-tune only if a specific behavioral gap remains. Petronella Technology Group provides RAG implementation services and LLM fine-tuning services, and the comparison between the two approaches is covered in more depth on the RAG versus fine-tuning page.

One consideration specific to regulated environments deserves attention. Retrieval systems must respect existing document permissions. If the index contains material a given user is not entitled to read, a well-phrased question can surface it, and the retrieval layer becomes an access control bypass rather than a productivity tool. Permission-aware retrieval needs to be designed in from the start, because retrofitting it onto a live index is considerably harder than building it correctly the first time.

Securing and Governing a Self-Hosted LLM

Self-hosting moves the risk inside your boundary. It does not remove it.

A self-hosted LLM is a server, and it inherits every ordinary server obligation: patching, hardening, network segmentation, backup, and monitoring. It also introduces obligations of its own. Inference endpoints have shipped with authentication disabled by default often enough that internet-exposed, unauthenticated model servers are a recurring finding in security research. An endpoint like that is both a free compute resource for whoever finds it and, more seriously, a window into whatever documents the retrieval layer indexes.

Prompt injection is the risk class that most distinguishes AI systems from conventional applications. When a model reads content, that content can contain instructions, and a model has no reliable way to separate the instructions it was given from instructions embedded in the material it was asked to summarize. A document, an email, or a web page can therefore attempt to redirect a model's behavior. The mitigations are architectural rather than clever prompting: treat all retrieved content as untrusted input, constrain what the model is allowed to do with the tools it can call, require human confirmation before consequential actions, and log every tool invocation. The LLM security page covers this class of attack in detail.

Governance is the other half. A self-hosted deployment should be covered by a written policy setting out what may and may not be submitted, which data classifications are permitted, how outputs must be reviewed before they are relied on, and who approves new use cases. Model provenance belongs in that policy too, since weights downloaded from a public repository should be verified and pinned rather than pulled fresh at each restart. For organizations formalizing this, the AI governance framework and ISO 42001 certification pages describe how these practices are structured, and ComplianceArmor® produces the policy and evidence documentation that assessors ask for.

For healthcare organizations, a self-hosted model handling protected health information sits squarely inside HIPAA scope, and the same risk analysis, access control, and audit logging requirements apply as they would to an electronic health record system. Craig Petronella is the author of How HIPAA Can Crush Your Medical Practice and a licensed North Carolina Digital Forensics Examiner, and the HIPAA-compliant AI page covers what that scoping involves.

What LLM Hosting Costs, and When Self-Hosting Wins

How to run the comparison honestly instead of comparing a purchase price to a monthly bill.

Cost comparisons in this space are usually wrong in the same direction, because a hardware purchase gets compared against an API invoice without accounting for everything else self-hosting requires. An honest comparison puts five things on the self-hosted side of the ledger: the hardware, amortized over a realistic useful life rather than treated as a single year's expense; power and cooling, which are meaningful for GPUs running continuously; the staff time to build and then operate the system; the software and monitoring around it; and the redundancy needed if the service is business critical, because one server is a single point of failure.

On the API side, the number that matters is not today's spend during a pilot. It is projected spend at the adoption level you actually want. Multiply realistic per-user usage by the number of people who will use the tool once it works, then note that reasoning-heavy models and long documents consume tokens far faster than a chat conversation does. Many organizations discover that the pilot invoice, multiplied out to full deployment, changes the conclusion entirely.

The structural difference is that one cost is variable and one is fixed. API spend rises with every additional user and every additional use case, so success increases the bill. Self-hosted capacity costs the same whether it is idle or saturated, so the effective cost per request falls as adoption grows, and the incentive is to use the system more rather than less. Where the crossover lands depends entirely on volume, which is why the useful exercise is estimating your own break-even point rather than looking for a general rule.

There is also a cost that appears on neither side of a spreadsheet: what it costs to be unable to use AI on your most valuable data. For a defense contractor, a hospital, or a law firm, the documents where a language model would deliver the most value are precisely the documents that cannot be sent to a public API. Self-hosting is often less a cost optimization than the only route to using the capability at all, and organizations that recognize this early tend to scope the project better than those that arrive at it through a billing dispute.

How Petronella Technology Group Deploys a Self-Hosted LLM

A sequence built to avoid the two common failure modes: buying hardware too early, and shipping a system nobody can audit.

Petronella Technology Group has been securing regulated businesses from Raleigh, North Carolina since April 2002, holds a BBB A+ rating dating to 2003, and operates as a CyberAB Registered Provider Organization under RPO #1449. The team runs its own AI agents on self-hosted infrastructure rather than public APIs, which means private AI engagements are informed by operating this class of system daily rather than only by deploying it for others. Craig Petronella is MIT-certified in artificial intelligence and cybersecurity, has been featured as a cybersecurity commentator on NBC, ABC, CBS, FOX, and WRAL, and hosts the Encrypted Ambition podcast.

Self-Hosted LLM Questions

The questions that come up most often in scoping conversations.

Explore Private AI at Petronella