How to Run Open Source LLMs Locally (2026)

01 / The story

Open source, open weights, and local are different

The phrase “open source LLM” is often used loosely. Some models publish weights under custom licenses; others use more permissive licenses. Ollama’s software license does not replace a downloaded model’s license. Check the exact checkpoint, license, acceptable-use policy, and redistribution conditions.

This guide keeps the requested Llama 3, Mistral, and Phi families as practical examples rather than presenting them as the newest or universally best models of 2026. A newer release can be larger, require a different license, or run worse on your hardware.

02 / The story

1. Install Ollama and start small

Use the official Ollama download and operating-system instructions, not an unknown installer mirror or a copied privileged shell command. Check the supported GPU documentation if you intend to use acceleration. After installation, test a small local text model before downloading a much larger one.

These commands are documented by the library’s model pages. Downloading weights needs a network connection, storage, and acceptance of the model’s terms. Do not append a cloud model tag when your goal is local inference.

ollama run llama3.2:1b
# Alternative local models to try separately:
ollama run mistral
ollama run phi4
# Inspect downloaded models:
ollama list
# Inspect what is currently loaded:
ollama ps

03 / The story

2. Plan storage, memory, and context separately

The checked Ollama listings show approximately 1.3 GB for llama3.2:1b, 2.0 GB for llama3.2:3b, 4.4 GB for mistral’s default 7B package, and 9.1 GB for phi4’s 14B package. These are listed package sizes, not guaranteed runtime memory requirements. Different tags and quantization choices can change them.

Weights, the key/value cache used during generation, runtime overhead, and your operating system all need memory. Longer context and multiple simultaneous requests increase the load. GPU VRAM and ordinary RAM are not interchangeable, though some systems can split work or share memory. Verify the chosen backend.

For rough weight storage only, parameter count × bits per parameter ÷ 8 gives bytes. A hypothetical 7-billion-parameter model at 4 bits is about 3.5 billion bytes before overhead. This is an illustration, not a promise that a 3.5 GB GPU can run it.

04 / The story

3. Use the loopback API without exposing it

Ollama documents its local API on port 11434. The request below uses a local model and disables streaming so a developer can inspect one JSON result. Only run it after the server is installed and the model is downloaded. It is not a hosted Qlauson endpoint.

Keep the service on loopback unless you have deliberately designed access controls. Binding it publicly is not a harmless convenience. A website or application that forwards requests needs its own authorization, input limits, network controls, and spending or compute limits.

curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.2:1b","stream":false,"messages":[{"role":"user","content":"Explain local inference in two sentences."}]}'

05 / The story

4. Check privacy and licenses before work data

Ollama’s FAQ distinguishes local inference from cloud features and documents disabling cloud features. Verify your settings and model choice, and remember that your own application can still send data through telemetry, plugins, remote retrieval, backups, or logs. “Runs locally” is not a complete privacy audit.

The Phi-4 repository’s checked license is MIT. The checked Mistral-7B-Instruct-v0.3 model card lists Apache-2.0; that does not describe every Mistral product. Llama uses a model-specific community license rather than the same permissive terms. Review the exact model card and license before redistribution or commercial deployment.

06 / The story

5. Compare full local cost with an API

Local inference avoids per-token provider charges for that local run, but hardware, electricity, maintenance, updates, and your time are not free. Measure your own power use and realistic throughput before buying hardware. Small models may also need more correction work.

Compare the cost per acceptable task using the same inputs, quality requirements, and context length. A hosted API can be easier for occasional work or higher-quality requirements; local inference can fit offline or controlled-data use. Neither automatically wins. No 2027 hardware-price or model-performance claim is made here.