Run Ollama in Docker: CPU, NVIDIA, AMD and Intel GPUs, Docker Compose With Open WebUI, Windows 11, Kali, Mac, Models, Updates and Fixes

To run Ollama in Docker, start the official ollama/ollama image with a volume for your models and port 11434: docker run -d -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollama, then load a model with docker exec -it ollama ollama run llama3.2. For an NVIDIA GPU, install the NVIDIA Container Toolkit and add --gpus=all. For AMD, use the ollama/ollama:rocm image with --device /dev/kfd --device /dev/dri. Intel and other GPUs work through Vulkan, which is built into the main image. On Windows 11, Docker Desktop passes through NVIDIA cards only, through WSL 2. On a Mac, Docker has no GPU at all, so the native app is much faster. One detail most guides skip: the image listens on every network address, so the plain -p 11434:11434 opens your models to the whole network. Below: Docker Compose with Open WebUI and a GPU, Windows and Kali steps, models and Modelfiles in a container, environment variables, updating and pinning versions, AWS, and the errors you are likely to hit.

Jake's computer shop has a spare tower in the back room with an RTX 3060 in it, and he wanted a private chat assistant for the staff: something that answers "what's the warranty on this model?" from the shop's own notes, with nothing leaving the building. Ethan could have installed Ollama directly. He chose Docker instead, for three reasons he explained over coffee: the whole setup would live in one file, it would come back by itself after a power cut, and an update would be one command that could be undone. This page is that setup, plus every wall he hit on the way.

New to running AI models on your own computer? Our free guide series explains models, memory and quantization in plain English, and the cards below cover the rest of the background.

Do you need Docker to run Ollama?

No. Ollama is an ordinary program for Windows, macOS and Linux, and on a laptop the native install is the simplest and usually the fastest choice. Docker is a way of packaging programs into containers: sealed boxes that carry everything the program needs and run the same way on any machine with Docker. If the idea is new, our explainer on containers vs virtual machines takes five minutes.

So "Ollama or Docker" is not a choice between two tools that do the same job. Docker is the box; Ollama is what you put in it. The real question is whether the box helps you:

If you are still choosing a local AI tool in the first place, our Ollama vs LM Studio comparison helps. The rest of this page assumes you have picked Ollama and want it in a container.

Jake: "So Docker doesn't make the AI smarter or faster?"

Ethan: "Neither. It makes it tidy. Same model, same speed on Linux with the GPU passed through, but the whole thing lives in one file I can rebuild in a minute."

The official Ollama Docker image: tags, size and versions

The official image is ollama/ollama on Docker Hub, published by the Ollama team. It is the only one you need; the many community images that wrap Ollama tend to fall behind. Its tags follow a simple pattern:

The image comes for both amd64 (ordinary PCs and most servers) and arm64 (AWS Graviton, Ampere servers, a Raspberry Pi 5), and Docker picks the right one automatically.

Why is the Ollama Docker image so big? The main image is about 3.5 GB compressed on amd64, before any model. It carries the GPU libraries Ollama needs, mainly NVIDIA's CUDA runtime libraries plus Vulkan, so you never install them yourself; that is the price of "add --gpus=all and it works." The AMD rocm image is smaller to download, about 1.35 GB compressed. Models are separate and much larger: they live in the volume, not the image, so updating the image never re-downloads them.

Run Ollama in Docker: the basic command

Start here even if you have a GPU; it proves Docker and the image work before you add drivers to the picture. You need Docker installed: Docker Desktop on Windows or Mac, or Docker Engine on Linux (on Kali, sudo apt install -y docker.io and sudo systemctl enable docker --now).

docker run -d --name ollama --restart unless-stopped \
  -v ollama:/root/.ollama \
  -p 127.0.0.1:11434:11434 \
  ollama/ollama

What each part does:

• -d runs it in the background, and --name ollama gives the container a name you can use in later commands.
• --restart unless-stopped brings it back after a reboot or a crash, unless you stopped it yourself.
• -v ollama:/root/.ollama keeps your models in a Docker volume named ollama. Without it, every model disappears when you remove the container.
• -p 127.0.0.1:11434:11434 makes Ollama's port reachable from this computer only. The security section explains why the 127.0.0.1: part matters.

Check that it answers:

curl http://localhost:11434
# Ollama is running

Now load a model and chat with it, using the ollama command inside the container:

docker exec -it ollama ollama run llama3.2

The first run downloads the model (about 2 GB for llama3.2) into the volume; later runs start in seconds. Type /bye to leave the chat. Any model from Ollama's library works the same way; swap the name.

If you already have Ollama installed on the same computer, its own ollama command talks to whatever answers on port 11434, which is now the container. That is handy: ollama list on the host shows the container's models. It is also why the native Ollama app and the container cannot both run on 11434 at once; more on that in the errors table.

Ollama in Docker with an NVIDIA GPU (Linux, Kali and WSL 2)

A container cannot see your graphics card unless Docker hands it over. For NVIDIA cards, the piece that does that is the NVIDIA Container Toolkit. You need three things on the host, in this order:

1. The NVIDIA driver. Check with nvidia-smi; it should print your card and a driver version. On Kali, install the driver with sudo apt install -y nvidia-driver nvidia-cuda-toolkit and reboot.
2. The NVIDIA Container Toolkit. On Ubuntu, Debian and Kali, add NVIDIA's repository and install it:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

On Fedora, RHEL, Rocky and Amazon Linux, use the RPM repository instead:

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo \
  | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo yum install -y nvidia-container-toolkit

3. Tell Docker about it, then restart Docker:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Before involving Ollama, prove that Docker can reach the GPU at all. This throwaway container should print the same table as nvidia-smi on the host:

docker run --rm --gpus all ubuntu nvidia-smi

If that fails, the problem is the toolkit or the driver, and no Ollama setting will fix it. If it works, start Ollama with the GPU (remove the CPU container first with docker rm -f ollama; the volume and your models stay):

docker run -d --name ollama --restart unless-stopped --gpus=all \
  -v ollama:/root/.ollama \
  -p 127.0.0.1:11434:11434 \
  ollama/ollama

To confirm the model really runs on the card, load one and look at the PROCESSOR column:

docker exec -it ollama ollama run llama3.2 "hello"
docker exec ollama ollama ps
# NAME             ID              SIZE      PROCESSOR    UNTIL
# llama3.2:latest  a80c4f17acd5    4.0 GB    100% GPU     4 minutes from now

100% GPU is what you want. A split such as 45%/55% CPU/GPU means the model did not fit in your card's memory and part of it runs on the processor, which is much slower. 100% CPU means the container cannot see the GPU. docker logs ollama shows what Ollama found when it started; our Ollama GPU guide explains how to read that log and fix each case.

To give the container only some of your cards, replace --gpus=all with --gpus '"device=0"' (or "device=0,1"). On an NVIDIA Jetson board, also pass -e JETSON_JETPACK=5 or 6 to match your JetPack version.

Jake: "It said 100% GPU. Is that good?"

Ethan: "That's the whole game. Your 3060 has 12 GB, so anything up to about 10 GB of model runs entirely on it. Bigger than that and it starts borrowing from the CPU."

AMD and Intel GPUs: the ROCm image and Vulkan

AMD Radeon on Linux. Use the rocm image and pass the two devices ROCm needs, /dev/kfd (the compute interface) and /dev/dri (the graphics devices):

docker run -d --name ollama --restart unless-stopped \
  --device /dev/kfd --device /dev/dri \
  -v ollama:/root/.ollama \
  -p 127.0.0.1:11434:11434 \
  ollama/ollama:rocm

The host needs a current AMD driver; the container brings its own ROCm libraries. Cards outside ROCm's supported list, such as many RX 6600 and RX 5700 models, can still work through Vulkan in the main image (next paragraph). On Fedora and other systems with SELinux, the container may be blocked from the devices until you run sudo setsebool -P container_use_devices=1.

Intel Arc, Intel integrated graphics and other GPUs (Vulkan). The main ollama/ollama image includes Vulkan support, which turns on automatically when the container can see a GPU device. Intel GPUs appear under /dev/dri, so pass only that:

docker run -d --name ollama --restart unless-stopped \
  --device /dev/dri \
  -v ollama:/root/.ollama \
  -p 127.0.0.1:11434:11434 \
  ollama/ollama

Copying an AMD example with --device /dev/kfd onto an Intel or NVIDIA machine fails at once, because that device file only exists with an AMD driver. If a machine has several GPUs and you want Vulkan to use only some of them, set -e GGML_VK_VISIBLE_DEVICES=0; to switch Vulkan off, set -e OLLAMA_VULKAN=0.

Vulkan is usually slower than CUDA on NVIDIA or ROCm on supported AMD cards, but it is far faster than the processor alone, and it is the only GPU route for most Intel graphics.

Ollama in Docker on Windows 11 (Docker Desktop and WSL 2)

On Windows, Docker Desktop runs containers inside a small Linux system managed by WSL 2. Three things follow from that.

1. GPUs: NVIDIA only. Docker Desktop passes NVIDIA cards into containers through WSL 2's GPU support. You need the WSL 2 backend turned on in Docker Desktop's settings (it is the default), a current NVIDIA driver installed in Windows (nothing to install inside WSL), and an up-to-date WSL kernel:

wsl --update

Then the same command works in PowerShell, written on one line:

docker run -d --name ollama --restart unless-stopped --gpus=all -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 ollama/ollama

AMD and Intel GPUs are not passed through by Docker Desktop on Windows. If you have one, the native Windows app (which uses ROCm or Vulkan directly) will be much faster than a container on the processor.

2. Memory comes from WSL. The container can only use the memory WSL 2 gives it, which by default is half of your PC's RAM. A model that loads fine in native Ollama can fail in Docker with "model requires more system memory." To raise the limit, create or edit %UserProfile%\.wslconfig:

[wsl2]
memory=24GB

Run wsl --shutdown, restart Docker Desktop, and the new limit applies. Our "requires more system memory" guide covers the other causes of that message.

3. Volumes beat Windows folders. A named volume such as ollama lives inside WSL's own disk and is fast. Mounting a Windows folder instead (-v D:\ollama:/root/.ollama) works, but every read crosses from Linux into Windows, and loading a multi-gigabyte model from it is noticeably slower. Use a named volume unless you need to see the files in Explorer.

Two smaller notes. If the native Ollama app is running in your system tray, it already holds port 11434, so quit it first (right-click the tray icon, then Quit). And on Windows 10, if model downloads inside WSL crawl, a known fix is turning off "Large Send Offload Version 2" (IPv4 and IPv6) in the advanced properties of the vEthernet (WSL) network adapter.

Ollama in Docker on a Mac: why it runs on the CPU

Docker Desktop on macOS runs containers inside a Linux virtual machine, and that machine has no access to the Mac's GPU. So Ollama in Docker on a Mac, even an M4 Max, runs entirely on the processor. It works, but it is several times slower than the native app, which uses Apple's Metal GPU and the unified memory directly.

For a Mac, install the native app from Ollama's website (macOS Sonoma 14 or later) and keep Docker for the parts that do not need the GPU, such as Open WebUI. Point Open WebUI at the native Ollama with http://host.docker.internal:11434, which on Docker Desktop always means "the Mac this container runs on."

Docker's own Docker Model Runner handles this differently: on macOS and Windows it runs its model engine outside a container, so it is not boxed in the same way. It is a separate product with its own commands (docker model run); the comparison is near the end of this page.

Docker Compose: Ollama plus Open WebUI, with an NVIDIA GPU

Most people who search for Ollama in Docker want a chat page in the browser, and Open WebUI is the usual choice. Docker Compose runs both from one file, starts them in the right order, and lets them find each other by name. Save this as compose.yaml in an empty folder:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=30m
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      - ollama
    ports:
      - "127.0.0.1:3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open-webui:/app/backend/data

volumes:
  ollama:
  open-webui:

Start it, load a model, and open the chat page:

docker compose up -d
docker compose exec ollama ollama pull llama3.2
# then open http://localhost:3000 and create the first (admin) account

The key line is OLLAMA_BASE_URL=http://ollama:11434. Inside Compose's network, each service is reachable by its name, so Open WebUI finds Ollama at ollama, not at localhost (inside a container, localhost means the container itself). That one word is behind most "Open WebUI cannot connect to Ollama" reports; our could not connect to Ollama guide covers the other setups, such as Open WebUI in Docker with Ollama installed natively.

No NVIDIA card? Delete the whole deploy: block for CPU only. For AMD, use the ROCm image and list the devices instead:

  ollama:
    image: ollama/ollama:rocm
    devices:
      - /dev/kfd
      - /dev/dri

For Intel or other Vulkan GPUs, keep ollama/ollama:latest and list only /dev/dri.

Day-to-day commands, run from the same folder: docker compose ps (status), docker compose logs -f ollama (live log), docker compose stop and docker compose start. docker compose down removes the containers but keeps both volumes, so your models, chats and accounts survive. Only docker compose down -v deletes the volumes, and with them everything you downloaded.

Open WebUI also publishes an all-in-one image, ghcr.io/open-webui/open-webui:ollama, with Ollama built in. It is convenient for a quick try, but the two-service file above is easier to update and debug, because each part has its own logs and its own version.

Jake: "So if the power goes out overnight?"

Ethan: "Docker starts on boot, and both containers have restart unless-stopped. You'll walk in, open the page, and it's there."

Pull, list and remove models in the container

Everything you know from the ollama command works inside the container through docker exec:

The -it flags are needed only for interactive chat. For a one-off command in a script, leave them out, or the script may complain that it is not running in a terminal.

The same works over the HTTP API from the host, which is what apps use: curl http://localhost:11434/api/tags lists models, and this downloads one without entering the container:

curl http://localhost:11434/api/pull -d '{"model": "llama3.2"}'

Where are the model files? In the volume. On Linux, a named volume called ollama lives at /var/lib/docker/volumes/ollama/_data; on Docker Desktop it sits inside Docker's own virtual disk. docker system df -v shows how much space each volume takes, which is worth checking: a few large models easily pass 50 GB.

Reusing models you already downloaded natively. Mount the existing folder instead of a named volume. On Linux, the standard native install keeps models in /usr/share/ollama/.ollama; on Windows, in C:\Users\YourName\.ollama:

docker run -d --name ollama --gpus=all -v /usr/share/ollama/.ollama:/root/.ollama -p 127.0.0.1:11434:11434 ollama/ollama

Stop the native Ollama service first (sudo systemctl stop ollama), so the two do not write to the same folder at the same time.

Models that download on startup, and images with a model built in

A common question: "Why does ollama pull fail inside my Dockerfile?" Because during an image build, no Ollama server is running, and the ollama command is only a client that talks to a server. The pull has nothing to talk to. Two patterns fix it, depending on what you want.

Pattern 1: pull at startup (recommended). Keep the image stock and pull models into the volume after the container starts. With Compose, a small second service can do it once and exit:

  ollama-pull:
    image: ollama/ollama:latest
    depends_on:
      - ollama
    environment:
      - OLLAMA_HOST=ollama:11434
    entrypoint: ["/bin/sh", "-c", "sleep 5 && ollama pull llama3.2"]
    restart: "no"

Setting OLLAMA_HOST tells this second copy of the ollama command to use the server in the ollama service. Because the model lands in the shared volume, it is downloaded once and then reused.

Pattern 2: bake the model into the image. For offline machines or air-gapped networks, start a temporary server during the build:

FROM ollama/ollama:0.40.6
RUN ollama serve & sleep 5 && ollama pull llama3.2

The image grows by the size of the model, and here is the trap: if you then run it with an empty bind mount (a host folder) on /root/.ollama, that folder hides the baked-in models, and the container looks empty. An empty named volume behaves differently: Docker copies the image's files into it the first time, so the models appear. If a baked model "vanished," check which kind of mount you used.

Custom models and Modelfiles in Docker

A Modelfile lets you give a model a fixed system prompt, settings such as context length, or point Ollama at a GGUF file you downloaded yourself. In Docker, the only extra step is getting the file into the container. The simplest way is to mount a folder:

docker run -d --name ollama --gpus=all \
  -v ollama:/root/.ollama \
  -v "$HOME/ollama-files":/files \
  -p 127.0.0.1:11434:11434 ollama/ollama

Put a Modelfile in ~/ollama-files, for example a shop assistant built on a small model:

FROM llama3.2
PARAMETER num_ctx 8192
PARAMETER temperature 0.3
SYSTEM "You are the assistant for a small computer repair shop. Answer briefly. If you are not sure, say so."

Then build and run it:

docker exec ollama ollama create shop-helper -f /files/Modelfile
docker exec -it ollama ollama run shop-helper

To import a GGUF file, put it in the same folder and write FROM /files/your-model.gguf as the first line. Without a mounted folder, docker cp Modelfile ollama:/tmp/Modelfile copies a single file in, and ollama create then reads it from /tmp.

The ollama create step stores the result in the volume, so the custom model survives container updates like any other.

Environment variables for Ollama in Docker

Ollama is configured with environment variables, and in Docker you set them with -e NAME=value on docker run, or under environment: in Compose. The ones worth knowing:

Environment variables are fixed when a container is created. To change one, remove the container and run it again with the new value (docker rm -f ollama, then your docker run command); with Compose, edit the file and run docker compose up -d, which recreates only what changed. Your models stay in the volume either way.

Behind a company proxy that inspects HTTPS with its own certificate, downloads fail with certificate errors. The fix is a tiny custom image that trusts your company's certificate:

FROM ollama/ollama
COPY my-ca.pem /usr/local/share/ca-certificates/my-ca.crt
RUN update-ca-certificates

Keep it private: port 11434 and the 0.0.0.0 default

This is the part most Ollama Docker guides get wrong, and it matters. Inside the official image, Ollama is set to listen on 0.0.0.0:11434, meaning every network address. That is necessary, because Docker's port forwarding arrives from outside the container. But it means the usual example, -p 11434:11434, publishes Ollama on every network interface of your computer: anyone on your Wi-Fi, office network or, on a cloud server, the internet can reach it.

And Ollama's API has no password. Anyone who can reach port 11434 can run your models on your hardware, download new ones to your disk, and delete the ones you have. Exposed Ollama servers are regularly found by internet scanners for exactly this reason.

Two habits fix it:

• Bind to this computer only: -p 127.0.0.1:11434:11434, as every command on this page does. Programs on the same machine, and other containers on the same Compose network, still reach it.
• On Linux, Docker bypasses the ufw firewall. Published Docker ports get their own firewall rules, and those take effect before ufw's. So "I published 11434 but ufw blocks it" is not true: the port is open. Binding to 127.0.0.1 is the reliable fix.

When other computers on your network do need to use it, publish it on one specific address (-p 192.168.1.10:11434:11434) and limit who can reach that address, or better, put a reverse proxy with a password in front, such as Caddy or nginx with basic authentication. For Open WebUI, people only need port 3000 (Open WebUI has its own accounts), and Ollama can stay private behind it.

Jake: "Wait, so with the normal command, the customer Wi-Fi could use our AI?"

Ethan: "And delete it. That's why every command I typed says 127.0.0.1. The staff use the chat page, and the chat page talks to Ollama inside Docker's own network."

Update Ollama in Docker, and pin a version

Updating a container means replacing it with one made from a newer image. Your models are in the volume, so they are untouched. With docker run:

docker pull ollama/ollama
docker rm -f ollama
docker run -d --name ollama --restart unless-stopped --gpus=all -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 ollama/ollama
docker exec ollama ollama -v

With Compose, from the folder with compose.yaml:

docker compose pull
docker compose up -d

Compose recreates only the containers whose image changed. Old images stay on disk until you remove them with docker image prune, which frees several gigabytes after a few updates.

When do you need to update? New model families often need a new Ollama version. If you pull a brand-new model and get an error saying it requires a newer version of Ollama, the image is behind; update and pull again.

Pinning and rolling back. latest moves every time Ollama releases. On a server you care about, use an exact tag such as ollama/ollama:0.40.6 in your command or Compose file. Then nothing changes until you edit the number, and rolling back is just putting the old number back and running docker compose up -d. Release candidates such as 0.40.7-rc0 are there when a new model needs the very latest build before it is final.

Ollama in Docker on an AWS GPU instance

If your own hardware is too small, a cloud GPU server runs exactly the same container. On AWS, the usual choice for one mid-sized model is a G6 instance: a g6.xlarge has one NVIDIA L4 card with 24 GB of GPU memory, 4 vCPUs and 16 GiB of RAM, for roughly $0.80 an hour on demand in US East (N. Virginia). A g5.xlarge with an A10G (also 24 GB) is the older alternative. Both are billed only while running, so stopping the instance stops the GPU charge; the EBS disk holding your models costs a little until you delete it.

1. Launch the instance from the AWS Deep Learning Base GPU AMI (Ubuntu 24.04), which comes with the NVIDIA driver, Docker and the NVIDIA Container Toolkit installed. On a plain Amazon Linux 2023 image, install Docker yourself with our Docker on Amazon Linux 2023 guide and add the toolkit from NVIDIA's RPM repository above.
2. Give it a root disk big enough for your models: 100 GB is a comfortable start.
3. In the security group, allow SSH from your own IP address only. Leave port 11434 closed.
4. Run the NVIDIA command from this page, with 127.0.0.1: in the port mapping.
5. From your own computer, reach it through an SSH tunnel, so Ollama is never exposed to the internet:

ssh -i my-key.pem -L 11434:localhost:11434 ubuntu@203.0.113.25

Replace 203.0.113.25 with your instance's public IP address, shown in the EC2 console. While that window is open, http://localhost:11434 on your computer is the Ollama on the instance, and apps like Open WebUI on your machine can use it as if it were local. When you are done for the day, stop the instance from the console or with aws ec2 stop-instances.

For a team service rather than one person's experiments, the same image runs on Amazon ECS with GPU container instances, and our Amazon ECS explainer is the place to start.

Common Ollama Docker errors and fixes

When none of these match, the log almost always says why: docker logs ollama, or docker compose logs ollama, then restart with -e OLLAMA_DEBUG=1 for more detail.

Ollama in Docker vs Docker Model Runner

Docker now has its own way to run models, Docker Model Runner, built into Docker Desktop and available for Docker Engine on Linux. It is a different product from Ollama, and the names make it easy to mix them up.

If your tools already speak Ollama, such as Open WebUI, n8n, Continue or LangChain's Ollama integration, stay with Ollama. If you are starting fresh inside Docker Desktop and want models next to your other containers with Docker's own commands, Model Runner is worth a look.

Frequently asked questions

By closing time, the back-room tower was running two containers, a model sat on the RTX 3060 at 100% GPU, and the staff had bookmarked a chat page on the shop's network. Ethan wrote the update routine on a sticky note: "docker compose pull, docker compose up -d." Jake added a second line underneath: "127.0.0.1. Always." Ethan said it was the best thing anyone had written on that monitor.

Revision note. Written October 19, 2026, with Ollama 0.40.6 as the current release. A container is a box you can lift, move and replace; the model inside it never notices.