Install Gemma 4 on Kali Linux: Run It Offline with Ollama (2026)

To install Gemma 4 on Kali Linux and run it offline, install Ollama with its official script and pull a model: curl -fsSL https://ollama.com/install.sh | sh, then ollama run gemma4:e4b on a laptop or ollama run gemma4:12b on a machine with an 8 GB graphics card. The first command downloads Ollama; the second downloads the model once and opens a chat in your terminal. After that, Gemma 4 runs with no internet at all, and nothing you type leaves the machine. The same two lines work on any Linux distribution. Here is the catch that decides your experience before anything else: if Kali is running inside VirtualBox or VMware, your graphics card is invisible to it, so the model runs on the processor alone, and only the two smallest sizes are pleasant to use. Choose the size for where Kali actually runs, not for the hardware you own.

Ethan's cousin, three months into a junior penetration-testing job, wanted a model he could paste client scripts and logs into without breaking the engagement's confidentiality rules. His Kali ran in a VM on a gaming laptop with a 12 GB card, and his first try, the 26B model, took a minute per answer. Ethan moved him to Kali installed directly on a second drive, where the card was visible, and the 12B answered at reading speed. Jake, on an old ThinkPad with no graphics card, runs the E4B on the CPU and uses it to explain commands he does not understand. Same model family, three very different setups. This page tells you which one you are.

Gemma 4 is Google's open-weight model family, released April 2, 2026 under the Apache 2.0 license. It is not Gemini; Gemini models, including the new Gemini 4 Argon, are cloud-only, as the Argon guide explains. Gemma is the Google model you can actually download.

Which Gemma 4 size fits your machine

Gemma 4 comes in five sizes. The numbers below are for the 4-bit versions Ollama downloads by default, which is what almost everyone should use.

Ollama tag	Size	Download	Memory needed	Context	Best for
gemma4:e2b	2.3B effective	about 4.6 GB	4 GB	128K	Old laptops, small VMs, CPU only
gemma4:e4b	4.5B effective	about 6.6 GB	5.5 to 6 GB	128K	Any laptop; Kali in a VM; the default choice
gemma4:12b	12B dense	about 7.7 GB	7 to 8 GB	256K	8 GB graphics card on bare-metal Kali
gemma4:26b	26B total, 3.8B active (MoE)	about 16 GB	16 to 18 GB	256K	32 GB RAM desktops; fast for its quality
gemma4:31b	31B dense	about 19 GB	17 to 20 GB	256K	24 GB graphics cards; the strongest

Three notes the table cannot hold. The E2B and E4B use a technique that makes them behave like larger models than their memory use suggests, so the E4B on a plain laptop is the pleasant surprise of the family. The 26B is a mixture-of-experts model that only uses 3.8B parameters per word, so on a machine that can hold it, it answers faster than the 12B. And only the three smaller sizes accept audio input; the 26B and 31B take text and images. If you are unsure, pull the E4B first: it downloads in minutes and tells you within ten whether local AI is for you.

The five-minute setup, in order

If you just want it working, here is the whole page as a checklist. Each step links to its section if something goes wrong.

1. Decide where Kali runs. VM: plan for E2B or E4B. Bare metal or WSL with an NVIDIA card: 12B or larger. Details.
2. Update Kali with sudo apt update && sudo apt full-upgrade -y.
3. Install Ollama with the official script. Details.
4. Pull your size with ollama run gemma4:e4b (or 12b, 26b, 31b). Size table.
5. Check the processor with ollama ps. If it says CPU and you expected GPU, fix the driver. GPU setup.
6. Test offline once, then keep the API on localhost. Why.

Install Ollama and Gemma 4 on Kali, step by step

Step 1: update Kali

sudo apt update && sudo apt full-upgrade -y

If apt update fails with signature or key errors, fix that first; the Kali signatures guide covers the 2026 key change. The apt and sudo explainer helps if these commands are new to you.

Step 2: install Ollama

curl -fsSL https://ollama.com/install.sh | sh

The official script installs the ollama binary, creates an ollama system user, and sets up a systemd service that starts at boot. It works on Kali's rolling release as well as numbered releases. Ollama is not in Kali's own repositories as a current package, which is why apt install ollama is not the route. If you prefer to read a script before running it, download it first with curl -fsSL https://ollama.com/install.sh -o install.sh, read it, then run sh install.sh.

Check it is running:

ollama --version
systemctl status ollama

Step 3: pull and run Gemma 4

ollama run gemma4:e4b

The first run downloads the model, then drops you at a >>> prompt. Type a question and press Enter. Type /bye to leave. Next time, the same command starts instantly from the local copy. Swap e4b for the size you chose from the table.

Step 4: confirm it works offline

Disconnect from the network (nmcli networking off), run ollama run gemma4:e4b again, ask something, then reconnect with nmcli networking on. If it answers, you have a fully offline model. That test is worth doing before you depend on it somewhere without connectivity.

Use your graphics card (NVIDIA and AMD)

Ollama uses a supported GPU automatically if the drivers are installed. Check what it is using while a model is loaded:

ollama ps

The PROCESSOR column says 100% GPU, 100% CPU, or a split such as 40%/60% CPU/GPU when the model does not fit in graphics memory. A split works but runs at roughly the speed of the CPU part.

• NVIDIA: install the proprietary driver from Kali's repositories with sudo apt install -y nvidia-driver nvidia-cuda-toolkit, reboot, and confirm with nvidia-smi. Then restart Ollama with sudo systemctl restart ollama. Kali's documentation covers Secure Boot and laptop hybrid-graphics cases.
• AMD: Ollama supports recent Radeon cards through ROCm; the install script fetches the extra components when it detects one. Older cards fall back to the CPU.
• Intel integrated graphics: treat it as CPU-only and use E2B or E4B.

Kali in VirtualBox, VMware or WSL: what changes

This is the section the Argon guide promised, and it is where most disappointing first attempts come from.

• VirtualBox and VMware Workstation do not pass your graphics card through to the guest for compute. Inside the VM, Ollama sees only virtual CPUs and the RAM you assigned. Give the VM as many CPU cores as you can spare and at least 8 GB of RAM, and use E2B or E4B. The 12B will run, slowly; anything larger is not worth it. The VirtualBox install guide shows where to set cores and memory.
• Kali on WSL 2 (Windows Subsystem for Linux) can use an NVIDIA card: with a current NVIDIA driver installed on Windows, CUDA is available inside WSL, and Ollama inside Kali uses it. Do not install a Linux NVIDIA driver inside WSL. If you want the simplest Windows route, installing Ollama for Windows directly is easier still; the Windows Gemma guide covers it.
• Bare-metal Kali (installed on a drive, or booted from a persistent USB) sees the GPU directly. This is what turned Ethan's cousin's minute-long answers into reading speed.
• Proxmox or KVM with GPU passthrough also gives the VM the real card, if you have set up passthrough. Then it behaves like bare metal.

Using Gemma 4 on Kali: practical commands

# one-off question without entering chat
ollama run gemma4:e4b "Explain what this nmap flag does: -sV"

# feed a file (a script or log) into the prompt
ollama run gemma4:e4b "Summarize the errors in this log:" < /var/log/syslog

# list, inspect and remove models
ollama list
ollama show gemma4:e4b
ollama rm gemma4:31b

Ollama also serves a local API on http://localhost:11434, which tools such as Open WebUI, VS Code extensions and your own Python scripts can use. The API understands the OpenAI request format, so many tools work by pointing their base URL at it. For a pentester, the value is simple: scripts, logs and client data you paste into a local model stay on your disk.

Models are stored, for the service install, in /usr/share/ollama/.ollama/models. To move them to an air-gapped machine, copy that folder to the same place on the target after installing Ollama there.

Prefer a window to a terminal? LM Studio and llama.cpp

Ollama is the simplest route on Kali, but not the only one. LM Studio offers a Linux AppImage with a chat window, a model browser and a built-in local server; search for "gemma 4" in its model list and pick a size from the table above. llama.cpp, the engine underneath many of these tools, runs the same GGUF files from the command line for people who want full control over threads and memory. Whichever you choose, the default 4-bit files (Ollama pulls Q4_K_M) are the sensible balance of quality and size; 8-bit versions roughly double the memory needed.

Keep the Ollama API private on a security box

By default Ollama listens only on 127.0.0.1, so nothing on the network can reach it. Many guides tell you to set OLLAMA_HOST=0.0.0.0 to use it from another device; on a Kali machine that you take onto client networks, do not. The API has no authentication, and an exposed one lets anyone on the network run models, read what is loaded and use your hardware. If you need remote access, put it behind an SSH tunnel: ssh -L 11434:localhost:11434 you@kali-box from the other machine, and leave the service on localhost.

When it does not work

Problem	Likely cause	Fix
Error: could not connect to ollama app, is it running?	The service is stopped	sudo systemctl start ollama, or run ollama serve in another terminal
model requires more system memory	The size is too big for your RAM	Pick a smaller tag; close browsers; give a VM more RAM
Answers take a minute or more	Running on CPU, often inside a VM	Check ollama ps; use E4B, or run Kali on bare metal
ollama ps says 100% CPU with an NVIDIA card	Driver missing or Ollama started before it	Install the driver, confirm nvidia-smi, restart Ollama
pull model manifest: file does not exist	Typo in the tag	Use gemma4:e4b, gemma4:12b and so on; there is no gemini4
Download stalls or fails	Network or proxy	Re-run the same command; it resumes. Set HTTPS_PROXY for the service if you use a proxy
Disk full after a few models	Each size is a separate multi-GB download	ollama list, then ollama rm the ones you do not use

Remove Gemma 4 or Ollama

To remove only a model, run ollama rm gemma4:e4b (or whichever size). To remove Ollama entirely:

1. Stop and disable the service: sudo systemctl disable --now ollama
2. Delete the service file: sudo rm /etc/systemd/system/ollama.service
3. Remove the program: sudo rm $(which ollama)
4. Delete the downloaded models: sudo rm -r /usr/share/ollama
5. Remove the service account: sudo userdel ollama

Nothing else on Kali depends on Ollama, so removing it leaves the rest of your tools untouched.

Ethan's cousin now keeps the 12B on his bare-metal Kali drive and runs every client script past it before asking a colleague, and nothing leaves the laptop. Jake's E4B on the old ThinkPad is slower, and he does not mind, because it explains things patiently at two in the morning. Neither setup needed a subscription or an account. The model is free, the install is two lines, and the only real decision is the size, which comes down to one question: can Kali see your graphics card?

Revision note. Written October 1, 2026, for Gemma 4 and the current Ollama on Kali. If your first try was painfully slow inside a VM, the model was fine; it just could not see your card.