Run IBM Granite 4.2 Locally on Windows or Kali, Free

On August 25, 2026, IBM released Granite 4.2 — three open AI models (3B, 8B, and 30B) that reason step by step before they answer, read up to 128,000 tokens of text in one go, and ship under the Apache 2.0 license, which means you can download them, run them on your own PC, and use them for anything, including commercial work, for $0 and with no account. This page is the hands-on version: what Granite actually is, which of the three sizes your machine can run, the exact install steps for Windows 11 and Kali Linux, how to switch the "thinking" mode on and off, what goes wrong on first run (and the fix for each), and the honest answer to "does running AI locally cost money." The part that surprises people: a reasoning model with a commercial license now runs on an ordinary 16 GB laptop with no graphics card at all — slowly, but it runs, and nothing you type ever leaves the machine. If you have ever hesitated to paste a customer file or a contract into a cloud chatbot, that last sentence is the whole reason to keep reading.

Jake has a drawer of PDFs at the repair shop — supplier invoices, warranty claims, the occasional angry letter from a customer whose data he is legally responsible for — and last month he asked whether he could "get one of those AI things to read them." He could. He had also, the week before, read one of the breach stories on this site and asked, in the same breath, whether uploading a customer’s warranty claim to a chatbot in California counted as sharing it. It does, in the sense that matters: it left his building. So we set him up with the 8B version of the model on the shop’s five-year-old desktop, which has 16 GB of memory and no graphics card worth the name. It took eleven minutes, most of that the download. It summarizes a warranty PDF in about forty seconds — slow enough to make coffee, fast enough that he has stopped reading them himself. And the PDFs never left the drawer. If you run any kind of small business, or you simply do not like the idea of your private files training someone else’s product, Jake’s setup is the one this page walks you through.

Ethan: "A cloud AI is a restaurant. You describe what you want, someone behind a wall makes it, you pay per plate, and you never see the kitchen — which is fine until you want to bring your own ingredients and you’re not sure who else is in there. A model like Granite is the cookbook. IBM spent the money writing it — fifteen trillion words of training, they say — and then handed it out with a license that says cook whatever you like, sell it if you want, no royalties. The catch is the same as with any cookbook: you have to own a stove. This page is about how big a stove you need, and it’s smaller than you think."

What IBM Granite actually is

Granite is IBM’s family of open language models — the same category of thing as the models behind the big chatbots, but published as downloadable files rather than sold as a service. IBM has been releasing them since 2023, mostly aimed at businesses that need a model they can run inside their own walls: banks, hospitals, government offices, anyone whose lawyers get nervous about data leaving the building. That heritage shapes everything about Granite. The training data is curated with an eye on licensing and provenance, the models are smaller than the giants on purpose, and the license is the most permissive one in mainstream use. The family also includes Granite Guardian, a separate set of safety models that check prompts and answers for harmful content, and a line of small speech models — the newest, Granite Speech 5.0 Turbo, is a 470-million-parameter transcriber IBM says can turn three hours of recordings into text in about a second on a data-center GPU. Those are worth knowing about; this page is about the language models.

"Is IBM Granite open source?" is a question people ask because the phrase has been stretched by other companies. Many "open" models ship under custom licenses with clauses about who may use them, for what, and above what size of business. Granite 4.2 ships under Apache 2.0 — a standard, decades-old open-source license with no such clauses. You may download it, modify it, fine-tune it on your own data, build a product on it, and sell that product, without asking IBM or paying anyone. For a hobbyist that is a nicety. For Jake’s shop, or a startup, or a hospital IT department, it is the difference between "we can use this" and "legal needs six weeks."

What is new in Granite 4.2

Three things, in order of how much they matter to someone running this at home. First, reasoning is built in. Earlier Granite models answered the way most chatbots did in 2024: straight to the reply. Granite 4.2 can "think" first — write out a chain of intermediate steps, check itself, then answer — and every size has a switch to turn that on, off, or down to a cheap low-effort mode for easy questions. That matters because thinking costs time on a home machine; you want it for a tricky math problem or a contract clause and not for "rewrite this email." Second, the context window is 128,000 tokens across all three sizes (roughly 90,000 English words, or a 300-page document), with the training extended toward 512K for the larger model. That is the number that lets Jake drop an entire warranty PDF in rather than pasting pages. Third, the two bigger sizes were trained to act, not just chat — IBM ran what it calls agentic reinforcement learning on the 8B and 30B, teaching them to edit code, drive a terminal, and run searches inside sandboxed environments. The 3B skips that stage; it is the small, fast, run-anywhere option.

Under the hood, all three are conventional dense transformer models (no mixture-of-experts tricks), trained from scratch on about 15 trillion tokens, including roughly a trillion tokens of synthetic code from IBM’s own pipeline. They handle twelve languages — English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese — with English strongest. IBM’s published numbers for the 30B, for anyone who wants them, are the kind that would have been frontier-model scores two years ago:

Two honest notes on that table. These are IBM’s numbers, published with the model; independent runs will follow over the coming weeks and usually land a little lower. And they are for the 30B, which most home machines will not run comfortably. The 8B — the size most readers will actually install — is meaningfully weaker on every row, and the 3B weaker still. What the smaller sizes keep is the license, the context window, and the reasoning switch. For summarizing, drafting, explaining, and asking questions about your own documents, that is most of what matters.

Which size your machine can run

The only question that matters before you install anything is memory. A model has to be loaded into memory to run — graphics-card memory (VRAM) if you have a decent card, otherwise ordinary system RAM — and the download sizes below are roughly what it takes, plus a margin for the context you feed it. The useful rule from the people who run these all day: budget about 1.2 to 1.4 times the file size in free memory. Ollama, the tool we use below, ships each size in a compressed ("quantized") form that keeps almost all the quality at a fraction of the memory, which is why an 8-billion-parameter model fits in 5.3 GB. 

Speed, so you know what to expect: on a mid-range graphics card the 8B produces text faster than you can read it — dozens of tokens per second. On a CPU alone, the same model runs at roughly 2 to 15 tokens per second depending on how new the processor is (a token is about three-quarters of a word). That is the "make coffee" speed Jake gets. It is not the speed of a cloud chatbot, and this page will not pretend otherwise; what it is, is free and private, and for a document you would otherwise have to read yourself, forty seconds is a bargain. If you have a graphics card from AMD or Intel rather than NVIDIA, it still works — Ollama uses a different acceleration path (Vulkan) — but expect somewhat lower speeds and check the notes in the failure-modes table.

Install on Windows 11 (and 10): the ten-minute version

There are two well-trodden ways to run AI locally on Windows. LM Studio is a point-and-click app with a built-in model catalog and chat window — the right choice if you never want to see a terminal. Ollama is a small background service with a one-line command interface and a local API that other programs can talk to; it has had a native Windows version since late 2024 and, more recently, its own simple chat window too. We use Ollama here because the commands are identical on Windows and Kali, which keeps this page honest for both halves of our readers. Either tool works with Granite 4.2; the model files are the same.

1. Download Ollama. Go to ollama.com/download, pick Windows, run OllamaSetup.exe. It installs like any app and puts a small llama icon in the system tray. That icon means the service is running.
2. Open a terminal. Right-click Start → Terminal (PowerShell is fine). Type ollama --version and press Enter. A version number means step one worked. "Not recognized" means the installer has not updated your PATH yet — close and reopen the terminal, or sign out and back in.
3. Pull and run the model. Type ollama run granite4.2:8b (or :3b for a lighter machine). The first run downloads 5.3 GB, so this is the coffee step. When you see >>>, you are talking to a model that lives on your PC.
4. Ask it something with a right answer. Try "If a $1,200 laptop is discounted 15% and then taxed at 8%, what do I pay?" You will see it think (a block of reasoning), then answer $1,101.60. That is the reasoning mode; more on controlling it below.
5. Windows-specific: if Windows Defender or a third-party antivirus shows a prompt about the Ollama service or its network access, allow it — the "network" is just the model talking to your own machine on port 11434. The one setting worth changing: add the model folder (C:\Users\YOU\.ollama\models) to Defender’s exclusions, otherwise real-time scanning of an 18 GB file makes the 30B load painfully.
6. Optional: move the models off drive C. Set a system environment variable OLLAMA_MODELS to a folder on a bigger drive (Settings → System → About → Advanced system settings → Environment Variables), then restart Ollama from the tray icon. Models are large; C: drives are not.

A note on WSL, because the guides that rank for this topic still argue about it: you do not need the Windows Subsystem for Linux to run Ollama or LM Studio. Both are native Windows programs now. WSL is only worth the trouble if you specifically want Linux-only serving tools for development, and if you do, you already know who you are.

Install on Kali Linux (and any Debian-family distro)

Kali readers: this is the same machine that the Kali guides on this site are written from, and the steps are the ones that worked on it. Ollama is not in the Kali repositories, so there is no apt install ollama; the supported route is the project’s own installer script, which sets up a systemd service and a dedicated user. If you are running Kali inside VirtualBox — the way our ten-minute VirtualBox install guide sets it up — read the VM note at the end of this list first, because the graphics card will not be visible inside the VM and you will be on CPU speed.

1. Update first. On a rolling distro, stale package lists cause more failed installs than anything else: sudo apt update && sudo apt full-upgrade -y. (Why full-upgrade and not upgrade is explained in our apt guide — on Kali it matters.)
2. Run the installer. curl -fsSL https://ollama.com/install.sh | sh. Yes, that is a pipe-to-shell; the security-minded reader (this is Kali, so that is you) can fetch the script first, read it, and then run it. It installs the binary to /usr/local/bin, creates an ollama system user, and enables the service.
3. Check the service. systemctl status ollama should show active (running). If not, sudo systemctl enable --now ollama.
4. NVIDIA users: the installer detects the proprietary driver if it is already installed. On Kali that is sudo apt install nvidia-driver nvidia-cuda-toolkit and a reboot, done before step 2 ideally. Run nvidia-smi; if it prints your card, Ollama will use it. AMD cards work through ROCm on supported models; Intel and older AMD fall back to CPU.
5. Pull and run. ollama run granite4.2:8b. Same download, same >>> prompt, same test question as the Windows steps.
6. VM note: inside VirtualBox or VMware, give the VM at least 12 GB of RAM and 4 cores for the 8B, or use the 3B. The host’s graphics card is not passed through by default, so expect CPU speeds. It works — Jake’s numbers came from a machine without a card — but the 30B in a VM is a test of patience, not a tool.

Controlling the "thinking" mode: on, off, and low

This is the feature that makes 4.2 different from the Granite you may have tried before, and it is also the one that will frustrate you if you leave it on for everything. With thinking enabled (the default), the model writes out its reasoning inside a marked block before the answer. On a graphics card that is a second or two of extra text; on a CPU it can be thirty seconds of watching it deliberate over "rewrite this paragraph in a friendlier tone." So learn the switches on day one:

1. Inside the chat: type /set nothink to turn reasoning off for the session, and /set think to turn it back on. Use off for drafting, rewriting, and simple questions; on for math, logic, code, and anything where a wrong answer costs you.
2. From the command line, one shot: ollama run granite4.2:8b --think=false "Summarize this in three bullets: …". Add --hidethinking if you want the reasoning to happen but not scroll past you.
3. Low-effort mode: the model card exposes a low effort setting that spends a short reasoning budget — enough to sanity-check, not enough to essay. In Ollama this is the --think=low form on models that support levels; in code, it is the low_effort flag in the chat template. It is the right default for most everyday questions.
4. From your own programs: Ollama listens at http://localhost:11434 with an API that mirrors the format most AI apps already speak. Send "think": false in the request to skip reasoning; the reply separates the reasoning (message.thinking) from the answer (message.content) so you can log one and show the other.

One thing the reasoning block is not: a guarantee. IBM’s own model card says the intermediate thoughts are just that — intermediate — and can wander before the final answer lands. Treat the visible reasoning as a way to catch the model going wrong, which is genuinely useful, and not as proof it went right. Our plain-English piece on why a language model can be confidently wrong explains why that holds for every model, thinking or not.

Making it useful: your own documents, offline

A chat prompt in a terminal is a demo. The thing Jake actually uses is a folder of PDFs he can ask questions about, and the 128K context window is what makes that work without any clever engineering: a 40-page warranty document is around 20,000 tokens, which the model simply reads in one go. The simplest way to get there on either OS is a front-end that talks to Ollama’s local API — Open WebUI (a browser chat that looks like the cloud ones, runs on your machine, and lets you upload files) or AnythingLLM (a desktop app built around "chat with your documents", which IBM lists as a launch partner for 4.2). Both install in a few minutes, both point at localhost:11434, and both keep everything on disk. Pick Open WebUI if you want the familiar chat feel; AnythingLLM if the documents are the point.

Two practical rules that save people grief. Feed the model text, not pictures of text: a scanned PDF is an image, and the language model cannot read it until something has done the character recognition — both front-ends handle common cases, but a bad scan is a bad scan. And keep the reasoning switch off for document questions on a CPU machine; summarizing does not need deliberation, and the thinking block on a 20,000-token input is where "forty seconds" becomes "four minutes." If you want to see how the bigger version of this idea works — the same model, the same document question, but running on rented cloud hardware with a bill attached — our plain-English guide to Amazon Bedrock covers the trade-off; Granite is one of the model families it offers, which is a useful hint about who IBM built this for.

Everything that goes wrong on the first run, and the fix

Does running AI locally cost money? The actual math

The model costs nothing. The software (Ollama, LM Studio, Open WebUI, AnythingLLM) costs nothing. There is no per-message fee, no monthly plan, no token meter. What you pay for is electricity and, if you go shopping for it, hardware. The electricity is smaller than people assume. A laptop working hard draws somewhere around 50 to 80 watts; a desktop with a mid-range graphics card under load, perhaps 250 to 350. At a typical U.S. residential rate of about 17 cents per kilowatt-hour, an hour of the laptop running the model is roughly a cent, and an hour of the gaming desktop flat out is about five or six cents. A month of Jake-style use — a few dozen document questions a day — rounds to well under a dollar, because the machine is idle between answers. Compare that with the $20-a-month subscriptions the cloud chatbots charge, and the break-even on doing it yourself is measured in days, not years.

Hardware is the honest cost, and only if you want speed. You do not need to buy anything to run the 3B or the 8B on a machine with 16 GB of RAM; you need patience. If you decide the speed matters, the budget answer that keeps coming up in every guide on this topic is a used or entry-level graphics card with 12 GB of memory, which turns the 8B from "make coffee" into "faster than reading." That is a few hundred dollars once, versus a subscription forever — and it is also the moment to be clear-eyed: if all you want is an occasional chatbot answer, the cloud is cheaper and better. Local wins when the work is regular, the documents are private, or the license matters.

Why "nothing leaves the machine" is the real feature

When Jake pastes a customer’s warranty claim into a cloud chatbot, three things happen that he never sees. The text travels to a data center. It sits in logs for some retention period set by someone else’s policy. And depending on the plan and the checkbox he did or did not find, it may be used to train the next version of the product. None of that is sinister; it is how a service works. It is also, for a customer’s name, address, and purchase history, a data transfer that his privacy obligations were written to prevent. With the model on his own disk, the pipeline is: the text goes from the file to the memory of the machine it is already on, and back. You can pull the network cable and it keeps working. That is not a marketing claim; it is the architecture.

This matters more now than it did two years ago because the threat model changed. Our recent write-up on how scam pages ride on legitimate infrastructure is one half of the story: the internet is full of look-alike services. The other half is that every account you hold is another place your data can leak from, and the AI account is the one people fill with their most sensitive material — contracts, medical letters, financial statements — because it is so useful. A local model gives you the usefulness without opening the new account. For a security-minded Kali user that argument needs no elaboration. For everyone else, it is worth one sentence: the safest cloud is the one you never upload to.

Granite 4.2 vs the other local option vs the cloud chatbots

Readers of this site have already met the other popular route to a local model: the distilled DeepSeek releases covered in our guide to running DeepSeek on a Windows PC. Here is the honest side-by-side, on the dimensions that do not need a benchmark to judge. We are deliberately not scoring "which is smarter" — independent numbers for 4.2 are days old at best, and the only fair answer today is "try the same three prompts on both; the download is free."

Android, iPhone, and Mac: the honest answer

"How to run AI locally on Android" is one of the most-typed versions of this question, and the answer is yes-with-an-asterisk. Phone apps exist that run small models on-device — the 3B class, heavily compressed — and on a recent flagship they are usable for short questions. Granite 3B is exactly the kind of model those apps load. What you will not do on a phone is run the 8B with a 40-page document, and any app that claims to is either sending the work to a server or about to make your phone very warm. The realistic mobile play is the one many people land on: run Ollama on the desktop at home, put Open WebUI in front of it, and reach it from the phone over your own Wi-Fi or a VPN back to your house. Your data still never leaves your network. iPhone is the same story with fewer apps. Macs with Apple silicon are, quietly, the easiest machines of all for this: the unified memory means a 16 GB MacBook runs the 8B at graphics-card speeds, and a 32 GB one runs the 30B. The install is identical — Ollama has a Mac build — and every command on this page works unchanged.

The honest limits

Balance, because this site does not write brochures. An 8-billion-parameter model is not a frontier model. It will make mistakes a cloud chatbot would not, especially on obscure facts, on long multi-step instructions, and in languages other than English. It does not browse the web, so it knows nothing after its training cutoff and nothing about today. The reasoning switch helps on math and logic and does little for factual recall — a model cannot think its way to a fact it never learned. The 30B closes much of the gap, and it also closes the door on most laptops. And "private" means private from the model’s maker; it does not make your PC secure. If the machine itself is compromised, so is everything on it, which is a Kali reader’s bread and butter and a Windows reader’s reminder to keep the OS patched.

The right mental model is the one Ethan gave up top. The cookbook is excellent and free; the meal depends on the stove and the cook. For summarizing, drafting, explaining, translating between the twelve languages it knows, first-pass code review, and asking questions about documents you cannot send anywhere, Granite 4.2 on a home machine is genuinely good and genuinely yours. For "what happened in the news today" or a research task that needs the whole web, use the cloud, and use it for the parts that do not contain your customers’ names.

For IT admins and small businesses

The release is aimed at you more than at hobbyists, and three things make it worth an afternoon. The license removes the procurement conversation: Apache 2.0 needs no vendor agreement, no seat count, and no review of an "acceptable use" policy that might change next quarter. The model runs on hardware you already own — one workstation with a 24 GB card can serve the 30B to a small office through Ollama’s API, and every request stays inside your network, which turns a data-residency headache into a non-issue. And Granite Guardian, the companion safety model, gives you a policy layer you control: run inputs and outputs through it before they reach users, log what it flags, and adjust. The same weights are available on Amazon Bedrock and IBM’s watsonx if you outgrow the workstation, so nothing you build locally is a dead end. Start with the 8B on one machine, point AnythingLLM at your internal documentation, and see how many helpdesk questions it answers before you decide whether the 30B is worth a card.

FAQ — IBM Granite 4.2 and running AI locally, answered straight

What is IBM Granite?

IBM’s family of open language models, built for businesses that want to run AI inside their own systems. Granite 4.2, released August 25, 2026, comes in 3B, 8B, and 30B sizes with built-in step-by-step reasoning and a 128K context window.

Is IBM Granite free?

Yes. The models are released under Apache 2.0, so you can download and use them for personal or commercial purposes with no fee, no account, and no royalties. The only costs are your electricity and, optionally, hardware.

Is IBM Granite open source?

The weights are released under a genuine open-source license (Apache 2.0), which is more permissive than the custom "open" licenses many other model families use. IBM publishes the training and post-training stages; the full training data itself is not released.

Can I run AI locally on my PC?

Almost certainly. A PC with 8 GB of RAM runs the 3B model; 16 GB runs the 8B; a 24 GB graphics card or a 32 GB machine runs the 30B. No graphics card is required — it is just slower without one.

How do I run AI locally on Windows?

Install Ollama from ollama.com, open a terminal, and type ollama run granite4.2:8b. The first run downloads the model; after that it starts in seconds. LM Studio is the point-and-click alternative if you prefer no terminal.

How do I install it on Kali Linux?

Update the system, run curl -fsSL https://ollama.com/install.sh | sh, confirm the service with systemctl status ollama, then ollama run granite4.2:8b. Install the NVIDIA driver first if you have a card so it is detected.

Does running AI locally cost money?

Only electricity: roughly a cent an hour on a laptop, five or six cents an hour on a gaming desktop at full load, at typical U.S. rates. There are no token or subscription fees. Buying a graphics card is optional and only for speed.

Which Granite size should I use?

The 8B for most people: it fits in 16 GB of RAM or an 8 GB graphics card, and it received the agent training the 3B did not. Use the 3B on older or smaller machines, and the 30B only with a 24 GB card or plenty of RAM and patience.

How do I turn the thinking mode off?

Type /set nothink in the chat, or start with ollama run granite4.2:8b --think=false. Turn it back on with /set think for math, logic, or code. Reasoning costs time on a CPU, so keep it off for drafting and summarizing.

Why is it so slow on my computer?

Because the model is running on the CPU, or it did not fit in graphics memory and spilled to RAM. Run ollama ps to see the split. A smaller size, a shorter context, or turning thinking off usually fixes it.

Can I run AI locally on Android?

Small models like the 3B, yes, through on-device apps on a recent phone. Larger models and long documents, no. The practical route is running Ollama on a home PC and reaching it from the phone over your own network.

Is IBM Granite better than the cloud chatbots?

No, in raw capability — frontier cloud models are larger and web-connected. It wins on license, cost, and privacy. For document questions, drafting, and summaries it is more than good enough; for news and deep research the cloud is the better tool.

What is IBM Granite Guardian?

A separate family of safety models from IBM that classify prompts and responses for harmful or off-policy content. Businesses run them alongside the language model as a filter they control.

Can I use it for my own documents?

Yes. The 128K context reads roughly 300 pages in one go. Put Open WebUI or AnythingLLM in front of Ollama to upload PDFs and ask questions; both keep everything on your machine. Feed it text, not scanned images.

Does it work offline?

Completely, once the model is downloaded. You can disconnect from the internet and it keeps working, which is the whole privacy argument in one sentence.

What languages does Granite 4.2 support?

English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese, with English the strongest.

Do I need WSL on Windows?

No. Ollama and LM Studio are native Windows programs. WSL is only for developers who specifically want Linux-only serving tools.

Is a local model actually private?

Private from the model’s maker and from any cloud service, yes — nothing is transmitted. It does not make your PC itself secure; if the machine is compromised, so is what is on it. Keep the OS patched.

Revision note. Written August 27, 2026, two days after the release, from IBM’s model cards and research post, the Ollama library listing (download sizes and the thinking controls are quoted from there), and the same Kali machine our Linux guides come from. The benchmark figures are IBM’s own and will be revisited when independent runs land; the Windows and Kali steps will be updated if the installer or the model tags change. And if you got here because you have a drawer of files you did not want to upload anywhere: you were right to hesitate, and you do not have to choose between useful and private anymore. Start with the 8B, turn the thinking off, and give it one document. Jake’s first one was a warranty claim. He has not read one since.In simple words, start using those AI models and save costs. I know it wont give claude like features immediately, but they are not waste as well, so give it a try.