Nemotron 3.5 Bolt 74B-A9B: Run NVIDIA's 24GB Vision Model Locally (Ollama, llama.cpp, LM Studio), Requirements, Benchmarks and License

NVIDIA Nemotron 3.5 Bolt 74B-A9B is an open-weight vision-language model built to run agents on your own PC: 74 billion parameters, 9 billion active per token, text, image and short video in, a 256K-token context, tool calling and switchable reasoning. NVIDIA compressed it from Nemotron 3.5 Super VL (120B-A12B) and ships an official "24GB" checkpoint whose experts are trained to work at about 2 bits per weight, so the whole model is roughly 22-25 GB. You can run it today with Ollama (ollama run nemotron-3.5-bolt, a 25 GB download), llama.cpp (NVIDIA's own recipes for 32 GB and 48 GB GPUs, with a speculative-decoding drafter), LM Studio or Jan. Best fit: an RTX 5090 (32 GB), a 48 GB workstation card, an RTX Spark AI PC or a DGX Spark; a 24 GB RTX 4090 runs it with a smaller context or a smaller quant. Official support covers NVIDIA Ada and Blackwell GPUs on Windows and Linux; Ollama does not support Apple Silicon for it yet. License: OpenMDW-1.1, commercial use allowed. There is no hosted API and no Amazon Bedrock listing yet, and the smaller Bolt E4B is announced but not released. Below: hardware sizes, every install route, the exact llama.cpp commands, Python, thinking on and off, benchmarks, and the errors you will meet.

Jake had just built a desktop around an RTX 5090 and wanted one model that could read screenshots, call tools and still leave room for his editor and browser. The 120B Super VL was too big for 32 GB. Bolt, released a few days later, was cut down from that same model for exactly his situation, and Ethan's question was the reasonable one: does a 2-bit model actually hold up, and which of the five ways to install it is the right one?

New to running models on your own PC? The cards below cover the Super VL model Bolt comes from, choosing a runner, updating Ollama and picking hardware.

What is Nemotron 3.5 Bolt?

If you are asking what Nemotron is in the first place: it is NVIDIA's family of open models, released with their weights so you can download and run them yourself. The 3.5 generation, out in October 2026, comes in several sizes, and Bolt is the one aimed at your own computer:

What makes Bolt unusual is how it was made. NVIDIA took Super VL and compressed it with a method it calls Star Elastic, cutting the experts each token uses from 22 to 9, then trained the result to run at very low precision (quantization-aware distillation). The architecture is a hybrid: Mamba-2 layers, which handle long inputs cheaply, mixed with Transformer attention and a "LatentMoE" mixture of experts, plus a C-RADIO v4-H vision encoder for pictures. It answers in text, including JSON, tool calls and an optional reasoning trace, and it handles English best, along with French, Spanish, Italian, German, Japanese, Hindi, Korean, Brazilian Portuguese and Chinese.

Why does a 74B model fit in about 24 GB? Arithmetic. The experts, which are most of the weights, are stored at roughly 2.06 bits each: 74 billion × 2.06 bits is about 19 GB. The Mamba layers use NVIDIA's 4-bit NVFP4 format and the embeddings use 8-bit, which brings the file to the low twenties. Most "2-bit" models are squeezed after training and lose a lot; this one was trained to work at that precision, which is the whole point of the release.

Jake: "Two bits per weight sounds like it should be terrible."

Ethan: "It is, if you squash a model afterwards. This one was taught to live at two bits. That's the bet."

Nemotron 3.5 Bolt requirements: will it fit on your PC?

The weights are only part of the memory. The context window (KV cache), the vision projector and, in NVIDIA's recipe, a small drafter model all add to it. Here is what to expect on common hardware:

A few practical notes for your own setup:

• GPU generation matters. NVIDIA supports Ada Lovelace (RTX 40 series) and Blackwell (RTX 50 series, RTX PRO, DGX Spark and RTX Spark) on Windows and Linux. An RTX 3090 has 24 GB but is older Ampere; use Unsloth's standard GGUF quants there rather than the official recipe.
• Leave room for your desktop. Windows, a browser and an editor easily take 1-2 GB of VRAM. That is why the 32 GB recipe is called "28GB": it leaves headroom.
• System RAM is the fallback. When a model does not quite fit, Ollama and llama.cpp put some layers in system memory. It still works, just slower; 64 GB of RAM makes that comfortable.

A memory budget for a 32 GB card, to show where the gigabytes go when you follow NVIDIA's "28GB" recipe:

1. Model weights: the official checkpoint, roughly 22-24 GB once loaded.
2. Vision projector: the mmproj file that turns images into tokens, a small extra slice.
3. DFlash drafter: about 1 GB, optional, and the first thing to drop when memory is tight.
4. KV cache: grows with context; the 8-bit cache settings in the recipe are what make 65,536 tokens fit. The hybrid design helps here, because only the few attention layers keep a growing cache while the Mamba layers do not.
5. Your desktop: 1-2 GB for Windows, the browser and your editor.

Add those up and a 32 GB card is comfortably full, which is why the 24 GB card story is "it runs, with less context or a smaller quant", and why 48 GB cards get the full 256K window.

Jake: "So on my 5090 the drafter and the context fight for the last few gigabytes?"

Ethan: "Exactly. Long documents: drop the drafter, keep the context. Fast chat: keep the drafter, shorten the context."

Shopping for hardware with this model in mind? Our local LLM laptop guide covers what 24 GB and 32 GB of VRAM cost in practice.

Install Nemotron 3.5 Bolt with Ollama

Ollama is the shortest route: one command, and it handles the download, the vision projector and the chat template. You need a recent Ollama, since support for Nemotron 3.5 Super and Bolt arrived in version 0.40.5 on October 14, 2026.

1. Update Ollama and check the version with ollama --version (0.40.5 or later). Our Ollama update guide covers Windows, Mac and Linux.
2. Pull and run it: ollama run nemotron-3.5-bolt. The download is about 25 GB.
3. Check it is on the GPU: in a second window, ollama ps should show 100% GPU. A split such as 15%/85% CPU/GPU means part of it is in system RAM.
4. Set the context you want: inside the chat, /set parameter num_ctx 32768, or start the server with OLLAMA_CONTEXT_LENGTH=32768. Bigger context needs more memory.
5. Give it an image: type your question and the path to a picture, for example What is wrong in this error dialog? C:\Users\jake\Pictures\error.png.
6. Turn reasoning off for quick answers with /set nothink (or run ollama run nemotron-3.5-bolt --think=false); /set think turns it back on.

The Ollama library lists three tags: latest, 74b-a9b and 74b-a9b-iq2, all 25 GB with a 256K context and vision, tools and thinking. They are the same 2-bit build under different names, so nemotron-3.5-bolt is all you need. If you prefer Unsloth's quants, Ollama can pull them straight from Hugging Face: ollama run hf.co/unsloth/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF:UD-IQ2_XXS.

Jake: "ollama ps says 100% GPU at 24 GB used. That's it?"

Ethan: "That's it. Now push the context up and watch that number."

Run it with llama.cpp: NVIDIA's recipe

For the best speed, use llama.cpp directly with NVIDIA's settings. The recipe adds DFlash speculative decoding: a small drafter model (the "LiLiCorr" file, about 1 GB) guesses several tokens ahead and Bolt checks them in one pass, which raises tokens per second without changing the answers.

1. Build the latest llama.cpp with CUDA (an old build fails on the nemotron_h_moe architecture): cmake -B build -DGGML_CUDA=ON then cmake --build build --config Release -j. On Windows, add -DCMAKE_CUDA_ARCHITECTURES=120 for an RTX 50 card, 89 for RTX 40, or 121 for RTX Spark and DGX Spark.
2. Download the three files (the model, its vision projector and the drafter):

pip install -U huggingface_hub
hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF \
  NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
  mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf --local-dir ./bolt
hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF \
  NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf --local-dir ./bolt

3. Start the server with the 32 GB GPU settings ("28GB" recipe):

./build/bin/llama-server \
  --model ./bolt/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
  --mmproj ./bolt/mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
  --spec-type draft-dflash --spec-draft-n-max 6 \
  --spec-draft-model ./bolt/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf --spec-draft-ngl 999 \
  --ctx-size 65536 --parallel 1 --batch-size 2048 --ubatch-size 512 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -ngl 999 --flash-attn on --jinja --temp 1.0 --top-p 0.95 --port 8080

4. On a 48 GB+ card or a 128 GB RTX Spark, use the "32GB" recipe instead: --ctx-size 262144, --spec-draft-n-max 7, --batch-size 4096 --ubatch-size 2048.
5. Open http://localhost:8080 for the built-in chat page, or point any OpenAI-compatible app at http://localhost:8080/v1.

Keep the recommended sampling (temperature 1.0, top_p 0.95). Lower temperatures, a habit from older models, tend to make reasoning models loop.

Running Bolt on Windows without building anything

Building llama.cpp on Windows needs Visual Studio's C++ tools and the CUDA toolkit, which is a lot for one model. Two shorter routes:

1. Ollama for Windows: install it from ollama.com, open PowerShell, and run ollama run nemotron-3.5-bolt. To raise the context for every app, set a user environment variable (setx OLLAMA_CONTEXT_LENGTH 65536), quit Ollama from the system tray, and start it again so it picks the value up.
2. Prebuilt llama.cpp: download the latest Windows CUDA release zip from the llama.cpp GitHub releases page (the file name contains win-cuda), plus the matching CUDA runtime zip if one is listed, and unpack both into one folder. Then run llama-server.exe with the same flags as the recipe above, using Windows paths such as .\bolt\NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf and a caret (^) instead of a backslash at the end of each line in Command Prompt, or a backtick in PowerShell.
3. Keep the NVIDIA driver current: a recent Game Ready or Studio driver from the NVIDIA App is enough; you do not need the full CUDA toolkit to run prebuilt binaries.
4. Check the GPU is used: Task Manager's Performance tab shows "Dedicated GPU memory" filling to about 24-28 GB when the model loads; if system memory climbs instead, the GPU is not being used.

On Linux the same prebuilt approach works with the Ubuntu CUDA release, or build from source in two commands as shown in the recipe; WSL2 on Windows also works with the Linux steps if you already use it.

A browser chat window for Bolt with Open WebUI

If you want a polished chat interface with history, file uploads and image paste instead of a terminal, put Open WebUI in front of Ollama. It runs in Docker, finds Ollama on your machine, and lists nemotron-3.5-bolt in its model menu. Our Ollama Docker Compose guide has a ready Compose file for Ollama and Open WebUI together, with GPU access.

Open WebUI can also talk to llama-server: add a connection of type "OpenAI API" with the URL http://host.docker.internal:8080/v1 and any key. Paste a screenshot into the chat box and Bolt reads it, the same as in code. If several people on your network will use it, turn on Open WebUI's sign-in and keep Ollama itself bound to localhost.

LM Studio, Jan and other apps

Prefer a window to a terminal? Bolt works in the usual local apps:

• LM Studio: search for "Nemotron 3.5 Bolt", pick the nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF download, and set GPU offload to the maximum. If loading fails, update the app's llama.cpp runtime first; our LM Studio load error guide covers the rest.
• Jan, Unsloth Desktop, Lemonade and Docker Model Runner (docker model run hf.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF) also load it.
• vLLM and SGLang are the choices for serving several users from one machine; they also handle video input, which the chat apps mostly do not yet.

Not sure which runner suits you? Our Ollama vs LM Studio vs Jan comparison goes through the trade-offs.

Use it from Python: chat, images, tools and thinking

Both Ollama (http://localhost:11434/v1) and llama-server (http://localhost:8080/v1) speak the OpenAI API, so the standard openai package works:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="local")   # any string works

reply = client.chat.completions.create(
    model="nemotron-3.5-bolt",
    messages=[{"role": "user", "content": "Explain a mixture-of-experts model in two sentences."}],
    temperature=1.0, top_p=0.95,
)
print(reply.choices[0].message.content)

For llama-server, switch reasoning off per request through the chat template, which saves tokens on simple questions (reasoning and answer share your max_tokens):

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")
reply = client.chat.completions.create(
    model="bolt",
    messages=[{"role": "user", "content": "Give me a regex for UK postcodes."}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

Images go in as a content part, the same way as with hosted models:

import base64
image_b64 = base64.b64encode(open("invoice.png", "rb").read()).decode()
reply = client.chat.completions.create(
    model="nemotron-3.5-bolt",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "List the line items and the total as JSON."},
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}},
    ]}],
)

Tool calling uses the standard tools parameter; with llama-server keep --jinja on so the model's own template formats the tool calls. A middle setting between full reasoning and none also exists in the template (medium_effort), useful for agent loops where full reasoning on every step is too slow.

Jake: "It read my invoice screenshot and gave back clean JSON. On my own desk."

Ethan: "And nothing left the machine. For invoices, that's half the reason to run it locally."

Nemotron tool calling: a complete working loop

Tool calling is what Bolt was built for, so it is worth seeing the whole loop rather than one request. You describe your functions, the model answers with a tool call instead of text when it needs one, your code runs the function, and you send the result back so the model can finish. This works the same against Ollama or llama-server:

import json
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="local")
MODEL = "nemotron-3.5-bolt"

def disk_usage(path):
    import shutil
    total, used, free = shutil.disk_usage(path)
    return {"path": path, "free_gb": round(free / 1e9, 1), "used_gb": round(used / 1e9, 1)}

tools = [{
    "type": "function",
    "function": {
        "name": "disk_usage",
        "description": "Free and used space for a folder or drive",
        "parameters": {
            "type": "object",
            "properties": {"path": {"type": "string", "description": "e.g. C:\\ or /home"}},
            "required": ["path"],
        },
    },
}]

messages = [{"role": "user", "content": "How much free space is on C:\\ and is it enough for a 25 GB model?"}]

while True:
    reply = client.chat.completions.create(model=MODEL, messages=messages, tools=tools,
                                           temperature=1.0, top_p=0.95)
    msg = reply.choices[0].message
    if not msg.tool_calls:
        print(msg.content)
        break
    messages.append(msg)
    for call in msg.tool_calls:
        args = json.loads(call.function.arguments)
        result = disk_usage(**args)
        messages.append({"role": "tool", "tool_call_id": call.id,
                         "content": json.JSONEncoder().encode(result)})

Three habits keep tool loops reliable with a local model:

1. Write tool descriptions for the model, not for people: say what the tool returns and when to use it. Vague descriptions are the main cause of wrong or missing calls.
2. Cap the loop: stop after, say, eight rounds, so a confused model cannot call tools forever.
3. Validate arguments before running anything: the model chooses the arguments, so treat them like user input, especially for tools that touch files, shells or the network.

If tool calls arrive as plain text (a JSON blob inside the answer instead of tool_calls), the chat template is not being applied: update Ollama, or start llama-server with --jinja.

Use Bolt as a local coding agent (OpenCode, Continue, Cline)

Because Ollama and llama-server speak the OpenAI API, any coding agent that accepts a custom OpenAI-compatible endpoint can use Bolt, with nothing leaving your machine:

• OpenCode: add Ollama as a provider in opencode.json with the base URL http://localhost:11434/v1 and the model nemotron-3.5-bolt, then pick it with /models. Our OpenCode guide has the full provider block.
• Continue, Cline and Roo Code in VS Code: choose Ollama (or "OpenAI compatible" for llama-server) as the provider and enter the model name.
• Any OpenAI SDK app: set the base URL to http://localhost:11434/v1 (Ollama) or http://localhost:8080/v1 (llama-server) and any placeholder key.

The one setting that matters most for agents is context length. An agent sends your files, tool results and the conversation history on every turn, which fills a small window within minutes. With Ollama, start the server with OLLAMA_CONTEXT_LENGTH=65536 (or higher if your memory allows); with llama-server, set --ctx-size as in the recipes above. When an agent "forgets" files it read earlier or repeats steps, a context window that is too small is the usual reason.

Jake: "OpenCode with Bolt fixed a failing test on its own, but it was slow to start."

Ethan: "The first turn reads the whole repository into context. After that the cache does the work."

Reading screenshots, scanned documents and PDFs

Bolt's vision encoder makes it useful for the document jobs people usually send to cloud APIs: invoices, receipts, forms, charts and screenshots of apps. A few practical rules:

• PDFs go in as page images. The chat APIs take images, not PDF files, so convert pages first, for example with pdftoppm -png -r 150 invoice.pdf page (Poppler), and send one page per message part.
• Resolution matters more than file size. Around 150 DPI keeps small print readable without wasting context; a full-page scan at 600 DPI costs far more tokens and rarely reads better.
• Ask for a fixed JSON shape ("return {vendor, date, total, lines:[{item, qty, price}]}") and parse it in code; then check totals with arithmetic rather than trusting the model's sum.
• Screenshots for agents: Bolt reads app and web page screenshots well enough to say which button to click, which is how its WebArena score was earned. Crop to the relevant window when you can.
• Video (up to 2 minutes) works through vLLM or SGLang; most desktop apps accept only images for now.

For a dedicated, CPU-only OCR tool instead of a 24 GB model, our Indra OCR guide covers the lighter option.

Thinking modes: on, off and medium effort

Bolt can reason step by step before answering, and you choose how much:

Reasoning tokens are real tokens: they take time to generate and count against max_tokens. If answers stop halfway, raise max_tokens or turn reasoning down. For document extraction, thinking off is usually both faster and just as accurate.

A simple way to choose: send the same ten real prompts from your own work in each mode and compare answers and time. If thinking off gets nine of ten right in a third of the time, use it and switch reasoning on only for the hard cases, for example with a router in your code that turns thinking on when a first, quick answer fails a check such as invalid JSON or a failing test.

Jake: "Thinking on made my invoice extraction three times slower and no better."

Ethan: "Reading a total off a page isn't a reasoning problem. Save thinking for the agent's planning steps."

How to measure Bolt's speed on your machine

Speed depends on your GPU, the context length, whether the drafter is on, and how much of the model sits in system RAM, so measure rather than guess:

1. Ollama: run ollama run nemotron-3.5-bolt --verbose, ask a question, and read the eval rate line (tokens per second for the answer) and prompt eval rate (how fast it reads your input).
2. llama.cpp: the server log prints prompt and generation speed for every request; for a clean benchmark, run llama-bench -m ./bolt/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf.
3. Compare with and without the drafter: start llama-server once with the --spec-* flags and once without, and send the same prompt. The gain is largest on predictable text such as code and JSON.
4. Watch memory while you test: nvidia-smi in another window shows VRAM use; if it is full and system RAM climbs, part of the model has spilled and speed drops sharply.

If generation is far slower than you expect, the cause is nearly always layers in system RAM; lower the context or close other GPU programs first.

Bolt vs Super VL vs Lightning vs Qwen3.8: which local model to pick

Our Nemotron family guide lists every model, with sizes, Ollama names and licenses.

Running local agents safely

A local model keeps your data at home, but an agent that can run commands can still damage your machine. Before you give Bolt tools that touch files or a shell:

• Keep the server private. Ollama listens on 127.0.0.1:11434 by default; setting OLLAMA_HOST=0.0.0.0 exposes it to your network with no password. llama-server is the same on port 8080. Use an SSH tunnel instead of opening ports.
• Sandbox the agent. Run coding agents in a container, a separate user account or a sandbox such as NVIDIA's OpenShell, with only the project folder mounted.
• Approve risky actions. Most agents can ask before running shell commands or deleting files; leave that on until you trust the setup.
• Treat web content as untrusted. A page the agent browses can contain instructions aimed at the model; an agent that reads the web should not also hold your credentials.

Nemotron 3.5 Bolt benchmarks

The launch scores compare the 2-bit Bolt against popular full-precision local models. These are vendor-run numbers, so treat them as a starting point and test on your own tasks:

Read the pattern, not the decimals: Bolt is strongest on agent work (browsing, terminal tasks, tool use, screenshots of web pages), which is what it was built for. It is not ahead everywhere: Qwen3.6 edges it on PinchBench and both Qwen and Lightning hallucinate less on the knowledge test, so for factual question-answering without tools, check its answers. The comparison models run at full precision and Bolt at 2 bits, which is the interesting part: a 24 GB file competing with models that need far more memory at the same quality.

License, price and "is it free?"

• License: OpenMDW-1.1, an open license that allows commercial and non-commercial use. Check the full text yourself before you ship it in a product, as with any model license.
• Price: the download is free. There is no metered API from NVIDIA or the major hosts yet, so there is no per-token price to compare; your cost is your electricity and your GPU. As a rough guide, an RTX 5090 working hard draws around half a kilowatt, so four busy hours a day is about 60 kWh a month: roughly $10 at typical US rates, and closer to $20 in the UK or Australia, where power costs more.
• What the license covers: OpenMDW is written for model releases as a whole (weights, code and data together), which makes it simpler to read than licenses with separate terms for each part. You can fine-tune Bolt and ship the result; NVIDIA publishes the BF16 weights for exactly that kind of customization.
• Data: post-training data runs to May 2026, so it knows nothing after that without tools or documents you provide.
• Images and video you feed it: the model does not blur faces or remove personal data, so the usual privacy care is on you.

Is running Bolt locally cheaper than an API?

Often not, and it is worth knowing before you buy a GPU for it. Bolt has no hosted API yet, but its parent, Nemotron 3.5 Super VL, costs about $0.10 per million input tokens and $0.40 per million output tokens on OpenRouter. Take an agent that reads 1.8 million tokens and writes 200,000 a day:

• Through the Super VL API: 1.8 × $0.10 + 0.2 × $0.40 = about $0.26 a day, under $8 a month, with no hardware.
• Bolt at home: an RTX 5090 drawing around 0.5 kW for four busy hours uses about 2 kWh, roughly $0.34 a day at a typical US rate of $0.17 per kWh, before you count the card itself.

So the case for local is not price per token. It is privacy (invoices, contracts and code that cannot leave the building), no rate limits or outages, offline work, and unlimited experimenting once the hardware is paid for. If none of those matter, an API is cheaper and simpler; if they do, Bolt is the strongest agent model that fits on one consumer card.

Can you fine-tune Nemotron 3.5 Bolt?

Yes in principle, but not on the PC that runs it. The 24 GB checkpoint is a deployment format; fine-tuning starts from NVIDIA's BF16 weights, and 74 billion parameters at 2 bytes each is about 148 GB for the weights alone, before gradients and optimizer state. Even parameter-efficient methods such as LoRA need that base model loaded, which means several data-center GPUs (rented 80 GB cards on AWS or elsewhere) rather than one RTX card.

Before you plan that, try the cheaper levers: a better system prompt, a few worked examples in the prompt, retrieval of your documents into the context, and tool definitions that do the precise work. Most "the model doesn't know our format" problems are solved there. If you do fine-tune, plan to compress the result again for local use, since a fine-tuned BF16 model does not inherit the 2-bit training that makes Bolt fit in 24 GB.

Running Bolt on AWS

Bolt is not on Amazon Bedrock today, so there is no serverless, pay-per-token way to call it on AWS. If you want it in the cloud, run it yourself on a GPU instance:

• Instance: a g6e (NVIDIA L40S, 48 GB, Ada Lovelace) fits the full-context "32GB" recipe on one GPU. A g6e.2xlarge gives you more system RAM than the smallest size, which helps if you also load bigger quants.
• Image: start from an AWS Deep Learning AMI, which already has the NVIDIA driver and CUDA, then build llama.cpp or pip install vllm and use the same commands as above.
• Access: keep port 8080 closed to the internet. Reach the server through Session Manager port forwarding or an SSH tunnel, since llama-server has no login of its own.
• Cost: stop the instance when you are done; an idle GPU instance bills by the hour, and for regular use a local card pays for itself quickly.
• Managed endpoint: the same files can be served from SageMaker AI with a vLLM container if several people or apps need it.

Keeping Bolt up to date

Model files get revised after launch: chat-template fixes, better quants and runtime tweaks are common in the first weeks. To pick them up:

• Ollama: run ollama pull nemotron-3.5-bolt again. It compares layers and downloads only what changed, so an unchanged model finishes in seconds. Update Ollama itself at the same time, because template and architecture fixes often ship in the app rather than the model.
• llama.cpp: rerun the hf download commands to refresh the files, and rebuild or redownload llama.cpp, since support for new hybrid models improves quickly.
• LM Studio: update the app and its llama.cpp runtime from the settings; then check the model page for a newer upload.
• Watch for Bolt E4B: when it lands, it will appear as its own model in the same places, not as an update to this one.

If a model that used to work starts failing after an update, check the release notes of your runner first; a newer runtime with an old model file, or the other way round, causes most "it broke overnight" reports.

Errors and fixes

• unknown model architecture: 'nemotron_h_moe' in llama.cpp or LM Studio: the runtime is older than the model. Update llama.cpp (or the LM Studio runtime) to the latest build.
• "model requires more system memory" in Ollama: lower num_ctx, close other GPU apps, or use a smaller Unsloth quant; our memory error guide walks through it.
• Very slow, ollama ps shows mostly CPU: the GPU is not being used or most layers spilled to RAM. Our Ollama GPU guide covers drivers and CUDA.
• CUDA out of memory with llama-server: drop --ctx-size, keep --cache-type-k q8_0 --cache-type-v q8_0, or remove the drafter (--spec-* flags) to save about 1 GB.
• It "thinks" for ages on simple questions: turn reasoning off (/set nothink in Ollama, enable_thinking: False in llama-server) or raise max_tokens so the answer is not cut off after the reasoning.
• Images ignored in llama.cpp: the --mmproj file is missing from the command.
• Not available on Mac: Ollama does not support this model on Apple Silicon yet; that is expected, not a broken install.
• Tool calls show up as text in the answer: the chat template is not applied; update Ollama, or add --jinja to llama-server.
• Answers loop or repeat: the temperature is set low (0.1-0.3 is a habit from older models); go back to 1.0 with top_p 0.95.
• llama runner process has terminated in Ollama: almost always memory; lower the context, close other GPU apps, and check nvidia-smi.
• The agent forgets earlier files: the context window is too small for the session; raise OLLAMA_CONTEXT_LENGTH or --ctx-size.
• Answer cut off after a long reasoning block: max_tokens ran out during thinking; raise it, or switch to medium effort or thinking off.
• Download stalls or fails halfway: rerun the same ollama pull or hf download command; both resume where they stopped.

Jake: "So for my 5090: Ollama for quick chats, llama-server with the drafter for the agent work?"

Ethan: "Exactly. Same 24 GB file underneath both. Pick by how fast you need it."

Frequently asked questions

Jake now runs Bolt in llama-server with the drafter for his screenshot-reading agent and keeps Ollama for quick questions, both from the same 24 GB file. With a 64K context it uses about 28 GB of his 32, which leaves room for his editor and browser. Ethan is waiting for Bolt E4B, to see whether a laptop GPU can do half of this.

Revision note. Written October 19, 2026, four days after the Hugging Face release. Bolt E4B and any hosted API will be added here when they arrive.