Hermes Agent with Ollama: run it on local models

What running Hermes on Ollama means

Ollama downloads open-weight models and serves them on your computer through an OpenAI-compatible API, by default at http://localhost:11434/v1. Hermes reaches it through its custom endpoint option. From then on Hermes behaves as it does with any other provider: it runs terminal commands, edits files, browses the web and runs scheduled jobs, with the model doing the thinking locally.

The Hermes guide for this sets out the hardware as a minimum and a recommendation. For small models of around 3B parameters it lists 8 GB of RAM, and for models of 27B and above it recommends 32 GB or more, plus 5 GB to 30 GB or more of disk. A GPU is not required, but the guide says an NVIDIA GPU with 8 GB or more of video memory speeds things up significantly.

Ollama is one of several routes. Hermes also supports LM Studio, vLLM, SGLang and llama.cpp’s llama-server, covered further down, and the Hermes desktop app can install and manage a local llama.cpp server for you.

Setting it up step by step

1. Install Ollama. On Linux the Hermes guide uses curl -fsSL https://ollama.com/install.sh | sh, and you can confirm it is running with curl http://localhost:11434/api/tags. Other systems: see ollama.com for the installer.
2. Pull a model that can use tools. For example ollama pull gemma4:31b. The next section explains why the choice matters.
3. Raise the context window to at least 64,000 tokens. This is the step people miss; see below.
4. Tell Hermes about the endpoint. Run hermes model, choose the custom endpoint option, enter http://localhost:11434/v1, leave the key empty, and enter the model name. Or set it in ~/.hermes/config.yaml:

   model:
     default: "gemma4:31b"
     provider: "custom"
     base_url: "http://localhost:11434/v1"
     context_length: 64000

5. Start Hermes with hermes. Inside a session, /model switches between the models you have pulled.

If you have not installed Hermes yet, start with how to install Hermes Agent. If you run Hermes inside WSL2 on Windows while Ollama runs on the Windows side, you need a little extra networking: the Hermes docs recommend mirrored networking mode on Windows 11 22H2 or later, so that localhost reaches Windows from inside WSL2.

The context window: a common cause of trouble

Hermes refuses to use a model with less than 64,000 tokens of context. In this setup, sessions, model switches and scheduled jobs all reject anything smaller (the source makes one exception, an explicit context_length on the LM Studio provider), because the system prompt, the tool descriptions and the working conversation need room.

Ollama does not use a model’s full context by default. According to the Hermes docs, the default depends on your video memory: 4,096 tokens under 24 GiB, 32,768 tokens from 24 GiB to under 48 GiB, and about 256,000 tokens at 48 GiB or more. So a machine with under 24 GiB of video memory starts well below what Hermes needs. The context length cannot be set through the OpenAI-compatible API that Hermes calls; it has to be set on the Ollama side, in one of two ways:

• Set it for the whole server. If you start Ollama yourself, stop it first and restart it with OLLAMA_CONTEXT_LENGTH=64000 ollama serve. If it runs as a systemd service on Linux, run sudo systemctl edit ollama.service, add Environment="OLLAMA_CONTEXT_LENGTH=64000" under [Service], then sudo systemctl daemon-reload and sudo systemctl restart ollama.
• Or bake it into a model: a Modelfile containing FROM gemma4:31b and PARAMETER num_ctx 64000, then ollama create gemma4-64k -f Modelfile, and use gemma4-64k as the model name in Hermes.

Check the result by sending one request to the model, then running ollama ps while it is still loaded: the CONTEXT column should show the value you set. A bigger window also needs more memory, so a model that fits at 4,096 tokens may not fit at 64,000.

Which local models work for an agent

An agent acts through tool calls: it asks to run a command, read a file or search the web, and Hermes carries it out. A model that cannot produce structured tool calls can chat but cannot do the work. The Hermes guide is blunt about this. In its table, gemma4:31b (about 20 GB on disk, 24 GB or more of RAM) is the one listed with strong tool use, while gemma2:27b and gemma2:9b are marked as unable to call tools and suited only to conversation. It also lists llama3.2:3b as lightweight and very limited. Ollama’s own model page for llama3.2:3b carries a tools label (checked 23 October 2026), so tool support and reliability on complex jobs are separate questions. It notes that tool-calling support in Ollama’s library is expanding, so check the Ollama model library for newer models and look for the tools label on a model’s page.

Other Hermes docs use qwen2.5-coder:32b as their Ollama example. For other servers, they name model families with native tool-calling support: Llama 3.x, Qwen 2.5 (including Coder), Hermes 2 and 3, Mistral and DeepSeek on llama.cpp, and Qwen 2.5, Llama 3.x, Mistral and Hermes on LM Studio. These lists describe tool-call format support, not how smart a model is. Whichever you pick, try it on a real job before relying on it.

The same docs say what local models are good at: file editing, code generation, terminal commands, web browsing and scheduled jobs, with models of about 9B and up handling the file work well. They say very complex multi-step reasoning is noticeably better on 70B-class or cloud models. If you cannot fit a big model, you can configure a cloud fallback with fallback_providers. It takes over when the main provider errors, for example on rate limits, server overload, authentication failures or dropped connections; it does not judge which jobs are hard. For a harder job, switch models yourself with /model.

LM Studio, llama.cpp, vLLM and Ollama Cloud

• LM Studio has its own entry in hermes model and defaults to http://localhost:1234/v1. Set the context length to at least 64,000 in LM Studio and reload the model, or load it with lms load <model> --context-length 64000.
• llama.cpp (llama-server) needs its Jinja chat template support enabled (the Hermes docs pass --jinja) and a model whose template supports tools; otherwise the model writes tool calls as plain text. Set -c to at least 64,000 yourself, and check how much context each request actually gets, because with parallel slots the total can be divided between them.
• vLLM is the option for GPU servers. Tool calling needs --enable-auto-tool-choice and a --tool-call-parser that matches the model, and --max-model-len must be large enough.
• Ollama Cloud is a built-in Hermes provider (ollama-cloud). It serves open-weight models from Ollama’s servers, so you need no GPU, but it is a hosted service with an OLLAMA_API_KEY, not local. Hermes lists the available models from the service, and the docs give gpt-oss:120b as an example.

On the desktop app, Settings, Providers, Local Models installs a llama.cpp server, offers a model catalogue that shows whether each model fits your machine, and sizes the context for you. The docs say every recommended model there gets at least a 64K window, and that Windows and Linux use an NVIDIA GPU (CUDA), an AMD GPU (Vulkan builds) or the CPU, while macOS needs Apple Silicon. A GPU with 8 GB or more of memory runs the small catalogue models; 16 GB or more runs the 27B to 35B ones.

Common problems and fixes

• Hermes rejects the model at startup. If the error mentions context length, the window is under 64,000 tokens; fix it on the Ollama side, as above.
• The model prints raw JSON such as a tool name instead of acting. Usually the server, not the model: tool calling is not switched on or the format is not parsed (for example llama.cpp without Jinja templates enabled, or vLLM without the tool flags). If it is the model itself, it probably lacks tool-call support; pick one that has it.
• The first reply takes minutes. Hermes sends the system prompt and the tool descriptions on every call, and on a CPU or a small GPU, processing that dominates the first turn. The docs call this expected, not a hang. Hermes raises its stream timeout to 1800 seconds for local endpoints, and you can set HERMES_API_TIMEOUT=1800 in ~/.hermes/.env. Ollama unloads an idle model after five minutes by default, so set OLLAMA_KEEP_ALIVE=24h if the agent should answer promptly all day. hermes prompt-size shows how large the fixed prompt is, and switching off toolsets you do not use shortens it.
• Slow replies throughout. If the model is bigger than your video memory, part of it runs on the CPU from system RAM, which is slower; if it is bigger than your system RAM too, expect swapping or a failed load. ollama ps shows the CPU and GPU split, so consider a smaller model if much of it is on the CPU. Hermes’s own example figures for CPU-only machines are about 10 tokens a second for a 9B model on a modern 8-core processor and 2 to 5 for a 31B model, so a long answer can take a minute or two. Treat them as rough.
• Connection refused. Check that the endpoint URL and port are right with curl http://localhost:11434/api/tags, then that Ollama is running (ollama serve, or sudo systemctl start ollama on Linux). If Hermes runs in WSL2 and Ollama on Windows, localhost may not reach it; see the networking note in the setup section.
• Chat apps stop answering. If you connect Telegram or Discord to a local agent, the agent only answers while the computer, Ollama and the Hermes gateway are all running.

Local or hosted: which fits

Local suits you if privacy is the point, if you already own a machine with plenty of memory or a good GPU, or if you are experimenting and want no usage bill. Local inference avoids per-token model charges, but your own electricity is a cost, and web search or other cloud tools you switch on may bill separately. Expect to spend time on context settings, server flags and model choice, and expect the results to be weaker than a leading cloud model on hard multi-step work.

It suits you less if you want an agent that keeps working overnight, answers Telegram or WhatsApp when your laptop is shut, or runs scheduled jobs without a machine you have to keep awake. A laptop that sleeps cannot do that, and a desktop that stays on 24 hours a day has its own cost. Switching a local Hermes install to a cloud model does not fix that, because the agent itself still runs on your machine; what keeps working while your laptop is off is an agent hosted on an always-on machine. Where the model runs is a separate choice, and you can mix them. Our comparison of self-hosting and managed hosting goes through the same question for the whole agent.

Hermes on ReadyHermes

To be plain about it: ReadyHermes does not run models on your computer, and it does not offer GPU hosting for local models. Your own Hermes Agent runs on our servers, always on, so scheduled jobs still run and chat messages still get answers when your own computer is shut down. You use it from a browser workspace and from Telegram, WhatsApp, Discord, WeChat and others.

• AI is included: paid plans come with US$1 of AI a month, renewing on your billing date. The free hour runs on included models too.
• If you already pay an AI provider, a paid plan lets you connect your own key and runs on your own account, with no AI limit from us. The list includes OpenRouter, OpenAI, Anthropic, Google Gemini, DeepSeek, Groq, Qwen, Z.AI, Moonshot and MiniMax, plus an “other endpoint” for an OpenAI-compatible API. That endpoint must be a public HTTPS hostname on port 443, is checked with a test message before it is saved, and a few hosts are blocked.
• Addresses such as localhost, IP addresses and private-network addresses are refused, so you cannot point ReadyHermes straight at an Ollama running on your own computer or home network. Ollama Cloud is not in the built-in provider list.

If you want local models, run Hermes yourself as described above. If you want the agent always on without keeping a machine running, see managed Hermes Agent hosting.