Using Groq with Hermes Agent: setup, models, limits and voice notes

What Hermes includes for Groq, and what it does not

For chat, Groq is not a built-in provider in Hermes Agent 0.22.5. It has no entry in the provider list that hermes model offers, it is not in the table of providers the fallback documentation supports, and there is no Groq provider plugin. Groq does run an OpenAI-compatible API, and Hermes’ own documentation lists Groq at https://api.groq.com/openai/v1 as one of the services that work through a custom endpoint, with a worked recipe. That custom endpoint is the route that really works for the main model.

The built-in part is speech-to-text. Hermes has a Groq transcription provider that readsGROQ_API_KEY and calls Groq’s Whisper models, and it is one of the choices for transcribing voice notes. That is a separate feature from the chat model. The two can share one Groq key, but nothing makes them do so; the voice-notes section below covers it.

Hermes also contains a few Groq-specific accommodations inside its custom-endpoint code, for example how it sends the reasoning setting to api.groq.com, and it knows Groq is a strict OpenAI-compatible host that rejects extra fields. So a Groq custom endpoint is a shape Hermes’ own code and documentation plan for, just not a named provider.

Get a key and point Hermes at Groq

Create an API key in the Groq console at console.groq.com/keys. Groq’s quickstart suggests keeping it in an environment variable called GROQ_API_KEY, and that is also the name the Hermes recipe uses. Put it in ~/.hermes/.env:

GROQ_API_KEY=your-groq-key

Then add a named custom provider and make it the main model in ~/.hermes/config.yaml:

providers:
  groq:
    api: https://api.groq.com/openai/v1
    key_env: GROQ_API_KEY

model:
  default: openai/gpt-oss-120b
  provider: custom:groq

This is the shape from Hermes’ own recipe, with one change: the recipe names llama-3.3-70b-versatile, and Groq retired that model for free and developer accounts on 16 August 2026 (see the next section), so the example above uses a current model instead. Restart the gateway if you run one, or start a new chat.

If you would rather not edit YAML, run hermes model, choose Custom endpoint, and enter the base URL https://api.groq.com/openai/v1, your key and a model name. The wizard also asks for the API mode and an optional context length; leave the context length blank to auto-detect. Once the named provider exists you can switch to it for one session with /model custom:groq:openai/gpt-oss-120b.

Groq’s /models endpoint lists every active model, speech and safety models included, so a model picker that reads it will show ids that are not chat models. Pick from the chat models in the next section. If you are new to the other provider routes, the Mistral page walks through the same custom-endpoint pattern for another strict OpenAI-compatible host.

Which Groq models work for an agent, and which support tool calling

An agent lives on tool calling: every file read, search or command is a tool call the model has to format correctly. Groq’s tool-use documentation says all models hosted on Groq support tool use, and its table marks the following as supporting local tool use and JSON mode. From Groq’s models page on 2 November 2026:

• openai/gpt-oss-120b: listed as a production model, about 500 tokens per second, 131,072-token context window, up to 65,536 completion tokens, US$0.15 input and US$0.60 output per million tokens. Tool use yes, parallel tool use no.
• openai/gpt-oss-20b: production, about 1,000 tokens per second, the same 131,072-token window and 65,536 completion limit, US$0.075 input and US$0.30 output per million. Tool use yes, parallel tool use no.
• qwen/qwen3.8-27b: listed as a preview model, about 450 tokens per second, 131,072-token window, up to 16,384 completion tokens, US$0.80 input and US$4.00 output per million. Tool use and parallel tool use yes, and it accepts images. Groq says preview models can be discontinued at short notice.

llama-3.3-70b-versatile and llama-3.1-8b-instant are still on the models page, but marked Enterprise with prices on request. Groq’s deprecations page says it announced their retirement on 17 June 2026, with a shutdown date of 16 August 2026 for free and developer-tier use andopenai/gpt-oss-20b or openai/gpt-oss-120b as the suggested replacements. If an old tutorial, or the Hermes recipe, gives you a Llama id and you get a model error, that is the likely reason.

Two Hermes details help. For this Groq setup, Hermes requires a context window of at least 64,000 tokens; all three models above clear that comfortably. And Hermes adds extra tool-use guidance to the system prompt for model names that contain strings such as gpt or qwen (the agent.tool_use_enforcement setting, on auto by default), which both families match.

One thing to watch is the reasoning setting. When a reasoning effort is set, and Hermes fills in medium for a custom endpoint when you have not set one, the code sends reasoning_effort: default to api.groq.com, because a comment in the Hermes source says Groq’s compatible wire accepted only none or default there and returned a 400 for graded levels. Groq’s current reasoning page, however, says none and default are supported only by Qwen 3.8 27B, while the GPT OSS models take low, medium and high. We have not tested this against a live Groq key. Hermes has a recovery path for a rejected reasoning setting: when it classifies the error that way, it drops the reasoning fields and retries once using the route’s default for the session. If the request still fails, read the returned error before changing models.

Rate limits and context on Groq's free tier

Groq’s rate-limits page has a Free Plan tab and a Developer Plan tab. Read on 2 November 2026, the Free Plan row for openai/gpt-oss-120b is 30 requests per minute, 1,000 requests per day, 8,000 tokens per minute and 200,000 tokens per day. openai/gpt-oss-20b and qwen/qwen3.8-27b have the same figures. Groq’s models page shows 250,000 tokens per minute and 1,000 requests per minute for the two GPT OSS models on the Developer plan. Groq says these are a high-level summary, that limits apply to the organisation rather than to each key or user, and that your exact numbers are on the Limits page of your console.

For an agent, the token figure is the one that bites. Each request carries the system prompt, the conversation so far and the definitions of every enabled tool, and one chat message from you can trigger several requests: Hermes’ own setup text warns that it typically makes 3 to 10 API calls per user turn. We have not measured the size of a Hermes request, but on a limit of 8,000 tokens per minute a few tool calls in a row are likely to hit it. The same Hermes text lists Groq among the alternatives with workable free usage, so it will run, but a busy assistant belongs on the Developer plan rather than the free one.

When you exceed a limit Groq returns 429 Too Many Requests with a retry-after header, and the other rate-limit headers are sent on every response. Cached tokens do not count against the limits. Hermes can switch to a configured fallback straight away on a rate limit, so it does not always use up its retries first. For the context window itself, the models above give you 131,072 tokens in and a completion cap set by the model. The Qwen model’s completion cap is 16,384 tokens, well below the GPT OSS models’ 65,536.

Fast voice-note transcription with Groq Whisper

This is the built-in half. When someone sends a voice note on Telegram or WhatsApp, Hermes transcribes it with whichever speech-to-text provider is set, and Groq is one of the choices alongside local faster-whisper, OpenAI, Mistral and others. Set it in config.yaml:

stt:
  provider: groq
  language: ""
  groq:
    model: whisper-large-v3-turbo

The key is GROQ_API_KEY in ~/.hermes/.env, the same variable the chat recipe above points at, so with that recipe one Groq key serves both jobs. The custom-endpoint wizard stores its key in a host-specific variable in ~/.hermes/.env instead, so set GROQ_API_KEY as well if you want transcription to have its own credential. You can use a different key for each job to separate usage. The Groq model defaults to whisper-large-v3-turbo, with whisper-large-v3 as the other option, and an old config naming the retired distil-whisper-large-v3-en is remapped to the default.

Two settings are worth knowing. First, Hermes ships a global language hint of en, and a blank per-provider value falls through to it, so voice notes in other languages are forced to English unless you blank the global one too, as above. Second, an explicit provider is honoured strictly: if the Groq key is missing, transcription fails with guidance rather than quietly switching. Without an explicit provider, Hermes picks local first, then Groq, so a machine with faster-whisper installed will not use Groq until you ask it to.

From Groq’s speech-to-text page on 2 November 2026: the two Whisper models cost US$0.04 per hour of audio (turbo) and US$0.111 per hour (large-v3); the maximum file size is 25 MB on the free tier and 100 MB on the developer tier; and flac, mp3, mp4, m4a, ogg, wav and webm among others are accepted, which covers the .ogg Opus files that WhatsApp voice notes arrive as. The Free Plan row is 20 requests per minute and 7,200 audio seconds per hour. For how voice notes reach the agent on each app, see the WhatsApp and Telegram guides.

A fallback for when Groq rate-limits you

Hermes can switch to a second provider in the middle of a session. Run the manager:

hermes fallback add
hermes fallback list

hermes fallback add opens the same picker as hermes model and appends your choice to a fallback_providers list in config.yaml; list, remove and clear manage the chain, and it skips a model deployment that matches the primary or an existing chain entry. The simplest setup for a Groq primary is a second provider you already pay for, such as OpenRouter. If Groq is the backup instead, add it as a custom entry, since Groq is not in the list of named fallback providers:

fallback_providers:
  - provider: custom
    model: openai/gpt-oss-120b
    base_url: https://api.groq.com/openai/v1
    key_env: GROQ_API_KEY

Fallback can happen straight away on rate limits, after recovery attempts for authentication failures, and after retries for server or connection errors. A recognised model-not-found error can trigger it, but a generic 404 does not. When a rate-limit response says when it resets, Hermes benches the primary until then; set fallback.min_switch_reset_seconds if you would rather wait out a short window than swap models. The commands and config shape here were checked against the 0.22.5 source, not run with a live Groq key, so after you add an entry confirm it with hermes fallback list and try a deliberate failure, for example a temporarily wrong key on the primary.

Groq on ReadyHermes

ReadyHermes runs your own Hermes Agent on our servers, always on, so you do not edit YAML or run a gateway. Groq is a named provider in the workspace’s Model key tab, labelled Fast inference. Choose Groq, paste your key, press Load models or type the model name, then press Check and switch. The tab sends one small test message with your key first, and if Groq refuses the key or the model you see the reason and nothing changes. Switching restarts the assistant, which takes about a minute. Groq bills you directly, the tab is open during the free trial as well as on paid plans, and a key added in the trial stays when you choose a plan.

Under the hood the tab sets the same kind of custom endpoint as the recipe above: the main model becomes https://api.groq.com/openai/v1 with your key and the model name you entered. Three things to know. Load models shows what Groq returns, speech and safety models included, so pick a chat model from the list in this page. The tab does not set up a fallback, and saving a key through it clears any fallback_providers list on the assistant. And it does not touch speech-to-text: it does not set GROQ_API_KEY or switch transcription to Groq. ReadyHermes provides its own speech-to-text route for voice notes and keeps a transcription provider you explicitly chose.

New assistants are ready in about ten seconds, and the free trial lasts one hour. If you want to see what a key costs against what a plan includes, the page on whether Hermes Agent is free explains both.