Skip to content

Provider System

Hyper-Extract supports these ways to connect to LLMs: OpenAI, Anthropic, Google Gemini, Alibaba Bailian, DeepSeek, OrcaRouter, Atlas Cloud, Runcrate, Cerebras, Mistral AI, Moonshot (Kimi), Cheaper Inference, CloudGPU, llm.buzz, LLMTR, and local Ollama or vLLM. All use the same create_client() interface — only the first line changes.


Verified Model Compatibility

Cloud API

Platform Model Function calling Compatible Notes
OpenAI gpt-4o / gpt-4o-mini / gpt-5 ✅ ✅ Native support, recommended
Anthropic claude-opus-4-8 / claude-sonnet-4-6 / claude-haiku-4-5 ✅ (tool calling) ✅ LLM only — no embeddings API (pair with an OpenAI-compatible embedder). Needs hyperextract[anthropic]
Google Gemini gemini-3.8-flash / gemini-2.5-flash / gemini-2.5-pro ✅ (tool calling) ✅ LLM only — no embedder path in this repo (pair with an OpenAI-compatible embedder). Needs hyperextract[google]. Keys: GOOGLE_API_KEY or GEMINI_API_KEY. Alias: gemini
DeepSeek deepseek-v4-flash / deepseek-v4-pro ✅ ✅ OpenAI-compatible. V4 models default to "thinking" mode, which Hyper-Extract auto-disables (extra_body) so structured extraction works. LLM only — no embeddings API. Keys: DEEPSEEK_API_KEY
OrcaRouter orcarouter/auto, openai/gpt-4o-mini, anthropic/claude-haiku-4-5, deepseek/..., gemini/... ✅ ✅ OpenAI-compatible gateway routing 150+ models from OpenAI, Anthropic, Google, DeepSeek, Qwen, MiniMax, xAI behind one endpoint and key. Keys: ORCAROUTER_API_KEY
Cheaper Inference gpt-5.4-mini ✅ ✅ OpenAI-compatible gateway. One key gives access to models from several labs. LLM only — no embeddings API (pair with an OpenAI-compatible embedder). Keys: CHEAPER_INFERENCE_API_KEY
API Route gpt-6.1-sol ✅ ✅ OpenAI-compatible LLM preset. Configure an embedding provider separately. Key: API_ROUTE_API_KEY. Setup guide
Atlas Cloud deepseek-ai/deepseek-v4-flash, deepseek-ai/deepseek-v4-pro, moonshotai/kimi-k3, qwen/qwen3.5-plus, openai/gpt-5.5, anthropic/claude-sonnet-4.6 ✅ ✅ OpenAI-compatible gateway serving DeepSeek, Z.ai, Moonshot, Qwen, MiniMax, OpenAI, Anthropic and Google models under vendor-prefixed ids. LLM only — no embeddings API. glm-5.x models reject the forced tool_choice structured extraction uses; pick another model. Keys: ATLASCLOUD_API_KEY
Runcrate deepseek-ai/DeepSeek-V4.1-Flash, openai/gpt-oss-120b ✅ ✅ OpenAI-compatible API for open models under vendor-prefixed ids. Also serves embeddings (BAAI/bge-m3, 1024 dims), so one key covers the LLM and the embedder. Open models are free with your GitHub account. zai-org/GLM-5.x returned an empty graph in testing; pick another model. Keys: RUNCRATE_API_KEY
Ollama (local) gpt-oss:20b, qwen3:8b, and anything else ollama pull has fetched ✅ ✅ Local runtime serving an OpenAI-compatible subset on http://localhost:11434/v1, /v1/embeddings included, so one endpoint covers the LLM and the embedder (embeddinggemma). The key is required by the API shape but its value is ignored. Keys: OLLAMA_API_KEY (only needed for Ollama Cloud)
Cerebras qwen-3.8-27b, gpt-oss-120b (shared); kimi-k2.7-code, gemma-4-31b (dedicated) ✅ ✅ OpenAI-compatible inference endpoint; tool_choice accepts none/auto/required or a named function on every served model. LLM only — no embeddings API (pair with an OpenAI-compatible embedder). Keys: CEREBRAS_API_KEY
Mistral AI mistral-large-latest, mistral-small-latest, mistral-large-4, ministral-3-8b ✅ ✅ OpenAI-shaped /v1/chat/completions; tool_choice accepts auto/none/any/required or a named function. Also serves /v1/embeddings (mistral-embed, 1024 dims), so one key covers the LLM and the embedder. *-latest ids are rolling aliases — pin a dated id to avoid drift. Keys: MISTRAL_API_KEY
Moonshot (Kimi) kimi-k3.1-pro, kimi-k3.1-pro-highspeed, kimi-k3, kimi-k2.7-code ✅ ✅ OpenAI-compatible chat endpoint; tool_choice accepts a named function. Bare model ids, no vendor prefix. LLM only — no embeddings API (pair with an OpenAI-compatible embedder). Keys: MOONSHOT_API_KEY
CloudGPU deepseek-v4.1-flash ✅ ✅ OpenAI-compatible LLM preset. Model ids have no vendor prefix. Configure an embedding provider separately. Key: CLOUDGPU_API_KEY. Setup guide
llm.buzz openai/gpt-5.4-mini ✅ ✅ OpenAI-compatible gateway with vendor-prefixed model ids (list: GET https://api.llm.buzz/api/v1/models). LLM only in this preset (pair with an OpenAI-compatible embedder). Keys: LLM_BUZZ_API_KEY
LLMTR qwen/qwen3.8-27b-free, llmtr/qwen3-8-27b ✅ ✅ OpenAI-compatible gateway. Default model is the free qwen/qwen3.8-27b-free. llmtr/... ids keep the llmtr/ prefix (override with llmtr:llmtr/qwen3-8-27b). Also serves embeddings (llmtr/embeddingmagibu-200m), so one key covers the LLM and the embedder. Keys: LLMTR_API_KEY
Alibaba Bailian qwen-plus / qwen-turbo / qwen3.6-plus / deepseek-r1 ✅ ✅ Works out of the box
Alibaba Bailian qwen-max / deepseek-v3 ❌ ❌ Only json_object; not compatible with function-calling structured output

Bailian users: qwen-max and deepseek-v3 only support json_object and are not compatible with Hyper-Extract's function-calling structured output. If you hit messages must contain the word 'json' or get non-JSON output, switch to qwen-plus, qwen-turbo, or deepseek-r1. See Issue #24.

Local Deployment

Service Model Quantization VRAM Verified
LLM Qwen3.5-9B GPTQ-Marlin 4bit ~8GB ✅ AutoList / AutoGraph
Embedding BAAI/bge-m3 None ~2GB ✅ Semantic search

Cloud Embedding

Platform Model Dimensions Verified
OpenAI text-embedding-3-small 1536 ✅
Alibaba Bailian text-embedding-v4 1024 ✅

Quick Start by Platform

from hyperextract import create_client

# OpenAI
llm, emb = create_client("openai", api_key="sk-xxx")

# Bailian
llm, emb = create_client("bailian", api_key="sk-xxx")

# Anthropic (Claude) — LLM only; Anthropic has no embeddings API, so
# pair it with an OpenAI-compatible embedder.
# Keys: ANTHROPIC_API_KEY (or CLAUDE_API_KEY) for the LLM, OPENAI_API_KEY for embeddings.
llm, emb = create_client(
    llm="anthropic",  # default model: claude-opus-4-8 (override with "anthropic:<model>")
    embedder="openai:text-embedding-3-small",
)

# Google Gemini — native ChatGoogleGenerativeAI (not the OrcaRouter gemini/...
# gateway). LLM only; pair with an OpenAI-compatible embedder.
# Keys: GOOGLE_API_KEY (or GEMINI_API_KEY). Alias: "gemini".
# Extra: pip install 'hyperextract[google]'.
llm, emb = create_client(
    llm="google",  # default model: gemini-3.8-flash (override with "google:<model>")
    embedder="openai:text-embedding-3-small",
)

# DeepSeek — OpenAI-compatible, but no embeddings API. The V4 models
# default to "thinking" mode, which Hyper-Extract auto-disables so that
# tool-calling (function_calling) structured extraction works out of the box.
# Key: DEEPSEEK_API_KEY. Override model with "deepseek:deepseek-v4-pro".
llm, emb = create_client(
    llm="deepseek",
    embedder="openai:text-embedding-3-small",  # or vllm:bge-m3@localhost:8001/v1
)

# OrcaRouter — OpenAI-compatible gateway routing 150+ models (OpenAI,
# Anthropic, Google, DeepSeek, Qwen, MiniMax, xAI) behind one key. The default
# "orcarouter/auto" model routes to a suitable upstream; override with
# namespaced ids like "openai:gpt-4o-mini" or "anthropic:claude-haiku-4-5".
# Key: ORCAROUTER_API_KEY (fallback: OPENAI_API_KEY).
llm, emb = create_client("orcarouter")

# Atlas Cloud — OpenAI-compatible gateway with vendor-prefixed model ids
# (deepseek-ai/..., moonshotai/..., qwen/..., openai/..., anthropic/...).
# LLM only — pair it with an OpenAI-compatible embedder.
# Key: ATLASCLOUD_API_KEY. Override model with "atlascloud:moonshotai/kimi-k3".
llm, emb = create_client(
    llm="atlascloud",
    embedder="openai:text-embedding-3-small",
)

# Runcrate — OpenAI-compatible API for open models with vendor-prefixed ids.
# Serves embeddings too, so one key covers both.
# Key: RUNCRATE_API_KEY. Override model with "runcrate:openai/gpt-oss-120b".
llm, emb = create_client(
    llm="runcrate",  # default model: deepseek-ai/DeepSeek-V4.1-Flash
    embedder="runcrate",  # default embedder: BAAI/bge-m3
)

# LLMTR — OpenAI-compatible gateway. Default model is the free
# qwen/qwen3.8-27b-free. Ids such as llmtr/qwen3-8-27b keep the llmtr/
# prefix; override with "llmtr:llmtr/qwen3-8-27b".
# Serves embeddings too, so one key covers both.
# Key: LLMTR_API_KEY.
llm, emb = create_client(
    llm="llmtr",  # default model: qwen/qwen3.8-27b-free
    embedder="llmtr",  # default embedder: llmtr/embeddingmagibu-200m
)

# Ollama — local runtime, OpenAI-compatible on http://localhost:11434/v1.
# Serves embeddings on the same endpoint. The key value is ignored locally.
# Override model with "ollama:qwen3:8b" — the colon in a tag is kept whole.
llm, emb = create_client(
    llm="ollama",  # default model: gpt-oss:20b
    embedder="ollama",  # default embedder: embeddinggemma
    api_key="ollama",
# Cerebras — OpenAI-compatible inference endpoint, bare model ids.
# No embeddings API, so pair it with an OpenAI-compatible embedder.
# Key: CEREBRAS_API_KEY. Override model with "cerebras:gpt-oss-120b".
llm, emb = create_client(
    llm="cerebras",  # default model: qwen-3.8-27b
    embedder="openai:text-embedding-3-small",
)

# Mistral AI — OpenAI-shaped chat and embeddings on one key.
# Key: MISTRAL_API_KEY. Override model with "mistral:mistral-small-latest".
llm, emb = create_client(
    llm="mistral",  # default model: mistral-large-latest
    embedder="mistral",  # default embedder: mistral-embed
)

# Moonshot / Kimi — OpenAI-compatible chat endpoint, bare model ids.
# No embeddings API, so pair it with an OpenAI-compatible embedder.
# Key: MOONSHOT_API_KEY. Override model with "moonshot:kimi-k3".
llm, emb = create_client(
    llm="moonshot",  # default model: kimi-k3.1-pro
    embedder="openai:text-embedding-3-small",
# CloudGPU — OpenAI-compatible LLM API. Model ids have no vendor prefix.
# CloudGPU does not serve embeddings, so pick a separate embedder.
# Key: CLOUDGPU_API_KEY. Override model with "cloudgpu:kimi-k2.6".
llm, emb = create_client(
    llm="cloudgpu",  # default model: deepseek-v4.1-flash
    embedder="openai",
)

# Cheaper Inference — OpenAI-compatible gateway, one key for models from
# several labs. No embeddings API, so pair it with an OpenAI-compatible embedder.
# Key: CHEAPER_INFERENCE_API_KEY. Override model with "cheaperinference:gpt-5.4".
llm, emb = create_client(
    llm="cheaperinference",
    embedder="openai:text-embedding-3-small",
)

# Local vLLM
llm, emb = create_client(
    llm="vllm:Qwen3.5-9B@http://localhost:8000/v1",
    embedder="vllm:bge-m3@http://localhost:8001/v1",
    api_key="dummy",
)

Full configuration options: Provider Configuration Guide.


vLLM Deployment

Start LLM Service

vllm serve /path/to/qwen3.5-9b-gptq-marlin \
  --served-model-name Qwen/Qwen3.5-9B \
  --trust-remote-code \
  --quantization gptq_marlin \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --port 8000 \
  --api-key dummy

Start Embedding Service

vllm serve BAAI/bge-m3 \
  --task embed \
  --dtype float16 \
  --max-model-len 8192 \
  --port 8001

Docker

docker run --runtime nvidia --gpus all \
  --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3.5-9B \
  --trust-remote-code

Recommendations

Use non-thinking models for local vLLM deployment.

When deploying locally, thinking models (e.g. Qwen3.5 thinking mode) output <think>...</think> tags, which conflict with constrained decoding's requirement for JSON from the first token. Disable thinking mode when deploying Qwen3.5-9B:

--default-chat-template-kwargs '{"enable_thinking": false}'

DeepSeek-R1 via Bailian is verified to work — Bailian filters out <think> tags on the backend, so AutoGraph and with_structured_output work correctly. If accessing DeepSeek-R1 through other channels, thinking output may still cause issues.

Prefer GPTQ-Marlin over AWQ for quantization — AWQ has compatibility issues with vLLM 0.21.0.


References