Provider System¶
Hyper-Extract supports these ways to connect to LLMs: OpenAI, Anthropic, Google Gemini, Alibaba Bailian, DeepSeek, OrcaRouter, Atlas Cloud, Runcrate, Cerebras, Mistral AI, Moonshot (Kimi), Cheaper Inference, CloudGPU, llm.buzz, LLMTR, and local Ollama or vLLM. All use the same create_client() interface — only the first line changes.
Verified Model Compatibility¶
Cloud API¶
| Platform | Model | Function calling | Compatible | Notes |
|---|---|---|---|---|
| OpenAI | gpt-4o / gpt-4o-mini / gpt-5 | ✅ | ✅ | Native support, recommended |
| Anthropic | claude-opus-4-8 / claude-sonnet-4-6 / claude-haiku-4-5 | ✅ (tool calling) | ✅ | LLM only — no embeddings API (pair with an OpenAI-compatible embedder). Needs hyperextract[anthropic] |
| Google Gemini | gemini-3.8-flash / gemini-2.5-flash / gemini-2.5-pro | ✅ (tool calling) | ✅ | LLM only — no embedder path in this repo (pair with an OpenAI-compatible embedder). Needs hyperextract[google]. Keys: GOOGLE_API_KEY or GEMINI_API_KEY. Alias: gemini |
| DeepSeek | deepseek-v4-flash / deepseek-v4-pro | ✅ | ✅ | OpenAI-compatible. V4 models default to "thinking" mode, which Hyper-Extract auto-disables (extra_body) so structured extraction works. LLM only — no embeddings API. Keys: DEEPSEEK_API_KEY |
| OrcaRouter | orcarouter/auto, openai/gpt-4o-mini, anthropic/claude-haiku-4-5, deepseek/..., gemini/... |
✅ | ✅ | OpenAI-compatible gateway routing 150+ models from OpenAI, Anthropic, Google, DeepSeek, Qwen, MiniMax, xAI behind one endpoint and key. Keys: ORCAROUTER_API_KEY |
| Cheaper Inference | gpt-5.4-mini | ✅ | ✅ | OpenAI-compatible gateway. One key gives access to models from several labs. LLM only — no embeddings API (pair with an OpenAI-compatible embedder). Keys: CHEAPER_INFERENCE_API_KEY |
| API Route | gpt-6.1-sol | ✅ | ✅ | OpenAI-compatible LLM preset. Configure an embedding provider separately. Key: API_ROUTE_API_KEY. Setup guide |
| Atlas Cloud | deepseek-ai/deepseek-v4-flash, deepseek-ai/deepseek-v4-pro, moonshotai/kimi-k3, qwen/qwen3.5-plus, openai/gpt-5.5, anthropic/claude-sonnet-4.6 |
✅ | ✅ | OpenAI-compatible gateway serving DeepSeek, Z.ai, Moonshot, Qwen, MiniMax, OpenAI, Anthropic and Google models under vendor-prefixed ids. LLM only — no embeddings API. glm-5.x models reject the forced tool_choice structured extraction uses; pick another model. Keys: ATLASCLOUD_API_KEY |
| Runcrate | deepseek-ai/DeepSeek-V4.1-Flash, openai/gpt-oss-120b |
✅ | ✅ | OpenAI-compatible API for open models under vendor-prefixed ids. Also serves embeddings (BAAI/bge-m3, 1024 dims), so one key covers the LLM and the embedder. Open models are free with your GitHub account. zai-org/GLM-5.x returned an empty graph in testing; pick another model. Keys: RUNCRATE_API_KEY |
| Ollama (local) | gpt-oss:20b, qwen3:8b, and anything else ollama pull has fetched |
✅ | ✅ | Local runtime serving an OpenAI-compatible subset on http://localhost:11434/v1, /v1/embeddings included, so one endpoint covers the LLM and the embedder (embeddinggemma). The key is required by the API shape but its value is ignored. Keys: OLLAMA_API_KEY (only needed for Ollama Cloud) |
| Cerebras | qwen-3.8-27b, gpt-oss-120b (shared); kimi-k2.7-code, gemma-4-31b (dedicated) |
✅ | ✅ | OpenAI-compatible inference endpoint; tool_choice accepts none/auto/required or a named function on every served model. LLM only — no embeddings API (pair with an OpenAI-compatible embedder). Keys: CEREBRAS_API_KEY |
| Mistral AI | mistral-large-latest, mistral-small-latest, mistral-large-4, ministral-3-8b |
✅ | ✅ | OpenAI-shaped /v1/chat/completions; tool_choice accepts auto/none/any/required or a named function. Also serves /v1/embeddings (mistral-embed, 1024 dims), so one key covers the LLM and the embedder. *-latest ids are rolling aliases — pin a dated id to avoid drift. Keys: MISTRAL_API_KEY |
| Moonshot (Kimi) | kimi-k3.1-pro, kimi-k3.1-pro-highspeed, kimi-k3, kimi-k2.7-code |
✅ | ✅ | OpenAI-compatible chat endpoint; tool_choice accepts a named function. Bare model ids, no vendor prefix. LLM only — no embeddings API (pair with an OpenAI-compatible embedder). Keys: MOONSHOT_API_KEY |
| CloudGPU | deepseek-v4.1-flash |
✅ | ✅ | OpenAI-compatible LLM preset. Model ids have no vendor prefix. Configure an embedding provider separately. Key: CLOUDGPU_API_KEY. Setup guide |
| llm.buzz | openai/gpt-5.4-mini |
✅ | ✅ | OpenAI-compatible gateway with vendor-prefixed model ids (list: GET https://api.llm.buzz/api/v1/models). LLM only in this preset (pair with an OpenAI-compatible embedder). Keys: LLM_BUZZ_API_KEY |
| LLMTR | qwen/qwen3.8-27b-free, llmtr/qwen3-8-27b |
✅ | ✅ | OpenAI-compatible gateway. Default model is the free qwen/qwen3.8-27b-free. llmtr/... ids keep the llmtr/ prefix (override with llmtr:llmtr/qwen3-8-27b). Also serves embeddings (llmtr/embeddingmagibu-200m), so one key covers the LLM and the embedder. Keys: LLMTR_API_KEY |
| Alibaba Bailian | qwen-plus / qwen-turbo / qwen3.6-plus / deepseek-r1 | ✅ | ✅ | Works out of the box |
| Alibaba Bailian | qwen-max / deepseek-v3 | ❌ | ❌ | Only json_object; not compatible with function-calling structured output |
Bailian users: qwen-max and deepseek-v3 only support
json_objectand are not compatible with Hyper-Extract's function-calling structured output. If you hitmessages must contain the word 'json'or get non-JSON output, switch to qwen-plus, qwen-turbo, or deepseek-r1. See Issue #24.
Local Deployment¶
| Service | Model | Quantization | VRAM | Verified |
|---|---|---|---|---|
| LLM | Qwen3.5-9B | GPTQ-Marlin 4bit | ~8GB | ✅ AutoList / AutoGraph |
| Embedding | BAAI/bge-m3 | None | ~2GB | ✅ Semantic search |
Cloud Embedding¶
| Platform | Model | Dimensions | Verified |
|---|---|---|---|
| OpenAI | text-embedding-3-small | 1536 | ✅ |
| Alibaba Bailian | text-embedding-v4 | 1024 | ✅ |
Quick Start by Platform¶
from hyperextract import create_client
# OpenAI
llm, emb = create_client("openai", api_key="sk-xxx")
# Bailian
llm, emb = create_client("bailian", api_key="sk-xxx")
# Anthropic (Claude) — LLM only; Anthropic has no embeddings API, so
# pair it with an OpenAI-compatible embedder.
# Keys: ANTHROPIC_API_KEY (or CLAUDE_API_KEY) for the LLM, OPENAI_API_KEY for embeddings.
llm, emb = create_client(
llm="anthropic", # default model: claude-opus-4-8 (override with "anthropic:<model>")
embedder="openai:text-embedding-3-small",
)
# Google Gemini — native ChatGoogleGenerativeAI (not the OrcaRouter gemini/...
# gateway). LLM only; pair with an OpenAI-compatible embedder.
# Keys: GOOGLE_API_KEY (or GEMINI_API_KEY). Alias: "gemini".
# Extra: pip install 'hyperextract[google]'.
llm, emb = create_client(
llm="google", # default model: gemini-3.8-flash (override with "google:<model>")
embedder="openai:text-embedding-3-small",
)
# DeepSeek — OpenAI-compatible, but no embeddings API. The V4 models
# default to "thinking" mode, which Hyper-Extract auto-disables so that
# tool-calling (function_calling) structured extraction works out of the box.
# Key: DEEPSEEK_API_KEY. Override model with "deepseek:deepseek-v4-pro".
llm, emb = create_client(
llm="deepseek",
embedder="openai:text-embedding-3-small", # or vllm:bge-m3@localhost:8001/v1
)
# OrcaRouter — OpenAI-compatible gateway routing 150+ models (OpenAI,
# Anthropic, Google, DeepSeek, Qwen, MiniMax, xAI) behind one key. The default
# "orcarouter/auto" model routes to a suitable upstream; override with
# namespaced ids like "openai:gpt-4o-mini" or "anthropic:claude-haiku-4-5".
# Key: ORCAROUTER_API_KEY (fallback: OPENAI_API_KEY).
llm, emb = create_client("orcarouter")
# Atlas Cloud — OpenAI-compatible gateway with vendor-prefixed model ids
# (deepseek-ai/..., moonshotai/..., qwen/..., openai/..., anthropic/...).
# LLM only — pair it with an OpenAI-compatible embedder.
# Key: ATLASCLOUD_API_KEY. Override model with "atlascloud:moonshotai/kimi-k3".
llm, emb = create_client(
llm="atlascloud",
embedder="openai:text-embedding-3-small",
)
# Runcrate — OpenAI-compatible API for open models with vendor-prefixed ids.
# Serves embeddings too, so one key covers both.
# Key: RUNCRATE_API_KEY. Override model with "runcrate:openai/gpt-oss-120b".
llm, emb = create_client(
llm="runcrate", # default model: deepseek-ai/DeepSeek-V4.1-Flash
embedder="runcrate", # default embedder: BAAI/bge-m3
)
# LLMTR — OpenAI-compatible gateway. Default model is the free
# qwen/qwen3.8-27b-free. Ids such as llmtr/qwen3-8-27b keep the llmtr/
# prefix; override with "llmtr:llmtr/qwen3-8-27b".
# Serves embeddings too, so one key covers both.
# Key: LLMTR_API_KEY.
llm, emb = create_client(
llm="llmtr", # default model: qwen/qwen3.8-27b-free
embedder="llmtr", # default embedder: llmtr/embeddingmagibu-200m
)
# Ollama — local runtime, OpenAI-compatible on http://localhost:11434/v1.
# Serves embeddings on the same endpoint. The key value is ignored locally.
# Override model with "ollama:qwen3:8b" — the colon in a tag is kept whole.
llm, emb = create_client(
llm="ollama", # default model: gpt-oss:20b
embedder="ollama", # default embedder: embeddinggemma
api_key="ollama",
# Cerebras — OpenAI-compatible inference endpoint, bare model ids.
# No embeddings API, so pair it with an OpenAI-compatible embedder.
# Key: CEREBRAS_API_KEY. Override model with "cerebras:gpt-oss-120b".
llm, emb = create_client(
llm="cerebras", # default model: qwen-3.8-27b
embedder="openai:text-embedding-3-small",
)
# Mistral AI — OpenAI-shaped chat and embeddings on one key.
# Key: MISTRAL_API_KEY. Override model with "mistral:mistral-small-latest".
llm, emb = create_client(
llm="mistral", # default model: mistral-large-latest
embedder="mistral", # default embedder: mistral-embed
)
# Moonshot / Kimi — OpenAI-compatible chat endpoint, bare model ids.
# No embeddings API, so pair it with an OpenAI-compatible embedder.
# Key: MOONSHOT_API_KEY. Override model with "moonshot:kimi-k3".
llm, emb = create_client(
llm="moonshot", # default model: kimi-k3.1-pro
embedder="openai:text-embedding-3-small",
# CloudGPU — OpenAI-compatible LLM API. Model ids have no vendor prefix.
# CloudGPU does not serve embeddings, so pick a separate embedder.
# Key: CLOUDGPU_API_KEY. Override model with "cloudgpu:kimi-k2.6".
llm, emb = create_client(
llm="cloudgpu", # default model: deepseek-v4.1-flash
embedder="openai",
)
# Cheaper Inference — OpenAI-compatible gateway, one key for models from
# several labs. No embeddings API, so pair it with an OpenAI-compatible embedder.
# Key: CHEAPER_INFERENCE_API_KEY. Override model with "cheaperinference:gpt-5.4".
llm, emb = create_client(
llm="cheaperinference",
embedder="openai:text-embedding-3-small",
)
# Local vLLM
llm, emb = create_client(
llm="vllm:Qwen3.5-9B@http://localhost:8000/v1",
embedder="vllm:bge-m3@http://localhost:8001/v1",
api_key="dummy",
)
Full configuration options: Provider Configuration Guide.
vLLM Deployment¶
Start LLM Service¶
vllm serve /path/to/qwen3.5-9b-gptq-marlin \
--served-model-name Qwen/Qwen3.5-9B \
--trust-remote-code \
--quantization gptq_marlin \
--dtype bfloat16 \
--max-model-len 8192 \
--gpu-memory-utilization 0.90 \
--default-chat-template-kwargs '{"enable_thinking": false}' \
--port 8000 \
--api-key dummy
Start Embedding Service¶
Docker¶
docker run --runtime nvidia --gpus all \
--ipc=host \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 8000:8000 \
vllm/vllm-openai:latest \
--model Qwen/Qwen3.5-9B \
--trust-remote-code
Recommendations¶
Use non-thinking models for local vLLM deployment.
When deploying locally, thinking models (e.g. Qwen3.5 thinking mode) output <think>...</think> tags, which conflict with constrained decoding's requirement for JSON from the first token. Disable thinking mode when deploying Qwen3.5-9B:
DeepSeek-R1 via Bailian is verified to work — Bailian filters out <think> tags on the backend, so AutoGraph and with_structured_output work correctly. If accessing DeepSeek-R1 through other channels, thinking output may still cause issues.
Prefer GPTQ-Marlin over AWQ for quantization — AWQ has compatibility issues with vLLM 0.21.0.