Inferrail MCP Server

Inferrail

A self-hosted, OpenAI-compatible gateway that sits between your app and your LLM provider, so every request produces a payload-free economic receipt — measured cost and business attribution, with no prompt or response ever stored.

Point your existing OpenAI client at Inferrail instead of directly at your provider. Nothing else changes — same request/response shape, same streaming, same tool calls — except now every call gets logged locally with its real token usage, its verified cost, and whatever customer/workflow/feature you attribute it to.

Developer preview (v0.1.1). Works today; CLI flags, config shape, and receipt fields may still change before 1.0 — see docs/PRODUCT.md.

Why Inferrail

• Know what each request cost, and who it was for — without adding a hosted observability vendor or logging prompts yourself.
• One place to point an OpenAI-compatible client instead of provider-specific SDK code sprinkled through your app.
• Runs entirely on your own machine. No Inferrail-operated service exists yet, and none of this depends on one.
• Structurally can't store your prompts. The receipt and telemetry schemas have no field capable of holding message content — not a setting, a guarantee.

Install

pip install inferrail

Requires Python 3.11+.

Quickstart

See the whole pipeline — request → receipt → cost report — with no API key and no network call:

inferrail demo

Runs canned requests through Inferrail's real engine using a fake in-memory provider. Every price and response is clearly labeled DEMO.

To try it for real, with your own key:

export OPENAI_API_KEY=<your-openai-api-key>
inferrail try "Reply with one word: ready" --customer acme

ready

Receipt ir_670b20135cfe4bcb8f6f
  Provider          openai
  Model             gpt-4o-mini
  Input tokens      12
  Output tokens     1
  Cost              $0.000002
  Customer          acme
  Prompt stored     no
  Response stored   no

Saved to ./inferrail-receipts.jsonl

Next:
  inferrail report --by customer

inferrail report --by customer (or provider, model, route, or any attribute you've attached) aggregates every receipt written so far into a table of requests, tokens, and cost.

Send a request

Run the gateway itself with the same zero-config defaults:

inferrail serve --quickstart

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Inferrail-Attribute-Customer: acme" \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'

The response is standard OpenAI choices/usage plus a non-standard inferrail block (route, provider, latency, retries) any OpenAI client already ignores. X-Inferrail-Attribute-* headers are optional attribution — never forwarded upstream. See examples/basic_chat_request.py for a minimal Python client, or point any existing OpenAI-compatible SDK (LangChain, LlamaIndex, CrewAI, ...) at http://127.0.0.1:8000/v1.

# LangChain
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="not-needed",  # or your INFERRAIL_GATEWAY_TOKEN if auth is enabled
    model="default",
)

# LlamaIndex
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="default",
    api_base="http://127.0.0.1:8000/v1",
    api_key="not-needed",
    is_chat_model=True,
    context_window=8192,
)

# CrewAI
from crewai import LLM

llm = LLM(
    model="openai/default",  # "openai/" prefix required by CrewAI
    base_url="http://127.0.0.1:8000/v1",
    api_key="not-needed",
)

What Inferrail records

Every request writes one payload-free JSON receipt — no field on it can hold a prompt or response:

{
  "receipt_id": "ir_1e6c916bac8940ca8a85",
  "route": "default",
  "provider": "openai",
  "model": "gpt-4o-mini",
  "status": "success",
  "prompt_tokens": 842,
  "completion_tokens": 191,
  "pricing": {
    "input_usd_per_million": "0.15",
    "output_usd_per_million": "0.60",
    "source": "https://developers.openai.com/api/docs/pricing",
    "verified_date": "2026-08-16"
  },
  "estimated_cost_usd": "0.000241",
  "attributes": { "customer": "acme", "workflow": "contract-review" },
  "total_latency_ms": 15.96,
  "retry_count": 0
}

If Inferrail can't verify a price for the (provider, model) pair, pricing and estimated_cost_usd are null — never a guessed or fabricated cost. All money math uses Decimal, never float. Details: docs/adr/0005.

Model routing

"model" normally selects a named route from inferrail.yaml (e.g. "default"), which maps to a provider + underlying model. If default_provider is set in your config, a model that matches no route is instead forwarded to that provider unchanged — so "model": "gpt-5.6-sol" works with no route pre-registered for it. Named routes always take priority. This passthrough is on by default for the zero-config quickstart path, off by default otherwise. Full design: docs/adr/0007.

MCP

pip install "inferrail[mcp]"

An MCP server (inferrail-mcp), published on the MCP registry as io.github.domondi1/inferrail, exposes Inferrail's local receipt ledger to any MCP-aware agent (Claude Code, Claude Desktop, Cursor, ...) as two read-only tools — neither executes inference or spends provider budget:

{
  "mcpServers": {
    "inferrail": { "command": "inferrail-mcp" }
  }
}

Claude Code: claude mcp add inferrail -- inferrail-mcp. Full contract: inferrail-mcp/README.md.

Configuration

For a real deployment instead of quickstart defaults:

cp inferrail.example.yaml inferrail.yaml
cp .env.example .env      # then add a real OPENAI_API_KEY
inferrail config check    # validate without starting a server
inferrail serve

inferrail.yaml only ever holds the name of an environment variable for a secret, never the secret itself. Full shape (providers, routes, telemetry, receipts, pricing overrides): inferrail.example.yaml.

By default the gateway binds to 127.0.0.1:8000 with no auth. Set INFERRAIL_GATEWAY_TOKEN to require callers to send Authorization: Bearer <token> — see SECURITY.md.

What works today

• POST /v1/chat/completions: streaming (stream: true, real SSE passthrough) and tool/function calling, single string message content, no n != 1
• GET /health
• One provider adapter, generic over any OpenAI-compatible HTTP endpoint
• Named-route + optional passthrough model routing (above)
• Per-route retry with backoff on transient provider errors
• Local structured telemetry and payload-free cost receipts for every request, plus inferrail report --by <dimension>
• CLI: inferrail demo, try, serve (--quickstart), config check, report

Not yet: multi-provider intelligent routing, cost estimates for models outside the built-in catalog or an explicit pricing: override, budgets/spend limits, any non-OpenAI-compatible provider. Full scope: docs/PRODUCT.md.

Documentation

• docs/PRODUCT.md — exact current scope
• docs/ARCHITECTURE.md — package layout, request lifecycle
• docs/adr/ — why specific structural decisions were made
• openapi.json / config.schema.json / llms.txt — machine-readable references for tooling and agents
• SECURITY.md

Development

git clone https://github.com/domondi1/inferrail.git && cd inferrail
pip install -e ".[dev,mcp]"
ruff check . && mypy && pytest

pytest needs no API key or network access — see CONTRIBUTING.md.

License

Apache License 2.0 — see LICENSE.