Back to Browse

Okareo MCP Server

Developer ToolsUse Caution4.2MCP RegistryLocalRemote
Free

Server data from the Official MCP Registry

Simulation, evaluation and monitoring for voice agents.

About

Simulation, evaluation and monitoring for voice agents.

Remote endpoints: streamable-http: https://tools.okareo.com/mcp

Security Report

4.2
Use Caution4.2High Risk

The Okareo MCP server is a well-structured tool for AI model evaluation with appropriate authentication mechanisms (OAuth for remote, API key for local) and reasonable permission scoping. However, there are concerns around sensitive credential handling in custom endpoint configurations, unclear data validation on user-supplied templates, and potential issues with how redacted values are managed. The codebase shows good error handling but has some gaps in input validation for complex nested structures. Supply chain analysis found 5 known vulnerabilities in dependencies (0 critical, 2 high severity).

4 files analyzed · 13 issues found

Security scores are indicators to help you make informed decisions, not guarantees. Always review permissions before connecting any MCP server.

Permissions Required

This plugin requests these system permissions. Most are normal for its category.

env_vars

Check that this permission is expected for this type of plugin.

HTTP Network Access

Connects to external APIs or services over the internet.

File System Read

Reads files on your machine. Normal for tools that analyze or process local data.

How to Install & Connect

Available as Local & Remote

This plugin can run on your machine or connect to a hosted endpoint. during install.

Documentation

View on GitHub

From the project's GitHub README.

Okareo MCP Server

The Okareo MCP server exposes Okareo's evaluation capabilities as MCP tools, allowing AI coding assistants to create and manage scenarios, register models, run evaluations, and execute multi-turn simulations directly from your editor.

For detailed documentation, see the Okareo MCP docs.

Prerequisites

  • An Okareo account at app.okareo.com
  • A copilot that supports MCP servers (Claude Code, Cursor, or VS Code)
  • Python 3.10–3.12 (only for the local install modes; not needed for remote)

Remote MCP (hosted) — recommended

The fastest way to get started. No local install, no Python, no API key in .mcp.json. Browser sign-in handles auth on first connect.

Recommended (OAuth — Claude Code, Claude Desktop, Cursor, VS Code 1.101+)

Add to your copilot's MCP config (typically .mcp.json):

{
  "mcpServers": {
    "okareo": {
      "type": "http",
      "url": "https://tools.okareo.com/mcp"
    }
  }
}

Reload the copilot. It will open a browser to Okareo sign-in once; thereafter the copilot stores the token itself. Your .mcp.json contains no secrets.

Fallback (Bearer header — older clients or headless / CI)

For clients that haven't shipped the MCP OAuth flow yet, paste your API key as a bearer header. Prefer the env-var form over an inline literal:

{
  "mcpServers": {
    "okareo": {
      "type": "http",
      "url": "https://tools.okareo.com/mcp",
      "headers": {
        "Authorization": "Bearer ${env:OKAREO_API_KEY}"
      }
    }
  }
}

Working across multiple Okareo organizations

If you have access to more than one Okareo account, simply Clear Authentication and then re-authenticate with the account you want to work in. You can always ask which account your are in and have acccess to.


Local install (alternative)

Run the MCP server on your own machine. Useful for offline / airgapped environments and for development.

Prefer the Remote MCP section above unless you have a specific reason to install locally (airgapped, custom build, development on this repo). The remote endpoint requires no install and stays current automatically.

For multi-org users, the remote endpoint also exposes tenant management tools (list_tenants, switch_tenant).

Step 1: Set Your API Key

export OKAREO_API_KEY="your-api-key"

Add this to your ~/.zshrc or ~/.bash_profile for persistence.

Step 2: Configure Your Copilot

Claude Code

Add to .mcp.json:

{
  "mcpServers": {
    "okareo": {
      "command": "uvx",
      "args": ["okareo-mcp"],
      "env": {
        "OKAREO_API_KEY": "${OKAREO_API_KEY}"
      }
    }
  }
}

No pre-install needed — uvx handles it automatically.

Cursor

Add to .cursor/mcp.json:

{
  "mcpServers": {
    "okareo": {
      "command": "uvx",
      "args": ["okareo-mcp"]
    }
  }
}

Cursor inherits OKAREO_API_KEY from your shell environment.

Alternative: pip install

If you don't have uv installed:

pip install okareo-mcp

Then use "command": "okareo-mcp" instead of "command": "uvx" with "args": ["okareo-mcp"].


Configuration Reference

VariableDefaultDescription
OKAREO_API_KEY(required)Your Okareo API key
OKAREO_BASE_URLhttps://api.okareo.comOverride for on-prem Okareo backend
TRANSPORTstdioTransport: stdio (local) or sse (Docker)
PORT8000Port for SSE transport

Available Tools

Scenarios

ToolDescription
save_scenarioSave a named scenario from rows of input/result data (idempotent)
list_scenariosList all scenarios in the project with names, IDs, and row counts
get_scenarioRetrieve a scenario's metadata and all data rows by name or ID
create_scenario_versionCreate a new version of an existing scenario with updated data
preview_delete_scenarioPreview what will be deleted before removing a scenario
delete_scenarioPermanently delete a scenario and all related test data

Generation Models

ToolDescription
list_available_llmsBrowse available LLMs from the Okareo registry
register_generation_modelRegister a generation model for testing by selecting an LLM from the registry
list_generation_modelsList all registered generation models in the project
get_generation_modelRead detailed information about a registered generation model
update_generation_modelChange the LLM a registered generation model points to
delete_generation_modelRemove a registered generation model and all its related test data

Tests & Checks

ToolDescription
list_checksList available quality checks (built-in and custom) for evaluating model outputs
run_testRun a quality test that evaluates a model against a scenario using specified checks
list_test_runsList past test runs with optional filters (model, scenario, simulation-only)
get_test_run_resultsLoad detailed per-row results of a test run or simulation by ID or name
get_conversation_transcriptRetrieve the full conversation transcript for a single data point
reevaluate_test_runRe-score a completed test run against a (possibly different) set of checks
create_or_update_checkCreate or update a quality check by name — model-based, code-based, or audio (upsert)
generate_checkGenerate a check from a natural-language description, then save it
get_checkRetrieve a check's full configuration, including its prompt template or code
delete_checkPermanently delete a check by name

Simulations (Multi-Turn)

ToolDescription
create_or_update_targetCreate or update a Target — generation model, custom endpoint, or voice (OpenAI, Deepgram, Twilio)
get_targetRetrieve a Target's configuration by name (all types)
list_targetsList all simulation targets (voice and custom_endpoint) in the project
delete_targetRemove a simulation target and all its related test data
create_or_update_driverDefine a simulated user persona that will interact with your target
get_driverRetrieve a Driver's full configuration including the persona prompt
list_driversList all Driver personas in the project
list_driver_voicesDiscover the voices, voice profiles, and languages available for voice drivers
run_simulationRun a multi-turn conversation evaluation (or rerun a previous one with overrides)
list_simulationsList past simulation runs with optional filters (target, scenario, limit)

Voice Monitoring

ToolDescription
ingest_conversationsSubmit completed voice conversations to Okareo for monitoring
connect_voice_integrationConnect a voice provider so its traffic flows into Okareo monitoring
list_voice_integrationsList the voice provider integrations in your project
get_voice_integrationRetrieve a voice provider integration by ID, including its status
update_voice_integrationUpdate a voice provider integration's metadata
rotate_voice_integration_secretRotate a voice provider integration's secrets
delete_voice_integrationDelete a voice provider integration by ID
get_voice_webhook_urlGet the inbound webhook endpoint for a voice provider

Projects

ToolDescription
list_projectsList every project you can work in — id, name, tags, archive state — with the active one marked
select_projectChoose the project to work in for this conversation
create_projectCreate a new project
update_projectRename a project, or replace its tags
archive_projectHide a project from the project picker. Reversible; nothing is deleted
unarchive_projectRestore an archived project to the picker
clone_projectCopy an existing project's scenarios into a new project (dry-run first)

Okareo has no project delete — in the app, the API, or here. Archiving is the removal, and it only hides the project from the picker: its scenarios, runs, and dashboards stay intact and every tool still works against it.

Analytics & Dashboards

ToolDescription
query_analyticsQuery Okareo's product analytics to understand evaluation trends
list_dashboardsList the analytics dashboards in your project
get_dashboardRetrieve a dashboard's full configuration by name
save_dashboardCreate or update an analytics dashboard by name (upsert)
reorder_dashboardsSet the display order of dashboards
delete_dashboardDelete a dashboard by name

Tenant Management (remote MCP only)

ToolDescription
list_tenantsList every Okareo organization you have access to in this MCP session
switch_tenantChange which Okareo organization subsequent tool calls operate against

Documentation & Templates

ToolDescription
get_docsQuery the Okareo documentation system for conceptual or user-legible explanations
get_templatesRetrieve prompt templates for common Okareo patterns (works offline)
get_reps_baselineServe REPS agent-evaluation baseline material (scenario banks, drivers, checks, eval configs) from the latest tagged okareo-tools release

Troubleshooting

SymptomCauseFix
okareo-mcp: command not foundNot installed or not in PATHRun pip install -e . (dev) or use uvx okareo-mcp (user)
Server exits with API key errorOKAREO_API_KEY not setExport it: export OKAREO_API_KEY="..."
pip install fails on Python 3.13+Okareo SDK requires Python <3.13Use Python 3.10–3.12
Copilot can't connect (Docker)Wrong URLEnsure URL ends with /sse and port matches
Cursor doesn't pick up API keyCursor launched from Dock, not terminalLaunch Cursor from terminal: cursor .

Contributing

This repository is a curated public mirror; the canonical source is maintained by Okareo. We welcome issues and consider community pull requests — see CONTRIBUTING.md for how proposed changes are reviewed and ported.

License & Trademarks

The Okareo MCP server source code is licensed under the Apache License 2.0 (see also NOTICE).

"Okareo", the Okareo logo, and Okareo product names are trademarks of Okareo, Inc. and are not covered by the Apache 2.0 license. See TRADEMARK.md for permitted use.

Reviews

No reviews yet

Be the first to review this server!