Server data from the Official MCP Registry
Version-controlled golden datasets and RAG evaluation, no API key needed.
Version-controlled golden datasets and RAG evaluation, no API key needed.
This is a well-structured MCP server for golden dataset management with sound security practices. All tools require explicit dataset_path parameters, preventing session-state vulnerabilities. Dependencies are minimal and legitimate, input validation uses Pydantic, and no credentials or external API calls are made. Minor code quality issues (broad exception handling, path traversal potential via user-supplied paths) are present but do not significantly impact security given the server's purpose as a local filesystem tool. Supply chain analysis found 7 known vulnerabilities in dependencies (1 critical, 3 high severity). Package verification found 1 issue.
7 files analyzed · 12 issues found
Security scores are indicators to help you make informed decisions, not guarantees. Always review permissions before connecting any MCP server.
This plugin requests these system permissions. Most are normal for its category.
Add this to your MCP configuration file:
{
"mcpServers": {
"io-github-nipunkhanderia-golden-dataset-mcp": {
"args": [
"golden-dataset-mcp"
],
"command": "uvx"
}
}
}From the project's GitHub README.
mcp-name: io.github.nipunkhanderia/golden-dataset-mcp
An MCP server wrapping golden-dataset-studio — version-controlled golden dataset management and semantic evaluation for RAG/LLM pipelines.
This is a thin protocol layer over the existing golden_dataset library (DatasetStore, Evaluator). It does not reimplement any logic — it exposes the library's existing Python API as MCP tools so an agent (Claude Desktop, Claude Code, or any MCP client) can manage golden datasets conversationally.
No LLM API key required. Evaluation uses TF-IDF cosine similarity (scikit-learn), not an LLM call.
golden-dataset-studio?golden-dataset-studio is a CLI tool — designed for a human typing golden add, golden commit, etc. in a terminal. golden-dataset-mcp exposes the same underlying operations as MCP tools so an LLM agent can drive them programmatically, e.g. as part of an automated RAG evaluation pipeline. Keeping them as separate PyPI packages means CLI users aren't forced to pull in fastmcp as a dependency, and MCP users get a clean, protocol-focused package.
| Tool | What it does |
|---|---|
init_dataset | Initialise a new dataset at a given path |
add_entry | Add a question/answer pair to the working tree |
update_entry | Edit fields of an existing working-tree entry |
delete_entry | Remove an entry from the working tree |
list_entries | List working-tree or committed-version entries |
commit_version | Snapshot the working tree as a new immutable version |
diff_versions | Show entries added/removed/changed between two versions |
evaluate_answers | Score actual answers against a version via TF-IDF cosine similarity |
dataset_status | Show current version, working tree size, and version history |
dataset_pathUnlike the CLI (which operates on the current working directory), every tool here requires an explicit dataset_path parameter. This keeps the server fully stateless between calls — no hidden "current dataset" session state to lose track of, and safe for one server instance to manage multiple datasets or serve multiple concurrent clients.
pip install golden-dataset-mcp
This pulls in golden-dataset-studio and scikit-learn automatically as dependencies.
{
"mcpServers": {
"golden-dataset": {
"command": "golden-dataset-mcp"
}
}
}
No environment variables needed — no API key, no config.
1. init_dataset(dataset_path="./my-rag-eval", name="support-bot-eval")
2. add_entry(dataset_path="./my-rag-eval", question="...", answer="...")
[repeat for each golden Q&A pair]
3. commit_version(dataset_path="./my-rag-eval", description="initial 50 questions")
4. [run your RAG pipeline, collect actual answers]
5. evaluate_answers(dataset_path="./my-rag-eval", actual_answers=[...])
-> avg_semantic_similarity, per-entry scores, pass/fail
As your RAG pipeline changes over time, commit_version again after edits and use diff_versions to see exactly what changed in your golden set between releases.
golden-dataset-studio | golden-dataset-mcp | |
|---|---|---|
| Interface | CLI (golden ...) | MCP tools |
| Driven by | A human typing commands | An LLM agent / MCP client |
| Path handling | Current working directory | Explicit dataset_path per call |
| Dependency direction | — | Depends on golden-dataset-studio |
If you want the human-driven CLI, use golden-dataset-studio directly. If you want an agent to drive it, use this package.
git clone https://github.com/nipunkhanderia/golden-dataset-mcp
cd golden-dataset-mcp
pip install -e ".[dev]"
pytest -v
Validate the MCP-facing contract:
npx @modelcontextprotocol/inspector golden-dataset-mcp
evaluate_answers uses TF-IDF cosine similarity, which captures lexical overlap better than deep semantic meaning. For embedding-based or RAGAS-style metrics, call the underlying library's Evaluator.ragas_evaluate() directly (requires pip install "golden-dataset-studio[ragas]" — not exposed as an MCP tool in this version).ValueError: empty vocabulary on inputs like a bare "4" or "the a an". Avoid single-token golden answers, or expect evaluate_answers to fail on them.<dataset_path>/.golden_dataset/; this server does no remote storage or syncing.MIT
Be the first to review this server!
by Modelcontextprotocol · Developer Tools
Web content fetching and conversion for efficient LLM usage
by Toleno · Developer Tools
Toleno Network MCP Server — Manage your Toleno mining account with Claude AI using natural language.
by mcp-marketplace · Developer Tools
Create, build, and publish Python MCP servers to PyPI — conversationally.