Canyougrab Api MCP Server

CanYouGrab API

Domain availability lookup API with subscription billing, built on FastAPI + DNS (Unbound resolver).

Live services:

• API: https://api.canyougrab.it
• Developer portal: https://portal.canyougrab.it
• Auth: https://auth.canyougrab.it (Auth0 custom domain)

Architecture Overview

┌──────────────────────────────────────────────────────────────────┐
│                        Developer Portal                          │
│              (Zudoku/React on portal.canyougrab.it)              │
│   Usage Dashboard · API Keys · Pricing · API Reference (OAS)    │
└──────────────────┬───────────────────────────────────────────────┘
                   │  Auth0 JWT (portal)  /  Bearer API key (API)
                   ▼
┌──────────────────────────────────────────────────────────────────┐
│                    FastAPI Backend (v5.0.0)                       │
│                    api.canyougrab.it:8000                         │
│                                                                  │
│  ┌─────────────┐  ┌──────────┐  ┌──────────┐  ┌─────────────┐  │
│  │ app.py      │  │ keys.py  │  │billing.py│  │ auth.py     │  │
│  │ /check/bulk │  │ /keys    │  │/billing  │  │ API key +   │  │
│  │ /usage      │  │ CRUD     │  │/stripe   │  │ JWT auth    │  │
│  │             │  │ rotate   │  │ webhook  │  │             │  │
│  └──────┬──────┘  └──────────┘  └────┬─────┘  └─────────────┘  │
│         │                            │                           │
└─────────┼────────────────────────────┼───────────────────────────┘
          │                            │
          ▼                            ▼
┌─────────────────┐          ┌─────────────────┐
│  Valkey (Redis)  │          │   Stripe API     │
│  Job queue +     │          │   Subscriptions   │
│  Rate limiting   │          │   Webhooks        │
└────────┬────────┘          └─────────────────┘
         │
         ▼
┌─────────────────┐          ┌──────────────────────────────┐
│  RQ Worker       │──DNS──▶ │   Unbound Resolver            │
│  (worker.py)     │          │   (dedicated droplet)         │
│  ThreadPool(10)  │          │   NS queries via VPC          │
│  via RQ queue    │          └──────────────────────────────┘
└─────────────────┘
                             ┌──────────────────────────────┐
         ───────────────────▶│   PostgreSQL                  │
          (auth, usage,      │   API keys + Usage logs       │
           billing only)     └──────────────────────────────┘

Directory Structure

zuplo/
├── backend/                    # Python FastAPI backend
│   ├── app.py                  # Main API: /check/bulk, /usage, /health
│   ├── auth.py                 # API key auth (SHA-256) + Auth0 JWT auth (RS256)
│   ├── billing.py              # Stripe checkout, portal, webhooks, card-on-file, usage details
│   ├── keys.py                 # API key CRUD: create, list, rotate, revoke (+ Turnstile)
│   ├── antifraud.py            # Anti-fraud: Turnstile, device fingerprints, risk scoring
│   ├── email_utils.py          # Email normalization + disposable email detection
│   ├── dns_client.py           # DNS-based domain availability checking via Unbound
│   ├── queries.py              # PostgreSQL queries: usage tracking, auth, billing
│   ├── valkey_client.py        # Redis/Valkey job queue client (RQ-backed)
│   ├── rq_tasks.py             # RQ task function: process_domain_job()
│   ├── worker.py               # RQ worker process (ThreadPoolExecutor per job)
│   ├── migrations/             # SQL migrations
│   │   └── 001_free_tier_antifraud.sql
│   └── requirements.txt        # Python dependencies
├── portal/                     # Developer portal (Zuplo + Zudoku)
│   ├── config/
│   │   ├── routes.oas.json     # OpenAPI 3.1 spec (public API documentation)
│   │   └── policies.json       # Zuplo policies (empty — all routing is direct)
│   ├── docs/                   # Zudoku documentation portal
│   │   ├── src/
│   │   │   ├── config.ts       # API_BASE, Turnstile site key, Stripe PK
│   │   │   ├── UsageDashboard.tsx   # Usage + billing dashboard component
│   │   │   ├── PricingPage.tsx      # Plan selection + Stripe checkout
│   │   │   ├── PricingPlans.tsx     # Pricing card grid component
│   │   │   └── CardSetupPage.tsx    # Stripe Elements card-on-file for Free+
│   │   ├── public/             # Static assets (logos, banners, CSS overrides)
│   │   ├── zudoku.config.tsx   # Portal config: theme, nav, Auth0, API key mgmt
│   │   └── package.json        # Frontend dependencies (React 19, Zudoku)
│   ├── package.json            # Workspace root (Zuplo v6, TypeScript v5)
│   └── README.md               # Zuplo boilerplate (not project-specific)
├── mcp-server/                 # MCP package for ChatGPT, Claude, and remote MCP clients
│   ├── pyproject.toml          # MCP package metadata + version
│   ├── server.json             # MCP registry/server metadata
│   ├── uv.lock                 # Locked MCP runtime dependencies
│   └── src/canyougrab_mcp/
│       ├── __init__.py
│       └── server.py           # stdio + streamable-http MCP entrypoint
├── .github/workflows/
│   ├── deploy.yml              # Production deploy (on tag push v*)
│   └── deploy-dev.yml          # Dev deploy (on push to dev branch)
├── .claude/
│   └── launch.json             # Local dev server config (port 9200)
└── package.json                # Root workspace config

API Endpoints

Public API (API key auth: Authorization: Bearer cyg_...)

Portal API (Auth0 JWT auth)

Internal / Webhook

Core Request Flow

Domain Availability Check

Client                    FastAPI                     Valkey                      Worker              Unbound
  │                          │                          │                          │                    │
  │  POST /api/check/bulk    │                          │                          │                    │
  │  { domains: [...] }      │                          │                          │                    │
  │─────────────────────────▶│                          │                          │                    │
  │                          │── validate key ──────────│                          │                    │
  │                          │── check minute rate ────▶│ INCR ratelimit:id:min   │                    │
  │                          │── check monthly quota ──▶│ (PostgreSQL)             │                    │
  │                          │── record usage ─────────▶│ (PostgreSQL)             │                    │
  │                          │── create_job() ─────────▶│ HSET job:{uuid}          │                    │
  │                          │                          │ RQ enqueue (with retry)   │                    │
  │                          │                          │                          │                    │
  │                          │                          │◀── RQ Worker dequeues ───│                    │
  │                          │                          │                          │                    │
  │                          │                          │──── claim_job() ────────▶│                    │
  │                          │                          │                          │── ThreadPool(10)   │
  │                          │                          │                          │── check_domain_dns()
  │                          │                          │                          │── NS query ───────▶│
  │                          │                          │                          │◀── NOERROR/NXDOMAIN│
  │                          │                          │◀── complete_job() ───────│                    │
  │                          │                          │                          │                    │
  │                          │◀─ poll get_job_status() ─│                          │                    │
  │                          │   (0.3s interval, 30s    │                          │                    │
  │                          │    max timeout)          │                          │                    │
  │◀─────────────────────────│                          │                          │                    │
  │  { results: [...] }      │                          │                          │                    │

Billing / Subscription Flow

User → Pricing Page → Select Plan → Auth0 login
  → POST /api/billing/checkout (JWT auth)
  → Find/create Stripe customer (linked by auth0_sub metadata)
  → Stripe Checkout Session created → redirect to Stripe
  → User pays → Stripe fires webhook
  → POST /api/stripe/webhook (HMAC-SHA256 verified)
  → checkout.session.completed → fetch subscription → get price ID
  → Map price to plan → UPDATE api_keys SET plan, lookups_limit

Subscription Plans

Authentication

API Key Auth (public API consumers)

• Format: Authorization: Bearer cyg_<token>
• Keys are SHA-256 hashed in the api_keys table (plaintext never stored)
• Key prefix (cyg_XXXXXXXX...) is stored for display in the portal
• Full key returned only once at creation time
• Soft-delete on revocation (revoked_at timestamp, not physically deleted)

JWT Auth (portal/dashboard)

• Auth0 tenant: dev-mqe5tavp6dr62e7u
• Custom domain: auth.canyougrab.it
• Algorithm: RS256 with JWKS validation (1-hour cache)
• Audience: https://api.canyougrab.it
• Social logins: Google, Apple (Sign in with Apple)

Domain Availability via DNS

Domain availability is checked by querying a dedicated Unbound recursive DNS resolver running on a separate DigitalOcean droplet. The worker sends NS record queries over the VPC private network and interprets the response:

The Unbound resolver caches aggressively (7 days for registered domains, 5 minutes for NXDOMAIN) and queries TLD authoritative servers directly, avoiding public resolver rate limits.

Database Schema

PostgreSQL is used for authentication, usage tracking, and billing — not for domain lookups.

PostgreSQL Tables

Valkey (Redis) Data Structures

Infrastructure

Servers

External Services

Deployment Pipelines

Production Deploy

Trigger: Push a git tag matching v* (e.g., v1.0.5)

git tag v1.0.5 && git push origin v1.0.5

Pipeline (.github/workflows/deploy.yml):

1. GitHub Actions triggers on v* tag push
2. SSH into production server (DEPLOY_HOST secret) using DEPLOY_SSH_KEY
3. Bootstraps the target ref on the server (git fetch + git checkout <tag>)
4. Runs the repo-managed deploy script: /opt/canyougrab-repo/scripts/deploy-host.sh

Dev Deploy

Trigger: Push to the dev branch

git push origin dev

Pipeline (.github/workflows/deploy-dev.yml):

1. GitHub Actions triggers on push to dev
2. SSH into dev server (DEV_DEPLOY_HOST secret) using same DEPLOY_SSH_KEY
3. Bootstraps the dev ref on the server (git fetch + git checkout dev)
4. Runs the repo-managed deploy script: /opt/canyougrab-repo/scripts/deploy-host.sh

Repo-Managed Host Deploy Script

Both pipelines use a small inline SSH bootstrap to update /opt/canyougrab-repo to the target ref, then call scripts/deploy-host.sh from that checked-out revision. The repo-managed deploy script handles:

• pip install -r requirements.txt for backend dependencies
• rsync of backend, portal, and MCP source trees into their runtime directories
• Reinstall or refresh the mcp-server package/runtime when the host also runs canyougrab-mcp.service
• Restart of FastAPI (uvicorn), worker, and MCP services via systemd

If /mcp is served by the same host as the API, backend-only deploys are not enough. The MCP service must be updated and restarted during the same deploy or the OAuth metadata and live MCP behavior can drift apart.

An existing /opt/deploy.sh can still be kept as a manual compatibility shim if desired, but the automated pipeline should treat the repo copy of scripts/deploy-host.sh as the source of truth.

Branching Strategy

Environment Variables

Backend (required on servers)

# DNS Resolver (Unbound on dedicated droplet)
DNS_RESOLVER_HOSTNAME=unbound.canyougrab.internal  # VPC internal hostname (resolved via socket.gethostbyname)
DNS_RESOLVER_PORT=53
DNS_QUERY_TIMEOUT=5.0  # Per-query timeout in seconds

# PostgreSQL (DigitalOcean Managed Database — auth, usage, billing only)
POSTGRES_HOST=         # DB cluster hostname
POSTGRES_PORT=5432
POSTGRES_DB=canyougrab
POSTGRES_USER=canyougrab
POSTGRES_PASSWORD=     # DB password
POSTGRES_SSLMODE=require

# Valkey / Redis (DigitalOcean Managed Database)
VALKEY_HOST=           # Valkey cluster hostname
VALKEY_PORT=25061
VALKEY_USERNAME=default
VALKEY_PASSWORD=       # Valkey password

# Auth0
AUTH0_DOMAIN=dev-mqe5tavp6dr62e7u.us.auth0.com
AUTH0_AUDIENCE=https://api.canyougrab.it

# Stripe
STRIPE_SECRET_KEY=     # sk_live_... (prod) or sk_test_... (dev)
STRIPE_WEBHOOK_SECRET= # whsec_... (per-environment)
STRIPE_PRICE_BASIC=    # price_... (live price ID for Basic plan)
STRIPE_PRICE_PRO=      # price_... (live price ID for Pro plan)
STRIPE_PRICE_BUSINESS= # price_... (live price ID for Business plan)

# Cloudflare Turnstile (bot prevention on key creation)
TURNSTILE_SECRET_KEY=  # 0x4AAAA... (from Cloudflare dashboard)

# Portal
PORTAL_URL=https://portal.canyougrab.it

# Worker
BATCH_CONCURRENCY=10   # Thread pool size for domain checks
VALKEY_QUEUE_NAME=canyougrab-jobs  # RQ queue name (default: canyougrab-jobs)

# Monitoring (optional — only if monitoring stack is installed)
SLACK_ALERTS_WEBHOOK_URL=  # Slack incoming webhook for Alertmanager
RQ_METRICS_PORT=9122       # Prometheus metrics exporter port

# Auto-scaler (optional — only on API host with autoscaler enabled)
DO_API_TOKEN=              # DigitalOcean API token
DO_WORKER_SNAPSHOT_ID=     # Snapshot ID for new worker droplets
AUTOSCALER_MIN_WORKERS=1
AUTOSCALER_MAX_WORKERS=5
AUTOSCALER_SCALE_UP_THRESHOLD=50
AUTOSCALER_SCALE_DOWN_IDLE_MINUTES=10

GitHub Actions Secrets

Local Development

Backend

cd backend
pip install -r requirements.txt

# Start API server
uvicorn app:app --reload --port 8000

# Start worker (separate terminal)
python worker.py

Requires DNS_RESOLVER_HOST pointing to an Unbound instance (or use 8.8.8.8 for basic testing), plus PostgreSQL and Valkey/Redis (local or tunneled to managed instances).

Portal

cd portal/docs
npm install
npm run dev          # Starts Zudoku dev server (via zuplo)

Or use the configured launch server:

# From repo root
npx zuplo dev        # Starts on port 9200

Portal dev server hardcodes API_BASE to https://api.canyougrab.it in portal/docs/src/config.ts. To develop against a local backend, change this to http://localhost:8000.

SEO: .it TLD Geo-Targeting Requirements

Because canyougrab.it uses a .it country-code TLD (Italy), all user-facing HTML (portal, docs) must include signals that the content targets English speakers:

1. <html lang="en"> on every page.
2. Hreflang tags in <head>:

   <link rel="alternate" hreflang="en-US" href="{page_url}" />
   <link rel="alternate" hreflang="en" href="{page_url}" />
   <link rel="alternate" hreflang="x-default" href="{page_url}" />
   

3. Canonical tag — Self-referencing <link rel="canonical" href="{page_url}" />.
4. All content in English — No Italian text on any page.

This applies to the portal (portal.canyougrab.it), API docs, and any other publicly rendered HTML.

Key Design Decisions

• Live DNS lookups: Domain availability is checked via NS queries to a dedicated Unbound recursive resolver. This gives real-time results (no 24-hour zone file lag), works for any TLD automatically, and avoids the operational complexity of daily batch zone file loading. Unbound caches aggressively (7 days for registered domains) so repeated queries are ~1ms.
• Nullable availability: DNS has more failure modes than a database lookup. When Unbound returns SERVFAIL or times out, the API returns available: null instead of a potentially dangerous false positive. Consumers should treat null as "could not determine."
• Job queue for bulk checks: The bulk endpoint uses RQ (Redis Queue) backed by Valkey to dispatch jobs with automatic retries (2 retries, 5s/30s backoff) and failed-job tracking. Workers parallelize DNS queries across a thread pool.
• Long-polling: The bulk check endpoint holds the HTTP connection open (polling Valkey every 0.3s, up to 45s) rather than requiring clients to implement polling. Job results are stored in custom Valkey hashes (not RQ's built-in result storage) so the poll logic is independent of the queue framework.
• API keys with SHA-256 hashing: Keys are hashed before storage (like passwords), so a database breach doesn't expose raw keys.
• Stripe metadata linking: Stripe customers are linked to Auth0 users via auth0_sub metadata on the Stripe customer object, avoiding a separate mapping table.
• Usage double-tracking: Both daily (usage_log_daily) and per-minute (usage_log_minute) usage are recorded for monthly quota and per-minute rate limiting respectively.
• No Zuplo gateway policies: The project originally used Zuplo as an API gateway but migrated to direct FastAPI (commit 3abef70). The Zuplo portal/docs framework is still used for the developer portal.

Processes on Servers

Core services (every host)

1. FastAPI (uvicorn): Serves all HTTP endpoints on port 8000
2. RQ Worker (worker.py): Processes domain check jobs from the RQ queue
3. MCP server (if /mcp is hosted on this server): Serves remote MCP clients

Monitoring stack (API host, optional)

1. Prometheus (localhost:9090): Scrapes metrics, evaluates alert rules
2. Alertmanager (localhost:9093): Routes alerts to Slack
3. redis_exporter (localhost:9121): Exports Valkey metrics to Prometheus
4. RQ metrics exporter (localhost:9122): Exports queue depth, worker count, failed jobs
5. Grafana (localhost:3000, nginx-proxied): Dashboards for queue health
6. Auto-scaler (optional): Scales worker droplets up/down via DO API

Install the monitoring stack with: sudo bash scripts/setup-monitoring.sh

All active services on the host must be managed via systemd and restarted during deployments. All automated deploys should route through scripts/deploy-host.sh, which is responsible for restarting whichever of these services are installed on the host.