bigquery-data-science

Production Agent Skills, zero-dependency CLI, stdio MCP server & Google ADK tools for BigQuery ML, BigQuery AI, and BigQuery MLOps — distilled directly from 108 live-executed notebooks and 89 SQL scripts in Google Cloud.

108 live-verified notebooks · 89 SQL scripts · 4 Composable Agent Skills · 20 Reference & Playbook Guides · 139 <25 KB syntax cards · zero-dependency CLI, MCP & ADK

%%{init: {'flowchart': {'wrappingWidth': 380, 'nodeSpacing': 24, 'rankSpacing': 28}}}%%
flowchart TD
    subgraph GT["1. Ground-Truth Notebooks, SQL & Reference Hubs (GitHub)"]
        direction LR
        BQML["<b>bigquery-ml/</b> (62 Notebooks · 58 SQL)<br/>23 CREATE MODEL · 12 ML.* Families<br/>18 Applied Workflows · 9 MLOps Pipelines"]
        BQAI["<b>bigquery-ai/</b> (46 Notebooks · 31 SQL)<br/>31 AI.* & Cloud AI SQL Functions<br/>15 Multi-Function Workflows"]
        REFS["<b>13 Deep Reference Guides</b><br/>bigquery-ml/reference/ (5 Guides)<br/>bigquery-ai/reference/ (8 Guides)"]
        BQML ~~~ BQAI ~~~ REFS
    end

    subgraph SKILLS["2. 4 Composable Agent Skills (skills/ · .agents/skills/ · .claude/skills/)"]
        direction TB
        ROUTER["<b>bigquery-data-science</b> (Umbrella Router)<br/>ML.* vs AI.* Matrix · 10 Head-to-Head Comparisons<br/>6 Multi-Skill Playbooks · 7-Stage Lifecycle Navigator"]
        S_ML["<b>bigquery-ml</b> (Trained Models)<br/>23 CREATE MODEL · 49 ML.* Functions<br/>Preprocessing · Causal · Explain"]
        S_AI["<b>bigquery-ai</b> (Foundation Models)<br/>31 AI.* & Cloud AI · TimesFM · TabFM<br/>Embeddings · Vector & Hybrid Search"]
        S_OPS["<b>bigquery-mlops</b> (Production Ops)<br/>9 Orchestrators · Vertex AI Registry<br/>EXPORT / IMPORT · Drift & Feature Store"]
        ROUTER --> S_ML
        ROUTER --> S_AI
        ROUTER --> S_OPS
    end

    subgraph LENS["3. 4-Lens Progressive Disclosure & Zero-Dependency Delivery"]
        direction LR
        L12["<b>Lens 1 & Lens 2 — Route & Compare</b><br/>4 SKILL.md Routers (~8 KB)<br/>20 Decision & Playbook Guides (~25 KB)"]
        L34["<b>Lens 3 & Lens 4 — Write & Build</b><br/>139 Surgical Syntax Cards (< 25 KB)<br/>108 Worked Walkthroughs (~20 KB)"]
        DELIV["<b>Zero-Dependency CLI, MCP & ADK</b><br/>uvx bigquery-data-science install · mcp<br/>triage · lookup · search · review-sql"]
        L12 --> L34 --> DELIV
    end

    GT --> SKILLS --> LENS

    style GT fill:#F8FAFF,stroke:#AECBFA,stroke-width:1.5px,color:#174EA6
    style SKILLS fill:#F8FAFF,stroke:#AECBFA,stroke-width:1.5px,color:#174EA6
    style LENS fill:#F8FAFF,stroke:#AECBFA,stroke-width:1.5px,color:#174EA6

    classDef data fill:#E8F0FE,stroke:#4285F4,stroke-width:2px,color:#174EA6
    classDef config fill:#FEF7E0,stroke:#F9AB00,stroke-width:2px,color:#7A5A00
    classDef route fill:#FCE8E6,stroke:#EA4335,stroke-width:2px,color:#A50E0E
    classDef compute fill:#E6F4EA,stroke:#34A853,stroke-width:2px,color:#0D652D
    class BQML,BQAI,S_ML,DELIV data
    class REFS,S_OPS,L12 config
    class ROUTER route
    class S_AI,L34 compute

This repository pairs three tightly connected layers that stay synchronized through automated tooling (bigquery-data-science sweep):

1. Ground-Truth Notebooks, SQL & Reference Hubs (bigquery-ml/ & bigquery-ai/): Every model type (23), ML.* function (12 families), AI.* / Cloud AI function (31), composed workflow (33), and MLOps orchestration pipeline (9) is validated end-to-end in a live-executed Jupyter notebook (.ipynb with committed outputs and one-click Colab / Vertex AI Workbench / BigQuery Studio headers) paired with a pure SQL companion script (89 .sql files) and 13 deep domain reference guides.
2. 4 Composable Agent Skills (skills/): An umbrella router and multi-skill composer (bigquery-data-science) routes across three specialist skills — trained models & causal inference (bigquery-ml), zero-training foundation models & vector/hybrid search (bigquery-ai), and production orchestration, model registry, hybrid serving, drift monitoring & point-in-time feature stores (bigquery-mlops) — and explicitly recommends multi-skill combinations via cross-skill-playbooks.md.
3. 4-Lens Progressive Disclosure, CLI, Stdio MCP & Google ADK (src/bigquery_data_science/): Instead of dumping 24 MB of raw notebooks or 200+ KB monolithic reference files into an AI coding agent's context window, bigquery-data-science compiles the repository into 4 progressive lenses (SKILL.md $\to$ 20 reference guides $\to$ 139 <25 KB syntax cards $\to$ 108 clean narratives) and exposes them through a zero-dependency CLI (uvx bigquery-data-science), stdio MCP server (bigquery-data-science mcp), and Google ADK toolset (get_bigquery_tools()).

The core bigquery-data-science package has zero third-party runtime dependencies (< 2.0 MB wheel) and installs the 4 skills (bigquery-data-science, bigquery-ml, bigquery-ai, bigquery-mlops) into .agents/skills/ and .claude/skills/:

# Option A — Zero-install ephemeral CLI via uvx (recommended for instant setup):
uvx bigquery-data-science install                     # installs into ./.agents/skills & ./.claude/skills
uvx bigquery-data-science install --mcp               # also registers stdio MCP server in ./.mcp.json
uvx bigquery-data-science install --global            # installs into ~/.agents/skills & ~/.claude/skills
uvx bigquery-data-science install --target agents     # only .agents/skills (Antigravity, Cursor, Codex, Windsurf)
uvx bigquery-data-science install --target claude     # only .claude/skills (Claude Code)

# Option B — Persistent CLI install via uv tool, pipx, or pip:
uv tool install bigquery-data-science && bigquery-data-science install
pipx install bigquery-data-science && bigquery-data-science install
pip install bigquery-data-science && bigquery-data-science install

# Option C — Add to an existing Python project (uv or pip):
uv add bigquery-data-science && uv run bigquery-data-science install
pip install bigquery-data-science && python3 -m bigquery_data_science.cli install

# Option D — skills.sh CLI (installs all 4 skills or an individual skill from GitHub):
npx skills add statmike/bigquery-data-science
npx skills add statmike/bigquery-data-science --skill bigquery-ml
npx skills add statmike/bigquery-data-science --skill bigquery-mlops

# Option E — Claude Code plugin marketplace & stdio MCP server:
# In Claude Code: /plugin marketplace add statmike/bigquery-data-science
#                 /plugin install bigquery-data-science
claude mcp add bigquery-data-science -- uvx bigquery-data-science mcp
# Or with a local pip/uv install:
claude mcp add bigquery-data-science -- python3 -m bigquery_data_science.cli mcp

Run any catalog command directly with uvx bigquery-data-science ... (zero install) or bigquery-data-science ... (after uv tool install or pip install):

# 1. List bundled skills and lens counts:
uvx bigquery-data-science list

# 2. Triage any BigQuery ML, BigQuery AI, or BigQuery MLOps task (10 comparisons + 6 multi-skill playbooks):
uvx bigquery-data-science triage "forecast daily retail sales with holiday promotions"
uvx bigquery-data-science triage "schedule retraining pipeline with drift gate and Vertex AI Model Registry"

# 3. Fetch the surgical syntax card (< 25 KB) for any model or function:
uvx bigquery-data-science lookup ARIMA_PLUS
uvx bigquery-data-science lookup AI.FORECAST
uvx bigquery-data-science lookup ML.ENTITY_FEATURES_AT_TIME

# 4. Search across all 139 syntax cards, 20 reference guides, and 108 narratives:
uvx bigquery-data-science search "point-in-time feature leakage"

# 5. Statically audit a BigQuery SQL query or .sql file for prerequisites & gotchas:
uvx bigquery-data-science review-sql "SELECT AI.GENERATE('Summarize', endpoint => 'gemini-3.8-flash')"

# 6. Add --json to any catalog command (list, lookup, search, triage, review-sql) for automation:
uvx bigquery-data-science lookup AI.AGG --json

Every notebook in bigquery-ml/ and bigquery-ai/ includes committed outputs for immediate reading on GitHub and one-click launch links for Colab, Colab Enterprise, Vertex AI Workbench, and BigQuery Studio.

To run notebooks locally, choose the dependency extra that matches the folders you want to execute:

git clone https://github.com/statmike/bigquery-data-science.git
cd bigquery-data-science

# With uv (recommended — uses the locked uv.lock environment):
uv sync --extra all
uv run bigquery-data-science kernel --user

# Or with standard Python venv + pip:
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[all]"
bigquery-data-science kernel --user

BigQuery exposes machine learning and generative AI across two SQL prefixes (ML.* and AI.*) that reflect when a function was introduced and whether it requires a CREATE MODEL DDL object. Full matrix: skills/bigquery-data-science/reference/namespace-matrix.md.

Whether you are reading the repository as a human practitioner or routing an AI coding agent, every workflow and pipeline maps to one of 7 problem-first stages (full playbook & SQL choreography: skills/bigquery-data-science/reference/cross-skill-playbooks.md):

Instead of scrolling through 108 individual rows here, jump directly into the domain hubs and category indexes below:

Contributions are welcome — see CONTRIBUTING.md, CODE_OF_CONDUCT.md, and SECURITY.md.

• GitHub Actions CI & Release Workflows: .github/workflows/ci.yml runs ruff, mypy, pytest, bigquery-data-science sweep, sync_headers.py --check, and ./scripts/check_install.sh across Python 3.11, 3.12, and 3.13 on every push and pull request. .github/workflows/release.yml publishes release candidates (v*rc*) to TestPyPI and production tags (v*) to PyPI via OIDC Trusted Publishing.
• Local Pre-Commit Hooks (.pre-commit-config.yaml): Install with uv run pre-commit install to run ruff, mypy, bigquery-data-science sweep, and sync_headers.py --check automatically before each commit.

Run the complete local verification and cold-install gate before submitting a pull request:

uv run ruff check src tests && \
  uv run mypy src && \
  uv run pytest -q && \
  uv run bigquery-data-science sweep && \
  python3 architectures/headers/sync_headers.py --check && \
  ./scripts/check_install.sh

bigquery-data-science adheres to Semantic Versioning (Major.Minor.Patch):

• Major (X.0.0) — Skills architecture, names & structural contract shifts: Changes in skills hierarchy/naming (skills/), packaging overhauls, or breaking changes to CLI subcommands, MCP tool schemas, or library exports.
• Minor (X.Y.0) — Multi-area monthly sweeps & capability rollouts: Scheduled monthly sweeps across BigQuery release notes — introducing new foundation models, options, notebooks, SQL workflows, and synchronized 4-lens skill updates.
• Patch (X.Y.Z) — Targeted single-area updates & fixes: Isolated bug fixes, typo corrections, single-notebook adjustments, or targeted hotfixes.

See CONTRIBUTING.md for the full release process and TestPyPI/PyPI tag flow.

Apache-2.0 — see LICENSE.

This is not an officially supported Google product. It is built on Google products and is neither endorsed nor supported by Google.