What it does. Query and download public cancer imaging data from NCI Imaging Data Commons. Invoke for any question about IDC collections, cancer imaging datasets, DICOM data access, radiology (CT, MR, PET) or pathology AI training sets, metadata queries, visualization, or license checks — even when the user doesn't explicitly mention "IDC". No authentication required. Part of K-Dense-AI/scientific-agent-skills (AI Scientist skills) (K-Dense-AI/scientific-agent-skills).

Upstream	K-Dense-AI/scientific-agent-skills
Skill file	skills/imaging-data-commons/SKILL.md
License	MIT
Author	K-Dense Inc.
Fetched	2026-09-10

Install

• npx skills add K-Dense-AI/scientific-agent-skills --skill imaging-data-commons, or copy the skill folder into ~/.claude/skills/imaging-data-commons/.
• Raw file: curl -sL https://raw.githubusercontent.com/K-Dense-AI/scientific-agent-skills/HEAD/skills/imaging-data-commons/SKILL.md

SKILL.md (verbatim)

name: imaging-data-commons
description: Query and download public cancer imaging data from NCI Imaging Data Commons. Invoke for any question about IDC collections, cancer imaging datasets, DICOM data access, radiology (CT, MR, PET) or pathology AI training sets, metadata queries, visualization, or license checks — even when the user doesn't explicitly mention "IDC". No authentication required.
license: This skill is provided under the MIT License. IDC data itself has individual licensing (mostly CC-BY, some CC-NC) that must be respected when using the data.
metadata:
  version: "1.5"
  source-skill-version: 1.8.1
  skill-author: Andrey Fedorov, @fedorov
  idc-index: "0.12.5"
  idc-data-version: "v24"
  repository: https://github.com/ImagingDataCommons/imaging-data-commons-skill

Imaging Data Commons

Overview

Query and download public cancer imaging data from the National Cancer Institute Imaging Data Commons (IDC). No authentication required for data access.

Expected network access: IDC metadata is reachable three ways — a local DuckDB index shipped with the idc-index Python package (no network), or the hosted IDC service over MCP or REST (api.imaging.datacommons.cancer.gov, no authentication). File downloads use public GCS (storage.googleapis.com) and AWS S3 (s3.amazonaws.com) — no authentication required. DICOMweb access uses either the public IDC proxy (proxy.imaging.datacommons.cancer.gov, no auth) or the Google Cloud Healthcare API (healthcare.googleapis.com, requires GCP authentication). Optional BigQuery queries (bigquery.googleapis.com) also require GCP authentication. No credentials or environment variables are accessed by this skill.

Current IDC Data Version: v24 (always verify — see Best Practices)

Choose the access path first. There is no single default: the cheapest correct path depends on the session and the task.

1. Session already has the IDC MCP server? Route discovery and metadata there — see IDC MCP Server.
2. Otherwise, is idc-index installed? Run python scripts/check_version.py. If it passes, use idc-index for everything.
3. Not installed, and the task is read-only metadata — counts, attribute values, collection lookups, SQL under 10 000 rows, licenses, citations, viewer URLs? Use the REST API over curl; do not install anything. Installing costs ~77 MB of packaged index data plus pandas, pyarrow, and duckdb, which a metadata question does not need. See Data Access Options.
4. Not installed, and the task needs more than metadata — downloading files, pandas or plotting, pydicom/SimpleITK, pathology tiling, results past 10 000 rows, or a version-pinned script the user re-runs? Install idc-index: check_version.py exits non-zero and prints the exact install command for the running interpreter. Prefer a virtual environment, then restart Python.

idc-index (GitHub) is still the most capable path and the only one that moves image bytes; the rule is just not to pay for it before the task calls for it. check_version.py never installs anything itself — it also flags a newer idc-index or skill release when one exists.

Setup for the idc-index path:

from idc_index import IDCClient
client = IDCClient()

# Verify IDC data version (should be "v24")
print(f"IDC data version: {client.get_idc_version()}")

Core workflow: query metadata with client.sql_query() → download with client.download_from_selection() → visualize with client.get_viewer_URL(). Python examples below assume this client; Data Access Options has the REST equivalents. For current data scale, run the summary query in references/sql_patterns.md or GET /v3/stats.

IDC MCP Server

IDC operates a hosted MCP server at https://api.imaging.datacommons.cancer.gov/mcp (streamable HTTP, no authentication). Where it is available it complements — it does not replace — the idc-index workflow below.

Identify it by the MCP resource idc://guide, or by three or more of the tool names build_cohort, get_cohort_urls, list_analysis_results, and get_idc_version. Generic names such as run_sql are not evidence on their own. If identification is ambiguous, use idc-index.

If this session has the server, treat it as authoritative for discovery and metadata — IDC version, counts, attribute values, cohort building, metadata SQL — and follow the server's own instructions rather than re-deriving them from this file. Its data version is whatever the server reports: call get_idc_version instead of relying on the version pinned in this file.

Return here for what the server does not do: downloading files, local pandas/notebook analysis, DICOMweb, BigQuery, digital pathology tiling, and reproducible scripts. Hand off by passing SeriesInstanceUIDs from the server to client.download_from_selection(...), and run scripts/check_version.py at that point.

If it is not available, the identical service is reachable with no configuration as a REST API at https://api.imaging.datacommons.cancer.gov/v3 — use it for read-only metadata rather than installing idc-index, per the routing gate in Overview. Suggest connecting the MCP server at most once, only for repeated interactive discovery, and never change the user's configuration yourself.

See references/mcp_guide.md for the tool inventory, handoff patterns, and per-host notes.

When to Use This Skill

• Finding publicly available radiology (CT, MR, PET) or pathology (slide microscopy) images
• Selecting image subsets by cancer type, modality, anatomical site, or other metadata
• Downloading DICOM data from IDC
• Checking data licenses before use in research or commercial applications
• Visualizing medical images in a browser without local DICOM viewer software

Quick Navigation

Inline below: the MCP/REST routing rules, the IDC data model, the index tables and how they join, the core API patterns (query, download, visualize, license, cite), best practices, and troubleshooting.

Reference Guides (load on demand):

Guide	When to Load
index_tables_guide.md	Complex JOINs, schema discovery, DataFrame access
use_cases.md	End-to-end workflows: training datasets, batch downloads, DICOM reading with pydicom/SimpleITK, pipeline integration
sql_patterns.md	Quick SQL patterns for filter discovery, annotations, size estimation
clinical_data_guide.md	Clinical/tabular data, imaging+clinical joins, value mapping
licensing_and_citation.md	Commercial-use questions, mixed-license cohorts, citation formats
cloud_storage_guide.md	Direct S3/GCS access, versioning, UUID mapping
dicomweb_guide.md	DICOMweb endpoints, PACS integration
digital_pathology_guide.md	Slide microscopy (SM), annotations (ANN), pathology workflows
bigquery_guide.md	Full DICOM metadata, private elements (requires GCP)
cli_guide.md	Command-line tools (idc download, manifest files)
parquet_access_guide.md	Direct Parquet queries via GCS (no idc-index install needed)
mcp_guide.md	Hosted IDC MCP server: tool inventory, identification, handoff to idc-index
rest_api_guide.md	Hosted IDC REST API: endpoints, filter syntax, SQL over HTTP, manifests

IDC Data Model

IDC adds two grouping levels above the standard DICOM hierarchy (Patient → Study → Series → Instance):

• collection_id: Groups patients by disease, modality, or research focus (e.g., tcga_luad, nlst). A patient belongs to exactly one collection.
• analysis_result_id: Identifies derived objects (segmentations, annotations, radiomics features) across one or more original collections. Use it to find AI-generated or expert annotations, while collection_id finds original imaging data (which may itself include deposited annotations).

Key identifiers for queries:

Identifier	Scope	Use for
collection_id	Dataset grouping	Filtering by project/study
PatientID	Patient	Grouping images by patient
StudyInstanceUID	DICOM study	Grouping of related series, visualization
SeriesInstanceUID	DICOM series	Grouping of related series, visualization

Index Tables

The idc-index package provides multiple metadata index tables, accessible via SQL or as pandas DataFrames. The REST API exposes the same tables through GET /tables and POST /sql.

Important: client.indices_overview is the authoritative source for current table descriptions, available columns, and their types — query it when writing SQL or exploring data structure. It also answers "which table contains column X"; see references/index_tables_guide.md for that search pattern and full schema discovery.

Available Tables

Always call client.fetch_index("table_name") before querying any index table — it is safe and idempotent for all tables, including those loaded automatically at startup.

Family	Tables	Granularity
Core	index (primary metadata for all current data), collections_index, analysis_results_index	series / collection / analysis result
Modality acquisition parameters	ct_index, mr_index, pt_index, contrast_index	1 row = 1 series of that modality
Derived objects	seg_index, rtstruct_index, ann_index, ann_group_index	1 row = 1 series (or annotation group)
Microscopy	sm_index, sm_instance_index	1 row = 1 SM series / instance
Geometry, clinical, history	volume_geometry_index, clinical_index, version_metadata_index, prior_versions_index	see guide

references/index_tables_guide.md has the full inventory with each table's columns and contents — load it when you need to know what a specialized table actually holds.

prior_versions_index is for reproducibility only. It contains series permanently removed from IDC, with zero overlap with index. Use it only to reproduce work against a prior IDC version. Do NOT use it for version history or "what's new" questions — those use series_init_idc_version / series_revised_idc_version in the main index table, which are not equivalent to this table's min_idc_version / max_idc_version.

Joining Tables

SeriesInstanceUID is the universal join key for all series-level specialized tables: sm_index, sm_instance_index, seg_index, ann_index, ann_group_index, contrast_index, volume_geometry_index, rtstruct_index, ct_index, mr_index, pt_index. Always join these to index on SeriesInstanceUID. The exceptions below use different column names.

Join Column	Tables	Use Case
collection_id	index, prior_versions_index, collections_index, clinical_index	Link series to collection metadata or clinical data
analysis_result_id	index, analysis_results_index	Link series to analysis result metadata (annotations, segmentations)
source_DOI	index, analysis_results_index	Link by publication DOI
segmented_SeriesInstanceUID	seg_index → index	Link segmentation to its source image series (seg_index.segmented_SeriesInstanceUID = index.SeriesInstanceUID)
referenced_SeriesInstanceUID	ann_index → index, rtstruct_index → index	Link annotation or RTSTRUCT to its source image series

Note: subjects, updated, and description appear in multiple tables but have different meanings (counts vs identifiers, different update contexts). Joining prior_versions_index to index on SeriesInstanceUID always returns zero rows — see the warning above.

For detailed join examples, schema discovery patterns, key columns reference, and DataFrame access, see references/index_tables_guide.md.

Clinical Data Access

Clinical (non-imaging) attributes — staging, demographics, therapy — live in per-collection tables. client.fetch_index("clinical_index") loads the dictionary mapping columns to collections; client.get_clinical_table(name) returns one table as a DataFrame.

See references/clinical_data_guide.md for the discovery workflow, coded-value mapping, and joining clinical data with imaging.

Data Access Options

Method	Auth	Best For	Reference
idc-index	No	Downloads, pandas analysis, unbounded queries — the most capable path	This document
IDC MCP server	No	Discovery, cohort building, metadata when the session already has it	mcp_guide.md
IDC REST API	No	Metadata with no install, from any language or shell — the default when idc-index is absent	rest_api_guide.md
Direct Parquet (GCS)	No	Version-pinned queries, or results past the REST row cap	parquet_access_guide.md
Cloud storage (S3/GCS)	No	Direct file access, bulk transfer, custom pipelines	cloud_storage_guide.md
DICOMweb via IDC proxy	No	Tool and PACS integration; daily quota, so testing and moderate use	dicomweb_guide.md
DICOMweb via Google Healthcare	Yes (GCP)	The same DICOMweb API at production volume, without the proxy quota	dicomweb_guide.md
SlicerIDCBrowser	No	3D visualization and analysis in 3D Slicer	https://github.com/ImagingDataCommons/SlicerIDCBrowser
BigQuery	Yes (GCP)	Full DICOM metadata, private elements, SR measurements — last resort	bigquery_guide.md

The IDC Portal (https://portal.imaging.datacommons.cancer.gov/) is interactive only — browser-based exploration, manual cohort selection, and download. Unlike every option above it has no programmatic interface, so point a user there to browse or click through data themselves; never use it as a step in a script or workflow.

REST API — the no-install metadata path

https://api.imaging.datacommons.cancer.gov/v3, no authentication: discovery, cohort counts and manifests, read-only SQL, clinical tables, viewer URLs, licenses, citations. It is the same service as the MCP server over plain HTTP, so it needs no configuration. It never moves image bytes — switch to idc-index to download, to get a DataFrame, or for results past 10 000 rows.

B=https://api.imaging.datacommons.cancer.gov/v3
curl -s $B/version   # idc_version, idc_index_data_version, api_version
curl -s $B/stats     # collections, patients, studies, series, instances, size_TB
curl -s "$B/attributes/Modality/values?limit=5"   # real filter values, with counts
curl -s $B/sql -H 'content-type: application/json' \
  -d '{"sql":"SELECT collection_id, COUNT(*) n FROM index GROUP BY 1 ORDER BY n DESC LIMIT 3"}'
curl -s $B/cohort/counts -H 'content-type: application/json' \
  -d '{"filters":{"terms":{"collection_id":["rider_pilot"]}}}'

The filter object always goes under filters — on cohort/counts, cohort/manifest, cohort/manifest.txt, licenses, and citations alike. A bare filter or an unrecognized key is a 422 naming the fix; an unfiltered series-enumerating request is a 400, not the whole archive. Every filtered response echoes filters_applied and warnings — read them, because they name any predicate the server dropped. A zero count with empty warnings therefore means the filter matched nothing, not that a value was miscased; miscasing produces a warning that says so.

POST /sql takes one read-only SELECT/WITH over the tables idc-index exposes plus clinical.<table>; max_rows defaults to 5 000, caps at 10 000, and truncated flags clipping. GET /attributes lists the 19 filterable attributes — clinical values, segmented anatomy, and acquisition parameters are not among them and need SQL. There is no rate limit or quota. Use v3 only: V1 and V2 are superseded and scheduled for shutdown, so port any /v1/- or Modality_btw-style example a user brings rather than extending it.

Both sides build on idc-index-data, so compare the API's idc_index_data_version against local idc_index_data.__version__ before mixing them: the major is the IDC data release (24.x.y serves v24), so differing minor/patch means the series are identical. If the API is a whole release ahead, idc-index cannot download the extra series — it silently skips what its own index does not list — so either upgrade it (run scripts/check_version.py for the right command) or transfer directly from the bucket with s5cmd --no-sign-request.

See references/rest_api_guide.md for the endpoint reference, filter grounding, limits, and the manifest-based download flow.

Cloud storage organization

All DICOM files live in public buckets mirrored between AWS S3 and GCS, organized by CRDC UUIDs (not DICOM UIDs) to support versioning, as <crdc_series_uuid>/<crdc_instance_uuid>.dcm. Access is free (no egress fees) via AWS CLI, gsutil, or s5cmd with anonymous access; use the series_aws_url column for S3 URLs. Note that idc-open-data-cr / idc-open-cr (~4% of data) is commercial-use restricted (CC BY-NC). See references/cloud_storage_guide.md for the full bucket list and UUID mapping.

DICOMweb access

IDC data is available via DICOMweb (Google Cloud Healthcare API) for PACS integration and DICOMweb-compatible tools: a public proxy (no auth, daily quota) for testing and moderate queries, or Google Healthcare (GCP auth) for production volumes. See references/dicomweb_guide.md.

Direct Parquet access

The idc-index metadata tables are also published as Parquet on a public GCS bucket (idc-index-data-artifacts), queryable with DuckDB or pandas. This needs DuckDB installed and cannot reach the per-collection clinical tables, so prefer REST /sql for ad-hoc metadata; choose Parquet to pin a data version or for results past the REST row cap. See references/parquet_access_guide.md.

Core Capabilities

The patterns below are the ones that go wrong when recalled from memory rather than checked. Worked examples for each area live in the reference guides named inline.

1. Discovery — enumerate values before filtering on them

Filtering on a guessed Modality or BodyPartExamined string is the most common cause of an empty result set. Enumerate first:

modalities = client.sql_query("""
    SELECT DISTINCT Modality, COUNT(*) as series_count
    FROM index
    GROUP BY Modality
    ORDER BY series_count DESC
""")
print(modalities)

The same pattern works for any filter column, optionally narrowed by another — BodyPartExamined within a Modality, Manufacturer, collection_id. On the REST path this grounding is a single call — GET /attributes/{attr}/values returns values with counts — and the cohort endpoints report a miscased value in warnings rather than as an empty result.

Two indices carry curated collection-level metadata the primary index does not, both requiring client.fetch_index(...) first: collections_index (cancer types, tumor locations, species, subject counts) and analysis_results_index (derived datasets — AI segmentations, expert annotations, radiomics — with their source collections and modalities).

Cancer type lives in collections_index.cancer_types, not in index — filtering by cancer type requires a join:

client.fetch_index("collections_index")
results = client.sql_query("""
    SELECT i.collection_id, i.PatientID, i.SeriesInstanceUID, i.Modality
    FROM index i
    JOIN collections_index c ON i.collection_id = c.collection_id
    WHERE c.cancer_types LIKE '%Breast%'
      AND i.Modality = 'MR'
    LIMIT 20
""")

client.sql_query() returns a pandas DataFrame. Confirm column names with client.get_index_schema('index') or client.indices_overview before writing a query rather than assuming them.

See references/sql_patterns.md for filter-value discovery, annotation and segmentation queries, size estimation, clinical linking, and version tracking ("what's new in vX" — use series_init_idc_version / series_revised_idc_version in index, never prior_versions_index).

2. Downloading DICOM files

The two download methods take their first two arguments in opposite order. This is the most common source of broken IDC code — check it rather than recalling it:

Method	First arg	Second arg	Use when
download_from_selection	downloadDir (required)	filter kwargs (optional)	Filtering by collection, patient, study, or series
download_dicom_series	seriesInstanceUID (required)	downloadDir (required)	Downloading specific series by UID only

download_from_selection takes filter keyword arguments, NOT a DataFrame. The name "from_selection" refers to filtering the IDC index by criteria — not to accepting a pandas DataFrame. To download query results, extract the UIDs into a list first:

# Step 1: Query for series UIDs
series_df = client.sql_query("""
    SELECT SeriesInstanceUID
    FROM index
    WHERE Modality = 'CT'
      AND BodyPartExamined = 'CHEST'
      AND collection_id = 'nlst'
    LIMIT 5
""")

# Step 2: Extract UIDs as a list from the DataFrame
uids = list(series_df['SeriesInstanceUID'].values)

# Step 3: Pass the list to download_from_selection (NOT the DataFrame itself)
client.download_from_selection(
    downloadDir="./data/lung_ct",
    seriesInstanceUID=uids       # list of strings, not a DataFrame
)

# Alternative: download_dicom_series has seriesInstanceUID as FIRST arg (different order!)
client.download_dicom_series(
    seriesInstanceUID=uids,      # FIRST arg here
    downloadDir="./data/lung_ct"
)

# Whole collection: downloadDir is still the FIRST positional argument
client.download_from_selection(downloadDir="./data/rider", collection_id="rider_pilot")

Both methods default to AWS; pass source_bucket_location="gcs" to pull from Google Storage.

Downloaded files are named <crdc_instance_uuid>.dcm, not by SOPInstanceUID. The DICOM UIDs are preserved inside the file metadata, not in the filename. Use the crdc_instance_uuid column to map files back to the series they came from.

idc download <collection|series-uid|manifest> --download-dir ./data does the same from a shell. See references/cli_guide.md for the dirTemplate hierarchy options (Python default: %collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID; dirTemplate="" flattens), manifest downloads with resume, and dry-run size estimation.

3. Visualizing IDC images

viewer_url = client.get_viewer_URL(seriesInstanceUID=uid)        # one series
viewer_url = client.get_viewer_URL(studyInstanceUID=study_uid)   # all series in a study

Returns a browser URL — nothing is downloaded. The method selects OHIF v3 for radiology or SLIM for slide microscopy automatically. Viewing by study is useful when a single DICOM Study holds several Series (T1, T2, and DWI from one MRI session).

4. Licenses and citations — obligations, not optional steps

IDC data carries license terms and attribution requirements that follow it into any downstream publication or product, and neither is inferable from the pixel data. Check the license before use, and generate citations for whatever you download.

# License breakdown for a selection
licenses = client.sql_query("""
    SELECT DISTINCT collection_id, license_short_name,
           COUNT(DISTINCT SeriesInstanceUID) as series_count
    FROM index GROUP BY collection_id, license_short_name
""")

# Citations for the same selection you downloaded (APA by default)
for citation in client.citations_from_selection(collection_id="rider_pilot"):
    print(citation)

About 97% of IDC data is CC BY (commercial use allowed with attribution) and about 3% is CC BY-NC (non-commercial only). Licenses attach to series, not collections — 39 of 176 collections carry more than one — so check the selection you actually intend to use, and note that the most restrictive term governs a mixed cohort.

Both tasks are available from all three access paths, so stay on whichever one the session is already using: idc-index as above, POST /v3/licenses and POST /v3/citations over REST, or the get_licenses and get_citations MCP tools. See references/licensing_and_citation.md for the full license inventory, all three routes, the citation formats (APA, BibTeX, CSL JSON, RDF Turtle), and what to include when publishing.

5. Reaching past the index

Pick the access path with the routing gate in Overview; Data Access Options above is the full routing table.

Before reaching for BigQuery (which needs a billing-enabled GCP account), check whether a specialized index table already has the column you want: search client.indices_overview, then client.fetch_index(...) and query locally for free. BigQuery is required only for private DICOM elements, per-segment anatomy (segmentations), and pre-extracted SR measurements (quantitative_measurements, qualitative_measurements) — these have no idc-index equivalent.

Best Practices

• Check schema before writing queries — Use client.get_index_schema('index') (reads cached metadata, no SQL executed) or client.indices_overview to see all available columns and their descriptions. The version-tracking columns series_init_idc_version and series_revised_idc_version in the main index table directly answer "what's new / when was this added" questions without touching prior_versions_index.
• Never use web search for IDC data content questions - Always query the IDC index directly, via client.sql_query() locally or POST /v3/sql over HTTP. Web sources (release notes, blog posts, documentation pages) are frequently out of date and will produce incorrect answers. The index is the authoritative source; use it even when web search is available.
• Verify the IDC data version at the start of a session - client.get_idc_version(), GET /v3/version, or the MCP get_idc_version tool, depending on the path in use (currently v24). For a stale local index, run scripts/check_version.py and use the upgrade command it prints
• Check licenses and generate citations - Query license_short_name and respect CC BY vs CC BY-NC terms; use citations_from_selection() to produce citations from source_DOI for publications
• Explore small, then commit - Use LIMIT (or a low max_rows) while exploring, and check collection size before downloading — some collections are terabytes. See references/cli_guide.md
• Keep downloads reproducible - Organize with dirTemplate (e.g. %collection_id/%PatientID/%Modality) and save the Series UIDs or manifest behind any dataset you build

Troubleshooting

Issue: ModuleNotFoundError: No module named 'idc_index'

• Cause: idc-index package not installed
• Solution: If the task is read-only metadata, do not install it — use the REST API instead (Data Access Options). Otherwise run scripts/check_version.py and use the install command it prints, which targets the running interpreter and pins the vetted version. For data analysis also add pandas, numpy, and pydicom (tested with pandas>=1.5, numpy>=1.23, pydicom>=2.3)

Issue: Download fails with connection timeout

• Cause: Network instability or large download size
• Solution: Download in smaller batches (10-20 series); see references/cli_guide.md for --use-s5cmd-sync resume and retry guidance

Issue: BigQuery quota exceeded or billing errors

• Cause: BigQuery requires billing-enabled GCP project
• Solution: Use idc-index mini-index for simple queries (no billing required), or see references/bigquery_guide.md for cost optimization tips

Issue: Series UID not found or no data returned

• Cause: Typo in UID, data not in the current IDC version, or wrong field name
• Solution: Test with LIMIT 5 first, check field names against client.indices_overview, and confirm the series is in the current version (some old data is deprecated)

Issue: Column not found in index table (e.g., SliceThickness, PixelSpacing, KVP, EchoTime, InjectedDose)

• Cause: The index table contains series-level metadata only; modality-specific acquisition and reconstruction parameters live in dedicated tables (ct_index, mr_index, pt_index)
• Solution: Search client.indices_overview for the column to find its table — the loop is under Finding which table contains a column in references/index_tables_guide.md — then fetch and join on SeriesInstanceUID:

  client.fetch_index("ct_index")
  result = client.sql_query("""
      SELECT i.SeriesInstanceUID, i.Modality, c.SliceThickness, c.KVP, c.PixelSpacing_row_mm
      FROM index i
      JOIN ct_index c USING (SeriesInstanceUID)
      WHERE i.collection_id = 'your_collection'
  """)
  

Issue: Downloaded DICOM files won't open

• Cause: Corrupted download, or an object type the viewer does not handle — SEG, RTSTRUCT, SR, and slide microscopy all need specialized tools
• Solution: Check Modality and SOPClassUID first, validate with pydicom.dcmread(file, force=True), try another viewer (3D Slicer, QuPath for pathology), then re-download

Resources

Reference guides and their decision triggers are listed in Quick Navigation above.

• IDC Portal: https://portal.imaging.datacommons.cancer.gov/explore/
• Documentation: https://learn.canceridc.dev/ — Tutorials: https://github.com/ImagingDataCommons/IDC-Tutorials
• User Forum: https://discourse.canceridc.dev/ — idc-index: https://github.com/ImagingDataCommons/idc-index
• indices_reference — external index-table docs (may be ahead of the installed version)
• Citation: Fedorov, A., et al. "National Cancer Institute Imaging Data Commons: Toward Transparency, Reproducibility, and Scalability in Imaging Artificial Intelligence." RadioGraphics 43.12 (2023). https://doi.org/10.1148/rg.230180
• Skill updates: releases page; watch the repository (Watch → Custom → Releases)

Other files in this skill

• references/bigquery_guide.md
• references/cli_guide.md
• references/clinical_data_guide.md
• references/cloud_storage_guide.md
• references/dicomweb_guide.md
• references/digital_pathology_guide.md
• references/index_tables_guide.md
• references/licensing_and_citation.md
• references/mcp_guide.md
• references/parquet_access_guide.md
• references/rest_api_guide.md
• references/sql_patterns.md
• references/use_cases.md
• scripts/check_version.py

references/cli_guide.md (verbatim)

idc-index Command Line Interface Guide

The idc-index package provides command-line tools for downloading DICOM data from the NCI Imaging Data Commons without writing Python code.

Installation

Needs idc-index installed — run python scripts/check_version.py, which reports the installed version and prints the install command for the interpreter you are running.

After installation, the idc command is available in your terminal.

Available Commands

Command	Purpose
idc download	General-purpose download with auto-detection of input type
idc download-from-manifest	Download from manifest file with validation and progress tracking
idc download-from-selection	Filter-based download with multiple criteria

---

idc download

General-purpose download command that intelligently interprets input. It determines whether the input corresponds to a manifest file path or a list of identifiers (collection_id, PatientID, StudyInstanceUID, SeriesInstanceUID, crdc_series_uuid).

Usage

# Download entire collection
idc download rider_pilot --download-dir ./data

# Download specific series by UID
idc download "1.3.6.1.4.1.9328.50.1.69736" --download-dir ./data

# Download multiple items (comma-separated)
idc download "tcga_luad,tcga_lusc" --download-dir ./data

# Download from manifest file (auto-detected by file extension)
idc download manifest.txt --download-dir ./data

Options

Option	Description
--download-dir	Destination directory (default: current directory)
--dir-template	Directory hierarchy template (default: %collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID)
--log-level	Verbosity: debug, info, warning, error, critical

Directory Template Variables

The same templates apply in Python, where the argument is dirTemplate= rather than the --dir-template flag. The default is %collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID:

# Simplified hierarchy (omit StudyInstanceUID level)
client.download_from_selection(
    downloadDir="./data",
    collection_id="tcga_luad",
    dirTemplate="%collection_id/%PatientID/%Modality"
)
# Results in: ./data/tcga_luad/TCGA-05-4244/CT/

# dirTemplate="" disables the hierarchy, writing every file straight into downloadDir

Use these variables in --dir-template to organize downloads:

• %collection_id - Collection identifier
• %PatientID - Patient identifier
• %StudyInstanceUID - Study UID
• %SeriesInstanceUID - Series UID
• %Modality - Imaging modality (CT, MR, PT, etc.)

Examples:

# Flat structure (all files in one directory)
idc download rider_pilot --download-dir ./data --dir-template ""

# Simplified hierarchy
idc download rider_pilot --download-dir ./data --dir-template "%collection_id/%PatientID/%Modality"

---

idc download-from-manifest

Specialized for downloading from manifest files with built-in validation, progress tracking, and resume capability.

Usage

# Basic download from manifest
idc download-from-manifest --manifest-file cohort.txt --download-dir ./data

# With progress bar and validation
idc download-from-manifest --manifest-file cohort.txt --download-dir ./data --show-progress-bar

# Resume interrupted download with s5cmd sync
idc download-from-manifest --manifest-file cohort.txt --download-dir ./data --use-s5cmd-sync

Options

Option	Description
--manifest-file	Required. Path to manifest file containing S3 URLs
--download-dir	Required. Destination directory
--validate-manifest	Validate manifest before download (enabled by default)
--show-progress-bar	Display download progress
--use-s5cmd-sync	Enable resumable downloads - skips already-downloaded files
--quiet	Suppress subprocess output
--dir-template	Directory hierarchy template
--log-level	Logging verbosity

Manifest File Format

Manifest files contain S3 URLs, one per line:

s3://idc-open-data/cb09464a-c5cc-4428-9339-d7fa87cfe837/*
s3://idc-open-data/88f3990d-bdef-49cd-9b2b-4787767240f2/*

How to get a manifest file:

1. IDC Portal: Export cohort selection as manifest
2. Python query: Generate from SQL results

from idc_index import IDCClient

client = IDCClient()
results = client.sql_query("""
    SELECT series_aws_url
    FROM index
    WHERE collection_id = 'rider_pilot' AND Modality = 'CT'
""")

with open('ct_manifest.txt', 'w') as f:
    for url in results['series_aws_url']:
        f.write(url + '\n')

---

idc download-from-selection

Download data using filter criteria. Filters are applied sequentially.

Usage

# Download by collection
idc download-from-selection --collection-id rider_pilot --download-dir ./data

# Download specific series
idc download-from-selection --series-instance-uid "1.3.6.1.4.1.9328.50.1.69736" --download-dir ./data

# Multiple filters
idc download-from-selection --collection-id nlst --patient-id "100004" --download-dir ./data

# Dry run - see what would be downloaded without actually downloading
idc download-from-selection --collection-id tcga_luad --dry-run --download-dir ./data

Options

Option	Description
--download-dir	Required. Destination directory
--collection-id	Filter by collection identifier
--patient-id	Filter by patient identifier
--study-instance-uid	Filter by study UID
--series-instance-uid	Filter by series UID
--crdc-series-uuid	Filter by CRDC UUID
--dry-run	Calculate cohort size without downloading
--show-progress-bar	Display download progress
--use-s5cmd-sync	Enable resumable downloads
--dir-template	Directory hierarchy template

Dry Run for Size Estimation

Use --dry-run to estimate download size before committing:

idc download-from-selection --collection-id nlst --dry-run --download-dir ./data

This shows:

• Number of series matching filters
• Total download size
• No files are downloaded

---

Common Workflows

1. Download Small Collection for Testing

# rider_pilot is ~1GB - good for testing
idc download rider_pilot --download-dir ./test_data

2. Large Dataset with Progress and Resume

# Use s5cmd sync for large downloads - can resume if interrupted
idc download-from-selection \
    --collection-id nlst \
    --download-dir ./nlst_data \
    --show-progress-bar \
    --use-s5cmd-sync

3. Estimate Size Before Download

# Check size first
idc download-from-selection --collection-id tcga_luad --dry-run --download-dir ./data

# Then download if size is acceptable
idc download-from-selection --collection-id tcga_luad --download-dir ./data

4. Download Specific Modality via Python + CLI

# First, query for series UIDs in Python
from idc_index import IDCClient

client = IDCClient()
results = client.sql_query("""
    SELECT SeriesInstanceUID
    FROM index
    WHERE collection_id = 'nlst'
      AND Modality = 'CT'
      AND BodyPartExamined = 'CHEST'
    LIMIT 50
""")

# Save to manifest
results['SeriesInstanceUID'].to_csv('my_series.csv', index=False, header=False)

# Then download via CLI
idc download my_series.csv --download-dir ./lung_ct

---

Built-in Safety Features

The CLI includes several safety features:

• Disk space checking: Verifies sufficient space before starting downloads
• Manifest validation: Validates manifest file format by default
• Progress tracking: Optional progress bar for monitoring large downloads
• Resume capability: Use --use-s5cmd-sync to continue interrupted downloads

---

Troubleshooting

Download Interrupted

Use --use-s5cmd-sync to resume:

idc download-from-manifest --manifest-file cohort.txt --download-dir ./data --use-s5cmd-sync

Connection Timeout

For unstable networks, download in smaller batches using Python to generate multiple manifests, then download sequentially.

---

See Also

• idc-index Documentation
• IDC Portal - Interactive cohort building
• IDC Tutorials

references/clinical_data_guide.md (verbatim)

Clinical Data Guide for IDC

Tested with: idc-index 0.12.5 (IDC data version v24)

Clinical data (demographics, diagnoses, therapies, lab tests, staging) accompanies many IDC imaging collections. This guide covers how to discover, access, and integrate clinical data with imaging data using idc-index.

When to Use This Guide

Use this guide when you need to:

• Find what clinical metadata is available for a collection
• Filter patients by clinical criteria (e.g., cancer stage, treatment history)
• Join clinical attributes with imaging data for cohort selection
• Understand and decode coded values in clinical tables

For basic clinical data access, see the "Clinical Data Access" section in the main SKILL.md. This guide provides detailed workflows and advanced patterns.

Prerequisites

Needs idc-index installed — run python scripts/check_version.py, which reports the installed version and prints the install command for the interpreter you are running.

No BigQuery credentials required - clinical data is packaged with idc-index.

Understanding Clinical Data in IDC

What is Clinical Data?

Clinical data refers to non-imaging information that accompanies medical images:

• Patient demographics (age, sex, race)
• Clinical history (diagnoses, surgeries, therapies)
• Lab tests and pathology results
• Cancer staging (clinical and pathological)
• Treatment outcomes

Data Organization

Clinical data in IDC comes from collection-specific spreadsheets provided by data submitters. IDC parses these into queryable tables accessible via idc-index.

Important characteristics:

• Clinical data is not harmonized across collections (terms and formats vary)
• Not all collections have clinical data (check availability first)
• All data is anonymized - dicom_patient_id links to imaging

The clinical_index Table

The clinical_index serves as a dictionary/catalog of all available clinical data:

Column	Purpose	Use For
collection_id	Collection identifier	Filtering by collection
table_name	Full BigQuery table reference	BigQuery queries (if needed)
short_table_name	Short name	get_clinical_table() method
column	Column name in table	Selecting data columns
column_label	Human-readable description	Searching for concepts
values	Observed attribute values for the column	Interpreting coded values

The values Column

The values column contains an array of observed attribute values for the column defined in the column field. Each entry has:

• option_code: The actual value observed in that column
• option_description: Human-readable description of that value (from data dictionary if available, otherwise None)

For ACRIN collections, value descriptions come from provided data dictionaries. For other collections, they are derived from inspection of the actual data values.

Note: For columns with >20 unique values, the values array is left empty ([]) for simplicity.

Core Workflow

Step 1: Fetch Clinical Index

from idc_index import IDCClient

client = IDCClient()
client.fetch_index('clinical_index')

# View available columns
print(client.clinical_index.columns.tolist())

Step 2: Discover Available Clinical Data

# List all collections with clinical data
collections_with_clinical = client.clinical_index["collection_id"].unique().tolist()
print(f"{len(collections_with_clinical)} collections have clinical data")

# Find clinical attributes for a specific collection
nlst_columns = client.clinical_index[client.clinical_index['collection_id']=='nlst']
nlst_columns[['short_table_name', 'column', 'column_label', 'values']]

Step 3: Search for Specific Attributes

# Search by keyword in column_label (case-insensitive)
stage_attrs = client.clinical_index[
    client.clinical_index["column_label"].str.contains("[Ss]tage", na=False)
]
stage_attrs[["collection_id", "short_table_name", "column", "column_label"]]

Step 4: Load Clinical Table

# Load table using short_table_name
nlst_canc_df = client.get_clinical_table("nlst_canc")

# Examine structure
print(f"Rows: {len(nlst_canc_df)}, Columns: {len(nlst_canc_df.columns)}")
nlst_canc_df.head()

Step 5: Map Coded Values to Descriptions

Many clinical attributes use coded values. The values column in clinical_index contains an array of observed values with their descriptions (when available).

# Get the clinical_index rows for NLST
nlst_clinical_columns = client.clinical_index[client.clinical_index['collection_id']=='nlst']

# Get observed values for a specific column
# Filter to the row for 'clinical_stag' and extract the values array
clinical_stag_values = nlst_clinical_columns[
    nlst_clinical_columns['column']=='clinical_stag'
]['values'].values[0]

# View the observed values and their descriptions
print(clinical_stag_values)
# Output: array([{'option_code': '.M', 'option_description': 'Missing'},
#                {'option_code': '110', 'option_description': 'Stage IA'},
#                {'option_code': '120', 'option_description': 'Stage IB'}, ...])

# Create mapping dictionary from codes to descriptions
mapping_dict = {item['option_code']: item['option_description'] for item in clinical_stag_values}

# Apply to DataFrame - convert column to string first for consistent matching
nlst_canc_df['clinical_stag_meaning'] = nlst_canc_df['clinical_stag'].astype(str).map(mapping_dict)

Step 6: Join with Imaging Data

The dicom_patient_id column links clinical data to imaging. It matches the PatientID column in the imaging index.

# Pandas merge approach
import pandas as pd

# Get NLST CT imaging data
nlst_imaging = client.index[(client.index['collection_id']=='nlst') & (client.index['Modality']=='CT')]

# Join with clinical data
merged = pd.merge(
    nlst_imaging[['PatientID', 'StudyInstanceUID']].drop_duplicates(),
    nlst_canc_df[['dicom_patient_id', 'clinical_stag', 'clinical_stag_meaning']],
    left_on='PatientID',
    right_on='dicom_patient_id',
    how='inner'
)

# SQL join approach
# Clinical tables loaded via get_clinical_table() are not automatically
# registered in DuckDB. Register the DataFrame manually before joining.
nlst_canc_df = client.get_clinical_table("nlst_canc")
client._duckdb_conn.register("nlst_canc", nlst_canc_df)

query = """
SELECT
  index.PatientID,
  index.StudyInstanceUID,
  index.Modality,
  nlst_canc.clinical_stag
FROM index
JOIN nlst_canc ON index.PatientID = nlst_canc.dicom_patient_id
WHERE index.collection_id = 'nlst' AND index.Modality = 'CT'
"""
results = client.sql_query(query)

Common Use Cases

Use Case 1: Select Patients by Cancer Stage

from idc_index import IDCClient
import pandas as pd

client = IDCClient()
client.fetch_index('clinical_index')

# Load clinical table
nlst_canc = client.get_clinical_table("nlst_canc")

# Select Stage IV patients (code '400')
stage_iv_patients = nlst_canc[nlst_canc['clinical_stag'] == '400']['dicom_patient_id']

# Get CT imaging studies for these patients
stage_iv_studies = pd.merge(
    client.index[(client.index['collection_id']=='nlst') & (client.index['Modality']=='CT')],
    stage_iv_patients,
    left_on='PatientID',
    right_on='dicom_patient_id',
    how='inner'
)['StudyInstanceUID'].drop_duplicates()

print(f"Found {len(stage_iv_studies)} CT studies for Stage IV patients")

Use Case 2: Find Collections with Specific Clinical Attributes

# Find collections with chemotherapy information
chemo_collections = client.clinical_index[
    client.clinical_index["column_label"].str.contains("[Cc]hemotherapy", na=False)
]["collection_id"].unique()

print(f"Collections with chemotherapy data: {list(chemo_collections)}")

Use Case 3: Examine Observed Values for a Clinical Attribute

# Find what values have been observed for a specific attribute
chemotherapy_rows = client.clinical_index[
    (client.clinical_index["collection_id"] == "hcc_tace_seg") &
    (client.clinical_index["column"] == "chemotherapy")
]

# Get the observed values array
values_list = chemotherapy_rows["values"].tolist()
print(values_list)
# Output: [[{'option_code': 'Cisplastin', 'option_description': None},
#           {'option_code': 'Cisplatin, Mitomycin-C', 'option_description': None}, ...]]

Use Case 4: Generate Viewer URLs for Selected Patients

import random

# Get studies for a sample Stage IV patient
sample_patient = stage_iv_patients.iloc[0]
studies = client.index[client.index['PatientID'] == sample_patient]['StudyInstanceUID'].unique()

# Generate viewer URL
if len(studies) > 0:
    viewer_url = client.get_viewer_URL(studyInstanceUID=studies[0])
    print(viewer_url)

Key Concepts

column vs column_label

• column: Use for selecting data from tables (programmatic access)
• column_label: Use for searching/understanding what data means (human-readable)

Some collections (like c4kc_kits) have identical column and column_label. Others (like ACRIN collections) have cryptic column names but descriptive labels.

option_code vs option_description

The values array contains observed attribute values:

• option_code: The actual value observed in the column (what you filter on)
• option_description: Human-readable description (from data dictionary if available, otherwise None)

dicom_patient_id

Every clinical table includes dicom_patient_id, which matches the PatientID column in the imaging index. This is the key for joining clinical and imaging data.

Troubleshooting

Issue: Clinical table not found

Cause: Using wrong table name or table doesn't exist for collection

Solution: Query clinical_index first to find available tables:

client.clinical_index[client.clinical_index['collection_id']=='your_collection']['short_table_name'].unique()

Issue: Empty values array

Cause: The values array is left empty when a column has >20 unique values

Solution: Load the clinical table and examine unique values directly:

clinical_df = client.get_clinical_table("table_name")
clinical_df['column_name'].unique()

Issue: Coded values not in mapping

Cause: Some values may be missing from the dictionary (e.g., empty strings, special codes like .M for missing)

Solution: Handle unmapped values gracefully:

df['meaning'] = df['code'].astype(str).map(mapping_dict).fillna('Unknown/Missing')

Issue: No matching patients when joining

Cause: Clinical data may include patients without images, or vice versa

Solution: Verify patient overlap before joining:

imaging_patients = set(client.index[client.index['collection_id']=='nlst']['PatientID'].unique())
clinical_patients = set(clinical_df['dicom_patient_id'].unique())
overlap = imaging_patients & clinical_patients
print(f"Patients with both imaging and clinical data: {len(overlap)}")

Resources

IDC Documentation:

• Clinical data organization - How clinical data is organized in IDC
• Clinical data dashboard - Visual summary of available clinical data
• idc-index clinical_index documentation

Related Guides:

• bigquery_guide.md - Advanced clinical queries via BigQuery
• Main SKILL.md - Core IDC workflows

IDC Tutorials:

• clinical_data_intro.ipynb
• exploring_clinical_data.ipynb
• nlst_clinical_data.ipynb

Back to K-Dense-AI/scientific-agent-skills (AI Scientist skills) or Agent skills.