CSDA Data Search Agent

01 — Problem & constraints

Who it was for Researchers who know what they need to see, but not which vendor collection holds it.

• ConstraintOnly public catalogue metadata. Downloads need the user’s own Earthdata login, which is kept in the chat session and never stored.
• ConstraintThe upstream `csda-client` library has no search method, so search has to talk to the STAC API directly.
• ConstraintOne `docker compose up` for the whole stack.

02 — My role

I built everything in the public repository, and I am its only contributor: the agent and its tools, the catalogue graph, the STAC client, the chat UI and map panel, and the Docker setup. It is a personal project, separate from my team’s work on NASA IMPACT, and it uses only public catalogue metadata.

03 — Architecture

flowchart TB
  U["Chat UI (Chainlit on FastAPI)"] --> G["LangGraph graph"]
  G --> A["Pydantic AI agent"]
  A -->|find_datasets| N["Neo4j catalogue graph with vector index"]
  A -->|set_search_filters, run_data_search, paginate_results| ES["Per-thread explore state"]
  ES -->|CQL2-JSON search| STAC["CSDA STAC API"]
  ES <-->|WebSocket| M["Map and results panel"]
  A -->|download_asset| C["csda-client with the user's Earthdata login"]
  A --> PG["Postgres and pgvector: users, threads, agent history"]
  A --> L["Ollama: chat model and embeddings"]

04 — Key decisions

1. D1

   Resolve vague requests through a catalogue graph

   Users name things ("optical imagery", a vendor’s product line), but the search needs an exact collection slug.

   Chose

   Ingest the vendor catalogue into Neo4j as a graph (vendors, product types, products, filters) with a vector index, and give the agent a `find_datasets` tool that returns candidate slugs with scores.

   Trade-off

   A second datastore and an ingest step to keep current. Filtering by kind happens after the vector lookup (it over-fetches four times the limit), which can still return too few results of one kind.

   Alternatives considered (2)

   • Let the model pick a slug from the collection list in its prompt — Slugs look alike and change; a guessed slug fails silently with zero results.
   • Keyword search over collection titles — Misses requests that describe the data rather than name it.
2. D2

   One explore state for the agent and the map panel

   Search results are spatial and long; a chat reply can’t show them well, and users want to tweak filters by hand.

   Chose

   The agent’s tools don’t return results as text. They change a per-thread explore state (filters, results, paging), which is broadcast to the panel over a WebSocket. The panel sends its own filter and search actions back to the same state.

   Trade-off

   State lives in the app process, so it’s per instance. The WebSocket protocol is one more thing to keep compatible.

   Alternatives considered (2)

   • Return results only in the chat reply — No map, no thumbnails, and the user can’t adjust a filter without another prompt.
   • A separate search UI with its own state — The agent and the UI would drift apart, and the agent couldn’t explain what the user sees.
3. D3

   Search the STAC API directly; download through csda-client

   `csda-client` handles Earthdata authentication but has no search method.

   Chose

   A small STAC client builds CQL2-JSON searches with token paging. Downloads go through `csda-client`, using credentials the user enters in the chat session, which are never written to disk or the database.

   Trade-off

   Users re-enter credentials after resuming a chat before they can download.

   Alternatives considered (1)

   • Store Earthdata credentials in the app database — It would hold other people’s credentials for a convenience feature.

05 — Outcomes & evidence

1. Measured · 23 Sep 2026

   1 request → find_datasets → set_search_filters → run_data_search → 20 results

   A plain-English request is turned into a real catalogue search end to end: the agent chose a collection, set filters and returned 20 low-cloud scenes.

   One local run on 23 Sep 2026 with `docker compose up` and the public CSDA STAC API. Request: "Find optical satellite imagery over the Willamette Valley, Oregon from August 2024 with low cloud cover". Model gpt-oss:120b-cloud through Ollama. No downloads (no Earthdata login). A single run, not an evaluation.

2. Measured · 23 Sep 2026

   The catalogue graph covers the whole CSDA vendor catalogue at ingest time: 20 vendors, 7 product types, 84 products and 239 product filters.

   Counts returned by the repository’s `neo4j_store.ingest_catalog()` against the public CSDA vendors API on 23 Sep 2026.

06 — Limitations & next steps

• —No evaluation set: there is no measured accuracy for dataset resolution or for the filters the agent sets. The outcomes above are one run.
• —Tests cover configuration and password hashing only; the agent, tools and STAC client have no automated tests.
• —The three performance changes in the repository’s ADR (async STAC client, no exact counts, vector pre-filtering in Neo4j) are proposed, not implemented.
• —The map needs a Mapbox token, and the chat model runs on a hosted open-weight model through Ollama.

Next steps

• —Build a small labelled set of requests with their expected collections and filters, and track accuracy.
• —Implement the ADR’s async STAC client and Neo4j pre-filtering, and measure latency before and after.
• —Add tests for the tools and the explore-state protocol.

07 — Related writing

• Building a Scientific Data Discovery Platform with a Team of AI AgentsMoon, Mars and Earth data in one traceable model: what I built, the problems I hit, and how I ran several coding agents without losing track of the work.