Building a Scientific Data Discovery Platform with a Team of AI Agents

1. The problem

Scientific data is heterogeneous in every way that matters:

"Temperature" means three different things in this list: regolith surface, buried probes down to −1.3 m, and air at 1.4–2 m. "Radiation" covers both solar irradiance (W/m²) and energetic particle counts. Those can't be averaged together.

The questions the platform had to answer were simple to ask:

"Compare near-surface temperatures on the Moon, Mars and Earth in kelvin."

The answer is only useful if every number in it can be traced back to the exact bytes it came from.

2. Principles I held myself to

I wrote these down before any code and put them in CLAUDE.md, so every agent read them first:

1. The raw record is never mutated or discarded. Normalization is never the only copy.
2. Every normalized value carries its transformation_history and a link to its raw record.
3. Standardize only what interoperability needs. Original field names, tokens and units stay alongside.
4. Nothing is dropped silently. Every problem is one of reject_record, reject_value, missing_value or warning, with file, locator, field, reason and the verbatim value.
5. One AI capability, with deterministic guardrails. An LLM never runs in the ingest path.
6. No infrastructure breadth for its own sake. No Kubernetes, no microservices, no overlapping databases.

Having these written down made reviews cheap. A reviewer, human or agent, could say "this violates principle 4" instead of arguing about taste.

3. Architecture: a modular monolith

flowchart TB
    subgraph SRC["Sources (inbox/)"]
        A["Apollo 17 HFE<br/>fixed-width"]
        B["InSight TWINS<br/>CSV"]
        C["NASA POWER<br/>GeoJSON"]
        D["LRO CRaTER<br/>headerless CSV + PDS4"]
    end
    subgraph DDP["ddp package"]
        CT["contracts<br/>DatasetConfig · Vocabulary · StructuredQuery"]
        IN["ingest<br/>adapters → validate → normalize → load"]
        ST["store<br/>only package that writes SQL"]
        CA["catalog<br/>dataset_measurement · STAC"]
        QU["query<br/>discover · observations · combine · trace"]
        AS["assistant<br/>NL → StructuredQuery + guardrails"]
        IF["interfaces<br/>Typer CLI · FastAPI · Chainlit"]
    end
    PG[("Postgres 16")]
    SRC --> IN
    IN --> ST
    ST --> PG
    CA --> ST
    QU --> ST
    AS --> QU
    IF --> AS
    IF --> QU
    IF --> CA

The important rules are the import boundaries: only store writes SQL, and assistant reaches data only through query. That second rule is what makes the AI safe. There is no code path where the model can issue a query the deterministic layer hasn't validated.

Declarative mapping configs instead of per-source code

Each dataset is one YAML file in config/datasets/: adapter, fields, units, sentinels, time parser, location strategy and vocabulary term. Four generic adapters read them (fixed_width_table, csv, pds4_headerless_csv, nasa_power_json).

The reference system I was drawing from, CSDA's Airflow pipeline, has a transform.py per vendor. That hides unit and time assumptions inside code. A YAML config can be reviewed in a diff, hashed into a config_version, validated before ingest, and cited by ddp trace for every value it produced.

Deterministic identifiers

file_id         = sha256(file bytes)
raw_record_id   = sha256(file_id | locator)        # locator like "line:57" or "hour:2024030105"
observation_id  = sha256(raw_record_id | source_field)

Re-running ingest gives the same IDs, so reruns are idempotent. A trace ID from my laptop resolves to the same row in the Docker container. The generated demo document also regenerates byte for byte, which made it a regression test for free.

A vocabulary with abstract terms

config/vocabulary.yaml is a small hierarchy. temperature groups surface_temperature, subsurface_temperature and air_temperature. combine keeps them as separate groups and never averages across them. radiation is marked abstract: discovery returns both irradiance and particle rate, and combine refuses it with a named guardrail code (abstract_term_combined).

The one AI capability

Pydantic AI with output_type=StructuredQuery, temperature 0 and one retry. The flow:

flowchart TB
    Q["Question in plain language"] --> M["LLM → StructuredQuery (typed JSON)"]
    M --> G["Deterministic guardrails<br/>unknown dataset? unit mismatch?<br/>abstract term combined? window outside extents?"]
    G -->|refused| R["Refusal with a named code"]
    G -->|valid| X["ddp.query runs it"]
    X --> T["Answer = template over result models<br/>+ the query shown + traced provenance"]

The model picks which query to run. It never writes the numbers in the answer. Every command except ask works with no model at all. No model is configured by default. The documented route is Ollama, so a reviewer can use a local model without an API key.

On an eval of held-out questions (3 trials each, majority pass) it scored 92% on validation and 93% on test with ollama:gpt-oss:120b-cloud, an Ollama-hosted model. A purely local model scores lower. Test questions were never used for tuning.

4. Problems I hit and how I solved them

This is the part I enjoyed most. Each of these came from the real data or the real deployment, not from a synthetic test.

4.1 A mission epoch that's off by a day

Apollo 17 HFE's TIME column counts milliseconds from an epoch. The PDS label text implies 1972-01-01. Checked against the archive's own row dates, that is off by a day. The correct epoch is 1971-12-31T00:00Z.

Fix: the epoch is declared in the dataset config rather than in code. The assumption is recorded in every value's transformation_history, and data/README.md documents it. The timestamps are also Earth-received, so every row carries time_kind = earth_received, and every combined result warns about it.

Lesson: metadata is data too, and it can be wrong. Keep the assumption visible so the next person can check it.

4.2 Coordinates that aren't on Earth

GeoJSON and STAC geometry is WGS84, which is Earth. Latitude 20° on the Moon is not latitude 20° on Earth. PostGIS can handle other reference systems, but setting up custom planetary CRSs was more than this prototype needed.

Fix: no PostGIS or pgSTAC. Lat/lon are plain columns read per body (target_body + location_frame), planetocentric and east-positive. No query spans bodies spatially. CRaTER's lat/lon/altitude are derived from its position vector with a lunar radius of 1737.4 km. In the STAC output, Moon and Mars Items have null GeoJSON geometry, and their native extents live in ddp:* properties.

Tradeoff (logged): no radius or polygon search, even within one body.

4.3 "Temperature" is three measurements, "radiation" is two

The first design for cross-dataset combine would have happily averaged lunar regolith with Martian air. The fix was in the vocabulary, not in code: leaf terms for the physically different quantities, and an abstract flag for words like radiation that are good for search but meaningless to average.

The cross-dataset demo expresses "near-surface" as vertical_m between −0.2 m and 2.5 m. That keeps the lunar surface thermocouple, the shallowest buried probe and both air temperatures. The deeper probes are excluded by the filter, not by the data.

4.4 Ingest replaced the whole split on every run (issue #17)

After the Docker setup landed, the ingest_run table showed the InSight split being fully replaced (30,382 rows inserted) on runs that should have been no-ops. Nothing was duplicated, because the IDs are deterministic, but rows were being destroyed and recreated.

The version triple gave it away:

16:59  cfg 021bcd97  vocab cf16c1a5  code ddp-0.1.0
16:51  cfg 021bcd97  vocab cf16c1a5  code 442f61981323   ← git sha
16:12  cfg 021bcd97  vocab cf16c1a5  code ddp-0.1.0

code_version came from git rev-parse HEAD, with ddp-0.1.0 as the fallback. The Docker image had no git, so container runs and host runs recorded different code versions, and any version difference triggered a replace.

Fix: only a config or vocabulary change replaces observations. A code-version change alone does not (decision log #4).

Lesson: the "why" was sitting in the provenance table. Recording versions per run paid for itself here.

4.5 Ingestion inside the app's start-up script (issue #18)

The first Docker version ran ingest in the app container's entrypoint. Every UI restart re-ran a load: 60 runs had built up from restarts alone. The input folder was fixed at build time, and run status existed only in the database.

Fix: ingestion became its own Compose service, orchestrated by Prefect (server + UI on :4200). The flow in ddp.ingest.flows is a thin wrapper: parameters in, run_ingest called, report out. The CLI, the tests and the service share one ingestion code path. I chose Prefect over Airflow because a typical Airflow deployment needs a scheduler, a webserver and a metadata database (plus a triggerer for deferrable tasks) and a DAG-file convention to run one function, while Prefect needs one server and one process running plain Python. This reversed my original "single process, no orchestrator" decision, and the decision log entry says so.

4.6 The chat UI lost provenance (issues #23, #24, #27; PR #31)

This took several rounds. First, the chat answer showed one dataset and told you to run a command to trace a row, instead of showing the trace. Then discovery reported a dataset count that didn't match its rows. Finally, combine samples turned out to come only from earth_nasa_power because of row ordering.

Fix, in stages: the chat now shows an automatically traced provenance table (one row per dataset/term/height). The transformation-history step is shown next to the result, and combine samples are balanced across groups.

Lesson: "traceable" isn't done when the API can trace. It's done when the person reading the answer can see the trace without extra work.

Workflow: managing a team of agents

Four agents worked on the project, each in its own git worktree. Claude, Codex and Gemini implemented tasks, often at the same time. Antigravity mostly verified, merged, coordinated and wrote handoffs. The hard part turned out to be keeping them coordinated, more than writing the code. I went through three setups.

flowchart TB
    subgraph P1["Phase 1 · Task files inside the code repo"]
        P1a["project-management/ in the repo<br/>README board · tasks/T01–T32 · journal.md"]
        P1b["Each worktree has its own copy of the board<br/>→ status diverges, merges conflict"]
        P1a --> P1b
    end
    subgraph P2["Phase 2 · Task files in a folder outside the repo"]
        P2a["~/Workspace/agent-project-management/<br/>one shared copy for every worktree"]
        P2b["Not connected to GitHub<br/>→ board says 'review', GitHub says 'merged'"]
        P2a --> P2b
    end
    subgraph P3["Phase 3 · GitHub issues as the source of truth"]
        P3a["Issue → assigned from Orca CLI → Claude builds the PR"]
        P3b["Codex reviews inline → Claude fixes"]
        P3c["Antigravity verifies: VERIFIED / NOT VERIFIED"]
        P3a --> P3b --> P3c
    end
    P1 --> P2 --> P3

Phase 1: local task files inside the repo

I started with plain files, because they don't depend on any particular tool: a project-management/ folder in the repo (commit eb0928c) holding

• README.md, a board of 32 tasks (T01–T32) grouped into waves by dependency;
• tasks/Txx-*.md, one file per task with "Files you own", acceptance criteria and a link to the spec section;
• PROTOCOL.md, the rules: claim before you edit, stay inside your files, worktree per task, second agent verifies;
• journal.md, an append-only log, one line per event: <UTC> <owner> claimed|wip|review|done|blocked|needs <task> — note.

The protocol itself worked. File ownership meant parallel agents rarely edited the same code, and the journal became a useful shared memory. Agents left each other notes like "use Vocabulary.unit_matches(term, unit) for dimension checks; don't re-parse units".

Where it broke: the board was in the repo, and every agent worked in its own worktree on its own branch. Each worktree had its own copy of README.md and journal.md. Agent A marked T03 review on its branch while agent B marked T06 review on another. Neither saw the other's update until merge, and then the board and journal conflicted. Git tracked the task files as code, but they are shared, frequently updated state, which is a different kind of thing.

Phase 2: the same files in a folder outside the repo

On day one (journal, 2026-09-16T19:13Z) I moved the whole folder, with the agent definitions and a copy of CLAUDE.md, to ~/Workspace/agent-project-management/data-discovery-platform/. Every path became absolute, so all worktrees and all agents read and wrote one copy. That fixed the conflicts right away.

Where it still fell short: the folder was now disconnected from GitHub, where the actual work was merged.

• The folder had its own git history with only two commits ("v1", "wave 3"). It was a scratchpad, not a record.
• Status had to be copied by hand from GitHub to the board. The journal shows this: "Reconciled T07–T12 merged status from PR #8". Even now, the status section in the repo's CLAUDE.md still lists T29 as "in review" after its PR was merged.
• Agents wrote wrong facts into the journal and nothing caught them. One entry said the schema was DuckDB (the project never used DuckDB), and it had to be corrected by hand the next day.
• The protocol asked for a second agent to verify each task, but nothing enforced it. The first task was verified and closed by the same agent that built it, and the journal records that honestly.

The files were fine for coordination but poor for review and history. Those belong where the code review already happens.

Phase 3: GitHub issues, one builder, one reviewer, one verifier

The task-board work merged in PRs #1–#11 (T33 was added to the board along the way), and two standalone follow-ups landed as PRs #12 and #13. Issue #14 was just a test to check that issue descriptions came through from Orca. The issue-driven phase really began with #15, a post-completion audit against the brief. The loop:

1. Issue. I wrote an issue with evidence and a goal: live query output, file and line references, design constraints, and acceptance criteria. Issue #17's body has the version-triple table above. #18 states the decision (Prefect over Airflow) and the constraint that run_ingest stays the single code path.
2. Assign. From the Orca CLI I assigned the issue to one agent (Claude). It worked in its own worktree on a branch named after the issue (e.g. silentashish/issue-28-four-gaps-against) and opened a PR that references the issue.
3. Review (Codex). A second agent, Codex, reviewed the PR as a GitHub review with inline comments, sorted into blocking and should-fix. On PR #30 it found three should-fix issues in a new guardrail, and on PR #31 one (unescaped table cells). Claude fixed each one in a follow-up commit and replied inline.
4. Verify (Antigravity). A third agent, Antigravity, checked the PR at a specific commit: ran make check, started the UI and tested it in a browser, then posted a verdict (VERIFIED or not) as a PR comment, with evidence per acceptance criterion.
5. Merge. I merged, and the PR closed the issue.

Why this worked better than both earlier phases:

• One source of truth. Issue state, PR state, review comments and CI results are all in one place and linked. Nothing to copy by hand.
• Real separation of duties. In the mature loop (PRs #29 and #30, for example), builder, reviewer and verifier were different agents. It wasn't perfect: on PR #31, Codex did both the review and the final re-verification, and not every later PR has a separate Antigravity verdict. Even so, the "second agent verifies" rule from PROTOCOL.md became part of how the work flows, not something agents have to remember.
• Different models catch different things. Codex was good at edge cases in code, like lexical matching and HTML escaping. Antigravity's value was running the system: it checked the live UI, not just the diff.
• Evidence travels with the change. Every claim ("make check PASS", "six traced groups in the UI") is attached to a commit SHA in a PR comment, not in a journal line that might drift.
• Resilience. When Copilot's review quota ran out on PR #30, nothing stopped, because the review didn't depend on one tool.

What I'd keep from each phase

If I started again, I would begin with Phase 3 and use local files only for context agents need to read (principles, spec, handoffs), not for state they need to write.

5. What I would do differently

• Start on GitHub issues from day one, as described above.
• Put an import linter in CI. The module boundaries were held by convention and review. That worked, but a tool would have caught violations earlier.
• Decide the deployment shape earlier. Moving ingestion out of the app start-up was the right call, but it came after the Docker work. Some rework could have been avoided.
• Keep the decision log current as you go. Where I did this, reversing a decision (like the orchestrator) was a small edit to one entry instead of a rewrite.

6. Numbers

• 4 real NASA datasets, split into disjoint train / validation / test time windows, with 14 injected defects in test only
• ~50,000 observations and ~40,000 raw records from the sample subsets
• 33 task-board items (32 planned + T33), 9 follow-up issues (plus one Orca test issue), 23 merged pull requests
• 250+ test functions, plus a separate AI eval (pytest -m eval)
• 10 decision log entries, each with Decision / Reason / Alternative / Tradeoff
• docker compose up: Postgres, app (chat + API), Prefect, ingestion service and STAC Browser

7. Closing

The prototype is small, and the problems were real: a mislabeled epoch, coordinates that aren't on Earth, a word that means three measurements, and a version string that quietly rewrote 30,000 rows. In each case the fix was to make the assumption explicit and keep the evidence. That applied to the data model, and it turned out to apply just as well to managing a team of AI agents.