DARPA program analysis sites: evidence you can audit

The situation

A program report names a number. Downstream, the number gets used as if someone had inspected it. Inspection is rare. The figure came from a consultant or an analyst's deck, and chasing it back through the chain of memos and slides requires a phone call and the willingness to wait days for an answer. By the time a hearing-room reviewer asks where the figure came from, the provenance has gone cold.

When the stakes are high and the review is hostile, that arrangement collapses. A program office needs to be able to open the source of any number in front of the person asking and watch it reproduce on the spot.

What we built

We built per-program analytical sites for DARPA. One site per program. The site walks a reader through the program's competitive picture, its research security exposure, the depth of engagement its publications have received, the commercial translation activity downstream of its funding, the case studies it produced, and the methods used to reach each conclusion.

Every quantitative sentence on every page is backed by an entry in an evidence ledger that lives next to the section it supports. Each entry has four fields. The exact claim text, written word for word as it appears on the page. A confidence level of HIGH, MODERATE, or LOW. A reproducible source, which can be the query that produced the number, the path to a data file with the steps to verify, or an external citation. The expected result the verification should return.

Before anything ships, a verification pass walks the entire site, reproduces every number, and compares the outputs against what the ledger expects. A discrepancy stops the build. Either the claim changes or the ledger gets corrected.

What happens when a number is challenged

A reviewer questions a figure on page four. The program office opens the ledger entry next to that figure. The entry names the source. The number reproduces in front of the reviewer. If it doesn't, the claim comes off the site before the meeting ends.

The same discipline runs deeper than reproducibility. The methods on these sites enforce what specific metrics actually mean. A "highly influential citation" in Semantic Scholar's algorithm measures how substantively a citing paper engages with the work it cites. It does not measure the cited paper's quality. It does not measure the citing institution's research impact. Mixing those readings up produces confident conclusions that are wrong in strategically important directions, and the analytical methods reject sentences that subtly flip them.

Defense and intelligence terminology gets the same treatment. "Threat," "risk," and "adversary" have formal meanings inside the IC, and the site keeps those words inside the IC's hands. A citation pattern gets called a citation pattern. A technology-transfer concern gets called a technology-transfer concern. The data dictates the vocabulary, and the vocabulary stays honest about what the data shows.

What it replaced

This work usually takes one of two forms. A PDF goes stale the day it's printed and asks the reader to take its numbers on faith. A live dashboard asks the same thing dressed in a chart. Either way, the program office spends its authority on the underlying question defending claims it can't reproduce on its own.

What a similar engagement looks like

Per-program, these run roughly two weeks. All we need from you is the program name or the BAA. We pull the publications, build the program metadata, and run the analysis ourselves. We deliver the deployed analytical site, the evidence ledger, the data pipeline that refreshes the citation and bibliometric inputs, and a defined process for making claim changes so the site stays honest after the engagement ends.

Program offices know what they funded. They know what the performers reported. What they almost never see is what the work did next. A program closes, the deliverable ships, and the research begins its second life, the one that runs without the program office watching. Five years out, a foreign defense lab cites the program's paper. The citation is a replication study. The lab wanted to know if the result held. It did. The next year, the same lab publishes five more papers, extending the technique and making it smaller. By year seven, seven more on hardening the technique for the conditions it would have to survive. Year eight brings five studies on prototype deployments. A single citation has become a research program in another country, and that's only one of the second lives. Ten years out, a startup somewhere else pitches a product that traces back through three layers of academic work to a paper the program seeded. The site we build is for that second life. It maps where the work diffused and who picked it up. That includes the institutions that built on it, the fast followers, the commercial translations that produced VC-funded products with a clear pedigree, the intellectual supply chains the program now sits at the head of, and the adversarial uptake that programs almost never get to inspect. The program office finally gets to see what its money accomplished. It's overkill if your work never leaves the inside of the program office.