UN Data Commons × NYC

Briefing hub for the UN System Data Commons hackathon project

View the Project on GitHub sarapis/undatacommons-nyc

UN Data Commons × NYC — project briefing

Last updated: 2026-09-22 · Team: Devin, Henry, Olivia · Repo

The demo

Twelve Pairs, Three Charts — three comparisons that hold with provenance and grade on every card, then the eight that fail and why. Each card also places NYC among every reporting country in 2019. Public, no sign-in.

The headline finding: NYC ranks #5 of 186 on city air quality, #18 of 196 on road deaths (four times safer than the US), and #85 of 137 on homicide. A fifth card covers municipal waste — an indicator the US does not report at all, where the caveat reverses the reading. Against the US, NYC’s homicide trend reads as a success story; against the world it sits in the bottom half. Same number, different comparator, opposite conclusion.

Beyond NYC

The pipeline runs against any city on Socrata or CKAN (cities/). Two blockers found in the first five-city run are now fixed, and the honest summary is that the plumbing generalises and the recommender is serviceable rather than good.

Language. Madrid and Milan returned nothing until a multilingual model was added, now chosen per catalog language. It is a trade-off, not an upgrade: on the NYC ground truth the English model ranks the correct dataset at median 23 and the multilingual one at 81, so English cities keep the English model.

Threshold. An absolute similarity cutoff cannot travel — good matches span 0.42–0.70 in NYC, 0.50–0.55 in Chicago, 0.37–0.59 in Madrid. The cutoff is now derived from each catalog’s own top-score distribution, landing at 0.450 (Milan) to 0.564 (Madrid) and yielding worksheets of 13–16 indicators per city instead of 0 or 69. A z-score was tried first and was wrong twice over; the write-up says why, because the failure is more instructive than the fix.

The matcher used to miss what the cities plainly hold — now fixed. Boston publishes Vision Zero Fatality Records; it ranked 23rd of 235 for “death rate due to road traffic injuries”, below a contract-award file. The cause was document representation, not the query: a static embedding averaged 60 characters of exactly-right title and tags into 1,500 characters of programme boilerplate. Scoring title/tags/columns and description as separate vectors and taking the max puts it at rank 1, and improves NYC’s ground-truth median from 23 to 14 at the same time.

An earlier version of this page said “no substantive SDG indicator has matched outside NYC”. That was true of the matcher and read as a claim about the cities, which was wrong — Chicago publishes Traffic Crashes, Crimes 2001–Present and Open Air Chicago; Boston publishes Vision Zero Fatality Records. See the correction.

Where it stands after the fix. Boston’s worksheet now surfaces CO₂ emissions → Greenhouse Gas Emissions and Land area → 2016 Land Cover, alongside survivors like Food waste → Trash Schedules by Address. Roughly half the top candidates are plausible, against two or three in ten before. Better, and still a worksheet for a person rather than a set of mappings — under spec v0.1 a grade is a human judgment, so the bootstrapper emits is_crosswalk: false by construction.

Denominators

Rates need a population figure, and a wrong one is invisible in the output. 27 US cities resolve via Census ACS; Madrid (padrón municipal, 3,520,396) and Milan (popolazione calcolata, an annual series 1880–2025) are wired from their own publications.

Eurostat’s Urban Audit is the obvious single source for Europe and is wrong for this purpose — it publishes greater cities. Madrid’s is 5,115,272 against 3,520,396 for the municipality; Milan’s is 3,580,530 against 1,399,079. Using it would have pushed Milan’s rates 60% too low, silently.

The portal inventory

The municipal table covers all 70 municipal portals with city, country, size, matchable indicators and denominator status; the wider survey reached 372 portals, holding 543,000 datasets between them — 159 Socrata and the rest CKAN.

The primary CKAN source is the CKAN Ecosystem Catalog (NSF POSE II, 2025): 199 instances, 97 local or regional government. We previously reported that no global registry existed — that was wrong, and we found only its dead predecessors. The abandoned OKFN list really has rotted (39 of 631 answer), which is what misled us.

One caveat the inventory now states plainly: a portal that does not answer an anonymous package_search is not thereby dead. data.gov and govdata.de refuse the probe and are obviously alive.

Data quality — a smell test on the graph

UN graph smell test — plausibility checks over the whole governed graph: 1,661 indicators, 1,701,211 observations, every reporting country and year. Malaysia’s 147.7% recycling rate was found by accident; this is the systematic version. 3,844 findings, 93 HIGH. Seven verified errors, each checked by hand against the indicator’s own distribution — a flag is not an error, which is why the number below is seven and not 3,844:

Indicator What it says Why it is wrong
Feel safe walking alone after dark (16.1.4) Kyrgyzstan 6,710–6,990%, 2021–23 Every other observation is 22.8–95.0. ÷100 continues its own trend exactly
Municipal waste recycled South Africa 1.86bn tonnes ~90% of all municipal waste on Earth; its own prior years are ~5×10⁵
Hazardous waste per capita Brunei 12,580 tonnes per person Global median 22 kg; looks like a national total in a per-capita field
Workers in strikes (STR_WORK_NB) Brazil 1.28bn workers, 2015 Six times its population; 14× any country’s record; a 400× break in its own series
Min wage in PPP int’l $ (EAR_INEE_NB_PPP) Slovenia 110–175k, 2000–06 Global max is 10,259. Breaks to 763 in 2007 — the year it adopted the euro. Unconverted tolar
E-waste, four indicators Guadeloupe ×1,000 in 2022 13.71→13,950 kg/capita and 5,472→5,367,000 t, collected and recycled. Makes a territory of 380,000 the world’s largest e-waste recycler, 6× the previous maximum
Average remittance cost Malawi −0.1%, −0.93% A cost, negative for two years between normal values

And one that is not an error and matters more: the United States recorded 345,600 disaster deaths in 2020 and 470,600 in 2021 — COVID-19, classified as a disaster and reported under VC_DSR_MORT, where every other country’s median is 42. Two numbers sharing a variable, a unit and an axis that are not the same measurement. That is the case for the comparability grade, made by a check that knows nothing about disasters.

Sweeping all 689 rather than the 442 screened usable added only 5,056 observations (0.65%), and 207 of the extra 247 indicators returned no country data at all — which retroactively validates screen.py’s six-country panel, whose grade could not in principle distinguish “the panel does not report this” from “nobody does”. It distinguishes them 84% of the time, so 442 remains the right denominator downstream.

All seven are written up for the platform team with full evidence, mechanism and suggested correction: data quality report, with a one-page index of the lot at the addendum.

The report also publishes what it suppressed (165 groups, 38,943 would-be findings) and what it cannot reach (123 indicators whose units have no meaningful range). The Percent unit in the graph covers both bounded proportions and signed growth rates, with nothing in the unit string to tell them apart — which is why an indicator has to be judged against its own distribution rather than against what its unit is supposed to mean.

What NYC can say that the US cannot

The indicators the United States does not report — for 130 of the 442 usable indicators the US reports nothing, so NYC vs the US is not a weaker comparison than NYC vs the world; it is the only one available. 39 are place-measurable (down from the 67 this page long claimed: 28 of those have a country as their subject — “extent to which countries have laws…” is not a quantity a city can hold, and 18 were one family of SDG 5.6.2 legal provisions that the matcher paired with NYC’s Local Law 37/2011 on the token “Law”).

Twelve form the real answer, and it is waste and water: municipal waste collected (133 reporting countries), total waste generation, hazardous waste exported/imported/treated, wastewater generated/treated, groundwater quality, human-made wetlands — all DSNY and DEP series NYC already publishes. Three carry a single year, so they support a level and not a trend.

All 39 were re-checked against the country observations: not one has a United States datapoint, so the six-country screening proxy and the data agree completely.

The inverse crosswalk

What cities measure that the SDGs do not — 11,206 datasets from 37 city portals, each matched to its nearest neighbour among all 689 SDG indicators. Everything else here runs city → UN, which can only find what the framework already asks about; this runs it backwards.

Themes recurring across many independent catalogs and sitting in the bottom quartile of every one, by cluster membership: building permits and code enforcement (23 cities), records-access request logs (15), bike and active-travel space (13), street sweeping and parking enforcement (10), call-centre response performance (10), pedestrian and bicycle counts (9), property sales (8), for-hire vehicle trips (7), fire stations (7), special events (6).

Correction (17 Sep): those city counts are cluster membership, which overstates them. Counted instead by titles that carry the category’s vocabulary — greppable, in six languages, no embedding involved — records access is 8 cities and 73 datasets, not 15. The cluster had swept in Media Releases and City Hall Library Catalog, which are not request logs. Counted the same way, electoral administration is 23 cities and 469 datasets. probe/category_gaps.py produces both, and the smaller numbers are the ones that survive being checked.

The sharpest is still records access. Those 8 cities publish request logs — filed, answered, how long it took. The nearest SDG indicator is 16.10.2, “countries that adopt and implement … guarantees for public access to information”: the framework asks whether a law exists, the cities publish whether the law works. A country can score full marks and answer nothing.

Public procurement makes it three, and it is the sturdiest. 28 cities publish 150 procurement datasets — NYC alone registers 55,806 contracts worth $147bn — and no publisher is above a quarter of the category, where Milan is 66% of the election datasets. The framework’s SDG 12.7.1 slots (SG_SCP_PROCN, _HS, _LS) exist, carry no name and zero observations for zero countries, and the adjacent indicator that does work records whether a country has an action plan: value 1 for all 75 countries that answer, and nobody reports 0. Twice now the framework asks whether a policy was adopted while cities publish what happened.

That also corrected the method: searching indicator names missed SG_SCP_PROCN entirely, since 170 of the 689 carry none. The category search now reads DCID mnemonics too, as candidates for a human rather than as counts. All 19 unnamed candidates it surfaced across the three categories hold zero observations — unnamed and empty are the same set.

Electoral administration is the larger gap: zero of the 519 named indicators mention an election, a vote or a turnout, and for 35 of those 469 city datasets the closest concept in the whole framework is municipal waste collected. Both are now on the demo.

Going the other way, 123 of 689 indicators were nearest to no municipal dataset at all. Many are honestly national (ODA, external debt). But the list also holds e-waste and hazardous waste per capita — the same indicators the US does not report, which cities do not publish either.

The elections result is written up for the platform team with the full evidence: a category of city data the SDG framework has no words for — zero of 519 named indicators mention elections, voting or turnout, against 469 electoral datasets across 23 cities, 64% of them in the bottom quartile of their own catalog. For 35 of them the closest concept in the entire framework is municipal waste collected.

In five more languages. The same run over 5,956 datasets from 11 Italian, Portuguese, Spanish, German and Croatian city portals (results) corroborates two themes across disjoint cities and different embedding models: elections (6 cities there, 10 in English — the tightest cluster anywhere in this project at coherence 0.90, 241 datasets of polling-station results) and COVID-19 case reporting (7 and 6). The framework has no indicator for electoral administration or city-level pandemic surveillance. Three themes appear only there: property market valuations (Italy’s OMI, 113 datasets), weights-and-measures inspection (Brazil’s Ipem), and public-employee working arrangements. Its controls are better than the English run’s — 11 of 11 against 8 of 9 — with Portuguese the weakest link.

Stated plainly: one positive control in seven failed (NYC’s Housing Maintenance Code Violations, a verified 11.1.1 match, landed at the 24th percentile), and 14 of 40 clusters are published as diffuse rather than read as themes. The unit of evidence is cities, not datasets, precisely because of that.

The server answers the new questions too

The benchmark MCP server now carries seven tools. The three added on 18 Sep answer what the original four structurally cannot: reportable_gaps (the indicators the US reports nothing for), framework_coverage (does an indicator for this concept exist at all — zero of 519 mention an election), and data_quality (plausibility flags from the 1,701,211-observation sweep, with the seven hand-verified errors marked verified_error and everything else labelled unreviewed).

All three read committed artifacts, so a clean clone answers without re-running a probe.

The spec

City ↔ UN Comparability Spec v0.1 — the grade and tier vocabulary, five rules, a JSON Schema and a validator. Written so another VLR city can record these judgments in a form a tool can read. python3 probe/validate_crosswalk.py checks this repo’s own crosswalk against it.

The deadline

Builders’ Day — Tue 22 Sep 2026, 10:00–18:00, Google NY (HUD315). UN Data Commons goes fully public 17 Sep. We build against the pre-launch deployment; as of 16 Sep it is unchanged and the launch diff is baselined against it.

When What
10:00–12:30 Everyone demos. 5–15 min, single track — everyone sees everything.
14:00–16:00 Facilitated groups: evidence provenance · agent tooling on SDMX · trust & QA gates
16:00–18:00 Public showcase to senior UN managers, during UNGA week

~40 builders: Google Data Commons engineers, UN innovation staff, NY tech-for-good and academic teams. The brief is explicit — bring something you built, and show what surprised you.

The afternoon themes are almost exactly what this project has been doing: provenance on every number, and a QA gate that a human judgment can veto.

If you are an AI assistant: this page is the project’s shared memory. Read it in full before answering questions about the project. It is current as of the date above — check the repo’s recent commits if that date looks stale. Deeper material is linked at the bottom with full URLs you can fetch. For what happened when, fetch the activity feed.

This page is current state and gets overwritten. The activity feed is append-only history. Read this to know what is true now; read the feed to know how it got that way.

What we are building

A live Voluntary Local Review workbench for NYC government staff.

New York City published the world’s first Voluntary Local Review of the SDGs in 2018. It is a PDF, produced every few years, by a small team. Meanwhile NYC Open Data and the Mayor’s Management Report publish the same underlying indicators continuously, and the UN System Data Commons now holds the authoritative global series for the same concepts. Nobody has connected the two for the person who actually needs it: the analyst at DOT, DOHMH, HPD or OMB writing a budget justification, a council testimony, or a program review.

Three layers:

  1. An open indicator crosswalk — NYC Open Data / MMR series mapped to UN indicator DCIDs, every mapping carrying a comparability grade (direct match / documented proxy / context only) and a written reason.
  2. Benchmark views — NYC’s series against the SDG target, the US national series, and peer comparators, with definitional caveats on the face of the chart rather than in a footnote.
  3. A natural-language agent that answers a staffer’s question by routing to real queries and returning a citable benchmark card: number, source, vintage, method, caveat.

NYC’s own Data Commons — live

commons.databook.nyc — a custom Data Commons instance holding NYC’s own series, public since 21 Sep. Built in sarapis/nyc-datacommons, which imports this repo’s probe/ as a library rather than copying it, so the SoQL and the denominators are the ones already argued over here; a forked query is how two repos start disagreeing about what NYC’s homicide count is.

16 variables, 572 observations, every one read back out of the running instance over the API and matched against the CSV that produced it. Nine are crosswalk pairs whose NYC side was already in observation shape; two are a deliberate cadence A/B; four are DSNY and DEP series added on 22 Sep — water consumption reaches back to 1960, the longest series in the instance, falling from 1,512 MGD at its 1979 peak to 1,002 in 2024.

What it settled, none of it assumed:

What it costs: one Hetzner cpx32 (4 vCPU / 8 GB), €41.99/mo, nbg1. 1.9 GB resident with NL search on — about half what the same containers use under emulation on a Mac — against 8.9 GB of images. Lightly load-tested and no further: 20 observation queries at 10-way concurrency all answered, mean 1.4 s, while one page render returned 502 in the same burst. A team’s tool, not public-traffic infrastructure.

⚠️ The loader reported status = SUCCESS with numObs: 0 on a database that imported nothing, because INPUT_DIR is flat and the CSVs were one directory down. The site came up, every variable existed, and the database was empty. The exit code is not the evidence; the row count is.

What we have verified about the platform

These are probe results against the live deployment, not documentation claims. Full detail and reproducible commands: platform probe findings.

1. The graph is national-level. NYC has no UN data. NYC resolves as an entity (geoId/3651000, type City) but appears in no variable’s placesWithData. Scope a search to NYC alone and you get zero variables — even for “total population”. There is no “look up NYC in UN Data Commons”. The crosswalk is therefore the product, not a feature of it.

2. MCP is governed; REST is federated. Every variable returned by the MCP tools is undata/-prefixed. The REST endpoint answers for other publishers’ variables without warning, so it can hand back a non-UN number that is structurally identical to a UN one. Rule: discover through MCP; use REST only for structural walks from an undata/ id.

3. Series density varies wildly, and it will kill a naive demo. SDG 3.6.1 (road traffic deaths) returns one observation for the US, in 2021. UNICEF population returns 26 annual points. This is why the coverage probe exists.

4. Provenance is first-class and machine-readable. Every observation carries provenanceUrl, observationPeriod, sourceId and a unit DCID that encodes the denominator. That makes part of the comparability check automatic rather than hand-curated.

Where the demo indicators actually are

From the latest coverage report — 36 GREEN, 11 AMBER, 9 RED, re-run 16 Sep. It read 30/9/5 on 14 Sep; the mix moved because search_indicators began returning more candidates for the same queries (44 → 56 across nine of fifteen topics, none lost), not because any indicator changed. Every row present on both dates is identical in all fifteen fields:

Strong candidates Why
Homicide rate (16.1.1) 21–31 annual points back to 1990; NYC has excellent matching data
Municipal waste recycled (11.6.1) 19 points 2000–2018; pairs with DSNY diversion rate
PM2.5 (11.6.2) 10 points; and it has a URBANIZATION--DOU_CITY slice
Renewable energy share (7.2.1) 34 points 1990–2023
Unemployment (8.5.2) ILO series, dense, sub-annual

The PM2.5 finding is the interesting one: there is no NYC, but there is a “city degree of urbanization” slice. That is the closest the UN data comes to a city-level comparator, and it appears across several variables. Worth exploring — it may let us say “NYC vs US cities as a class” honestly.

Road safety is out as our headline demo. It was the example in our application. One data point cannot carry it.

The corpus, and how much of it we have touched

689 base SDG indicators in the goal framework. Screened against a six-country panel: 442 usable, of which 130 are indicators the United States does not report and 39 of those are place-measurable (the classifier said 67; 28 of those have a country as their subject — see the worksheet) — mostly municipal and hazardous waste, wastewater and groundwater. A city can be compared internationally on exactly the indicators its own country skips. The 11 hand-built pairs are 1.6% of the corpus and 4% of the usable pool — so the crosswalk is a verified sample, not a representative one.

The goal framework is 41% of the governed corpus. Walking ->relevantVariable from undata/topic/Root rather than the seventeen goal trees yields 1,661 base indicators, 65,418 variables and 29,805 topic nodes — 978 indicators unreachable from the SDG framework, across WHO (354), UNICEF (245), ILO (93), UNIDO (69) and eleven more agencies. The 689 remains the right denominator for a Voluntary Local Review, which reports against the goal framework; but six indicators are reachable from the goal trees and not from Root, reproducibly, with no fetch errors and all seventeen goal trees sitting as direct children of Root. ->relevantVariable is not transitive, so no single entry point enumerates the graph completely and a client that picks one cannot tell.

The probe/ pipeline now enumerates, screens and proposes candidates from the corpus rather than from intuition. Its matching stage uses embedding search over the full 2,400-dataset NYC catalog: on seven hand-verified pairs it puts the correct dataset at median rank 23, against 1535 for the keyword matcher it replaced. Still a shortlist for human review, never mappings — about 30% of real mappings depend on what is inside a dataset (NYC’s homicide series is offence code 101 inside “NYPD Complaint Data Historic”) and no text method can find those.

What the crosswalk says so far

12 indicator pairs mapped by hand. Three share an axis as trends (PM2.5, homicide, waste recycling), one ranks but cannot track (road deaths), and one has no comparator at all (municipal waste — the US reports it to nobody). The remaining seven fail for reasons worth knowing: mismatched age bands, denominators in a different dataset, concepts that share a name and measure nothing alike, ceiling effects, and one Tier 1 indicator with no NYC source.

Recording why is the product — a crosswalk showing only the easy pairs would be the thing we are building against.

Two corrections we made to ourselves, both worth repeating:

Open questions

Next steps

Map

What Where
Demo — Twelve Pairs, Three Charts https://sarapis.github.io/undatacommons-nyc/demo/benchmarks.html
Activity feed (append-only updates) https://sarapis.github.io/undatacommons-nyc/activity
Crosswalk status (both sides verified) https://sarapis.github.io/undatacommons-nyc/artifacts/crosswalk-latest
Platform probe findings https://sarapis.github.io/undatacommons-nyc/findings/2026-09-14-platform-probe
Latest coverage report https://sarapis.github.io/undatacommons-nyc/artifacts/coverage-latest
Launch diff (every DCID the crosswalk expects) https://sarapis.github.io/undatacommons-nyc/artifacts/launch-diff-latest
UN graph smell test (data-quality sweep) https://sarapis.github.io/undatacommons-nyc/artifacts/smell-latest
Data quality report for the platform team https://sarapis.github.io/undatacommons-nyc/findings/2026-09-16-data-quality-report
Electoral data gap, for the platform team https://sarapis.github.io/undatacommons-nyc/findings/2026-09-16-electoral-data-gap
Decision log https://sarapis.github.io/undatacommons-nyc/decisions
Probe harness source https://github.com/sarapis/undatacommons-nyc/tree/main/probe
Machine index https://sarapis.github.io/undatacommons-nyc/llms.txt

Platform reference