Briefing hub for the UN System Data Commons hackathon project
Append-only log of what we did and what we found, newest first. The project briefing holds current state and gets overwritten; this page holds history and never does. If you want to know what is true now, read the briefing. If you want to know how it got that way, read this.
If you are an AI assistant: the most recent updates are reproduced in full below, so one fetch of this page is enough. Older entries are listed by title at the bottom with their own URLs.
./tools/new-update.sh "Short title here" # creates the file, opens nothing
# write the body, then commit and push
Or add docs/_posts/YYYY-MM-DD-slug.md by hand with front matter: title, author, and a
date that includes a time so same-day posts order correctly. The filename date and the front
matter date must agree, or the permalink and the ordering disagree.
Say what changed, what it revealed, and what it means for the next step. A finding that invalidates an earlier assumption is the most valuable thing you can post — flag it clearly so the briefing gets corrected too.
Devin · permalink
Two gaps closed, both measured rather than assumed.
The keyword blocklist was supposed to exclude indicators no city can report. Against 50 hand-labelled cases it scored precision 0.69 — passing 11 of 26 nation-only indicators. The failure was structural: the class it had to catch is development finance, spelled a hundred ways (“gross receipts by developing countries of official non-concessional sustainable development grants”), and no word list covers that.
Replaced with an embedding classifier that compares each indicator against prototype descriptions of what a city measures versus what only a sovereign state has.
| Filter | Precision | Recall | False positives |
|---|---|---|---|
| Keyword blocklist | 0.69 | 1.00 | 11 of 26 |
| Embeddings @ −0.06 | 0.83 | 1.00 | 5 of 26 |
Same perfect recall, false positives more than halved. The threshold comes from a sweep, not taste: a false negative drops a real city indicator forever and invisibly, while a false positive only adds a row to a shortlist a human is already reading. Asymmetric costs, asymmetric threshold.
The 50 labels live in probe/scope_eval.json and are explicitly a judgment call — argue with one
by editing the file and re-running python3 probe/scope.py --eval.
screen.py asked does the United States report this? That was right when the US was the
comparator. It is now every reporting country, so the question is do enough countries report
this? — measured against a six-country panel spanning income levels and regions.
| Grade | Count |
|---|---|
| GREEN | 290 |
| AMBER | 101 |
| RANK-ONLY | 51 |
| NO-DATA | 233 |
| THIN-COVERAGE | 14 |
442 usable, of which 130 are indicators the US does not report — invisible to the old screen. After the scope filter, 67 are genuinely city-scoped. That is the real unlock, and it is smaller than the raw 130 while being far more real than the 328 the keyword filter would have waved through.
RANK-ONLY is a new grade for single-observation indicators. The old grading collapsed those into
RED, which is how we wrote off road deaths before discovering 195 countries report it.
Mostly waste and water — and that is a finding about the United States, not about NYC:
The US does not report municipal waste collected to the UN. Dozens of countries do, and NYC has DSNY tonnage updated monthly. A city can be compared internationally on exactly the indicators its own country skips. That inverts the usual assumption about who has the data, and it is the second time this project has found the gap on the UN/national side rather than the city side.
A panel of twelve countries was my first attempt and it was wrong — the response cap applies to variables × entities together, so every batch overflowed and split down to single variables. Correct but needlessly slow. Six countries runs clean with zero short responses, which is the signal that nothing is being silently truncated.
And the classifier is not clean at 0.83: “Total inbound official flows for infrastructure” and “International financial flows to developing countries” both survive into the 67. That is the expected error rate showing up exactly where predicted, and it is why the output stays a shortlist for human review.
Devin · permalink
We graded road deaths BLOCKED because the UN holds one observation for the United States. That is true of a trend and irrelevant to a ranking: WHO publishes a single global round, and in 2021 195 countries reported. One year is more than enough to place a city.
The blocker was our framing, not the data.
NYC: 297 road deaths in 2021, 3.51 per 100,000 — 18th of 196.
| per 100k | |
|---|---|
| Germany | 3.30 |
| Netherlands | 3.40 |
| Spain | 3.50 |
| New York City | 3.51 |
| Cyprus | 3.90 |
| United States | 14.20 |
NYC’s streets are as safe as Western Europe’s and roughly four times safer than the country it sits in. That finding was completely invisible while the comparator was the US — which is the best argument yet for the world view existing at all.
NYC’s eleven years of rates and the UN’s single 2021 observation share one axis. The asymmetry is the argument: a dense blue line running low and flat, and a lone orange dot four times higher.
The grade is new — RANK ONLY. Not blocked, not chartable as a trend. NYC can be placed among 195 countries and cannot be tracked against them, because the world has one snapshot and NYC has eleven years. The inversion panel now says that instead of claiming the comparison fails.
The first version drew NYC’s line straight through 2020 while annotating “no ACS 2020” beside it. The series omitted 2020 rather than carrying an explicit null, so the line bridged the gap. The homicide chart already broke correctly.
On a page whose entire argument is that missing data should stay visible, that was the wrong bug to ship. Fixed: an explicit null, a real break.
The screening criterion should change too: screen.py asks does the US report this? when the
comparator is now the world. The right question is do enough countries report it?
Devin · permalink
Every chart in the demo compared NYC to the United States. That was a limit of my framing, not of
the data: get_child_observations over Earth / Country returns every reporting country in
one call, and the response carries entityMetadata names, so no separate lookup is needed.
Each card now shows where NYC sits among all reporting countries in 2019.
| Indicator | NYC | Rank | Nearest neighbours |
|---|---|---|---|
| PM2.5 (city aggregates) | 6.60 µg/m³ | #5 of 186 | Finland, Estonia, Iceland |
| Homicide | 3.63 per 100k | #85 of 137 | Pakistan, Montenegro |
| Waste recycled | 17.3% | #38 of 64 | Bahrain, Greece |
Against the United States, NYC crossed below the national rate in 2013 and stayed below — a success story, and the one we put on the chart yesterday.
Against the world, NYC sits in the bottom half, 85th of 137, between Pakistan and Montenegro, with 84 countries reporting a lower rate.
Same number. Different comparator. Opposite conclusion. Neither is wrong. A tool that shows only the first is not neutral — it is flattering, and it is flattering by omission. This is the definitional-caveat lesson again, arriving through a different door: the choice of who you compare to is as load-bearing as what you measure.
Tier still governs. PM2.5 is Tier 1, so NYC is measured against other countries’ city aggregates — genuinely like-for-like. Homicide and waste are Tier 3: a city against whole nations, which is real context and not a peer comparison, since cities generally run above their national averages.
date: "latest" returns each country’s own latest vintage — Afghanistan 2023 sitting beside
Aruba 2014. That is precisely the mixed-vintage comparison this project exists to catch, so
everything is pinned to 2019, the last year with wide coverage across all three.
Malaysia reports 147.7% of its municipal waste recycled in 2019. You cannot recycle more waste than exists. It is in the UN SDG database, and one bad figure was flattening the entire distribution on the chart.
We clipped it off the axis, drew it in red at the edge, kept it in the data and named it in the
caption. Deleting it would have produced a cleaner chart and a dishonest one — a silent drop is
how a dataset launders its own errors. Two countries also return empty names from
entityMetadata; those fall back to their ISO codes rather than rendering blank.
For a room asked how do you hold the line on truth in a world flooded with synthetic data, a demonstrably impossible figure sitting in authoritative UN statistics is a more useful exhibit than anything we could have contrived.
Devin · permalink
Expanded the crosswalk from 3 pairs to 11. Three are chartable. That ratio is the finding.
| Pair | SDG | Tier | Grade | Chartable |
|---|---|---|---|---|
| PM2.5 annual mean | 11.6.2 | 1 | PROXY | yes |
| Intentional homicide | 16.1.1 | 3 | PROXY | yes |
| Municipal waste recycled | 11.6.1 | 3 | PROXY | yes |
| Road traffic deaths | 3.6.1 | 3 | BLOCKED | no |
| Child mortality | 3.2.1 | 3 | CONTEXT | no |
| Maternal mortality | 3.1.1 | 3 | BLOCKED | no |
| Deaths from air pollution | 3.9.1 | 3 | CONTEXT | no |
| Poverty | 1.2.1 | 3 | CONTEXT | no |
| Inadequate housing | 11.1.1 | 2 | CONTEXT | no |
| Built-up area per capita | 11.7.1 | 1 | NO-NYC-SOURCE | no |
| Safely managed drinking water | 6.1.1 | 3 | NO-SIGNAL | no |
First run reported BLOCKED and CONTEXT pairs as “chartable: yes”. The mechanical checks — does it resolve, do units agree, is there overlap — all passed, so the probe waved them through over the top of a human judgment that said these must not share an axis.
Road traffic deaths was the clearest case: marked chartable on a single overlapping year.
Fixed two ways. The human grade now vetoes the mechanical result, and a trend needs at least five overlapping years rather than a shared endpoint. Worth dwelling on, because it is the product’s whole thesis reproduced in our own tooling: every automated check passed, and the answer was still wrong. Units agreeing is necessary and nowhere near sufficient.
Road traffic deaths is blocked by the UN side, not by us. NYC has Vision Zero collision data updated daily, every fatality geocoded. The UN has one modelled estimate from 2021. For the city’s flagship street-safety programme, the international comparison simply does not exist.
That is the argument for city data flowing toward the UN system rather than only the reverse, and it is a better conversation to have with UNSD in the room than another dashboard.
Devin · permalink
Builders’ Day is Tue 22 Sep, the morning is single-track demos, and until today we had no demo — a research pipeline and a docs site, neither of which is watchable for ten minutes in a room of forty builders. Now there is one.
Three charts that hold, each carrying its source, vintage, unit and grade on the face of the card. Then the eight that don’t, with the reason each one fails. The blocked ledger is not an appendix — it is the claim.
Municipal waste recycling. NYC at 17.2% against 23.6% for the US in 2018: a six-point gap and an easy headline. Except DSNY tonnage covers residential collection only and the UN definition includes commercial waste, so the two numbers do not cover the same universe.
Same units. Same period. Same apparent shape. Every automated check passes. Only a person reading both definitions catches it — which is the argument for the entire project, in one chart, in about ninety seconds.
Published as a Claude Artifact and mirrored on GitHub Pages. The Pages copy is public and needs no sign-in, which is what a showcase screen should depend on.
Devin · permalink
Census key in hand, so the denominator problem is solved and every Tier 3 rate indicator is unblocked, not just homicide. All three crosswalk pairs now resolve end to end.
NYC counts murders; the UN publishes a rate per 100,000. With ACS annual population as the denominator, the two finally share an axis — and the result is a real finding:
| Year | NYC | US |
|---|---|---|
| 2006 | 6.93 | 5.79 |
| 2010 | 6.50 | 4.73 |
| 2013 | 3.83 | 4.47 |
| 2017 | 3.22 | 5.21 |
| 2021 | 5.62 | 6.78 |
| 2023 | 4.66 | 5.76 |
NYC crossed from above the national homicide rate to below it around 2013, and stayed below through the 2021 spike. That is the kind of statement a city analyst can actually use, and it is invisible from either dataset alone.
There is no ACS 1-year release for 2020 — the Bureau withheld the standard product after COVID disrupted collection. So 2020 has no denominator, and therefore no rate.
2020 is the year NYC homicides jumped to 456. The gap lands precisely on the most interesting year in the series.
We leave it empty. An interpolated denominator produces a rate that is visually indistinguishable from a measured one, and this is exactly the case where someone would quote it. The probe records dropped years explicitly rather than silently omitting them. 2025 is also absent for the ordinary reason that the vintage is not published yet.
The Census key lives in a gitignored .env and is read from the environment. It is in no
tracked file and no commit — verified against the full history, not just the working tree.
This repo is public and its parent workspace has a live incident from committed keys, so the
check is deliberate rather than assumed.
Collaborators: get your own free key at https://api.census.gov/data/key_signup.html and export
CENSUS_API_KEY. The probe explains this if the key is missing.
Devin · permalink
Earlier today I reported that URBANIZATION--DOU_CITY recovers peer-city comparison and
recommended it as the primary framing. That was an over-generalisation from the one indicator I
tested it on. Scoping it properly changes the picture.
Scanned 23 topic areas, 5,320 variables. Indicators carrying a DOU_CITY slice: five.
undata/sdg/EN_ATM_PM25 — PM2.5undata/unicef/DM_BU_PC_DOU — built-up area per capitaundata/unicef/DM_POP — populationundata/sdg/AG_PRD_FIESS / AG_PRD_FIESMS — food insecurityThey cluster in indicators derived from gridded geospatial data (the GHSL settlement layer), which is the only family where a national figure can be cut by settlement type. Homicide, waste, unemployment, poverty and renewable energy have no such dimension and will not get one — they come from administrative reporting with no spatial component.
About eight more indicators carry only DOU_U (urban vs rural). DEGURBA “urban” bundles cities
with towns and suburbs, so it is a coarser class, and the indicators it covers skew toward
electricity access, handwashing and open defecation — little NYC relevance. Slums and school
completion are the exceptions.
| Tier | Comparator | Coverage |
|---|---|---|
| 1 | NYC vs national city aggregates, many countries | ~5 indicators |
| 2 | NYC vs national urban aggregates | ~8, mostly low NYC relevance |
| 3 | NYC vs national totals | everything else — the large majority |
Tier 1 is real and PM2.5 is an excellent demo of it. The general case is still Tier 3.
That is arguably a better story than “we found city-level UN data.” It means the tool’s core job is telling a user which tier they are in and what that permits them to claim. A naive dashboard renders all three tiers as identical bar charts, and that is precisely the failure we are building against.
Devin · permalink
Two probes done. One unblocks the open decision; the other says the NYC side needs the same rigour we gave the UN side.
URBANIZATION--DOU_CITY is a real population-weighted aggregate, not a token. For the US it
covers 121M of 347M people, and the city figure differs meaningfully from the national one
(PM2.5 2019: 7.57 city vs 7.18 national).
The important part: it exists for other countries too, with full 10-year series. PM2.5 city aggregate, 2019 — US 7.57, Canada 6.87, UK 10.06, Japan 11.32, Germany 11.92, France 12.03, Colombia 15.97, Mexico 19.04.
So we can compare NYC against the city aggregate of other countries using nothing but UN Data Commons — one source, one method, one unit, consistent DEGURBA definitions. That is arguably better than the NYC-vs-London comparison we originally promised: no cherry-picked comparator, no second source to reconcile. Recommending this as the primary framing.
Caveat to state on any such chart: NYC is itself inside the US city aggregate (~7% of it).
Built a pair probe that verifies both sides of every crosswalk entry. It immediately found things a human eye would have missed:
month field is text ("2026 / 08"), so date_extract_y fails outright — and
rows before 1993 predate curbside recycling entirely.Two of three pairs now resolve end to end. Homicide is blocked: NYC publishes a count, the UN publishes a rate per 100,000, and the Census ACS denominator now needs a free API key.
PM2.5 — NYC is below the US city average and the gap is widening: 8.93 vs 9.23 in 2014, 6.60 vs 7.57 in 2019.
Waste recycling — NYC at 17.2% against 23.6% for the US in 2018. But NYC’s figure is DSNY residential collection only, while the UN municipal-waste definition includes commercial. The gap is probably overstated, and the honest answer is that these are not measuring the same universe.
That contrast is the demo. One indicator where NYC looks good, one where it looks bad, and the bad one carries a caveat that changes the interpretation. It is the argument for the whole project in two charts.
Crosswalk status ·
python3 probe/pair_probe.py
Devin · permalink
Replaced the keyword matcher. The headline is not that embeddings are better — it is how much, and that we measured it instead of assuming.
Ground truth is the seven NYC datasets already verified by hand in crosswalk.json. The
question: out of 2,400 datasets, where does each method rank the correct one?
| Method | Median rank | top-10 | top-50 |
|---|---|---|---|
| Keyword overlap | 1535 | 2/7 | 2/7 |
| Embeddings (name + description) | 27 | 2/7 | 4/7 |
| Embeddings (+ columns, tags, category) | 23 | 3/7 | 5/7 |
A ~65× improvement in median rank. Keyword overlap was not merely imperfect — at a median rank of 1535 out of 2400 it was worse than useless, since it ranked the right answer below the midpoint of a random shuffle. Every “candidate” it produced was effectively arbitrary.
Retrieval over the whole catalog. The old matcher could only re-rank whatever Socrata’s keyword search returned, so retrieval was the real ceiling, not scoring. All 2,400 datasets are now cached and embedded locally.
Columns, tags and category are worth as much as the description. Adding them moved Proportion of municipal waste recycled → DSNY Monthly Tonnage from rank 882 to 23, because the dataset’s title never says “recycled” — the concept lives in its fields and its tags.
Two of the seven stay unfindable at any rank. NYC’s homicide series is offence code 101 inside “NYPD Complaint Data Historic” — a fact that appears nowhere in that dataset’s metadata. No text method can recover a mapping that depends on knowing what is inside a dataset.
So roughly 30% of real mappings need human domain knowledge and always will. That is the argument for the shortlist being a triage aid with a person in the loop, which is the same conclusion as the grade veto and for the same reason.
248 GREEN → 73 excluded as inherently national → 53 with a candidate above the similarity floor, 50 not yet in the crosswalk. Eyeballing the top of the list, most are now plausible rather than most being noise.
New finds the keyword matcher never surfaced:
pip3 install model2vec — static embeddings, CPU-only, no torch, tens of megabytes. Without it
the pipeline falls back to keyword matching and prints exactly how bad that is. The repo keeps
its clone-and-run property; it is just measurably worse in that mode.
The similarity floor of 0.50 is a calibration choice with no theory behind it — the median top candidate across all indicators is 0.46, so it keeps roughly the better half.
Devin · permalink
Repo and briefing hub are up at sarapis/undatacommons-nyc, everything public.
The briefing page is written to be read by an AI assistant as well as by a person — paste the URL into Claude and ask it to get up to speed. Verified that flow actually works before calling it done, rather than assuming.
Two pages, two jobs. The briefing holds current state and gets overwritten, so it is always true now. This feed holds history and never gets overwritten. When a finding invalidates something in the briefing, post here and correct there.
Open decision that blocks the benchmark view design: do we compare NYC to peer cities (needs a non-UN source — OECD metro, Eurostat Urban Audit, UN-Habitat) or to nations (free and honest today: “NYC’s rate sits between Portugal and Slovenia”)? Logged in the decision log.
Devin · permalink
Built the coverage probe harness and ran it across 15 candidate topics. It sweeps indicators, pulls each full series, and grades them on whether they can actually carry a chart. First run: 30 GREEN, 9 AMBER, 5 RED.
Road safety is dead as our headline demo. All three road-traffic variables came back RED with a single observation each — SDG 3.6.1 has exactly one US data point, 2021. That was the worked example in our application. Better to find out now than at 4pm on demo day, which is the whole reason the harness exists.
The replacements are stronger anyway. Homicide rate (16.1.1) has 21–31 annual points back to 1990 and NYC publishes closely matching data. Municipal waste recycled (11.6.1) has 19 points and pairs naturally with the DSNY diversion rate. Renewable energy share and unemployment are both dense too.
The genuinely interesting find: several variables carry an URBANIZATION--DOU_CITY slice, e.g.
undata/sdg/EN_ATM_PM25.URBANIZATION--DOU_CITY. It is a national figure cut by degree of
urbanization — “the city parts of the US”, not any particular city — but it may be the most
honest comparator available for a NYC number, and it partially softens the no-city-data finding.
Worth an hour before we commit to a framing.
Two limitations, documented rather than hidden: the semantic search drifts (querying “urban
public open space” returned PM2.5 variables, which is exactly why the crosswalk needs human
review), and the modelled-estimate flag is name-based and under-detects — AIR_DEATH_R is
modelled but does not say so in its name.
Latest coverage report ·
regenerate with python3 probe/coverage_probe.py.
Devin · permalink
Probed the live staging deployment before writing any integration code. The headline result changes the shape of the project.
NYC resolves as an entity in the graph (geoId/3651000, type City) but appears in no
variable’s placesWithData — not for road deaths, not for PM2.5, not even for total population.
Scoping a search to NYC alone returns zero variables and zero topics. places turns out to be a
hard availability filter, not a hint.
So there is no “look up NYC in UN Data Commons,” and anything that assumed we would query both sides and join is wrong. The crosswalk is the product, not a feature of it. That is awkward for the build but good for the pitch — the gap we proposed to fill is verifiably there.
Two supporting findings. The MCP surface only ever returned governed undata/ variables, while
REST is federated with the wider Data Commons graph and will answer for other publishers without
warning — so discovery goes through MCP, always. And provenance is first-class: every
observation carries a provenanceUrl, an observationPeriod, and a unit DCID that encodes the
denominator, which means part of our comparability check can be automatic rather than
hand-curated.
Full detail and reproducible commands: platform probe findings.
Devin · permalink
The first eleven crosswalk pairs came from topics I chose out of my head, which meant the crosswalk could only ever confirm my assumptions about NYC’s data. This replaces that with a three-stage pipeline driven by the UN corpus.
python3 probe/corpus.py # enumerate -> 689 base SDG indicators
python3 probe/screen.py # US coverage -> 248 GREEN
python3 probe/match_nyc.py # NYC candidates -> ranked shortlist
689 base SDG indicators. The eleven hand-picked pairs were 1.6% of it.
| Screen | Count |
|---|---|
| GREEN — usable US series | 248 |
| AMBER | 37 |
| RED | 27 |
| NO-US-DATA | 377 |
55% of SDG indicators have no US data at all. The real candidate pool is 248, of which my hand-picking sampled 4%.
The goal trees do not expose variables. Walking sdgf/goal-* yields undata/svpg/... nodes
— StatVarPeerGroups — which carry no observations. The tempting move is rewriting
svpg/sdg/X to sdg/X, which looks right and is precisely the DCID guessing the platform
forbids. The correct path is following each group’s ->member arc. Verified the svpg nodes
return nothing before building on them.
get_variable_metadata silently truncates above ~10 variables per call — status: None and
an empty map, not an error. The first screening run used batches of 40 and reported
“screened 689/689” having actually recorded nine. Now capped at 10, with any short response
treated as failure and re-split, and anything the graph never returned reported explicitly.
That is the third time on this project that every automated check passed and the number was still wrong. It keeps being the same lesson.
First run produced mostly false positives. Short indicator names have few keywords, so one coincidental word scored 1.0:
Tightened two ways: a match now needs at least two distinct indicator terms rather than high proportional coverage alone, and inherently national indicators (balance of payments, ODA, tariffs, treaties, fisheries) are excluded before searching, since a city does not publish them and matching could only manufacture noise.
Result: 248 GREEN → 73 not city-scoped → 43 with candidates, 39 not yet in the crosswalk. Roughly a third of those look plausible on inspection. Term overlap is a genuinely poor proxy for semantic equivalence and no amount of tuning will fix that; the output is labelled a shortlist for human review and should be read as nothing more.
EN_ATM_CO2, 24 obs) against NYC’s Climate Budgeting
emission factors.EG_EGY_PRIM, 34 obs) against Local Law 84 building
energy benchmarking — an area where NYC’s data is unusually strong.SG_XPD_ESSRV, 24 obs) against Agency Spending
by Budget Function.Those are the pipeline earning its keep. None were on my list.
The honest improvement is not more scorer tuning. It is using a real embedding model against NYC dataset titles instead of my keyword overlap — the platform already does this properly for its own search, and I am reimplementing it badly.