Briefing hub for the UN System Data Commons hackathon project
Append-only log of what we did and what we found, newest first. The project briefing holds current state and gets overwritten; this page holds history and never does. If you want to know what is true now, read the briefing. If you want to know how it got that way, read this.
If you are an AI assistant: the most recent updates are reproduced in full below, so one fetch of this page is enough. Older entries are listed by title at the bottom with their own URLs.
./tools/new-update.sh "Short title here" # creates the file, opens nothing
# write the body, then commit and push
Or add docs/_posts/YYYY-MM-DD-slug.md by hand with front matter: title, author, and a
date that includes a time so same-day posts order correctly. The filename date and the front
matter date must agree, or the permalink and the ordering disagree.
Say what changed, what it revealed, and what it means for the next step. A finding that invalidates an earlier assumption is the most valuable thing you can post — flag it clearly so the briefing gets corrected too.
Devin Balkind · permalink
We have been arguing that a city should publish indicators rather than only tables. As of today there is one to look at: commons.databook.nyc, a custom Data Commons instance holding NYC’s own series.
11 variables, 410 observations, every one read back out of the running instance over the API and matched against the CSV that produced it. Nine are the crosswalk pairs whose NYC side was already in observation shape — PM2.5, homicide, waste recycled, road deaths, child and maternal mortality, air-pollution deaths, inadequate housing, municipal waste. Two more exist only to answer a question we refused to reason about.
Nobody had checked what the platform does below annual granularity. NYC’s structural advantage over a national graph is frequency — daily and monthly data the UN system does not hold — so if that degraded on load, the case for this whole approach would have been weaker than we were saying.
So it was tested rather than argued: DSNY’s tonnage file is natively monthly and both waste series already sum it to years, which means the same rows could be loaded twice, differing only in the granularity of the date column. 260 month-granularity observations came back as months, rendered as months, with the seasonal cycle intact. The advantage survives.
One Hetzner cpx32 — 4 vCPU, 8 GB, €41.99/mo. Memory is about half what the same containers use under emulation on a Mac: 1.9 GB resident with NL search on, against 8.9 GB of images.
Capacity is a belief we tested lightly and will not overstate: 20 observation queries at 10-way concurrency all answered, mean 1.4 s — and one page render returned 502 during that burst. A team’s tool, not public-traffic infrastructure, until somebody runs a real load test.
The caveat is one click further away than the citation. Each variable’s comparability grade,
its reason, its tier and the UN series it maps to are carried in the MCF description, land in
the database, and are displayed in full on the knowledge-graph page. They are absent from the
chart’s “About this data” dialog, which shows source and citation only. The thing this project
most wants attached to a number is the one thing the chart surface does not carry.
A blank unit renders as Count. On a per-100,000 rate that is not a missing label but an
incorrect one, which is worse than the blank it replaced. There is no unit in the base graph
meaning “per 100,000 people”, so we defined Per100kPeople and KgPerPerson as UnitOfMeasure
instances in MCF. Establishing that took an experiment rather than a guess: the timeline axis
renders a unit’s DCID and ignores both name and shortDisplayName, so a custom unit’s DCID
is its display string. The first attempt produced the axis label
Count (PerOneHundredThousandPeople).
The first load reported status = SUCCESS, metadata = {'numVars': 0, 'numObs': 0}, and exited
0. Every variable existed, the embeddings built, the site came up — against an empty database.
INPUT_DIR is flat, the loader globs it for *.csv and never resolves config.json’s file keys
as paths, and our CSVs were one directory down.
The exit code is not the evidence; the row count is. The runner now greps both numbers out of
the log and fails loudly on a zero, and verify.py reads every series back out of the instance
before a load is called done.
Council district, NTA, community district and police precinct do not exist as classes in the base graph at all — city, borough, census tract, ZCTA, school district and state do. Publishing NYC data by council district means defining the class and its entities in MCF, which is supported work with a known shape rather than an open question.
The code lives in a separate workspace, ~/Antigravity/nyc-datacommons, which is local only
and has no remote yet — so unlike this repo, the instance’s build is not public.
Devin Balkind · permalink
Every count this project publishes rests on one number: 689 base indicators, enumerated by
walking ->relevantVariable from the seventeen SDG goal trees. Ran the same walk from the graph’s
actual root to see what else is there.
| From the 17 SDG goal trees | From undata/topic/Root |
|
|---|---|---|
| Topic nodes | 3,761 | 29,805 |
| Peer groups | 1,916 | 19,532 |
| Variables | 6,025 | 65,418 |
| Base indicators | 689 | 1,661 |
978 base indicators are unreachable from the SDG goal framework — WHO 354, UNICEF 245, ILO 93, UNIDO 69, UNAIDS 41, OHCHR 36, UNFPA 32, UNDP-HDRO 28, ITU 25, UNESCO 18, UNODC 12, ECLAC 6, UNDRR 5, IOM-DTM 1, UNHCR 1. The goal framework is 41% of the governed corpus.
Six base indicators are reachable from the goal trees and not from Root:
undata/sdg/SG_DSR_SILN undata/sdg/VC_DSR_AGLH
undata/sdg/SG_DSR_SILS undata/sdg/VC_DSR_CHLN
undata/sdg/SM_POP_REFG_OR undata/sdg/VC_DSR_HOLH
All seventeen goal trees are direct children of Root — we checked, they are 17 of its 42 children. So a traversal from Root should be a strict superset of a traversal from the goal trees, and it is not.
Three things rule out the boring explanations:
undata/sdg/ indicators are reachable from Root and
not from the goal trees — the youth-in-parliament series SG_DMK_PARLYTH* and the global
citizenship education series SE_SGE_* among them.So ->relevantVariable is not transitive across these hierarchies, and no single entry point
enumerates the graph completely. A client that picks one — as we did, for principled reasons —
gets a silently incomplete corpus and no way to know it. That is worth the platform team knowing,
and it is the third structural item we have for them, after the 170 unnamed indicators and the
Percent unit covering both bounded proportions and signed rates.
The 689 is still the right denominator for this project, and nothing published needs correcting. A Voluntary Local Review reports against the SDG goal framework; enumerating from those seventeen trees is the correct scope, not a shortcut. The demo’s masthead says “689 base indicators in the goal framework”, which is exactly what it is.
What changes is that we now know the framework’s share: 689 of 1,661, and the other 978 are the agency series — WHO’s health indicators, ILO’s labour series, UNICEF’s child statistics — which a city might well want and which no SDG-scoped enumeration will ever surface.
The smell test is running against the 978 now. The first twenty-five rows already show ILO series
with 1,067 and 1,471 observations across 135 countries, so this is not a thin surface. All five
data errors found so far came from undata/sdg/ alone, which was 41% of the graph; this is the
other 59%.
probe/corpus.py --roots all now writes to its own file and keeps its own resume state, so a
whole-graph walk can no longer overwrite the SDG corpus that every downstream count resolves
against. It would have, silently, before today.
Devin Balkind · permalink
Added public procurement as a third category to
probe/category_gaps.py,
expecting the elections result again — no indicator, lots of city data. The first run said exactly
that: zero of 519 named indicators, 150 datasets across 28 cities.
It was wrong, and finding out why produced a better result.
SG_SCP_PROCN sits in the SDG goal tree. It is the 12.7.1 series — sustainable public procurement
— and it carries no name, so a search over indicator names cannot see it. Neither can it see
SG_SCP_PROCN_HS or SG_SCP_PROCN_LS.
We have known since Wednesday that 170 of the 689 base indicators return no name. What we had not
done is account for them when claiming a concept is absent. The category search now looks at
DCID mnemonics too, and reports them as candidates for a human to read rather than counting
them — mnemonics are noisy, and SE_ACS_ELECT is schools with access to electricity, not
elections.
All three procurement DCIDs exist. All three hold zero observations for zero countries.
The one adjacent indicator that does carry data is SG_SCP_CNTRY — countries with sustainable
consumption and production national action plans — and its value is 1 for all 75 countries
that report it. Nobody reports a 0. It is a list of countries that have a plan, recorded as a
number.
So the framework does not lack a procurement indicator. It has three, and they are empty; and the thing next to them that works counts whether a country has written a policy down.
Against that, 28 cities publish 150 procurement datasets — tenders, awards, vendors, purchase orders. New York City alone registers 55,806 contracts worth $147 billion, every one with an agency, a vendor, a value and a status.
| Category | What the framework measures | What cities publish |
|---|---|---|
| Records access | 16.10.2 — whether a country has adopted guarantees | 8 cities: requests filed, answered, how long they took |
| Procurement | 12.7.1 — three empty slots; the neighbour counts action plans | 28 cities: 150 datasets of actual transactions |
| Elections | nothing at all | 23 cities: 469 datasets of results by polling station |
Twice the framework asks has a policy been adopted and the cities answer here is what happened. Once it does not ask.
By this project’s own evidence rule — count cities, not datasets — procurement is the best of them:
One honest wrinkle: procurement datasets are not unusually far from the framework in embedding space — mean affinity 0.393 against 0.407 for municipal data generally, and only 32% fall in the bottom quartile of their own catalog, against 64% for elections. The reason is that a procurement dataset is about something — road contracts, health contracts, water contracts — and those subjects do have indicators. The data gets absorbed by its topic rather than recognised as procurement. The lexical search is what makes the gap visible; the embedding would have missed it.
Every unnamed DCID the mnemonic search surfaced across all three categories — 19 of them, including
the democratic-institutions and judiciary series SG_DMK_JDC* and SG_DMK_PARLCC_* — holds
zero observations. Unnamed and empty are the same set. That is consistent with Wednesday’s
--all sweep, where 207 of the 247 indicators outside the usable set returned no country data at
all, and it is worth stating plainly: the framework’s governance slots are largely unfilled, not
merely unnamed.
Procurement is now a third block on the demo,
and mcp/smoke.py (32 checks) asserts the empty-slot claim against the artifact, so it cannot
quietly become false if the platform fills them in.
Devin Balkind · permalink
Wiring the inverse crosswalk’s two headline categories into the demo meant checking their numbers a second way, and one of them did not survive.
The 16 Sep write-up said fifteen cities publish records-access request logs. Counted properly it is eight.
The fifteen was cluster membership — how many cities contributed a dataset to the k-means cluster whose distinctive terms were foia, request, log. That is not the same as how many cities publish a request log. The cluster had swept in Edmonton’s Media Releases and NYC’s City Hall Library Catalog, which sit near request logs in embedding space and are not request logs.
probe/category_gaps.py now counts the same categories without any clustering: search every named
SDG indicator for the category’s vocabulary, and count the municipal datasets whose titles
carry it, in six languages. Both halves are greppable.
| Cluster membership | Vocabulary count | |
|---|---|---|
| Records-access requests | 15 cities | 8 cities · 73 datasets |
| Electoral administration | 10 + 6 cities, two runs | 23 cities · 469 datasets |
Elections went up, which is the useful part of the check: clustering was splitting one category across two runs and under-counting it, while over-counting the other. A method that is wrong in both directions is not a method you can read a number off.
The argument for both gaps stands, and the elections one is stronger than before:
A cluster is not a count. k-means returns k groups whether or not k categories exist, and its membership is the nearest thing to a centroid, not the things that belong to a category. We had already made it declare its own coherence and publish 14 of 40 clusters as diffuse. This goes further: where a category can be counted lexically, the lexical count is the one that gets published, and the cluster is what found it rather than what measures it.
Both categories are now on the demo,
with the smaller, checkable numbers — and mcp/smoke.py asserts the page against
probe/category_gaps.py, so they cannot drift apart.
Devin Balkind · permalink
The asymmetry this project keeps circling: the UN graph is national-level, so a city can only be placed against countries — and for 130 of the 442 usable SDG indicators the United States reports nothing at all. On those, NYC vs the US is not a weaker comparison than NYC vs the world. It is the only one available, and the US is not in it.
Built the worksheet:
us-silent-latest ·
python3 probe/us_silent.py
The briefing has said 67 since the corpus was first screened — 130 US-silent indicators, of which the scope classifier judged 67 city-scoped. Building the worksheet showed that number is inflated, and why.
28 of the 67 have a country as their subject, not a place. “Extent to which countries have laws and regulations that guarantee full and equal access to sexual and reproductive health care” is not a quantity New York City can hold a value for; the question is whether a national legislature passed something. Eighteen of the 28 are that one family — the SDG 5.6.2 legal provisions — and the matcher paired every one of them with NYC’s Local Law 37/2011 Temporary Housing Assistance, on the token “Law”.
The scope classifier could not have caught this. It reads an indicator’s topic — health, water, education — and judges it municipal, which for 5.6.2 it is. What it cannot see is that the sentence measures a legislature. So that filter is now a lexical rule rather than a classifier: “extent to which…”, “countries that…”, “proportion of countries…” are excluded, and “extent of human made wetlands” is not. A rule you can read and argue with beats a score you cannot.
39 survive. 38 have a peer group of 15+ reporting countries, and 26 also have an NYC candidate above the noise floor.
Twelve of the 39 form one coherent group, and it is the group the briefing predicted:
| Countries | Indicator | Years | NYC candidate |
|---|---|---|---|
| 133 | Municipal waste collected | 2000–2023 | Recycling Diversion and Capture Rates (0.63) |
| 107 | Hazardous waste generated per unit of GDP | 2000–2024 | Wastewater Co-digestion and Biogas (0.53) |
| 97 | Hazardous waste exported | 2000–2024 | DSNY Disposal Sites Used by Facilities (0.52) |
| 97 | Total waste generation | 2000–2023 | DSNY Waste Characterization (0.52) |
| 94 | Total wastewater treated | 2022 only | Watershed Water Quality – Wastewater (0.54) |
| 93 | Hazardous waste treated or disposed | 2000–2024 | Dewatered Solids and Biosolids (0.51) |
| 88 | Proportion of hazardous waste treated or disposed | 2000–2024 | Dewatered Solids and Biosolids (0.51) |
| 85 | Hazardous waste imported | 2000–2024 | DSNY Disposal Sites Used by Facilities (0.49) |
| 84 | Groundwater bodies with good ambient water quality | 2017–2023 | Watershed Water Quality – Limnology (0.52) |
| 84 | Total wastewater generated | 2022 only | Wastewater Co-digestion and Biogas (0.59) |
| 72 | Proportion of wastewater treated | 2022 only | Dewatered Solids and Biosolids (0.52) |
| 55 | Extent of human made wetlands | 2021–2025 | NYC Wetlands (0.50) |
These are DSNY and DEP series. NYC genuinely publishes them, and 133 countries report municipal waste collected while the United States reports none of it. That is a comparison NYC can make and its own country cannot.
Three of the twelve carry one year. That supports a level comparison and not a trend — the same finding that removed road safety as our headline demo, and the worksheet marks it rather than letting a reviewer discover it after choosing the indicator.
Two results from the last two days point at the same set from opposite directions.
The corpus smell test found that sweeping all 689 indicators rather than the 442 usable ones adds almost nothing — but the 40 that do hold data are almost entirely e-waste and municipal waste import/export.
The inverse crosswalk found e-waste and hazardous waste per capita among the 123 indicators no municipal dataset comes near — nobody publishes them at city level either.
So this band of the framework is thinly reported by everyone: the US skips it, cities mostly skip it, and it is exactly where a city that does publish has something unusual to say.
screen.py decides does the US report this from entityCoverage on a six-country panel — a
proxy we have flagged as a proxy. Checked all 39 against the actual country observations:
not one has a single United States datapoint. The proxy and the data agree completely.
That is the second independent confirmation of that panel this week; yesterday’s --all sweep
showed its NO-DATA grade predicts global emptiness 84% of the time.
graded: false. Nothing here is a mapping until a person says it is.The platform went fully public this morning. python3 probe/launch_diff.py: 57 series, 12
variables and 689 corpus indicators, zero drift against yesterday’s pre-launch baseline, and
mcp/smoke.py 8/8. No public hostname resolves yet — undatacommons.unicc.biz and
datacommons.un.org both still refuse — and the deployment we have always used is still answering.
The demo is safe.
Devin Balkind · permalink
Ran the inverse crosswalk over the
non-English catalogs: 5,956 datasets from 11 cities in Italian, Portuguese, Spanish, German and
Croatian, pooled into one run under the multilingual model.
Results ·
python3 probe/inverse.py --language non-en
Pooled rather than run per language, because cosine similarities from two different models are not comparable and two-to-four cities per language could not support an analysis whose unit of evidence is how many cities. Pooled, they share one space and eleven cities.
The tightest cluster in anything this project has produced: 241 datasets across 6 cities, coherence 0.90 — Milan’s Elezioni Politiche: Risultati di Sezione, Madrid’s Elecciones Autonómicas, polling-station-level results going back to 1996.
And the English run found the same thing separately: election results by voting station, 10 cities, coherence 0.71 — Calgary, Edmonton, and others.
Two disjoint sets of cities, two different embedding models, one category. The SDG framework has no indicator for electoral administration at all; the nearest thing the matcher can find for the English cluster is “number of local governments”. Turnout, results, polling-station coverage — cities publish all of it and the framework has no word for any of it.
COVID-19 case reporting corroborates the same way: 6 English cities and 7 non-English cities, independently. The nearest indicators the matcher offers are “new HIV infections per 1,000 uninfected population” and “number of total conflict-related deaths”. There is no city-level pandemic-surveillance indicator to match.
| Cities | Datasets | Theme |
|---|---|---|
| 5 | 113 | Property market valuations — Italy’s OMI quotazioni immobiliari, sale and rental prices by zone, half-year by half-year since 2004 |
| 6 | 69 | Weights-and-measures inspection — Brazil’s Ipem verifications of pre-measured goods |
| 5 | 33 | Public-employee working arrangements — Milan’s personale a tempo indeterminato in telelavoro, permanent staff on telework, by year |
The property-valuation series is the interesting one. Housing affordability is squarely an SDG 11 concern (11.1.1 is inadequate housing), but the framework counts people in inadequate housing and the Italian cities publish what property costs, by neighbourhood, twice a year, for twenty years. Those are not the same measurement and only one of them has an indicator.
11 of 11 passed, against 8 of 9 in English. These are hand-built: no city outside NYC has a
graded worksheet, so the controls are datasets whose titles plainly correspond to an SDG concept,
chosen across all five languages and recorded in probe/inverse_controls.json.
The two weakest are both Portuguese — Fortaleza’s Número de Óbitos por Acidentes de Trânsito at the 44th percentile and Recife’s Acidentes de Trânsito com Vítimas at the 32nd. Those are about as unambiguous as a match gets (deaths from traffic accidents ↔ SDG 3.6.1) and they sit in the bottom half. Portuguese retrieval is the weakest link, so the Brazilian portion of the tail is the least trustworthy part of this run, and anything resting on Fortaleza, Recife, Belo Horizonte or São Paulo alone should be read with that in mind.
Both runs improved when I stopped matching against indicators that have no name.
170 of the 689 base indicators carry no name in any source we hold — they return neither
metadata nor observations. The first version fell back to the DCID mnemonic, so a quarter of “the
framework” was represented by query strings like DI ILL OUT. Nothing can match that, which
inflates the tail — datasets look further from the framework than they are — and it put mnemonics
in the nearest-indicator column, where VC VAW SXVLN was being reported as a cluster’s closest
SDG concept.
Excluding them moved every weak control up: Fortaleza from the 28th percentile to the 44th, Recife from the 27th to the 32nd. The framework is now represented by its 519 named indicators, and the count is stated on the page rather than folded into a total.
Two smaller fixes in the same pass. Cluster examples were being spread one-per-city to show reach, which made them unrepresentative of clusters dominated by one city — a cluster labelled for Italian property valuations was illustrated with a Madrid shop register. Each cluster now shows its most central members as well as the city spread. And the phrase view returns nothing here, which the page now says is expected by construction: Spanish, Italian, Portuguese, German and Croatian titles share essentially no bigrams.
| English | Non-English | |
|---|---|---|
| Datasets / cities | 11,206 / 37 | 5,956 / 11 |
| Controls passing | 8 of 9 | 11 of 11 |
| Coherent clusters | 23 of 40 | 8 of 20 |
| Indicators nearest to nothing | 69 of 519 | 183 of 519 |
The non-English run has better controls and worse coverage — 183 indicators are nearest to no dataset at all, against 69 in English, which is what you would expect from half the datasets and a weaker cross-lingual model. It is not evidence that those cities publish less.
Devin Balkind · permalink
Every analysis in this repo has run city → UN: take an SDG indicator, find the municipal dataset that matches it. That direction can only ever discover what the framework already asks about. So: run it backwards. Take 11,206 datasets from 37 city portals, find each one’s nearest neighbour among all 689 SDG indicators, and look at what is left over.
The question is not which cities are behind. It is what would SDG 11 look like if it had been written from municipal data upward?
Full results ·
python3 probe/inverse.py
Themes that recur across many independent city catalogs and sit in the bottom quartile of every one of them:
| Cities | Theme |
|---|---|
| 23 | Building permits, licences and code enforcement |
| 15 | Records-access request logs (FOIA and equivalents) |
| 13 | Bike parking, bike share, active-travel space |
| 10 | Street sweeping schedules and parking enforcement |
| 10 | Call-centre response performance — answer times, 311 service levels |
| 9 | Pedestrian and bicycle counts |
| 8 | Property sales and assessment |
| 7 | For-hire vehicle and taxi trip records |
| 7 | Fire stations and emergency facilities |
| 6 | Special events permitting |
The sharpest one is records access. Fifteen cities publish request logs — how many were filed, how many answered, how long it took. The nearest SDG indicator the matcher can find is 16.10.2, “countries that adopt and implement constitutional, statutory and/or policy guarantees for public access to information.” The framework asks whether a law exists. The cities publish whether the law works. A country can score full marks on 16.10.2 and answer nothing, and no indicator in the framework would notice.
The same shape appears in call-centre performance: the SDGs have no concept of how quickly a government responds to its residents, and ten cities publish exactly that, in seconds.
123 of the 689 indicators were the nearest neighbour of no municipal dataset at all. Many are honestly national — ODA flows, external debt, climate finance, which no city could report. But the list also contains e-waste collected, generated and recycled per capita, and hazardous waste generated — which is a double gap. Those are the same indicators we found the United States does not report, and it turns out cities do not publish them either. Nobody is measuring them at any level of government.
One positive control in seven failed. NYC’s Housing Maintenance Code Violations is a hand-verified match for SDG 11.1.1 inadequate housing, and it landed at the 24th percentile — inside the tail. So the tail contains real matches, and any individual dataset in it may simply have been missed. That is why the unit of evidence above is cities, not datasets: one dataset scoring low is retrieval failure, and twenty cities independently publishing the same category is not.
Fourteen of forty clusters are noise, and the report says which. k-means returns k clusters whether or not k themes exist; each one now carries the mean cosine of its members to its own centroid, and below 0.62 it is published as diffuse rather than read as a theme. The first run had no such measure and cheerfully labelled a cluster “school” that contained building violations and lobbyist registrations.
English-language portals only. Comparing cosine similarities across two embedding models is meaningless, so the multilingual catalogs need their own run.
The first run’s clusters were incoherent, and the reason was not the clustering.
embed.Index decided whether cached vectors could be reused by checking len(cached_ids) ==
len(datasets). Re-fetch a portal months later and it returns the same datasets in a different
order — measured: 24 of 45 city catalogs did exactly that. Same count, same IDs, different
sequence. The length check accepted it, and every dataset was handed another dataset’s vector.
There is no symptom. Similarity scores stay in range, rankings look ordinary, nothing errors. It surfaced only because a cluster labelled “school” was full of facade-compliance filings and I looked at the rows.
It now matches on IDs, and when the set is identical but re-ordered it permutes the cached vectors instead of re-embedding. Worth noting what this didn’t affect: the published municipal table was computed on the run that created its cache, in the order it created it, so those numbers were right. The bug only bites on a later re-run — which is to say, it was waiting for the next person to re-render the table.
That is the eighth entry in this project’s list of automated checks that produced a confident wrong answer, and the fourth caught by reading rows rather than totals.
Devin Balkind · permalink
The smell test ran against
the 442 indicators screened usable. That left 247 graded NO-DATA or THIN-COVERAGE — and
the grade was assigned against a six-country panel, so it only ever meant “no data for those
six”, not “no data anywhere”. Ran it with --all to find out.
689 indicators · 773,335 observations · 2,503 findings.
The extra 247 indicators contributed 5,056 observations — 0.65% of the corpus — and 207 of the 247 returned no country-level data at all. A grade assigned on six countries predicted global emptiness correctly 84% of the time.
That closes a question I had raised against our own method. The six-country panel was chosen for speed, with the honest caveat that it could not distinguish “the panel does not report this” from “nobody does”. It turns out to distinguish them almost perfectly, and 442 remains the right denominator for everything downstream.
The 40 indicators that do hold data are worth knowing about, because of what they are: e-waste collected, recycled and its proportions; municipal waste imported and exported. Those sit exactly in the band we identified months ago as the interesting one — indicators a city can report internationally precisely because its own country skips them.
The previous sweep found Guadeloupe’s e-waste collection multiplying by a thousand in 2022. With the full corpus it turns out the same error propagates through every derived indicator:
| Indicator | Unit | 2021 | 2022 |
|---|---|---|---|
| Electronic waste collected per capita | kg | 13.71 | 13,950 |
| Electronic waste recycled per capita | kg | 13.71 | 13,950 |
| Total electronic waste collected | tonnes | 5,472 | 5,367,000 |
| Total electronic waste recycled | tonnes | 5,472 | 5,367,000 |
The largest total any other country has ever recorded for e-waste recycled is 899,300 tonnes. So Guadeloupe — population about 380,000 — is published as the world’s biggest e-waste recycler, by a factor of six. One unit slip in a 2022 submission, carried into four indicators.
This is also the first time the checks corroborated each other: the same anomaly surfacing in four related series is much stronger evidence than the same anomaly surfacing once.
The new coverage brought a pattern rather than a single error. Municipal waste exported and imported, both flatlined at exactly zero for very long runs: Mauritius 23 years, Dominica 22, Singapore 21 and 22, Cuba 20, Palestinian Territories 19, Jamaica 18, Saint Lucia 16, Liechtenstein 24.
Some of that is surely true — a small island genuinely exports no municipal waste. But a twenty-year run of exact zeros is also what “no submission” looks like when it is stored as a number instead of a gap, and the two are indistinguishable from outside. Worth one question to the platform team: is zero here a measurement or a default? Our own rule is that a missing year stays missing and is never interpolated; the same distinction matters just as much in the other direction.
Alongside it, “Proportion of electronic waste that is collected” sits at exactly 100% for Niger for eight consecutive years and Iran for six.
The finding counts barely moved — 2,457 to 2,503, with one new HIGH — which is the correct result for adding 0.65% more data. The five verified errors from the earlier sweep stand unchanged.
Devin · permalink
Most SDG indicators are rates per 100,000, so every non-US city needs a population figure before it can produce a chart. Two are now wired, and the route we did not take is the more useful finding.
urb_cpop1 covers 969 European cities annually and is exactly the single source this needs. It
reports greater cities, not municipalities:
| City | Eurostat “greater city” | Municipality | If used |
|---|---|---|---|
| Madrid | 5,115,272 | 3,520,396 | rates ~31% too low |
| Milan | 3,580,530 | 1,399,079 | rates ~60% too low |
For Madrid the table contains no municipality entry at all — the greater city is the only option. A city’s open data covers its municipality, so this is the denominator mismatch the whole project exists to catch, and it would have been completely invisible in the output.
Better provenance as well: numerator and denominator then share a publisher.
Madrid — Padrón municipal (200076-0-padron), summing the four Spanish/foreign ×
male/female columns of a 34 MB CSV: 3,520,396 as of 2026-09-01. A snapshot, so Madrid supports
levels and not trends until the historic padrón is wired.
Milan — Popolazione calcolata (ds1494), year-end population 1880–2025, ISTAT to 2002
then the city’s own anagrafe: 1,399,079 in 2025. A series, so Milan supports trends — the
first non-US city that does.
Two cities, two shapes, two resolvers. City statistical publications do not share a format, and pretending otherwise is how you end up with Eurostat.
Building a summary table showed Chicago and Boston both reporting 8,478,072 — New York’s population.
The census resolver fell through to NYC’s place code whenever a city declared none of its own.
Chicago’s registry entry named its place code in a prose source string but carried no structured
field, so it silently inherited NYC’s 8.5 million. Every Chicago rate would have been three
times too low, and nothing in the output would have looked wrong.
There is now no fallback. A city using the census module without explicit state and place FIPS raises, with the reason: refusing to guess, a wrong denominator is invisible in the output.
That is the fifth automated path on this project that produced a confident wrong answer, and the second where the failure mode was a silent default rather than an error.
| City | 2023 population | Source |
|---|---|---|
| New York | 8,258,035 | Census ACS |
| Madrid | 3,520,396 | Padrón municipal |
| Chicago | 2,664,454 | Census ACS |
| Milan | 1,417,597 | Popolazione calcolata |
| San José | 969,615 | Census ACS |
| Boston | 652,442 | Census ACS |
Plus 23 more US cities resolvable by ACS place code. Buenos Aires declares no source and is refused rather than estimated. 24 non-US cities still need one, and each needs a person to find its statistical publication — the part that does not automate.
Devin Balkind · permalink
Malaysia reports recycling 147.7% of its municipal waste. That figure is in authoritative UN data, it is on our demo, and we found it by accident while building a chart. This is the obvious follow-up, run properly: 442 indicators, 768,279 observations, every reporting country and every year, against checks dumb enough that they need no subject-matter knowledge — a percentage above 100, a negative count, a rate exceeding its own denominator, a value repeated across countries that should not agree.
Full results ·
python3 probe/smell.py
The tool flags; it does not judge. Everything below that I call an error, I checked by hand against the indicator’s own distribution first.
1. Kyrgyzstan feels 6,990% safe walking home. SDG 16.1.4, proportion of population that feel
safe walking alone around their local area after dark, unit Percent:
| Kyrgyzstan | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 |
|---|---|---|---|---|---|---|
| 57.9 | 64.35 | 66.8 | 6710 | 6840 | 6990 |
The other 252 observations, across 55 countries, run from 22.8 to 95.0, median 72.0. Divide the last three by 100 and you get 67.1, 68.4, 69.9 — which continues Kyrgyzstan’s own trend from 66.8 exactly. Live for three consecutive years.
2. South Africa recycles 1.86 billion tonnes of municipal waste. Global municipal solid waste generation is roughly 2 billion tonnes a year, worldwide. South Africa’s own earlier figures are 260,600 t (2005) and 520,800 t (2006); from 2018 they are 1.0–3.4 billion. Read as kilograms they are entirely sensible.
3. Brunei generates 12,580 tonnes of hazardous waste per person. EN_HAZ_PCAP, unit
kilograms, where the global median is 22 kg and the maximum outside Brunei is 212. Brunei reports
8.7–36 million kg per capita, every year from 2016 to 2023. Against a population of ~450,000
that is 5.7 billion tonnes. It looks like a national total sitting in a per-capita field.
4. Guadeloupe’s e-waste collection multiplies by a thousand in one year. 8.46, 8.54, 10.03, 10.36, 10.58, 11.97, 13.10, 13.71 — then 13,950 in 2022. The cleanest of the five: the series resumes perfectly if you divide the last point by 1,000.
5. Malawi is paid to send remittances. SI_RMT_COST, the average cost of sending $200, as a
percentage. Malawi runs 16.95, 15.82, 14.47, 16.26, 14.79, 13.13 — then −0.1 and −0.93 —
then 31.48. A cost of minus one percent, for two years, between normal values.
The United States recorded 345,600 disaster deaths in 2020 and 470,600 in 2021. Every other
country in VC_DSR_MORT has a median of 42. Those are COVID-19 deaths: the US classified the
pandemic as a disaster and reported it here. Most countries did not.
That is not a data error. It is the exact failure this whole project exists to prevent — two numbers that share a variable, a unit and an axis, and are not the same measurement. A chart of “disaster deaths, US vs peers” would be perfectly well-formed and would tell you something false. It is the strongest argument yet for the comparability grade, and we found it with a check that knows nothing about disasters.
Each of these I chased and dropped, which is most of what the afternoon consisted of:
Percent unit covers both bounded proportions and signed rates, and no
unit string distinguishes them.The first full sweep returned 41,350 findings — 5.4% of all observations, which is not a result, it is a broken instrument. Three false-positive classes, each caught by reading rows rather than totals:
| Check said | Actually |
|---|---|
| 351 jumps in one indicator (12% of its rows) | the Indicator of Food Price Anomalies is a signed index centred on zero; a ratio means nothing on it |
| 23,172 negative percentages | growth rates and balances are signed by construction |
| four scaled rates treated as plain counts | my substring match on "COUNT" swallowed RATIO_COUNT_PER_100_COUNT_POP, so its cap never applied |
The first fix over-corrected: it gated the jump check on a unit whitelist, and thereby discarded
Mauritius’ food waste going 207 tonnes → 177,570 tonnes in a year because WEIGHT_TN was not
on the list. Gating on the shape of the quantity — does this indicator ever go negative
anywhere? — keeps it.
What came out of that is a principle worth more than any individual finding: an indicator is its
own control group. A rule broken by most of an indicator’s observations is its definition. A
rule broken by three country-years out of three thousand is Kyrgyzstan. So checks are now
suppressed per-indicator when they fire often enough to be structural — 165 suppressed groups,
38,943 would-be findings — and the suppressions are published, because the list of indicators
that publish signed values under a Percent unit is itself a finding about the graph’s
vocabulary.
That left an outlier check that works against each indicator’s own distribution rather than
against what a unit is supposed to mean: 20× the indicator’s 99th percentile. It survives
suppression, which matters — “proportion of hazardous waste treated” exceeds 100% for 37
countries, so its range check is structural and gets dropped, and Guatemala’s 44,825% would
have gone with it. It is also the only check that reaches the 123 indicators and 248,457
observations whose units (CR_USD, WEIGHT_TN, INDEX, SCORE) have no meaningful range at
all. The report states that coverage gap on its own face.
Final count: 2,457 findings, 92 of them HIGH. Small enough to read.
Two things. It is a QA gate on the data our own crosswalk depends on — and it is the kind of thing only an outsider runs, because it requires no authority and no access, just the published graph and a willingness to look at 768,279 numbers. The five errors above are reportable to the platform team as they stand.
Devin · permalink
Boston’s Vision Zero Fatality Records ranked 23rd of 235 for “death rate due to road traffic injuries”, below a contract-award file. It now ranks 1st. NYC’s ground-truth median improved from 23 to 14 at the same time.
First instinct was query expansion — add “crashes, collisions, fatalities, vision zero” to the indicator name. It made things worse (rank 23 → 30). So did rephrasing: “traffic fatalities” scored worse (32) than the clinical UN wording (23).
The problem was the document, not the query. Vision Zero Fatality Records carries excellent metadata:
title: Vision Zero Fatality Records
tags: accidents bikes cars crashes fatalities pedestrians safety streets traffic vision zero
desc: 1,500 characters of programme mission statement — "our commitment to focus
the city's resources on proven strategies… we are inspired by…"
A static embedding averages over every token. Sixty characters of exactly-right title and tags were being drowned by 1,500 characters of boilerplate. The dataset that beat it, My Neighborhood Dataset, won partly by having a shorter description.
Two vectors per dataset instead of one concatenated blob:
Score is max(head·q, body·q), so a dataset surfaces on whichever field actually carries its
signal. Programme-named datasets are found by their tags; thinly-titled ones by their prose.
Measured both ways before adopting it:
| Scheme | NYC median rank | top-10 | top-50 | Boston: Vision Zero |
|---|---|---|---|---|
| concatenated (before) | 23 | 3/7 | 5/7 | #23 |
| head only | 57 | 3/7 | 3/7 | #1 |
| max(head, body) | 14 | 3/7 | 5/7 | #1 |
Head-only fixes Boston and wrecks NYC — PM2.5 falls from rank 30 to 397, because NYC’s air-quality dataset is found through its description. Taking the max gets both.
Boston now returns Vision Zero Fatality Records (0.544) and Vision Zero Crash Records (0.541) as the top two for road deaths.
Vector caches were keyed by city and model but not by the text representation. Changing how documents are built silently reused vectors from the old scheme — wrong rankings, no symptom, no error. Some of the earlier Boston numbers were affected by exactly this.
The cache key now includes a REPR_VERSION that must be bumped when the representation changes.
Fourth entry in this project’s running list of automated checks that were confidently wrong.
This improves retrieval; it does not make the matcher trustworthy. The cities now surface their obvious datasets, and a person still has to decide whether Vision Zero Fatality Records is comparable to SDG 3.6.1 — which, per the spec, is the one thing a machine may not do.
Devin · permalink
The briefing and the last two posts carried this line:
No substantive SDG indicator has matched outside NYC.
True of our matcher. Read as a statement about the cities, it is wrong, and it was phrased so it would be read that way. Correcting it.
Searching Chicago’s and Boston’s catalogs directly, by hand, for the four indicators NYC is crosswalked on:
| Chicago (915 datasets) | Boston (235) | |
|---|---|---|
| Crime / homicide | Crimes — 2001 to Present | Shootings · Homicide Clearance Rate |
| Road deaths | Traffic Crashes — Crashes / People | Vision Zero Fatality Records |
| Air quality | Open Air Chicago (measurements, hourly, daily) | only building emissions (BERDO) |
| Waste | Christmas tree recycling; no municipal tonnage | Trash schedules, not tonnage |
Chicago’s Traffic Crashes — Crashes is structurally the same thing as the NYC dataset behind our road-deaths card. Boston publishes a dataset literally called Vision Zero Fatality Records — the same programme, the same concept, named almost identically to what we used for NYC.
Querying Boston for “Death rate due to road traffic injuries”:
0.486 Traffic-Related Data
0.467 City of Boston Contract Award
0.444 My Neighborhood Dataset
0.387 Trash Collection Days
...
#23 Vision Zero Fatality Records (0.286)
The correct dataset ranked 23rd of 235, below a contract-award file. No threshold rescues that; it is a retrieval failure, not a calibration one.
The matcher is not uniformly blind — querying homicide returns Homicide Clearance Rate at rank 1, because the word “homicide” appears in the title. It works when the vocabulary happens to line up and fails when a city names the same concept differently, which cities routinely do. “Vision Zero” is a programme name, not a description of its contents.
This is the third time on this project that an automated check produced a confident answer a person overturns by reading for five minutes: the chartable flag that ignored human grades, the screen that reported 689 having recorded 9, and now a matcher whose silence was mistaken for absence.
A null result from a tool you have measured at ~50% precision is not evidence of absence. We wrote that lesson into the spec as R1 and then made the mistake anyway, in our own prose, about our own tool.
Devin Balkind · permalink
The platform goes fully public tomorrow, 17 Sep. Every figure this project publishes was measured against the pre-launch deployment, and the open question in the briefing was blunt: will staging DCIDs survive the launch? A withdrawn DCID does not raise an error. It returns an empty result, which is the same shape as a country that does not report — so the demo would show a blank chart to a room at Google NY and nothing would look broken until someone asked.
So: re-ran every probe against the live deployment and diffed it against 14 Sep, and built
probe/launch_diff.py so that tomorrow’s version of this is one command rather than an afternoon.
| What | 14 Sep | 16 Sep |
|---|---|---|
| SDG base indicators enumerated | 689 | 689 |
| Variant DCIDs beneath them | 6,025 | 6,025 |
| Indicators screened, every field compared | 689 | 689, zero changed |
| Crosswalk series (variable × place), value by value | — | 57 checked, zero changed |
| Crosswalk variables, via metadata | — | 12 checked, zero changed |
Demo figures (mcp/smoke.py) |
8/8 | 8/8 |
Not one DCID added, not one removed, not one value revised, not one unit changed. The three
EMPTY series are the ones we already knew about — the UN holds no municipal-waste observations
for the US, Japan or Mexico, which is the finding the municipal-waste card is built on.
The only differences anywhere in the crosswalk were ours, not the platform’s: two figures in NYC’s housing dataset moved by one (297,753 → 297,754 — NYC Open Data is live), the road-deaths pair was regraded BLOCKED → RANK-ONLY by hand yesterday, and the municipal-waste pair is new since the 14th.
The coverage probe grades whatever search_indicators returns for fifteen topic queries. Same
code, same queries, same host, two days apart:
The one worth noting: searching “road traffic deaths” on 16 Sep returns
undata/sdg/SH_STA_TRAF — the exact DCID our road-deaths pair uses — and on the 14th it did not.
Recall improved. That is good news and a warning in the same breath: the graph is frozen, the
retrieval over it is not. Anything that resolves a DCID by searching at runtime can change its
answer without the data changing at all. Our crosswalk hardcodes DCIDs that a human resolved once,
which is exactly why the demo is insulated from this. It was a chore to do it that way. It is not
a chore any more.
I checked the obvious alternative explanation before writing this down: search_indicators
returns byte-identical results across three consecutive calls today, and neither
coverage_probe.py nor candidates.json has been touched since the initial commit.
The first run of launch_diff.py reported, confidently, that all twelve crosswalk DCIDs were
missing from the graph — while get_observations was returning data for every one of them.
get_variable_metadata requires entity_dcids. Omit it and the server answers 200 with a
completely empty body — no error, no status field, no variables key at all. A caller that
trusts the response shape reads that as “every variable has been withdrawn”. Two days before the
launch it was written to detect, the launch detector was ready to cry wolf about the entire
crosswalk.
That is the seventh entry in this project’s list, and it was caught the same way as four of the other six: by printing a table of all twelve rows and reading it, rather than trusting a summary count. A checker that reports catastrophe is at least loud. The version of this bug that would have actually hurt is the mirror image — a diff that stays quiet because it cannot see.
So launch_diff.py --self-test now injects each drift class into a copy of the baseline and
asserts the diff reports it: a revised value, a series going empty, a series coming back, a
changed unit, a shortened span, a renamed variable, a withdrawn variable, an indicator dropping
out of the corpus, a tool disappearing from the server. Plus the converse — an unmutated copy must
report nothing. 10/10. A diff that returns “no change” and a diff that cannot see change are
indistinguishable from the outside, and this project has shipped that mistake often enough to stop
paying for it twice.
The baseline is recorded as of today, pre-launch. After the platform goes public:
python3 probe/launch_diff.py # exits non-zero if anything moved
python3 mcp/smoke.py # the demo's published figures
If the public deployment answers on a different host, UNDC_ENDPOINT and UNDC_REST override it
without a code edit, and the diff reports the host change as drift in its own right. As of this
afternoon no public hostname resolves yet — undatacommons.unicc.biz and datacommons.un.org
both refuse to connect, and the deployment we have been building against is still answering
normally.
Devin · permalink
Two corrections and one fix. The corrections are the useful part.
We reported that no global registry of CKAN portals survives — that every canonical one had rotted. That was wrong.
The CKAN Ecosystem Catalog is a 2025 NSF POSE II project from
the CKAN core team, WPRDC and datHere. It lists 199 instances, 97 of them local or regional
government, machine-readable at
ckan/ckan-instances.
We missed it because we searched for the registries we already knew about — ckan.org/instances,
dataportals.org, opendatainception — found all three dead, and concluded the category was dead.
The correct inference was that we were looking at the previous generation. (The site sits behind
Cloudflare and refuses both curl and a headless browser; the GitHub repo is the way in.)
The abandoned OKFN list really has rotted — 39 of 631 answer — which is what made the wrong conclusion feel supported.
Inventory now: 372 portals surveyed, 70 municipal, 543,000 datasets.
And a related correction: a portal that does not answer an anonymous package_search is not
dead. data.gov, govdata.de and data.overheid.nl all refuse the probe and are plainly alive.
The inventory now reports what responds to one specific API call, which is a much weaker claim
than liveness.
An absolute similarity cutoff cannot travel between catalogs. Measured on hand-judged matches, good pairs span 0.42–0.70 in NYC, 0.50–0.55 in Chicago, 0.37–0.59 in Madrid. One number is simultaneously too strict and too loose.
The first fix was wrong, and instructively so. We calibrated a z-score — how many standard deviations the top match sits above the catalog’s own distribution — measured F1 0.68 against a labelled set, wired it in, and it admitted 60 of 80 indicators. Two independent errors:
What actually varies between catalogs — and is the right thing to normalise — is the distribution of top-1 scores across all probed indicators. The cutoff is now the best 20% of that, with an absolute floor so a uniformly hopeless catalog cannot contribute its least-bad rows anyway.
| City | Cutoff | Candidates |
|---|---|---|
| Milan | 0.450 | 12 |
| Boston | 0.457 | 16 |
| Chicago | 0.473 | 16 |
| Madrid | 0.515 | 16 |
Bounded worksheets that adapt to each catalog, instead of 0 or 69.
No substantive SDG indicator has matched outside NYC. Better calibration produces a better worksheet; it has not produced a good mapping. NYC’s pairs were found by a person who knew the data, with the matcher confirming them, and nothing yet shows the matcher can lead.
That is worth saying plainly a week out from Builders’ Day: the infrastructure generalises, the judgment does not, and the judgment is the part that matters.
Devin · permalink
The blocker for every non-anglophone city was the embedding model, not the pipeline. Adding
potion-multilingual-128M unblocks them — and measuring it showed the fix is not free.
| City | Lang | Before | After | Best sim |
|---|---|---|---|---|
| Madrid | es | 0 | 19 | 0.59 |
| Milan | it | 0 | 5 | 0.67 |
On the seven hand-verified NYC pairs, against the full 2,400-dataset catalog:
| Model | Median rank | top-10 | top-50 |
|---|---|---|---|
potion-base-32M (English) |
23 | 3/7 | 5/7 |
potion-multilingual-128M |
81 | 3/7 | 3/7 |
Swapping wholesale would have bought non-English coverage with English accuracy — a 3.5× worse median rank on the catalog we know best. Interestingly it is not uniform: the multilingual model improves the two hardest cases (homicide 786→401, slums 1033→716) while degrading the easy ones.
So the model is chosen per catalog language. English cities keep the English model; everything else gets the multilingual one. The vector cache is keyed by model, because vectors from two models are not interchangeable and reusing them across would be a silent, invisible corruption.
Milan is good — 4 of 5 plausible:
Madrid is noisy — six near-identical “water area of lakes and rivers” indicators all matched Fuentes de agua para mascotas, drinking fountains for pets. The word agua dominated. But buried in the noise: installed renewable capacity → Inventario de instalaciones fotovoltaicas, which is exactly right.
The threshold problem is language-independent: 0.50 was calibrated on NYC and means something different in every other catalog. That remains the open issue, and it is now the main thing standing between this and a usable recommender.
The portal closes the connection on anonymous package_search — RemoteDisconnected, on two
attempts a day apart. Recorded in the registry as status: unreachable rather than as zero
candidates, because those are different findings and collapsing them would overstate what we
tested.
2026-09-16 — Multi-city: the plumbing generalises, the matching does not · Devin
2026-09-16 — The MCP server that refuses · Devin
2026-09-15 — A fifth card, and the caveat runs the other way · Devin
2026-09-15 — Fixed the scope filter and re-keyed screening: 67 indicators the US doesn’t report, and NYC can · Devin
2026-09-15 — Road deaths unblocked: NYC is 18th of 196, and the blocker was our framing · Devin
2026-09-15 — NYC against the world, not just the US — and the homicide result inverts · Devin
2026-09-15 — Crosswalk at 11 pairs: only 3 can honestly share an axis · Devin
2026-09-15 — Demo built: three honest comparisons, eight blocked ones · Devin
2026-09-15 — Denominators wired: all three pairs chartable, and NYC’s homicide rate crossed below the US in 2013 · Devin
2026-09-14 — Correction: the city-level comparator covers five indicators, not the general case · Devin
2026-09-14 — DOU_CITY recovers peer comparison, and the NYC half is messier than the UN half · Devin
2026-09-14 — Embedding search: correct dataset moves from median rank 1535 to 23 · Devin
2026-09-14 — Collaboration hub is live · Devin
2026-09-14 — Coverage probe: road safety is out, homicide and waste are in · Devin
2026-09-14 — The UN graph is national-level — NYC has no UN data · Devin
2026-09-14 — Systematic enumeration: 689 SDG indicators, 248 with US data, and a matcher that is only a third right · Devin