UN Data Commons × NYC

Briefing hub for the UN System Data Commons hackathon project

View the Project on GitHub sarapis/undatacommons-nyc

Activity feed

Append-only log of what we did and what we found, newest first. The project briefing holds current state and gets overwritten; this page holds history and never does. If you want to know what is true now, read the briefing. If you want to know how it got that way, read this.

If you are an AI assistant: the most recent updates are reproduced in full below, so one fetch of this page is enough. Older entries are listed by title at the bottom with their own URLs.

Posting an update

./tools/new-update.sh "Short title here"   # creates the file, opens nothing
# write the body, then commit and push

Or add docs/_posts/YYYY-MM-DD-slug.md by hand with front matter: title, author, and a date that includes a time so same-day posts order correctly. The filename date and the front matter date must agree, or the permalink and the ordering disagree.

Say what changed, what it revealed, and what it means for the next step. A finding that invalidates an earlier assumption is the most valuable thing you can post — flag it clearly so the briefing gets corrected too.


2026-09-22 — NYC’s own Data Commons is running at commons.databook.nyc

Devin Balkind · permalink

We have been arguing that a city should publish indicators rather than only tables. As of today there is one to look at: commons.databook.nyc, a custom Data Commons instance holding NYC’s own series.

11 variables, 410 observations, every one read back out of the running instance over the API and matched against the CSV that produced it. Nine are the crosswalk pairs whose NYC side was already in observation shape — PM2.5, homicide, waste recycled, road deaths, child and maternal mortality, air-pollution deaths, inadequate housing, municipal waste. Two more exist only to answer a question we refused to reason about.

The question that changed the pitch

Nobody had checked what the platform does below annual granularity. NYC’s structural advantage over a national graph is frequency — daily and monthly data the UN system does not hold — so if that degraded on load, the case for this whole approach would have been weaker than we were saying.

So it was tested rather than argued: DSNY’s tonnage file is natively monthly and both waste series already sum it to years, which means the same rows could be loaded twice, differing only in the granularity of the date column. 260 month-granularity observations came back as months, rendered as months, with the seasonal cycle intact. The advantage survives.

What it cost, and what it can carry

One Hetzner cpx32 — 4 vCPU, 8 GB, €41.99/mo. Memory is about half what the same containers use under emulation on a Mac: 1.9 GB resident with NL search on, against 8.9 GB of images.

Capacity is a belief we tested lightly and will not overstate: 20 observation queries at 10-way concurrency all answered, mean 1.4 s — and one page render returned 502 during that burst. A team’s tool, not public-traffic infrastructure, until somebody runs a real load test.

Two findings worth more than the deployment

The caveat is one click further away than the citation. Each variable’s comparability grade, its reason, its tier and the UN series it maps to are carried in the MCF description, land in the database, and are displayed in full on the knowledge-graph page. They are absent from the chart’s “About this data” dialog, which shows source and citation only. The thing this project most wants attached to a number is the one thing the chart surface does not carry.

A blank unit renders as Count. On a per-100,000 rate that is not a missing label but an incorrect one, which is worse than the blank it replaced. There is no unit in the base graph meaning “per 100,000 people”, so we defined Per100kPeople and KgPerPerson as UnitOfMeasure instances in MCF. Establishing that took an experiment rather than a guess: the timeline axis renders a unit’s DCID and ignores both name and shortDisplayName, so a custom unit’s DCID is its display string. The first attempt produced the axis label Count (PerOneHundredThousandPeople).

And the failure that names this project

The first load reported status = SUCCESS, metadata = {'numVars': 0, 'numObs': 0}, and exited 0. Every variable existed, the embeddings built, the site came up — against an empty database. INPUT_DIR is flat, the loader globs it for *.csv and never resolves config.json’s file keys as paths, and our CSVs were one directory down.

The exit code is not the evidence; the row count is. The runner now greps both numbers out of the log and fails loudly on a zero, and verify.py reads every series back out of the instance before a load is called done.

Next

Council district, NTA, community district and police precinct do not exist as classes in the base graph at all — city, borough, census tract, ZCTA, school district and state do. Publishing NYC data by council district means defining the class and its entities in MCF, which is supported work with a known shape rather than an open question.

The code lives in a separate workspace, ~/Antigravity/nyc-datacommons, which is local only and has no remote yet — so unlike this repo, the instance’s build is not public.


2026-09-18 — Enumerating the corpus depends on where you start

Devin Balkind · permalink

Every count this project publishes rests on one number: 689 base indicators, enumerated by walking ->relevantVariable from the seventeen SDG goal trees. Ran the same walk from the graph’s actual root to see what else is there.

  From the 17 SDG goal trees From undata/topic/Root
Topic nodes 3,761 29,805
Peer groups 1,916 19,532
Variables 6,025 65,418
Base indicators 689 1,661

978 base indicators are unreachable from the SDG goal framework — WHO 354, UNICEF 245, ILO 93, UNIDO 69, UNAIDS 41, OHCHR 36, UNFPA 32, UNDP-HDRO 28, ITU 25, UNESCO 18, UNODC 12, ECLAC 6, UNDRR 5, IOM-DTM 1, UNHCR 1. The goal framework is 41% of the governed corpus.

The part worth reporting

Six base indicators are reachable from the goal trees and not from Root:

undata/sdg/SG_DSR_SILN   undata/sdg/VC_DSR_AGLH
undata/sdg/SG_DSR_SILS   undata/sdg/VC_DSR_CHLN
undata/sdg/SM_POP_REFG_OR  undata/sdg/VC_DSR_HOLH

All seventeen goal trees are direct children of Root — we checked, they are 17 of its 42 children. So a traversal from Root should be a strict superset of a traversal from the goal trees, and it is not.

Three things rule out the boring explanations:

So ->relevantVariable is not transitive across these hierarchies, and no single entry point enumerates the graph completely. A client that picks one — as we did, for principled reasons — gets a silently incomplete corpus and no way to know it. That is worth the platform team knowing, and it is the third structural item we have for them, after the 170 unnamed indicators and the Percent unit covering both bounded proportions and signed rates.

What it does not change

The 689 is still the right denominator for this project, and nothing published needs correcting. A Voluntary Local Review reports against the SDG goal framework; enumerating from those seventeen trees is the correct scope, not a shortcut. The demo’s masthead says “689 base indicators in the goal framework”, which is exactly what it is.

What changes is that we now know the framework’s share: 689 of 1,661, and the other 978 are the agency series — WHO’s health indicators, ILO’s labour series, UNICEF’s child statistics — which a city might well want and which no SDG-scoped enumeration will ever surface.

Next

The smell test is running against the 978 now. The first twenty-five rows already show ILO series with 1,067 and 1,471 observations across 135 countries, so this is not a thin surface. All five data errors found so far came from undata/sdg/ alone, which was 41% of the graph; this is the other 59%.

probe/corpus.py --roots all now writes to its own file and keeps its own resume state, so a whole-graph walk can no longer overwrite the SDG corpus that every downstream count resolves against. It would have, silently, before today.


2026-09-18 — Procurement: the framework has the slot and it is empty

Devin Balkind · permalink

Added public procurement as a third category to probe/category_gaps.py, expecting the elections result again — no indicator, lots of city data. The first run said exactly that: zero of 519 named indicators, 150 datasets across 28 cities.

It was wrong, and finding out why produced a better result.

The method was searching names, and a quarter of the framework has none

SG_SCP_PROCN sits in the SDG goal tree. It is the 12.7.1 series — sustainable public procurement — and it carries no name, so a search over indicator names cannot see it. Neither can it see SG_SCP_PROCN_HS or SG_SCP_PROCN_LS.

We have known since Wednesday that 170 of the 689 base indicators return no name. What we had not done is account for them when claiming a concept is absent. The category search now looks at DCID mnemonics too, and reports them as candidates for a human to read rather than counting them — mnemonics are noisy, and SE_ACS_ELECT is schools with access to electricity, not elections.

What the slots actually contain

All three procurement DCIDs exist. All three hold zero observations for zero countries.

The one adjacent indicator that does carry data is SG_SCP_CNTRY — countries with sustainable consumption and production national action plans — and its value is 1 for all 75 countries that report it. Nobody reports a 0. It is a list of countries that have a plan, recorded as a number.

So the framework does not lack a procurement indicator. It has three, and they are empty; and the thing next to them that works counts whether a country has written a policy down.

Against that, 28 cities publish 150 procurement datasets — tenders, awards, vendors, purchase orders. New York City alone registers 55,806 contracts worth $147 billion, every one with an agency, a vendor, a value and a status.

The pattern, now three for three

Category What the framework measures What cities publish
Records access 16.10.2 — whether a country has adopted guarantees 8 cities: requests filed, answered, how long they took
Procurement 12.7.1 — three empty slots; the neighbour counts action plans 28 cities: 150 datasets of actual transactions
Elections nothing at all 23 cities: 469 datasets of results by polling station

Twice the framework asks has a policy been adopted and the cities answer here is what happened. Once it does not ask.

Procurement is the sturdiest of the three

By this project’s own evidence rule — count cities, not datasets — procurement is the best of them:

One honest wrinkle: procurement datasets are not unusually far from the framework in embedding space — mean affinity 0.393 against 0.407 for municipal data generally, and only 32% fall in the bottom quartile of their own catalog, against 64% for elections. The reason is that a procurement dataset is about something — road contracts, health contracts, water contracts — and those subjects do have indicators. The data gets absorbed by its topic rather than recognised as procurement. The lexical search is what makes the gap visible; the embedding would have missed it.

Also true of the other two

Every unnamed DCID the mnemonic search surfaced across all three categories — 19 of them, including the democratic-institutions and judiciary series SG_DMK_JDC* and SG_DMK_PARLCC_* — holds zero observations. Unnamed and empty are the same set. That is consistent with Wednesday’s --all sweep, where 207 of the 247 indicators outside the usable set returned no country data at all, and it is worth stating plainly: the framework’s governance slots are largely unfilled, not merely unnamed.

Procurement is now a third block on the demo, and mcp/smoke.py (32 checks) asserts the empty-slot claim against the artifact, so it cannot quietly become false if the platform fills them in.


2026-09-17 — Correction: records access is eight cities, not fifteen

Devin Balkind · permalink

Wiring the inverse crosswalk’s two headline categories into the demo meant checking their numbers a second way, and one of them did not survive.

The 16 Sep write-up said fifteen cities publish records-access request logs. Counted properly it is eight.

The fifteen was cluster membership — how many cities contributed a dataset to the k-means cluster whose distinctive terms were foia, request, log. That is not the same as how many cities publish a request log. The cluster had swept in Edmonton’s Media Releases and NYC’s City Hall Library Catalog, which sit near request logs in embedding space and are not request logs.

probe/category_gaps.py now counts the same categories without any clustering: search every named SDG indicator for the category’s vocabulary, and count the municipal datasets whose titles carry it, in six languages. Both halves are greppable.

  Cluster membership Vocabulary count
Records-access requests 15 cities 8 cities · 73 datasets
Electoral administration 10 + 6 cities, two runs 23 cities · 469 datasets

Elections went up, which is the useful part of the check: clustering was splitting one category across two runs and under-counting it, while over-counting the other. A method that is wrong in both directions is not a method you can read a number off.

What this does not change

The argument for both gaps stands, and the elections one is stronger than before:

The rule this leaves behind

A cluster is not a count. k-means returns k groups whether or not k categories exist, and its membership is the nearest thing to a centroid, not the things that belong to a category. We had already made it declare its own coherence and publish 14 of 40 clusters as diffuse. This goes further: where a category can be counted lexically, the lexical count is the one that gets published, and the cluster is what found it rather than what measures it.

Both categories are now on the demo, with the smaller, checkable numbers — and mcp/smoke.py asserts the page against probe/category_gaps.py, so they cannot drift apart.


2026-09-17 — What NYC can say that the United States cannot

Devin Balkind · permalink

The asymmetry this project keeps circling: the UN graph is national-level, so a city can only be placed against countries — and for 130 of the 442 usable SDG indicators the United States reports nothing at all. On those, NYC vs the US is not a weaker comparison than NYC vs the world. It is the only one available, and the US is not in it.

Built the worksheet: us-silent-latest · python3 probe/us_silent.py

The number is 39, not 67

The briefing has said 67 since the corpus was first screened — 130 US-silent indicators, of which the scope classifier judged 67 city-scoped. Building the worksheet showed that number is inflated, and why.

28 of the 67 have a country as their subject, not a place. “Extent to which countries have laws and regulations that guarantee full and equal access to sexual and reproductive health care” is not a quantity New York City can hold a value for; the question is whether a national legislature passed something. Eighteen of the 28 are that one family — the SDG 5.6.2 legal provisions — and the matcher paired every one of them with NYC’s Local Law 37/2011 Temporary Housing Assistance, on the token “Law”.

The scope classifier could not have caught this. It reads an indicator’s topic — health, water, education — and judges it municipal, which for 5.6.2 it is. What it cannot see is that the sentence measures a legislature. So that filter is now a lexical rule rather than a classifier: “extent to which…”, “countries that…”, “proportion of countries…” are excluded, and “extent of human made wetlands” is not. A rule you can read and argue with beats a score you cannot.

39 survive. 38 have a peer group of 15+ reporting countries, and 26 also have an NYC candidate above the noise floor.

The real answer is waste and water

Twelve of the 39 form one coherent group, and it is the group the briefing predicted:

Countries Indicator Years NYC candidate
133 Municipal waste collected 2000–2023 Recycling Diversion and Capture Rates (0.63)
107 Hazardous waste generated per unit of GDP 2000–2024 Wastewater Co-digestion and Biogas (0.53)
97 Hazardous waste exported 2000–2024 DSNY Disposal Sites Used by Facilities (0.52)
97 Total waste generation 2000–2023 DSNY Waste Characterization (0.52)
94 Total wastewater treated 2022 only Watershed Water Quality – Wastewater (0.54)
93 Hazardous waste treated or disposed 2000–2024 Dewatered Solids and Biosolids (0.51)
88 Proportion of hazardous waste treated or disposed 2000–2024 Dewatered Solids and Biosolids (0.51)
85 Hazardous waste imported 2000–2024 DSNY Disposal Sites Used by Facilities (0.49)
84 Groundwater bodies with good ambient water quality 2017–2023 Watershed Water Quality – Limnology (0.52)
84 Total wastewater generated 2022 only Wastewater Co-digestion and Biogas (0.59)
72 Proportion of wastewater treated 2022 only Dewatered Solids and Biosolids (0.52)
55 Extent of human made wetlands 2021–2025 NYC Wetlands (0.50)

These are DSNY and DEP series. NYC genuinely publishes them, and 133 countries report municipal waste collected while the United States reports none of it. That is a comparison NYC can make and its own country cannot.

Three of the twelve carry one year. That supports a level comparison and not a trend — the same finding that removed road safety as our headline demo, and the worksheet marks it rather than letting a reviewer discover it after choosing the indicator.

What corroborates it

Two results from the last two days point at the same set from opposite directions.

The corpus smell test found that sweeping all 689 indicators rather than the 442 usable ones adds almost nothing — but the 40 that do hold data are almost entirely e-waste and municipal waste import/export.

The inverse crosswalk found e-waste and hazardous waste per capita among the 123 indicators no municipal dataset comes near — nobody publishes them at city level either.

So this band of the framework is thinly reported by everyone: the US skips it, cities mostly skip it, and it is exactly where a city that does publish has something unusual to say.

The screen held

screen.py decides does the US report this from entityCoverage on a six-country panel — a proxy we have flagged as a proxy. Checked all 39 against the actual country observations: not one has a single United States datapoint. The proxy and the data agree completely.

That is the second independent confirmation of that panel this week; yesterday’s --all sweep showed its NO-DATA grade predicts global emptiness 84% of the time.

Where it is still weak

Also today

The platform went fully public this morning. python3 probe/launch_diff.py: 57 series, 12 variables and 689 corpus indicators, zero drift against yesterday’s pre-launch baseline, and mcp/smoke.py 8/8. No public hostname resolves yet — undatacommons.unicc.biz and datacommons.un.org both still refuse — and the deployment we have always used is still answering. The demo is safe.


2026-09-17 — The inverse crosswalk in five more languages

Devin Balkind · permalink

Ran the inverse crosswalk over the non-English catalogs: 5,956 datasets from 11 cities in Italian, Portuguese, Spanish, German and Croatian, pooled into one run under the multilingual model. Results · python3 probe/inverse.py --language non-en

Pooled rather than run per language, because cosine similarities from two different models are not comparable and two-to-four cities per language could not support an analysis whose unit of evidence is how many cities. Pooled, they share one space and eleven cities.

Elections, in both language groups, independently

The tightest cluster in anything this project has produced: 241 datasets across 6 cities, coherence 0.90 — Milan’s Elezioni Politiche: Risultati di Sezione, Madrid’s Elecciones Autonómicas, polling-station-level results going back to 1996.

And the English run found the same thing separately: election results by voting station, 10 cities, coherence 0.71 — Calgary, Edmonton, and others.

Two disjoint sets of cities, two different embedding models, one category. The SDG framework has no indicator for electoral administration at all; the nearest thing the matcher can find for the English cluster is “number of local governments”. Turnout, results, polling-station coverage — cities publish all of it and the framework has no word for any of it.

COVID-19 case reporting corroborates the same way: 6 English cities and 7 non-English cities, independently. The nearest indicators the matcher offers are “new HIV infections per 1,000 uninfected population” and “number of total conflict-related deaths”. There is no city-level pandemic-surveillance indicator to match.

What only the non-English catalogs show

Cities Datasets Theme
5 113 Property market valuations — Italy’s OMI quotazioni immobiliari, sale and rental prices by zone, half-year by half-year since 2004
6 69 Weights-and-measures inspection — Brazil’s Ipem verifications of pre-measured goods
5 33 Public-employee working arrangements — Milan’s personale a tempo indeterminato in telelavoro, permanent staff on telework, by year

The property-valuation series is the interesting one. Housing affordability is squarely an SDG 11 concern (11.1.1 is inadequate housing), but the framework counts people in inadequate housing and the Italian cities publish what property costs, by neighbourhood, twice a year, for twenty years. Those are not the same measurement and only one of them has an indicator.

The controls came out better than the English run

11 of 11 passed, against 8 of 9 in English. These are hand-built: no city outside NYC has a graded worksheet, so the controls are datasets whose titles plainly correspond to an SDG concept, chosen across all five languages and recorded in probe/inverse_controls.json.

The two weakest are both Portuguese — Fortaleza’s Número de Óbitos por Acidentes de Trânsito at the 44th percentile and Recife’s Acidentes de Trânsito com Vítimas at the 32nd. Those are about as unambiguous as a match gets (deaths from traffic accidents ↔ SDG 3.6.1) and they sit in the bottom half. Portuguese retrieval is the weakest link, so the Brazilian portion of the tail is the least trustworthy part of this run, and anything resting on Fortaleza, Recife, Belo Horizonte or São Paulo alone should be read with that in mind.

A quarter of the framework was unmatched text

Both runs improved when I stopped matching against indicators that have no name.

170 of the 689 base indicators carry no name in any source we hold — they return neither metadata nor observations. The first version fell back to the DCID mnemonic, so a quarter of “the framework” was represented by query strings like DI ILL OUT. Nothing can match that, which inflates the tail — datasets look further from the framework than they are — and it put mnemonics in the nearest-indicator column, where VC VAW SXVLN was being reported as a cluster’s closest SDG concept.

Excluding them moved every weak control up: Fortaleza from the 28th percentile to the 44th, Recife from the 27th to the 32nd. The framework is now represented by its 519 named indicators, and the count is stated on the page rather than folded into a total.

Two smaller fixes in the same pass. Cluster examples were being spread one-per-city to show reach, which made them unrepresentative of clusters dominated by one city — a cluster labelled for Italian property valuations was illustrated with a Madrid shop register. Each cluster now shows its most central members as well as the city spread. And the phrase view returns nothing here, which the page now says is expected by construction: Spanish, Italian, Portuguese, German and Croatian titles share essentially no bigrams.

Where it stands

  English Non-English
Datasets / cities 11,206 / 37 5,956 / 11
Controls passing 8 of 9 11 of 11
Coherent clusters 23 of 40 8 of 20
Indicators nearest to nothing 69 of 519 183 of 519

The non-English run has better controls and worse coverage — 183 indicators are nearest to no dataset at all, against 69 in English, which is what you would expect from half the datasets and a weaker cross-lingual model. It is not evidence that those cities publish less.


2026-09-17 — The inverse crosswalk: what cities measure that the SDGs do not

Devin Balkind · permalink

Every analysis in this repo has run city → UN: take an SDG indicator, find the municipal dataset that matches it. That direction can only ever discover what the framework already asks about. So: run it backwards. Take 11,206 datasets from 37 city portals, find each one’s nearest neighbour among all 689 SDG indicators, and look at what is left over.

The question is not which cities are behind. It is what would SDG 11 look like if it had been written from municipal data upward?

Full results · python3 probe/inverse.py

The answer, as far as it goes

Themes that recur across many independent city catalogs and sit in the bottom quartile of every one of them:

Cities Theme
23 Building permits, licences and code enforcement
15 Records-access request logs (FOIA and equivalents)
13 Bike parking, bike share, active-travel space
10 Street sweeping schedules and parking enforcement
10 Call-centre response performance — answer times, 311 service levels
9 Pedestrian and bicycle counts
8 Property sales and assessment
7 For-hire vehicle and taxi trip records
7 Fire stations and emergency facilities
6 Special events permitting

The sharpest one is records access. Fifteen cities publish request logs — how many were filed, how many answered, how long it took. The nearest SDG indicator the matcher can find is 16.10.2, “countries that adopt and implement constitutional, statutory and/or policy guarantees for public access to information.” The framework asks whether a law exists. The cities publish whether the law works. A country can score full marks on 16.10.2 and answer nothing, and no indicator in the framework would notice.

The same shape appears in call-centre performance: the SDGs have no concept of how quickly a government responds to its residents, and ten cities publish exactly that, in seconds.

The other direction

123 of the 689 indicators were the nearest neighbour of no municipal dataset at all. Many are honestly national — ODA flows, external debt, climate finance, which no city could report. But the list also contains e-waste collected, generated and recycled per capita, and hazardous waste generated — which is a double gap. Those are the same indicators we found the United States does not report, and it turns out cities do not publish them either. Nobody is measuring them at any level of government.

What this is worth, stated honestly

One positive control in seven failed. NYC’s Housing Maintenance Code Violations is a hand-verified match for SDG 11.1.1 inadequate housing, and it landed at the 24th percentile — inside the tail. So the tail contains real matches, and any individual dataset in it may simply have been missed. That is why the unit of evidence above is cities, not datasets: one dataset scoring low is retrieval failure, and twenty cities independently publishing the same category is not.

Fourteen of forty clusters are noise, and the report says which. k-means returns k clusters whether or not k themes exist; each one now carries the mean cosine of its members to its own centroid, and below 0.62 it is published as diffuse rather than read as a theme. The first run had no such measure and cheerfully labelled a cluster “school” that contained building violations and lobbyist registrations.

English-language portals only. Comparing cosine similarities across two embedding models is meaningless, so the multilingual catalogs need their own run.

The bug underneath it

The first run’s clusters were incoherent, and the reason was not the clustering.

embed.Index decided whether cached vectors could be reused by checking len(cached_ids) == len(datasets). Re-fetch a portal months later and it returns the same datasets in a different order — measured: 24 of 45 city catalogs did exactly that. Same count, same IDs, different sequence. The length check accepted it, and every dataset was handed another dataset’s vector.

There is no symptom. Similarity scores stay in range, rankings look ordinary, nothing errors. It surfaced only because a cluster labelled “school” was full of facade-compliance filings and I looked at the rows.

It now matches on IDs, and when the set is identical but re-ordered it permutes the cached vectors instead of re-embedding. Worth noting what this didn’t affect: the published municipal table was computed on the run that created its cache, in the order it created it, so those numbers were right. The bug only bites on a later re-run — which is to say, it was waiting for the next person to re-render the table.

That is the eighth entry in this project’s list of automated checks that produced a confident wrong answer, and the fourth caught by reading rows rather than totals.


2026-09-16 — The whole corpus: what the other 247 indicators hold

Devin Balkind · permalink

The smell test ran against the 442 indicators screened usable. That left 247 graded NO-DATA or THIN-COVERAGE — and the grade was assigned against a six-country panel, so it only ever meant “no data for those six”, not “no data anywhere”. Ran it with --all to find out.

689 indicators · 773,335 observations · 2,503 findings.

The screening panel was right, and now that is measured

The extra 247 indicators contributed 5,056 observations — 0.65% of the corpus — and 207 of the 247 returned no country-level data at all. A grade assigned on six countries predicted global emptiness correctly 84% of the time.

That closes a question I had raised against our own method. The six-country panel was chosen for speed, with the honest caveat that it could not distinguish “the panel does not report this” from “nobody does”. It turns out to distinguish them almost perfectly, and 442 remains the right denominator for everything downstream.

The 40 indicators that do hold data are worth knowing about, because of what they are: e-waste collected, recycled and its proportions; municipal waste imported and exported. Those sit exactly in the band we identified months ago as the interesting one — indicators a city can report internationally precisely because its own country skips them.

Guadeloupe, again, four times over

The previous sweep found Guadeloupe’s e-waste collection multiplying by a thousand in 2022. With the full corpus it turns out the same error propagates through every derived indicator:

Indicator Unit 2021 2022
Electronic waste collected per capita kg 13.71 13,950
Electronic waste recycled per capita kg 13.71 13,950
Total electronic waste collected tonnes 5,472 5,367,000
Total electronic waste recycled tonnes 5,472 5,367,000

The largest total any other country has ever recorded for e-waste recycled is 899,300 tonnes. So Guadeloupe — population about 380,000 — is published as the world’s biggest e-waste recycler, by a factor of six. One unit slip in a 2022 submission, carried into four indicators.

This is also the first time the checks corroborated each other: the same anomaly surfacing in four related series is much stronger evidence than the same anomaly surfacing once.

Small states reporting zero for two decades

The new coverage brought a pattern rather than a single error. Municipal waste exported and imported, both flatlined at exactly zero for very long runs: Mauritius 23 years, Dominica 22, Singapore 21 and 22, Cuba 20, Palestinian Territories 19, Jamaica 18, Saint Lucia 16, Liechtenstein 24.

Some of that is surely true — a small island genuinely exports no municipal waste. But a twenty-year run of exact zeros is also what “no submission” looks like when it is stored as a number instead of a gap, and the two are indistinguishable from outside. Worth one question to the platform team: is zero here a measurement or a default? Our own rule is that a missing year stays missing and is never interpolated; the same distinction matters just as much in the other direction.

Alongside it, “Proportion of electronic waste that is collected” sits at exactly 100% for Niger for eight consecutive years and Iran for six.

Everything else held

The finding counts barely moved — 2,457 to 2,503, with one new HIGH — which is the correct result for adding 0.65% more data. The five verified errors from the earlier sweep stand unchanged.


2026-09-16 — Denominators: Eurostat is the obvious answer and it is wrong

Devin · permalink

Most SDG indicators are rates per 100,000, so every non-US city needs a population figure before it can produce a chart. Two are now wired, and the route we did not take is the more useful finding.

Eurostat’s Urban Audit publishes greater cities

urb_cpop1 covers 969 European cities annually and is exactly the single source this needs. It reports greater cities, not municipalities:

City Eurostat “greater city” Municipality If used
Madrid 5,115,272 3,520,396 rates ~31% too low
Milan 3,580,530 1,399,079 rates ~60% too low

For Madrid the table contains no municipality entry at all — the greater city is the only option. A city’s open data covers its municipality, so this is the denominator mismatch the whole project exists to catch, and it would have been completely invisible in the output.

So denominators come from the cities themselves

Better provenance as well: numerator and denominator then share a publisher.

Madrid — Padrón municipal (200076-0-padron), summing the four Spanish/foreign × male/female columns of a 34 MB CSV: 3,520,396 as of 2026-09-01. A snapshot, so Madrid supports levels and not trends until the historic padrón is wired.

Milan — Popolazione calcolata (ds1494), year-end population 1880–2025, ISTAT to 2002 then the city’s own anagrafe: 1,399,079 in 2025. A series, so Milan supports trends — the first non-US city that does.

Two cities, two shapes, two resolvers. City statistical publications do not share a format, and pretending otherwise is how you end up with Eurostat.

The bug this work exposed

Building a summary table showed Chicago and Boston both reporting 8,478,072 — New York’s population.

The census resolver fell through to NYC’s place code whenever a city declared none of its own. Chicago’s registry entry named its place code in a prose source string but carried no structured field, so it silently inherited NYC’s 8.5 million. Every Chicago rate would have been three times too low, and nothing in the output would have looked wrong.

There is now no fallback. A city using the census module without explicit state and place FIPS raises, with the reason: refusing to guess, a wrong denominator is invisible in the output.

That is the fifth automated path on this project that produced a confident wrong answer, and the second where the failure mode was a silent default rather than an error.

Where it stands

City 2023 population Source
New York 8,258,035 Census ACS
Madrid 3,520,396 Padrón municipal
Chicago 2,664,454 Census ACS
Milan 1,417,597 Popolazione calcolata
San José 969,615 Census ACS
Boston 652,442 Census ACS

Plus 23 more US cities resolvable by ACS place code. Buenos Aires declares no source and is refused rather than estimated. 24 non-US cities still need one, and each needs a person to find its statistical publication — the part that does not automate.


2026-09-16 — A smell test for the UN graph, and five things it found

Devin Balkind · permalink

Malaysia reports recycling 147.7% of its municipal waste. That figure is in authoritative UN data, it is on our demo, and we found it by accident while building a chart. This is the obvious follow-up, run properly: 442 indicators, 768,279 observations, every reporting country and every year, against checks dumb enough that they need no subject-matter knowledge — a percentage above 100, a negative count, a rate exceeding its own denominator, a value repeated across countries that should not agree.

Full results · python3 probe/smell.py

The tool flags; it does not judge. Everything below that I call an error, I checked by hand against the indicator’s own distribution first.

Five that are wrong

1. Kyrgyzstan feels 6,990% safe walking home. SDG 16.1.4, proportion of population that feel safe walking alone around their local area after dark, unit Percent:

Kyrgyzstan 2018 2019 2020 2021 2022 2023
  57.9 64.35 66.8 6710 6840 6990

The other 252 observations, across 55 countries, run from 22.8 to 95.0, median 72.0. Divide the last three by 100 and you get 67.1, 68.4, 69.9 — which continues Kyrgyzstan’s own trend from 66.8 exactly. Live for three consecutive years.

2. South Africa recycles 1.86 billion tonnes of municipal waste. Global municipal solid waste generation is roughly 2 billion tonnes a year, worldwide. South Africa’s own earlier figures are 260,600 t (2005) and 520,800 t (2006); from 2018 they are 1.0–3.4 billion. Read as kilograms they are entirely sensible.

3. Brunei generates 12,580 tonnes of hazardous waste per person. EN_HAZ_PCAP, unit kilograms, where the global median is 22 kg and the maximum outside Brunei is 212. Brunei reports 8.7–36 million kg per capita, every year from 2016 to 2023. Against a population of ~450,000 that is 5.7 billion tonnes. It looks like a national total sitting in a per-capita field.

4. Guadeloupe’s e-waste collection multiplies by a thousand in one year. 8.46, 8.54, 10.03, 10.36, 10.58, 11.97, 13.10, 13.71 — then 13,950 in 2022. The cleanest of the five: the series resumes perfectly if you divide the last point by 1,000.

5. Malawi is paid to send remittances. SI_RMT_COST, the average cost of sending $200, as a percentage. Malawi runs 16.95, 15.82, 14.47, 16.26, 14.79, 13.13 — then −0.1 and −0.93 — then 31.48. A cost of minus one percent, for two years, between normal values.

One that is not an error, and matters more

The United States recorded 345,600 disaster deaths in 2020 and 470,600 in 2021. Every other country in VC_DSR_MORT has a median of 42. Those are COVID-19 deaths: the US classified the pandemic as a disaster and reported it here. Most countries did not.

That is not a data error. It is the exact failure this whole project exists to prevent — two numbers that share a variable, a unit and an axis, and are not the same measurement. A chart of “disaster deaths, US vs peers” would be perfectly well-formed and would tell you something false. It is the strongest argument yet for the comparability grade, and we found it with a check that knows nothing about disasters.

Four that look wrong and are not

Each of these I chased and dropped, which is most of what the afternoon consisted of:

What the checks learned about themselves

The first full sweep returned 41,350 findings — 5.4% of all observations, which is not a result, it is a broken instrument. Three false-positive classes, each caught by reading rows rather than totals:

Check said Actually
351 jumps in one indicator (12% of its rows) the Indicator of Food Price Anomalies is a signed index centred on zero; a ratio means nothing on it
23,172 negative percentages growth rates and balances are signed by construction
four scaled rates treated as plain counts my substring match on "COUNT" swallowed RATIO_COUNT_PER_100_COUNT_POP, so its cap never applied

The first fix over-corrected: it gated the jump check on a unit whitelist, and thereby discarded Mauritius’ food waste going 207 tonnes → 177,570 tonnes in a year because WEIGHT_TN was not on the list. Gating on the shape of the quantity — does this indicator ever go negative anywhere? — keeps it.

What came out of that is a principle worth more than any individual finding: an indicator is its own control group. A rule broken by most of an indicator’s observations is its definition. A rule broken by three country-years out of three thousand is Kyrgyzstan. So checks are now suppressed per-indicator when they fire often enough to be structural — 165 suppressed groups, 38,943 would-be findings — and the suppressions are published, because the list of indicators that publish signed values under a Percent unit is itself a finding about the graph’s vocabulary.

That left an outlier check that works against each indicator’s own distribution rather than against what a unit is supposed to mean: 20× the indicator’s 99th percentile. It survives suppression, which matters — “proportion of hazardous waste treated” exceeds 100% for 37 countries, so its range check is structural and gets dropped, and Guatemala’s 44,825% would have gone with it. It is also the only check that reaches the 123 indicators and 248,457 observations whose units (CR_USD, WEIGHT_TN, INDEX, SCORE) have no meaningful range at all. The report states that coverage gap on its own face.

Final count: 2,457 findings, 92 of them HIGH. Small enough to read.

What this is for

Two things. It is a QA gate on the data our own crosswalk depends on — and it is the kind of thing only an outsider runs, because it requires no authority and no access, just the published graph and a willingness to look at 768,279 numbers. The five errors above are reportable to the platform team as they stand.


2026-09-16 — Fixed: the matcher now finds Vision Zero Fatality Records — and NYC got better too

Devin · permalink

Boston’s Vision Zero Fatality Records ranked 23rd of 235 for “death rate due to road traffic injuries”, below a contract-award file. It now ranks 1st. NYC’s ground-truth median improved from 23 to 14 at the same time.

The diagnosis was not the query

First instinct was query expansion — add “crashes, collisions, fatalities, vision zero” to the indicator name. It made things worse (rank 23 → 30). So did rephrasing: “traffic fatalities” scored worse (32) than the clinical UN wording (23).

The problem was the document, not the query. Vision Zero Fatality Records carries excellent metadata:

title: Vision Zero Fatality Records
tags:  accidents bikes cars crashes fatalities pedestrians safety streets traffic vision zero
desc:  1,500 characters of programme mission statement — "our commitment to focus
       the city's resources on proven strategies… we are inspired by…"

A static embedding averages over every token. Sixty characters of exactly-right title and tags were being drowned by 1,500 characters of boilerplate. The dataset that beat it, My Neighborhood Dataset, won partly by having a shorter description.

The fix: score each field separately, take the best

Two vectors per dataset instead of one concatenated blob:

Score is max(head·q, body·q), so a dataset surfaces on whichever field actually carries its signal. Programme-named datasets are found by their tags; thinly-titled ones by their prose.

Measured both ways before adopting it:

Scheme NYC median rank top-10 top-50 Boston: Vision Zero
concatenated (before) 23 3/7 5/7 #23
head only 57 3/7 3/7 #1
max(head, body) 14 3/7 5/7 #1

Head-only fixes Boston and wrecks NYC — PM2.5 falls from rank 30 to 397, because NYC’s air-quality dataset is found through its description. Taking the max gets both.

Boston now returns Vision Zero Fatality Records (0.544) and Vision Zero Crash Records (0.541) as the top two for road deaths.

A silent cache bug found on the way

Vector caches were keyed by city and model but not by the text representation. Changing how documents are built silently reused vectors from the old scheme — wrong rankings, no symptom, no error. Some of the earlier Boston numbers were affected by exactly this.

The cache key now includes a REPR_VERSION that must be bumped when the representation changes. Fourth entry in this project’s running list of automated checks that were confidently wrong.

Still not solved

This improves retrieval; it does not make the matcher trustworthy. The cities now surface their obvious datasets, and a person still has to decide whether Vision Zero Fatality Records is comparable to SDG 3.6.1 — which, per the spec, is the one thing a machine may not do.


2026-09-16 — Correction: the other cities do have comparable data — our matcher just can’t find it

Devin · permalink

The briefing and the last two posts carried this line:

No substantive SDG indicator has matched outside NYC.

True of our matcher. Read as a statement about the cities, it is wrong, and it was phrased so it would be read that way. Correcting it.

The data is there

Searching Chicago’s and Boston’s catalogs directly, by hand, for the four indicators NYC is crosswalked on:

  Chicago (915 datasets) Boston (235)
Crime / homicide Crimes — 2001 to Present Shootings · Homicide Clearance Rate
Road deaths Traffic Crashes — Crashes / People Vision Zero Fatality Records
Air quality Open Air Chicago (measurements, hourly, daily) only building emissions (BERDO)
Waste Christmas tree recycling; no municipal tonnage Trash schedules, not tonnage

Chicago’s Traffic Crashes — Crashes is structurally the same thing as the NYC dataset behind our road-deaths card. Boston publishes a dataset literally called Vision Zero Fatality Records — the same programme, the same concept, named almost identically to what we used for NYC.

What our matcher did with it

Querying Boston for “Death rate due to road traffic injuries”:

0.486  Traffic-Related Data
0.467  City of Boston Contract Award
0.444  My Neighborhood Dataset
0.387  Trash Collection Days
...
#23    Vision Zero Fatality Records        (0.286)

The correct dataset ranked 23rd of 235, below a contract-award file. No threshold rescues that; it is a retrieval failure, not a calibration one.

The matcher is not uniformly blind — querying homicide returns Homicide Clearance Rate at rank 1, because the word “homicide” appears in the title. It works when the vocabulary happens to line up and fails when a city names the same concept differently, which cities routinely do. “Vision Zero” is a programme name, not a description of its contents.

The honest position

This is the third time on this project that an automated check produced a confident answer a person overturns by reading for five minutes: the chartable flag that ignored human grades, the screen that reported 689 having recorded 9, and now a matcher whose silence was mistaken for absence.

A null result from a tool you have measured at ~50% precision is not evidence of absence. We wrote that lesson into the spec as R1 and then made the mistake anyway, in our own prose, about our own tool.


2026-09-16 — The launch diff, and the bug it found in itself

Devin Balkind · permalink

The platform goes fully public tomorrow, 17 Sep. Every figure this project publishes was measured against the pre-launch deployment, and the open question in the briefing was blunt: will staging DCIDs survive the launch? A withdrawn DCID does not raise an error. It returns an empty result, which is the same shape as a country that does not report — so the demo would show a blank chart to a room at Google NY and nothing would look broken until someone asked.

So: re-ran every probe against the live deployment and diffed it against 14 Sep, and built probe/launch_diff.py so that tomorrow’s version of this is one command rather than an afternoon.

The graph has not moved

What 14 Sep 16 Sep
SDG base indicators enumerated 689 689
Variant DCIDs beneath them 6,025 6,025
Indicators screened, every field compared 689 689, zero changed
Crosswalk series (variable × place), value by value — 57 checked, zero changed
Crosswalk variables, via metadata — 12 checked, zero changed
Demo figures (mcp/smoke.py) 8/8 8/8

Not one DCID added, not one removed, not one value revised, not one unit changed. The three EMPTY series are the ones we already knew about — the UN holds no municipal-waste observations for the US, Japan or Mexico, which is the finding the municipal-waste card is built on.

The only differences anywhere in the crosswalk were ours, not the platform’s: two figures in NYC’s housing dataset moved by one (297,753 → 297,754 — NYC Open Data is live), the road-deaths pair was regraded BLOCKED → RANK-ONLY by hand yesterday, and the municipal-waste pair is new since the 14th.

The search surface has moved

The coverage probe grades whatever search_indicators returns for fifteen topic queries. Same code, same queries, same host, two days apart:

The one worth noting: searching “road traffic deaths” on 16 Sep returns undata/sdg/SH_STA_TRAF — the exact DCID our road-deaths pair uses — and on the 14th it did not. Recall improved. That is good news and a warning in the same breath: the graph is frozen, the retrieval over it is not. Anything that resolves a DCID by searching at runtime can change its answer without the data changing at all. Our crosswalk hardcodes DCIDs that a human resolved once, which is exactly why the demo is insulated from this. It was a chore to do it that way. It is not a chore any more.

I checked the obvious alternative explanation before writing this down: search_indicators returns byte-identical results across three consecutive calls today, and neither coverage_probe.py nor candidates.json has been touched since the initial commit.

The bug it found in itself

The first run of launch_diff.py reported, confidently, that all twelve crosswalk DCIDs were missing from the graph — while get_observations was returning data for every one of them.

get_variable_metadata requires entity_dcids. Omit it and the server answers 200 with a completely empty body — no error, no status field, no variables key at all. A caller that trusts the response shape reads that as “every variable has been withdrawn”. Two days before the launch it was written to detect, the launch detector was ready to cry wolf about the entire crosswalk.

That is the seventh entry in this project’s list, and it was caught the same way as four of the other six: by printing a table of all twelve rows and reading it, rather than trusting a summary count. A checker that reports catastrophe is at least loud. The version of this bug that would have actually hurt is the mirror image — a diff that stays quiet because it cannot see.

So launch_diff.py --self-test now injects each drift class into a copy of the baseline and asserts the diff reports it: a revised value, a series going empty, a series coming back, a changed unit, a shortened span, a renamed variable, a withdrawn variable, an indicator dropping out of the corpus, a tool disappearing from the server. Plus the converse — an unmutated copy must report nothing. 10/10. A diff that returns “no change” and a diff that cannot see change are indistinguishable from the outside, and this project has shipped that mistake often enough to stop paying for it twice.

Tomorrow

The baseline is recorded as of today, pre-launch. After the platform goes public:

python3 probe/launch_diff.py        # exits non-zero if anything moved
python3 mcp/smoke.py                # the demo's published figures

If the public deployment answers on a different host, UNDC_ENDPOINT and UNDC_REST override it without a code edit, and the diff reports the host change as drift in its own right. As of this afternoon no public hostname resolves yet — undatacommons.unicc.biz and datacommons.un.org both refuse to connect, and the deployment we have been building against is still answering normally.


2026-09-16 — Threshold calibration, and a registry we said didn’t exist

Devin · permalink

Two corrections and one fix. The corrections are the useful part.

The CKAN registry exists. We said it didn’t.

We reported that no global registry of CKAN portals survives — that every canonical one had rotted. That was wrong.

The CKAN Ecosystem Catalog is a 2025 NSF POSE II project from the CKAN core team, WPRDC and datHere. It lists 199 instances, 97 of them local or regional government, machine-readable at ckan/ckan-instances.

We missed it because we searched for the registries we already knew about — ckan.org/instances, dataportals.org, opendatainception — found all three dead, and concluded the category was dead. The correct inference was that we were looking at the previous generation. (The site sits behind Cloudflare and refuses both curl and a headless browser; the GitHub repo is the way in.)

The abandoned OKFN list really has rotted — 39 of 631 answer — which is what made the wrong conclusion feel supported.

Inventory now: 372 portals surveyed, 70 municipal, 543,000 datasets.

And a related correction: a portal that does not answer an anonymous package_search is not dead. data.gov, govdata.de and data.overheid.nl all refuse the probe and are plainly alive. The inventory now reports what responds to one specific API call, which is a much weaker claim than liveness.

The threshold fix, and the wrong turn before it

An absolute similarity cutoff cannot travel between catalogs. Measured on hand-judged matches, good pairs span 0.42–0.70 in NYC, 0.50–0.55 in Chicago, 0.37–0.59 in Madrid. One number is simultaneously too strict and too loose.

The first fix was wrong, and instructively so. We calibrated a z-score — how many standard deviations the top match sits above the catalog’s own distribution — measured F1 0.68 against a labelled set, wired it in, and it admitted 60 of 80 indicators. Two independent errors:

  1. The labelled set was biased. Every case in it came from candidates that had already passed the old 0.50 filter, so we measured on a pre-selected population and learned nothing about what the filter should have rejected.
  2. z measures the wrong thing. Against a catalog whose similarities cluster near zero, the top hit sits many standard deviations above the mean whether it is right or garbage. z describes how peaked a distribution is, not how good its winner is.

What actually varies between catalogs — and is the right thing to normalise — is the distribution of top-1 scores across all probed indicators. The cutoff is now the best 20% of that, with an absolute floor so a uniformly hopeless catalog cannot contribute its least-bad rows anyway.

City Cutoff Candidates
Milan 0.450 12
Boston 0.457 16
Chicago 0.473 16
Madrid 0.515 16

Bounded worksheets that adapt to each catalog, instead of 0 or 69.

What has not changed

No substantive SDG indicator has matched outside NYC. Better calibration produces a better worksheet; it has not produced a good mapping. NYC’s pairs were found by a person who knew the data, with the matcher confirming them, and nothing yet shows the matcher can lead.

That is worth saying plainly a week out from Builders’ Day: the infrastructure generalises, the judgment does not, and the judgment is the part that matters.


2026-09-16 — Multilingual embeddings: Madrid 0 → 19, Milan 0 → 5, and a trade-off we kept

Devin · permalink

The blocker for every non-anglophone city was the embedding model, not the pipeline. Adding potion-multilingual-128M unblocks them — and measuring it showed the fix is not free.

City Lang Before After Best sim
Madrid es 0 19 0.59
Milan it 0 5 0.67

Why it is not a straight swap

On the seven hand-verified NYC pairs, against the full 2,400-dataset catalog:

Model Median rank top-10 top-50
potion-base-32M (English) 23 3/7 5/7
potion-multilingual-128M 81 3/7 3/7

Swapping wholesale would have bought non-English coverage with English accuracy — a 3.5× worse median rank on the catalog we know best. Interestingly it is not uniform: the multilingual model improves the two hardest cases (homicide 786→401, slums 1033→716) while degrading the easy ones.

So the model is chosen per catalog language. English cities keep the English model; everything else gets the multilingual one. The vector cache is keyed by model, because vectors from two models are not interchangeable and reusing them across would be a silent, invisible corruption.

The matches, read honestly

Milan is good — 4 of 5 plausible:

Madrid is noisy — six near-identical “water area of lakes and rivers” indicators all matched Fuentes de agua para mascotas, drinking fountains for pets. The word agua dominated. But buried in the noise: installed renewable capacity → Inventario de instalaciones fotovoltaicas, which is exactly right.

The threshold problem is language-independent: 0.50 was calibrated on NYC and means something different in every other catalog. That remains the open issue, and it is now the main thing standing between this and a usable recommender.

Buenos Aires is unreachable, not empty

The portal closes the connection on anonymous package_search — RemoteDisconnected, on two attempts a day apart. Recorded in the registry as status: unreachable rather than as zero candidates, because those are different findings and collapsing them would overstate what we tested.


Earlier