Institute For The Future

Smart Government and Big, Open Data: The Trickle-Up Effect

As we grow numb to the daily headlines decrying the unimaginable scope of data being collected from Internet companies by the National Security Agency's Prism program, its worth remembering that governments themselves also produce mountains of data too. Tabulations of the most recent U.S. census, conducted in 2010, involved billions of data points and trillions of calculations. Not surprisingly, it is probably safe to assume that the federal government is also the world's largest spender on database software—its tab with just one company, market-leader Oracle, passed $700 million in 2012 alone. Government data isn't just big in scope. It is deep in history—governments have been accumulating data for centuries. In 2006, the genealogical research site Ancestry.com imported 600 terabytes of data (about what Facebook collects in a single day!) from the first fifteen U.S. censuses (1790 to 1930).

But the vast majority of data collected by governments never sees the light of day. It sits squirreled away on servers, and is only rarely cross-referenced in ways that private sector companies do all the time to gain insights into what's actually going on across the country, and emerging problems and opportunities. Yet as governments all around the world have realized, if shared safely with due precautions to protect individual privacy, in the hand of citizens all of this data could be a national civic monument of tremendous economic and social value.

This is why the unveiling earlier this month of the new design for Data.gov, the federal governments central repository for public data is so promising - because it is finally starting to move the federal government away from thinking about the site as a digital distribution site for a boundless supply of data, but rather a network of community forums built around bundles of data and software that provide tools for civic hackers to tackle that nation's problems. Gone is the hierarchical tree of agencies; in are clusters around issues: energy, health, safety. And the federal government is also figuring out that apps contests and hackathons need real problems to chew on. For the National Day of Civic Hacking held June 1-2 of this year, federal agencies identified dozens of challenges for coders to address and supplied many valuable new data sets to support their work.

Since its launch on May 21, 2009, Data.gov has grown to house freely downloadable data from 174 federal agencies and sub-agencies. In May 2013, four years after the launch, the site reported 213,000 visitors per month, with traffic doubling year-on-year. It's a success story of progressive governance and innovation at the federal level that has been widely told.

But the single-point-of-service model for Data.gov didn't come from the federal level. Rather, it was local governments in some of America's largest cities that led the way making government data more accessible and more relevant to solving real problems, and they continue to do so.

The movement began in November 2008, not far from the White House, when Washington, DC's Chief Technology Officer Vivek Kundra published more than 200 real-time data feeds as fuel for the first city government app design contest, "Apps for Democracy". The incoming Obama administration tapped Kundra the following March to join his administration as the nation's Chief Information Officer. So, in a very literal way, Data.gov was a national solution spawned from a very local civic laboratory.

Throughout 2009, other big American cities, from San Francisco to New York, quickly followed suit, throwing up their own open data portals.

But a disappointing pattern quickly emerged, most apparent on Data.gov, but broadly seen throughout the whole world of open government data. The portals and the data they held within didn't really add much value. At least in the beginning, there was very little new data being published - the portals were actually little more than aggregation points for existing data sets already posted across an archipelago of government agency websites.

There was also no obvious strategy behind what data was being published, how priorities for release were being set. As I wrote at the time on Planetizen, a leading news site for urban planners, New York City's Data Mine, as its portal was called, was more like a "Data Dump... the collected attachments received in reponse to the poor bureaucrat who had to twist every department's arms for one dataset, so the city could say every department contributed." (view the article here). One can only imagine the scene—late on the night before the portal's launch, deep in a cube farm at the Department of Buildings or Public Health, some over-worked mid-level paper shuffler frantically searching their agency's records for a risk-free, job-preserving database to contribute.

The data governments publish on these sites is growing fast and improving rapidly, mostly in response to citizen requests. But the supply-driven model of those early initiatives has taken longer to undo.

This can be seen in the app design contests that have accompanied the launch of many government data sites. These were promising experiments, aimed at spurring civic innovation at very little cost to the public purse. But beyond throwing open a bunch of data, and dangling a carrot of modest prize money and venture capitalist mentoring for software developers, those apps contests were totally unstructured. There were no grand challenges, like fixing education or health care, nor any more tractable local problems like battling graffiti or finding affordable housing.

To no one's surprise, as these contests demonstrated how America's youthful digital elite could solve their own problems. We saw lots of apps for biking, nightlife, and recreation. Not the kind of stuff that would entice a single mom holding down two jobs to trot down to the Apple store for a smart phone upgrade. The nascent "civic hacking" movement, as many are calling it, had a major inclusion problem. It was too focused on playing with new technologies, and only tangentially in finding out what people's actual problems were.

Again, this shortcoming was recognized first at the city level, and there is where it is being aggressively addressed. Today, you can't do an apps contest in a major American city without an extensive and broadly inclusive round of problem-scoping. Different cities have taken different approaches. For the fourth Big Apps contest in New York, the city recruited local foundations and non-profits to set the agenda for the geeks to work on. San Francisco's Urban Prototyping initiative, spearheaded by a local arts organization, held a series of hackathons during the summer of 2012 that brought together app developers, but also mid-level managers from city government, and community stakeholders from across the city. All of the half-dozen candidates running in that year's mayoral election showed up, and some even rolled up their sleeves and pitched in. These are the new models that the National Day of Civic Hacking has re-purposed on a nationwide scale.

There's a lot of really good work going in this country at the municipal level that is using technology and big, open data to make local government leaner, more innovative, and more transparent. Thank heavens that some of it is trickling up to Washington.