Accountability in government code: the need for due process
From Berkman Fellow Gene Koo...
In the Terminator movie series, an artificially intelligent defense network concludes that the greatest threat to world peace is human beings and proceeds to launch preemptive world-wide nuclear strikes. This Oedipal fear that our silicon-and-code children might one day overthrow us – a favorite among dystopic science fiction authors – may be closer at hand than we’d like to think. Predator drones have yet to evolve to Schwarzenegger clones, but our fears about the inner workings of Diebold voting machines are real and understandable. Thus Professor Danielle Citron’s forthcoming article, “Technological Due Process,” provides a timely examination of an urgent problem challenging our computer-dependent society.
Professor Citron, of the University of Maryland School of Law, has written the opening chapter to an important new field of legal study, one hinted at in Lawrence Lessig’s “Code is Law” formulation and which I had dubbed “Law as Code,” but which until now has not received direct scholarly investigation. Her article, forthcoming in the Washington University Law Review, identifies the need for “technological due process” in software that executes government policies – from public benefits to no-fly lists.
In brief, Professor Citron describes how software code increasingly executes our public laws. Decision support systems, she convincingly argues, quickly become decision making systems. And invariably, the vagaries of the legislative and administrative processes leave large gaps in the specifics of how a given law should be executed. Without firmer guidance from proper governmental bodies, the programmers charged with translating legal code into software code essentially wind up creating law to fill the gaps. (I describe this as “shoving analog pegs into digital slots”). From a procedural – even a Constitutional – perspective, this is a grievously inappropriate delegation of governmental functions to the private sector, not unlike the hiring of Blackwater mercenaries to achieve military objectives. Professor Citron finds, therefore, the need for “technological due process”: safeguards to ensure that software is literally up to code.
In exploring what I’d described as “Law as Code,” and now significantly better-informed by the invaluable analysis of “Technological Due Process,” this fall the Berkman Center started seeking out areas of public law at risk of being improperly executed by unaccountable software code. In my work in legal aid I’d already identified the distribution of federal food stamps as one such area. Here in Massachusetts, the BEACON software system is riddled with errors and misinterpretations of the law. In New York State, Federal District Court Judge Rakoff had similar invalidated similar software allocating that state’s food stamps (445 F.Supp.2d 400). Rather than going after broken code, we’re much more keen on identifying code not yet in place, where identifying and implementing some best practices can go much farther than litigating for change in a system already starved for resources.
- continued -
In the Terminator movie series, an artificially intelligent defense network concludes that the greatest threat to world peace is human beings and proceeds to launch preemptive world-wide nuclear strikes. This Oedipal fear that our silicon-and-code children might one day overthrow us – a favorite among dystopic science fiction authors – may be closer at hand than we’d like to think. Predator drones have yet to evolve to Schwarzenegger clones, but our fears about the inner workings of Diebold voting machines are real and understandable. Thus Professor Danielle Citron’s forthcoming article, “Technological Due Process,” provides a timely examination of an urgent problem challenging our computer-dependent society.
Professor Citron, of the University of Maryland School of Law, has written the opening chapter to an important new field of legal study, one hinted at in Lawrence Lessig’s “Code is Law” formulation and which I had dubbed “Law as Code,” but which until now has not received direct scholarly investigation. Her article, forthcoming in the Washington University Law Review, identifies the need for “technological due process” in software that executes government policies – from public benefits to no-fly lists.
In brief, Professor Citron describes how software code increasingly executes our public laws. Decision support systems, she convincingly argues, quickly become decision making systems. And invariably, the vagaries of the legislative and administrative processes leave large gaps in the specifics of how a given law should be executed. Without firmer guidance from proper governmental bodies, the programmers charged with translating legal code into software code essentially wind up creating law to fill the gaps. (I describe this as “shoving analog pegs into digital slots”). From a procedural – even a Constitutional – perspective, this is a grievously inappropriate delegation of governmental functions to the private sector, not unlike the hiring of Blackwater mercenaries to achieve military objectives. Professor Citron finds, therefore, the need for “technological due process”: safeguards to ensure that software is literally up to code.
In exploring what I’d described as “Law as Code,” and now significantly better-informed by the invaluable analysis of “Technological Due Process,” this fall the Berkman Center started seeking out areas of public law at risk of being improperly executed by unaccountable software code. In my work in legal aid I’d already identified the distribution of federal food stamps as one such area. Here in Massachusetts, the BEACON software system is riddled with errors and misinterpretations of the law. In New York State, Federal District Court Judge Rakoff had similar invalidated similar software allocating that state’s food stamps (445 F.Supp.2d 400). Rather than going after broken code, we’re much more keen on identifying code not yet in place, where identifying and implementing some best practices can go much farther than litigating for change in a system already starved for resources.
- continued -