Skip to main content area Skip to main content area Skip to institutional navigation Skip to search Skip to section navigation
Feedback

Optimizing Government Project

Reading Room

Notable research on the use of machine learning in government, policy-making, and regulation. Please email optimizing@law.upenn.edu if you would like to submit additional sources.

General Interest

Challenges

Applications

Methods

General Interest

U.S. Executive Office of the President, Preparing for the Future of Artificial Intelligence (2016). 

U.S. Executive Office of the President, The Administration’s Report on the Future of Artificial Intelligence (2016). 

U.S. Executive Office of the President, Big Data: Seizing Opportunities, Preserving Values (2014).  

U.K. House of Commons, Science and Technology Committee, Robotics and Artificial Intelligence (2016). 

Stuart Minor Benjamin, Algorithms and Speech, 161 U. Pa. L. Rev. 1445 (2013).

Sarah Bird, Solon Barocas, Kate Crawford, et al., Exploring or Exploiting? Social and Ethical Implications of Autonomous Experimentation in AI (Oct. 2016). 

Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (2014). 

danah boyd & Kate Crawford, Six Provocations for Big Data (presented at A Decade in Internet Time: Symposium on the Dynamics of the Internet and Society, 2011).

John Brockman (editor), What to Think About Machines That Think (2015).

Erik Brynjolfsson & Andrew McAffee, The Second Machine Age (2014). 

Ryan Calo, Robotics and the Lessons of Cyberlaw, 103 Calif. L. Rev. 513 (2015).

Cary Coglianese, Optimizing Government for an Optimizing Economy, University of Pennsylvania Institute for Law and Economics (June 2016). 

Cary Coglianese & David Lehr, Regulating by Robot: Administrative Decision-Making in the Machine-Learning Era, Geo. L.J. (forthcoming 2016). 

Pedro Domingos, The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World (2015).

The Return of the Machinery Question, The Economist (June 25, 2016).

Ed Felten, Accountable Algorithms, Freedom to Tinker (Sep. 12, 2012).

Martin Ford, Rise of the Robots: Technology and the Threat of a Jobless Future (2015). 

Dirk Helbing et al., Will Democracy Survive Big Data and Artificial Intelligence?, Scientific American (Feb. 25, 2017). 

Joshua A. Kroll et al., Accountable Algorithms, 165 U. Pa. L. Rev. 633 (2017).

Rory Van Loo, Rise of the Digital Regulator, 66 Duke L. J. 1267 (2017).

John Markoff, Machines of Loving Grace: The Quest for Common Ground Between Humans and Robots (2015).

Viktor Mayer-Schönberger & Kenneth Cukier, Big Data: A Revolution That Will Transform How We Live, Work, and Think (2013).

Cathy O’Neil, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (2016).

Cassidy R. Sugimoto, Hamid R. Ekbia & Michael Mattioli (editors), Big Data is Not a Monolith (2016). 

 

 

Challenges

Equal Protection and Bias

Solon Barocas & Andrew Selbst, Big Data’s Disparate Impact, 104 Cal. L. Rev 671 (2016). 

Nanette Byrnes, Why We Should Expect Algorithms to Be Biased, MIT Tech. Rev. (June 24, 2016).

Cynthia Dwork, et al., Fairness Through Awareness (Nov. 29, 2011).

Sorelle Friedler et al., Certifying and Removing Disparate Impact (presented at Fairness, Accountability, and Transparency in Machine Learning, 2014).

Toshihiro Kamishima et al., Fairness-Aware Classifier with Prejudice Remover Regularizer, in Machine Learning and Knowledge Discovery in Databases (Peter Flach, Tijl De Bie & Nello Cristianini eds., 2012).

Hannah Laqueur & Ryan Copus, Synthetic Crowdsourcing: A Machine-Learning Approach to Problems of Inconsistency and Bias in Adjudication (Oct. 21, 2016). 

Claire Cain Miller, Algorithms and Bias: Q. and A. With Cynthia Dwork, N. Y. Times (Aug. 10, 2015).

Katherine Noyes, Will Big Data Help End Discrimination—or Make It Worse?, Fortune (Jan. 15, 2015). 

Michael Schrage, Big Data’s Dangerous New Era of Discrimination, Harv. Bus. Rev. (Jan. 29, 2014).

Sonja Starr, Evidence-Based Sentencing and the Scientific Rationalization of Discrimination, 66 Stan. L. Rev. 804 (2014).

Latanya Sweeney, Discrimination in Online Ad Delivery, 56 Comms. of the Ass’n for Computer Machinery 44 (May 2013).

Laura Sydell, Can Computers Be Racist? The Human-Like Bias of Algorithms, NPR (March 14, 2016).

Kim Zetter, Researchers Sue the Government Over Computer Hacking Law, Wired (July 29, 2016).

Privacy

Steven M. Bellovin et al., When Enough is Enough: Location Tracking, Mosaic Theory, and Machine Learning, 8 N.Y.U. J. L. & Liberty 555 (2014).

Kate Crawford & Jason Schultz, Big Data and Due Process: Toward a Framework to Redress Predictive Privacy Harms, 55 B.C. L. Rev. 93 (2014).

Roger Allan Ford & W. Nicholson Price II, Privacy and Accountability in Black-Box Medicine, 22 Mich. Telecomm. & Tech. L. Rev. (forthcoming 2016).

Neil Richards & Jonathan King, Three Paradoxes of Big Data, 66 Stan. L. Rev. Online 41 (2013).

Omer Tene & Jules Polonestky, Privacy in the Age of Big Data: A Time for Big Decisions, 64 Stan. L. Rev. Online 63 (2012).

Applications

Criminal Justice

Julia Angwin et al., Machine Bias, ProPublica (May 23, 2016).

Julia Angwin & Jeff Larson, Bias in Criminal Risk Scores is Mathematically Inevitable, Researchers Say, ProPublica (Dec. 30, 2016). 

Richard Berk, Susan B. Sorenson & Geoffrey Barnes, Forecasting Domestic Violence: A Machine Learning Approach to Help Inform Arraignment Decisions, 13. J. Empirical L. Studies 94 (2016).

Richard Berk, Lawrence Sherman, Geoffrey Barnes, Ellen Kurtz & Lindsay Ahlman, Forecasting Murder Within a Population of Probationers and Parolees: A High Stakes Application of Statistical Learning, 172 J. Royal Stat. Soc’y Series A Stat. in Soc’y 191 (2009). 

Richard Berk & Jordan Hyatt, Machine Learning Forecasts of Risk to Inform Sentencing Decisions, 27 Federal Sentencing Reporter 222 (2015).

Richard Berk & Justin Bleich, Statistical Procedures for Forecasting Criminal Behavior, 12 Criminology & Public Policy 513 (2013).

Tim Brennan & William Oliver, The Emergence of Machine Learning Techniques in Criminology: Implications of Complexity in Our Data and in Research Questions, 12 Criminology & Pub. Pol’y 551 (2013).

Tim Brennan et al., Evaluating the Predictive Validity of the COMPAS Risk and Needs Assessment System, 36 Crim. Just. & Behav. 21 (2009).

Maurice Chammah, Policing the Future, The Verge (Feb. 3, 2016).

Alexandra Chouldechova, Fair prediction with disparate impact: A study of bias in recidivism prediction instruments (Oct. 2016). 

Andrew Guthrie Ferguson, Big Data and Predictive Reasonable Suspicion, 163 U. Pa. L. Rev. 327 (2015).

Maria Konnikova, The Future of Fraud-Busting, The Atlantic (March 2016).

Brian Kriegler & Richard Berk, Small Area Estimation of the Homeless in Los Angeles: An Application of Cost-Sensitive Stochastic Gradient Boosting, 4 Annals Applied Stat. 1234 (2010).

Michael L. Rich, Machine Learning, Automated Suspicion Algorithms, and the Fourth Amendment, 164 U. Pa. L. Rev. 871 (2016).

Andrew D. Selbst, Disparate Impact in Big Data Policing, Yale Information Society Project (Oct. 3, 2016). 

Matt Stroud, Chicago’s New Police Computer Predicts Crimes, But is it Racist?, The Verge (Feb. 19, 2014).

Environmental Protection

U.S. Environmental Protection Agency, Toxicology Testing in the 21st Century (Tox21).

Richard S. Judson et al., Estimating Toxicity-Related Biological Pathway Altering Doses for High-Throughput Chemical Risk Assessment, 24 Chemical Res. in Toxicology 451, 457-60 (2014).

Robert Kavlock et al., Update on EPA’s ToxCast Program: Providing High Throughput Decision Support Tools for Chemical Risk Management, 25 Chemical Res. in Toxicology 1287 (2012).

Cleridy Lennert-Cody & Richard Berk, Statistical Learning Procedures for Monitoring Regulatory Compliance: An Application to Fisheries Data, 170 J. Royal Stat. Soc. 671 (2007).

Huanxiang Liu, Xiaojun Yao, and Paola Gramatica, The Applications of Machine Learning Algorithms in the Modeling of Estrogen-Like Chemicals, 12 Combinatorial Chem. & High Throughput Screening 490 (2009).

Matthew Martin et al., Economic Benefits of Using Adaptive Predictive Models of Reproductive Toxicity in the Context of a Tiered Testing Program, 58 Systems Biology in Reproductive Med. 3 (2012).

Financial Regulation and Tax

Taxpayer Advocacy Service, IRS Policy Implementation Through Systems Programming Lacks Transparency And Precludes Adequate Review, in 2010 Annual Report to Congress 71 (2010).

U.S. Federal Deposit Insurance Corporation, Business Technology Strategic Plan 2013-2017 (2013).

The Government Office for Science, London, Foresight: The Future of Computer Trading in Financial Markets (2012).

Scott W. Bauguess, The Hope and Limitations of Machine Learning in Market Risk Assessment (Mar. 6, 2015).

Christopher Condon, Quest for Robo-Yellen Advances as Computers Gain on Rate Setters, Bloomberg (May 24. 2016).

David DeBarr et al., Relational Mining for Compliance Risk (presented at the Internal Revenue Service Research Conference, 2004).

Steve Donoho, Early Detection of Insider Trading in Option Markets (presented at the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2004).

Mikella Hurley & Julius Adebayo, Credit Scoring in the Era of Big Data, 18 Yale J.L. & Tech. 148 (2016). 

Amir Khandani et al., Consumer Credit-risk Models via Machine-learning Algorithms, 34 J. Banking & Fin. 2767 (2010).

Andrei A. Kirilenko & Andrew W. Lo, Moore’s Law vs. Murphy’s Law: Algorithmic Trading and Its Discontents, 27 J. Econ. Perspectives 51 (2013).

Jane Martin & Rick Stephenson, Risk-Based Collection Model Development and Testing (presented at the Internal Revenue Service Research Conference, 2005).

Shawn Mankad, George Michailidis & Andrei A. Kirilenko, Discovering the Ecosystem of an Electronic Financial Market with a Dynamic Machine-Learning Method, 2 Algorithmic Finance 151 (2013).

Scott D. O’Malia, Opening Statement at 12th Meeting of the Technology Advisory Committee (June 3, 2014).   

Johan Perols, Financial Statement Fraud Detection: An Analysis of Statistical and Machine Learning Algorithms, 30 Auditing: J. Practice & Theory 19 (2011).

Gregory Scopino, Preparing Financial Regulation for the Second Machine Age: The Need for Oversight of Digital Intermediaries in the Futures Markets, 2015 Col. Bus. L. Rev. 439 (2015).

Laws and Courts

Sean Braswell, All Rise for Chief Justice Robot!, OZY (June 7, 2015).

Mariano-Florentino Cuéllar, Cyberdelegation and the Administrative State, in Administrative Law From the Inside Out: Essays on Themes in the Work of Jerry Mashaw (Nicholas R. Carillo ed., forthcoming).

Of Prediction and Policy, The Economist (Aug. 20, 2016).

If Computers Wrote Laws: Decisions Handed Down by Data, The Economist (May 16, 2016).

Stephen Goldsmith & William D. Eggers, Governing by Network: The New Shape of the Public Sector (2004).

Wallis M. Hampton, Predictive Coding: It’s Here to Stay, Practical L. (May 5, 2014).

Smart Cities

Gabe Cherry, Google, U-M to Build Digital Tools for Flint Water Crisis, University of Michigan News (May 3, 2016).

Bechara Choucair, Jay Bhatt & Raed Mansour, How Cities Are Using Analytics to Improve Public Health, Harv. Bus. Rev. (Sept. 15, 2014).

Will Davies, Robot Amelia - A Glimpse of the Future for Local Government, The Guardian (July 4, 2016).

Brian Heaton, New York City Fights Fire with Data, Gov’t Tech (May 15, 2015).

Edward Glaeser et al., Crowdsourcing City Government: Using Tournaments to Improve Inspection Accuracy (National Bureau of Economic Research Working Paper No. 22124, 2016).

Stephen Goldsmith & Susan Crawford, The Responsive City: Engaging Communities Through Data-Smart Governance (2014).

Will Knight, Can Machine Learning Help Lift China’s Smog?, MIT Tech. Rev. (Mar. 28, 2016).

Ian Lovett, To Fight Gridlock, Los Angeles Synchronizes Every Red Light, N.Y. Times (Apr. 1, 2013).

David Morris, How Swarming Traffic Lights Could Save Drivers Billions of Dollars, Fortune (July 13, 2015).

Mohana Ravindranath, In Chicago, Food Inspectors Are Guided by Big Data, Wash. Post (Sept. 28, 2014).

Nick Rojas, Chicago and Big Data, TechCrunch (Oct. 22, 2014).

Methods

Martín Abadi, et al., Deep Learning with Differential Privacy (July 2016).

Julius Adebayo & Lalana Kagal, Iterative Orthogonal Feature Projection for Diagnosing Bias in Black-Box Models (Nov. 2016).

Emrah Akyol, Cedric Langbort & Tamer Basar, Price of Transparency in Strategic Machine Learning (Oct. 2016).

Aws Albarghouthi, Loris D’Antoni, Samuel Drews & Aditya Nori, Fairness as a Program Property (Oct. 2016).

Ethem Alpaydin, Introduction to Machine Learning (2d ed. 2009).

David Barber, Bayesian Reasoning and Machine Learning (2012).

Richard Berk, Statistical Learning from a Regression Perspective (2008).

Richard Berk, Criminal Justice Forecasts of Risk (2012).

Christopher Bishop, Pattern Recognition and Machine Learning (2006).

Leo Breiman, Statistical Modeling: The Two Cultures (With Comments and a Rejoinder by the Author), 16 Stat. Sci. 199 (2001).

L. Elisa Celis, Amit Deshpande, Tarun Kathuria & Nisheeth K. Vishnoi, How to be Fair and Diverse (Oct. 2016). 

Bertrand Clarke, Ernest Fokoué & Hao Helen Zhang, Principles and Theory for Data Mining and Machine Learning (2009). 

Miguel Ferreira, Muhammad Bilal Zafar & Kirshna P. Gummadi, The Case for Temporal Transparency: Detecting Policy Change Events in Black-Box Decision Making Systems (Oct. 2016).

Sorelle Friedler et al., On the (Im)possibility of Fairness (Sept. 2016).  

Ian Goodfellow, et al., Deep Learning (2016). 

Alex Graves, Supervised Sequence Labelling with Recurrent Neural Networks (2008).

Trevor Hastie, et al., The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2009).

Moritz Hardt, Eric Price & Nathan Srebo, Equality of Opportunity in Supervised Learning (2016).  

Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern & Aaron Roth, Fair Learning in Markovian Environments (Nov. 2016). 

I. Jordan & T. M. Mitchell, Machine Learning: Trends, Perspectives, and Prospects, 349 Science 255 (2015).

Matthew Joseph, Michael Kearns, Jamie Morgenstern, Seth Neel & Aaron Roth, Rawlsian Fairness for Machine Learning (Nov. 2016). 

Matthew Joseph, Michael Kearns, Jamie Morgenstern & Aaron Roth, Fairness in Learning: Classic and Contextual Bandits (May 2016).

Jon Kleinberg, Sendhil Mullainathan & Manish Raghavan, Inherent Trade-Offs in the Fair Determination of Risk Scores (Nov. 2016).

Pat Langley, The Changing Science of Machine Learning, 82 Machine Learning 275 (2011).

Kristian Lum & James Johndrow, A statistical framework for fair predictive algorithms (Oct. 2016). 

David MacKay, Information Theory, Inference, and Learning Algorithms (2003).

William Rand, Machine Learning Meets Agent-Based Modeling: When not to Go to a Bar (Nw. Univ. working paper, 2006).

Richard S. Sutton & Andrew G. Barto, Reinforcement Learning: An Introduction (2015).

Ke Yang & Jula Stoyanovich, Measuring Fairness in Ranked Outputs (Oct. 2016). 

Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, et al., Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Measurement (Oct. 2016).