Every day, companies are carrying out thousands of tests to develop and refine their products and increase their profitability. How can the same practices be used to guide the evaluation and development of administrative policy and the law, improving their effectiveness for the American people?

This question motivated a one-day, invitation-only workshop that brought together federal officials, academics, and members of the civil and private sectors to share best practices, increase capacity, and build a community of practice around the use of rigorous policy pilots in making law and policy.

The event was held on May 30, 2019 in Washington, DC. The proceedings were closed to the media and observed the Chatham House rule. The program featured lightning talks, panels, and interactive discussion.

The workshop was co-sponsored by the Administrative Conference of the United States , the Partnership for Public Service , the Columbia Center for Constitutional Governance , the Santa Clara University High Tech Law Institute , and the Penn Program on Regulation .

Participants from over a dozen federal agencies as well as academic and non-profit institutions addressed the following topics:

• Agency Experiences with Rigorous Policy Pilots and Lessons Learned
• Prospective and Retrospective Evaluation of Policy to Advance Mission
• Cross-Agency Coordination 
• Embedding RIgorous Evaluation in Policy- and Law-Making and Implementation

 If you would like more information, please contact the workshop’s organizing co-chairs, Colleen Chien (Columbia/Santa Clara) at cchien@scu.edu, or Todd Rubin (Administrative Conference of US) at trubin@acus.gov.

 

Workshop materials:

• Workshop Agenda
• Presentation slides - Colleen Chien (Santa Clara)
• Presentation slides - Christina Yancey (Dept. of Labor)
• Presentation slides - Dan Ryman (PTO)
• Presentation slides - Sharon Marsh & Cynthia Lynch (PTO)
• Presentation slides - Emma Rackstraw (J-PAL)
• Presentation slides - Michael Abramowicz (George Washington Law) & Ian Ayres (Yale Law)
• Presentation slides - Brian Scholl (SEC)

 

Federal Pilots and Evaluation Programs:

• USPTO, TM Post Registration Randomized Audit Program
• USPTO, TTAB Expedited Cancellation Pilot Program
• EPA, Compliance Assistance in the Auto Body Sector: A Statistically Valid Pilot Project
• DOL, DOL 2018 Evaluation Program (includes evaluations of numerous pilots)
• SEC, Regulation SHO — Pilot Program , Economic Analysis
• SEC, Tick-size Pilot Program , Statement of Expiration
• HUD, Moving to Opportunity Demonstration Project
• OCC, Lending Limits Pilot Program
• GSA, Office of Evaluation Sciences Website of Agency Program Implementation Pilots 
• Education, Institute of Education Sciences
• VA, Quality Enhancement Research Initiative
• EPA, EPA Evaluate
• CFPB, “Pitch a Pilot” program

 

Related readings:

• Colleen Chien, Rigorous Policy Pilots, Iowa Law Review (2019) (draws on case law and agency experience to show how using pilots to develop policy is feasible, legal, and useful).
• Foundations for Evidence-Based Policymaking Act of 2018, Public Law 115-435
• Administrative Conference of the United States (ACUS), Recommendation 2017-6: Learning from Regulatory Experience (2017)
• Zachary J. Gubler, Regulatory Experimentation (2017) (report to ACUS)
• Zachary J. Gubler, “Experimental Rules,” 55 B.C. L. REV. 129 (2014)
• Zachary J. Gubler, “Making Experimental Rules Work,” 67 ADMIN. L. REV. 551 (2015). 2
• Administrative Conference of the United States (ACUS), Recommendation 2014-5: Retrospective Review of Agency Rules (2014)
• Joseph A. Aldy, Learning from Experience: An Assessment of the Retrospective Reviews of Agency Rules and the Evidence for Improving the Design and Implementation of Regulatory Policy (2014) (report to ACUS)  
• Cary Coglianese & Todd Rubin, Learning What Works in Regulation , The Regulatory Review (2018)
• Todd Rubin, Hurdles in Building Public-Private Partnerships , The Regulatory Review (2019)
• Cary Coglianese, Measuring Regulatory Performance (OECD Report) 
• Cary Coglianese, Richard Zeckhauser & Edward A. Parson, Seeking Truth for Power: informational Strategy and Regulatory Policy , Minnesota Law Review (2004)
• Dan Ho & Becky Elias, Government Under Review, Boston Review (2016) (argues for government peer review)

• Why Economists Should Conduct Field Experiments and 14 Tips for Pulling One Off (provides an overview of the spectrum of experimental methods in economics)

• Michael Abramowicz, Ian Ayres & Yair Listokin, Randomizing Law , University of Pennsylvania Law Review (2011)
• Michael Abramowicz, Ian Ayres & Yair Listokin, Randomizing Regulation , The Regulatory Review (2012)

• Lisa Ouellette, Patent Experimentalism , Virginia Law Review (2015)

• Matthew Spitzer & Talley, Eric, “On Experimentation and Real Options in Financial Regulation,” 43 J. LEGAL STUD. S151 (2014)
• Yoon-Ho Alex Lee, “An Options Approach to Agency Rulemaking,” 65 ADMIN. L. REV. 881 (2013)

• Erin Winick, Universal Basic Income Had a Rough 2018 , MIT Technology Review (Dec. 27, 2018)

• Results for America, Invest in What Works Federal Standard of Excellence (highlights the work of 9 federal agencies to build infrastructure for evidence, data and evaluation in budget, policy and management decisions)

•  Phil Ames & James Wilson, Unleashing the Potential of Randomised Controlled Trials in Australian Governments , M-RCBG Associate Working Paper Series No. 55 (April 2016)

 

Other related links:

• Legal Services of America Research Study Opinion
• The Access to Justice Lab at Harvard
• Jameel Lateef Poverty Action Lab - North America
• Copyright Office Modernization Regulations and plan

• Results for America

*** 

 

 

 

-