Encouraging Experimentation

… a core concept used in Implementation and Delivery and Atlas107

Click for document website

Concept description

Experimentation is a key activity for a learning organizations (see Building a Learning Organization) and governments can take active measures to encourage experimentation, such as the Experimentation direction for Deputy Heads referenced below and linked on right.

The experimentation direction provides context and directions for deputy heads and departmental managers in the government of Canada on how to implement the “Government’s commitment to devote a fixed percentage of program funds to experimenting with new approaches and measuring impact to instill a culture of measurement, evaluation and innovation in program and policy design and delivery.”

Defining experimentation

The document defines experimentation as “[t]esting new approaches to learn what works and what does not work using a rigorous method that could feature:

  • deliberate, thoughtful, and ethical experimental design;
  • comparisons between interventions and base cases to capture evidence (e.g., randomized controlled trials, A/B testing, counterfactual experiments, baseline performance data, pre- and post-tests);
  • randomized assignment to test and control groups, whenever possible;
  • rigorous impact measurement and causality assessment; and
  • transparent publication of positive, negative and neutral results.”

The document states that:

“Features such as randomization, analysis to demonstrate statistical significance of results, and strong counterfactuals may not always be possible; smaller-scale or other experiments that do not meet such requirements should not necessarily be ruled out, but rather every effort should be made to establish as rigorous as possible baseline information and impact assessment strategies.

“In this context, new and innovative approaches to be tested include but are not limited to tools and methods such as:

  • user-centred design;
  • behavioural insights;
  • open policy-making, including co-creation approaches to develop policy, programs, and services with stakeholders, civil society, citizens, and other governments;
  • staged funding approaches to enable scaling;
  • outcomes-based and pay-for-performance funding, including social and development finance and Social Impact Bond-like instruments;
  • gamification;
  • data analytics and modelling;
  • investing in pooled funds that use these tools and methods.”
Requirements

The document sets out the expectations for deputy heads and departmental mangers, which include:

“Deputy Heads are expected to identify a percentage of program funds that their organization is currently devoting or plans to devote to experimenting with new approaches and to report on their experimentation efforts in the 2017-18 Departmental Plan.

“Departmental managers are expected to foster work environments that are conducive to experimentation, innovation and intelligent risk-taking so that public servants try new approaches and are not reprimanded for well-managed risks that fail to produce improvements, so long as lessons are captured and reflected in subsequent plans. …

“Departments are expected to share the results of their experiments, positive, negative or neutral/null, as broadly as possible, with a strong default to public release.

“Central agencies and enabling departments are responsible for establishing or providing access to experimentation training and resources for public servants. Where they have the appropriate expertise and specialized skills, federal hubs and labs are expected to support department-led experiments and disseminate lessons learned. Departments are also encouraged to consult and partner with external experts and organizations, as appropriate, to support high quality, ethical experimentation. …

“Performance information from experiments, regardless of whether they result in demonstrable improved results, will be shared openly by Deputy Heads and results will be reported publicly through Departmental Plans and Departmental Results Reports.”

Central agency support

The document states:

“TBS and PCO will work to help create the conditions for implementing rigorous experimentation approaches into the core business of departments and agencies. This includes working with Departments to ensure the enabling authorities are in place to support experimentation; helping to build capacity; providing practical tools and resources; and leveraging existing platforms and reporting structures so that departments can track and share experiences and showcase success. To support this work, TBS and PCO will establish an ADM-led interdepartmental mechanism on experimentation.”

Atlas topic, subject, and course

Promoting Learning (core topic) in Implementation and Delivery and Atlas107

Sources

Innovation Hub, at https://www.canada.ca/en/innovation-hub/services/reports-resources/experimentation-direction-deputy-heads.html, accessed 16 October 2017.

Page created by: Ian Clark, last modified 19 October 2017.

Image: Innovation Hub, at https://www.canada.ca/en/innovation-hub.html, accessed 16 October 2017.