An Outcomes-based Framework for Evaluating MPP and MPA Programs

… back to Competencies or to Core Curriculum Project

Figure 1. The three foundational elements of an evaluation framework where credible student learning outcomes can link second-level professional competency categories to second-level curricular content categories

Using detailed learning outcomes to relate curricular content to the specific skills and knowledge that underlie the competencies that are judged important to professionals

This page proposes a framework for evaluating the effectiveness of an MPP or MPA program at producing graduates with the skills and knowledge that professionals deem to be important in their work.

Our proposed framework has three foundational elements, which are depicted in Figure 1 and can be summarized as follows:

  1. a typology of professional competencies (skills and knowledge) where:
    1. top-level competency categories are collectively exhaustive – so that they cover all the areas deemed important by professionals
    2. second-level professional competencies are sufficiently fine-grained so that each can be associated with a specific skill or topic
    3. each second-level professional competency is articulated in a way that facilitates its association with at least one specific learning outcome
  2. a typology of curricular content where:
    1. top-level content categories are collectively exhaustive – so that they cover the MPP and MPA core curriculum
    2. second-level content categories, called topics, are sufficiently fine-grained so that at least one can be associated with each course-week of instruction
    3. each topic is articulated in a way that facilitates its association with at least one specific learning outcome
  3. a set of learning outcomes where:
    1. student mastery of each learning outcome can be credibly assessed
    2. the set of learning outcomes is collectively exhaustive – so that it covers all the core courses and all the professional competencies of interest
    3. each learning outcome is articulated in a way that facilitates its association with the curricular content addressed in a specific week of study in a core course and at the same time facilitates its association with a specific second-level professional competency

This description illustrates the idealized nature of the foundational elements. The evaluation framework requires that the two typologies and the set of learning outcomes use categories that are collectively exhaustive, i.e., that they cover the whole field of interest. In an idealized version of the framework, the categories would also be mutually exclusive within their higher-level category and unique to that higher-level category. However, the mutual exclusivity and uniqueness properties are not crucial to the framework and are very hard to achieve in practice, as we will see below in the typologies used to describe professional competencies or to review programs.

Examples of professional competency typologies

The U.S. Office of Personnel Management has developed a two-level typology to describe the core qualifications for senior executives (in all fields, not just the policy profession). It is summarized at SES Executive Core Competencies. The 6 top-level categories (“fundamental competencies” and 5 “executive core qualifications (ECQs)”) encompass a total of 29 second-level categories. The second-level categories (for example, “interpersonal skills”) within the top-level category, fundamental competencies, are specifically designed not to be unique to that category but to apply to all of the other 5 top-level categories. But most of the second-level categories in these 5 top-level categories are not unique to their top-level category. For example  “flexibility” is not unique to its first-level category, “ECQ1 Leading Change,” and “decisiveness” is not unique to its first-level category, “ECQ3 Results Driven.”

The most finely-grained and collectively exhaustive typology of competencies for policy professionals is likely UK Policy Professional Standards 2019. This is a three-level typology with 3 top-level categories known as “areas of competence” (analysis and use of evidence; politics and democracy; and policy delivery), each of which has 6 second-level categories known as “sub-areas” and each of these 18 second-level categories has 4-9 third-level categories known as “skills.” The typology articulates third-level categories (skills) at each of three levels in policy profession hierarchy: an entry-level policy professional (level 1); a “skilled policy professional” (level 2); and “effective policy leader (level 3).” The second- and categories are relatively mutually exclusive within their first-level category and relatively unique to their first-level category. This is less the case for third-level categories. For example, the third-level categories, “Uses decision-making tools such as Impact Assessments to analyse the feasibility of options,” and “Facilitates effectively the early involvement of the people who will deliver the policy, and those impacted by it” are not unique to their second-level category, “Understanding the Delivery Context and Effective Implementation Planning.”

Examples of competency typologies created by accrediting bodies

As described on the Atlas page, NASPAA Competencies, the Commission on Peer Review and Accreditation (COPRA) of the Network of Schools of Public Policy, Affairs and Administration (NASPAA) has developed the NAPAA Standards. These include five first-level competency categories that NASPAA calls “domains of universal required competencies” (Standard 5.1). The entities that constitute second-level competency categories are referred to as “competencies” and also as “student learning outcomes.” Table 1 lists the top-level competencies and the number of second-level competencies in each.

Table 1: NASPAA’s competency typology
Top-level competency category
(domains of universal required competencies)
Number of second-level categories
(competencies / learning outcomes)
To lead and manage in the public interest

18

To participate in, and contribute to, the policy process

11

To analyze, synthesize, think critically, solve problems and make evidence-informed decisions in a complex and dynamic environment

11

To articulate, apply, and advance a public service perspective

10

To communicate and interact productively and in culturally responsive ways with a diverse and changing workforce and society at large

12

As described on the Atlas page, CAPPA Standards for Student Competencies, in 2016, the Canadian Association of Programs in Public Administration (CAPPA) published on its website a new set of standards and procedures. It has only one level of competency categories, which are somewhat similar to the NASPAA top-level categories:

  1. The ability to analyze and think critically about public sector problems;
  2. The ability to lead and manage within public organizations;
  3. Knowledge and understanding of the tools and techniques required to engage stakeholders in policy and governance processes;
  4. An appreciation of the purpose of public service and associated standards of ethical behaviour;
  5. A capacity to communicate and interact both professionally and productively with a diverse and changing citizenry.

As described on the Atlas page, UNDESA/IASIA Standards, “the Task Force on Standards of Excellence for Public Education and Training was initiated by the Division of Public Administration and Development Management (DPADM), Department of Economic and Social Affairs (DESA) of the United Nations (UN) in partnership with the International Association of Schools and Institutes of Administration (IASIA) in July of 2005.” The task force produced a report in 2008 that includes a typology of “curriculum components” with five top-level categories, each of which has 4-7 second-level categories.

Typologies that include magnitude

If we are eventually to evaluate effectiveness, we will need to quantify some of the inputs and outputs.

How much instruction or work experience would be required to attain a specific competency? The typologies above logically imply that a higher-level competency requires more instruction and experience than any of its second-level competencies but none of the typologies offers and indication of relative magnitude among categories at the same level.

Two typologies that do include an indication of magnitude are the typology of curricular content described in Light’s Typology of Standard Core and Skills for Success, which provides 8 top-level categories, and the Atlas typology of core curricular content, which has three levels: the top-level categories are the 17 core Subjects; the second-level categories are the 120 core Topics; and the third-level categories are the roughly one thousand core Concepts. As described in the Atlas Framework for Curricular Analysis, the top-level categories are reasonably close to being collectively exhaustive and the second-level categories are somewhat less so. The exclusivity and uniqueness properties are very hard to achieve for most topics and concepts and so each topic and concept is assigned to the higher level category to which they are “most strongly associated.”

Table 2 compares magnitudes, expressed as number of one-semester courses, in the two typologies.

Table 2: Comparison of Light’s Standard Curriculum Topics
and the most closely associated Atlas Core Subjects
with number of one-semester required courses
Light’s Standard Curriculum Topic
Atlas Core Subject
Micro-Economics/Policy Analysis
(1.6 courses)
Economic Analysis (1.0 courses); Analytic Methods (0.5 courses); Evaluation and Performance Measurement (0.5 courses)

(Total = 2.0 courses)

Quantitative Methods
(1.6 courses)
Quantitative Methods
(1.0 courses)
Organization Theory/Public Management
(1.0 courses)
Implementation and Delivery (1.0 courses); Governance and Institutions (1.0 courses); Human Resource Management (0.25 courses); Information and Technology Management (0.25 courses)

(Total = 2.5 courses)

Policy Process
(0.5 courses)
Policy Analysis and Process
(1.0 courses)
Leadership
(0.2 courses)
Leadership Skills (0.75 courses); Communication Skills (0.25 courses)

(Total = 1.0 courses)

Ethics
(0.4 courses)
Ethics, Rights, and Accountability
(0.5 courses)
Public Budgeting and Finance
(0.8 courses)
Public Financial Management
(0.5 courses)
General Introduction
(.4 courses)
Socioeconomic and Political Context (0.5 courses); Global Context (0.25 courses); Macroeconomic Policy (0.5 courses); Environment and Sustainability (0.25 courses)

(Total = 1.5 courses)

Table 2 illustrates that the two typologies are fairly closely aligned given the differences in in sample populations. The Light typology was developed in 1998 from the curricula of top-ranked American MPP and MPA programs while the Atlas typology was developed in 2015 from the curricula of the 120 MPP and MPA programs in 17 countries listed in Programs.

The Atlas core subjects are organized into 7 top-level categories in MPP/MPA Core Competencies, as displayed in Table 3, which includes our estimate of the number of course-weeks in an MPP or MPA program required to reach a professional level of mastery.

Table 3: The 7 top-level categories in MPP/MPA Core Competencies,
with number of course-weeks to achieve a professional level of mastery
  1. Applying economics and statistics concepts to public management problems (30 course-weeks)
  2. Applying analytic methods and evaluation concepts to public management problems (12 course-weeks)
  3. Understanding the socioeconomic, political and global contexts of public management (12 course-weeks)
  4. Applying policy and management analysis concepts to public management problems (18 course-weeks)
  5. Understanding democratic institutions and processes, and acting ethically (24 course-weeks)
  6. Managing financial, human and information resources in public and nonprofit institutions (12 course-weeks)
  7. Leading and communicating in public management (12 course-weeks)

Looking at Light’s Typology of Standard Core and Skills for Success, it would be interesting to try to place the 13 “skills for success” into a new, collectively exhaustive, typology. Some of the skills (for example, “influencing policy makers”) might turn out to be top-level categories and others (for example, “writing regulation and legislation”) might turn out to be second-level categories. And some (for example, “managing medial relations”) may turn out to be relatively unique to a single first level category (such as communicating with the public) while others (for example, “maintaining ethical standards”) might turn out to be a second-level category designed to be associated with all the first-level categories.

The challenge of developing detailed learning outcomes

In our proposed framework, detailed learning outcomes would be developed to describe the specific second-level professional competencies that specific topics in the core courses of an MPP/MPA curriculum are trying to teach. The learning-outcome-writing task would be integrated with the curriculum-design task – to determine, in consultation with practitioners, the most important skills and knowledge needed, and then to express them through approximately 120 detailed learning outcomes, one for each course-week of instruction in the equivalent of 10 one-semester core courses.

As discussed below, we estimate that about 120 such detailed learning outcomes would be needed. This is roughly twice as many as NASPAA’s 62 learning outcomes (second-level competencies) within its 5 top-level competencies (see Table 1 above and NASPAA Competencies).

Each learning outcome would need to be articulated in sufficient detail to guide the creation of a student learning assessment tool to assess mastery of the competency. The assessment approaches noted in Table 1 of NASPAA Competencies show how this might be done. The approach used on the Atlas makes heavy use of Concepts, the third-level curricular content categories. Most of the 120 Topics have an Atlas page with a Topic learning outcome that is expressed in terms of attaining comprehension of a collection of 5-20 specified Core concepts associated with this topic. Many of the Atlas topic pages also include a set of Concept comprehension questions, which provide “for each core topic, a set of multiple choice questions that will enable both the learner and the instructor to assess the comprehension of the core concepts associated with the topic” [see Concept Comprehension Questions]. What is still lacking in the Atlas approach is a rigorous connection between each topic and a specific second-level professional competency. Some progress can be seen at Policy Profession Competencies and Atlas Topics, where each of the second-level competency categories in the typology derived from UK Policy Profession Standards 2019 has associated with it the Atlas’s Most aligned subject and topics.

Creating detailed learning outcomes is, of course, easier said than done. Although devotees of Scholarship of Teaching and Learning [see, for example, the University of Washington’s list of annual conferences at https://www.washington.edu/teaching/programs/teaching-and-learning-symposium/scholarship-of-teaching-and-learning/sotl-annual-conferences/] have long encouraged instructors to include learning outcomes in their syllabi, it is hard to find many syllabi that contain detailed learning outcomes. Similarly, although governments have, from time to time, encouraged publicly-funded universities to articulate learning outcomes for their programs [see, for example, ANU Crawford Learning Outcomes and Melbourne MPPM Learning Outcomes], relatively few among the 120 listed in Programs appear to have done so.

Effectiveness and efficiency

If one were able to evaluate the extent to which a program or a course contributed to specific professional competencies, it would be possible to make comparisons of effectiveness (say, the contribution to the desired outcome per unit of cost) and efficiency (say, the contribution to a specific output per unit of cost). Costs could be normalized to the number of one-semester courses, or to the monetary costs borne by students and other funders, particularly governments.

Page by: Ian Clark, last modified 8 March 2020.

Image: Created on PowerPoint by Ian Clark, 3 March 2020.