Atlas Examination Standards

examhallTesting learning outcomes for the MPP/MPA core

As described in Atlas Standards, our goal is to develop Atlas Examination Standards – a set of 120 topic-level learning outcomes, with a grading rubric and Concept Comprehension Questions, that will cover the full MPP/MPA Core Curriculum.

Learning outcomes on the Atlas can be articulated at three different levels: 1) for each of the 7 MPP/MPA Core Competencies; 2) for each of the 17 core Subjects; and 3) for each of the 120 core normed Topics. A first draft of learning outcomes at the first two levels can be found at MPP/MPA Core Learning Outcomes. The Atlas Examination Standards would provide learning outcomes at the third level – a level where assessment can be concrete and measurable.

Program assessment – How would our graduates do on the Atlas exam?

Could the Atlas examination standards help with program assessment? It depends. For many programs, the standards will be overly demanding because students are not expected to learn all that we have included in the MPP/MPA core curriculum. The core curriculum and the Atlas standards are designed to be “NASPAA Plus” in that they are informed by what is taught in the required courses of leading programs, specifically the four Reference Programs of Harvard, NYU, Michigan, and Wisconsin. Indeed, a cursory review of the course descriptions on program websites would suggest that relatively few of the 119 programs in the Atlas database currently offer a curriculum and course of study that would meet the Atlas standards. On the other hand, for programs that aspire to raise their academic standards closer to those in the leading MPP and MPA schools, the Atlas standards could provide useful insights, even as a work in progress. For these programs, the question “How would our graduates do on the Atlas exam?” could contribute to the self-assessment process.

Student assessment – A grading rubric from B- to A+

Different students will acquire the necessary knowledge and skills associated with each topic with different levels of comprehension, ranging from just passable (B-) to exceptional (A+). The Atlas grading rubric distinguishes between vocabulary knowledge, subject-matter intuition, and ability to calculate:

“B” level comprehension – knowing the vocabulary

  • B-: demonstrable by achieving passing marks on Concept Comprehension Questions
  • B: demonstrable by achieving high marks on Concept Comprehension Questions
  • B+: demonstrable by the ability to write out coherent a definition (such as that in the first sentence of Atlas concept entries) for a high proportion of core concepts

“A” level comprehension – knowing the vocabulary and applying the concepts

  • A-: demonstrable by achieving high marks on multiple choice questions requiring subject-matter intuition, such as the impacts of a policy intervention or a change in a variable
  • A: demonstrable by achieving high marks on problem sets requiring calculations or essays relating core concepts to policy and management questions
  • A+: demonstrable by achieving very high marks on problem sets requiring calculations or essays relating core concepts to policy and management questions
What is meant by mastery?

This grading rubric helps to define what we mean by mastery in our Atlas U claim that Atlas Courses provide “free online access to the learning materials needed to master the core topics and concepts taught at the world’s leading MPP and MPA programs.” There are different levels of mastery. We believe that it should be possible for a diligent learner to attain, using Atlas resources alone, an A- mastery of some subjects in the MPP/MPA core (particularly Economic AnalysisMacroeconomic Policy, and Quantitative Methods where the Atlas courses provide links to superb online pedagogy) and at least B+ mastery of the remainder of the MPP/MPA core. For most learners, we believe that achieving mastery at the A or A+ level would require enrolment in a leading MPP or MPA program.

Applying the standards to assess learning comprehension

Below are two examples, one from economic analysis and the other from leadership skills, of how the learning of a core topic could be assessed.

Example 1 – Producer Theory and Competition, a core topic in Economic Analysis

Learning outcome: Appropriately utilize and interpret results of applying the principles of producer theory and competition, including the identified core concepts, to the analysis of public policy and management problems. (The identified core concepts include: Production Possibility Frontier; Factors of Production; Production Function; Variable Input; Fixed Input; Short Run; Long Run; Economies of Scale; Returns to Scale; Isoquant Curve.)

  • Students could demonstrate B- level comprehension of producer theory and competition by achieving 80% to 89% on Concept Comprehension Questions (without access to reference material) of the meaning of all the core concepts and terms associated with this topic.
  • Students could demonstrate B level comprehension of producer theory and competition by achieving 90% to 100% on Concept Comprehension Questions (without access to reference material) of the meaning of all the core concepts and terms associated with this topic.
  • Students could demonstrate B+ level comprehension of producer theory and competition by being able to write (without access to reference material) a coherent one-sentence definition for 90% of the core concepts and terms associated with this topic.
  • Students could demonstrate A- level comprehension of producer theory and competition by achieving 90% to 100% on both the Concept Comprehension Questions plus multiple choice questions requiring subject-matter intuition, such as the following (from Cowen and Tabarrok at MRU university, and reproduced at Producer Theory and Competition):

  • Students could demonstrate A level comprehension of producer theory and competition by achieving results above plus achieving 80% to 90% on questions requiring calculation such as the following (from Jonathan Gruber in MIT OpenCourseWare and reproduced at Producer Theory and Competition):

ProbSet4

  • Students could demonstrate A+ level comprehension of producer theory and competition through the same tests as those for A level comprehension, but with even higher marks.

Example 2 – Recognizing Values, a core topic in Leadership Skills

Learning outcome: Recognition of the importance of taking account of differences in value systems and personality types and awareness of the most commonly referenced frameworks for characterizing these differences, including the identified concepts. (The identified concepts include: Concept Creep; Haidt’s 6 Innate Moral Foundations; Intrinsic Motivation; Motivation; Myers-Briggs Personality Type Indicator (MBTI).)

  • Students could demonstrate B- level comprehension of the role played by values in public policy and management by achieving 80% to 89% on Concept Comprehension Questions (without access to reference material) of the meaning of all the core concepts and terms associated with this topic (e.g.,
  • Students could demonstrate B level comprehension of the role played by values in public policy and management by achieving 90% to 100% on Concept Comprehension Questions (without access to reference material) of the meaning of all the core concepts and terms associated with this topic.
  • Students could demonstrate B+ level comprehension of the role played by values in public policy and management by being able to write (without access to reference material) a coherent one-sentence definition for 90% of the core concepts and terms associated with this topic.
  • Students could demonstrate A-, A, or A+ level comprehension of the role played by values in public policy and management through two-page-essay responses to questions such as:
  1. Summarize the survey findings on gender differences in Haidt’s 6 Innate Moral Foundations and Myers-Briggs Personality Type Indicator (MBTI), and describe how you think this might influence gender differences in styles of leadership such as Lewin’s 3 Leadership Styles.
  2. Critique Jonathan Haidt’s claim (The Most Dangerous Creep On Campus (http://heterodoxacademy.org/2016/04/09/the-most-dangerous-creep/) that innate moral foundations help to explain recent university policies in a number of areas such as trigger warnings, and comment on the extent to which the claim may be relevant for regulation making in government.

Page created by: Ian Clark, last modified 2 February 2017.

Images: AusTop Group, at http://www.austop.com.au/%E7%95%99%E5%AD%A6/2015-hsc%E7%B2%BE%E8%8B%B1%E8%BE%85%E5%AF%BC/, accessed 21 May 2016.