Architecting Evaluation and Assessment—Using a Design-based Approach to Develop a Complete and Systematic Approach Presenters: Dr. Douglas Gilbert – University of the Rockies Dr. Peter Pintus – University of the Rockies Dr. Olin Oedekoven – Peregrine Academics @ACBSPAccredited #ACBSP2015 Agenda The Challenge of Assessment ACBSP Assessment Requirements Using a Design Approach to Architect a Model Multi-Tiered Assessment Instrumentation Examples Conclusions 2 @ACBSPAccredited #ACBSP2015 The Challenge of Assessment ACBSP Assessment Requirements Using a Design Approach to Architect a Model Multi-Tiered Assessment Instrumentation Examples Conclusions 3 @ACBSPAccredited #ACBSP2015 The Assessment Challenge: Four Key Questions 1. Do we know what graduates need to know and be able to do? 2. Does the curriculum provide what graduates need to know and be able to do? 3. Are students on a learning path for relevant knowledge, skills, and abilities? 4. Do graduates have the competencies demanded by employers? 4 @ACBSPAccredited #ACBSP2015 The Challenge of Assessment ACBSP Assessment Requirements Using a Design Approach to Architect a Model Multi-Tiered Assessment Instrumentation Examples Conclusions 5 @ACBSPAccredited #ACBSP2015 Key ACBSP Assessment Requirements • Criterion 4. 1 – Must have a “ learning outcomes assessment program” a. b. c. d. e. Program learning obj ectives (PLOs) An assessment process Internal assessment approach External assessment approach Formative and summative assessment approach • Criterion 4. 2 – Trends • Criterion 4. 3 – Time series benchmarked comparisons and also… 6 @ACBSPAccredited #ACBSP2015 Setting strategic direction • Standard 2 – “ Process … to better address key student and program performance requirements” 7 @ACBSPAccredited #ACBSP2015 The Challenge of Assessment ACBSP Assessment Requirements Using a Design Approach to Architect a Model Multi-Tiered Assessment Instrumentation Examples Conclusions 8 @ACBSPAccredited #ACBSP2015 What is a “ complete and systematic approach” ? • Complete —“ [ H] aving all necessary part s : not lacking anyt hing” (Merriam-Webst er. com. 2015. ht t p: / / www. merriam-webst er. com) • Systematic —“ The t erm "syst emat ic" ref ers t o approaches t hat are repeatable and use data and information so learning is possible . ” (ACBSP Bac/ Grad St andards and Crit eria—Rev J. , December 2014, p. 71). —“ Well-ordered, repeatable , and exhibit ing t he use of dat a and inf ormat ion so t hat learning is possible. ” (Baldrige Excellence Framework 2015-16, p. 53). • Approach —“ [ T] he methods used by an organizat ion t o address t he crit eria requirement s. Approach includes t he appropriateness of t he met hods t o t he requirement s and t he effectiveness of t heir use. ” (ACBSP Bac/ Grad St andards and Crit eria—Rev J. , December 2014, p. 62). 9 @ACBSPAccredited #ACBSP2015 Is this your assessment design? 10 @ACBSPAccredited #ACBSP2015 A Problematic Assessment System Assessment Advisory Committee Provost / VPAA Dean / Dept. Chair Strategic Planning VPs Institutional Research Faculty Governance Program Chairs Curriculum approval 11 Data: Student surveys and classroom work, accumulated by faculty in classroom Data: Departmental studies, e.g. accounting, OB, etc. Data: Programmatic assessments, e.g. portfolios. Uses: Individual Faculty Efforts Uses: Departmental Improvement Uses: Program reviews Adapt ed f rom Wolvoord (2010) Data: Institutional res-• Surveys (NSSE, EOCS, etc.) • Std-ized tests • IPEDS • Alumni Uses: Reporting Data: Academic support and co-curricular, e.g. library, IT, student affairs, etc. @ACBSPAccredited Uses: Individual dept. improvement #ACBSP2015 Common “ Disconnects” • Structure —Academic data islands of classroom, departments, and programs —Institutional and instructional data not integrated • Process —Blurred or non-existent assessment roles and responsibilities —Non-systematic or isolated cycles of improvement • Technology —“ One-off” faculty classroom technology approaches —Wide range of surveys and assessment automation • People —Separated, disj ointed assessment responsibilities —Holistic knowledge management competencies lacking —Little cross-functional / inter-departmental perspective 12 @ACBSPAccredited #ACBSP2015 A Design-Based Assessment Architecture Mission Vision Values Business Unit Strategy Process Structure Technology People Performance Measurement Data Definitions Technology Standards Knowledge Mgmt Competencies Performance Analysis / Review KPIs / SLOs Collection Technology Organizational Learning Focus Performance Improvement Control Limits Analysis Technology Process-based Organization Architecture Enablers Policies 13 Procedures Instruments @ACBSPAccredited #ACBSP2015 The Common Singular Focus Mission Vision Values Business Unit Strategy Process Structure Technology People Performance Measurement Data Definitions Technology Standards Knowledge Mgmt Competencies Performance Analysis / Review KPIs / SLOs Collection Technology Organizational Learning Focus Performance Improvement Control Limits Analysis Technology Process-based Organization Architecture Enablers Policies 14 Procedures Instruments @ACBSPAccredited #ACBSP2015 The Challenge of Assessment ACBSP Assessment Requirements Using a Design Approach to Architect a Model Multi-Tiered Assessment Instrumentation Examples Conclusions 15 @ACBSPAccredited #ACBSP2015 Outcome / Instrumentation Overview or Business Unit Standard 2 Today Tomorrow Student Focus Workforce Focus Customer Focus Leadership and Governance Focus Performance Management Financial Focus Yest erday performances (internal to school only) • Student publications or conference presentations • Employer and internship supervisor ratings of students' performance etc. • Job Placement • Department or program review data subject area tests • Employer or alumni surveys • Focus group interviews with students, faculty members, • Student perception surveys or employees • • • • • • Direct • Course evaluations • Grades not based on explicit criteria related to clear • Test blueprints (outlines of the concepts and skills learning goals • • • • covered on tests) Percent of class time spent in active learning Number of student hours spent on service learning Number of student hours spent on homework Number of student hours spent at intellectual or cultural activities related to the course Indirect Formative • Exams / Standardized tests Case study analyses Rubric scores for assignments, reports, and research projects Rubric scores for writing, oral presentations, and performances Observations of field work, internship performance, service learning, consulting projects Quizzes Class discussion participation External Summative Course Internal Assessment Academic Program • Capstone projects, senior theses, exhibits, or • Pass rates or scores on licensure, certification, or Formative 16 • Externally normed assessments, e.g. Peregrine, ETS, Summative Standard 4. 1c, d, e Standard 4. 1a, b Operational Environment Course Level Elements 17 Program Level Elements 18 Business Unit Elements 19 The Challenge of Assessment ACBSP Assessment Requirements Using a Design Approach to Architect a Model Multi-Tiered Assessment Instrumentation Examples Conclusions 20 @ACBSPAccredited #ACBSP2015 Overview of Examples 1. 2. 3. 4. 5. 21 College-level KPI Reports (Standard 2) Signature assignments using standard rubric (Standard 4. 1c) NPS – External Measure (Standard 4. 1d) Trending (longitudinal) dashboard (Standard 4. 2) External test benchmarking approach (Standard 4. 3) @ACBSPAccredited #ACBSP2015 College Dashboard - Layer 1 (National Baldrige Recipient Example) ACBSP Standard 2 22 Source: Dallas Communit y College Dist rict @ACBSPAccredited #ACBSP2015 College Dashboard - Layer 2 (National Baldrige Recipient Example) 23 Source: Dallas Communit y College Dist rict @ACBSPAccredited #ACBSP2015 College Dashboard - Layer 3 (National Baldrige Recipient Example ) 24 Source: Dallas Communit y College Dist rict @ACBSPAccredited #ACBSP2015 Signature Assignments Using a Standardized Rubric (Standard 4. 1c) Standardized Rubric 25 @ACBSPAccredited #ACBSP2015 NPS as an External Measure (Standard 4. 1d) Source: Bain & Co. ht t p: / / www. net promot ersyst em. com/ about / measuring-your-net -promot er-score. aspx 26 @ACBSPAccredited #ACBSP2015 Trending (longitudinal) dashboard (Standard 4. 2) 27 @ACBSPAccredited #ACBSP2015 External test benchmarking approach (Standard 4. 3) 28 @ACBSPAccredited #ACBSP2015 The Challenge of Assessment ACBSP Assessment Requirements Using a Design Approach to Architect a Model Multi-Tiered Assessment Instrumentation Examples Conclusions 29 @ACBSPAccredited #ACBSP2015 Putting it all together Signature Assignment Example Signature assignment aligned to Mission, Vision, & Values Process • Signature assignment selection Structure People • Rubric structure • LMS • Knowledge mgt competencies • Relevant student learning outcomes • Rubric software • CQI approach • Analysis software • SIPOC for fit of process • Scoring / rubric process • Cycles of review / improvement Technology • Targets Architecture Enablers Assessment Policies 30 Overall assessment cycle Rubric types @ACBSPAccredited #ACBSP2015 Additional Resources • Presentation available at http:/ / bitly. com/ ACBSP2015 • White paper to be available at same location 31 @ACBSPAccredited #ACBSP2015