La Cuestión Universitaria
Boletín Electrónico de la Cátedra UNESCO de Gestión y Política Universitaria
Universidad Politécnica de Madrid

Accountability and student learning outcomes in the USA

Resumen / Abstract

Over the past two decades, there have been a variety of efforts to shift the primary purpose of US higher education institutions to the production of learning rather than the delivery of instruction (Barr and Tagg, 1995). In 1983, A Nation at Risk (U.S. Department of Education, 1983) raised concerns about elementary, secondary and postsecondary educational outcomes. Other reports such as Access to Quality Undergraduate Education (Southern Regional Education Board, 1985), Integrity in the College Curriculum (Association of American Colleges, 1985), and To Reclaim a Legacy (Bennett, 1984) called for assessment of student learning (Ewell, 2002; Huba & Freed, 2000). Recently, a commission convened by US Education Secretary Margaret Spellings released A Test of Leadership: Charting the future of U.S. higher education (2006). The Spellings Commission noted "...a remarkable absence of accountability mechanisms to ensure that colleges succeed in educating students" (US Department of Education, p. xii).

These national calls for institutional accountability for student learning outcomes occur in a complex context where there is no direct federal control over higher education. The US Constitution gives the states responsibility for education. Authority structures across the 50 states range from consolidated governing boards over all higher education institutions in a state, to coordinating boards to planning boards which may offer advice but have little direct control over a state´s public colleges and universities (McGuiness, 2003). In addition, many private US colleges and universities operate independent of state control. Quality of academic programs in public and private institutions is assured by voluntary regional and specialized accrediting agencies rather than by government control. Six major regional accrediting agencies review public and private institutions. Many professions, ranging from medicine to engineering to business, have specialized accrediting agencies that review academic preparation programs in their fields across institutions.

Although there is no central locus of quality control for higher education, there is increasing convergence of opinion and policy efforts to assess student learning outcomes. For the purposes of this paper, learning outcomes will be defined as "the knowledge, skills and abilities that a student has attained at the end (or as a result) of his or her engagement in a particular set of higher education experiences" (CHEA, 2006, p. 1). The federal government now requires all accreditation agencies to incorporate assessment of student learning as part of their quality reviews. Associations such as the State Higher Education Officers Association and the Education Commission of the States provide opportunities for state executive officers and legislators to exchange information and learn from each others´ policy experiences. A recent policy brief from another such group, the National Governors´ Association (NGA), highlighted two primary reasons for US states to increase their efforts to have institutions assess student learning outcomes (NGA, 2007): the need to ensure college access and achievement for an increasingly diverse student population who will comprise the future US workforce, and to facilitate mobility for the many students who attend more than one institute as they progress toward their degrees.

The NGA issue brief, Higher Education for Student Learning, also reported results of a study of institutional assessment practices (2007). More than three-fourths of the institutions surveyed have explicit learning goal statements, and 55 percent have, or are working on institution-level plans to assess the extent to which the goals are achieved. Given the lack of centralized governance of US higher education, as well as the large number of US colleges and universities, the diversity of control, size, and mission, the study found remarkable consistency among institutions for the types of general learning goals held for students, including "communication, critical thinking, and numeracy skills; computer/information technology skills, and ethics" in addition to mastery of major subject areas (NGA, 2007, p. 3). The NGA issue brief recommends several ways to incorporate student learning outcomes into state accountability systems for higher education, including creating new state-wide surveys or examinations of student learning, aggregating data from existing nationally-normed exams already conducted by professional associations, or mandating institutions to develop assessments and report the results publicly. These policies reflect NGA´s increased reliance upon regulatory methods for inducing faculty to adopt learning outcomes as a primary method for assessing the quality of the educational environment.

1. Case Studies Examining Policies to Enhance Learning: Top-down Policies and Bottom-up Practice

Likely institutional and faculty responses to similar policy models addressing learning were investigated in a recent study, Enhancing Faculty Contributions to Learning Productivity, funded by the US Department of Education. A top-down analysis of policy implementation examined three primary state policy initiatives designed to improve undergraduate teaching and learning at institutions in states with differing governance structures and policy environments. Ohio offered an example of a "regulatory" environment in which faculty not only must report their work activities at the state level, but substantive measures were enacted to regulate faculty work. In the "monitoring" environment in Texas, faculty must report their work activities at the state level. Tennessee provided an example of an "inducement" environment; state policies placed more emphasis on outcomes assessment than on faculty work, and any monitoring of faculty work was done at the institutional rather than at the state level. By contrast, a bottom-up analysis (Colbeck, Fairweather, Brown, Beach, & Fingers, 2001), began with the end in mind, asking: what should new graduates have learned? The next step involved ascertaining how faculty should teach to foster productive student learning.

In each state, case studies were conducted at the premier public research university, a public Master´s level university, and a private liberal arts college. On each campus, the research team interviewed administrators concerned with undergraduate education and students, faculty, chairs, and deans from four departments selected for disciplinary variation: English (soft-pure), Physics (hard-pure), Business Management (soft-applied), and Mechanical Engineering (hard-applied). A total of 338 interviews were conducted. Individual interviews with administrators and faculty and focus group interviews with students elicited information about teaching methods, perceptions of ways that faculty efforts contribute to student learning, and perceptions of how university, disciplinary, and state policy contexts enable or constrain the production of learning.

Top-down Analysis: Responses to State Policies: Administrators at public universities ensured their universities complied with the workload reporting and performance funding reporting requirements. Developing criteria, collecting data, and preparing reports were more onerous tasks for the Tennessee universities than for the Ohio and Texas institutions. The Ohio workload mandate requirements (mission review, development of department workload guidelines, and faculty time use reports) and the Texas reporting requirements were less demanding than Tennessee´s ten performance standards, which included major field tests, measurement of general education outcomes, and peer reviews of undergraduate and graduate programs.

Administrators also, however, used managerial strategies which seemed to mitigate direct impact of these policies on faculty. At both Ohio universities, the most tangible result of the workload mandate was a change in the way staff reported faculty time use to the state. Faculty said that the workload mandate had little direct impact on the ways they interacted with students or on how they allocated time to teaching and their other faculty responsibilities. Administrators at the public universities in Texas found ways to meet state workload requirements even while providing their faculty with some flexibility.

Unlike their counterparts in Ohio and Texas, Tennessee administrators did believe the practices encouraged by their state´s performance funding policy could and did have a positive impact on teaching and learning in their universities. The performance funding initiative involved administrators and faculty in creating academically meaningful measures and determining how the assessment results should be used. Because of the context of extreme state budget constraints, however, implementation of the policy broke down between departmental performance and reward. Although departments reported performance, cash-starved central administrators felt they had no choice but to use "additional funds" for general university operating expenses rather than to reward high-performing departments. Therefore department chairs and faculty did not experience the direct benefit of performance inducements.

At all six public universities, faculty said that state policies had no effect on what they did in classroom. Administrators and faculty in the private institutions felt little need to know about public policies that did not directly affect them. Their market niches and their approaches to teaching varied, but the missions and activities of these liberal arts colleges already focused on enhancing student learning.

Bottom-up Analysis: Effective Teaching for Productive Learning: This part of the analysis assumed that effective teaching and learning involves more than simple knowledge transfer. Respondent´s conceptions of productive learning were quite consistent across the four disciplines. They said that disciplinary (and sometimes interdisciplinary) knowledge provides the framework within which students develop and apply skills essential for scholarly inquiry, career, personal life, and informed citizenship. By the time they receive their baccalaureate degrees, students across all disciplines should have developed analysis skills, whether for ill-defined problem solving, critical thinking, or inductive and deductive reasoning. They should develop these skills as they apprehend, integrate, and apply a range of discipline-based knowledge and theories to real world contexts. Students should also develop writing, presentation and personal skills necessary to communicate effectively the process and results of their analyses. Beyond understanding foundational knowledge and principles, memorization of specific facts and examples may be less important than development of the desire and competence to discover what they have not yet learned or need to know in a given situation.

Effective teaching for productive undergraduate learning, according to respondents, requires a variety of teaching methods, all of which help students make connections between context, content and application and encourage students to think critically. As they introduce students to new ideas and concepts, instructors should relate the new material to students' experiences, to possible applications in concurrent labs, future classes, or in the "real world." They should explain processes or give demonstrations to provide immediate context for the ideas. By asking meaningful questions, effective teachers check to ensure students' understanding. While communicating new ideas to students is important, it is not enough, according to many of our respondents, to teach students what they should learn by the time they graduate. Effective teaching depends on providing students with complex assignments that become opportunities to apply and integrate their content and skill learning, on coaching students through the steps necessary to identify, analyze, and solve problems, and on providing students with prompt and detailed feedback about their progress. In the process, instructors demonstrate their caring for students, and model their own enthusiasm for learning. All these teaching practices take more time and effort than the age-old image of the busy professor dusting off yellowed lecture notes. Instructional productivity measured in terms of student credit hours does not address effective teaching that leads to learning productivity.

Influences on Effective Teaching for Productive Learning: The case studies revealed that faculty motivation and opportunities to engage in effective teaching practices likely to foster productive undergraduate learning are shaped by a range of factors. Variations in singular contextual factors at the student, colleague, department, disciplinary, institution, and state levels shape faculty opportunities to contribute to undergraduate students' learning. Because faculty work is conducted within multiple embedded contexts, however, faculty responses to varied factors at one level are also shaped by variations in other levels.

• Students: Who students are and what they have to say about teaching have a profound influence on the ways faculty teach. Within and across institutions, college students vary widely in their preparation for college work, motivation, maturity, and non-college responsibilities.
• Faculty colleagues: Individual faculty member's approaches to teaching are also influenced by their colleagues within and outside their departments. Faculty sometimes mentioned department colleagues who were willing to take risks and use innovative methods in their classrooms. According to comments made by many respondents, innovators were likely to be either junior or very senior faculty.
• Departments: Collective interaction among colleagues serves to communicate the norms and values within an academic department. Variations in resulting department cultures, evident in chairperson support, informal messages, autonomy, and participation in decision making can influence individual faculty members' efforts to contribute to undergraduate student learning. In addition, variations in department policies regarding workload, particularly scheduling and weighting of courses, shape faculty opportunities to engage in the types of teaching practices that foster productive learning.
• Disciplines: Aspects of disciplines that influence the ways and the extent to which faculty contribute to productive student learning include their location in the undergraduate curriculum, their access to external resources, and whether or not they are subject to specialized accreditation.

This study yielded several implications for making policies about improving teaching and learning. First, policies designed to improve undergraduate education must consider the interactions between multiple overlapping contexts within which faculty work and contribute to undergraduate student learning. Second, undergraduate education is ultimately about learning-individual, organizational, and societal. Therefore policies to improve undergraduate education should be based on goals for outcomes, not on desired inputs. Third, both public considerations and professional expertise are required to develop effective educational policy. Therefore policy design and policy implementation should involve representatives of both perspectives.

2. Specialized Accreditation, Student Learning Assessment, and Evidence of Change

Professional disciplines and their associated specialized accrediting agencies are leading the way in integrating academic and professional expertise, and public considerations to improve student learning and to document the improvements. Many professional degree programs are revising their curricula in response to societal changes, industry demands, changing bodies of knowledge, and accrediting requirements to assess graduates´ competencies. A competency is "a combination of skills, abilities, and knowledge necessary to perform a specific task" (U.S. Department of Education, 2001, p. 1). Competency-based education is influencing curricular change in fields ranging from medicine (American Council of Graduate Medical Education, 2006) to engineering. Undergraduate engineering programs have modified curricula in response to shifts in accreditation criteria from input indicators to assessments of graduates´ professional competencies (Prados, Peterson, and Lattuca, 2005). Students develop competencies from engaging in "integrative learning experiences in which skills, abilities and knowledge interact to form learning bundles that have currency in relation to the task for which they are assembled" (Voorhees, 2001, p. 9).

A competency-based approach was adopted in 1996 for implementation by 2000 by the Accrediting Board for Engineering and Technology (ABET). ABET required that by 2001 all engineering programs in the U.S. should demonstrate that they meet 11 learning outcomes for their Engineering Criteria 2000 (EC2000). The outcomes include proficiency in engineering, math, and science, but now additionally includes solving unstructured problems, communicating effectively, working in teams, understanding professional and ethical responsibility, knowledge of contemporary issues, and understanding the global context for engineering.

ABET solicited input from industry, academic, and public stakeholders about proposed changes to the accreditation standards from member institutions and professional societies for more than a year. By allowing for broad stakeholder input, ABET achieved engineering faculty support for the new criteria. Similarly, ABET allows for academic engineering programs to define how they will meet define and assess student competency for each of the requirements. As Colbeck (2002) reports, curricular changes best occur when faculty are involved in generating the recommendations, an assertion consistent with ABET´s approach.

In addition to encouraging engineering programs to assess student learning and competencies via its new accreditation criteria, ABET took the next step to secure external evaluation of the impact of its changes on programs, faculty, and students. A team of researchers from The Pennsylvania State University´s Center for the Study of Higher Education conducted a comprehensive, nation-wide evaluation: Engineering Change: A Study of the Impact of EC2000 (Lattuca, Terenzini, & Volkwein, 2006). The study included engineering degree programs at more than 40 US higher education institutions. Surveys were collected from 1,243 faculty members, 147 program chairs, 39 deans, 1,622 employers, 5,494 graduates from the class of 1994, and 4,330 graduates of the class of 2004.

The findings of this comprehensive study show that the shift from accounting for inputs (e.g. terminal degrees of faculty, facilities, student credit hours) to assessing students´ competencies in the 11 broad areas had significant effects on the content of curricula in programs, on teaching methods used by faculty, and on students´ learning. Lattuca et al. (2006) observe that " Three-quarters or more of the chairs report moderate or significant increases in their programs´ emphasis on communication, teamwork, use of modern engineering tools, technical writing, lifelong learning, and engineering design" (p.3) Perhaps because of the shift in curricular content, faculty members reported shifts in the ways they taught and interacted with students. As they provided students opportunities to learn teamwork, design, and communication skills, the majority of faculty also reported increases in their use of group work, design projects, and active learning exercises. Such practices are consistent with the findings for "effective teaching for productive learning" in the study conducted by Colbeck, et al. (2001), summarized above. Also similar to the faculty in that study, the engineering faculty in the ABET study were more likely to attribute their changes to their own agency or their responsiveness to students than to the external policy influence of ABET. Program chairs, however, overwhelmingly cited ABET as a primary influence for curricular and pedagogical changes. Whether or not they gave ABET the credit for initiating changes, 90 percent of faculty reported they were personally involved in assessment and 60 percent said they were moderately or strongly supportive of assessing student learning for continuous improvement. A majority of faculty reported engaging in efforts to enhance their own teaching, such as reading about teaching, or attending professional development workshops on teaching, learning, and assessment.

These curricular and assessment change efforts at the levels of academic program and individual faculty had positive effects on student learning. Lattuca et al.(2006) compared surveys of the 1994 (pre EC2000) and 2004 (post EC2000) cohorts. After controlling for graduates´ and institutional characteristics, they found that the post-EC2000 graduates were more actively engaged in their own learning, had more interaction with instructors, were more involved in engineering design competitions, spend more time studying abroad, and noticed more emphasis in their degree programs on diversity than members of the 1994 (pre-EC2000) cohort. As a result, they reported larger gains in five EC2000 competency criteria: awareness of global issues related to engineering, applying engineering skills, group skills, awareness of ethics, and ability to apply math and sciences. Thus, adding "soft skills" to the curriculum did not hinder students´ learning of basic engineering, math, and science skills. Faculty use of active learning pedagogies was positively associated with students´ learning gains; use of lecture had a negative association with student learning.

Conclusion

The U.S. Department of Education case studies reveal that while state policy makers are primarily utilizing regulatory, monitoring and inducement strategies to gain greater faculty compliance with the use of learning outcomes, those strategies may not be the most efficient way to produce the desired effects. Consistent with Colbeck, Fairweather, Brown, Beach & Fingers (2001), policy makers should consider policies that foster a bottom-up approach to learning outcomes. Additionally, both the Colbeck et al. (2001) and the Lattuca et al. (2006) studies provide evidence that many faculty philosophically agree with the aims of learning outcomes and state policy strategies should focus upon this common ground.

Learning outcomes may best be implemented within the context of faculty culture. EC2000, for example, represents an accreditation agency´s efforts to facilitate a dialogue among faculty across the nation. That dialogue led to new standards that incorporated engaged learning outcomes into the accreditation process including those outcomes that are more consistent with the educational aims of employers, legislators and a changing work environment. ABET´s accreditation criteria delineate that students must gain competencies in working in teams, communicating effectively, understanding ethics, knowledge of contemporary issues and understanding the global context for engineering (EC2000). The success of the new ABET learning outcomes, as evidenced by the work of Lattuca, Terenzini, & Volkwien (2000), achieved success precisely because faculty assumed a leadership role in developing the changes.

In contrast to the findings highlighted by this article, the recent Spellings Commission´s report, A Test of Leadership: Charting the future of U.S. higher education (2006) calls for greater federal reliance upon regulatory, monitoring and inducement strategies to improve the educational outcomes of higher education. A more effective approach, according to the studies described here involves a more collaborative approach indicative the approach utilized by ABET. Utilizing a collegial model for curricular and pedagogical reform, accrediting associations must encourage their members to work together to develop more comprehensive learning outcomes. This model for creating change may well represent the best method for encouraging faculty to adopt learning outcomes and meeting the needs of the 21st century economy.

Bibliographic references

• AMERICAN COUNCIL OF GRADUATE MEDICAL EDUCATION (2006). Outcome project. Enhancing residency education through outcomes assessment. Retrieved October 29, 2007 from: (http://www.acgme.org/Outcome/)
• ASSOCIATION OF AMERICAN COLLEGES & UNIVERSITIES (1985). Integrity in the college curriculum: A report to the academic community. Washington, D.C.: Association of American Colleges.
• BARR, R. B. & TAGG, J. (1995). From teaching to learning ?A new paradigm for undergraduate education. Change, 27(6), 12-25.
• BELL, T. H. (1993). Reflections one decade after a nation at risk. Phi Delta Kappan, 74 (8), 592-598.
• BENNETT, W. J. (1985). To reclaim a legacy. American Education, 21(1), 4-15.
• COLBECK, C. L., FAIRWEATHER, J. S., BROWN, D. F., BEACH, A. L., & FINGERS, E. A. (2001). Final report to OERI: Enhancing faculty contributions to learning productivity. University Park, PA: Center for the Study of Higher Education, The Pennsylvania State University.
• COUNCIL OF HIGHER EDUCATION ACCREDITATION (2006). 2007 CHEA Award for Institutional Progress in Student Learning Outcomes (Publication no. http://www.chea.org/award/CHEA_Award_Application_2007.doc). Retrieved October 25, 2007:
• EWELL, P. T. (2002). An emerging scholarship: A brief history of assessment. Banta and Associates (Eds). Building a scholarship of assessment. San Francisco: Jossey-Bass. pp. 1-25.
• HUBA, M. E., & FREED, J. E. (2000). Learner-centered assessment on college campuses: Shifting the focus from teaching to learning. Needham Heights, MA: Allyn & Bacon/Longman Publishers
• LATTUCA, L. R., TERENZINI, B. T., AND VOLKWEIN, J. F. (2006). Engineering change: A study of the impact of EC2000. Executive Summary. Baltimore, MD: ABET, Inc. Retrieved October 28, 2007 from (http://www.abet.org/Linked%20Documents-UPDATE/White%20Papers/Engineering%20Change.pdf)
• MCGUINNESS, A. (2003). Models of postsecondary education coordination and governance in the states. Denver, CO: Education Commission of the States. Available at: (http://www.ecs.org/clearinghouse/34/23/3423.pdf)
• NATIONAL GOVERNOR´s ASSOCIATION (2007). Higher education accountability for student learning. Washington, D.C.: National Governor´s Association Center for Best Practices, Policies Studies Division. (http://www.nga.org/Files/pdf/0702HIGHERED.PDF)
• PRADOS, J. W., PETERSON, G. D., AND LATTUCA, L. R. (2005). Quality assurance of engineering education through accreditation: The impact of engineering criteria 2000 and its global influence. Journal of Engineering Education, 94 (1), 165-184.
• U.S. DEPARTMENT OF EDUCATION (2006). A Test of Leadership: Charting the future of U.S. higher education. Report of the Commission appointed by Secretary of Education Margaret Spellings. Retrieved October 29, 2007 from: (http://www.ed.gov/about/bdscomm/list/hiedfuture/reports/final-report.pdf)
• U. S. DEPARTMENT OF EDUCATION, NATIONAL CENTER FOR EDUCATION STATISTICS (2001). Defining and assessing learning: Exploring competency-based initiatives (by E. Jones, R. A. Voorhees, and K. Paulson for the Council of the National Postsecondary Education Cooperative Competency-Based Initiatives Working Group). Washington, D. C.: U. S. Department of Education, National Center for Education Statistics.
• VOORHEES, R. A. (2001). Competency-based learning models: A necessary future. In R. A. Voorhees (Ed.) Competency-based learning models in higher education, New Directions for Institutional Research, No. 110. pp. 5-13. San Francisco: John Wiley & Sons.

• Valoración: 5
• Identifíquese para valorar artículo
• Recomendar

Compártelo

Traducir