Agnes Scott College, in partnership with Converse College, the University of North Carolina, Asheville, and Wofford College requests $300,000 from the Teagle Foundation to develop and implement a three-year project designed to assess the impact of integrative learning experiences on students’ intellectual growth. Our proposal emerges from a series of in-depth discussions, including the Teagle Foundation Listenings (February and September 2004), the “Project DEEP/Liberal Arts Colloquium" (June 2004), the planning conference held by the consortium in November 2005, and the February 2006 AAC&U/Teagle Foundation working conference.
Our institutions share an important focus programmatically: we all offer some type of educational experience in a student’s first year that is designed to integrate academic learning–across disciplines and/or in conjunction with co-curricular or service learning experiences. Assessment of these experiences is critical to understanding their value to liberal arts education in general and for continued program improvement on each campus. In seeking to enhance our colleagues’ understanding of the importance and power of effective assessment, we will: 1) develop local, direct assessment measures of particular types of programs (e.g., integrative learning) to complement large-scale national assessment measures; 2) engage faculty and administrators in the iterative process of using assessment results for program improvement; and 3) coordinate assessment efforts at the course, department and co-curricular level with institutional goals, objectives, and strategic planning.
Problem Statement
Assessment has taken on a greater urgency in light of the federal government’s recent attention to accountability in higher education. Rather than wait for the calls for accountability to lead to externally imposed, poorly designed solutions, it is important that liberal arts institutions work together to develop assessment strategies that demonstrate the value-added of a liberal arts degree.
Much of the initial assessment done on campuses has come in response to accrediting agencies. These initial assessments have analyzed readily accessible data: size of endowment, student faculty ratios, standardized test scores of admitted students, graduation rates, and faculty salaries. While these measures are important barometers of financial and administrative efficiency, they do not tell us the extent to which students are learning, maturing, and developing skills that will lead them to be productive and responsible citizens within our democracy and throughout the world. In short, they do not speak to the issues that underscore the mission of liberal arts colleges.
Faculty understand that the kind of teaching that happens at liberal arts institutions—intense involvement with students and attention to their development as learners—is key to shaping those students’ intellectual futures. This includes providing students the opportunity to integrate their learning in many different ways—across courses, outside the classroom, and in the greater community. The challenge and opportunity is that there are few, if any, standard instruments available which attempt to assess teaching methods or integrated learning at the individual, departmental, or institutional levels.
Effective assessment is possible on campuses that embrace inquiry, evidence, and reflection, and such terminology can appease faculty concerns about the purpose of assessment. Because professors are trained to embrace empirical evidence and assessment of their scholarship (e.g., peer review), assessment naturally assumes a vital and integral role in their professional lives once they see the transformative power of evidence of student learning outcomes. It provides faculty with meaningful data that simultaneously inform classroom practices and aid in the planning and revision of academic programs
Purpose of the Consortium: Assessing and Improving Integrative Learning
We propose to use a combination of national survey instruments, in-depth interviews, focus group discussions (with students and faculty) and direct examination of student work to measure general liberal education outcomes as they emerge through integrative learning experiences in the first year. Recent efforts to measure student engagement in their own educations have yielded some exciting results for liberal arts institutions that will help guide our work. The National Survey of Student Engagement (NSSE) measures student perceptions of their learning gains, including several commonly associated with integrative learning (e.g. critical thinking, ability to work on problems with complex solutions, collaborative learning, etc.). We will also employ and compare findings from several surveys developed by the Higher Education Research Institute (HERI), including the CIRP Freshman Survey, the Your First College Year (YFCY), and the College Student Survey (CSS). They will help us define both the overlap and unique characteristics of our student populations. They will also provide a means of focusing our exploration of student learning outcomes in our specific integrated programs within and among our collaborating institutions. We seek to identify which aspects of the integrative learning experiences contribute to students’ “growth and development”—including their critical thinking and writing skills, their creativity, their collaborative abilities, their ethical judgment.
Because we want to truly engage in pedagogical change and program improvement, we will also plan for how to best incorporate assessment information into planning efforts concerning the curriculum, co-curriculum, residence life, faculty development, and strategic planning at our institutions. While receiving high marks on a national survey or satisfying an accreditation team brings a certain set of benefits for an institution, these achievements alone will not strengthen the student experience. They also cannot ensure that all levels within an institution are identifying and pursuing complementary teaching and learning goals. Achieving the aim of aligning course goals and learning outcomes with the multivalent learning experiences of all students requires a deeper understanding of, and involvement with, assessment data at all levels of the institution.
In November 2005, a group of faculty and administrators from the four institutions met at Wofford College with several external consultants: Charles Blaich, Director of the Center of Inquiry in the Liberal Arts, Wabash College; Ed Neal, Director of Faculty Development, University of North Carolina Center for Teaching and Learning; Eric Newhall, former Director of the Core Program, Occidental College; Steven Weisler, Dean of Academic Development, Hampshire College. During this planning session and in several follow-up exchanges, our consultants critiqued the original proposal and helped us refocus our initiative. The meeting at Wofford helped us identify the common curricular elements we share as well as the challenges we face in assessing them. This meeting confirmed our belief that a consortium of these institutions would have much to offer each other as well as to the national conversation about the importance of assessing the outcomes of integrative learning experiences.
Our consortium represents a unique partnership among southeastern institutions that, while diverse in type, share a historical and contemporary commitment to providing the highest quality liberal education for our students. Members include Agnes Scott College (private women’s, GA), Converse College (private women’s, SC), the University of North Carolina, Asheville (public liberal arts university, NC), and Wofford College (private co-ed, SC). All these institutions have undertaken specific initiatives designed to strengthen integrative learning. Moreover, as we discovered at our November planning meeting, each of these initiatives models a curricular practice whose efficacy and outcomes the other campuses have a serious interest in learning from and perhaps emulating. While our consortium will collectively administer the Cooperative Institutional Research Project (CIRP) Survey, the Your First College Year (YFCY), National Survey of Student Engagement (NSSE) each institution will also carry out a targeted assessment in an area of special interest:
Agnes Scott College will evaluate the benefits of linking its interdisciplinary First-Year seminars (FYS) to Living & Learning communities (L&LC) in the context of a broader assessment of its First-Year seminar program, in existence since 2001. The seminars are designed to help students analyze a topic from different disciplinary perspectives and improve their critical thinking and writing skills. The assessment project will compare those students taking FYS linked to L&LC with those not linked to FYS to determine whether the former 1) perform better academically; 2) engage in higher levels of active and collaborative learning, interaction with faculty members, and diversity-related activities; and 3) report increased satisfaction with the institution and their academic program. Faculty will use the data collected as a result of this project to decide whether or not to make L& LC an integral and required component of the FYS program.
Converse College will assess the impact of its new leadership and service learning initiative, a partnership between academic and student affairs. Whereas many colleges throughout the U.S. have separate leadership and service programs, the Converse initiative is distinctive in that leadership and service are combined. While Converse has a tradition of educating women for leadership and service, the new initiative assists faculty with integrating service learning into the curriculum by facilitating training and discussion groups and offering site-matching services, regional conference opportunities, a resource library, and individual consultations. During the Teagle funding cycle Converse will evaluate the impact of the new service learning curriculum on its first-year students by tracking student progress over three years.
The University of North Carolina, Asheville will assess its Liberal Studies Introductory Colloquium (LSIC)—the first component of the university’s new Integrative Liberal Studies program—for the impact of integrative academic experiences on student learning and student satisfaction. The LSICs are topical seminars taken by all new freshmen and most transfer students, featuring multidisciplinary approaches and materials. LSICs are writing intensive in pedagogy and place faculty in the role of academic advisers. These courses also serve to acculturate students to the liberal arts environment and approach to higher education.
Wofford College will undertake a learning outcomes assessment of its first-year learning communities initiative, an optional program linking laboratory science courses for non-science majors and freshman humanities seminars. In this program, begun in 2001, two general education courses, laboratory science for non-science majors (Science-104) and a humanities course (usually the first semester seminar course, Humanities-101) are linked by a common theme and developed and implemented as an integrated learning experience. Since its inception, the program has been evaluated "in house" and by external reviewers. Focus groups interviews with students and participating faculty have recorded high levels of satisfaction with the program, but learning outcomes, especially critical thinking skills and teamwork, have not been compared between students enrolled or not enrolled in the LC program. As Wofford seeks to adapt the lessons of the LC program more broadly across campus, evidence of the value-added of such resource intensive programs is imperative.
In addition to their value for each institution, we believe these individual projects will also yield important comparative data for the consortium as a whole. For example, we expect to compare the ways in which an academically integrated service learning program and learning communities affect student learning outcomes.
Proposed Activities
A. Program Design
One of the foremost challenges faced in assessment is getting good data into the hands of faculty who will use it in their planning and teaching. For this reason, we will involve faculty centrally in the selection of specific learning outcomes and the development of assessments regarding the integrative learning experiences offered on our campuses. During the first year of our three-year grant, each campus will carefully develop strategies and instruments for assessing the specific integrative learning experiences underway on their campuses. Agnes Scott College will host a September 2006 planning session to complete a census of data, including survey data, in order to provide the consortium with a better understanding of what information is already available with and between institutions. Since all four of these initiatives are part of the first year general education curriculum, the campuses will develop a set of shared “best practices” for assessing first year programs as well as assessment methods unique to their specific initiatives.
B. Implementation, Assessment, and Data Sharing
Integrative Learning Assessments: Faculty at each institution will develop a set of questions that they would like answers to regarding their specific integrative learning program. These questions will be developed in light of each of the institutions’ current assessment programs, and reflect what faculty would like to know about their own integrative learning initiatives. A group of faculty and administrators representing each of the institutions will then develop a shared set of assessments which can be utilized at the individual institutions. Agnes Scott College will supervise a post-doctoral researcher to combine and analyze the shared assessment data according to the needs of the individual institution.
Shared National Survey Instruments: The institutions will agree to share a common set of institutional survey instruments for the term of the grant. These instruments might include: CIRP, CSS, YFCY, NSSE, and any common questions or instruments developed in collaboration. In the first phase of the grant, the institutions will identify the instruments and local items to be added to these instruments and to create databases designed to integrate and analyze data.
Shared Student Data: The institutions will agree to share individual student data on some set of characteristics (e.g. race, academic major, residence, etc.) that could be linked with student responses to assessments and survey data. We anticipate that this shared dataset will include measures of individual students’ academic performance and the results of direct assessments of learning. These measures could include SAT scores, GPA, course-embedded assessments, etc.
Confidentiality: Each college will link survey data and other information before sending unit record data to the host institution (Agnes Scott). Individuals would be identified by home college and some unique number assigned by the college to facilitate later linking of additional data as it becomes relevant.
C. Organization
We propose that Agnes Scott College serve as the recipient institution for the grant, providing overall administrative and financial oversight. One key advantage of this arrangement is that Agnes Scott College has a full-time Director of Assessment who is a committed member of the collaborative. This expertise will be used to guide the assessment efforts on individual campuses and help the consortium identify and evaluate value added gains among students.
Each member institution will identify a primary contact for its campus, along with one or more faculty and administrators who will serve on the consortium steering committee and oversee consortium-related projects. The steering committee will identify emerging
needs and priorities of the group, develop the agendas for consortium meetings, and link the efforts of the consortium to ongoing assessment projects and committees on each campus.
As the attached budget indicates, each of the campuses involved in each of the specific integrative learning initiatives will host at least one meeting in each of the three years. A single, all-consortium meeting will take place at the end of each year. These end-of-year meetings will allow participants to report on progress made in each initiative, share insights across project working groups, and possibly develop new collaborative projects as opportunities arise. A different campus will host the meetings each year, which will enable additional faculty on each hosting campus to take part.
D. Outcomes and Dissemination
By working together, the participating colleges will be able to share successful models, such as Agnes Scott College’s FYS-L&LC program, Converse College’s service learning model, UNC-A’s interrdisciplinary first-year curriculum, and Wofford’s learning communities linking laboratory science courses for non-science majors and freshman humanities seminars. Through the consortium, we will create an ongoing forum for the discussion of emerging assessment methods, instruments, and strategies.
We see numerous opportunities to disseminate the work of the collaborative. We look forward to sharing the results of our work through AAC & U, Higher Education Data Sharing, and National Resource Center for The First-Year Year Experience meetings, publications, and with other national organizations concerned with liberal arts education, value-added assessment, and integrative learning opportunities.
Budget
Project: “Improving and Assessing Integrative Learning Experiences”: Agnes Scott College, Converse College and University of North Carolina – Asheville, and Wofford College.
Budget Narrative
The first year of the grant aims to establish a common data set and other local evaluative measures that will be useful across campuses of the consortium for the specific purpose of improving our understanding of the value-added of integrative learning experiences. We will begin by evaluating institutional indices (e.g. students, staff, and faculty survey results, analyses of pedagogy and learning outcomes, other assessment information) on a comparative basis. We will then work with faculty and those responsible for assessment and data gathering to develop assessment measures that create sustainable assessment initiatives on each of our campuses that will be useful in improving integrative learning programs. Year two focuses on implementing the agreed-upon local and national assessment measures and organizing and analyzing initial data from those assessments, with an eye toward refining any measures to improve upon their ability to yield useful assessment information. In year three the focus shifts to additional data analysis and synthesis, with special emphasis on providing results to faculty in ways that they find useful for discussion of pedagogical change.
See Karen W. Arenson, “Panel Explores Standard Tests for Colleges,” New York Times, 9 February 2006, A.1.1.
Liberal Education Outcomes: A Preliminary Report on Student Achievement in College. Washington, D.C.: AAC&U, 2005.
Evidence from within and beyond the collaborative supports the premise that integrative experiences provide an important component of the “value added” of a liberal arts education. See Kuh, G. D., Kinzie, J., Schuh, J.H., Whitt, E.J., Student Success in College: Creating Conditions That Matter. AAHE, 2005. Jossey-Bass, p. 142-144, 198.
