|
Assessment Practice at JMU
The following provides a detailed description of the procedures for assessment at James Madison University. Assessment practice has been a hallmark of JMU for over 18 years, and over this time, the University has received considerable recognition for its long-standing commitment to quality assessment. James Madison University's assessment practice and the guidance of the Center for Assessment and Research Studies (CARS) have been recognized repeatedly by such external agencies as the State Council for Higher Education in Virginia (SCHEV), the National Council for Accreditation of Teacher Education (NCATE), and the Southern Association of Colleges and Schools Commission on Colleges (SACSCOC).
Purpose or Objectives:
In 1985, the Commonwealth of Virginia chose several Virginia institutions to explore issues in accountability and assessment in higher education; James Madison University was one of those institutions. As a result of those investigations, a Director of Assessment, T. Dary Erwin, was hired in 1986. In 1988, the State Council of Higher Education (SCHEV) allocated funding for an Assessment Center at James Madison University, with permanent funding for full-time positions to work in Academic Affairs and Student Affairs assessment work. These faculty members have taught and enjoyed dual academic rank in the Department of Psychology since 1986. All CARS faculty members are affiliated with the Department of Graduate Psychology.
In 1987, the Commonwealth required every public institution to submit assessment plans for review by SCHEV staff and an external consultant. These assessment plans were to include assessment in the major, general education, at-risk students, and alumni. Within two years, SCHEV added assessment of student affairs and off-campus instruction. JMU has a history of taking this mission seriously and attempting to operate from a scientific foundation. The issue of appropriate sampling of students became an active issue as early as 1987 when the 'incentive' sampling practice began to break down. The Academic Council of Deans designated a day specifically for student assessment to facilitate collection of credible assessment data to support inferences about general education, the major, and student development. On this Assessment Day, all undergraduate courses are canceled to assure no room or time conflicts. From the start, students were required to participate, and student registration for courses was blocked until their assessment data was collected. This institutional commitment to assessment is the envy of many, many institutions nationwide. The Office of Student Assessment became the Center for Assessment and Research Studies (CARS) in 1997. This Center now is comprised of eight full-time doctoral level faculty, one Masters level faculty member, and three staff members. The Center also administers, in collaboration with the Department of Graduate Psychology, a PhD program in Assessment and Measurement, as well as a concentration within the Psychological Sciences MA program.
James Madison University demonstrates excellence in assessment through scheduled data collection at four stages of the academic career: 1) entering first-year student assessment conducted during fall orientation activities; 2) mid undergraduate point assessment in conducted in mid-February; 3) graduating senior assessment conducted within the academic major(s); and 4) regular surveys of alumni. The Center for Assessment and Research Studies manages these data collection efforts and works closely with faculty on the development and review of information. The role of CARS' assessment activities with James Madison University's Academic Program Review (APR) procedures is integral to monitoring and enhancing educational quality. Every academic program on this campus is scheduled on a six-year cycle for systematic review. Each program is expected to provide five years of assessment data as part of their self-study. They are also expected to indicate how they use this data to monitor and improve their curriculum, instructional delivery, and assessment design. It should be noted that the Division of Student Affairs also engages in systematic program review, and similar procedures and review are imposed. A central part of CARS' mission is to provide support for assessment activities and program review across both academic affairs and student affairs. One of the defining properties of assessment practice at JMU is that student affairs is and always has been an equal partner in assessment.
Procedures:
The assessment procedures are mentioned in the JMU undergraduate catalogs. Page 28 of JMU's 2004-2005 undergraduate catalog states: "JMU requires students to take a series of assessment tests prior to graduation. JMU uses information from the tests results to measure how its programs and services contribute to individual student learning and development and for student progression based on competency attainment in selected programs. The university encourages students to review program requirements for further details."
Consistent with these aims, James Madison University has designated two formal assessment days for each academic year. The first assessment day involves all first-year entering students and takes place during the orientation activities in August. On this day, all entering students are randomly assigned to testing locations on the basis of the last two digits of their JMU student IDs. In this way, the students assigned to any testing location can be considered large, representative samples of the entering student cohort. This sampling procedure provides a strong scientific foundation upon which the credibility of assessment results can withstand scrutiny. The testing sessions generally span a period of 2 to 2 ½ hours in duration. The second assessment day activities take place each spring semester in February. On this date, all undergraduate classes are canceled, and all sophomore students with 45-70 cumulative credit hours (including transfer credits) are assigned to testing rooms. As stated previously, participation in assessment day activities is required; therefore, bias resulting from nonparticipation is controlled. The room assignments are made on the basis of the last two digits of JMU student IDs, thus, assuring large representative samples of students assigned to the various assessment instruments to be administered. Since student IDs do not change over time, it is possible to link the assessment performances of entering students with their later performances as sophomores. At a minimum, this design allows cross-sectional analyses to be conducted. If our instruments remain stable, this design can provide very powerful repeated-measures analyses that allow faculty to make more confident inferences about student growth and development, particularly when these analyses are coupled with variability in student exposures to relevant courses, grades in those courses, and other important variables.
It should also be noted that since classes are canceled on this spring assessment day, many academic programs use this occasion for data collection with their graduating senior majors. The academic programs are responsible for their assessment data collection strategies; however, CARS liaisons are available for consultation. JMU relies heavily on all of these data collection processes for quality assurance and research on student affairs and academic program success.
CARS is responsible for the data collection efforts for entering first-year students and sophomores on these formal assessment days. CARS hires and trains all proctors employed to administer assessment tests. These procedures are in compliance with The Standards for Educational and Psychological Tests (APA, AERA, NCME, 1999). We protect test security, and carefully archive the data. The instruments used for assessment day all link back to JMU educational and developmental learning outcomes. Test and data security procedures are carefully observed, and the Center for Assessment and Research Studies has extensive physical and cyber security procedures in place.
CARS faculty work in close collaboration with other faculty members from across our campus in developing instruments that are tailored to meet our specific learning and developmental goals and objectives. We have found that CARS' faculty expertise in measurement and instrument construction when combined with the content expertise of other JMU faculty members provides a potent interaction. Many of the instruments we have developed at JMU are now being actively sought by other institutions across the Commonwealth of Virginia and beyond. Over 90% of the instruments employed for JMU assessment have been locally developed.
The assessment work conducted across the Division of Student Affairs is so abundant it defies description. Assessment consultation is provided for Multicultural Awareness & Student Health; Student Life; and Student Success. Within each of these areas, several assessment programs thrive. In an effort to provide for these many programs, CARS has allocated multiple faculty members to serve as consultants for Student Affairs assessment activities.
An important component to our assessment procedures will be the implementation of a new University Assessment Committee. The Associate Vice President for Academic Affairs for Assessment and Program Evaluation, Dr. T. Dary Erwin, has made plans to formulate this committee. The University Assessment Committee will be chaired by the Executive Director of the Center for Assessment and Research Studies (CARS), Dr. Donna L. Sundre, and will have broad campus representation. This group will be charged with overseeing JMU assessment practice and reviewing the assessment designs. More details will be forthcoming to the University on this committee and its composition.
Data Analysis:
As mentioned above, the sampling procedures employed for assessment at JMU provide rich opportunities for exploring the extent to which our programs achieve their goals and objectives. A wide variety of data analytic techniques are used. However, of primary importance are the means by which confidentiality of subjects and their responses are maintained. JMU ID numbers are systematically collected; they are absolutely necessary to assure student participation and to link student performances with student course taking experiences and grades. While JMU IDs are collected, individual students are not the primary unit of measurement. Our analyses are primarily concerned with program evaluation; therefore, the analyses focus on groups of students. For example, when analyzing data for general education, an analysis might explore relationships between performances on a test instrument with grades in particular courses. Individual students are never identified or reported; further, only groups larger than 20 will be used for any analysis. Assessment data is archived on protected network servers that are password protected. Access to this server is allowed only to those authorized by CARS.
There are occasions when individual students are required to demonstrate competencies. JMU has implemented several individual student requirements as part of the General Education program. More specifically, in Cluster One: Skills for the 21st Century, students must complete the information literacy tests during the first year of college. This mandate includes passing the Information Seeking Skills Test and the Tech Level I exam (word processing, presentation software, spreadsheets). This testing is conducted in JMU's Ashby Computer Lab, which is dedicated to assessment. Students are individually informed of their test results immediately upon completion of the test. This testing program is overseen and administered jointly by CARS and the General Education program. It is possible that additional required competencies will be identified either by the State Council for Higher Education in Virginia (SCHEV) or by James Madison University. CARS faculty members work carefully and diligently with any program that chooses to use assessment results to make decisions about individual students, as these decisions must bear the weight of greater psychometric rigor.
Reporting Procedures:
Assessment data is reported extensively both internal and external to JMU. These data serve many stakeholders: individual JMU academic and student affairs programs; Academic Program Reviews (which include internal and external site visitors); the State Council for Higher Education in Virginia (SCHEV), and many regional and national accreditation organizations. We are proud to note that the last two accreditation visiting teams for the Southern Association of Colleges and Schools Commission on Colleges (SACSCOC) awarded rare commendations to JMU in recognition of the quality of its assessment work. In addition, assessment results and processes are frequently reported and published in JMU, state, regional, national, and international forums. Again, the unit of measurement is generally group level data. On rare occasions, qualitative data collected at the individual level may be collected and reported; however, no student is individually identified.
The presentation methods employed include formal written and oral reports, web-based summaries of reports, grant proposals and reports, informal written and oral presentations, and refereed publications. In the past, we were once able to provide feedback to students at their request via email on the Raven mainframe computer; however, this is no longer possible. Our email vendor has changed many times in recent years; we have been unable to identify a stable feedback platform. At this time, we are exploring a new means by which JMU students might gain access to their individual assessment results. With the recent implementation of eCampus at JMU, the long held desire to provide assessment results to students may become a reality. We will continue to explore this new method.
Experience of the Researcher(s):
Click here for information on the Assessment Specialists and graduate students at CARS.
|
 |
Assessment Practice at JMU
Memorandum of Information Cover Letter - August 2004
Memorandum of Information August 2004
Memorandum of Information - Updated April 2005
- Appendices A and B
- Appendix C
- Appendix D
Memorandum of Information - Updated January 2012
- Appendicies A and B
- Appendix C
- Appendix D
|
