Completer effectiveness is an important yet challenging standard to measure. The Commonwealth of Virginia does not make teacher or student performance data available to the institutions at which the teachers were trained. Accordingly, Virginia’s EPPs have had to find different and innovative ways to address completer effectiveness. To counteract this data gap, the Virginia Education Assessment Collaborative (VEAC; https://projectveac.org), a volunteer team of assessment and accreditation professionals at EPPs across Virginia, formed a collaborative to begin coordinating completer data collection. VEAC provides a centralized assessment structure for Virginia EPPs that standardizes and reduces the complexity of data collection for the Virginia Department of Education (VDOE) and CAEP. VEAC administers common surveys to completers of most of the 36 EPPs in Virginia.

VEAC’s annual Completer Survey was designed to satisfy the CAEP sufficiency criteria and has the benefit of facilitating benchmarking of EPP Completer ratings to VEAC-wide data. The Completer Survey consists of 12 items aligned to the 10 Interstate Teacher Assessment and Support Consortium core teaching standards (InTASC, 2013) and the 8 Virginia Uniform Performance Standards on which in-service Virginia teachers are evaluated (VUPS, 2021). The VEAC surveys were revised in 2023 to reflect changes to Virginia’s Uniform Performance Standards, including the new professional standard 6, Culturally Responsive Teaching and Equitable Practices. Table 1 shows the alignment of VEAC Completer Survey items to the InTASC and VUPS standards. The Virginia Uniform Performance Standards are:

Performance Standard 1: Professional Knowledge

The teacher demonstrates an understanding of the curriculum, subject content, and the developmental needs of students by providing relevant learning experiences.

Performance Standard 2: Instructional Planning

The teacher plans using the Virginia Standards of Learning, the school’s curriculum, student data, and engaging and research-based strategies and resources to meet the needs of all students.

Performance Standard 3:  Instructional Delivery

The teacher uses a variety of research-based instructional strategies appropriate for the content area to engage students in active learning, to promote key skills, and to meet individual learning needs.

Performance Standard 4:  Assessment of/for Student Learning

The teacher systematically gathers, analyzes, and uses all relevant data to measure student progress, guide instructional content and delivery methods, and provide timely feedback to students, parents/caregivers, and other educators, as needed.

Performance Standard 5:  Learning Environment

The teacher uses resources, routines, and procedures to provide a respectful, positive, safe, student-centered environment that is conducive to learning.

Performance Standard 6:  Culturally Responsive Teaching and Equitable Practices (New in 2022; first mapped to VEAC surveys in 2022-23)

The teacher demonstrates a commitment to equity and provides instruction and classroom strategies that result in culturally inclusive and responsive learning environments and academic achievement for all students.

Performance Standard 7: Professionalism

The teacher demonstrates a commitment to professional ethics, collaborates and communicates appropriately, and takes responsibility for personal professional growth that results in the enhancement of student learning.

Performance Standard 8:  Student Academic Progress

The work of the teacher results in acceptable, measurable, and appropriate student academic progress.

Table 1. VEAC revised survey items and VUPS / InTASC alignment

Survey results from 2022-23 JMU program completers indicated that JMU graduates in their first three years of teaching feel well-prepared to positively impact their students. Completers respond to all items on a scale ranging from 1 (Unacceptable) to 4 (Exemplary), following the general prompt, “Based on your preparation at James Madison University, how would you rate your performance in each of these teaching areas.” The expected performance for each item is an average score of 3, or a rating of “Proficient.” Of the 12 items on the VEAC Completer Survey, JMU completers scored at or above an average of 3/Proficient on all and scored at or above the VEAC mean on most. No differences were statistically significant. One item (IG) particularly addresses completers’ impact on student academic progress: “Engaging in practices that results in acceptable, measurable, and appropriate student academic progress.” The JMU average on that item was 3.40, which was higher than the VEAC-wide average of 3.28, suggesting a favorable impact of JMU completers on their students. Scores for all VEAC Completer Survey items for JMU and VEAC-wide are reported in Table 2.

Table 2. 2022-23 VEAC Completer Survey Results for JMU and VEAC (Completer Effectiveness)

Of course, the VEAC surveys have limitations in addressing student effectiveness. Most significantly, the surveys offer an indirect measure of perceptions around effectiveness without using direct empirical assessment data. In addition, the sample of respondents to the VEAC surveys is limited, which impacts representativeness and generalizability. Accordingly, to buttress information about completer impact, JMU is pursuing several different avenues to gather additional data. In Spring 2023, we piloted a VEAC-based focus-group protocol with a small sample of new teachers (i.e., recent JMU graduates) to gather more information about their impact. In Spring 2024, our university Center for Assessment and Research Studies will partner with us to use this protocol more broadly to conduct focus groups on representative samples of completers.

We are pursuing a second avenue for obtaining teacher evaluation data for our recent graduates through the Mid-Valley Consortium (MVC), our consortium of four locals EPPs (JMU, Bridgewater College, Eastern Mennonite University, and Mary Baldwin University) and seven local school divisions (Augusta County, Harrisonburg City, Page County, Rockingham County, Shenandoah County, Staunton City, and Waynesboro City). We are gathering in-service teacher evaluation data from these school systems. Examining real evaluation data for our in-service graduates will provide our programs and unit with more objective, direct information about areas where our teachers are excelling and areas where they may have more room for growth.

Finally, we are piloting an exit survey for program graduates in Spring 2024 to facilitate communication with our program completers. Two recent trends have led to additional challenges communicating with our program completers: the Virginia Department of Education not providing timely data about the employment of Virginia EPP graduates, and a short-notice decision by our university to discontinue student access to their alumni email accounts after graduation. We are optimistic that attaching an exit survey to completers’ licensure application process will improve our ability to follow up with our graduates after they leave JMU.

JMU gathers information about employer satisfaction in a variety of ways. One method is the VEAC Employer Survey, which asks employers to rate in-service teachers who completed one of JMU’s licensure programs on key knowledge and skills. For initial-licensure programs, principals typically complete surveys to evaluate new in-service teachers. Initial-licensure items mirror the items on the Completer Survey, aligned to InTASC standards and the Virginia Uniform Performance Standards. An advanced licensure program Employer Survey was piloted in 2022-23. For advanced licensure programs, employers completing the survey might be principals or district administrators (e.g., superintendents). Advanced licensure survey items align to the general CAEP competencies for advanced programs and are mapped to professional standards for each program. Both initial and advanced Employer Surveys are designed to satisfy CAEP sufficiency standards for surveys.

The 2022-23 pilot and initial administration of VEAC surveys for Advanced Licensure programs were small, resulting in limited data for JMU completers and their employers. Employers provided data for three Educational Leadership program completers, rating them all as Proficient or Exemplary in all categories. In 2023-24, we are making a concerted effort to gather contact information for program completers and their employers from all our Advanced Licensure programs (Educational Leadership, Literacy, School Counseling, and School Psychology) and will endeavor to acquire representative employer ratings for completers from each of these programs.

The Initial Licensure Employer Survey provided more robust data, with ratings for 129 JMU program completers. Employers respond to a parallel survey to program completers, rating the same items on a scale ranging from 1 (Unacceptable) to 4 (Exemplary), following the general prompt, “Please rate [Completer Name’s] performance on each of the following:” (see Table 1). The expected performance for each item is an average score of 3, or a rating of “Proficient.” Of the 12 items on the VEAC Employer Survey, JMU completers scored at or above an average of 3/Proficient on all items, ranging from a low of 3.02 on Item ID (“Systematically gathers, analyzes, and uses all relevant data to measure student academic progress, guides instructional content and delivery methods, and provides timely feedback to students, parents, caregivers, and other educators”) to a high of 3.30 on Items IF (“Maintains a commitment to professional ethics, communicates effectively, and takes responsibility for and participates in professional growth that results in enhanced student learning”) and IL (“Collaborates with the learning community (e.g. school personnel, caregivers, and volunteers) to meet the needs of all learners and contributes to a supportive culture”). Despite these proficient ratings, we are sensitive to a trend that conflicts with our usual results: in 2022-23, our completers’ average ratings were lower than the VEAC-wide averages across items. In previous years, ratings for JMU completers have been higher than the VEAC average for all items. We are paying attention to this anomaly and will be investigating whether it becomes a trend. Given that the completers who were evaluated in the 2022-23 cohort spent considerable time in their teacher education programs at the height of the COVID-19 pandemic, coinciding with our programs beginning the transition from primarily five-year MAT programs to four-year BS programs, we are hopeful that these data reflect unusual circumstances rather than the beginning of a new trend. Still, we will be watchfully waiting for the next round of VEAC employer data. Scores for all VEAC Employer Survey items for JMU and VEAC-wide are reported in Table 3.

Table 3. 2022-23 VEAC Initial Licensure Employer Survey Results for JMU and VEAC

Additionally, on the VEAC Employer Survey, employers are asked to respond to the following question for each completer: “Based on your experiences with this teacher, what best describes the extent to which they were ready to meet the needs of students in your school?” Responding employers were presented with the options:

• 5 – Fully ready (able to have an immediate impact on student learning)
• 4 – Mostly ready (able to successfully meet the needs of most students
• 3 – Moderately ready (in order to be successful, needed additional training, support, and coaching beyond what is typically provided to beginning teachers)
• 2 – Minimally ready (limited success meeting the needs of students and improving outcomes even with additional supports)
• 1 – Not ready (unable to meet the needs of students even with additional supports)

Table 4 provides descriptive statistics on the overall satisfaction item. James Madison University’s mean has consistently been greater than 4.0, indicating employers observed that their JMU-prepared teachers have a high average readiness level to impact student learning. However, for the most recent cycle of data (2022-23) JMU’s average is slightly below that of other EPPs in the Commonwealth of Virginia. As mentioned in the previous section, this is attributed to disruptions brought about by COVID-19 and the programmatic changes from 5-year to 4-year programs. We will monitor these results to determine whether this is an anomaly.

Table 4. 2022-23 (and Historical) VEAC Initial Licensure Employment Readiness Ratings for JMU and VEAC

Data are provided here for teachers who completed initial licensure programs. The modest Advanced Licensure survey piloted in 2022-23 resulted in three employer ratings with an average readiness rating of 4.67. In 2023-24, we anticipate having more robust results for completers employed in advanced licensure areas.

JMU also monitors employer satisfaction through regular and frequent communication with local school divisions who employ our graduates. This communication happens in both formal and informal ways. JMU faculty prioritizes strong relationships with local school divisions and ensuring that JMU is meeting their needs.

Stakeholder involvement is a key element of ensuring that JMU’s programs are designed to prepare program completers for success. There are many layers of stakeholder involvement at JMU:

Stakeholder Involvement

Professional Education Coordinating Council (PECC)

Stakeholders serve on the JMU’s Professional Education Coordinating Council (PECC), a group that meets monthly to review policies, needs of school partners, and changes in curriculum, assessment, and program/professional requirements. PECC includes program coordinators and/or assistant academic unit heads from every initial and advanced licensure education program at JMU. It also includes representatives from local school divisions.

More information about the PECC is available in the Professional Education Handbook: https://www.jmu.edu/coe/esc/_files/2020-21-Professional-Education-Handbook1.pdf

MidValley Consortium (MVC)

JMU is part of the MidValley Consortium (MVC) that collaborates with three other area EPPs (Mary Baldwin University, Bridgewater College, and Eastern Mennonite University) and seven partner school divisions. The MVC Mentorship and Clinical Experience Team (MCET) representing teachers, administrators, and teacher educators meets monthly to plan consortium activities. The MCET and MidValley Consortium Advisory Council meet to evaluate consortium projects, set policy, and approve the annual budget. Each school division and college or university supports consortium activities. In addition to maintaining regular collaborative workflows and relationships, the MVC engages in timely special projects. In 2022-23, the MVC (a) revised the collaborative Student Teaching Performance Assessment to align to the updated Virginia Uniform Performance Standards (to reflect the additional of Culturally Responsive Teaching as Standard 6); and (b) convened to discuss opportunities for innovative field placement formats, including the possibility of leveraging field placement experiences to meet long-term substitute needs in local schools facing teacher shortages.  

More information about the MVC is available online and in the MVC Handbook:

https://www.jmu.edu/coe/esc/consortium.shtml

College of Education Quality Assurance Advisory Team (CoEQAAT)

In 2022-23, the Director of Assessment, Accreditation, and Accountability for the College of Education convened a new panel of faculty, staff, and students from the College to provide high-level guidance and input about quality assurance issues. The purpose of the College of Education Quality Assurance Advisory Team (CoEQAAT) is to advise on issues related to gathering, summarizing, disseminating, and using educational data. This team of faculty, staff, and students brings diverse perspectives from across the College of Education to guide assessment and accreditation practices to optimize the value and effectiveness of our work. The CoEQAAT includes internal stakeholder representation from each education department, college staff, as well as an initial licensure and advanced licensure student representative.

Future Teachers Advisory Council (FTAC)

In 2023-24, the College of Education Quality Assurance Advisory Team (CoEQAAT) recommended creation of a better avenue for the Dean of the College of Education to hear directly from students about issues related to education programs. In Spring 2024, we are piloting our inaugural Future Teachers Advisory Council (FTAC), an open forum for current initial licensure education students to have dinner and share input with the Dean. Facilitated by a faculty member, this dinner provides an opportunity for students to share directly with the Dean about what is working well and what suggestions they have for program and College improvement.

Advanced Programs and Stakeholder Involvement

Although the PECC, MVC, and CoEQAAT involve advanced programs in some ways, they are often more targeted to the immediate needs of initial licensure programs. Most candidates in advanced licensure programs are already employed in schools; typically, candidates in these programs enter in cohorts of colleagues already holding teaching and other academic roles. Accordingly, the needs and nature of stakeholder involvement in advanced licensure programs differ.

In JMU’s Educational Leadership programs (certificate and M.Ed.) JMU faculty are retired school superintendents with rich networks of professional colleagues. The Educational Leadership program also utilizes approximately 20 adjunct professors who all hold high-level positions in school divisions across Virginia. This model ensures that local school divisions are directly involved in the creation and dissemination of program content and structure. Having such extensive buy-in from Virginia administrators strengthens the program and ensures it is responsive and proactive to current issues of practice in the schools. Educational Leadership program faculty also meet with a network of superintendents informally at twice-annual meetings of the Virginia Association of School Superintendents.

The Literacy Program has an Annual Stakeholder Meeting to discuss the program structure, assessment, trends in literacy education, policy, and other relevant topics. Participating stakeholders in this advisory group include current students, alumni who are working as Reading Specialists, and other faculty and administrators from Virginia schools. In 2022, the Literacy program adjusted its process for acquiring stakeholder input. In even years, they will hold synchronous meetings, and in odd years they will use surveys to gather stakeholder input asynchronously. The first stakeholder input survey will be piloted in 2023. The Literacy program also initiated an annual event, the JMU Literacy Leader Awards, in 2022. This annual event convenes local reading specialists for networking and learning and helps publicize the JMU Literacy Program while celebrating high-achieving literacy educators in Virginia schools.

The School Counseling Program utilizes a variety of methods for connecting with stakeholders:

About once a month throughout the school year, program faculty, staff, and students meet over Pizza and Process Lunches to share food, information, reflections, feedback, and suggestions on their experiences working together. Each semester, the Empathic Times [program newsletter] extends an invitation and provides specific dates and locations. The Counseling Programs Committee meets on the first and third Mondays of each month during the academic year. Locations are announced at the beginning of each semester. Student representatives attend all meetings except those involving comps evaluation and student progress review. Each summer, the School Counseling Program organizes a daylong Counseling Programs Committee Retreat to review the year, discuss revisions to its mission statement and other policies, and brainstorm possibilities. They invite students to join for a potluck lunch and to meet with program faculty during part of the day to offer ideas and perspective. The School Counseling External Advisory Committee consists of counseling professionals, some of whom are graduates of the program, who are active in the field. This advisory group meets each year to study and make recommendations for strengthening the program. The group meets during the Department of Graduate Psychology and Counseling Symposium, which is held each April. As part of its Comprehensive Assessment Plan, the program has developed several formal procedures to receive student feedback and suggestions. First, in courses, faculty members ask students to evaluate their teaching performance and the class activities. Second, when students apply for graduation, they complete an exit questionnaire assessing the entire training program. Finally, the program contacts students within three years after graduation to request they complete an alumni survey. This feedback has helped the program improve courses, the curriculum, and the program. The School Counseling program encourages students to offer constructive feedback and practical suggestions when we request their assessment of their training experiences.

The School Psychology Program faculty meet monthly to plan and revise policy and curriculum, discuss professional and training issues, and to review student progress. One first year and one second year student (selected by their classmates or on a rotating basis) serve on the program committee as student representatives. They participate in all committee activities, except student review and award recipient selection. The program welcomes suggestions and feedback from students to assist in on-going evaluation and improvement. Student representatives are responsible for soliciting questions and comments from their peers to bring to committee meetings. Additionally, an external advisory committee, composed of area practitioners and field-based supervisors, meets once per year to offer suggestions and evaluative feedback. The program initiates an alumni survey every 3 years or as needed, with feedback reviewed and revisions made in the program as needed.

James Madison University triangulates data from several sources, both internal and external, to ensure candidate competency at completion. These measures build from internal, program-level gateways and key assessments during the program, to student teaching data gathered during final initial licensure field placements, to external proprietary assessments necessary for initial and advanced licensure.

Each education program at JMU uses a variety of program- and unit-level gateways and key assessments to ensure candidates are making sufficient progress throughout the program and well-prepared for employment at program completion. Programs design and administer their own program-level measures and evaluate student progress as part of program-level meetings and college-level annual program assessment requirements.

There are also unit-wide assessments that measure candidate competency across programs. The Student Teaching Performance Assessment (STPA), created collaboratively through our MidValley Consortium of four local EPPs and seven school divisions, is administered for all students in their student teaching placements. Midpoint and Final assessments of student teaching performance are completed by cooperating teachers and university supervisors. A Dispositions assessment is also administered to cooperating teachers with the final STPA. Students complete the same measure as a final self-assessment to encourage reflection and to foster discussion with university supervisors.
More information about the STPA and Dispositions measures are available online:
https://www.jmu.edu/coe/esc/fieldexperience.shtml

Student Teaching Performance Assessment (STPA) indicators and final Dispositions are rated on a four-point scale, where 1 = Does Not Meet Expectations (DN); 2 = Developing Towards Expectations (DE); 3 = Meets Expectations (ME); and 4 = Exceeds Expectations (EE). Each indicator has its own specific criteria for each scoring level, reflected in the measure. The target score on each indicator is Meets Expectations. Accordingly, JMU hopes to see an average score of 3 or higher on each indicator by the time candidates complete student teaching. The STPA was revised in 2022-23 in response to new Virginia Performance standards in cultural competency; members of IHEs and LEAs in the MidValley Consortium convened across several sessions to update the STPA accordingly. During that process, we identified Culturally Responsive Practice items in each of our component areas, allowing us to compile data – and highlight importance – for culturally responsive teaching. This revised version of the STPA was piloted with a small administration in Spring 2023 and thoroughly piloted in Fall 2023.

Table 5 shows midblock and final STPA scores and final Dispositions scores for the Fall 2023 cycle, the most recent available data. Dispositions are rated only at the end of the student teaching placement and only by cooperating teachers. This table shows a snapshot of EPP-wide initial licensure data across programs. Average final ratings for all performance indicators and dispositions were greater than 3, suggesting that initial licensure teacher candidates are meeting expectations for program completers at the conclusion of their field placements. The lowest-scored indicators relate to facilitating students’ higher level thinking skills (M = 3.04, SD = 0.40), providing specific evidence to document student learning (M = 3.05, SD = 0.40), and systematically using data to understand student learning (M = 3.06, SD = 0.41). The lowest-rated disposition was also related to student assessment: “Values the use of self and peer assessment and uses data ethically to identify learner strengths and to promote learner growth” (M = 3.16, SD = 0.43). The highest-rated indicators were, “Establishes a climate of trust and teamwork” (M = 3.42, SD = 0.52) and “Establishes a safe physical and psychological environment for all students” (M = 3.42, SD = 0.50). The highest-rated disposition was, “Promotes inclusive learning environments that enable each learner to meet high standards by valuing different backgrounds and abilities” (M = 3.33, SD = 0.52). The high performance on this disposition reflects the arduous work of college faculty to focus on inclusive and culturally responsive teaching.

Table 5. Student Teaching Performance Assessment and Dispositions Scores for Initial Licensure Programs in Fall 2023

In addition to these unit-wide measures, JMU recently developed unit-wide practicum-level field placement performance assessment and dispositions measures. Both measures were piloted in Fall 2023, allowing another level of triangulation for measuring unit-wide candidate progress and ensuring candidates are on track to be well-prepared and highly competent at program completion. The Practicum Performance Assessment (PPA) uses the same four-point scale as the Student Teaching Performance Assessment. Pilot data from the Fall 2023 administration indicate that students are performing as intended in their practicum placements, with most items scoring above the “Meets Expectations” benchmark of 3.0. The lowest-scoring item on the PPA pertains to instructional planning (“Plans appropriate instructional strategies to meet the learning outcomes,” M=2.96) and the highest is about meeting professional expectations (“Demonstrates the expectations of the profession including codes of ethics, professional standards or practice and relevant law and policy,” M=3.42). Table 6 shows results for all items on the pilot administration of the PPA. Results for the pilot administration of the Practicum Dispositions Assessment are being compiled and will be included in our next reporting cycle.

Table 6. Pilot Results of Practicum Performance Assessment in Fall 2023

JMU annually completes reports documenting its candidates’ program success and ability to meet licensure requirements. The U.S. Department of Education annually gathers data for Title II of the Higher Education Act from all institutions of higher education with teacher preparation programs. These reports provide publicly-available data about teacher preparation and certification, including licensure pass rates. JMU also provides data to the Virginia Department of Education (VDoE) for biennial reports of teacher readiness.

All our completers have achieved all the state’s licensing requirements before leaving their education program. As the most recent (2023) VDoE Biennial Measures report indicates, 100% of our initial completers passed their licensure assessments. For advanced licensure candidates, 99% of Educational Leadership candidates passed their School Leadership Licensure Assessment (SLLA) by the time of graduation; sample sizes for our advanced licensure Literacy program remain too small to report (n=5). Although Virginia does not require the Praxis for School Psychologists, the School Psychology program reports Praxis data as part of its National Association of School Psychologists (NASP) accreditation; the pass rate for calendar year 2022 (the most recently reported year) was 100%.

JMU’s Title II annual reports and VDoE Biennial Measures reports are available the College of Education assessment reports site:

https://www.jmu.edu/coe/accreditation-and-assessment/state-federal-reports.shtml

All our completers have achieved all the state’s licensing requirements before leaving their education program. Accordingly, JMU’s program completers typically enjoy success finding employment upon graduation.

The Commonwealth of Virginia does not provide comprehensive P-12 employment data to EPPs, and obtaining complete and comprehensive data for program graduates is not currently possible. JMU is piloting an exit survey to track this information more thoroughly in completers. However, the JMU Career Center reports annually on career outcomes for graduating students with known data. The most recent available report for the College of Education, for 2022 graduates, reports outcome data for Bachelor’s-level and graduate-level students. For Bachelor’s-level graduates from the College of Education, no students with known data were still seeking employment 6 months after graduation; 33% were employed, and 67% were pursuing continuing education [at the time of the reported data, most education programs required a master's degree]. For Master’s-level graduates from the College of Education, 95% of students with known data were employed; 2% were pursuing continuing education, and 3% were still seeking employment.

More information about career outcomes for graduates from the College of Education is available at the JMU Career Center website: https://www.jmu.edu/career/careeroutcomes/_files/2022/2022_career_outcomes_-_coe.pdf