February 27, 2005

EVI-2 Report

This report is for the EVI-2 administration in the spring of 2003.  There were 37 pre-tests and 42 post-tests administered and completed.  A total of 34 students completed both the pre- and post-test.  Repeated measures t-tests were conducted on the item and total scores in order to assess if there was a significant increase in the scores after participation in the program. Table 1 provides descriptive statistics for each item and the total score at both pre- and post-test for the 34 students who completed both tests.  The difference between these means (calculated as post-test mean minus pre-test mean) is also presented, along with the results of the t-tests.  There was a significant increase in the total score from the pre-test to post-test.

Table 1 reveals that, for some items, student performance at pre-test is actually better than performance at post-test.  Although not determined to be statistically significant, more than 5% of the students who answered the following items correctly at pre-test subsequently answered them incorrectly at post-test:  3, 7, 19, 38, and 44.  However, several items show the opposite and desired effect; i.e., more students answer correctly at post-test than at pre-test.  Significant differences were found for the following 15 items for which more than 15% of students who answered incorrectly at pre-test subsequently answered correctly at post-test:  6, 13, 18, 20-23, 25, 26, 29-31, 33, 35, and 36.

Note that the percentage of students answering each item correctly is the same as the mean for each item. 

An examination of Table 1 reveals that there are several items that a majority of students answer correctly at the pre-test, indicating that they enter the program with this knowledge.  If 70% or more of the students answered an item correctly, it was considered that they had prior knowledge of this item.  The following 32 items (68% of the items on the test) show this effect:  1-5, 7-12, 14, 15, 17, 24, 26-28, 31, 32, 34-37, 39-44, 46, and 47. This indicates that students already know the majority of the concepts assessed by the test before completing the program.

A reliability estimate (Cronbachs coefficient alpha) was computed for the total scores from both pre- and post-testing occasions. The result of this analysis is presented in Table 3.  Reliability is the degree to which the items representing a construct are answered consistently.  The more reliable a test, the more confidence we can have that the scores obtained from the administration of the test are essentially the same scores that would be obtained if the test were readministered.  Reliability is expressed numerically, usually as a coefficient that ranges from 0 to 1.00.  A high coefficient indicates high reliability or consistency.  If the test scores were perfectly reliable, the coefficient would be 1.00.  However, test scores are seldom perfectly reliable.  Scores can be affected by errors in measurement or characteristics of the test itself (e. g., ambiguous items, items with no variance).  In general, reliabilities above .70 are considered adequate for program evaluation or research.  For evaluation of individual students, reliabilities closer to .80 are desirable.  Examining the results in Table 3, we can see that scores from the total test can be considered reliable. 

Appendix A contains the frequency distributions of the raw data (i.e., prior to scoring) both before and after the program. Examining these tables will reveal which distracter options were chosen most often at each time period (the option with the * next to it was the correct answer). This can help identify misconceptions students bring to the program and those with which they leave.  Specifically, these results indicate that students may be unclear about the definition of ethics (items 13, 19, and 38) and may not understand the concept of fidelity (item 18) even after completion of the program.

Finally, Table 4 displays the results for the demographic items (48-51).  For this administration of the EVI-2, more males than females participated, over 50% of the participants were freshmen, and most students had not completed the By the Numbers or the Calling the Shots programs.

Table 1

EVI-2 Item and Total Descriptive Statistics, Mean Differences, and t-test Results

Table 3

Reliability of EVI-2

Table 4

Results of Demographics Items 48-51

The breakdown of the total number of students who responded to these demographic items at pre-test and post-test is reported.  The Combined columns are the results for those who responded at both pre- test and post-test.  Some students did not answer these or used a response that is not included in the choice of responses.

Appendix A

Frequency Tables for Items 1-47

The letter P in the item number indicates that this is a post-test item.  The asterisk indicates the correct response. Examine the valid percent column as it does not include missing data.