James Madison University
Office of Institutional Research

Use Of Results For Program Improvement

One of the primary reasons an area conducts a program review is to improve its services to its constituents. Since 1993 OIR has routinely sought external input about the quality of its services and how we might improve. Below is a summary of the recommendations for improvement and the actions that have been taken by OIR to address the recommendations.

Year Recommendation Actions Taken
1993 New personnel are needed to handle demands of users. In 1995 a Statistical Analyst Senior position was added. This position has been upgraded to a Policy and Planning Specialist II.

In 2000 a secretary position was obtained to support OIR and the SACS Self-Study. We hope to keep this position after the Self-Study ends.

1993 Office members should upgrade skills and cross train so that major functions can be continued should a staff member leave. At the time of the 1993 study the director was responsible for most of the analysis work. Since the Policy and Planning Specialist II was hired, analysis activities have been shared. The only major function that the director manages that could not be done easily by other staff is enrollment projections. However, the enrollment spreadsheet is well documented.
1993 The director should focus more on management issues and develop skills which support strategic decision-making. This means moving to where the director is more involved in the strategic processes of the university. Since 1993 the director has become involved in many important functions of the university. In 1998 the office became part of the Division of Institutional Effectiveness. Since then the director has become increasingly involved in strategic decisions, including enrollment forecasting and the development of a Web-based planning process. See resume for a summary of his activities.
1993 Involvement of the office in the facilities inventory needs to be reviewed. The office continues to be involved in the inventory, but another office, Resource Planning, has a staff member that assists with the inventory. Eventually the inventory maintenance may become more decentralized. We are attempting to make the inventory more Web-accessible.
1993 The office should develop a priority system to identify what the key projects are and what will be done on resources available. Each year the director submits objectives to the Vice President for Institutional Effectiveness. Through working with the Vice President, the objectives for each year are developed. Also, annually the office conducts a customer satisfaction survey in which the respondents are asked to prioritize the usefulness of OIR's activities and suggest changes.
1993 The office should get more involved in supporting key management decisions. As a unit in the Division of Institutional Effectiveness the office is continually involved in key decisions about a variety of issues. Over the years the office has been involved in many different issues. See Research Studies and Research Notes for example of studies in which the office has been involved. Also, the director is the point person for the Planning Database, a Web-based planning system that will change dramatically the way JMU plans and evaluates. The director is responsible for enrollment projections, a key management decision that must be made annually.
1993 As the office works with the decision makers, it should engage in activities which put more IR support in the departments. The office needs to translate data into usable information. Progress has been made in this area, but resources have been a limitation. During the past three years the director has chaired a small group of users to create downloads from the administrative systems to support decision-making. Several offices now have this capability. However, funds have not been available to support the development of a university-wide data warehouse. In 2001 a focused program review of OIR's data management capabilities will be completed. One of the objectives will be to determine how OIR can best provide data to decision-makers.
1993 There needs to be a greater facility in obtaining, analyzing, and distributing data extracts to meet data needs. There need to be better access tools. OIR provides a significant amount of data to Academic Affairs. In 2000 Academic Affairs hired a data analyst to collect data on faculty productivity. OIR has attempted to work together with Academic Affairs to provide useful data to the deans and departments. Better tools are now available through PeopleSoft. OIR will be setting up a Web server with historical data in summer 2002 to give users better access to data.
1993 There needs to be cost-benefit data to support decisions. This function has been taken over by Academic Affairs.
1993 The office should establish a set of indices which will measure the amount and quality of the key processes of the university. These indices are similar to critical success factors or key performance indicators. The office has been involved significantly in gathering performance information. A  section of OIR's Web site is devoted to these indicators. Also, OIR has developed a database of performance measures that will be ported to the Web and integrated with the Planning Database. OIR also has an extensive amount of key performance information on peers.

In 2000-01 the State Council of Higher Education for Virginia developed a set of performance measures for all higher education institutions. These measures are known as "Reports of Institutional Effectiveness."

1994

Consistent Student Data on All Students. This indicates an interest in obtaining consistent demographic data about all undergraduate and graduate students across the University.

The office has data on all students in the form of the SCHEV data files. The office is attempting to make the data more accessible through its Web site and Statistical Summary. However, the office needs to do a better job of getting the data to the departments. The focused program review of the office in 2001 should provide better guidance about how the Web can be used for this.
1994

Faculty Activity. We have been collecting information on faculty activity for many years. There appears to be a need to work with the deans and department heads to clarify the definitions so that we can accurately report this information consistently on all faculty.

The office made several clarifications to the Faculty Activity Survey to define better how faculty spend their time. Academic Affairs is now responsible for providing faculty use data. An example of the data provided to the departments can be seen in the annual reports of the academic units.
1994

Staff. There is concern that there is insufficient staff in OIR to provide services to its clients.

OIR has gained additional staff since 1994. In 1995 a data analyst position was added and in 2000 a secretary was hired. At this time (2002) OIR probably has sufficient staff for current functions. It may be useful in future years to add staff who have very strong database and Web skills to assist with providing data to the users. Through the development of the PeopleSoft system more complete data are now available to the university community. The results of the focused program review should provide guidance about how the data can better be shared with the university community. Some data on staff have been added to OIR's Web site. A searchable Web-based database would improve the ability of JMU administrators to gather useful information.
1994

Timely Information. There are many competing demands for information from OIR. There is a need to prioritize these needs so that the most important information can get to those who need it in a timely manner.

OIR's annual survey helps prioritize the information for departments. However, significant improvement is needed in providing timely information to decision-makers. OIR's annual evaluation indicates that the office is seen as providing timely data.

OIR now places its Statistical Summary on the Web. Prior to the Web the annual factbook was printed in February. While we print fewer copies, Statistical Summaries since 1995-96 are on the Web site. Also, pages are placed on the Web as they are finished. Users are notified of completed pages through the quarterly newsletters.

1994

Latest Technology. This office must handle a wide variety and amount of information. The information which must be handled by OIR is increasing tremendously. This goal reflects the need for OIR to gather and report information using the most advanced technology possible.

OIR has made significant progress in using advanced technology. All staff and GAs have adequate computer technology. The Web has provided significant functionality to the office's ability to share data. The office is moving as quickly as it can to provide data via the Web. The tools to provide database access to data have only recently been made available. OIR plans to make major improvements in the sharing of these data.
1994

Agreed upon Definitions. There are definitions of information which relate to important areas of interest like faculty productivity and students. There is a need to ensure that all who use this information understand the definitions in the various databases on campus and in OIR.

The introduction of the PeopleSoft system has greatly improved the definitions of faculty and students. Academic Affairs has been working with the departments to agree on common definitions of faculty productivity.

OIR needs to develop a legend of definitions, sometimes known as a glossary.

1998

Clarify and focus, in consultation with higher levels of administration and in light of limited resources, its mission within the growing and changing needs of the university by:

  • reexamining the primary functions and responsibilities of OIR,

  • identifying the primary customers that OIR is to serve,

  • reviewing specific OIR activities, such as the facilities inventory, to determine if they may be more effectively administered by other units of the university.

Annually OIR reexamines its priorities through the annual customer survey and discussions with the Vice President for Institutional Effectiveness. Summaries of these surveys are on the Web.

The facilities inventory is now a shared responsibility with Resource Planning. Eventually it will be also shared with Academic Affairs as the Resource 25 inventory and scheduling system is fully functioning.

1998 Clarify in all communications the source of initiatives (e.g., SCHEV, the legislature, local administration) OIR is undertaking and the role OIR plays in advancing their implementation. This is being accomplished through the quarterly newsletter and in conversations with JMU staff as needed. Each newsletter has been viewed by more than 350 individuals via the Web. This may need to be clarified more in the newsletter.
1998 Continue to raise the profile of OIR's informational services within the university community, especially awareness of its web site offerings. OIR send periodic email messages to various groups about the web site. OIR annually does a mini-evaluation of its services. In October 2000 OIR instituted a quarterly newsletter to all JMU staff.
1998 Examine the kinds of managerial information those below the vice-president level need and determine whether these are kinds of information OIR can or should provide. Informal conversations have been held with various offices. OIR also coordinates the Data Analysis Team that has developed the specifications for common downloads of information for decision-making across multiple divisions.

OIR will be setting up a Web server in 2002 to provide data to users. Also, SCHEV has put all of the data sent to them by JMU since 1992 into a data warehouse that will be provided to JMU in spring 2002.

1998 Continue to pursue the recommendations of OIR's June 1997 evaluation of the planning process. OIR is working very closely with the Division of Institutional Effectiveness staff to revise the planning process. The web-based planning process (Planning Database) should effectively address many of these concerns. Another evaluation of the planning process will occur within the next 12 months.
1999 From an analysis of the annual evaluation, it is recommended that OIR continue to improve the information is provides on faculty. The Division of Academic Affairs assumed some of this function. OIR continues to work with Academic Affairs to facilitate the creation of useful definitions and data. This is an area that still needs a lot of effort on our part. OIR continues to coordinate external reporting of faculty data.
1999 OIR needs to improve its communication with the university community. In 2000-01 OIR instituted a quarterly newsletter. Each newsletter was read by more than 300 individuals via the Web. The reviews of the newsletters were very positive.
2001 Consider the possibility of making the Statistical Summary more interactive and searchable via the Web. We continue to research data warehousing. Have secured a computer that can become a server in 2002. Exploring the software that should manage the databases.
2001 Continue to develop ROIE measures for SCHEV and monitor JMU performance. Update JMU performance measures database and develop an interactive Web site of measures. Strongly promote ROIE in OIR's quarterly newsletters Updates are being developed this spring (2002). Will report in June to SCHEV. Interactive Web site of measures will be part of Planning Database. Was promoted in 2001 newsletters and will be promoted in May 2002 newsletter.
2001 Review suggestions for additional data for the OIR Web site. Continue to advertise new information about the site. Keep better statistics on site usage. A new table was added for majors by gender was added. TABLE 3-12  On-Campus Undergraduate and Graduate Majors by College, Gender, and Full-Time and Part-Time, Fall 2001.
2001 1. Continue to collect and display peer data on web site. Explore the possibility of creating an interactive web site of peer data. Add counters to Web site for 2001-02. 2. Educate users about the Peer Analysis Tool provided by the National Center for Education Statistics Will explore interest in interactive peer data in March newsletter.
2001 Produce four Research Notes on important topics for the university. Put counters on the Research Notes to indicate number of visits to sites. Research Notes were produced on the following topics:
1. Enrollment projections (October)
2. Fall 2001 enrollment (December)
2001 Complete longitudinal study of faculty at JMU. The first annual report was completed in November. It was shared with Academic Affairs, Academic Council, the Planning Advisory Committee, and the university community (through OIR's quarterly newsletter). It was very favorably received, and suggestions for inclusion in the second report were gathered. The next report is scheduled to be completed in the summer. Study is located on the Web.
2001 Continue to assist with the collection of data for the SACS Self-Study. Develop a document database to contain major university documents for historical purposes. The vast majority of data for the self-study have been collected. We are still working with HR and Academic Affairs to collect data on faculty qualifications. All data have been added to the SACS library. We are planning on developing a CD to share the final report and major documents with the visiting team. These are on schedule. The scheduling of visiting team members with JMU faculty, administrators, and staff will be handled through a variety of electronic software, primarily spreadsheets. We have received samples of scheduling spreadsheets from at least one other institution. Other large institutions have assured us that the scheduling can be managed in this manner.
2001 Complete the OIR program review. The program review was completed. The review focused on the structure of OIR's data files. This was very helpful. See recommendations below
2001 OIR should more strongly promote the use of the CDS by units that must report statistics. This will be promoted in the March 2002 newsletter.
2001 Develop an interactive, searchable database of the fall facilities inventory for deployment on the Web. Explore the feasibility of developing a means for departments to suggest updates to the inventory online. In Fall 2001 Web-based access was developed for searching of inventory by building and department. This is on OIR's web site.
2001 Conduct an evaluation of the planning process and provide feedback to senior management and the Planning Advisory Council At this time this objective has not been formally started. However, it may be wise to defer completion until Phase II of the Planning Database has been completed by IT and fully deployed. A focus group of users of the current system are helping IT with the development of Phase II. This focus group is currently serving as an informal evaluation of the planning process as these individuals represent the major users of the system, including Dr. Rose's executive assistant.
2001 The Board of Visitors financial book should be more heavily promoted in OIR's newsletters in 2001-02. The BOV book will be completed in February 2002 and placed on OIR's web site and promoted in newsletter.
2001 Continue to assist administrative and student support areas with program reviews. Build into the Planning Database reports of accomplishments that can be used in program reviews. Provide two training workshops (one in the fall and one in the spring) on program reviews for administrative and student support units. OIR in the fall supported program reviews by conduct surveys for Student Accounts/Cashiers and Orientation. As yet OIR has not participated on a program review team. Provided consultation to Development and Alumni Relations on program reviews. Offered to provide assistance as needed. Expect to conduct some training for them in the spring. Will approach UR about offering training/consultation. A policy on program has been developed that includes the requirement that Characteristics of Effective Office/Department be included. The policy has been reviewed by the division heads and is awaiting final approval of the President. It will then be placed on the Policies and Procedures Web site. The Planning Database will include an indicator for program review recommendations. Criteria for acceptable objectives have not yet been developed. This will be a late winter/early spring activity. Training for the PD includes references to the incorporation of program review recommendations. This will be emphasized in the training for Version 2.0.
2001 Implement Phase II of the Planning Database with the following features: university Web planning site; budget initiatives; and document database. The Planning Database is being completely rewritten in 2001-02 by Information Technology. The planning Web site for external view will be developed along with the document database. The budget initiative functionality will be developed in 2002-03.
2001-02 Program Review of OIR's database structures Continue using the same processes you are currently using. This recommendation is based
upon the following conclusions:
a. The data provided by Administrative Computing is in the format you are currently using.
b. The data that must be provided to SCHEV is in a flat file format similar to the data you are currently using.
c. The majority of the savings in a redesign would be realized in space savings.
d. The current limit on a Microsoft Access database is 2 gigabytes. The sample database, un-normalized, is only 22 megabytes.
OIR will continue to keep the same processes. However, OIR plans to install a server for these data within the next six months.
2001-02 Keep archive copies of previous years compressed in ZIP files. The sample database, uncompressed, is 21.5 megabytes. Compressed, the database occupies 2.68 megabytes. This compression reduces the storage requirements by almost 90%. These are being kept by the assistant director.
2001-02 The regular use of a database requires both considerable processing power and quick disk
access to be efficient. We recommend that staff members who execute database queries
regularly be equipped with new, powerful computers on a regular basis to maintain
productivity. As the processing demands increase, hardware capabilities must be increased
concomitantly.
The assistant director's PC was upgraded in Fall 2001. Each staff member's PC will be upgraded at least once every three years..