PAPSA PEER REVIEW PROCESS FOR PROGRAM EVALUATION PLANNING MANUAL
Overall program evaluation in a field as diverse as Pupil Services has always been a difficult task. Typical educational program review procedures, such as those conducted by regional accrediting organizations, have tended to be more appropriate to curricular components than to services, which were touched upon briefly if at all. Some professional associations have designed self-evaluation materials for specific aspects of Pupil Services such as guidance or nursing, but self-evaluation is always open to criticism due to bias on the part of the program staff. An effective antidote for these problems of existing evaluation systems can be found in a peer review process.
Development of the Pennsylvania Model
In 1985 the Pennsylvania Association of Pupil Services Administrators undertook the development of a peer review process that would be available to any school district whose Pupil Services administrator was a member of the organization. The PAPSA annual conference, a three-day affair held in Lancaster that year, was devoted to designing the process. Each participant at the conference was assigned to a subcommittee, whose task was to develop a list of the key issues to be reviewed for a specific area of Pupil Services and a list of those publications and materials apropos to that area, which should be available to reviewers. Subcommittees included guidance, health, special education, psychology, social work/attendance, and speech/language services. At various points during the conference, the subcommittees reported back to the entire group and their findings were critiqued. By the end of the three days, a rough draft was in place and a volunteer was found to draw the separate threads of the review process together into a coherent package. Dr. Robert Cormany, the PAPSA Executive Director at the time, accepted the assignment.
After a final version was approved, by the Association’s Executive Committee, districts were solicited to serve as pilot sites. State College and Lancaster City School Districts volunteered. Since they provided very different settings, one in a more suburban, university environment, and the other with a more inner-city, special needs population, they were deemed to offer an appropriate challenge to the new system. PAPSA members were solicited to serve as members of the visiting teams. The two reviews were successfully conducted in the fall of 1985 and spring of 1986 after which the review procedures were fine tuned based upon the experience of those two on-site visits. An offer was then placed in the association’s newsletter, inviting districts to apply as peer review sites. Each year a number of districts have participated with evident satisfaction.
It should be noted that in order to be eligible for a peer review a member of the school entity’s staff must have been a member of PAPSA for two consecutive years. Otherwise, there will be a charge for the review in addition to the expenses of the team members.
The final process to some extent resembles those used by regional accrediting bodies, but focuses entirely on Pupil Services. The following sections of the chapter will deal with the various aspects of the peer review process.
Application Process/Team Selection
In order to emphasize the district’s commitment to improvement of its Pupil Services program as a result of the peer review, both the district superintendent and the administrator of the Pupil Services department are required to sign a request form indicating their support. Realizing that district Pupil Services programs differ and that the needs for evaluation may vary, the district is asked to indicate which Pupil Services components they wish to have evaluated, the number of staff employed in those programs, and an approximate time of year when they would prefer the review to take place. Districts are encouraged to look at their entire range of Pupil Services, but this is not mandatory.
The request form (Appendix A) is sent to the association’s Executive Director, who then selects a review chairperson from among those individuals who have previously served on review teams. The chairperson, usually in consultation with the Executive Director, selects the remainder of the team and notifies the district of the team’s composition. No person will be included on the team without the concurrence of the district.
Teams typically run from five to eight members depending upon the size of the Pupil Services staff in the district. In cases where only one component, such as Counseling, is being reviewed, the teams may be as small as three members. Teams are composed of individuals with expertise in the various areas to be examined. A typical five-person team for example might consist of a Pupil Services administrator, guidance counselor, school nurse, special education supervisor, and a psychologist. Whereas the original teams used in the Pennsylvania model were comprised solely of Pupil Services administrators from the organization, each district, which has participated in a peer review has been asked to agree that they will permit staff members to serve on future teams. This has provided a growing pool of potential team members upon which chairpersons can draw.
The expenses to the district of a peer review include only the following: (1) rooms for the team for two nights, (2) all meals for team members during the stay, and (3) mileage and tolls incurred during travel. No honorariums are accepted since this process is seen as a professional responsibility and the team members are receiving their regular salaries through their home districts.
Prior to the team’s visit, the district staff will conduct a self-evaluation using the checklists provided by the association. The district may exercise some discretion as to how widely the checklists will be disseminated. At a minimum one should be completed by each Pupil Services staff member and by each district/building administrator. The discrepancies that exist between staff and administration and among staff members provide direction to the visiting team for developing interview questions.
Copies of the self-evaluation forms will be provided to each team member at least two weeks prior to the on-site visit. They will be accompanied by a final agenda of activities, directions for reaching the site, and descriptive materials related to the program.
Time Frame for the Review
Once dates for the visit have been agreed upon, the district contact person and the team chairperson work together to establish an agenda for the review. The following model is typical of that used by visiting teams.
Pre-Session The team will assemble on-site the evening prior to the review for dinner and an orientation by the team chairperson and the district Pupil Services administrator.
First Day Staff interviews, calls to key persons, and visits to programs will be conducted in the morning and afternoon. In the evening the team will discuss its findings and plan for the following day’s activities.
Second Day In the morning the team will continue to meet with staff, parents and students or visit other program components. In the afternoon team members will draw up their commendations and recommendations, after which the team chairperson will provide the district with an oral summary.
A more-detailed sample agenda appears in Appendix B, although these will vary greatly depending on the size of the district, number of pupil services programs to be reviewed, number of buildings, and number of staff.
During the two days of the on-site visit the team members will generally disperse to achieve maximum coverage of program components. Among the activities that they will undertake will be discussions with each member of the Pupil Services staff, all administrators, a sampling of students, parents and teachers from each of which will be gathered information pertaining to the functioning of the Pupil Services program. Visitations will be made to any program components that may be in session such as student programs, staff in-service, or advisory committees. The district shall provide the team with a workroom at a central location in which will be housed any materials and publications pertinent to the Pupil Services program. The team members may schedule time to examine these materials during the day or the two evenings that they are on site.
Preparation of the Report
Before leaving the site, each team member will provide the chairperson with their commendations and recommendations pertaining to the area that they have examined. These will be on the forms provided (Appendix C). In the case of every recommendation, there will be an accompanying rationale explaining what conditions observed during the visit led to that recommendation and what strategies are suggested for implementing improvements. In addition, team members will provide the chairperson with an interview list, which indicates the number of individuals spoken with during the review. These forms will comprise the basis for the final report developed by the chairperson.
The chairperson will select key commendations and recommendations from among those submitted by the team to be presented to the district staff and administration at the exit interview. This selection should be made in cooperation with the team members the final afternoon of the review so that the chairperson may clarify any points pertaining to those items before the team members leave.
One consideration that is be kept in mind, when preparing recommendations, is the potential cost to the district. The suggestion that additional staff or major expenditures are the only method of resolving an issue is avoided wherever possible. If that is indeed seen as the only workable strategy, then it should be offered as one among several alternatives and perhaps indicated as the most feasible in the view of the auditors.
Within two weeks after the completion of the on-site visit, the chairperson shall submit an electronic report to the district contact person. That report shall follow the framework established by the association and shall be signed by the chairperson.
It is acceptable for the chairperson to share copies of the report with team members for comment prior to finalizing it, however, no member of the team will be permitted to retain a copy of the report in the interest of confidentiality. Once the report has been dispatched to the district, a copy should be sent to the Executive Director for his files. Any further dissemination of the content would be at the discretion of the district.
Results and Followup
Each district that has participated in the Pennsylvania peer review process receives a follow-up survey six months later in which they are asked to comment on the process and the value to the district. Out of the first fifty districts to be visited, forty-eight reported that the experience had been very positive. Two were less enthusiastic, basically because the findings reflected negatively on certain staff members, which created some internal difficulties. The teams visiting those districts considered the impact of their recommendations carefully, but felt in both instances that to overlook or sugarcoat the facts was inappropriate. The decisions of what to do with the recommendations remains with the district.
One dichotomy became clear as the peer review process progressed. There are typically three reasons why a district will choose to have an outside team examine its program. Most common is a desire for improvement, second is a wish for a “pat on the back” because the district feels it has a good program, and third is the negative feeling that something or someone is at fault and that this is a way to confirm that feeling before proceeding to make a move against the program or person. All three are legitimate to a greater or lesser extent and should not deter an organization from conducting the review. Even when the review is not being held for the best of reasons, it does provide an opportunity to alleviate the situation by carefully worded commendations and recommendations. It should be noted, however, that the purpose of these reviews is to scrutinize program content and delivery, not individuals. Under no circumstances should the district use the report as a component of a staff member’s evaluation.
In forty-four of the fifty follow-up reports, districts indicated that they had implemented at least some of the recommendations. In twenty-two cases, the districts invited team members back to help them redesign portions of their Pupil Services program. In four instances, the chairperson of the team was invited to discuss the report with the district’s board of education at a public meeting, which had a very positive result. Six districts chose not to implement the recommendations of the teams. In two of those cases, as mentioned previously, the district staff were unhappy with the recommendations, while in four other cases the administration or board of the district felt they could not implement the changes suggested for reasons of cost or political consideration.
Advantages and Disadvantages
In considering whether to implement a peer review process, there are certain inherent advantages and disadvantages to consider.
Advantages include the credence generally given to outside evaluators over an internal evaluation, which may be seen as self-serving. When the peer review is sponsored by a professional organization, this advantage is multiplied. Recommendations of an outside team can be implemented or rejected without the same degree of hard feelings that can exist when an internal committee’s suggestions are being considered. The opportunity to call upon the broader professional community for support in program redesign efforts is a valuable adjunct to the review process itself. The outside review provides a wider perspective on Pupil Services issues from the experience of individuals in other districts.
Disadvantages center around the lack of clout on the part of the outside team to make things happen following the review. Their recommendations can be dismissed out-of-hand. There is a degree of lack of control on the part of the district that may be uncomfortable to some. It is difficult in two days to examine all the factors that may contribute to a certain situation and this may lead to erroneous conclusions on the part of the visiting team.
All in all, the advantages of an outside peer review outweigh any disadvantages, real or fancied, that may be presented. As with any such process, its success depends upon the persons involved and their intent. This should not prove to be a deterrent, but rather a challenge, which can be successfully met by the procedures described above.
Further information can be obtained by contacting the PAPSA Executive Director, Dr. Douglas Arnold, at (570) 656-4286.
Creating an Agenda
While the district will be provided with a sample agenda drawn up by the team chair, it is unlikely that such an agenda can be adopted without revision to meet the district’s needs and timeframe. For example, the sample agenda usually breaks the visit into one-hour intervals, while the district will more likely base the time around the length of class periods.
The following five categories of stakeholders must be included in the process. When it comes to selecting individuals to be interviewed by the team, the district is free to use its discretion. Experience shows that certain approaches seem to yield the best results.
(a). Pupil Services Staff – All pupil services staff in the field(s) to be reviewed should be scheduled for interviews in groups, e.g., elementary counselors, high school counselors, nurses, psychologists, etc. Administrative, supervisory and clerical staff may need to be scheduled separately. Contracted staff, whose duties relate to the program being reviewed may also be included.
(b). Administrators – At a minimum, the team should interview the superintendent and pupil services administrator separately. Building administrators can be scheduled in groups, e.g., elementary principals, middle school principals/assistant principals, etc.
(c). Parents – It is often difficult to get parents into the schools during the day. The district may wish to schedule a meeting for parents on the second evening of the visit. This could involve parents from all grade levels, who the team may then choose to break into levels depending upon the number in attendance. There is no answer to the question of how many parents to invite. Generally the more the better is desirable. An effort should be made to assure that there is a variety of parents included. Those who have had positive experiences with the pupil services and those who have had negative experiences. Those whose children have been successful and those whose children have experienced difficulties. If the meetings are scheduled during the day, then the parents should probably be invited to their child’s school, where they will meet with one team member, rather than combined at a central location. If a parent advisory committee for pupil services exists, that group should be included.
(d). Students – The team will wish to interview a sampling of students at each level. This may be accomplished in two ways. First, the team members could visit classrooms and speak to an entire class at once. This should be a class at the highest grade level in the building, since those students will have had the greatest exposure to the pupil services program. Second, students could be selected across classes to form a heterogeneous group. It is not recommended that individual interviews be scheduled with students, since that is too time-consuming and students are generally more comfortable with their peers.
(e). Teachers/Faculty – Teachers and other members of the faculty should be scheduled in groups. In most instances, schools have chosen to do this at the beginning or end of the day when it is most convenient for the staff to be made available. Often an announcement will be made to the faculty, encouraging any who wish to share comments with the team to be present at a certain location and time.
Districts may choose to have the team interview other stakeholders as well. In some instances, school board members have chosen to participate In other situations, districts may have felt that certain specialists or community agency representatives would provide useful input.
If possible, team members like to see the staff in action. This could involve observing counselors conducting groups or classroom lessons, nurses doing screenings, psychologists participating in student assistance team meetings, etc. These opportunities should only be included if they are already planned during the time the team is on site. Districts are not encouraged to schedule such activities just for purposes of the team visit.
In most districts there are a number of elementary schools. It is not necessary for the team to visit every school. Since they will have interviewed the pupil service staff and building principals in group sessions, and since the purpose is to obtain an overview of the entire program, not to critique distinctions among buildings, there is no need to meet with parents, students and faculty at every site.
Materials to Be Provided
In addition to the self-study instruments, to be completed by each administrator and pupil services staff member, there are a number of other information items that can assist the team in its deliberations. The district’s Chapter 12 Student Services Plan is particularly useful. Copies of job descriptions, brochures, policies and manuals are desirable. If a written program description, scope and sequence, or curriculum exists, the team would benefit from seeing a copy. These may either be sent to the team members beforehand or provided at a central location for the team to peruse during the visit.
Accommodations for the Team
While PAPSA does not assess a fee for its peer reviews, the district is expected to provide for the needs of the team members. This includes the following:
(a). Separate hotel rooms at a nearby facility. The district contact person should make certain that the hotel is aware that the district is paying for the rooms.
(b). Dinner the evening the team arrives. The district should arrange for a dinner at a local restaurant. It is the district’s option as to who will be invited from their staff. In some instances only the contact person has attended, while others have included staff and administration. The purpose of this dinner is to get to know each other informally and become acquainted with the community the district serves.
(c). Meals during the visit. On the first day that team members are in the buildings, interviewing stakeholders, they should be provided with a meal at no cost in the school cafeteria. It is not appropriate to take the team members out for lunch at a restaurant, since this takes a considerably longer time period than eating in-house, which interferes with time for data gathering. Someone on staff should accompany each team member to lunch. The second day, the team will eat together and the district may choose to have a meal catered from a nearby restaurant or from a school cafeteria. Generally districts have provided the team members with a menu from a restaurant or sandwich shop from which they could choose a sandwich and drink.
(d). Travel expenses. The district is expected to reimburse team members individually for their travel expenses to and from the site. Usually this only involves mileage and turnpike tolls. The team chairperson will usually pay for dinner the second night the team is on site and attach that receipt to his/her expense voucher. PAPSA provides an expense voucher to each team member. These will be collected by the chairperson and presented to the district contact person before leaving.
(e). Working space. The team should have a working area that can be used to examine materials and to prepare commendations and recommendations. Preferably this area should be available during the entire visit, but if this isn’t feasible, then the space must be available from 11:00 AM until 3:00 PM on the final day.
* Agenda for the review has been finalized.
* Staff have been made aware of the upcoming review and its purpose.
* All persons to be interviewed have been notified of the time and location for meeting with the team.
* Self-evaluation forms have been completed by staff and administration.
* Hotel reservations have been made and payment by the district confirmed.
* Arrangements for the initial dinner have been made.
* Team members have been sent the required preliminary materials at least two weeks in advance of the visit.