UNCW Home UNCW HomeBreaking News! Click here for details.

 Faculty Senate

Minutes

|

 
Tuesday, 27 April 2004
 
See Agenda
 
Meeting 04-09

Roll call

Absent: Adams (Nursing), Cami-Vela (FLL), Chen (History), Clark (Business), Clements (Psychology), Collins (Curricular Studies), Conner (Art & Theater), DePaolo (Chancellor), Dworkin (Psychology; Bookstore Committee), Hosier (Provost), Hurdle (Social Work), Kubasko (Specialty Studies), Maume (Sociology & Criminal Justice), McGiboney (FLL), Olsen (Communication Studies), Rishel (Information Systems), Roer (Graduate School), Schmid (Philosophy & Religion), Seaton (Chemistry), Seiple (CAS), Simmons (Anthropology), Usilton (History), Walker (Curricular Studies), White (Creative Writing), Whitehead (Economics & Finance), Wilson (Philosophy & Religion)

Approval of minutes

April 20 minutes were approved.

Old Business

In this special session, the Senate considered and passed one motion (04-08-29, below) revising the current SPOT instrument and procedures. The Senate first acted on the following amendments to the motion:

  1. Amendments [insertions, deletions]
    1. [Amendment carried:] Change the final question (Q16) as follows: "Rate your instructor’s overall teaching effectiveness in making this course a valuable learning experience for you."
    2. [Amendment failed:] Delete the final question (Q16).
    3. [Amendment failed:] Change the final question (Q16) from a 7-point to a 5-point scale.
    4. [Amendment carried:] Invert response scales for all 16 questions so that the positive responses, such as "strongly agree" and "highly effective," will appear on the right, rather than the left, of the instrument.
    5. [Amendment carried:] Revise Q10 as follows: "The instructor made appropriate use of used appropriate educational resources (for example, field experiences, library, technology) to promote learning."
    6. [Amendment failed:] Delete hyphens in "well-organized" (Q2) and "well-explained" (Q3).
    7. [Amendment carried:] Revise Q14 as follows: "Grading policies were clearly explained and uniformly applied clear."
    8. [Amendment failed:] Add new question: "I was fairly graded."
    9. [Amendment carried:] Add new question, numbered as Q15: "Course policies were uniformly applied."
    10. [Amendment failed:] Remove the Q number from the final question (Q16) so that it appears as an unnumbered question.
    11. [Amendment failed:] Add new question: "The instructor was knowledgeable about the subject matter of the course."
    12. [Amendment failed:] Require that full implementation of the revised SPOT survey be delayed until the survey instrument’s validity and reliability have been assessed, and that the required use of the SPOT for the full faculty only follow an analysis of the results from the pre-tests.
       
  2. The main motion, as revised above, then carried without a dissenting vote:

    [Motion 04-08-29; revise SPOTs; carried] That the following changes in the SPOT instrument and guidelines be adopted for implementation in the Fall Semester 2004:

I.       Changes to the SPOT instrument

    Replace the current instructions and Questions 1-16 by the following:

    To the student:  Please respond honestly and fairly. Your responses to these questions will be combined with perceptions of other students to identify your instructor’s teaching strengths, to suggest ways your instructor might improve his or her teaching, and to provide information that is used to evaluate faculty teaching performance. 

    Special instructions:  To ensure anonymity, please do not identify yourself by name or number on any part of this questionnaire. This form is already precoded with your course and section numbers and with your instructor’s name. Your instructor will not receive results of this evaluation until after final grades are turned in, and will not be present when you complete this survey.

    Think about each of the instructor’s practices as it contributed to your learning in this course.

    Organization and Clarity

      1. The instructor gave clear explanations.

    Strongly disagree         Disagree       Neither agree nor disagree      Agree           Strongly agree     Not applicable

      1. The course was well-organized.

    Strongly disagree         Disagree       Neither agree nor disagree      Agree           Strongly agree     Not applicable

      1. The objectives of the course were well-explained.

    Strongly disagree         Disagree       Neither agree nor disagree      Agree           Strongly agree     Not applicable

      1. The instructor’s teaching methods were effective.

    Strongly disagree         Disagree       Neither agree nor disagree      Agree           Strongly agree     Not applicable

    Interest in the Subject

      1. The instructor helped to make the subject matter interesting.

    Strongly disagree         Disagree       Neither agree nor disagree      Agree           Strongly agree     Not applicable

      1. The instructor showed enthusiasm when teaching.

    Strongly disagree         Disagree       Neither agree nor disagree      Agree           Strongly agree     Not applicable

    Learning

      1. The instructor motivated me to learn in this course.

    Strongly disagree         Disagree       Neither agree nor disagree      Agree           Strongly agree     Not applicable

      1. The course materials helped me understand the course concepts.

    Strongly disagree         Disagree       Neither agree nor disagree      Agree           Strongly agree     Not applicable

      1. The instructor provided sufficient feedback in this course.

    Strongly disagree         Disagree       Neither agree nor disagree      Agree           Strongly agree     Not applicable

      1. The instructor used appropriate educational resources to promote learning.

    Strongly disagree         Disagree       Neither agree nor disagree      Agree           Strongly agree     Not applicable

    Instructor Interaction with Students

      1. The instructor was concerned with whether or not the students were learning.

    Strongly disagree         Disagree       Neither agree nor disagree      Agree           Strongly agree     Not applicable

      1. Questions were answered satisfactorily by the instructor.

    Strongly disagree         Disagree       Neither agree nor disagree      Agree           Strongly agree     Not applicable

      1. Assistance from the instructor outside of class was readily available.

    Strongly disagree         Disagree       Neither agree nor disagree      Agree           Strongly agree     Not applicable

      1. Grading policies were clear.

    Strongly disagree         Disagree       Neither agree nor disagree      Agree           Strongly agree     Not applicable

    1. Course policies were uniformly applied.

    Strongly disagree         Disagree       Neither agree nor disagree      Agree           Strongly agree     Not applicable

    Overall Evaluation

      1. Rate your instructor’s overall teaching effectiveness in this course.

                1          2          3          4          5          6          7
         
    Very ineffective                                                       Very effective  

    Background Information

        Delete these questions and renumber the remaining questions as necessary:

    Comments:   The comments section is unaltered.

II. Changes in the Faculty Handbook [insertions, deletions]:

    Guidelines for the administration, use, and interpretation of the "Student Perception of Teaching" (SPOT) evaluation

    Revised by the Faculty Senate spring 2000 and spring 2004

    The reliability of data gathered by way of student evaluation instruments depends, in part, on the establishment of a set of common practices for administration and use. The following statements constitute a set of guidelines for the administration and use of SPOT.


    A. Administration
    1. Administration of the instrument shall ordinarily be conducted during the last ten class days of the semester (last five class days in a summer session) at a time convenient to the instructor. Administration at other times will be determined by agreement of the dean, the department chairperson and the faculty member. Days when tests are being given or returned shall be avoided when possible.
    2. SPOT shall ordinarily be used by all instructors in all courses every semester including summer sessions. Paper versions of SPOT will be used in traditional classroom settings, and online courses will use an online version of SPOT. Recognizing, however, that some courses rely heavily on specialized, non-classroom learning experiences (e.g., field-based; hospital-based; laboratory-based; performance-based), exceptions may be established at the departmental level by mutual consent of a faculty member and the appropriate chairperson. In such cases, some method of student evaluation shall be implemented by the department chairperson.
    3. Should departments wish to use additional evaluation instruments, these departmental instruments shall be administered after the administration of SPOT.
    4. Administration of the paper SPOTs shall be delegated to an individual other than the instructor. That individual may be a student or another faculty member.
    5. A brief standardized statement of instruction shall be presented to each class prior to the administration of SPOT.
    6. During the administration of the paper SPOTs, the instructor shall leave the classroom and its vicinity.
    7. Departments shall avoid practices which compromise student anonymity (i.e., student names and/or identification numbers shall not appear on evaluation forms).
    8. Following administration of the paper SPOTs, the evaluation forms shall be sealed in an envelope and returned immediately to the departmental office. Department chairpersons will keep these secure and will forward them for processing. No analysis or interpretation is to be made by anyone prior to processing of the SPOT forms by Computing Services.

    B. Analysis and reporting
    1. Academic departments/units shall deliver the administered forms, with blank forms removed, to the Office of Computing and Information Systems (OCIS) Operations, Hoggard Room 206, Computing Services by the last working day of final exams (within one week of final exams for summer sessions) for analysis. The analysis will not be done until after all grades have been submitted to the registrar.
    2. Three copies of a course section summary (for each instructor, if team-taught) shall be prepared; one for the instructor, one for the department chairperson, and one for the instructor's dean. This summary shall contain, for each item Q1 through Q16 and for any optional supplemental items the percentage of responses in each response category. For Q16 the summary shall contain the individual's response mean, the individual's response standard deviation, the individual's minimum and maximum responses, the number of students enrolled in the section, the number of students responding, the departmental response mean, and the departmental response standard deviation. In addition, the instructor shall receive one copy of the response frequencies of all SPOT items, including the demographic information.
    3. OCIS Computing Services shall also provide to each instructor and his or her department chairperson and dean a Question 16 Section Summary for each section evaluated by SPOT. That summary shall contain:
      1. course and section number, instructor's name, and semester (or summer session);
      2. the section mean on Question 16;
      3. quintiles for the distribution of Question 16 section means within the instructor's department or comparable academic unit (provided that section means for at least ten courses and/or course sections are available), within the instructor's college or school, and campus-wide for that semester or summer session (these quintiles should be reported visually on a scale that also shows the instructor's Question 16 mean);
      4. a verbal characterization of the student responses, that recognized the imprecision of the evaluation instrument and incorporates an objective interpretation of the Question 16 mean and the standard error of student responses;
      5. a histogram of the responses to Question 16 by students in this section.
    4. Every personnel action recommendation for reappointment, promotion, or tenure or post-tenure review should contain a summary, in a standard format, of the individual's SPOT results for Q16 (at least) over the most recent two-and-one-half years, together with a visual representation of trends and the relevant departmental and university quintiles. (An accumulation of Question 16 Section Summaries over that period would accomplish this.) All RTP recommendations shall include a qualitative interpretation of SPOT results by the department chairperson, and may include—at the individual's discretion—the individual's own qualitative interpretation. All statistical calculations and quantitative analysis processed by anyone other than OCIS Computing Services (which is discouraged) must be clearly identified as such.

    C. Guidelines for appropriate use of SPOT results
    1. Data from individual faculty gathered through the use of SPOT shall be treated with confidentiality and with recognition of the need for continued study of the meaning and validity of these data. The data shall not be released by anyone other than the faculty member to anyone who is not directly involved with evaluation for the purpose of reappointment, promotion, tenure, post-tenure review, or annual departmental review, or to anyone who is not directly involved with the development of norms, without the written permission of the faculty member. Each department shall use a release form that will enable instructors to designate other individuals or groups who may have access to evaluation information. In addition, quantitative data shall not be released from the department, or comparable administrative unit, without an accompanying written interpretation of the data by the appropriate evaluating officer and, if he/she chooses, by the faculty member. The evaluating officer's interpretation shall explain how an instructor's scores compare with peers in the same department, discipline, or course assignment, as appropriate. Because numerous studies have indicated that both peer and student evaluations are necessary for the equitable evaluation of teaching effectiveness, it is strongly suggested that peer and student evaluations be given similar emphasis in personnel recommendations.
    2. The Evaluation Committee of the Faculty Senate is charged with regularly reviewing both student and peer evaluation procedures, and with reporting and making recommendations for improvement to the Senate.
    3. Instructors shall be given no access to individual response forms prior to submission of grades and completion of processing by OCIS Computing Services.
    4. In the case of a formal appeal of a reappointment, promotion, or tenure, or post-tenure review recommendation, all parties involved directly in the appeal process shall be allowed access to the archived data pertinent to that case.
    5. Individual SPOT results, when combined with qualitative interpretation by the department chairperson and with peer evaluations of teaching, can contribute to measuring an individual's teaching effectiveness and to identification of areas of strength and areas where improvement is possible. Under those conditions, SPOT results are appropriately used for annual merit evaluation summaries, consideration for salary raises and, RTP, and post-tenure review decisions.

    D. Warnings against inappropriate use of SPOT results
    1. Standard deviations that are reported by section (resp., department) for each item measure the extent to which student responses are "scattered" within that section (resp., department). They do not measure the manner in which instructor means are distributed, hence should not be used to conjecture what percentile an instructor's mean score represents (or even how good or how bad a mean score is). Quintiles as described in Item 11 Section B are used for that purpose.
    2. Means for the sixteen SPOT items must not be "averaged" to produce a "combined SPOT score." (There are lots of reasons for this. Among them are: Question 5 and 6 have a different response scale, campus-wide means are not the same for each item and so they are not weighted properly when averaged, etc.)
    3. Mean scores for two or more courses must not be averaged to obtain a "semester SPOT score" for an individual. (Again, there are many reasons. Responses in different courses measure different things, it ignores class size, and it can be used to obscure instances of particularly good or poor performance.)
    4. Averaging SPOT scores from several different courses across several semesters to obtain an "overall individual SPOT score" is improper.
    5. Direct comparisons of ratings from the version of SPOT used from fall 1992 through summer 2004 to ratings from the revised version implemented in fall 2004 are not appropriate.

    E. Guidelines for interpretation of SPOT results

    Guidelines for SPOT ratings collected 1992-2004:

    1. There is strong evidence that the SPOT questions as a whole give a valid measure of characteristics of effective teaching, and that the results are reliable. Moreover, there are ample reasons to support the use of the Question 16 section mean as the best single measure of student perception of teaching.
    2. SPOT scores should, whenever possible, be viewed in the context of the immediately preceding five semesters. Comparisons should be general and should not ascribe meaning to the precision with which means are reported. (For example, a mean of 4.22 on Question 16 for a certain course might properly be described as lying in the second highest quintile of UNCW Question 16 scores, but should not be viewed as different from a score of, say, 4.18.)
    3. The receipt of a Q16 section mean in the lowest quintile is not necessarily an indication of poor teaching. Only 2.6% of student responses campus-wide to Question 16 are "poor", and if every student were to answer "average" to Question 16, the mean (3.00) would lie in the lowest quintile. However, receipt of Question 16 means in the lowest quintile over a period of several semesters may indicate an opportunity for improvement. Examination of results of other SPOT items, consultation with the department chairperson, and peer evaluations may reveal ways to improve student perception of an individual's teaching.

    Guidelines for SPOT ratings collected from the revised version implemented in fall 2004:

    Revised SPOT questions have been selected from reliability-tested instruments at UNCW and other institutions, and have been edited by the best judgment and experience of UNCW faculty. Revisions to the 1992-2004 SPOT instrument have been made in every case to improve the philosophy of the survey as a whole, the survey questions themselves, and the quality of the information collected.

III.  Review of SPOT revisions

The Faculty Senate Evaluation Committee is charged to perform statistical reliability and validity tests on SPOT data collected for the 2004-2005 academic year (using the newly revised version of SPOT) and report their findings to the Faculty Senate in spring 2006. At that time, the Evaluation Committee shall also recommend to the Senate guidelines for interpretation of SPOT to be posted in the Faculty Handbook, and to be applied in all uses of SPOT for faculty evaluation. The Evaluation Committee is further charged with making recommendations not later than spring 2006 for specific ways to balance student evaluation of teaching with peer evaluation of teaching in the evaluation of faculty.

Adjournment

Minutes by Richard Veit, Senate Secretary