How To Understand And Use Your UTEI Scores

Like any statistics, the UTEI scores have their limitations. But if you take account of a few simple factors you can get some useful information from your UTEI scores.

Using the UTEI results

Perhaps the most useful thing to do with your UTEI scores is to use them for comparisons. You should compare your unit or teaching scores in a particular semester with your scores in previous semesters to see whether any improvement is showing up. If the scores are already good then it may not be possible to achieve much improvement. If the scores show up some weakness you could work out how to go about improving for next time, perhaps in discussion with colleagues who have achieved better results.

You can compare your scores with the scores for your school and faculty (given at the bottom of the report) to see whether your scores are similar, better or worse than these means (or averages). If your scores are much lower than the school or faculty average you could again look into how you might improve for next time.

Once you have compared the Overall Satisfaction (OS) you could look at scores on individual questions and it may be that certain questions have much lower scores than the remaining questions. If so, this may give some lead to things that you could/should address. Another useful source of information - some staff would say the most useful - is the write-in comments by students. You need to look through the actual survey forms to see these. (These forms are usually sent back to the unit coordinator after processing—contact your school UTEI contact officer.)

At the same time you should recognise that the UTEI only attempts to measure student perceptions. There are many other crucial elements to good teaching and effective units. Other feedback (eg from colleagues or from other professional persons) could be quite useful to you and may be constructive in what you are doing, even when student perceptions are not as positive as you might like.

Reporting back to students

A very important step in using the UTEI scores is to report to each new class of students about what you have done to improve the unit as a result of feedback from the UTEI reports from the last group of students in the unit. This shows students that we do take these evaluations seriously and will, we hope, encourage them to provide meaningful feedback when their turn comes to fill in the survey.

Reading the UTEI scores

The UTEI scores are in two parts. In the second column from the right of the individual unit result tables you find the Mean or average score for each separate question. In the far right hand column of the table you will find % Agreement for each question. This refers to the proportion of students who answer positively (either Agree or Strongly Agree) rather than neutral or negative (Disagree or Strongly Disagree). The % Agreement scores typically look larger than the Mean scores, so some staff prefer to use them. But the best measure to use is actually the mean score.

Another important bit of information (see below) is the number of responses. This appears in the column with the heading N on the left. This does not indicate the total number of students enrolled but only includes the number of responses that were actually used in compiling the results—sometimes a lot fewer than the total enrolment. The key part of the report on the individual unit or on the lecturing/tutoring in an individual unit comes at the bottom of the page in a table with column headings in a shaded box. Here the results are summarised into different sections of the survey or ‘Scales’: LS = Learning Support; LG = Learning Guidance; OS = Overall Satisfaction. Apart from Overall Satisfaction each scale incorporates the results from a group of related questions.

These summarised scales results are presented again under the headings Mean (on the left) and % Agreement (on the right). Most importantly these give you a comparison of your unit result (and even different campus groups in the same unit where applicable) with School and Faculty Mean and % Agreement scores, so that you can see whether your scores are close to the school and faculty average or higher or lower.

What is a ‘good’ score on the UTEI?

The Mean scores in these tables are on a 200 point scale of –100 to +100 but the university average hovers around +50. So if your score is close to 50 you are about average. If your score is close to zero or even a negative number you would be well advised to have a good look at whether you need to make some changes. Once again it is appropriate to remember that a whole range of factors go into these student perception scores and low scores could conceivably still occur with excellent teaching in an excellent unit. But if you find low scores you should carefully examine what is happening in the unit and why, in order to ensure that all fixable problems are eliminated.

How different is different? or What is a meaningful change?

We all know that a mean or an average score summarises a whole range of scores, some of which may be much higher and some much lower than the average. What is less well recognised is that, unless every single person in the group responds to every question, even the means themselves actually represent a statistical estimate of the mean. And it is almost unknown that every student enrolled in the unit will be ‘present and voting’ on the day the survey is carried out. And sometimes a number of the responses of the students present on the day end up not being included because of errors in coding/reading the forms. (If you find any such problems please bring them to the attention of your Head of School or Program Director so that they can be rectified.)

Basically the range of this estimated mean is determined by how many responses are included (N on the results tables) and by the proportion of the total class/unit that that represents (not currently shown on the results reports). The more responses, the smaller the range. The higher the proportion of the total group that the responses represent, the smaller the range.

An example

Let’s explain this by means of a simple example of overall satisfaction using tables of what is called ‘sampling error’ calculated in relation to the UTEI results. Sampling error is the statistical factor behind the range that we are talking about. It recognises that the result came from a particular group of students out of the whole unit enrolment and that if a different or partially different group had answered instead, their results might have been a bit different. The sampling error tries to quantify how different the results could be as a result of different groups of the same size.

If you get a mean Overall Satisfaction score of 50 in a particular unit, for instance, this could mean a range of different things depending on the circumstances. If this score comes from 25 responses then the sampling error is ±15 if the total class is 62 but ±8 if the total class/unit is only 31. That is, if the class is 62, then 25 responses represents 40% of the total number of students in the unit and so the score of 50 actually represents a mean with a range from 35 to 65 (=50 ±15). If the total unit/class is 31, then 25 responses represents 80% of the unit and the score of 50 represents a range from 42 to 58 (=50±8).

An example of improvement

What this means, then, is that a change or difference of less than the sampling error is not statistically significant. If you get 25 responses from a class of 62 in two successive offerings of the unit and you see an improvement of, say, 12, this is less than the sampling error and effectively means no significant change, because it falls within the range of variation of the mean. If you see an improvement of 18 or 20, however, then this is greater than the sampling error and represents a small but genuine improvement. If the same improvements are observed, on the other hand, with 25 responses in a total enrolment of 31, then even an improvement of 12 in the overall satisfaction score does represent an improvement because it is greater than the sampling error of ±8.

An example of comparison

The same applies to comparisons with the school or faculty means. If you have a unit of 62, with 25 responses (40% response rate), and your score is 12 better than the school mean, your score is not to be considered significantly above the school mean in statistical terms, because this score falls within the sampling error of ±15. But if the 25 responses come from a class of 31 (80% response rate) the same difference of 12 would be statistically significant because it is greater than the sampling error of ±8.

The extremes

At the extremes as calculated for the UTEI, 5 responses out of a unit of 25 (20% response rate) has a sampling error of ±38, so a score of 50 actually represents a range of means from 12 to 88. 2500 responses out of a group of 3125 (eg a school with an 80% response rate across all its units overall) has a sampling error of ±1. That is a score of 50 still indicates a range between 49 and 51. The sampling error is still the same (±1) even if 2500 responses represents 40% response rate in a group of 6250 students). For more detail see the appendix.

How you should treat these issues of significance

In terms of statistical significance you should treat scores with caution unless you have a large number of responses in your unit and unless the number of responses received represents a relatively high proportion of the students enrolled in the unit or unless the difference/change in scores is very large. But when score differences are large but not statistically significant you might also want to investigate the cause of the differences.

Because sampling errors can be quite large with small samples, it is important to encourage students to participate fully in the UTEI survey—it means that the results are more reliable and thus more useful to us all, especially at the broader levels such as school and faculty, where the number of responses can be very large.

APPENDIX

Sampling Error of a UTEI Scale Mean

Note: The Sampling Errors are smaller for the scales on the right (UO, LS etc.) than for Overall Satisfaction (OS), because the scales are made up of several questions each while Overall Satisfaction is just one question.