Abstract
This article sets out a preliminary balance of the pros and cons following three
full years of experience of using a wireless multiple-choice system in a service
course of statistics teaching. As pointed out by several commentators, the
system can easily be applied to statistics as well as any other subject, and
the multi-choice system does not restrict the types of questions that can
be asked. The output files from the personal response system have been used
for some preliminary post-processing of the students’ answers. This provides
some idea of how the correct and incorrect answers are distributed over the
class. Student feedback on the use of the system is almost exclusively positive.
In the opinion of the present author, the only negative aspect of the system,
is the set-up and break-up time as well as the amount of time it takes to
record an answer from the full class. If these time-constraints can be overcome,
then the aim is to expand the use of the personal response system to all of
the lectures here in Glasgow.
1. Introduction
Over the course of the last five years the Personal
Response System (PRS) has received steady attention and the number of publications
on the use of the system seems to suggest that there is something exciting
going on in the classroom ([1,2]). The PRS is a simple technology to give
students the option of anonymous intervention during a lecture. The low threshold
to implementing the system, nothing more than a collection of remote controls
connected to a computer, means that participation is high and unthreatening.
It has been successfully used in philosophy teaching [3],computing science
lectures [4], psychology [5], economics [6]and statistics [7]. There is no reason why
it should not work for any other subject.
From 1947 until 2000 a group of academics in Holland
and Belgium used a one-button, wired rather than wireless
system to gauge students’ understanding of the lecture. Similar systems were
introduced in the United States in the 1960s. The main aim of such systems
was feedback to the lecturer, rather than an active learning tool, as the
PRS aims to be. Professor Jim Boyle (Mechanical Engineering, Strathclyde University),
as one of the first in the UK, has been using the PRS since 1998, after experimenting
with another handset system (Classtalk) and non-technological alternatives
before then. A good academic website about all aspects of the PRS and similar
systems is provided by Steve Draper (at http://www.psy.gla.ac.uk/~steve/ilig/).
A detailed description of the current PRS and some of its implementations
in statistics can be found in a previous article by the present author [7].
2. The PRS system in practice
2.1. Some recent history
of using the PRS in statistics teaching
Service teaching is an integral part of the teaching by
many statistics departments. The statistics department of the University of
Glasgow provides statistics instruction for students in the life sciences,
chemistry, nursing and others. The largest of these service courses is the
one taught to 200-270 undergraduate psychology students. Throughout one academic
year the students receive four hours per week of statistics instruction together
with a weekly two-hour computer practical, in which the lecture material is
put in a practical context. The computer practicals serve to bridge a gap
between the theoretical lectures and the ultimate use students make of statistics
within their psychology studies. Techniques discussed in the course are wide-ranging
and include, among others, sampling, design of experiments, one-sample and
two-sample inference, ANOVA, simple and multiple regression, repeated measure
analysis, categorical data analysis and multivariate analysis. A team of four
staff members from the statistics department in the University of Glasgow
teaches the course.
Whereas the students appreciate the problem-solving
approach to statistics, it is felt that some appreciation for the central
theoretical concepts of the course is important as well. For example, although
it is important to know that the null hypothesis can be rejected if the p-value
is small a form of performative knowledge, it is important
to know that the p-value does not stand for, for example, the probability
that the null-hypothesis is true. The peculiar nature of Statistics and human
understanding of concepts of uncertainty require some careful focus on theoretical
interpretations. For this reason, in addition to the lectures and the computer
laboratories, a one-hour mass tutorial is held every two weeks.
Whereas the course, both in terms of formal lectures
and computer practicals, has been a sustained success over almost ten years
by all available standards such as student evaluations at the end
of the course as well as the perceived impact of the course after several
years by these same students the tutorial system has had a difficult
and unimpressive record. After a few weeks into the year, attendance rates
would go down to a mere 10-15% and during the tutorials students seemed too
shy to ask questions or respond to questions that lecturers posed them. Clearly
the class size was a problem. There were no resources for additional small
tutorial groups, it was decided three years ago to introduce the PRS system.
Immediately attendance rates to the tutorial went up as reported in [7] and
have stayed up.
2.2. The current situation
Most of the experience with the handset system
has been within a mass tutorial setting, which takes place approximately once
every eight lectures. Rather than integrating the handset system in a lecture
setting, it has been the preference to concentrate the use of the handsets
in a formative assessment session at the end of every two-weekly lecture cycle.
This was more out of necessity, rather than choice for several reasons. Although
the handset receivers have been installed permanently within the lecture room,
the system requires an operator of the computer system as well as the distribution
and collection of the handsets to all the students in the class, which, certainly
in the beginning of the year when most of the 250+ students actually show
up, is a problem.
The question to address here is whether in principle
it would be useful to integrate the handset system in a lecture setting. There
are several advantages of doing so. Integration of lectures and handset assessment
would give immediate feedback not only to the student, but also to the lecturer.
The lecturer can address in the next or even in the very same lecture apparent
difficulties experienced by the students. It can give large lecture courses
the feel of interactiveness, which may well increase the sense of ownership
of the class among students and stimulate their attendance and involvement.
A disadvantage of integrating the PRS in a lecture
setting is the technical difficulty of collecting the answers quickly and,
therefore, the resulting disruption to the pace of the lecture. In order to
ask a single question, elicit the answers and discuss the results in a moderately
size class, it takes at least five minutes. This could be a useful way to
build a moment of rest halfway in a 50-minute lecture, given that, as widely
reported, students’ attention spans typically do not exceed 20-30 minutes.
However, in the personal experience of the present
author when experimenting with the PRS in a lecture, it seemed detrimental
to the quality of the lecture to ask more than one or two strategic questions.
The main practical problem of using the PRS in lectures is that, currently,
it requires to hand out a handset to each student at the beginning of the
lecture and collect it at the end, which for one or two questions is not very
efficient. Accordingly, the most effective use of the handset system in its
current configuration is within a large tutorial setting, which punctuates
the lecture series at regular intervals.
Nevertheless, as the lack of interaction in large
lectures settings is typically perceived rather negatively (evidenced by student
questionnaires, about which more below), a further evaluation of the system
in lectures as pointed out in references [1,6] might be considered in the
future. Handsets are perhaps not an efficient vehicle of delivering a lot
of information, large lectures without much interaction suffer from a strong
negative perception which might have the very same effect if not
worse. A very simple way in which the PRS may be effective in lectures is
by using it to measure understanding: in each lecture segment, consisting
of some didactic unit, the students can press a button whenever they get lost.
When the number of responses, which are shown on the screen in real time,
exceeds a certain threshold (e.g. 20% of all those present), the lecturer
can go back and redo that particular part of the lecture. It is planned to
experiment with this system in the coming academic year.
The handset system by its very nature can only
ask multiple-choice questions. As discussed before [7], this does not necessarily
have to be a limitation to the type of questions that can be asked. Multiple-choice
questions can be framed in such a way as to ask about simple facts, about
application of knowledge or understanding and even about interpretation issues.
Depending on the aim of the tutorial and the aim of the subject, a particular
type of multiple-choice question might be better suited than another. However,
irrespective of subject or aim, the sheer time it takes for asking, processing
and discussing a single PRS question means that the questions need to be framed
carefully by dealing only with important issues. It is particularly useful
to put questions about persistent misunderstandings in a PRS tutorial. By
failing such questions students will be confronted with their misconceptions
and are, thereby, hopefully motivated to correct them.
Merely using a handset tutorial to discuss a large
number and range of questions, including less important details, is particularly
inappropriate. A face-to-face tutorial, or even a handout, would address such
questions better. In the experience of the present author it is impossible
to ask more than 7-9 questions during a single hour. This includes repeats
of questions that were particularly badly answered. In those cases it is worthwhile
to present the class with additional explanation on the general topic of the
question and let the students vote again.
3. Student opinion on PRS
One component of the impact of the personal response
system relies on measuring the opinion of those that have been subjected to
it. It is clear that the PRS has a high novelty value, particularly because
this statistics course was the only place where these students encountered
it. As a variation of the Hawthorne effect, novelty, good or bad, typically
sparks some positive effect for some time. In order to account for this a
sample of 21 students were asked their opinion of only the final semester,
after a full year experience with the handset system in the statistics tutorials.
To avoid an unrepresentative sample, students present at an obligatory attendance
session of the course unrelated to the use of the PRS were selected (nevertheless,
despite best efforts, there was a 20% absence ratio, who were unlikely to
be missing at random).
Table 1. Opinion of 21 mostly second year psychology students on the use and benefits
of the PRS system during the second semester of a statistics service course.
|
Attendance of PRS sessions
|
mean
|
51%
|
SE
|
7%
|
|
|
|
|
|
Influence PRS on attendance
|
more likely
|
52%
|
no effect
|
48%
|
less likely
|
0%
|
|
|
|
Overall opinion PRS
|
very helpful
|
5%
|
helpful
|
75%
|
not helpful
|
10%
|
no opinion
|
10%
|
|
Benefits outweigh disadvantages
|
yes
|
67%
|
neutral
|
33%
|
No
|
0%
|
|
|
As Table 1 suggests the PRS seems in general to
be appreciated. In particular, the PRS seemed to stimulate attendance for
about half of the participants. Attendance to handset sessions was good (50%),
particularly in comparison with the general attendance for the lectures (10-40%
in the second term). Four out of five students found the PRS helpful or very
helpful and none of the students thought that the disadvantages of using the
PRS outweigh the benefits.
Some of the additional comments the students made
put this in an even starker contrast. A comment, such as “[Handsets] make
for good interaction between class + lecturer and also hold my attention better”
shows that the handsets somehow bridge the considerable distance between a
student and a lecturer in front of a large class. A stronger sentiment, such
as reflected in “tutorials are more useful than lectures” and “lectures need
more interaction”, suggests that perhaps it might well be useful to include
the PRS routinely in the lectures.
4. Post-analysis of PRS data
Although the PRS system is anonymous, most systems
do collect and store per student response information. This can be useful
to analyse the data after a handset session. Three questions are particularly
of interest for the lecturer:
(i) which questions caused the most
difficulty
(ii) are there any clusters of questions that
are answered consistently correctly or incorrectly by the students
(iii) are there any subgroups of students that
do better or worse than others.
The first question will probably be obvious from
the handset session, although it is always good to have a record of this fact.
For questions two and three, it can be useful sometimes to visualize the data
via simple clustering.
Fig. 1 Two-way clustering of the answers (correct/incorrect)
to 7 questions over 85 students in a statistics PRS tutorial. Roughly four
groups of students and three groups of questions are distinguished.
Figure1 shows the two-way clustering for a handset
tutorial in a Statistics course for psychology students. The tutorial discussed
7 questions (excluding 2 repeats of questions 5 and 6, question cluster 2).
The figure suggests that there may be 3 clusters of questions and 4 clusters
of students. Question cluster 1 consists of questions 1 through 4, which seems
to have been done well by a large fraction of the students, except those in
student cluster 1. Questions 5 and 6 in question cluster 2 seem to have been
poorly done by most students, whereas cluster 3 or question 7 was done well
by most, except, perhaps surprisingly, by those in student cluster 3, who
otherwise did very well in this tutorial.
Some of these results could be explained. Questions
1 through 4 (question cluster 1) were relatively straightforward, stressing
important points of the course in general. Questions 5 and 6 (question cluster
2) had been discussed in the lectures previously and addressed several common
misunderstandings. The final question (question cluster 3) was a bit tricky
and it seems that the good students were particularly fooled by it.
As the students in this course do not own their
own PRS handset, but have to pick one up every time the PRS is used, it is
unfortunately impossible to follow individual students up over time, except
within a single session. If each student could be equipped with an individual
handset, this would open up not only the possibility of a richer analysis,
but also more detailed feedback to the students as well as the lecturer. It
would be interesting to see, for example, how dropout rates are related to
PRS scores and whether these can be used for early intervention.
5. Discussion and conclusions
Although some studies have suggested that handset
systems might improve examination pass rates [8, 9], use of the PRS in the
Statistics Department at Glasgow University has not been employed in a controlled
fashion in order to suggest any effect of this kind. However, feedback from
students suggests that they enjoy using the PRS in large statistics tutorials.
Moreover, improved attendance rates suggest that the PRS is helpful in stimulating
active learning. Finally, the immediate feedback the PRS provides to the lecturing
staff is helpful in adjusting the delivery of the lectures and support to
students.
This author strongly recommends using the PRS
to anyone. The PRS is being utilised in a service course environment, where
a fraction of the students is expected to be unmotivated to the subject. However,
evidence has shown that the PRS can be used in any didactic environment. Two
warnings from the Glasgow experience are:
(i) Do not underestimate the amount
of time it takes to use the PRS; and
(ii) Prepare carefully the multiple-choice questions
in advance in order to avoid strategic voting and to encourage learning of
the specific type of knowledge that was intended.
To summarise the conclusions of this article a
series of commandments can be stated:
Thou shalt use the personal response system in…
- … a large tutorial environment (full
hour).
- … a short interval halfway during a lecture
or practical.
Why shalt thou use it?
- For formative (rather than summative)
assessment of the students.
- To give students a sense of ownership
of the course.
- To increase student involvement.
- To maintain an individual student’s anonymity.
- For feedback to the lecturer with regard
to student understanding.
How shalt thou use it?
- Using well-designed multiple-choice questions
suited for the particular type of knowledge to be tested.
- Focusing on important points of the subject.
- Dealing with persistent misconceptions
and mistakes.
- Providing lots of feedback to the answers.
- Redoing badly answered questions after
having provided feedback.
How shalt thou not use the PRS?
- As a rapid question-answer series.
- Dealing with minor points or unimportant
subtleties of the subject.
- With insufficient technical support or
knowledge of the system.
As with any “rule” in education, breaking a couple
of these commandments can actually be quite effective from time to time. Moreover,
they do not sum up comprehensively what can be done with the PRS. Having said
this, this author does believe that a lot of personal experience, both positive
and negative, with the PRS is contained in them.
References
[1] Boyle, J.T. and Nicol, D. J. “Using classroom
communication systems to support interaction and discussion in large class
settings”, Association for Learning Technology Journal, 11(3), pp.
43-57 (2003).
[2] Draper, S.W., Cargill, J., & Cutts, Q. "Electronically enhanced
classroom interaction," Australian journal of educational technology,
vol.18 no.1, pp.13-23 (2002).
[3] Stuart, S. A. J., Brown, M. I., Draper, S.
W. (2004) “Using an electronic voting system in logic lectures: one practitioner's
application”, Journal of Computer Assisted Learning, vol. 20, no. 2, pp. 95-102.
[4] Cutts, Q.I., Carbone, A., van Haaster, K.
(2004) “Using an Electronic Voting System to Promote Active Reflection on
Coursework Feedback”, Proceedings of the International Conference on Computer
in Education, Melbourne, Australia.
[5] Draper, S.W. (1998) "Niche-based success in CAL" Computers
and Education, vol. 30, pp.5-8
[6] Elliott, C. (2003) “Using a Personal Response
System in Economics Teaching” International Review of Economics Education,
volume 1, issue 1, pp. 80-86
[7] Wit, E. (2003) “Who wants to be… The use of
a personal response system in statistics teaching”, MSOR Connections, Vol.
3, No. 2, pp. 14-20.
[8] Poulis, J., Massen, C., Robens, E., Gilbert,
M. (1998) “Physics lecturing with audience paced feedback,” Am. J. Physics,
vol.66, pp. 439-441.
[9] Hake, R.R. (1998) “Interactive-engagement
versus traditional methods: a six-thousand-student survey of mechanics test
data for introductory physics courses”, American Journal of Physics,
vol. 66, no. 1, pp. 64–74.
