Maths CAA Series: July 2005

Page Guide: Home > Articles > maths-caa-series >

Use and Reuse of Simulations in Mathematical Online Assessment

by

Ruth Thomas

JeLSIM Partnership
Web: www.jelsim.org
Email: rct@jelsim.org

and

Helen Ashton

School of Mathematical and Computer Sciences,
Heriot-Watt University, Edinburgh.
Email: H.S.Ashton@hw.ac.uk

 

Index to article

1. Introduction
2. Use of simulations in online assessment
3. Integrating simulations and assessment
4. Applications of the system
5. Future
Further Information
References

1. Introduction

The focus of this paper is the use of simulations in assessment.  The paper describes the work undertaken by Heriot-Watt University and the JeLSIM partnership.  It looks at the drivers behind the pilot project integrating simulations and online assessment, explains the technology and then explores and demonstrates the potential of the system within a mathematical domain. 

2. Use of simulations in online assessment

Simulations are, as Alessi and Trollip [1] point out, one of the few media that have been “embraced equally by behavioural versus cognitive psychologists and by instructivist versus constructivist educators”.  They are a uniquely interactive medium in that they allow the learner to manipulate and explore a system but also provide learners with feedback from their actions as they do so.

Simulations are accordingly used to enhance education in a wide range of domains and in many different ways.  They can be used to facilitate visualisation of dynamic or complex behaviour.  Simulations are used to allow students to develop a feel for the relationships between the underlying factors governing a system, develop an understanding of what constitutes an actual condition within the system, to promote an appreciation of appropriate ranges for system parameters [2] and to give a qualitative feel for the system before the introduction of underlying theory [3].  From a constructivist perspective simulations can be used to provide experiences that will help students revise and build on their current understanding of the world using techniques such as Predict-Observe-Explain (e.g.  [4, 5]).  Simulations allow students to manipulate the parameters to test hypotheses, trying out “what if” scenarios, so they have a central role in scientific discovery learning [6] that is characterised by learners discovering concepts for themselves by designing and performing scientific experiments.  They can be used to develop thinking and problem solving skills such as troubleshooting and faultfinding.  In training, they are used to teach learners about processes and procedures.  (Also, though not the focus of this paper, they are used to teach students about simulation and modelling).

The opportunities for using simulations in assessment should be equally broad.  By combining simulations with assessment technology the same tools can be used for learning and assessment.  This can be particularly beneficial where learning outcomes require more than the demonstration of knowledge.

3. Integrating simulations and assessment

To take advantage of the potential offered by using simulations within assessment a pilot project funded by the PASS-IT project [7-9] was undertaken to integrate simulations within an online assessment system.  In this section, the two individual systems (The PASS-IT Assessment Engine (PAE) and the JeLSIM simulation toolkit) are first described, followed by the functionality of the integrated system.

3.1.  The PASS-IT Assessment engine

The PASS-IT Assessment System consists of a number of components:

Descriptions of some of these components in both the PASS-IT system, and its forerunner (CUE) can be found elsewhere (Paterson [10], Ashton and Schofield [11] and in other group publications [12]).

There are a number of areas of PAE functionality that have proven to be useful for the assessment of Mathematics:

Displaying mathematical expressions: question authors and students both communicate mathematical expressions to the system in the form of the single line string mathematical expression (i.e.  x^2 for x2).  The system has a tool that converts this into a more familiar form for presentation to the student, either within the context of the question, or to confirm interpretation of a students answer.

Random Variables: the system allows question authors to create random variables for use within questions.  These random variables can be used both in displaying information to the student, and in specifying the acceptable answers to the marking system.  In particular, these randoms can easily be included in mathematical expressions.  To enhance this feature even more, mathematical expressions can have simplification settings applied to them, ensuring that mathematical expressions are displayed in an appropriate form.  For example, consider the expression ax, where a is a random integer value between –5 and +5.  Simplification settings can be applied to ensure that:

The use of random variables is particularly useful to provide repeated formative practice for students, and to enable individualised questions to be presented in summative assessments, avoiding some of the issues involved in the close proximity of computer screens in many computer rooms.

Marking Mathematically Equivalent Answers: the marking system for mathematical expressions allows mathematically equivalent expressions to be marked.  This means that a marking scheme answer of 2x will also mark other mathematically equivalent expressions as correct, such as 2.0x, x+x and so on.  Where mathematical equivalence is undesirable, additional features can be used to exclude certain types of answers.

This feature is very useful when specifying the marking scheme since, for more complex expressions, there may be many ways in which the answer can be written, and this feature removes the need to specify them.

The use of steps for awarding partial credit: questions can be authored with optional steps which a student can choose if they need help in completing the required task.  This has proved to be useful in both formative and summative assessment: in the former providing learning support and in the latter, enabling students to receive credit for the parts of work they can complete.  More on this topic can be found in Ashton et al.  [13] and Beevers et al.  [14].

3.2.  JeLSIM toolset

The JeLSIM toolkit is a suite of software written to support the easy creation and customisation of simulations.  It provides a toolkit for the rapid creation of Java applets through a visual interface not unlike a traditional drawing package.  The simulations produced by the toolkit are small to medium desktop applications usable as components within an eLearning module or problem-solving environment.  The key feature of the tools is that they separate the model of the simulated system from the visualisation of the system.  The system is illustrated in figure 1. 

Fig 1 shows a block diagram showing how a simple system code has several views for teacher, designer, learner etc

Figure 1: The Separation of Model and Visualisation

A model is written by a Java programmer based on a design specification in consultation with a design team.  The aim is that the model produced will be re-usable for a wide range of scenarios and levels.  The design of the JeLSIM toolkit means that even novice programmers can create effective models because the Java code describes only the algorithm and not the user interface.  The intention is that once written, the model code does not change and the programmer's involvement ends. 

A visualisation is created by a teacher or instructional designer.  The tools themselves present a visual environment for the creation of new visualisations, rather like a computer drawing package.  A set of common visualisation objects – graphs, tables, digital inputs and outputs etc.  are provided – which the visualisation author can use to display the value of any of the input and output properties for the simulation.  A typical interface would consist of a number of input parameters and visualisation objects to illustrate the outputs.  Once finished, the simulation is easily deployed (as a Java applet embedded in a web page, or even an IMS Content Package). 

The tools (which are being moved to open source) together with a number of simulations, are freely available from http://www.jelsim.org.

3.3.  The Integrated System

The combined system integrates the PASS-IT Assessment Engine (PAE) and JeLSIM simulations. 

Figure 2 shows the PAE–JeLSIM system diagrammatically.  JeLSIM Java simulations are made available as applets for use in questions that are set, answered and marked using PAE.  The question setting teams have control of the simulation interface and starting state as well as the types of questions posed and the way in which the questions are asked.  Communication between the simulation and the assessment system allows initial state information to be sent from the assessment system into the simulation, or student answers and feedback given by the simulation to be sent to the assessment system for recording and marking.

Fig 2 shows how question setting team interacts with JeLSIM simulation (interface and state) and then onward to the Assessment Engine (PAE) to set Single/multiple variables, expressions, feedback and set or randomise initial state.

Figure 2: The PAE - JeLSIM integration

The combination of tools provides a system that empowers the non-programmer to produce questions utilising sophisticated interactivity.

In summary, the system is used in the following manner:

    1. The system allows the use of Java simulations as a component of questions.  A Java programmer is required initially to program the JeLSIM simulation models or new question templates (see the following section);
    2. The question setter produces a simulation visualisation using a JeLSIM model or template.  Visualisations are built up using the JeLSIM library of ready made interface components for data input and output (e.g.  sliders, graphs, meters, tables etc);
    3. The system allows the question setter to define which of the simulation variables will be used for marking by the assessment engine.  This could be a single variable or a combination of variables;
    4. The question setter defines the marks PAE will assign for the answers to different question parts;
    5. The question setter can control the state in which the simulation starts – either by editing the simulation, or by setting an initial state for it in the assessment engine.  The assessment engine's randomisation capability can, if required, be used to randomise the simulation starting state;
    6. When answering the question, the learner uses the simulation, manipulating it to a desired state, and then “submits” an answer to the assessment engine by pressing a button within the simulation or within PAE.  (This passes the relevant simulation variables to the assessment engine for marking).

Templates

It is easier to modify an existing resource than create a new one and it eases the simulation question production process, particularly for novice users, if template question types for specific questions can be created.  In the JeLSIM-PAE project a number of such templates were created which provided a straightforward resource that can be modified by non-programmers to create new questions.  Each resource consisted of:

A template effectively provides a way of creating a highly specialized new question type.

4. Applications of the system

It is useful to consider the way in which the system can be applied by using the following four categories:

  1. Using the simulation to aid in setting the question
  2. Using the simulation to provide new answer mechanisms
  3. Using the simulation to allow exploration and provide feedback
  4. Using the simulation to set more complex questions

Each of these categories is now considered in more detail. 

4.1.  Using the simulation to aid in setting the question

In mathematics and science it is important that diagrams and graphs are accurate.  It is often useful to be able to reuse diagrams and graphs with different variables.  Handcrafting these can be time consuming and the PAE-JeLSIM system provides a way of easily accomplishing this.

Fixed or randomised graphics

The question setter can control the state in which the simulation starts – either by editing the simulation, or by setting an initial state in the assessment engine.  The assessment engine’s randomisation capability can, if required, be used to randomise the simulation starting state. 

The randomised variables could be labels on a diagram or graph, data points, or system variables.  If the simulation visualisation is designed with no input capability, then this effectively allows the creation of randomised images.  This is particularly useful where questions are reused frequently, or where the simulation applies to a number of different situations.

Example: In this example a very simple simulation is used to plot the two lines on the graph.  The template, based on this simulation can be used to create a graphic of any number of randomised data arrays to be plotted as points or line graphs.  The display of the information (i.e.  the axis labels, line colours and thicknesses etc.) is set within the visualisation in JeLSIM.  The information for the points for each line is set by the question author in the assessment system.  In addition in this example, the starting time of the greyhound is randomised by the assessment system.  This example can be explored online. 

Fig 3 shows a simulation as a customised, fixed graph to support the question: At a greyhound track, the greyhounds are automatically released when an artificial hare crosses the starting line. The speed time graph shows the motion of one greyhound and the hare from the time when the hare crosses the starting line. The question asks: How long does it take for the greyhound to start moving after the hare crosses the starting line?

Figure 3: An example of using a simulation as a customised, fixed graphic

Another useful template for creating graphics is one that makes use of an expression parser that allows the question setter to include expressions that are displayed graphically.  The variables within the expression can be randomised and this allows the production of randomised functions.  The example here is used to display a cubic polynomial of the form d(x-a)(x-b)(x-c), where a, b, c and d are random variables.  This example can be explored online.

Fig 4 shows an example cubic simulation to demonstrate customised graphical information.. The screen example reads: The cubic polynomial P(x)=d(x-a)(x-b)(x-c) is shown below. Write down P as a function of x in factorised form. (a,b,c and d are integers. The graph in the figure shows a sample plot from x=-6 to +6 but the user can alter this range to zoom in and out; also placing the cursor on graph gives a readout of the x,y co-ordinates. An input box and submit button are shown under the graph for the user to enter an answer.

Figure 4: An example of using a simulation to display customised graphical information

4.2.  Using the simulation to provide new answer mechanisms

When answering the question, the learner uses the simulation, manipulating it to a desired state and then “submits” an answer to the assessment engine by pressing a button within the simulation or within PAE.  This passes one or more simulation variables back to the assessment engine for marking.  This effectively allows the full range of interactive input objects in a simulation to be available as an answer mechanism.  The simulation might allow students to draw a graph by inputting lines or individual points, input a shape etc.  This takes the questioning process beyond simple multiple-choice but without the need for programming.  Whenever generic requirements for a new input mechanism is established, a simple simulation can be created and a template produced.

Example: In this example the student uses the simulation to draw the appropriate straight line for the expression y=mx+c where m and c are randomised.  This example can be explored online.

Fig 5 shows an example of a simulation as an answering mechanism.. The question reads: Construct a line with the equation: y=2x+9. Either drag the red circles or type values into the table. Then press the submit button. The graph has a red line with 2 red circles at each end; moving these circles allows user to construct the line to answer the question OR x, y values for each point can be manually entered into the input table for each of the 2 "circled" ends of the line.

Figure 5: An example of using a simulation as an answering mechanism

4.3.  Using the simulation to allow exploration and provide feedback

This type of usage takes advantage of the main educational benefits of simulations.  It is often useful to allow students to explore and test their own conceptions in a formative mode.  Intrinsic feedback from the learner’s actions is provided by changes in simulation state.  It is also possible to update the simulation, after the learner has submitted an answer, to provide feedback as to the correctness of that answer.

In this section use of simulations for assessing the outcomes of exploration in two main categories is considered (other categories have been explored elsewhere [15]):

For example, asking the student to draw a graph showing the relationship between 2 variables (for more details see the answer mechanism, section 4.2)

In other words, if condition A is changed what will the outcome be?

Example: Figure 6 shows a simulation to display and explore the effects of the coefficients a, b, c and d in the expression a cos(bx + c) + d.  This simulation could be used in a number of ways: to allow exploration of the effects of modifying the parameters; as a randomised graphic; or as a mechanism to provide feedback on the expression a student believes to be correct.  Similarly the cubic example mentioned previously could be used in any of these ways (see Figure 4). 

Fig 6 shows a simulation for explorative and/or feedback use. The polynomial from Fig 4 P(x)=d(x-a)(x-b)(x-c) is used but this screen allows the user to enter a, b, c and d and see the effect on the graph displayed.

Figure 6: An example of a simulations for explorative and/or feedback use

4.4.  Using the simulation in more complex questions

Simulations can be used to provide problems that encourage students to use mathematical skills in their solution.  In the PAE-JeLSIM system, students can obtain the solution by using the simulation that can be passed to the assessment engine for marking.  It is also possible for the assessment engine to collect the state of simulation variables at stages throughout a complex task.

5. Future

In this paper some simple examples of the use and reuse of simulations in assessment in the mathematical domain have been demonstrated.  Reusability of simulations is crucial if the use of simulations is to become widespread.  Reusability is fostered by:

However, it is not just the potential use of simulations, or the technical communication mechanisms that need to be explored if simulation use is to become more viable.  The use of simulations, and the type and format of information that needs to be communicated must be brought into discussions and work on assessment specifications (IMS QTI [16]), student profiles and assessment reporting [11].

Further Information

More information about:

Further examples of simulations and assessments (including accompanying commentary) can be found, and explored, at http://www.calm.hw.ac.uk/sims-asses.html. 

References

[1] Alessi, S. M., Trollip, S. R. (2001). Multimedia for Learning: methods and development. (3rd Edition) Allyn and Bacon

[2] Laurillard, D. (1993). Rethinking University Education: a framework for effective use of educational technology, Routledge

[3] Thomas, R., Neilson, I. (1995). Harnessing Simulations in the Service of Education: The Interact Simulation Environment. Computers and Education 25(1/2): 25-29

[4] White, R., Gunstone, R. (1992). Probing Understanding, The Falmer Press, London

[5] Jimoyiannis, A., Komis, V. (2001). Computer simulations in physics teaching and learning: a case study on students' understanding of trajectory motion. Computers and education 36: 183-204

[6] de Jong, T., van Joolingen, W. R. (1998). Scientific discovery learning with computer simulations of conceptual domains. Review of Educational Research 68(2): 179-201

[7] H S Ashton, C E Beevers, A A Korabinski, M A Youngson, PASS-IT ON, LTSN MSOR Maths CAA series, March 2005, http://mathstore.ac.uk/articles/maths-caa-series/mar2005/

[8] PASS-IT Website: http://www.pass-it.org.uk

[9] CALM (Computer Aided Learning in Mathematics) group webpage for PASS-IT: http://www.calm.hw.ac.uk/pass-it.html

[10] J S Paterson, The CUE Assessment System, MSOR Connections newsletter vol 2 no 2 May 2002, Birmingham.  http://ltsn.mathstore.ac.uk/articles/maths-caa-series/apr2002/index.shtml

[11] Ashton, H S and Schofield, D K, Effective reporting for online assessment - shedding light on student behaviour, http://mathstore.ac.uk/articles/maths-caa-series/feb2005/

[12] CALM (Computer Aided Learning in Mathematics) group publications webpage: http://www.calm.hw.ac.uk/publications.html

[13] H S Ashton, C E Beevers and M A Youngson, Incorporating partial credit in computer-aided assessment of Mathematics in secondary education, To be published, British Journal of Educational Technology

[14] Beevers, C.E., Youngson, M.A., McGuire, G.R., Wild, D.G. and Fiddes, D.J. (1999) Issues of Partial Credit in Mathematical Assessment by Computer, ALT-J 7, 26-32

[15] R Thomas, H Ashton, B Austin, C Beevers, D Edwards, C Milligan, Cost Effective use of Simulations in Online Assessment, 9th International CAA Conference, July 2005

[16] IMS Question and Test Interoperability (QTI) Specifications webpage: http://www.imsglobal.org/question/index.html