A simulator can reproduce every button on the console, every screen and every sound of the equipment and still teach very little. Another, visually simple, can prepare someone better for the first time in front of a patient. The difference lies in how closely the decisions the student makes in the simulator match the ones they would make on the equipment.

Before putting a simulator in a classroom, a teaching institution needs to know what it trains, how it measures what the student did and on what basis it claims that this result means anything.

What research says about fidelity

In health professions education, the fidelity of a simulator is usually understood as how closely it looks like the real situation, behaves like it and conveys the same feel. The definition is useful for design, but it values physical resemblance above what produces learning.

In 2014, a group of researchers who had just completed a systematic review of simulation in health care published a proposal in Academic Medicine to abandon the term. According to them, several studies show that the degree of fidelity appears to be independent of educational effectiveness. The proposal was to separate two concepts that the word blended together: physical resemblance, how closely the simulator looks like the equipment, and functional task alignment, how closely what the student does in the simulator corresponds to what they will need to do in practice.

A review published two years earlier in Medical Education already pointed in the same direction. It brought together 24 studies that compared high- and low-fidelity simulators using measures of clinical performance, in areas such as auscultation, surgical technique and cardiac resuscitation. Both types produced consistent gains compared with groups that received no training. But almost no study found a significant advantage for the more sophisticated simulator, and the mean differences were between 1% and 2%.

Appearance helps, but alignment with the task needs to be demonstrated before it.

The functional question: is the decision the same?

In the operation of imaging equipment, functional task alignment can be translated into a direct question: is the decision the operator makes in the simulator the same one they would make on the equipment, with the same information available and the same consequences?

This question changes how one assesses what is missing and what is superfluous. A feature can be absent without harm when the decision it represents does not exist in the simulated workflow. And it can be present without teaching anything when changing it does not change the result.

Answering this question requires the simulator to declare, item by item, what is calculated, what is modeled with simplifications and what is only represented to complete the workflow. That declaration was the subject of an earlier article. Without it, the instructor has no way of knowing what can be expected of the student.

What the evidence says makes simulation work

In 2005, a systematic review published in Medical Teacher analyzed 109 comparative studies of simulation in medical education to identify what distinguished effective uses. The most frequently cited features were:

  • feedback, identified by 47% of the studies as the most important feature;
  • repetitive practice, cited by 39%;
  • curriculum integration, rather than occasional use;
  • range of difficulty across exercises;
  • variety of clinical situations, rather than a narrow repertoire;
  • a controlled environment, where the student can make a mistake, notice it and correct it without consequences for anyone.

The authors also noted that the methodological quality of the available research was generally weak. Six years later, a meta-analysis published in JAMA brought together 609 studies with more than 35,000 participants and found large effects of simulation on knowledge, skills and behaviors, and moderate effects on patient-related outcomes. The comparison, however, was against no intervention. That kind of comparison shows that training with simulation is better than not training, but it does not say which simulator is better, or why.

Making mistakes for the right cause

An environment where the student can make mistakes is only useful if the mistake appears for the right reason. When a poorly chosen parameter produces an artifact, the artifact has to come from the physics that explains it, not from a ready-made image attached to an “error” label. Only then does the student learn to recognize the cause and correct it, instead of memorizing what the problem looks like.

Useful feedback shows what happened, what would have been appropriate and the physical reason for the difference, based on the record of the decisions the student made during the exercise. That record is also what allows the instructor to review the session afterwards, instead of relying only on the final score.

There is also a distinction that assessment must respect: not every difficulty is the operator’s mistake. A patient who moves, who cannot hold their breath or who interrupts the exam creates problems that the operator has to manage, and the assessment must separate what was well managed from what the operator caused.

What to measure, and with what weight

Every score is a claim about someone’s performance, and that claim depends on choices that need to be explicit.

The first is what goes into the calculation. In the operation of imaging equipment, there are criteria for patient safety, image quality and efficiency, and they do not carry the same weight. Forgetting to ask about an implant is not comparable to taking a few extra minutes to complete the exam.

The second is whether one criterion can compensate for another. In a compensatory model, excellent performance in one dimension can make up for a failure in another. In a conjunctive model, each critical dimension must reach a minimum on its own. The assessment literature has compared the two models for decades. In training people to operate equipment that can injure patients, it makes sense for safety items to follow the conjunctive model: speed and a good image should not compensate for a screening failure.

The third choice is where the thresholds are set, that is, the point at which a performance is sufficient. The thresholds are an educational policy decision: the literature justifies using the conjunctive model, but the values need to be defined and reviewed by those who teach and those who practice.

Validity depends on how the score is used

Saying that a simulator “is validated” without explaining for what purpose says very little. Contemporary assessment research treats validity as a hypothesis about how a result is used: if the student received a given score, is it fair to draw a given conclusion about them and make a given decision?

A guide published in 2016 in the journal Advances in Simulation describes how to build this validity argument for simulation-based assessments. First, you define what you want to measure and what decision will be made with the result. Then you identify the weakest assumptions behind that conclusion and look for evidence for each one. The guide draws on a model that organizes this evidence into five sources:

  • content: are the tasks and criteria the ones that matter in practice?
  • response process: does the student reason in the simulator as they would in front of the equipment, or do they learn to exploit the software’s logic?
  • internal structure: do the criteria coherently measure what they claim to measure?
  • relationships with other variables: do experienced professionals perform better than beginners? Does performance in the simulator track supervised performance in the exam room?
  • consequences: do the decisions made on the basis of the score, such as clearing someone for supervised practice, turn out to be appropriate?

The role of experts

Part of this evidence can only come from people who practice. Experienced technicians and technologists, faculty members, and physicians who oversee exams know which mistakes are frequent, which are serious and which situations come up day to day without appearing in textbooks. When these people review the cases, the assessment criteria and the acceptance thresholds, other institutions gain a basis for adopting them.

This work follows established methods: expert panels that rate each criterion in successive rounds until they reach consensus, comparisons between the performance of experienced professionals and that of students, and pilot studies that follow a class before and after use. Until this stage is complete, the most honest thing is to say that the criteria have not yet been validated, and who will review them.

Questions worth asking of any simulator

An institution evaluating a simulator for teaching can put these questions directly to whoever offers it:

  • Which operator decisions does it reproduce, and which does it leave out?
  • What is calculated, what is simplified and what is only represented?
  • When the student makes a mistake, does the mistake appear because of its physical cause or because of an image attached to a label?
  • Does the feedback explain the cause of the mistake, or does it only report that a mistake occurred?
  • What is measured, with what weight, and can safety failures be offset by other criteria?
  • Who defined the acceptance thresholds, and on what basis?
  • What validity evidence exists, and what is still pending?

References

#Simulation#Training#Assessment#Research