An anesthesiology resident struggles on multiple-choice, clinical-vignette-based knowledge tests. He has performed well in the OR but tends to call his attending frequently to confirm management changes and has had consistently low scores on multiple In-Training Exams. To evaluate his test taking skills, the program director uses a think-aloud exercise by having the resident complete the following practice test question while the resident reads and thinks aloud.
A 58 year-old man with prior medical history of hypertension, well controlled with Lisinopril, and a strong family history of myocardial infarction, presents to the preoperative clinic for a planned knee arthroscopy after a fall while playing basketball. His exercise tolerance is greater than 4 METs and his review of systems is otherwise negative. What is the next best step?
- A.) Obtain 12-lead ECG
- B.) Start perioperative beta blockade on the day of surgery.
- C.) Start perioperative beta blockade two weeks before surgery.
- D.) No further action is required.
The resident predicts that D is the correct answer, but then rereads the answer choices and decides to choose A after going back and forth on the decision.
What type of test-taking deficiency does this resident most likely display?
- A.) Lack of script recognition
- B.) Lack of script specificity
- C.) Premature closure
- D.) Under-confidence
Andrews, Kelly and DeZee (2018) have described use of Self Regulated Learning Microanalysis to identify and remediate specific test taking deficiencies. Self regulated learning has been defined as a cyclical process of self generated thoughts and actions that a learner plans and adapts to achieve a personal goal. (Zimmerman, 2000) This process involves forethought, performance and self-reflection. The method of microanalysis uses questions posed to the learner during an educational task to elucidate subprocesses of self regulated learning. This semi-structured, think-aloud protocol has been used in multiple educational arenas including studying, procedural skills and clinical reasoning. (DiBenedetto and Zimmerman, 2010; Cleary and Sandars, 2011; Artino, et.al., 2014) Andrews, Kelly and DeZee describe the process to include:
- Description of a learner’s prior history on multiple-choice, clinical-vignette-based medical knowledge exams.
- Read and think aloud exercise while the learner completes a sample question.
- Repeat of think aloud exercise, but interjecting questions throughout:
- After just reading the clinical stem, questions are aimed at assessing skill and specificity of the use of disease scripts and confidence in identification of a diagnosis.
- Further questions after revelation of the test question are aimed at prediction of the answer and confidence in that answer.
- Revealing the answer choices, questions are aimed at asking further confidence in the answer and if the learner would change his answer.
- After revealing the correct answer, questions ask why right/why wrong of the answers and how to plan to improve.
- The process is repeated with new test questions until enough evidence is accumulated to identify a test-taking problem.
Test taking deficiencies include:
- Lack of script recognition. This learner struggles to identify the diagnosis presented in the clinical stem. He will read and reread without prioritizing and interpreting the information given in terms of the most likely script. These learners need to remediate by sorting clinical information based upon a script and change scripts as needed to accommodate new information. Clinical exposure and practice questions to study disease in the context of the clinical presentation can help.
- Lack of script specificity. This learner recognizes the general disease script but not the severity or specific subtype in the clinical stem. He can often narrow down the answers to two choices but then guesses. These learners need to refine their disease scripts through increased clinical exposure and further knowledge of disease severity/subtype.
- Premature closure. This learner makes an early decision on the diagnosis and downplays incongruent information given later in the clinical stem. These learners need to slow down after reading the stem and ensure diagnosis is consistent with all findings listed.
- Under-confidence. This is the learner who “over-thinks” the question. The learner knows the correct answer, but talks himself out of it when he sees the answer choices. He will score his confidence of predictions and final answers lower in the microanalytic questions. This learner needs to remediate by re-calibrating his own confidence. Leggett, Sandars and Burns (2012) have described a self-monitoring exercise that uses comparison of prediction and correct choices during practice questions that can help these learners.
- Incorrect causal attribution. This learner cannot explain why he got the answer right or wrong. This learner often completes many practice questions, but continues to struggle on exams. To remediate, this learner should cut back on the number of questions per session and allow for in-depth review of the reasoning for correct and incorrect answers.
- Inappropriate adaptive inferences. This learner can identify knowledge gaps but cannot generate a learning plan. Often he will answer, “I need to read more.” This learner needs prompts for a specific study plan to interact with the content, organize the content in a way that makes sense for the learner, and elaborate upon the content to facilitate memory encoding. (Cornell note-taking is a methodology that incorporates all of these).
- Isolated knowledge deficit. This learner understands the script in detail, can explain why the answer is right or wrong and knows how to develop a study plan, but hasn’t spent the time to learn the material. This learner’s remediation often includes time management skills, but the 7 D’s of poor performance should be ruled out: Depression, Distraction, Deprivation, Drugs, Disease, personality Disorder, and learning Disability. (Lucey and Boote, 2008)
For the anesthesiology resident described in the question, he displays typical under-confidence in just the initial read and think aloud exercise. Often the under-confident learner has a history of multiple failures and has consequently learned to distrust his reasoning, even if correct. He will overthink a question and change answers.
- Andrews, M. A., Kelly, W. F., & DeZee, K. J. (2018). Why Does This Learner Perform Poorly on Tests? Using Self-Regulated Learning Theory to Diagnose the Problem and Implement Solutions. Academic Medicine, 93(4), 612-615.
- Artino, A. R., Cleary, T. J., Dong, T., Hemmer, P. A., & Durning, S. J. (2014). Exploring clinical reasoning in novices: a self-regulated learning microanalytic assessment approach. Medical Education, 48(3), 280-291.
- Cleary, T. J., Sandars, J. (2011). Assessing self-regulatory processes during clinical skill performance: A pilot study. Medical Teacher, 33, e368–e374.
- DiBenedetto, M. K., & Zimmerman, B. J. (2010). Differences in self-regulatory processes among students studying science: A microanalytic investigation. The International Journal of Education and Psychological Assessment, 5, 2-24.
- Leggett, H., Sandars, J., & Burns, P. (2012). Helping students to improve their academic performance: A pilot study of a workbook with self-monitoring exercises. Medical Teacher, 34(9), 751-753.
- Lucey, C. R., & Boote, R. M. Working with Problem Residents: A Systematic Approach. In: Holmboe, E. S., & Hawkins, R. E. eds. (2008). Working with Problem Residents: A Systematic Approach. In Practical Guide to the Evaluation of Clinical Competence. Philadelphia, PA: Mosby, Inc.
- Zimmerman, B. J. Attaining self-regulation: A social-cognitive perspective. In: Boekaerts, M., Pintrich, P., & Zeidner, M., eds. (2000). Handbook of self-regulation. Orlando, PA: Academic Press,
Carol Ann B. Diachun, MD, MSEd
Clinical Professor of Anesthesiology
Associate Chair for Education and Residency Program Director
University of Florida College of Medicine – Jacksonville
Dr. Carol Diachun is Clinical Professor of Anesthesiology at the University of Florida College of Medicine - Jacksonville. She completed a BS in Health Sciences at SUNY-Buffalo in 1990, medical school at Stanford University School of Medicine in 1994, internal medicine internship in San Jose, CA and anesthesia residency at the University of Pennsylvania and Stanford University. She was a Clinical Instructor at Stanford University until 2000, when she joined the University of Rochester anesthesiology department. After 14 years in Rochester, NY, she was recruited to Jacksonville to become the Associate Chair for Education and Anesthesia Residency Program Director in 2014. Her main clinical interests have been in trauma, vascular and thoracic anesthesia. She has served as chief of both the divisions of vascular anesthesia and thoracic anesthesia during her tenure in Rochester. Since coming to Florida, she has mainly concentrated in trauma and obstetrical anesthesia. But foremost, Dr. Diachun is an educator. She served as residency curriculum director, associate residency program director, and chair of the Clinical Competency Committee at the University of Rochester. After moving to Florida, she has served as residency program director and faculty development director at UFCOM-J. She has pursued additional education training as the Lowell A. Glasgow Dean’s Teaching Fellow in 2009-2011 and completed a Master of Science in Health Professions Education from the University of Rochester Warner School of Education in October 2014. Dr. Diachun has been involved in anesthesia education at the national level giving numerous presentations at the Society for Education in Anesthesia (SEA) and American Society of Anesthesiology. She has been Chair and Chair-Designee of the SEA Educational Meetings Committee, and has been program director for three national meetings for the SEA. Dr. Diachun presently serves on the Board of SEA, has served as an invited education study section grant reviewer for the Foundation for Anesthesia Education and Research (FAER) and is on the advisory board for Anesthesia Toolbox, a collaborative effort by ~60 anesthesia programs to design and implement an international-level anesthesia residency curriculum.