The Emergency Medicine Oral Examination is a controversial topic: I want to start a dialogue on this particular topic. I have studied this issue for decades. Some will disagree with me because they will say I have a financial incentive to see this process continue. I’m going to start with a basic position…my commentary below. I’m hoping every proponent or opponent physician will join in and comment on this issue. I’m hoping if you have been a candidate or examiner you will join in. Here we go—in simple terms: I stand with ABEM…let me explain:

In the ever-evolving landscape of emergency medicine, the need for rigorous and reliable assessment tools to gauge the competence of our healthcare professionals remains paramount. The American Board of Emergency Medicine (ABEM) Oral Certification Examination stands as a testament to this commitment, serving as a comprehensive evaluation of emergency medicine knowledge, skills, and judgment. While artificial intelligence (AI) has made remarkable strides in recent years, its capabilities still fall short of replicating the intricate cognitive abilities, empathy, and adaptability demanded by the Oral Exam, highlighting the irreplaceable role of human examiners in this critical process.

The ABEM Oral Certification Examination: A Multifaceted Assessment

The ABEM Oral Certification Examination employs a meticulously designed scoring system that delves into eight key performance ratings: Data Acquisition, Problem-Solving, Patient Management, Resource Utilization, Healthcare Provided, Interpersonal Relations and Communication Skills, Comprehension of Pathophysiology, and Professionalism (overall clinical competence). These ratings are further evaluated based on critical actions, inappropriate actions, and even dangerous actions—specific behaviors that distinguish acceptable or unacceptable performance.

This comprehensive evaluation extends beyond the mere assessment of medical knowledge and technical skills. It encompasses the ability to gather information effectively, formulate sound clinical judgments under pressure, communicate clearly and empathetically with patients and colleagues, and adapt to the dynamic and unpredictable nature of emergency medicine settings.

The Limitations of AI in Assessing Human Expertise

While AI can assess the presence or absence of specific actions, it cannot adequately interpret the nuances of candidate language skills, bedside manner, adaptability, and their impact on the overall assessment. Human examiners, on the other hand, are adept at discerning subtle variations in communication, recognizing when a candidate’s statements or responses may indicate underlying gaps in knowledge, reasoning, or empathy.

Human examiners can also assess a candidate’s ability to adapt to dynamic and unpredictable emergency medicine settings, where split-second decisions often hinge on incomplete or ambiguous information. AI, with its current limitations in understanding real-world context and adapting to unpredictable scenarios, cannot adequately assess this crucial aspect of emergency medicine practice.

The Human Touch: Simulating Real-World Patient Encounters

Furthermore, human examiners can interact with candidates in a way that simulates real-world patient encounters, providing specific data as simulated patients, nurses, paramedics, consultants, or admitting physicians. This allows examiners to assess a candidate’s ability to communicate effectively with patients, families, and healthcare colleagues, as well as their ability to gather information, formulate diagnoses, and make sound treatment plans under pressure.

Addressing the Intimidation Factor

The perception that the ABEM Oral Certification Examination is intimidating often stems from self-imposed anxiety rather than the examiner’s demeanor. ABEM examiners undergo rigorous training and are carefully instructed to maintain a fair, objective, and empathetic approach throughout the evaluation process. Their interactions with candidates are designed to elicit the necessary information to assess their competence, not to create an intimidating atmosphere.

In fact, a simulated human, an AI, could easily be just as intimidating to candidates if not more so. Remember, perception is reality. A polygraph test, for example, often elicits physiological responses not because of the human administering the questions but because of the examinee’s perception of the machine’s intrusive ability to detect truth vs. lies. Yet, a polygraph machine doesn’t interact with speech or make scenario adjustments. It is silent. Furthermore, scientific and governmental assessments of polygraphy generally indicate that it is highly inaccurate, susceptible to deception-mitigating countermeasures, and an unreliable means of truthfulness evaluation. Now ABEM is not primarily assessing for deception, although I have experienced some of this in oral examination simulations: physicians sometimes talk just to talk when they lack specific knowledge, e.g. as a response to a pathophysiology question. A human examiner can definitely perceive this and it is why I teach candidates to simply be honest. This would be yet another challenge for an AI.

Empathy and Understanding

Examiners are trained to recognize signs of stress and anxiety in candidates. They can tell when a candidate is having difficulty thinking clearly or expressing themselves effectively. In these cases, examiners will often give the candidate a moment to collect their thoughts or provide them with additional support.

Recognizing and Addressing Unnecessary Self-Deprecation

Examiners can also recognize when a candidate is being unnecessarily self-deprecating. This can sometimes be a sign of anxiety or low self-esteem. In these cases, examiners do try to redirect the candidate’s focus to allow them to maximize their score.

The Importance of Human Judgment

The ability to recognize and respond to these human factors is one of the reasons why human examiners are still essential for the ABEM Oral Certification Examination. AI is not yet able to replicate this level of empathy and understanding.

Safeguarding Fairness and Transparency

Moreover, ABEM employs verifying physicians to monitor the examination process, ensuring that examiners adhere to established guidelines and maintain a professional and respectful environment. This additional layer of oversight further highlights ABEM’s commitment to fairness, transparency, and a focus on patient safety.

Furthermore, examiners disqualify themselves from testing any candidate they know in advance, eliminating any potential for bias or conflict of interest.

The Future of AI in Medical Assessment

The question of how AI would be implemented in the Oral Exam process raises significant concerns about its effectiveness and appropriateness. A machine that spits out images of rhythm strips and radiographs would currently fail to assess the candidate’s ability to interpret and communicate their findings effectively with patients and colleagues. Even if AI were given a human appearance, it would lack the ability to empathize with candidates, understand their concerns and objectives, and make sound clinical assessment decisions based on incomplete or ambiguous information.

In conclusion, the ABEM Oral Certification Examination serves as an indispensable tool for ensuring the competence of emergency physicians, safeguarding patient well-being. While AI holds immense promise for the future, its limitations in replicating human cognitive abilities, empathy, adaptability, and communication skills render it unsuitable for administering the Oral Exam in 2023. Until AI can bridge the chasm between data processing and human expertise as a physician examiner, the irreplaceable role of human examiners in this critical assessment will remain paramount. Now I’m not saying that AI could not eventually simulate a physician as an examiner…just not today.

I hope to hear from the emergency medicine community. Pro or con it doesn’t matter. I’m interested in hearing from everybody. Be well.