Applied AI Summit Healthcare
Free online conference | April 14-15, 2026Testing 1, 2, 3: Using the GenAI Lab to Advance Assessment in Medical Education
This session explores how NBME leverages the GenAI Lab to enhance medical education through innovative data generation. We use generative AI to create high-quality training data for short-answer questions to evaluate response correctness and Named Entity Recognition (NER) to extract key concepts from a formative communication assessment. These techniques enable scalable item development while maintaining rigor and validity. To ensure content quality and alignment with best practices, we incorporate structured feedback from external subject matter experts (SMEs) during item review. This accelerates item development and supports more authentic, competency-based assessments. Attendees will gain insights into practical applications of John Snow Labs’ Generative AI Lab for assessment design and learn strategies for integrating human expertise to advance medical education.
Key Takeaways:
- Learn how GenAI workflows can streamline the creation training data using classification and NER.
- Discover best practices for incorporating SME feedback to ensure quality and educational relevance.
About the speaker
Janet Mee
Measurement Scientist at NBME
Janet Mee, Ph.D. is a Measurement Scientist II in NBME’s Office of Research Strategy, where she advances innovative assessment and research capabilities—especially the design and evaluation of technology enabled measurement and automated scoring. Her work spans cross functional initiatives in Growth & Innovation, with a focus on annotation pipelines, NLP supported scoring, and evidence based process improvement. Her scholarly contributions include co authoring a chapter in the Handbook of Automated Scoring: Theory into Practice, which surveys foundational theory and practical applications of automated scoring across education and assessment. Janet regularly shares work through invited and internal talks. Recent highlights include NBME’s Latest in Assessment Research & Innovation session on the Workplace Based Assessment Creative Community (with John Moore, Monica Cuddy, and Stephanie Iaccarino), an ORS Research Recap presentation on comparative item formats, and an IAMSE workshop on best practices for reviewing workplace based assessment instruments. She also serves the scholarly community as an invited peer reviewer for journals including NPJ Digital Medicine, Scientific Reports, and Discover Education, reflecting her expertise at the intersection of measurement and AI in medical education. Beyond her research portfolio, Janet contributes to community and culture efforts at NBME (e.g., SWAN: Supporting Women at NBME), helping facilitate inclusive conversations and collaborative learning across teams.