Comparing AI-generated and human-written A3/A4 medical licensing questions
Scientific Validity Assessment and Optimization of AI-Generated A3/A4 Type Questions for the Chinese Medical Licensing Examination: An Empirical Analysis Based on the "Answering-Generating" Closed Loop and Transformation of Competency Assessment
The project will test whether AI-created A3/A4‑type questions perform like human-written ones for medical students in standardized residency training preparing for the Chinese Medical Licensing Examination.
Quick facts
| Study type | Observational |
|---|---|
| Enrollment | 20 (estimated) |
| Ages | 18 Years to 60 Years |
| Sex | All |
| Sponsor | Guangdong Provincial People's Hospital Academic / other |
| Locations | 1 site (Guangzhou, Guangdong) |
| Trial ID | NCT07505862 on ClinicalTrials.gov |
What this trial studies
This cross-sectional, two-stage project combines quantitative analysis with qualitative review to compare AI model outputs to human-authored A3/A4 exam items. Phase I performs a randomized model performance comparison, and Phase II conducts an item quality evaluation in which AI-generated and human-written items are interspersed for blinded testing. Single-blinding, randomization, and standardized procedures are used to reduce bias and improve reproducibility. Outcomes include item performance metrics, participant scoring, and qualitative feedback to guide optimization of AI-generated content.
Who should consider this trial
Good fit: Ideal participants are medical students enrolled in a Standardized Residency Training program who hold a Bachelor of Medicine degree or higher and can use digital platforms to complete assessments.
Not a fit: Those involved in training the AI or creating the human question bank, or individuals unable to complete digital assessments due to impairments or severe illness, are unlikely to benefit from participation.
Why it matters
Potential benefit: If successful, AI could help produce high-quality licensing exam items faster, easing test development and expanding practice resources for trainees.
How similar studies have performed: Similar work on AI-generated exam questions has shown mixed results with some comparable performance but remaining quality gaps, so the approach is partially validated but still evolving.
Eligibility criteria
Show full inclusion / exclusion criteria
Inclusion Criteria: * 1.Professional Status: Medical students currently enrolled in a Standardized Residency Training (SRT) program. 2.Educational Background: Holders of a Bachelor of Medicine degree or higher, with foundational clinical knowledge. 3.Informed Consent: Voluntarily participate in the study and provide written informed consent. 4.Technical Competency: Proficient in using digital platforms to complete assessments and scoring. Exclusion Criteria: * 1.Conflict of Interest: Individuals involved in the AI model training, prompt engineering, or the creation of the human-authored question bank for this study. 2.Inability to Complete: Presence of visual/auditory impairments or severe illness that precludes completion of the assessment within the specified time. 3.Investigator's Discretion: Any other condition that, in the opinion of the investigator, renders the participant unsuitable for the study.
Where this trial is running
Guangzhou, Guangdong
- Guangdong Provincial People's Hospital — Guangzhou, Guangdong, China (Recruiting)
Study contacts
- Study coordinator: Zhuoyi Chen MD
- Email: 18737552662@163.com
- Phone: +86 18737552662
How to participate
- Review the eligibility criteria above with your treating physician.
- Visit the official trial page on ClinicalTrials.gov for the most current contact information and recruitment status.
- Contact the listed study coordinator or principal investigator to request pre-screening. Pre-screening is free and never obligates you to enroll.