Hamadan University of Medical Sciences Avicenna Journal of Dental Research 2423-7582 16 1 2024 04 01 Development of an Instrument for the Online Evaluation of Orthodontic-Treated Cases: A Pilot Study 9 15 10.34172/ajdr.1595 EN Amirfarhang Miresmaeili https://orcid.org/0000-0002-1796-7512 Hana Salehisaheb https://orcid.org/0000-0002-7387-1371 Journal Article 10.34172/ajdr.1595 2022 02 26 2022 05 02 Background: It is necessary to create a sound method for the optimal evaluation of treated cases of orthodontic board exam candidates. This cross-sectional study aimed to design and validate a new method for the national orthodontic board exam using a web-based program. Methods: Complete documents of 10 patients randomly selected from a pool of the previously presented cases at the national board examination were entered into a web-based program called "the Orthoboard". The documents were arranged according to the European Board of Orthodontist Standards, and 15 related questions were asked based on the index of complexity, outcome, and need index and the American Board of Orthodontics evaluation standards. A customized grading system was used for the finalized questionnaire. They were asked to be evaluated by 10 orthodontists (5 with less and 5 with more than 10 years of experience). The content validity of the questionnaire was analyzed by the content validity index and content validity ratio. The reliability of this questionnaire was measured using the Kappa statistical test. Results: Most evaluators do not consider it necessary to ask questions about the diagnostic resume, problem list, treatment plan, and treatment resume. Crowding after treatment and cross-biting before and after treatment had the best interexaminer reliability. The least agreement among examiners was between the pre-treatment aesthetic index and the pre-treatment buccal segment relationship. Conclusion: Inter-examiner reliability was lower than expected, indicating that the orthodontic board test scoring is too subjective. The addition of a 3-dimensional cast is recommended for better objective evaluation.