MDPI Acta Medica Okayama 2072-6694 17 1 2024 The Three-Class Annotation Method Improves the AI Detection of Early-Stage Osteosarcoma on Plain Radiographs: A Novel Approach for Rare Cancer Diagnosis 29 EN Joe Hasei Department of Medical Information and Assistive Technology Development, Graduate School of Medicine, Dentistry and Pharmaceutical Sciences, Okayama University Ryuichi Nakahara Science of Functional Recovery and Reconstruction, Graduate School of Medicine, Dentistry and Pharmaceutical Sciences, Okayama University Yujiro Otsuka Department of Radiology, Juntendo University School of Medicine Yusuke Nakamura Plusman LCC Kunihiro Ikuta Department of Orthopedic Surgery, Graduate School of Medicine, Nagoya University Shuhei Osaki Department of Musculoskeletal Oncology and Rehabilitation, National Cancer Center Hospital Tamiya Hironari Department of Musculoskeletal Oncology Service, Osaka International Cancer Institute Shinji Miwa Department of Orthopedic Surgery, Kanazawa University Graduate School of Medical Sciences Shusa Ohshika Department of Orthopaedic Surgery, Hirosaki University Graduate School of Medicine Shunji Nishimura Department of Orthopaedic Surgery, Kindai University Hospital Naoaki Kahara Department of Orthopedic Surgery, Mizushima Central Hospital Aki Yoshida Science of Functional Recovery and Reconstruction, Graduate School of Medicine, Dentistry and Pharmaceutical Sciences, Okayama University Tomohiro Fujiwara Science of Functional Recovery and Reconstruction, Graduate School of Medicine, Dentistry and Pharmaceutical Sciences, Okayama University Eiji Nakata Science of Functional Recovery and Reconstruction, Graduate School of Medicine, Dentistry and Pharmaceutical Sciences, Okayama University Toshiyuki Kunisada Science of Functional Recovery and Reconstruction, Graduate School of Medicine, Dentistry and Pharmaceutical Sciences, Okayama University Toshifumi Ozaki Science of Functional Recovery and Reconstruction, Graduate School of Medicine, Dentistry and Pharmaceutical Sciences, Okayama University Background/Objectives: Developing high-performance artificial intelligence (AI) models for rare diseases is challenging owing to limited data availability. This study aimed to evaluate whether a novel three-class annotation method for preparing training data could enhance AI model performance in detecting osteosarcoma on plain radiographs compared to conventional single-class annotation. Methods: We developed two annotation methods for the same dataset of 468 osteosarcoma X-rays and 378 normal radiographs: a conventional single-class annotation (1C model) and a novel three-class annotation method (3C model) that separately labeled intramedullary, cortical, and extramedullary tumor components. Both models used identical U-Net-based architectures, differing only in their annotation approaches. Performance was evaluated using an independent validation dataset. Results: Although both models achieved high diagnostic accuracy (AUC: 0.99 vs. 0.98), the 3C model demonstrated superior operational characteristics. At a standardized cutoff value of 0.2, the 3C model maintained balanced performance (sensitivity: 93.28%, specificity: 92.21%), whereas the 1C model showed compromised specificity (83.58%) despite high sensitivity (98.88%). Notably, at the 25th percentile threshold, both models showed identical false-negative rates despite significantly different cutoff values (3C: 0.661 vs. 1C: 0.985), indicating the ability of the 3C model to maintain diagnostic accuracy at substantially lower thresholds. Conclusions: This study demonstrated that anatomically informed three-class annotation can enhance AI model performance for rare disease detection without requiring additional training data. The improved stability at lower thresholds suggests that thoughtful annotation strategies can optimize the AI model training, particularly in contexts where training data are limited. No potential conflict of interest relevant to this article was reported.