Mixed reality infrastructure based on deep learning medical image segmentation and 3D visualization for bone tumors using DCU-Net

Abstract

• Automatic segmentation and 3D reconstruction of osteosarcoma images are achieved by optimizing feature extraction and improving target space clustering capabilities. • A variety of indicators such as similarity (DSC), recall (R), precision (P) and three-dimensional vertex distance error (VDE) are combined to evaluate segmentation performance and 3D reconstruction effect. • The DCU-Net operation model proposed in this study has significant advantages compared with the segmentation and reconstruction results of models such as U-Net and Attention-Uet. • A mixed-reality three-dimensional visualization infrastructure was constructed, and preliminary testing shows that the facility enhances clinicians’ understanding of tumor morphology and spatial relationships, which in turn is expected to facilitate bone tumor clinical practice and improve outcomes. Segmenting and reconstructing 3D models of bone tumors from 2D image data is of great significance for assisting disease diagnosis and treatment. However, due to the low distinguishability of tumors and surrounding tissues in images, existing methods lack accuracy and stability. This study proposes a U-Net model based on double dimensionality reduction and channel attention gating mechanism, namely the DCU-Net model for oncological image segmentation. After realizing automatic segmentation and 3D reconstruction of osteosarcoma by optimizing feature extraction and improving target space clustering capabilities, we built a mixed reality (MR) infrastructure and explored the application prospects of the infrastructure combining deep learning-based medical image segmentation and mixed reality in the diagnosis and treatment of bone tumors. We conducted experiments using a publicly available dataset for bone tumor segmentation, used the optimized DCU-Net and 3D reconstruction technology to generate bone tumor models, and used set similarity (DSC), recall (R), precision (P), and 3D vertex distance error (VDE) to evaluate segmentation performance and 3D reconstruction effects. Then, two surgeons conducted clinical examination experiments on patients using two different methods, viewing 2D images and virtual reality infrastructure, and used the Likert scale (LS) to compare the effectiveness of surgical plans of the two methods. The DSC, R and P values of the model introduced in this paper all exceed 90%, which has significant advantages compared with methods such as U-Net and Attention-Uet. Furthermore, LS showed that clinicians in the DCU-Net-based MR group had better spatial awareness of tumor preoperative planning. The deep learning DCU-Net algorithm model can improve the performance of tumor CT image segmentation, and the reconstructed fine model can better reflect the actual situation of individual tumors; the MR system constructed based on this model enhances clinicians’ understanding of tumor morphology and spatial relationships. The MR system based on deep learning and three-dimensional visualization technology has great potential in the diagnosis and treatment of bone tumors, and is expected to promote clinical practice and improve efficacy.

Topics & Concepts