Identification of ulcerative colitis and Crohn's disease based on spatial and bilinear attention network

Jing QI; Guangcong RUAN; Yi YANG; Yi WU; Qian CAO; Yanling WEI; Yongjian NIAN

doi:10.16016/j.2097-0927.202209187

您当前的位置：

首页 >

文章列表页 >

Identification of ulcerative colitis and Crohn's disease based on spatial and bilinear attention network

Basic Medicine | 更新时间：2026-01-09

- Identification of ulcerative colitis and Crohn's disease based on spatial and bilinear attention network
- Journal of Army Medical University Vol. 45, Issue 3, Pages: 227-234(2023)
- 作者机构：
  
  1.400038 重庆, 陆军军医大学(第三军医大学)生物医学工程与影像医学系数字医学教研室
  2.400042 重庆, 陆军特色医学中心消化内科
  3.310016 杭州, 浙江大学医学院附属邵逸夫医院消化内科
- 作者简介：
- 基金信息：
  
  the Project of Postgraduate Scientific Research Innovation of Chongqing(CYS22746)
- DOI：10.16016/j.2097-0927.202209187
  CLC： R319;R447;R574.62
- Received：21 September 2022，
  
  Revised：2022-10-18，
  
  Published：15 February 2023
- 稿件说明：
移动端阅览
Jing QI, Guangcong RUAN, Yi YANG, et al. Identification of ulcerative colitis and Crohn's disease based on spatial and bilinear attention network[J]. Journal of Army Medical University, 2023, 45(3): 227-234.
DOI：

Jing QI, Guangcong RUAN, Yi YANG, et al. Identification of ulcerative colitis and Crohn's disease based on spatial and bilinear attention network[J]. Journal of Army Medical University, 2023, 45(3): 227-234. DOI： 10.16016/j.2097-0927.202209187.

摘要

目的

利用深度学习技术辅助内镜医师识别溃疡性结肠炎(ulcerative colitis

UC)与克罗恩病(Crohn's disease

CD)。

方法

收集2018年1月至2020年11月陆军特色医学中心消化内科与邵逸夫医院消化内科共1 576例受试者的内镜图像

包括CD、UC与正常三类共计34 300幅

并按照9:1的比例随机划分训练集与测试集

用于对网络进行训练与测试。在ResNet50基础上构建新颖的空间双线性深度网络(SABA-ResNet)

引入空间注意机制

通过膨胀卷积扩大感受野以联系上下文信息

并与普通卷积局部归纳特性相配合

自适应聚焦病变区域。利用双线性注意提高网络的特征表示能力

以二阶信息加权特征映射的通道信息

从而提高模型的分类性能。

结果

SABA-ResNet在测试集上对CD、UC和正常识别的总体准确率为92.67%(95%

: 91.91~93.37)

AUC分别为0.978(95%

: 0.972~0.983)、0.977(95%

: 0.971~0.982)和0.999(95%

: 0.998~1.000)

灵敏度分别为88.40%、89.07%、98.89%

特异性分别为95.49%、94.88%、98.93%

F1值分别为88.80%、89.01%和98.60%。消融实验与类激活映射图表明空间注意与双线性注意可帮助模型捕获更多病变区域的特征。

结论

所构建的网络将空间注意与双线性注意相结合

在CD、UC与正常的识别中取得了良好的性能

可以有效辅助内镜医师对UC与CD进行识别。

Abstract

Objective

To identify ulcerative colitis (UC) and Crohn's disease (CD) with aid of deep learning technology for endoscopists.

Methods

From January 2018 to November 2020

the endoscopic images of 1 576 subjects (including 34 300 CD

UC and normal images) were collected from the Department of Gastroenterology of Army Medical Center of PLA and Sir Run Run Shaw Hospital.The training set and test set were randomly divided according to the ratio of 9:1 to train and test the neural network.A novel spatial and bilinear deep network (SABA-ResNet) was constructed on the basis of ResNet50.The spatial attention mechanism was introduced

and the receptive field was expanded by dilated convolution to leverage contextual information

which was combined with the local induction of standard convolution to adaptively focus the lesion region.Bilinear attention was applied to improve the feature representation ability of the network

and the second-order information was used to weight the channel information of the feature map

so as to improve the classification performance of the model.

Results

The overall accuracy of SABA-ResNet for the recognition of CD

UC and normal tissues on the test set was 92.67%(95%

: 91.91~93.37)

the AUC value was 0.978(95%

: 0.972~0.983)

0.977(95%

: 0.971~0.982) and 0.999(95%

: 0.998~1.000)

the sensitivity was 88.40%

89.07% and 98.89%

the specificity was 95.49%

94.88% and 98.93%

and the F1 value was 88.80%

89.01% and 98.60%

respectively.The ablation experiment and the class activation map suggested that spatial attention and bilinear attention could help the model capture more features of the lesion region.

Conclusion

Our constructed network combines spatial attention and bilinear attention

achieves excellent performance in the recognition of CD

UC and normal tissue

and effectively assist endoscopists in the diagnosis of UC and CD.

关键词

Keywords

references

MCDOWELL C, FAROOQ U, HASEEB M. Inflammatory bowel disease[M]. Treasure Island: StatPearls Publishing, 2022.

MALIK T F, AURELIO D M. Extraintestinal manifestations of inflammatory bowel disease[M]. Treasure Island: StatPearls Publishing, 2022.

PARK J, CHEON J H. Incidence and prevalence of inflammatory bowel disease across Asia[J]. Yonsei Med J, 2021, 62(2): 99-108. DOI: 10.3349/ymj.2021.62.2.99.

FLYNN S, EISENSTEIN S. Inflammatory bowel disease presentation and diagnosis[J]. Surg Clin North Am, 2019, 99(6): 1051-1062. DOI: 10.1016/j.suc.2019.08.001.

SMOLANDER J, DEHMER M, EMMERT-STREIB F. Comparing deep belief networks with support vector machines for classifying gene expression data from complex disorders[J]. FEBS Open Bio, 2019, 9(7): 1232-1248. DOI: 10.1002/2211-5463.12652.

TONG Y R, LU K M, YANG Y Y, et al . Can natural language processing help differentiate inflammatory intestinal diseases in China? Models applying random forest and convolutional neural network approaches[J ] . BMC Med Inform Decis Mak, 2020, 20(1): 248. DOI: 10.1186/s12911-020-01277-w.

WANG L J, CHEN L P, WANG X Y, et al . Development of a convolutional neural network-based colonoscopy image assessment model for differentiating Crohn's disease and ulcerative colitis[J ] . Front Med, 2022, 9: 789862. DOI: 10.3389/fmed.2022.789862.

[YU F, KOLTUN V. Multi-scale context aggregation by dilated convolutions[EB/OL ] . 2015, arXiv: 1511.07122. https://arxiv.org/abs/1511.07122 https://arxiv.org/abs/1511.07122 .

SELVARAJU R R, COGSWELL M, DAS A, et al . Grad-CAM: visual explanations from deep networks via gradient-based localization[C ] . IEEE International Conference on Computer Vision, 2017: 618-626.

[VASWANI A, SHAZEER N,PARMAR N, et al . Attention is all You need[C ] //Proceedings of the 31st International Conference on Neural Information Processing Systems. New York: ACM, 2017: 6000-6010. DOI: 10.5555/3295222.3295349 http://dx.doi.org/10.5555/3295222.3295349 ..

GUO M H, XU T X, LIU J J, et al . Attention mechanisms in computer vision: a survey[J ] . Comput Vis Media, 2022, 8(3): 331-368. DOI: 10.1007/s41095-022-0271-y.

[HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C ] //2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018: 7132-7141. DOI: 10.1109/CVPR.2018.00745 http://dx.doi.org/10.1109/CVPR.2018.00745 .

[JADERBERG M, SIMONYAN K, ZISSERMAN A, et al . Spatial transformer networks[C ] //Proceedings of the 28th International Conference on Neural Information Processing Systems. New York: ACM, 2015: 2017-2025. DOI: 10.5555/2969442.2969465 http://dx.doi.org/10.5555/2969442.2969465 ..

[WOO S, PARK J, LEE J Y, et al . CBAM: convolutional block attention module[EB/OL ] . arXiv, 2018: 1807.06521. https://arxiv.org/abs/1807.06521 https://arxiv.org/abs/1807.06521 ..

[GAO Z L, XIE J T, WANG Q L, et al . Global second-order pooling convolutional networks[C ] //2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020: 3019-3028. DOI: 10.1109/CVPR.2019.00314 http://dx.doi.org/10.1109/CVPR.2019.00314 ..

[QIN Z Q, ZHANG P Y, WU F, et al . FcaNet: frequency channel attention networks[C ] //2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2022: 763-772. DOI: 10.1109/ICCV48922.2021.00082 http://dx.doi.org/10.1109/ICCV48922.2021.00082 ..

[LIN T Y, ROYCHOWDHURY A, MAJI S. Bilinear CNN models for fine-grained visual recognition[C ] //2015 IEEE International Conference on Computer Vision (ICCV). IEEE, 2016: 1449-1457. DOI: 10.1109/ICCV.2015.170 http://dx.doi.org/10.1109/ICCV.2015.170 .

[KIM J H, JUN J, ZHANG B T. Bilinear attention networks[C ] //Proceedings of the 32nd International Conference on Neural Information Processing Systems. New York: ACM, 2018: 1571-1581. DOI: 10.5555/3326943.3327087 http://dx.doi.org/10.5555/3326943.3327087 .

[FANG P F, ZHOU J M, ROY S, et al . Bilinear attention networks for person retrieval[C ] //2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2020: 8029-8038. DOI: 10.1109/ICCV.2019.00812 http://dx.doi.org/10.1109/ICCV.2019.00812 ..

XIAO B, YANG Z Y, QIU X M, et al . PAM-DenseNet: a deep convolutional neural network for computer-aided COVID-19 diagnosis[J ] . IEEE Trans Cybern, 2022, 52(11): 12163-12174. DOI: 10.1109/TCYB.2020.3042837.

ZHANG J P, XIE Y T, XIA Y, et al . Attention residual learning for skin lesion classification[J ] . IEEE Trans Med Imaging, 2019, 38(9): 2092-2103. DOI: 10.1109/TMI.2019.2893944.

XING X H, YUAN Y X, MENG M Q H. Zoom in lesions for better diagnosis: attention guided deformation network for WCE image classification[J]. IEEE Trans Med Imaging, 2020, 39(12): 4047-4059. DOI: 10.1109/TMI.2020.3010102.

SEYEDIAN S S, NOKHOSTIN F, MALAMIR M D. A review of the diagnosis, prevention, and treatment methods of inflammatory bowel disease[J]. J Med Life, 2019, 12(2): 113-122. DOI: 10.25122/jml-2018-0075.

RUAN G C, QI J, CHENG Y, et al . Development and validation of a deep neural network for accurate identification of endoscopic images from patients with ulcerative colitis and Crohn's disease[J ] . Front Med (Lausanne), 2022, 9(8): 854677. DOI: 10.3389/fmed.2022.854677.

[LI P H, XIE J T, WANG Q L, et al . Towards faster training of global covariance pooling networks by iterative matrix square root normalization[C ] //2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018: 947-955. DOI: 10.1109/CVPR.2018.00105 http://dx.doi.org/10.1109/CVPR.2018.00105 ..

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Integrative model combining deep learning, clinical and radiomic features enhances EGFR mutation prediction in non-small cell lung cancer

Predictive performance of CT images-based 3D ResNet18 model for identifying lung tuberculosis drug resistance

Advances in deep learning and MRI co-assisted genotyping of brain gliomas

Automatic segmentation of female urine control anatomical elements and related structures in MRI images based on deep learning

Ultrasound thyroid nodule segmentation algorithm based on wavelet transform and CNN-Transformer

Related Author

YANG Taotao

CHEN Wei

CHEN Cancan

YAN Wanying

WANG Dawei

XIONG Kunlin

SUN Zhiyuan

WANG Xianqi

Related Institution

Department of Diagnostic Radiology， Eastern Theatre General Hospital

Department of Radiology，First Affiliated Hospital， Army Medical University （Third Military Medical University）

Infervision Medical Technology Co.， Ltd

Department of Radiology， Daping Hospital， Army Medical University

Department of Medical Imaging， Chongqing Public Health Medical Center

⁰