- Open Access
Performance of automated CT ASPECTS in comparison to physicians at different levels on evaluating acute ischemic stroke at a single institution in China
Chinese Neurosurgical Journal volume 7, Article number: 40 (2021)
Our aim was to evaluate the sensitivity and specificity of the automated computer-based Alberta Stroke Program Early CT Score (e-ASPECTS) for acute stroke patients and compare the result with physicians at different levels.
In our center, e-ASPECTS and 9 physicians at different levels retrospectively and blindly assessed baseline computed tomography (CT) images of 55 patients. Sensitivity, specificity, receiver-operating characteristic curves, Bland–Altman plots with mean score error, and Matthews correlation coefficients were calculated. Comparisons were made between the scores by physicians and e-ASPECTS with diffusion-weighted imaging (DWI) being the ground truth. Two methods for clustered data were used to estimate sensitivity and specificity in the region-based analysis.
In total, 1100 (55 patients × 20 regions per patient) ASPECTS regions were scored. In the region-based analysis, sensitivity of e-ASPECTS was better than junior doctors and residents (0.576 vs 0.165 and 0.111, p < 0.05) but inferior to senior doctors (0.576 vs 0.617). Specificity was lower than junior doctors and residents (0.883 vs 0.971 and 0.914) but higher than senior doctors (0.883 vs 0.809, p < 0.05). E-ASPECTS had the best Matthews correlation coefficient of 0.529, compared to senior doctors, junior doctors, and residents (0.463, 0.251, and 0.087, respectively).
e-ASPECTS showed a similar performance to that of senior physicians in the assessment of brain CT of acute ischemic stroke patients with the Alberta Stroke Program Early CT score method.
Computed tomography (CT) is still the most widely used tool for AIS because it is fast, efficient, easy to access, and reliable to rule out hemorrhage, while others are more time consuming and contraindicated for some patients [1,2,3]. Despite its advantage compared to other imaging modalities, subsequent research showed that there was still inconsistency between observer to recognize and quantify these changes. Therefore, a useful scoring system, the Alberta stroke program early CT score (ASPECTS), was designed to semi-quantify and describe the topography of cerebral tissue damage caused by AIS [4, 5]. Unfortunately, the interrater variability and modest interobserver agreement which depend on their experiences have become a serious limitation for the scoring methods [6,7,8,9,10]. Hence, an automated software application based on ASPECTS scoring system (e-ASPECTS) was made to enhance the usefulness of CT imaging and to optimize the evaluation of AIS patients. In this study, we compared the scoring performance of e-ASPECTS to those of nine independent physicians with different working experiences.
Fifty-five patients participated in this study (Fig. 1); a CT and an MRI were obtained for stroke diagnosis. The automatic device e-ASPECTS (UGuard V184.108.40.206d9fb70114, Union Strong (Beijing) Technology Co. Ltd, China) and nine doctors interpreted the CT scanners using ASPECTS independently. All physicians were instructed to evaluate the CT in the correct use of the ASPECTS scoring system according to www.aspectsinstroke.com. Scorers were allowed to view the whole brain scan and to scroll the images backward and forward and to adjust the contrast, brightness, window/level, and magnification of the images. The physicians were grouped according to their working experiences. The senior doctors had extensive experiences in neurovascular disease for at least 10 years. The junior doctors’ experiences were varied, with at least 5 years in neurovascular disease. The resident doctors had experiences in neurovascular disease for at least 3 years.
We developed 1 classifier and 2 segmentation models to accomplish the task. The classifier used visual geometry group (VGG) model pre-trained on a large dataset as feature extractor, through which we can fine tune to extract target slices from the CT head series as in stage 1. In stage 2, segment model 1 detects 14 areas in the nucleus mass layer, and segment model 2 detects 6 areas in the nucleus mass upper layer. The segmentation models had the encoder-decoder architecture as U-Net and made better use of features through dense connections (Fig. 2).
The ASPECTS score of the ground truth was determined by Tiantan Hospital Stroke Centre based on medical professional interpretation of the MRI scanner as the gold standard. The ground truth was defined as the ASPECTS on DWI and was scored on consensus basis by two experts who were not blinded to the clinical information. Any detectable DWI lesion attributable to acute cerebral ischemia, independently of its size, was scored within the different ASPECTS regions by assessment of the whole DWI scan. We evaluated the performance of the automatic device by comparing how well the ASPECTS aligned with the ground truth. The study was approved by the committee of ethics in Beijing Tiantan Hospital. All patients were treated according to the stroke management protocols.
The automatic device produces binary scores of 20 regions between the cerebral hemispheres of every patient. The physicians determined the stroke by region as well. The ASPECTS score was calculated by summing up the binary scores of all the regions. We evaluated the alignment between each group and the ground truth using the concordance correlation coefficient (CCC). CCC ranges from − 1 to 1, and a higher value indicates better alignment. We also produced Bland-Altman plots to compare the alignment between each group and the ground truth. The dots in a Bland-Altman plot with a random pattern indicate good alignment between the two methods. Besides, the histograms of the difference in ASPECTS scores were produced to assess measurement accuracy.
As we were interested in diagnosing large infarct core (ASPECTS < 6), we dichotomized ASPECTS using the cut-off ASPECTS < 6 and evaluated the performance of the automatic device as well as the physicians. We illustrated the performance using the receiver-operating characteristic (ROC) curves, where the curve plots showed sensitivity changed relative to specificity. We compared sensitivity as well as specificity between the device and any of the three groups of physicians using a generalized linear mixed-effects model. Specifically, we utilized a logistic regression model. We used a non-inferiority test to examine whether the automatic device was no worse than the physicians by groups. The device was considered as not worse than its counterpart if the sensitivity was at most 0.1 less. Similar criteria applied to the specificity.
Since the lack of efficiency in time was a concern when diagnosing acute stroke, we also estimated the average processing time and compared the result between the device and doctor groups. An analysis of variance (ANOVA) model was used for the comparison. The analysis was performed in R version 3.3.1.
We summarized the concordance correlation coefficients (CCC) by measurement methods in Table 1. The estimated CCCs, as well as the corresponding 95% confidence interval, were produced. We found the automatic device provided the most accurate ASPECTS score (0.529), followed by the senior doctors (0.463). The ASPECTS score was least accurate in residents group (0.087). Figure 3 summarized the Bland-Altman (BA) plots by group. We found the BA plots for all physician groups showed some systemic patterns while the plot for the device was random. Figure 4 summarized the difference in ASPECTS score between each group and the gold standard. The difference was closer to 0 between the automatic device and the ground truth. These findings indicated that the device was more accurate compared to the physicians in determining the ASPECTS score.
The ROC curve was shown in Fig. 5. The automatic device performed better than the residents and junior doctors in correctly detecting the large infarct core; it performed equally well to the senior doctors (0.576 vs 0.617), in terms of sensitivity. We also found the specificity to be higher with the device, compared to the senior doctors (0.883 vs 0.809), while being slightly lower than the residents (0.883 vs 0.914) and junior doctors (0.883 vs 0.971). Besides, sensitivity in the automatic device was no worse than that with the residents (0.576 vs 0.111) as well as the junior doctors (0.576 vs 0.165) at 0.05 significance level; specificity in the automatic device was at least as good as that with the senior doctors at 0.05 significance level (Table 2).
The mean and standard deviation of processing time were summarized in Table 3. We also illustrated the processing time by group in Fig. 6. In addition, the difference in time between the group of doctors and the device was summarized in Fig. 7. We found that the junior doctors needed the least processing time on average, while the residents took longer time (3.148 min, mean average). However, the processing time was not statistically significant among other groups (2.193 s, 2.753 min, and 1.667 min, for device, senior doctors, and junior doctors, respectively).
In the developing country, different cities provide different quality of medical service. Since the devices are not well-distributed, they can affect the treatment outcome . In acute ischemic stroke (AIS), it is important to achieve optimal and good functional outcome in early reperfusion [12,13,14,15]. This has to be supported by the assessment device that provides less time and revealed AIS accurately. CT perfusion (CTP), magnetic resonance imaging (MRI) perfusion, and diffusion weighted imaging (DWI)/fluid attenuation inversion recovery (FLAIR) mismatch have been suggested prior to reperfusion [16,17,18,19,20,21]. However, CT is still the most widely used tool for AIS because it is fast, efficient, easy to access, and reliable to rule out hemorrhage, whereas others are more time-consuming and contraindicated for some patients [22,23,24]. The capability of doctors in hospital plays an important role in supporting the treatment. Although CT can reveal a significant change of AIS signs such as edema and hypoperfusion , not all doctors in developing and developed country are sensitive and aware of these clinical changes [2, 26], particularly those who are not specialized in stroke care diagnosis and management. Therefore, a machine-learning method aiming to help doctors make better clinical decisions is urgently needed and should be disseminated worldwide. Consequently, patients with AIS will benefit from the corresponding treatment.
Our current study aimed to evaluate the performance of an automated software application based on ASPECTS scoring system (e-ASPECTS) in assessing patients with AIS. We compared e-ASPECTS to physicians with different experiences from the same center. Our study showed that e-ASPECT performance was not only as good as the less-experienced doctors, but also not inferior to the senior doctors. The findings were further supported by the Bland-Altman plots and CCC. Both the relatively random pattern in the BA plots and the higher values in CCC (0.529) suggested good performance of the device. From this result, e-ASPECTS provided the best agreement with the ground truth data as compared to physicians. Specificity of e-ASPECTS was lower than senior doctors, but higher than junior doctors and residents, because these two groups were less sensitive. The difference in specificities was significant although it was quite small; it might be due to the unaffected regions that the majority of the data predominated when calculating the specificity. Meanwhile, the differences in sensitivities were quite large. The sensitivity of e-ASPECTS was lower than senior doctors, but significantly higher than residents and junior doctors. This result suggested that physicians with different experiences perform differently, i.e., a relatively large increase in sensitivity at a cost of small decrease in specificity for senior doctors. Besides, the device required least processing time in e-ASPECTS when analyzing ischemic area, compared to other groups. Although e-ASPECTS was superior in processing time when analyzing ischemic changes, this result was only considered as a reference. However, this result showed that in the future e-ASPECTS could be suitable in supporting early diagnosis and management of acute ischemic case, in order to achieve good prognosis.
Our finding with respect to sensitivity and specificity is in accordance with previous reported literatures [1, 2, 26,27,28]. This electronic ASPECTS could be suitable to be applied in routine diagnosis of acute ischemic stroke because it is CT based, which is widely used in most institutions. Even in recent study, it has been used to predict the prognosis of acute ischemic stroke patients undergoing endovascular reperfusion therapies. Research shows that it can provide important technical support in estimating patient’s prognosis [29, 30]. Moreover, CT is not only capable to shorten time management due to its easy accessibility and fewer physical limitations compared to MRI, but it is also more widely available in most hospitals . On the basis of these studies, the application of e-ASPECTS should be promoted more in clinical routine, especially in developing countries, where medical services and quality are not well-distributed, especially in the rural area. It requires at least 5 years of training to get familiar with diagnosing neurovascular diseases for physicians, while the incidence of acute ischemic stroke is increasing. Thus, more applications of electronic-based ASPECTS are urgently needed. It should be noted that the aim of e-ASPECTS is not to replace expert assessment of the scan or junior physicians, but instead to assist them in clinical routine and research. Although the objectivity of e-ASPECTS was higher compared to physicians, the pre-existing conditions and various changes in human brain are still challenging for the computer-aided ASPECTS assessment because the software is only capable to distinguish acute ischemic changes, and incapable to differentiate the etiologies of brain tissue damage. Hence, a check for plausibility is still needed by the physicians.
In summary, e-ASPECTS showed non-inferiority in applying the ASPECTS to acute ischemic changes compared to experienced doctors and superior compared to moderately and less experienced doctors. Despite this promising result, our study has had several limitations, such as restricted number of patients and lack of generalization as a single center study. A future study involving multiple centers in different regions is in preparation. Besides, the physicians participated in this study were mainly neurologists; more physicians from different departments, such as neuroradiology, neurosurgery, emergency room, and intensive care, should be included. Another limitation was that we did not evaluate the prognosis of patient with or without endovascular treatment based on the e-ASPECTS score. Further studies are required to verify the performance of e-ASPECTS in clinical routine.
E-ASPECTS performance was shown to be non-inferior to senior physicians and better than junior physicians in assessing the ASPECTS score of AIS patients. E-ASPECTS should be widely applied in routine diagnosis of AIS.
Availability of data and materials
All data are available to researchers on request for purposes of reproducing the results or replicating the procedure by directly contacting the corresponding author.
Alberta Stroke Program Early CT Score
Fluid attenuation inversion recovery
Magnetic resonance imaging
Visual geometry group
Pfaff J, Herweh C, Schieber S, Schonenberger S, Bosel J, Ringleb PA, et al. e-ASPECTS correlates with and is predictive of outcome after mechanical thrombectomy. AJNR Am J Neuroradiol. 2017;38(8):1594–9. https://doi.org/10.3174/ajnr.A5236.
Herweh C, Ringleb PA, Rauch G, Gerry S, Behrens L, Mohlenbruch M, et al. Performance of e-ASPECTS software in comparison to that of stroke physicians on assessing CT scans of acute ischemic stroke patients. Int J Stroke. 2016;11(4):438–45. https://doi.org/10.1177/1747493016632244.
Bentley P, Ganesalingam J, Carlton Jones AL, Mahady K, Epton S, Rinne P, et al. Prediction of stroke thrombolysis outcome using CT brain machine learning. Neuroimage Clin. 2014;4:635–40. https://doi.org/10.1016/j.nicl.2014.02.003.
Pexman JH, Barber PA, Hill MD, Sevick RJ, Demchuk AM, Hudon ME, et al. Use of the Alberta Stroke Program Early CT Score (ASPECTS) for assessing CT scans in patients with acute stroke. AJNR Am J Neuroradiol. 2001;22(8):1534–42.
Barber PA, Demchuk AM, Zhang J, Buchan AM. Validity and reliability of a quantitative computed tomography score in predicting outcome of hyperacute stroke before thrombolytic therapy. ASPECTS Study Group. Alberta Stroke Programme Early CT Score. Lancet. 2000;355:1670–4.
Demaerschalk BM, Silver B, Wong E, Merino JG, Tamayo A, Hachinski V. ASPECT scoring to estimate >1/3 middle cerebral artery territory infarction. Can J Neurol Sci. 2006;33(2):200–4. https://doi.org/10.1017/S0317167100004972.
Mak HK, Yau KK, Khong PL, Ching AS, Cheng PW, Au-Yeung PK, Pang PK, Wong KC, Chan BP, Alberta Stroke Programme Early CTS. Hypodensity of >1/3 middle cerebral artery territory versus Alberta Stroke Programme Early CT Score (ASPECTS): comparison of two methods of quantitative evaluation of early CT changes in hyperacute ischemic stroke in the community setting. Stroke. 2003;34:1194–6.
Finlayson O, John V, Yeung R, Dowlatshahi D, Howard P, Zhang L, et al. Interobserver agreement of ASPECT score distribution for noncontrast CT, CT angiography, and CT perfusion in acute stroke. Stroke. 2013;44(1):234–6. https://doi.org/10.1161/STROKEAHA.112.665208.
Farzin B, Fahed R, Guilbert F, Poppe AY, Daneault N, Durocher AP, et al. Early CT changes in patients admitted for thrombectomy: Intrarater and interrater agreement. Neurology. 2016;87(3):249–56. https://doi.org/10.1212/WNL.0000000000002860.
McTaggart RA, Jovin TG, Lansberg MG, Mlynash M, Jayaraman MV, Choudhri OA, et al. Alberta stroke program early computed tomographic scoring performance in a series of patients undergoing computed tomography and MRI: reader agreement, modality agreement, and outcome prediction. Stroke. 2015;46(2):407–12. https://doi.org/10.1161/STROKEAHA.114.006564.
Grotta JC, Hacke W. Stroke neurologist’s perspective on the new endovascular trials. Stroke. 2015;46(6):1447–52. https://doi.org/10.1161/STROKEAHA.115.008384.
Albers GW, Marks MP, Kemp S, Christensen S, Tsai JP, Ortega-Gutierrez S, et al. Thrombectomy for stroke at 6 to 16 hours with selection by perfusion imaging. N Engl J Med. 2018;378(8):708–18. https://doi.org/10.1056/NEJMoa1713973.
Nogueira RG, Jadhav AP, Haussen DC, Bonafe A, Budzik RF, Bhuva P, et al. Thrombectomy 6 to 24 hours after stroke with a mismatch between deficit and infarct. N Engl J Med. 2018;378(1):11–21. https://doi.org/10.1056/NEJMoa1706442.
Powers WJ, Derdeyn CP, Biller J, Coffey CS, Hoh BL, Jauch EC, et al. 2015 American Heart Association/American Stroke Association Focused Update of the 2013 Guidelines for the early management of patients with acute ischemic stroke regarding endovascular treatment: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke. 2015;46:3020–35.
Wahlgren N, Moreira T, Michel P, Steiner T, Jansen O, Cognard C, et al. Mechanical thrombectomy in acute ischemic stroke: consensus statement by ESO-Karolinska Stroke Update 2014/2015, supported by ESO, ESMINT, ESNR and EAN. Int J Stroke. 2016;11(1):134–47. https://doi.org/10.1177/1747493015609778.
Millan M, Aleu A, Almendrote M, Serena J, Castano C, Roquer J, et al. Safety and effectiveness of endovascular treatment of stroke with unknown time of onset. Cerebrovasc Dis. 2014;37(2):134–40. https://doi.org/10.1159/000357419.
Barreto AD, Martin-Schild S, Hallevi H, Morales MM, Abraham AT, Gonzales NR, et al. Thrombolytic therapy for patients who wake-up with stroke. Stroke. 2009;40(3):827–32. https://doi.org/10.1161/STROKEAHA.108.528034.
Michel P, Ntaios G, Reichhart M, Schindler C, Bogousslavsky J, Maeder P, et al. Perfusion-CT guided intravenous thrombolysis in patients with unknown-onset stroke: a randomized, double-blind, placebo-controlled, pilot feasibility trial. Neuroradiology. 2012;54(6):579–88. https://doi.org/10.1007/s00234-011-0944-1.
Natarajan SK, Snyder KV, Siddiqui AH, Ionita CC, Hopkins LN, Levy EI. Safety and effectiveness of endovascular therapy after 8 hours of acute ischemic stroke onset and wake-up strokes. Stroke. 2009;40(10):3269–74. https://doi.org/10.1161/STROKEAHA.109.555102.
Ma H, Parsons MW, Christensen S, Campbell BC, Churilov L, Connelly A, et al. A multicentre, randomized, double-blinded, placebo-controlled phase III study to investigate EXtending the time for Thrombolysis in Emergency Neurological Deficits (EXTEND). Int J Stroke. 2012;7(1):74–80. https://doi.org/10.1111/j.1747-4949.2011.00730.x.
Ebinger M, Scheitz JF, Kufner A, Endres M, Fiebach JB, Nolte CH. MRI-based intravenous thrombolysis in stroke patients with unknown time of symptom onset. Eur J Neurol. 2012;19(2):348–50. https://doi.org/10.1111/j.1468-1331.2011.03504.x.
Olive-Gadea M, Martins N, Boned S, Carvajal J, Moreno MJ, Muchada M, et al. Baseline ASPECTS and e-ASPECTS correlation with infarct volume and functional outcome in patients undergoing mechanical thrombectomy. J Neuroimaging. 2019;29(2):198–202. https://doi.org/10.1111/jon.12564.
Padroni M, Boned S, Ribo M, Muchada M, Rodriguez-Luna D, Coscojuela P, et al. CBV_ASPECTS improvement over CT_ASPECTS on determining irreversible ischemic lesion decreases over time. Interv Neurol. 2016;5(3-4):140–7. https://doi.org/10.1159/000446969.
Chang CC, Yeh HL, Hsiao CY, Lien LM. 10-point CT-ASPECTS-based reperfusion therapy for unknown onset stroke. J Formos Med Assoc. 2018;117:640–5.
del Zoppo GJ, von Kummer R, Hamann GF. Ischaemic damage of brain microvessels: inherent risks for thrombolytic treatment in stroke. J Neurol Neurosurg Psychiatry. 1998;65(1):1–9. https://doi.org/10.1136/jnnp.65.1.1.
Nagel S, Sinha D, Day D, Reith W, Chapot R, Papanagiotou P, et al. e-ASPECTS software is non-inferior to neuroradiologists in applying the ASPECT score to computed tomography scans of acute ischemic stroke patients. Int J Stroke. 2017;12(6):615–22. https://doi.org/10.1177/1747493016681020.
Mitomi M, Kimura K, Aoki J, Iguchi Y. Comparison of CT and DWI findings in ischemic stroke patients within 3 hours of onset. J Stroke Cerebrovasc Dis. 2014;23(1):37–42. https://doi.org/10.1016/j.jstrokecerebrovasdis.2012.08.014.
Nezu T, Koga M, Nakagawara J, Shiokawa Y, Yamagami H, Furui E, et al. Early ischemic change on CT versus diffusion-weighted imaging for patients with stroke receiving intravenous recombinant tissue-type plasminogen activator therapy: stroke acute management with urgent risk-factor assessment and improvement (SAMURAI) rt-PA registry. Stroke. 2011;42(8):2196–200. https://doi.org/10.1161/STROKEAHA.111.614404.
Goyal M, Menon BK, van Zwam WH, Dippel DW, Mitchell PJ, Demchuk AM, et al. Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials. Lancet. 2016;387(10029):1723–31. https://doi.org/10.1016/S0140-6736(16)00163-X.
Yoo AJ, Berkhemer OA, Fransen PSS, van den Berg LA, Beumer D, Lingsma HF, et al. Effect of baseline Alberta Stroke Program Early CT Score on safety and efficacy of intra-arterial treatment: a subgroup analysis of a randomised phase 3 trial (MR CLEAN). Lancet Neurol. 2016;15(7):685–94. https://doi.org/10.1016/S1474-4422(16)00124-1.
The authors would like to express their gratefulness to research and development team of Union Strong (Beijing) Technology Co. Ltd, China, for their support in this work.
This work was supported by the National Key Research and Development Program of China (2016YFC1301500), Beijing Hospitals Authority Youth Programme (QML20170502), and China Postdoctoral Science Foundation (2020-YJ-008).
Ethics approval and consent to participate
The present study was approved by the ethics committee of Beijing Tiantan Hospital (KY2014-051-01), and informed consent was obtained from all participants prior to commencing the study.
Consent for publication
The authors declare that they have no conflict of interest.
About this article
Cite this article
Huo, X., Raynald, Jin, H. et al. Performance of automated CT ASPECTS in comparison to physicians at different levels on evaluating acute ischemic stroke at a single institution in China. Chin Neurosurg Jl 7, 40 (2021). https://doi.org/10.1186/s41016-021-00257-x
- Computed tomography
- Ischemic stroke
- Automatic device