Machine learning for the real-time assessment of left ventricular ejection fraction in critically ill patients: a bedside evaluation by novices and experts in echocardiography
Critical Care volume 26, Article number: 386 (2022)
Machine learning algorithms have recently been developed to enable the automatic and real-time echocardiographic assessment of left ventricular ejection fraction (LVEF) and have not been evaluated in critically ill patients.
Real-time LVEF was prospectively measured in 95 ICU patients with a machine learning algorithm installed on a cart-based ultrasound system. Real-time measurements taken by novices (LVEFNov) and by experts (LVEFExp) were compared with LVEF reference measurements (LVEFRef) taken manually by echo experts.
LVEFRef ranged from 26 to 80% (mean 54 ± 12%), and the reproducibility of measurements was 9 ± 6%. Thirty patients (32%) had a LVEFRef < 50% (left ventricular systolic dysfunction). Real-time LVEFExp and LVEFNov measurements ranged from 31 to 68% (mean 54 ± 10%) and from 28 to 70% (mean 54 ± 9%), respectively. The reproducibility of measurements was comparable for LVEFExp (5 ± 4%) and for LVEFNov (6 ± 5%) and significantly better than for reference measurements (p < 0.001). We observed a strong relationship between LVEFRef and both real-time LVEFExp (r = 0.86, p < 0.001) and LVEFNov (r = 0.81, p < 0.001). The average difference (bias) between real time and reference measurements was 0 ± 6% for LVEFExp and 0 ± 7% for LVEFNov. The sensitivity to detect systolic dysfunction was 70% for real-time LVEFExp and 73% for LVEFNov. The specificity to detect systolic dysfunction was 98% both for LVEFExp and LVEFNov.
Machine learning-enabled real-time measurements of LVEF were strongly correlated with manual measurements obtained by experts. The accuracy of real-time LVEF measurements was excellent, and the precision was fair. The reproducibility of LVEF measurements was better with the machine learning system. The specificity to detect left ventricular dysfunction was excellent both for experts and for novices, whereas the sensitivity could be improved.
Trial registration: NCT05336448. Retrospectively registered on April 19, 2022.
The assessment of left ventricular ejection fraction (LVEF) is part of the point of care echocardiographic evaluation of critically ill patients [1,2,3]. It has the disadvantage of being time-consuming and operator dependent. Machine learning algorithms have recently been developed to facilitate, automate, and decrease the variability of echocardiographic measurements [4,5,6,7]. Several algorithms have been designed specifically for the real-time assessment of LVEF [8,9,10]. They have been trained to recognize specific ultrasound images, enable instantaneous image quality control, and measure LVEF automatically in just a few seconds. However, clinical validation studies remain scarce and have been done in ambulatory cardiac patients [8,9,10].
In critically ill patients, we compared real-time LVEF measurements taken with a new machine learning algorithm to reference manual measurements taken by experts in echocardiography.
We prospectively studied critically ill patients who required an echocardiographic evaluation during their ICU stay and in whom it was possible to obtain transthoracic images enabling a manual and quantitative evaluation of left ventricular systolic function. Real-time LVEF measurements were taken with a machine learning algorithm (Real-Time EF, GE Healthcare, Chicago, USA) installed on a cart-based ultrasound system (Venue, GE Healthcare). The real-time LVEF software is a neural network algorithm which has been trained with thousands of cardiac images to automatically detect the 4-chamber view of the heart, locate landmarks on the left ventricular wall and detect end-diastolic and end-systolic times from the mitral valve motion. Once the endocardial border is detected, the algorithm provides immediate user feedback regarding image quality using color-coding. When image quality is considered acceptable (green or yellow endocardial border displayed on screen), left ventricular volumes are automatically estimated from the single-plane Simpson disk method, enabling LVEF calculation from real-time end-diastolic and end-systolic volumes.
Real-time LVEF measurements obtained by a novice (LVEFNov) and by an expert (LVEFExp) were compared with LVEF measurements taken manually by an expert in critical care echocardiography (LVEFRef). Seven novices (all residents in our department and beginners in echocardiography) and two experts (senior intensivists with the European Diploma in Advanced Critical Care Echocardiography) participated in data collection. Measurements taken in triplicate were averaged for comparisons, and the intra-operator reproducibility was assessed by calculating the coefficient of variation (standard deviation divided by the mean) expressed as a percentage.
The quality of echo images was classified as good, fair, or poor by the experts, and as green (optimal), yellow (acceptable), or red (not acceptable for real-time LVEF measurements) by the machine learning algorithm.
Results are expressed as mean ± standard deviation (SD). Agreement between real-time and reference LVEF measurements was tested using the Bland–Altman method. Statistical comparisons were made with a t-test. A p value < 0.05 was considered statistically significant.
We prospectively enrolled 95 patients (mean age 60 ± 17 yr) over a 9-month period. Most patients were admitted for medical reasons and 32 (34%) were mechanically ventilated at the time of the ultrasound evaluation (Additional file 1: Table S1). Reference LVEF ranged from 26 to 80% (mean 54 ± 12%) and the reproducibility of manual measurements was 9 ± 6%. Thirty patients (32%) had a LVEFRef < 50% (left ventricular systolic dysfunction).
Real-time LVEFExp ranged from 31 to 68% (mean 54 ± 10%). We observed a strong relationship (r = 0.86, p < 0.001) between reference and real-time LVEFExp (Fig. 1). The average difference (bias) between real-time LVEFExp and reference LVEF was 0 ± 6% with 95% limits of agreement of − 12 to + 11% (Fig. 1). The intra-operator reproducibility of measurements was better for real-time LVEFExp than for reference manual measurements (5 ± 4% vs. 9 ± 6%, p < 0.001). The sensitivity and specificity of real-time LVEFExp to detect systolic dysfunction were 70% and 98%, respectively.
Real-time LVEFNov ranged from 28 to 70% (mean 54 ± 9%). We observed a strong relationship (r = 0.81, p < 0.001) between LVEFRef and real-time LVEFNov (Fig. 1). The average difference (bias) between real-time LVEFNov and LVEFRef was 0 ± 7% with 95% limits of agreement of − 14 to + 13% (Fig. 1). The intra-operator reproducibility of measurements was better for real-time LVEFNov than for reference manual measurements (6 ± 5% vs. 9 ± 6%, p < 0.001). The sensitivity and specificity of real-time LVEFNov to detect systolic dysfunction were 73% and 98%, respectively.
According to experts’ judgement, the quality of echo images was good, fair, and poor in 41, 43, and 11 patients, respectively. The average difference (bias) between real-time and reference LVEF measurements was comparable when images were of good quality (n = 41) and of fair or poor quality (n = 54), both for experts and novices (Table 1). And results did not change significantly after excluding the 11 patients with poor image quality (Table 1).
According to the machine learning algorithm, the quality of echo images was green, yellow, and red flagged in 80, 15 and 0 patients, respectively. Results did not change significantly after excluding the 15 patients in whom images were non-optimal/yellow flagged (Table 1).
The average difference (bias) between real-time and reference LVEF measurements was slightly higher in mechanically ventilated (n = 32) than in non-mechanically ventilated patients (n = 63), both for experts (− 2 ± 7% vs. 0 ± 5%) and novices (− 1 ± 8% vs. 0 ± 6%). However, observed differences did not reach statistical significance.
An increasing number of anesthesiologists and intensivists have been trained to perform qualitative echocardiographic assessments [1,2,3]. However, quantitative evaluations remain challenging for many, particularly for novices. In the present study, we tested an artificial intelligence-enabled tool specifically designed to facilitate and automatize the bedside measurements of LVEF. Our findings suggest that this tool enables a clinically acceptable estimation of LVEF when compared to manual measurements. They also suggest that the real-time LVEF tool enables novices to assess LVEF with a better reproducibility than what experts can achieve manually.
Several machine learning algorithms have been designed to assess LVEF from a parasternal long axis view or from an apical 2 or 4-chamber view [8,9,10]. Comparison studies published so far yielded promising results. Indeed, close correlations and good agreements have been reported between LVEF measurements taken by skilled operators and by machine learning algorithms, particularly when the algorithm detects and analyze the apical 4-chamber view [9, 10]. However, clinical validation studies remain scarce and have been done in ambulatory cardiac patients. Our study appears to be the first evaluation done in critically ill patients in whom transthoracic echocardiography is often challenging, in particular when patients are mechanically ventilated. Our findings suggest that the real-time LVEF algorithm may help clinicians, including beginners in echocardiography, to accurately measure LVEF in just a few seconds. Such a tool may contribute to further increase the adoption of point of care echocardiographic evaluations in critically ill patients.
Our study has limitations. Because ultrasound evaluations are time-consuming, we studied hemodynamically stable patients to ensure comparability between measurements taken at each step of the evaluation (LVEF measurements were first taken by a trainee, then by an expert both manually and with the automatic method). Also, we did not assess the ability of the new real-time LVEF method to track changes in LVEF. A small number of patients had a severely impaired left ventricular systolic function (LVEFRef < 30%, n = 4) or a hyperkinetic ventricle (LVEFRef > 70%, n = 2). Therefore, future studies will need to assess the clinical value of the real-time LVEF algorithm during hemodynamic instability, in patients with a very low or supranormal LVEF, and during therapeutic interventions (e.g., inotropic stimulation) known to induce significant changes in systolic function.
Machine learning-enabled real-time measurements of LVEF were strongly correlated with manual measurements obtained by experts. The accuracy of real-time LVEF measurements was excellent, and the precision was fair. The reproducibility of LVEF measurements was better with the machine learning system, including for novices. The specificity to detect left ventricular systolic dysfunction was excellent both for experts and novices, whereas the sensitivity could be improved. Studies are needed to confirm our findings in mechanically ventilated patients with cardiogenic shock or hyperdynamic states.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Left ventricular ejection fraction
- LVEFRef :
Reference left ventricular ejection fraction
- LVEFExp :
Real-time left ventricular ejection fraction measured by experts
- LVEFNov :
Real-time left ventricular ejection fraction measured by novices
Orde S, Slama M, Hilton A, et al. Pearls and pitfalls in comprehensive critical care echocardiography. Crit Care. 2017;21:279.
Vieillard-Baron A, Millington SJ, Sanfilippo F, et al. A decade of progress in critical care echocardiography: a narrative review. Intensive Care Med. 2019;45:770–88.
Marbach JA, Almufleh A, Di Santo P, et al. A shifting paradigm: the role of focused cardiac ultrasound in bedside patient assessment. Chest. 2020;158:2107–18.
Dey D, Slomka PJ, Leeson P, et al. Artificial intelligence in cardiovascular imaging. J Am Coll Cardiol. 2019;73:1317–35.
Narang A, Bae R, Hong H, et al. Using a deep-learning algorithm to guide novice to acquire echocardiograms for limited diagnostic use. JAMA Cardiol. 2021;6:624–32.
Nabi W, Bansal A, Xu B. Applications of artificial intelligence and machine learning approaches in echocardiography. Echocardiography. 2021;38:982–92.
Gonzalez FA, Varudo R, Leote J, et al. The automation of sub-aortic velocity time integral measurements by transthoracic echocardiography: clinical evaluation of an artificial intelligence-enabled tool in critically ill patients. Br J Anaesth. 2022;129:e116–9. https://doi.org/10.1016/j.bja.2022.07.037.
Asch FM, Poilvert N, Abraham T, et al. Automated echocardiographic quantification of left ventricular ejection fraction without volume measurements using a machine learning algorithm mimicking a human expert. Circ Cardiovasc Imaging. 2019;12:e009303.
Schneider M, Bartko P, Geller W, et al. A machine-learning algorithm supports ultrasound-naïve novices in the acquisition of diagnostic echocardiography loops and provides accurate estimation of LVEF. Int J Cardiovasc Imaging. 2021;37:577–86.
Asch FM, Mor-Avi V, Rubenson D, et al. Deep-learning based automated echocardiographic quantification of left ventricular ejection fraction: a point-of-care solution. Circ Cardiovasc Imaging. 2021;14:e012293.
We thank Hugo Moreira, Francisco d’Orey, Vânia Brito and Marta Najarro for participating in echo evaluations as trainees, and Rui Gomes and Vera Pereira for their support as active members of the EchoCrit Group from Hospital Garcia de Orta.
The authors did not receive any funding for the present study. The ultrasound device used in the study belongs to the ICU department.
Ethics approval and consent to participate
The study has been conducted in accordance with the Declaration of Helsinki principles and was approved by the ethical committee of hospital Garcia de Orta, Almada (# TI 71/2021) on July 27, 2021, and written informed consent was obtained for all patients.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Varudo, R., Gonzalez, F.A., Leote, J. et al. Machine learning for the real-time assessment of left ventricular ejection fraction in critically ill patients: a bedside evaluation by novices and experts in echocardiography. Crit Care 26, 386 (2022). https://doi.org/10.1186/s13054-022-04269-6