FibriCheck Compared With FDA-Approved heart rate Devices
Smartphone apps are capable of detecting heart rate. However validation of the accuracy is key for clinical implementation. Therefore in this study a comparison is made between FibriCheck and 2 existing, but fundamental different technologies Alivecor and a Nonin PulseOx meter.
Two FDA-cleared HR measurement devices were used, that is, Nonin oximeter and AliveCor. These 2 devices, which employ different measurement methods, were used to validate a novel smartphone app that measured the participant's heart rate (HR) based on the PPG principle. Nonin uses the transmission PPG method as a stand-alone device, whereas AliveCor uses the ECG as a method measured with a smartphone. The participant's HR was measured 3 times with each measuring device according to the protocol of Terbizan et al. Both FibriCheck and AliveCor were installed on an iPhone 5 (Apple Inc). Participants were recruited in the tertiary care center Ziekenhuis Oost-Limburg (ZOL, Genk, Belgium) in 2015. Inclusion criteria were 18 years or older and able to provide the Dutch written informed consent. Exclusion criteria were failure to obtain valid data with any device or failure to correctly follow the protocol.
A normalization period of 10 min before the first measurement was used to obtain a resting HR. For standardization, all measurements were performed in the same order, that is, FibriCheck app, Nonin oximeter, and AliveCor. The FibriCheck app measures the HR for 10 s by placing the index finger over the rear camera and LED while holding the smartphone in the other hand. Nonin and AliveCor measurements were performed according to the manufacturers’ guidelines.
Figure 1 represents a graphical overview of the step-by-step approach. In case of the FibriCheck app, the shown HR result value in beats per minutes (BPM) was used, whereas for both the Nonin oximeter and AliveCor app, the minimum and maximum HR during a 10 s measurement were averaged. Subsequently, all results of HRs measured by the different devices were statistically compared with each other.
Figure 1. Graphical overview measurement-process
The Shapiro-Wilk test was performed to test for normality. Different tests were performed to analyze the results. First, a Pearson correlation test of each possible pair of methods was performed to assess correlation. Second, the agreement between methods was assessed by the construction of Bland-Altman plots of the same pairs. Finally, a paired student t test and single-way analysis of variance (ANOVA) test were executed to see whether there was a significant difference between the HR as measured by the different methods. Statistical analysis and generation of Bland-Altman plots were performed by using R statistical software (version 3.2.2).
In total, 91 persons were included in the study. A total of 3 persons were excluded from analysis because of failure to obtain valid data with 1 or more devices. This resulted in a final study population of 88 subjects. Table 1 shows the characteristics of these patients. Data are expressed as mean (standard deviation [SD]).
Table 1. Characteristics of patients in study 1.
|Age in years, mean (SD)
|Height, mean (SD)
|Weight, mean (SD)
Table 2 gives an overview of the measured HRs. The average HR is 69 BPM for the Nonin, 71 BPM for the FibriCheck app, and 69 BPM for the AliveCor. Data are expressed as mean (SD).
The HR measurements as acquired by the three different methods were compared for assessing the ability of the FibriCheck app to correctly measure subjects’ HR. First, two-sided Pearson correlation tests were performed to evaluate the correlation between each possible pair of devices. Second, a paired student t test was performed. Thereafter, the RMSE and NRMSE were calculated (Table 3).
Table 2. Heart rates from Nonin, FibriCheck, and AliveCor.
||Heart rate, mean (SDa)
aSD = standard deviation
bSD: standard deviation.
Table 3. Correlation coefficients, statistical significance, root mean square error, and normalized root mean square error for each pair of devices.
|Pair of devices
||Correlation coefficient (r)
|Root mean square error (beats per minute)
||Normalized root mean square error (beats per minute)
aDemographics of 4 patients were reported as missing data.
bSD: standard deviation.
cSystolic and diastolic blood pressure were not included for patients who underwent the sport session.
dCHA2DS2-VASc calculates the stroke risk for patients with atrial fibrillation.
Finally, an ANOVA test was performed to evaluate whether there was a significant difference between the results of the HR measurements of the different devices. The results indicate no significant difference (P=.61) between the HRs measured by the 3 different devices.
Results show high correlations without significant differences for all device pairs. However, since correlation does not necessarily imply agreement, Bland-Altman plots were constructed to evaluate agreement between each pair of devices (Figure 2).
The mean bias ranged from 0.29 bpm (Nonin–AliveCor) to 1.42 bpm (FibriCheck–AliveCor) and 1.72 bpm (FibriCheck–Nonin). Some measurements were not situated between the lower limit of agreement (LLA) and the upper limit of agreement (ULA).
Figure 2. Bland-Altman plots for each device pair. The mean difference (bias), 1.96 (lower limit of agreement, LLA) and +1.96 standard deviations (upper limit of agreement, ULA) are plotted as full lines.
We sought to determine an approach to validate an HR-measuring app. For this experiment, we set up two different studies for determining the correct approach to answer the research question. The results were interpreted on the criterion validity (demonstrated by statistical test for a high correlation between new tool and the existing standard) and construct validity (refers to the systematic change in results when the input variable is under varying conditions) as described by Franko et al.
Study 1, FibriCheck compared with FDA approved HR devices, compared 3 tools for measuring HRs in a large sample of volunteers. The tools (Nonin and AliveCor) are approved by the FDA and are already used in clinical practice. The third one is the FibriCheck app. The results of the study, for criterion validity, show a correlation coefficient of .834 between FibriCheck and Nonin, .88 between FibriCheck and AliveCor, and .897 between Nonin and AliveCor. A single way ANOVA, P=.61 was executed to construct validity indicating that there is no significant difference between the HRs as measured by the 3 devices.
Study 2, FibriCheck beat-to-beat accuracy compared with wearable ECG, compared the RRI-PPI intervals at the same moment from the FibriCheck app in relation to the data of a wearable ECG. The results of the study show a positive correlation of .993 between RRIs and PPIs. This result supports the validity criteria. For construct validity, no significant difference (P=.92) was shown between the intervals from FibriCheck and the intervals from the wearable ECG.
Terbizan et al suggested a minimum correlation of .9 for heart monitors to be clinically reliable. On the basis of the measured results in study 1, no pair of devices complies with this correlation. Terbizan et al suggest to interpret the device as “not reliable.” This is contradictory because both AliveCor and Nonin have an FDA approval. Bland-Altman plots showed some outliers between the devices. In this study, outliers need to be included in the dataset because of the legitimate character of the observation.
A “not reliable” correlation could have multiple causes. For example, there are device-related (eg, different hardware) causes that could influence the signal of the measurement. Furthermore, algorithms converting the PPG signal into HR measurements differ between manufacturers, including in the way they cope with nonperfect measurements. Therefore, when the captured PPG signal is incomplete, for example, because of vibrations or movement by the finger, resulting HR measurements can differ between HR apps and monitors, even when the raw data are identical.
These differing results can be assessed by running the algorithms on a reference database such as the MIT-BIH arrhythmia database for ECG records.
In addition, it is important to consider device specifications when evaluating an HR app on the smartphone. The app therefore needs to be validated on a smartphone with minimal device requirements. Smartphones with lower system specifications than required could result in “‘not reliable” results of the app. It is important for the manufacturer of that HR app to ensure the minimal hardware requirements of hardware. This creates the obligation for manufacturers to evaluate apps on multiple smartphones.
Besides possible hardware and algorithm explanations, there could be time-related causes (eg, measurement on different time) that could result in physiological changes causing a change in HR. This could be eliminated by doing synchronous measurements with these devices.
Another explanation could be that taking average of the minimum and maximum HR during a 10-s interval is not the optimal procedure to obtain a reading from these devices. The FibriCheck app gives a single result after a 10-s measurement, whereas the Nonin oximeter gives a continuous reading and the AliveCor a minimum and maximum HR result after 10 s. To address this mismatch, the average of the minimum and maximum HR of a 10-s reading was used in case of the Nonin oximeter and AliveCor.
To further investigate the correlation between FibriCheck and existing technologies a more fundamental approach of comparison is carried out by have a beat-to-beat comparison between the PPG derived from FibriCheck and the simultaneous ECG derived from a wearable sensor. Read more in our next post!
Techcrunch. Smartphone users globally by 2020, overtaking basic fixed phone subscriptions 2015 URL: https://techcrunch. com/2015/06/02/6-1b-smartphone-users-globally-by-2020-overtaking-basic-fixed-phone-subscriptions/ [WebCite Cache ID 6si3uHw3X]
Pelegris P, Banitsas K, Orbach T, Marias K. A novel method to detect heart beat rate using a mobile phone. 2010 Presented at: Annual International Conference of the IEEE Engineering in Medicine and Biology Society; August 31, 2010; United States. [doi: 10.1109/IEMBS.2010.5626580]
Kumar S, Nilsen WJ, Abernethy A, Atienza A, Patrick K, Pavel M, et al. Mobile health technology evaluation: the mHealth evidence workshop. Am J Prev Med 2013 Aug;45(2):228-236 [FREE Full text] [doi: 10.1016/j.amepre.2013.03.017] [Medline: 23867031]
Kroll RR, Boyd JG, Maslove DM. Accuracy of a wrist-worn wearable device for monitoring heart rates in hospital inpatients: a prospective observational study. J Med Internet Res 2016 Sep 20;18(9):e253 [FREE Full text] [doi: 10.2196/jmir.6025] [Medline: 27651304]
Gregoski MJ, Mueller M, Vertegel A, Shaporev A, Jackson BB, Frenzel RM, et al. Development and validation of a smartphone heart rate acquisition application for health promotion and wellness telehealth applications. Int J Telemed Appl 2012;2012:696324 [FREE Full text] [doi: 10.1155/2012/696324] [Medline: 22272197]
Politi MT, Ghigo A, Fernández JM, Khelifa I, Gaudric J, Fullana JM, et al. The dicrotic notch analyzed by a numerical model. Comput Biol Med 2016 May 01;72:54-64. [doi: 10.1016/j.compbiomed.2016.03.005] [Medline: 27016670]
Friberg L, Engdahl J, Frykman V, Svennberg E, Levin L, Rosenqvist M. Population screening of 75- and 76-year-old men and women for silent atrial fibrillation (STROKESTOP). Europace 2013 Jan;15(1):135-140. [doi: 10.1093/europace/eus217] [Medline: 22791299]
Couetil JP. [Studies of mitral valve insufficiency and indications for surgery]. Soins Chir 1987 Apr(74):5-8. [doi:10.1186/1475-925X-12-19] [Medline: 3649882]
Pinheiro N, Couceiro R, Henriques J, Muehlsteff J, Quintal I, Gonçalves L. Can PPG be used for HRV analysis? 2016 Aug Presented at: 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) Cite this publication; August 2016; United States. [doi: 10.1109/EMBC.2016.7591347]
Elgendi M. On the analysis of fingertip photoplethysmogram signals. Curr Cardiol Rev 2012 Feb;8(1):14-25 [FREE Full text] [Medline: 22845812]
Hertzman A. Maxpulsecordurohealth. 1938. The blood supply of various skin areas as estimated by the photoelectric plethysmograph URL: http://www.maxpulsecordurohealth.com/files/skinBloodSupply.pdf [accessed 2017-08-21] [WebCite Cache ID 6ssdbyFBg]
Jonathan E, Leahy M. Investigating a smartphone imaging unit for photoplethysmography. Physiol Meas 2010 Nov;31(11):N79-N83. [doi: 10.1088/0967-3334/31/11/N01] [Medline: 20871134]
Ho C, Fu Y, Lin M, Chan S, Hwang B, Jan S. Smartphone applications (apps) for heart rate measurement in children: comparison with electrocardiography monitor. Pediatr Cardiol 2014 Apr;35(4):726-731. [doi: 10.1007/s00246-013-0844-8] [Medline: 24259012]
Jo E, Lewis K, Directo D, Kim MJ, Dolezal BA. Validation of biofeedback wearables for photoplethysmographic heart rate tracking. J Sports Sci Med 2016 Sep;15(3):540-547 [FREE Full text] [Medline: 27803634]
Giles D, Draper N, Neil W. Validity of the polar V800 heart rate monitor to measure RR intervals at rest. Eur J Appl Physiol 2016 Mar;116(3):563-571 [FREE Full text] [doi: 10.1007/s00421-015-3303-9] [Medline: 26708360]
Gamelin FX, Berthoin S, Bosquet L. Validity of the polar S810 heart rate monitor to measure R-R intervals at rest. Med Sci Sports Exerc 2006 May;38(5):887-893. [doi: 10.1249/01.mss.0000218135.79476.9c] [Medline: 16672842]
Selvaraj N, Jaryal A, Santhosh J, Deepak KK, Anand S. Assessment of heart rate variability derived from finger-tip photoplethysmography as compared to electrocardiography. J Med Eng Technol 2008;32(6):479-484. [doi:10.1080/03091900701781317] [Medline: 18663635]
Torfs T, Smeets CJ, Geng D, Berset T, Van DA, Vandervoort P, et al. Clinical validation of a low-power and wearable ECG patch for long term full-disclosure monitoring. J Electrocardiol 2014;47(6):881-889. [doi:10.1016/j.jelectrocard.2014.08.012] [Medline: 25201418]
Terbizan DJ, Dolezal BA, Albano C. Validity of seven commercially available heart rate monitors. Meas Phys Educ Exerc Sci 2002 Dec;6(4):243-247. [doi: 10.1207/S15327841MPEE0604_3]
Laskowski ER. Mayoclinic. 2016. What's a normal resting heart rate? URL: http://www.mayoclinic.org/healthy-lifestyle/ fitness/expert-answers/heart-rate/faq-20057979 [accessed 2017-08-14] [WebCite Cache ID 6si60rCBl]
Franko. Validate an app: how to design your study and get published. Journal MTM 2012 Jun 01;1(2):1-4. [doi:10.7309/jmtm.9]
Moody GB, Mark RG. The impact of the MIT-BIH arrhythmia database. IEEE Eng Med Biol Mag 2001;20(3):45-50. [doi:10.1109/51.932724] [Medline: 11446209]