Is Your Smartband Smart Enough to Know Who You Are: Continuous Physiological Authentication in The Wild

1 Is Y our Smartband Smart Enough to Kno w Who Y ou Are: Continuous Physiological Authentication in The W ild Deniz Ekiz, Y ekta Said Can, Y a ˘ gmur Ceren Darda ˘ gan and Cem Ersoy Abstract —The use of cloud services that pr ocess privacy- sensitive information such as digital banking, pervasiv e healthcare, smart home applications requires an implicit continuous authentication solution which will make these systems less vulnerable to the spoofing attacks. Physio- logical signals can be used f or continuous authentication due to their personal uniqueness. Ubiquitous wrist-worn wearable devices are equipped with photoplethysmogram sensors which enable to extract heart rate variability (HR V) features. In this study , we show that these devices can be used for continuous physiological authentication, for enhancing the security of the cloud, edge services, and IoT devices. A system that is suitable for the smartband frame- work comes with new challenges such as relatively lo w signal quality and artifacts due to placement which were not encountered in full lead electrocardiogram systems. After the artifact removal, cleaned ph ysiological signals are fed to the machine learning algorithms. In order to train our machine learning models, we collected physiological data using off-the-shelf smartbands and smartwatches in a real-life event. Perf ormance evaluation of selected machine learning algorithms shows that HR V is a strong candidate f or continuous unobtrusive implicit physiological authentication. Index T erms —smartband, smartwatch, heart rate vari- ability , continuous authentication I . I N T R O D U C T I O N I MPLICIT continuous authentication is required for cloud oriented services which grant access to the priv acy sensiti ve information domains such as mobile banking, pervasi ve healthcare [1], [2]. Smartphones, computers, smartwatches and Internet-of-Things (IoT) devices become more dependent on these services. It is expected that the number of IoT devices will be more than 75 Billion in 2025 [3]. Howe ver , these ser- vices are vulnerable to attacks once users authenticate. For example, a smartphone can be forgotten logged- in, the priv acy-sensiti ve services and information can Deniz Ekiz, Y ekta Said Can, Y a ˘ gmur Ceren Darda ˘ gan and Cem Ersoy were with the Department of Computer Engineering, Bo ˘ gazic ¸ i Univ ersity , Turk ey . E-mail: deniz.ekiz@boun.edu.tr Manuscript received XXX, XXX be stolen by the attackers. One simple mechanism can be asking a password to the user frequently . Howe ver , this is annoying for the service users. Continuous au- thentication should be implicit. Face-based systems can be tricked by using presence attacks, such as printing the face of the victim on a paper . Storing the face pictures of the users also create a priv acy concern [4]. Furthermore, fingerprint which is another prominent traditional biometrics modality along with the face-based systems can be easily manipulated [5] and fails on liv eness detection tests. On the other hand, biosignals are difficult to temper with and they inherently hav e liv eness detection feature [6]. Heart activity is unique to individuals and biosignal authentication research has started in vestigating this signal [6]. One of the most important properties of this signal is the Heart Rate V ariability (HR V). Although research in the past mostly focused on the connection between HR V and different types of health disorders [7], the validity of using HR V for biometric recognition is supported by the fact that the physiological and geometrical differences of the heart in different individuals display certain uniqueness in their HR V features [8]. High-end wearable systems are expensi ve and provide low comfort for the users, which limit their wide range application. Recently , smart bands and smartwatches became widely adopted by consumers. These devices are equipped with a rich set of sensors such as accelerometer, heart rate monitor and skin conductance. These advances create an opportunity to build a continuous implicit au- thentication system. Howe ver , these devices are prune to activity related errors [9] unlike full lead ECG systems. Modality specific artifact detection and remov al mecha- nisms should be developed for accurate measurements. A solution suitable for IoT connected devices should be context-independent because every service may require different types of behavior . Therefore, systems that only work in certain scenarios such as while typing or walk- ing are very limited in terms of the application area. The physiological parameters like hidden heart-related 2 biometrics are more suitable for this purpose due to their uniqueness and activity independence [10], [11]. W e propose an unobtrusiv e, low cost, activity-independent continuous authentication system with smartbands. W e implemented our solution on Empatica E4 Smartband [12], Samsung Gear S and Gear S2 [13] which are equipped with a photoplethysmography (PPG) sensor . Let’ s think about a scenario, where John has to make money transactions through a mobile banking smart- phone application. He logins to the system via two-f actor authentication. After he successfully makes the transac- tion, he forgets to logout. Then, an attacker withdraws all the cash from John’ s account by using his session. If John has a continuous biometric authentication system, he would not hav e this problem. Such a system can be also used in services like city bikes and electric scooter rental services which have become popular in recent years. The literature on continuous authentication with physiological signals gathered from smartbands is limited. The effecti veness of HR V features derived from PPG sensors of smartwatches and smartbands is still unknown for continuous authentication [4]. Our proposed system enhances the state of the art as follows: • One of the previous studies employed mean heart rate per minute which requires longer recordings and achie ves approximately five minutes [14]. The proposed system uses more sophisticated HR V fea- tures deri ved from inter-beat intervals which are extracted from the raw PPG sensor data. • Most of the previous works with smartbands only focused on continuous authentication while using a laptop, a smartphone, an A TM and used lifestyle re- lated metrics. Our solution is not acti vity dependent. For example, a user may use this system during any activity such as live streaming, on social media, cycling, presentation or working in an office. • Our solution is comfortable, unobtrusiv e, seamless and works without interrupting the ordinary pattern of any activity . • Real-life data contains artifacts. W e ev aluate the effect of data quality on system performance. In Section II we provide background information on the smartwatch framew ork and heart rate variability . In Section III, the related work on continuous authenti- cation and the comparison with our work in terms of its novelty are presented. In Section IV, we describe the proposed system for continuous authentication with smartwatches using heart rate variability . In Section V, we explain the conducted experiment for the proposed system. In Section VI, we provide the results of our system. In Section VII, we present the conclusion and future works of our study . I I . B AC K G RO U N D A. Smartband F rame work Smartbands (some times called wristbands or smart- watches) are comfortable devices that can be attached to the wrist or arm. The de vices used in this study are sho wn in Figure 1. Recently , the battery life of smartbands is increased to days. For example, the battery life of Empatica E4 is two days when all sensors are used [12]. The extended battery life of smartbands enables monitoring physiological signals gathered from individuals for long periods. Most of the wristbands are equipped with PPG which is an optically obtained signal that can be used to detect blood volume changes in the microvascular bed of tissue [15]. From the PPG signal, the time between the beats (RR intervals) can be computed. Most of the modern smartbands provide RR intervals thanks to their APIs. For example, Sam- sung smartwatches use the T izen framework which has Human Activity Monitor API to gather the RR intervals. The sampling frequenc y of the PPG sensor can vary from 20Hz to 100Hz (64Hz for E4, 100Hz for Samsung Gear Series) in many smartbands. Cubic spline interpolation is used to detect the beats more accurately and most of the devices correct the heartbeats by using an accelerometer sensor for detecting the movements of the subject. This functionality is av ailable in Tizen and Empatica API [12]. These devices are also equipped with Bluetooth and recently NFC chips which enable them to connect to smartphones, edge and cloud services. These short- range network interfaces can be used to check if the user is in the close proximity of the computer they are using. Fig. 1. Empatica E4, Samsung Gear S2 and Samsung Gear S are shown in the order . 3 B. Heart Rate V ariability The variability of RR intervals is called HR V [16]. It is a very important feature for recognition of certain psychological, physiological and personal properties of an individual [16]. In the literature, Kubios [17] is a popular HR V feature extraction tool to compute the HR V . Non-linear , time domain and frequency domain features can define the variability of the heart [17]. The calculation of frequency domain and time domain features of HR V is computationally effecti ve, thanks to Fast Fourier T ransform (FFT) O ( n log n ) for frequency domain and O ( n ) for time-domain features. I I I . R E L AT E D W O R K The behavioral and physiological biometrics from the wearable devices via sensors hav e become popular in individual recognition and authentication models. Some models focus on biometrics such as face [25], voice [26], fingerprints [27], [28], electroencephalography (EEG) [29], [30], [31], ECG [32], [33], [28] and phonocardiog- raphy (PCG), [34]. The continuous authentication field is a fast-gro wing field, howe ver , the literature on a system that is not dependent on a certain type of task is limited. Most of the previous work using physiological signals hav e been done on laboratory-grade equipment. Some of these sensors are not av ailable in unobtrusiv e devices such as smartphones, smartwatches, and smart bands. On the other hand, fingerprints and face-based authentication systems can be easily decei ved. Authentication with voice has pri vac y issues, which requires continuous v oice recording of the environment. As a method for recognizing indi viduals, Elkader et al. [18] presented a sensor-based motion biometric model that is suitable for 20 sedentary and non-sedentary activities (V acuuming, Sweeping, W alking Downstairs, W alking Upstairs, Dusting, Iron Cloth, Folding Cloth, W ashing Hands, Brushing T eeth, W ashing Dishes, W ash- ing V egetables, Dicing, Peeling V egetables, Grating, Stirring, W atching TV , Using PC, T alking on Phone T exting on Phone, Writing with Pen). They used different combinations of 3 sensors (acceleration, gyroscope and magnetometer sensors) on 6 different body positions (dominant wrist, dominant upper arm, non-dominant wrist, chest, thigh, and ankle positions). They concluded that features e xtracted from the combination of six sensors reach the best classification accuracy in ov erall (98.3%). These activities are gathered in a laboratory en vironment with manual segmentation of the signals. Another approach for implicit identification and au- thentication based on acti vity information, W earAI, Zeng et al. [19], proposed a biometric model that utilizes ac- celerometer and gyroscope sensors from fiv e body loca- tions such as left wrist (Shimmer 6DoF IMU), right ankle (Shimmer 6DoF IMU), center right hip/torso (Samsung Galaxy S4 i9500), left thigh/front pocket (Samsung Galaxy Nexus i9250), right upper arm (Samsung Galaxy Nexus i9250)). They achieved 97% accuracy with less than 1% false-positiv e rate. Howe ver , in both methods, placing many sensors on the body can be disturbing for the user in daily life usage. Acar et al. [35] used smartwatches with keystroke dynamics for continuous authentication. Musale et al. [23] proposed a continuous authentication system based on Motorola 360 Sport by using accelerometer and gyro- scope features. Vhaduri et al. [20] proposed continuous user authentication scheme that uses 44 features ex- tracted from various biometrics (calorie burn, metabolic equiv alent of task (MET), heart rate and step count) using Fitbit Char ge HR de vice and they achie ved a verage accuracy of 87.37% with Quadratic SVM classifier in one-to-many approach and av erage accuracy of 93% with Quadratic SVM classifier in one-to-one approach. In their revised scheme [14], they adopted more features (65) with dif ferent feature selection approaches and 93% (sedentary) and 90% (non-sedentary) with equal error rates of 5% is obtained. Howe ver , the Fitbit framew ork only provides only one sample each minute and access to the raw data is not possible. A system for continuous authentication with physiological signals should be lo w- cost and unobtrusiv e, and should not be dependent on certain activity for the sake of universality . W e com- pared the proposed system with the related work in T able I in terms of device and device position, features, unobtrusiv eness, en vironment, and dependency to the activity type. Our system outperforms other studies when feature engineering complexity , activity independence and unobtrusiveness are taken into consideration. Fig. 2. The system diagram of our proposed solution. I V . P RO P O S E D S Y S T E M In this section, we explain our continuous authentica- tion scheme. In Figure 2, we show the data collection application, preprocessing for artifact detection, feature extraction, feature selection and classification units of our system. The overall multi-factor authentication di- 4 T ABLE I P R EV I O U S W O R KS O N C O N TI N U O US AU TH E N T IC ATI O N . Article Y ear Device(s) Device Position(s) Featur es Unobtrusive En vironment Activity Independent S. A. Elkader et al. [18] 2018 A combination of acceleration, gyroscope and magnetometer sensors Dominant wrist, dominant upper arm, non- dominant wrist,thigh, chest, ankle Accelerometers, gyroscope and magnetometers features extracted from six sensor-based body locations No Laboratory Kitchen area No Y . Zeng et al. [19] 2017 Shimmer 6DoF IMU Samsung Galaxy S4 i9500 Samsung Galaxy Nexus i9500 Wrist,ankle, hip/torso, thigh/front pocket, upper arm Statistical features extracted from accelerometer sensor Y es Laboratory No S. Vhaduri et al. [20] 2017 Fitbit Charge HR device Wrist Statistical features extracted from step counts, heart rate, calorie burn, metabolic equivalent of task information Y es Daily Life No S. Vhaduri et al. [14] 2019 Fitbit Charge HR device Wrist Statistical features extracted from step counts, heart rate, calorie burn, metabolic equivalent of task information Y es Daily Life No Matsuyama et al. [21] 2015 Near-infrared Spectroscopy (NIRS) Fore Head Low-frequenc y brainwav es No Laboratory No Sarkar et al. [22] 2016 Front facing camera of mobile device In front of face Deep Face Features Y es Daily Life No Ramli et al. [8] 2016 Hearbeat detection kit as a wearable bracelet Wrist ECG W avelet Features No Laboratory No Ntantogian et al. [2] 2015 Camera In front of a person Gait and gesture features Y es Laboratory No Musale et al. [23] 2019 Motorola 360 Sport 2nd Gen (smartwatch) Motorola G4 plus (smartphone) Wrist Statistical and human-action-related features from accelerometer and gyroscope sensor Y es Daily Life No Peng et al. [24] 2017 Google Glass Head T ouch gestures (single-tap, swipe forward, swipe backward, swipe down, two-finger , swipe forward, and two-finger swipe backward) and voice commands No Laboratory No Our W ork 2019 Samsung Gear S Samsung Gear S2 Empatica E4 Wrist Heart Rate V ariability features derived from PPG sensor Y es Real Life Y es agram where a user initiates his/her session with a password, or fingerprint, is shown in Figure 3. A. Data Collection Application W e de veloped a data collection application in T izen 3.0 framework [36] for Samsung Gear S and S2. The ap- plication collects inter-beat intervals and 3D accelerom- eter data and stores them as do wnloadable comma- separated values (CSV files). Empatica E4 has a cloud based data collection application. The physiological sig- nals can be downloaded as a CSV file. The gathered RR intervals from two different participants are shown in Figure 4. B. Pr epr ocessing and Artifact Removal W e implemented our preprocessing module in MA T - LAB [37]. First, we loaded the CSV file provided from the smartbands. The signal is segmented into non- ov erlapping time windows of 120 seconds. According to 5 the HR V guidelines, 2 minutes is the minimum window length for calculation of short-term HR V features [16]. Since response time is important for a continuous au- thentication system, we selected the minimum possible duration. Therefore, the minimum required duration of physiological data for authentication is 2 minutes. The artifacts in the RR intervals are detected by checking the difference between the consecutive points. W e labeled the points exceeding more than 20% of the local average as artifacts, and the other points as the validated RR in- tervals, this threshold is selected from the previous works [38]. The points labeled as artifacts are deleted. After the remov al, we implemented two different techniques. The first one is to interpolate the missing data points using a cubic spline interpolation algorithm which is commonly used [17]. The second technique is to apply the minimum consecutiv e time and sample constraints on the remaining data to be regarded as meaningful. For example, if the minimum sample constraint is set to 5, we do not count three consecuti ve samples followed by a missing data point because the sequence is too short to be ev aluated. In this study , we applied the former technique because it achieved better results [9]. C. F eature Extraction W e extracted time and frequency domain heart rate variability features from the segmented time windows. W e used Marcus V olmer’ s toolbox [39] which is imple- mented in MA TLAB. W e selected the features which are commonly used in the previous works related to heart rate variability [9], [38] and [40]. In order to compute the frequency domain features, the RR intervals are interpolated using 4Hz cubic spline interpolation, because RR intervals are unev enly sampled. W e applied FFT to the interpolated windows. The computed features are shown in T able II. The total number of extracted features is 11 for each window . D. Dimensionality Reduction It is known that dimensionality reduction leads to better performance for the machine learning systems Fig. 3. Our proposed solution continuously authenticate the user by processing the RR intervals coming from the smartband. T ABLE II H E ART R A T E V A R IA B I L IT Y F E A T U R ES A N D TH E I R D EFI N I TI O N S . HR V Feature Description Mean RR Mean value of the RR intervals STD RR Standard deviation of the RR intervals RMSSD Root mean square of succes- siv e difference of the RR in- tervals pNN50 Percentage of the number of successiv e RR intervals vary- ing more than 50ms from the previous interval HR V triangular index T otal number of RR intervals divided by the height of the histogram of all RR intervals measured on a scale with bins of 1/128 s TINN T riangular interpolation of RR interval histogram SDSD Related standard deviation of successiv e RR interval differ- ences LF Power in low-frequency band (0.04-0.15 Hz) HF Power in high-frequency band (0.15-0.4 Hz) LF/HF Ratio of LF-to-HF VLF Power in very low-frequency band (0.00-0.04 Hz) since it removes the unrelated features with the desired prediction [41]. W e used Principal Component Analysis (PCA) based dimensionality reduction which is av ailable in the W eka toolkit [42]. PCA is a very powerful tool when applied with machine learning (ML) classifiers. It con verts the set of vectors to uncorrelated v ariables. W e explored the effect of different selection of cov ered Fig. 4. The RR interv als gathered from two participants using Samsung Gear S. The difference can be seen by looking at the raw signals. 6 Fig. 5. The MLP model used in the proposed system. variance and PCA v ariables. It is known that the covered variance affects the classification performance, therefore different values are ev aluated (0.8, 0.85, 0.9 and 0.95) we reported the best results when the variance is selected as 0.9. E. Handling Class Imbalance Since some of the windows are deleted due to im- proper placement of the devices or heavy movements. There is a class imbalance between participants. W e applied the majority class subsampling to equalize the number of windows for each participant. This method is the most commonly used one in the literature [43]. F . Machine Learning W e used k-Nearest Neighbour (kNN), Random Forest (RF), Multi-Layer Perceptron (MLP) and Linear Dis- criminant Analysis (LDA) classifiers which are av ailable in the W eka Machine Learning software [42]. W e fine tuned the parameters for different classifiers. The best performing feature set are as follows: N selected as 3 for the kNN, the number of trees is selected as 100 for the random forest and the hidden layer is selected as 1 and hidden unit as 5 for the MLP as shown in Figure 5. W e created a binary authentication model for each user . The selected user’ s label is set to 1 and others as 0. W e applied 10 fold stratified cross-validation (the distribution of class labels are equal in each fold) for ev aluating our system and fine tuned the parameters where 90% of the dataset is used for training and the rest is used for testing by changing the folds. G. Evaluation Metrics In order to present the results of our authentication system, we provide the performance metrics used in the literature [8], [14], [18], [19], [34] . In authentication systems, there are two types of error which are False Acceptance Rate (F AR) and False Rejection Rate (FRR). These errors are depend on selection of the threshold which can be between 0 and 1 for the ML classifiers. A smaller value will cause a low F AR but high FRR. The point of equilibrium is important for such a system. This point is called Equal Error Rate (EER). The definitions are provided below: • False Acceptance Rate (F AR): It is the ratio of false acceptance divided by the total attempts. • False Rejection Rate (FRR): It is the ratio of denied legitimate attempts to the total number of attempts. • Equal Error Rate (EER): The common value when FRR and F AR are equal, is called EER [44]. V . D AT A C O L L E C T I O N In this section, we describe the data collection in real life and the ethics procedure. W e collected physiological data from 28 people in controlled real-life settings, during a summer school for teachers. All of the participants are healthy teachers who have no prior medical condition. Before the data collection, subjects receiv ed and filled a consent. The gender of participants are 16 male and 12 female, the ages are between 25 and 35. The data collection procedure is sho wn in Figure 6. The duration of the total data collection is 110 minutes. The dataset has a baseline (20 minutes), lecture (40 minutes), free-time (10 minutes), examination (20 minutes) and recov ery session (20 minutes). W e did not use the free-time session which might create a bias on the results. The reason that we had these different scenarios is to create a daily life sequence. A system should take different states into consideration, because HR V can be affected by valance and arousal. During the free-time participants were allowed to take a break from the lecture. W e applied our implementation of T rier Social Stress T est [45] (TSST) which is frequently used for inducing stress. W e selected questions from the mathematics Olympics (which is very hard for the normal population). W e told the subjects that this is a test for measuring their intelligence, and we said that a moderate person achieves at least a 75% score. Subjects participated in ev ery session and they did not know the objectiv e of the study . The physiological data is gathered with different brands of commercial smartwatches (8 Empatica E4, 3 Samsung Gear S and 17 Samsung Gear S2). Ethics The procedure of the methodology used in this study is approv ed by the Institutional Revie w Board for Research with Human Subjects of Bogazic ¸ i University with the approv al number 2018/16. Prior to the data acquisition, 7 Fig. 6. Data collection procedure for the physiological continuous authentication system. each participant recei ved a consent form which explains the experimental procedure and its benefits and implica- tions to both the society and the subject. The procedure was also explained vocally to the subject. The data collection procedure and all of the interventions in this research fully meet the 1964 Declaration of Helsinki [46]. The data is stored anonymously . V I . E X P E R I M E N TA L R E S U LT S A N D D I S C U S S I O N W e examined the results in two different subsections. In the first one, we presented and ev aluated the authenti- cation results of different devices and the whole system performance. In the second part, by applying a signal data quality filter, we improved the performance of the system. A. Effect of Device T ype on the Biometric Authentication P erformance EER results for all 28 subjects are gi ven in T able III. These results are calculated by one vs. all tests for all subjects. A verage EER results for four different classifiers are presented in T able IV. W e also added the device type and average data quality columns to this table. Data quality presents the non-interpolated percentage of the data after the removal of artifacts. As an example, if the av erage data quality is 70%, the remaining 30% of the data is interpolated. Data quality along with the de vice type affects the EER results significantly (see Figure 7). W e achie ved the best performance with Gear S as 98.48% and 3.96% EER. The selection of classifier has also an important effect on the EER results. For example, Empatica E4 achieves 19.43% EER with kNN and 6.77% with RF classifier . The best classifier is selected as RF in terms of EER. Design of the watch strap as sho wn in Figure 1, PPG sensor quality , built-in processing algorithms of devices might be the factors for the difference in EER results. B. Effect of Data Quality Constraint F ilter In daily life, seamless wrist-worn de vices can get noisy signals, which drops the quality of the deriv ed features. It is not possible to collect high-quality data all the time during a day because of various reasons such as high acti vity lev el and improper use of smartwatches. T ABLE III T H E E E R M ET R I C S O F E AC H P A RT IC I PAN T W I T H D I FF ER E N T C L AS S I FI ER S . Participant/Classifier kNN RF MLP LD A Device A verage Quality P1 20.00 1.67 3.30 5.00 E4 96.00 P2 22.84 5.00 10.00 8.73 E4 94.00 P3 5.68 3.23 5.65 9.95 E4 96.00 P4 31.30 12.52 15.52 15.52 E4 93.00 P5 30.17 9.45 27.27 27.27 E4 91.50 P6 23.96 8.30 15.87 19.05 E4 97.00 P7 17.96 11.03 22.22 25.04 E4 92.00 P8 3.58 2.99 4.48 5.97 E4 93.00 P9 4.00 5.33 5.33 6.67 Gear S 87.00 P10 1.33 2.67 4.00 5.33 Gear S 95.00 P11 9.20 3.39 5.08 9.32 Gear S 80.00 P12 37.47 18.73 23.26 21.13 Gear S2 85.00 P13 30.15 10.41 21.13 22.54 Gear S2 80.00 P14 13.52 3.80 8.45 9.86 Gear S2 62.00 P15 31.35 4.69 12.50 17.19 Gear S2 68.00 P16 18.43 6.78 20.59 20.59 Gear S2 54.00 P17 36.25 18.23 27.78 29.17 Gear S2 64.00 P18 43.34 24.55 43.75 40.63 Gear S2 79.00 P19 21.88 2.82 8.45 11.27 Gear S2 69.00 P20 34.50 11.43 18.75 20.31 Gear S2 66.00 P21 39.30 19.55 21.13 22.54 Gear S2 66.00 P22 25.60 13.41 18.75 18.75 Gear S2 68.00 P23 31.67 14.08 23.94 21.13 Gear S2 62.00 P24 33.10 14.10 23.44 31.25 Gear S2 53.00 P25 20.81 7.04 15.49 19.72 Gear S2 65.00 P26 33.22 16.45 24.23 23.44 Gear S2 69.00 P27 37.48 20.54 28.17 29.58 Gear S2 76.00 P28 36.63 19.18 23.44 26.56 Gear S2 64.00 T ABLE IV A U T HE N T I CATI O N P E RF O R MA N C E R ES U LTS , E E R V A L UE S O F E M P AT IC A E 4 , S A M S UN G G E AR S A N D S AM S U N G G E A R S2 A R E P R ES E N T ED . Device kNN RF MLP LD A A verage Quality Empatica E4 19.43% 6.77% 13.03% 14.56% 94.06% Gear S 4.84% 3.96% 4.80% 7.10% 87.33% Gear S2 30.86% 13.28% 21.36% 22.68% 67.64% All Devices 24.37 % 10.08% 16.98% 18.69 % 77.30% 8 Fig. 7. The change of EER metric with respect to minimum quality thresholds. Best achieved EER is 3.96%. After observing that the data quality has a major ef fect on the authentication performance, we applied a data quality constraint on our data. Suppose that the data quality of a device is 50%. This means that the other half of the data is obtained by synthetic cubic interpolation data. Therefore, we expect that when the data is compared with other participants’ data, it could not be discrimi- nated, because it lost most of the unique characteristics of the PPG data. In Figure 7, we ev aluated the effect of a quality threshold on EER. As we in vestigate the EER results of different device types, Samsung Gear S giv es the smallest error rate 3.796% when compared to Empatica E4 (6.77%) and Samsung Gear S2 (13.66%) in low data qualities. As the quality increases, while the error rates of Samsung Gear S (2.67%) and Empatica E4 (4.4%) decrease at 95% quality threshold, Samsung Gear S2 is unable to show the same progress and ev entually reaches 18.557% equal error rate. None of the windows of Samsung Gear S2 has a higher than 95% quality . The performance ev aluation shows that the proposed system can effecti vely authenticate with small and consistent error rate which makes it reliable. V I I . C O N C L U S I O N W e proposed a scalable, unobtrusiv e and seamless continuous authentication system with commercial grade smartwatches and smartbands. W e collected physiolog- ical data from 28 participants and demonstrated the EER measures for each of the participants in a real- life scenario. W e proposed state-of-the art preprocessing for signals coming from real-life data with artifacts due to the physical construction of the smartwatches. W e achiev ed promising results by using our system (4.4% EER with Empatica E4). W e sho wed the ef fect of different smartwatches. The selection of the classifier for the proposed system is very important. W e applied feature based signal processing along with machine learning classifiers (kNN, RF , MLP and LDA). Even- though, Gear S2 is a newer model of Gear S, due to its leather strap, the signals coming from the heart rate monitoring unit contained higher amount of arti- facts, therefore it af fects the ov erall quality of the RR intervals and the authentication system’ s performance. For the authentication systems based on PPG sensors, sport straps can be a better choice, as shown in Figure 1. W e showed that HR V can be used for continuous authentication without interrupting the activity of the user . W e applied a signal remov al procedure by using the ov erall RR interv al quality measure, a higher quality leads to better performance after 80% quality threshold. The performance of the scheme v aries between indi- viduals. This conclusion is aligned with the literature [8], [14]. The minimum required amount of recording to apply our system for authentication is 2 minutes, once that is satisfied, authentication can be validated in seconds thanks to the sliding window approach. It logouts the user , once he/she leav es off the watch. Our system can be implemented on any wrist-worn device which can provide RR intervals without a need for the raw PPG. The proposed methodology can be used with various applications requiring continuous authentication. This study also has some limitations. The performance of the system on the data coming from different days is still unknown. As future works, we plan to apply our system completely in the wild settings with more partic- ipants and longer physiological recordings and show the performance of the framework. All of the ev aluations are done in the same context, therefore in different types of contexts, the system might achie ve better performance. R E F E R E N C E S [1] A. C. Baktir, C. Tunca, A. Ozgovde, G. Salur, and C. Er- soy , “SDN-Based Multi-Tier Computing and Communication Architecture for Pervasiv e Healthcare, ” IEEE Access , vol. 6, pp. 56765–56781, 2018. [2] C. Ntantogian, S. Malliaros, and C. Xenakis, “Gaithashing: A two-factor authentication scheme based on gait features, ” Com- puters & Security , vol. 52, 04 2015. [3] Statista Research Department, “Internet of Things (IoT) con- nected devices installed base worldwide from 2015 to 2025 (in billions), ” 2016. [4] S. W . Shah and S. S. Kanhere, “Recent trends in user authen- tication – a survey , ” IEEE Access , vol. 7, pp. 112505–112519, 2019. [5] A. Ross, K. Nandakumar, and A. K. Jain, “Introduction to multi- biometrics, ” in Handbook of biometrics , pp. 271–292, Springer , 2008. [6] N. Akhter, H. Gite, G. Rabbani, and K. Kale, “Heart rate variability for biometric authentication using time-domain fea- tures, ” in International Symposium on Security in Computing and Communication , pp. 168–175, Springer, 2015. [7] N. Akhter, J. F . Mahdi, and G. R. Manza, “Microcontroller based data acquisition system for heart rate variability (hrv) 9 measurement, ” International Journal of Applied Science and Engineering Researc h , vol. 1, no. 4, pp. 576–583, 2012. [8] D. Ramli, M. Hooi, and K. Chee, “Dev elopment of heartbeat de- tection kit for biometric authentication system, ” v ol. 96, pp. 305– 314, 12 2016. [9] Y . S. Can, N. Chalabianloo, D. Ekiz, and C. Ersoy , “Continuous stress detection using wearable sensors in real life: Algorithmic programming contest case study , ” Sensors , vol. 19, p. 1849, Apr. 2019. [10] K. Phua, J. Chen, T . H. Dat, and L. Shue, “Heart sound as a biometric, ” P attern Recognition , vol. 41, no. 3, pp. 906–919, 2008. [11] S. Pirbhulal, H. Zhang, S. Mukhopadhyay , C. Li, Y . W ang, G. Li, W . W u, and Y .-T . Zhang, “ An efficient biometric-based algorithm using heart rate variability for securing body sensor networks, ” Sensors , vol. 15, no. 7, pp. 15067–15089, 2015. [12] Empatica , 2018. accessed at December 2019. [13] Samsung , 2019. accessed at December 2019. [14] S. Vhaduri and C. Poellabauer, “Multi-modal biometric-based implicit authentication of wearable device users, ” IEEE Tr ansac- tions on Information F or ensics and Security , vol. 14, pp. 3116– 3125, Dec. 2019. [15] Y . Sun and N. Thakor, “Photoplethysmography revisited: From contact to noncontact, from point to imaging, ” IEEE T ransactions on Biomedical Engineering , vol. 63, pp. 463–477, Mar . 2016. [16] M. Malik, J. T . Bigger , A. J. Camm, R. E. Kleiger, A. Malliani, A. J. Moss, and P . J. Schwartz, “Heart rate variability: Standards of measurement, physiological interpretation, and clinical use, ” Eur opean Heart Journal , vol. 17, pp. 354–381, 03 1996. [17] “Kubios user guide, ” 2019. [18] S. A. Elkader, M. Barlow , and E. Lakshika, “W earable sensors for recognizing individuals undertaking daily acti vities, ” in Pr o- ceedings of the 2018 ACM International Symposium on W earable Computers - ISWC , ACM Press, 2018. [19] Y . Zeng, A. Pande, J. Zhu, and P . Mohapatra, “W earIA: W earable device implicit authentication based on activity information, ” in 2017 IEEE 18th International Symposium on A W orld of W ir eless, Mobile and Multimedia Networks (W oWMoM) , IEEE, June 2017. [20] S. Vhaduri and C. Poellabauer , “W earable device user authenti- cation using physiological and behavioral metrics, ” Oct. 2017. [21] Y . Matsuyama, M. Shozaw a, and R. Y okote, “Brain signal’ s lo w- frequency fits the continuous authentication, ” vol. 164, 03 2015. [22] S. Sarkar, V . M. Patel, and R. Chellappa, “Deep feature-based face detection on mobile devices, ” in 2016 IEEE International Confer ence on Identity , Security and Behavior Analysis (ISBA) , pp. 1–8, Feb 2016. [23] P . Musale, D. Baek, N. W erellagama, S. S. W oo, and B. J. Choi, “Y ou walk, we authenticate: Lightweight seamless authentication based on gait in wearable IoT systems, ” vol. 7, pp. 37883–37895, 2019. [24] G. Peng, G. Zhou, D. T . Nguyen, X. Qi, Q. Y ang, and S. W ang, “Continuous authentication with touch behavioral biometrics and voice on wearable glasses, ” IEEE T ransactions on Human- Machine Systems , vol. 47, pp. 404–416, June 2017. [25] M. Ghayoumi, “ A revie w of multimodal biometric systems: Fusion methods and their applications, ” in 2015 IEEE/A CIS 14th International Conference on Computer and Information Science (ICIS) , IEEE, June 2015. [26] R. Brunelli and D. Fala vigna, “Person identification using mul- tiple cues, ” IEEE T ransactions on P attern Analysis and Machine Intelligence , vol. 17, pp. 955–966, Oct. 1995. [27] E. Camlikaya, A. Kholmato v , and B. Y anikoglu, “Multi-biometric templates using fingerprint and voice, ” in Biometric T echnolo gy for Human Identification V (B. V . Kumar , S. Prabhakar , and A. A. Ross, eds.), SPIE, Mar. 2008. [28] M. Hammad, Y . Liu, and K. W ang, “Multimodal biometric authentication systems using convolution neural network based on different level fusion of ECG and fingerprint, ” IEEE Access , vol. 7, pp. 26527–26542, 2019. [29] M. D. Bugdol and A. W . Mitas, “Multimodal biometric system combining ECG and sound signals, ” P attern Recognition Letters , vol. 38, pp. 107–112, Mar . 2014. [30] Q. Gui, Z. Jin, and W . Xu, “Exploring EEG-based biometrics for user identification and authentication, ” in 2014 IEEE Signal Pr ocessing in Medicine and Biology Symposium (SPMB) , IEEE, Dec. 2014. [31] M. Rangoussi, N. Alexandris, A. Evangelou, and M. Poulos, “Person identification from the EEG using nonlinear signal classification, ” Methods of Information in Medicine , vol. 41, no. 01, pp. 64–75, 2002. [32] C. ZHANG, Y .-M. TIAN, and H.-W . W ANG, “Revie w of ECG signal identification research, ” DEStech T ransactions on Com- puter Science and Engineering , Mar . 2017. [33] H.-S. Choi, B. Lee, and S. Y oon, “Biometric authentication using noisy electrocardiograms acquired by mobile sensors, ” IEEE Access , vol. 4, pp. 1266–1273, 2016. [34] A. Sarkar, A. L. Abbott, and Z. Doerzaph, “Biometric authenti- cation using photoplethysmography signals, ” in 2016 IEEE 8th International Conference on Biometrics Theory , Applications and Systems (BT AS) , IEEE, Sept. 2016. [35] A. Acar, H. Aksu, A. S. Uluagac, and K. Akkaya, “W aca: W earable-assisted continuous authentication, ” in 2018 IEEE Se- curity and Privacy W orkshops (SPW) , pp. 264–269, May 2018. [36] Tizen, T izen API , 2015. accessed at December 2018. [37] MA TLAB, version 7.10.0 (R2010a) . Natick, Massachusetts: The MathW orks Inc., 2010. [38] B. Cinaz, B. Arnrich, R. Marca, and G. Tr ¨ oster , “Monitoring of mental workload levels during an everyday life office-work scenario, ” P ersonal Ubiquitous Comput. , vol. 17, pp. 229–239, Feb . 2013. [39] M. V ollmer, MarcusV ollmer/HRV T oolbox , 2017. accessed at December 2019. [40] M. Gjoreski, M. Lu ˇ strek, M. Gams, and H. Gjoreski, “Monitoring stress with a wrist device using context, ” Journal of biomedical informatics , vol. 73, pp. 159–170, 2017. [41] D. Barber, Bayesian Reasoning and Machine Learning . Cam- bridge University Press, 2012. [42] F . Eibe, M. Hall, and I. Witten, “The weka workbench. online appendix for” data mining: Practical machine learning tools and techniques, ” Morgan Kaufmann , 2016. [43] S. Kotsiantis, D. Kanellopoulos, and P . Pintelas, “Handling im- balanced datasets: A revie w , ” GESTS International T ransactions on Computer Science and Engineering , vol. 30, pp. 25–36, 11 2005. [44] S. Mondal, “Performance ev aluation of continuous authentication systems, ” IET Biometrics , 09 2015. [45] C. Kirschbaum, K.-M. Pirke, and D. H. Hellhammer, “The ‘trier social stress test’–a tool for inv estigating psychobiological stress responses in a laboratory setting, ” Neur opsychobiology , vol. 28, no. 1-2, pp. 76–81, 1993. [46] W . M. Association et al. , “W orld medical association declaration of helsinki. ethical principles for medical research in volving human subjects., ” Bulletin of the W orld Health Organization , vol. 79, no. 4, p. 373, 2001. Deniz Ekiz receiv ed the MS degree from Computer Engineering Department, Bogazici Univ ersity , T urkey in 2019. He is a Ph.D. candidate in the Computer Engineering De- partment of Bogazic ¸ i Univ ersity , Turke y . His research is focused on the health-related ap- plications of wearable technology 10 Y ekta Said Can studied Computer Engineer - ing in Bogazici University , Istanbul, T urkey where he obtained his B.Sc Degree in 2012. He obtained his M.Sc degree in the same department, in 2014 while working as a researcher at TUBIT AK BILGEM for two years. He is pursuing a PhD degree right now in Computer Engineering at Bogazici Univ ersity . His research interest includes wa- termarking, speech and speaker recognition, physiological signal processing and machine learning. Y agmur Ceren Dardagan is a senior stu- dent in Computer Engineering Department of Bogazici Univ ersity , Turk ey . Her research interests include physiological signal process- ing, machine learning and pervasi ve health applications. Cem Ersoy receiv ed the Ph.D. degree from Polytechnic University , New Y ork in 1992. He was an R&D Engineer in NE- T AS A. S. from 1984 to 1986. He is a Professor of computer engineering at Bogazici Univ ersity , T urkey . He is also the V ice Director of the T elecommunications and Informatics T echnologies Research Cen- ter , TET AM. His research interests include wireless/cellular/ad-hoc/sensor networks, ac- tivity recognition, and ambient intelligence for pervasiv e health applications, green 5G and beyond networks, and mobile cloud/edge/fog computing. He was the Chairman of the IEEE Communications Society Turkish Chapter eight years. He is a member of the IFIP .