Scalable and Adaptable Cross-Domain Autonomous Health Assessment
CAREER Project
The wide availability of commodity smart home sensor systems (Google Home, Amazon Echo, etc.) and internet-of-things (IoT) devices (Fitbit, Actigraph, etc.) is making it easier to continuously monitor individuals' health-related vital signals, activities, and behaviors to provide just-in-time health intervention to the aging population. This CAREER project seeks to design, implement, and evaluate heterogeneous sensor systems in smart homes that help ameliorate the progressive functional and behavioral health decline of older adults.
In order to realize autonomous health assessment methodologies in practice, it is necessary to build an activity and behavior recognition system across multiple inhabitants and various connected consumer devices that can select, adapt, and cope with device and user heterogeneities, privacy characteristics, resource constraints and scarcity of labeled data. To address the above-mentioned problems, this research project contributes to new methodology in four ways.
First, it is introducing deep transfer learning activity recognition model and multi-user multi-device optimization-based heuristics that automatically help adapt the inherent variations across different domains, including user/device-type/device-instance.
Second, it is designing a spatio-temporal dynamical system approach based on fractal dynamics to mitigate the variability in various sensor signals, and capture the self-similarity of human physiological health markers and establish the parametric task performance dependency between functional and behavioral health measurements.
Third, it posits an opportunistic sensing architecture and human-in-the loop activity model for real-time data sharing and annotation that help optimize the user interruption and system performance.
Fourth, it is designing a distributed implementation of tailored-computational techniques in actual smart home deployments, and evaluating the effectiveness of sensor-based functional and behavioral models and algorithms for just-in-time health assessment in actual living environments.
Cross domain activity recognition scenarios: the same set of activities can be performed by different persons and captured using a variety of smart devices across different body positions, introducing a wide range of heterogeneities e.g. device type/sampling rate, personal and positional and any combination of them.
Award Title
CAREER: Scalable and Adaptable Cross-Domain Autonomous Health Assessment
Acknowledgement
This material is based upon work supported by the National Science Foundation under Grant Numbers: 1750936 (University of Maryland Baltimore County)
Disclaimer
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Duration
May 1, 2018 to April 30, 2023 (Estimated)
Team
Investigators
Nirmalya Roy, University of Maryland, Baltimore County (Principal Investigator)
Graduate Students
Zahid Hasan
Graduated PhD Students
H M Sajjad Hossain, Microsoft Research [Thesis: Active Activity Recognition with Context-Aware Annotator Selection][PDF][PPT]
MD Abdullah Al Hafiz Khan, Kennesaw State University [Thesis: Cross-Domain Scalable Activity Recognition Models in Smart Environments][PDF][PPT]
Abu Zaher Md Faridee, University of Maryland Baltimore County [Thesis: Building Robust Human Activity Recognition from Unlabeled Data]
Sreenivasan Ramasamy Ramamurthy, Bowie State University [Thesis: A Counterfactual Verified Semi-Supervised Learning Framework for Older Adults' Functional and Cognitive Health Assessment]
Collaborators
Elizabeth Galik, University of Maryland, School of Nursing
Diane Ancel, Clinical Evaluator
Archan Misra, Singapore Management University
Project Goals
The goals of this project are to develop novel deep transfer learning techniques, optimization-based heuristics, opportunistic sensing architecture, and spatiotemporal dynamical systems-based approaches to address the diversity, adaptability, and reliability of activity and behavior recognition models across different users and technologies, while leveraging a human-in-the-loop control for improving the performance of the sensor systems.
Research Challenges
The project specifically looks at cross-domain approaches that can accommodate variability in behavior, activity, and physiological health conditions across a large population and diverse set of smart home sensor systems. The inability to build scalable and adaptable activity and behavior monitoring models across domains such as multi-occupant homes with heterogeneous internet-of-things devices is a major impediment to adoption of smart home technologies for healthcare applications.
Summary of Current Results
Our current project research and development activities are helping to automate activity and physiological health monitoring at scale, and thereby improving the design and study of adaptive interventions for elderly people, their families, and professional caregivers.
AugToAct [13] At a Glance
Semi-supervised learning architecture for scalable human activity recognition introduced in AugToAct [13] leveraging unlabeled data to improve ADL and iADL classification
Accuracy comparison of semi-supervised and augmentation modules in AugToAct[13] with different ratio of labeled and unlabeled samples on two datasets: DAR and HHAR.
Note: The labeled data percentage is expressed in log scale on the x axis.
AugToAct [13] and HDCNN [18] architectures working in tandem in a semi-supervised transfer learning setup with limited labeled data in both source and target domain
Accuracy comparison of AugToAct[13] and other transfer learning approaches on the DAR dataset.
Publications
Indrajeet Ghosh, Adam Goldstein, Avijoy Chakma, Jade Freeman, Timothy Gregory, Niranjan Suri, Sreenivasan Ramasamy Ramamurthy, and Nirmalya Roy. HeteroSys: Heterogeneous and Collaborative Sensing in the Wild, in Proceedings of 9th IEEE International Workshop on Sensors and Smart Cities, co-located with the 9th IEEE International Conference on Smart Computing (SmartComp), Nashville, Tennessee, USA, June 2023.
Avijoy Chakma, Abu Zaher Md Faridee, Indrajeet Ghosh, Nirmalya Roy. Domain Adaptation for Inertial Measurement Unit-based Human Activity Recognition: A Survey in arXiv preprint, April 2023
Zahid Hasan, Masud Hasan, Abu Zaher Md Faridee, Sanjay Purushotham, Heesung Kwon, Hyungtae Lee and Nirmalya Roy. NEV-NCD: Negative Learning, Entropy, and Variance regularization based novel action categories discovery, in arXiv preprint, April 2023
Zahid Hasan, Abu Zaher Md Faridee, Masud Ahmed and Nirmalya Roy. Self-rPPG: Learning the Optical and Physiological Mechanics of Remote Photoplethysmography with Self -Supervision, in Proceedings of the IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE), Washington D.C., Nov 2022
Zahid Hasan, Emon Dey, Sreenivasan Ramasamy Ramamurthy, Nirmalya Roy, Archan Misra. RhythmEdge: Enabling Contactless Heart Rate Estimation on the Edge, in Proceedings of the 8th IEEE International Conference on Smart Computing (SmartComp), Espoo, Finland, June 2022 Best Paper Award
Abu Zaher Md Faridee, Avijoy Chakma, Zahid Hasan, Nirmalya Roy, Archan Misra. CoDEm: Conditional Domain Embeddings for Scalable Human Activity Recognition, in Proceedings of the 8th IEEE International Conference on Smart Computing (SmartComp), Espoo, Finland, June 2022
Md Abdullah Al Hafiz Khan and Nirmalya Roy. Cross-Domain Unseen Activity Recognition Using Transfer Learning, in Proceedings of IEEE Computers, Software, and Applications Conference (COMPSAC), June 2022
Avijoy Chakma, Abu Zaher Md Faridee, Nirmalya Roy and Raghuveer Rao. Semi-supervised Multi-source Domain Adaptation in Wearable Activity Recognition, in Proceedings of 18th International Conference on Distributed Computing in Sensor Systems (DCOSS), Marina Del Rey, LA, California, June 2022
Sreenivasan Ramasamy Ramamurthy, Soumyajit Chatterjee, Elizabeth Galik, Aryya Gangopadhyay, Nirmalya Roy, Bivas Mitra, Sandip Chakraborty. CogAx: Early Assessment of Cognitive and Functional Impairment from Accelerometry, IEEE PerCom 2022, Pisa, Italy, March 2022
Avijoy Chakma, Abu Zaher Md Faridee, Md Abdullah Al Hafiz Khan, Nirmalya Roy. Activity Recognition in Wearables Using Adversarial Multi-Source Domain Adaptation, in Smart Health, December 2021
Abu Zaher Md Faridee, Avijoy Chakma, Archan Misra, Nirmalya Roy. STranGAN: Adversarially-Learnt Spatial Transformer for Scalable Human Activity Recognition, in Proceedings of the IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE), Washington D.C. Dec 2021 Best Paper Award
Zahid Hasan, Sreenivasan Ramasamy Ramamurthy, Nirmalya Roy. CamSense: A Camera-Based Contact-less Heart Activity Monitoring, in Proceedings of the IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE), Washington D.C. Dec 2021
Nhan Nguyen, Avijoy Chakma, Nirmalya Roy. A Scalable and Domain Adaptive Respiratory Symptoms Detection Framework using Earables, in Proceedings of the National Symposium for NSF REU Research in Data Science, Systems, and Security in conjunction with IEEE Big Data Conference, Dec 2021
Sreenivasan Ramasamy Ramamurthy, Indrajeet Ghosh, Aryya Gangopadhyay, Elizabeth Galik, Nirmalya Roy. STAR: A Scalable Self-taught Learning Framework for Older Adults’ Activity Recognition, in Proceedings of the 7th IEEE International Conference on Smart Computing (SmartComp), Irvine, USA, August 2021
Soumyajit Chatterjee, Avijoy Chakma, Aryya Gangopadhyay, Nirmalya Roy, Bivas Mitra, Sandip Chakraborty. LASO: Exploiting Locomotive and Acoustic Signatures over the Edge to Annotate IMU Data for Human Activity Recognition, in Proceedings of the 22nd ACM International Conference on Multimodal Interaction (ICMI), Netherlands, October 2020
Emon Dey and Nirmalya Roy. OMAD: On-device Mental Anomaly Detection for Substance and Non-Substance Users, in Proceedings of the 20th IEEE International Conference on Bioinformatics and Bioengineering (BIBE), October 2020
Faridee, Abu Zaher and Khan, Md Abdullah and Pathak, Nilavra and Roy, Nirmalya. "AugToAct: scaling complex human activity recognition with few labels," 16th ACM/EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous), 2019. doi:10.1145/3360774.3360831 Citation details
Hossain, H M and Roy, Nirmalya. "Active Deep Learning for Activity Recognition with Context Aware Annotator Selection," Proceedings of the 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), Anchorage, Alaska, August 2019, 2019. doi:10.1145/3292500.3330688 Citation details
Hossain, H. M. and Al Haiz Khan, MD Abdullah and Roy, Nirmalya. "DeActive: Scaling Activity Recognition with Active Deep Learning," Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, v.2, 2018. doi:10.1145/3214269 Citation details
Ramasamy Ramamurthy, Sreenivasan and Roy, Nirmalya. "Recent trends in machine learning for human activity recognition-A survey," Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, v.8, 2018. doi:10.1002/widm.1254 Citation details
Hossain, S and Roy, N.. "SocialAnnotator: Annotator Selection by Exploiting Social Relationships in Activity Recognition," Proceedings of the IEEE AAAI 2018 Fall Symposium, Oct 2018, 2018. Citation details
Khan, Md Abdullah and Roy, Nirmalya and Misra, Archan. "Scaling Human Activity Recognition via Deep Learning-based Domain Adaptation," 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom), 2018. doi:10.1109/PERCOM.2018.8444585 Citation details
Khan, Md Abdullah and Roy, Nirmalya. "UnTran: Recognizing Unseen Activities with Unlabeled Data Using Transfer Learning," 2018 IEEE/ACM Third International Conference on Internet-of-Things Design and Implementation (IoTDI), 2018. doi:10.1109/IoTDI.2018.00014 Citation details
Presentations
PhD student, Mr. Zahid Hasan invited to present RhythmEdge -- IEEE SmartComp 2022 Best Paper to Aberdeen Proving Ground, ARL. Congratulations Zahid!
Graduate Student Abu Zaher Md Faridee attended IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE) in Washington D.C. December 2021 in person and presented his work on Spatial Transformer for Scalable Human Activity Recognition and received Best Paper Award. Congratulations Zaher!
Dr. Roy discussed AI + IoT + ML for Innovative Smart Computing Applications at Colorado School of Mines, August 2021.
Dr. Roy presented on Multi-modal Sensing, Cross-Domain Modeling, and Community Deployments in Smart Environments at Army Research Lab, Adelphi, MD, July 2020.
Dr. Roy delivered a Keynote talk on Cross-Domain Machine Learning Framework for Pervasive Sensing at the IEEE PerIoT workshop in conjunction with IEEE PerCom 2020 conference on March 27 in Austin, USA.
Dr. Roy delivered an invited talk on Cross-Domain Machine Learning Framework for Scalable Human Activity and Behavior Recognition at the 12th ACM/IEEE International Conference on COMmunication Systems & NETworkS (COMSNETS), Bangalore, India, January 2020.
Graduate Student Abu Zaher Md Faridee attended 16th ACM/EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous) in Houston, Texas, Nov 10-14, 2019, and presented his work on Scaling Complex Human Activity Recognition with Few Labels
Dr. Roy attended NSF CSR 2019 PI meeting, Nov 4-5, in Arlington, VA, and presented a poster on Scalable and Adaptable Cross-Domain Autonomous Health Assessment.
Dr. Roy presented on IoT for Longitudinal Health Monitoring and Assessment at THINK GLOBALLY ACT LOCALLY UMBC-Italy Cooperation Meeting, UMBC, April 2018
Dr. Roy outlined the Journey of my CAREER Award at the UMBC CAREER workshop for Junior Faculty, March 2018
Vidoes
"STranGAN: Adversarially-Learnt Spatial Transformer for Scalable Human Activity Recognition" Presented at IEEE/ACM CHASE December 16, 2021, Washington D.C., USA.
Zaher's Dissertation Defense: "Building Robust Human Activity Recognition Models from Unlabeled Data"
Zahid's Invitation Talk at Aberdeen Proving Ground, ARL-"RhythmEdge: Enabling Contactless Heart Rate Estimation on the Edge" ( IEEE SmartComp 2022 Best Paper)
Datasets
Zahid Hasan, Masud Hasan, Nirmalya Roy (2023). Multi-view Dataset: 1st release Feb 202 [URL]
Zahid Hasan, Sreenivasan Ramasamy Ramamurthy, Nirmalya Roy. (2021). MPSC-rPPG Dataset. IEEE Dataport. [URL]
Indrajeet Ghosh, Sreenivasan Ramasamy Ramamurthy, Avijoy Chakma, Emon Dey, Zahid Hasan, Nirmalya Roy. (2020). Badminton Activity Recognition (BAR). IEEE Dataport. [URL]
Software Downloads
To be added
Patents
To be added
Broader Impacts
Algorithmic Advances
The project is investigating various state-of-the-art machine learning techniques such as transfer learning, active learning, deep learning with ambient and wearable sensors, IoT technologies and clinical tools in practice to improve the quality-of-life of older adults living independently in retirement community centers, assisted living and smart home environments.
Dataset Advances
Work in progress.
Integration with the broader research community
The PI and his research team have been collaborating with the University of Maryland, School of Nursing for real deployment of smart home sensor systems and technologies at three retirement community centers and senior homes in the greater Baltimore area.
Educational Material
Dr. Roy has designed a new course on Smart Home Health Analytics and has been offering this course at UMBC every alternative semester. The course has generated many publications related with the broader application of the CAREER project to health and other research areas such as sports analytics, cyber-physical systems and smart city. Here are some publications which have been inspired by the various Smart Home Health Analytics course research projects.
David Welsh and Nirmalya Roy. Smartphone-based Mobile Gunshot Detection, In Proceedings of the 13th Workshop on Context and Activity Modeling and Recognition (CoMoRea’17), co-located with PerCom, March 2017. [pdf]
H M Sajjad Hossain, Md Abdullah Al Hafiz Khan, and Nirmalya Roy. SoccerMate: A Personal Soccer Attribute Profiler using Wearables, in Proceedings of the 1st IEEE PerCom International Workshop on Behavioral Implications of Contextual Analytics (BICA), co-located with PerCom, March 2017. [pdf]
Abu Zaher Md Faridee, Sreenivasan Ramasamy Ramamurthy, H M Sajjad Hossain, and Nirmalya Roy. HappyFeet: Recognizing and Assessing Dance on the Floor, in Proceedings of the ACM 19th International Workshop on Mobile Computing Systems and Applications (HotMobile), Feb. 2018 [pdf]
Varun Mandalapu, Lavanya Elluri and Nirmalya Roy. Developing Machine Learning based Predictive Models for Smart Policing, in Proceedings of the 1st IEEE International Students Workshop on Smart Computing (SmartStudents), co-located with SmartComp, pp. 1-6, Washington D.C., June 2019
Highlights and Press Releases
Point of Contact
Nirmalya Roy, University of Maryland, Baltimore County (Principal Investigator)
Last Updated
04/18/2023