Scalable and Adaptable Cross-Domain Autonomous Health Assessment
The wide availability of commodity smart home sensor systems (Google Home, Amazon Echo, etc.) and internet-of-things (IoT) devices (Fitbit, Actigraph, etc.) is making it easier to continuously monitor individuals' health-related vital signals, activities, and behaviors to provide just-in-time health intervention to the aging population. This CAREER project seeks to design, implement, and evaluate heterogeneous sensor systems in smart homes that help ameliorate the progressive functional and behavioral health decline of older adults.
In order to realize autonomous health assessment methodologies in practice, it is necessary to build an activity and behavior recognition system across multiple inhabitants and various connected consumer devices that can select, adapt, and cope with device and user heterogeneities, privacy characteristics, resource constraints and scarcity of labeled data. To address the above-mentioned problems, this research project contributes to new methodology in four ways.
- First, it is introducing deep transfer learning activity recognition model and multi-user multi-device optimization-based heuristics that automatically help adapt the inherent variations across different domains, including user/device-type/device-instance.
- Second, it is designing a spatio-temporal dynamical system approach based on fractal dynamics to mitigate the variability in various sensor signals, and capture the self-similarity of human physiological health markers and establish the parametric task performance dependency between functional and behavioral health measurements.
- Third, it posits an opportunistic sensing architecture and human-in-the loop activity model for real-time data sharing and annotation that help optimize the user interruption and system performance.
- Fourth, it is designing a distributed implementation of tailored-computational techniques in actual smart home deployments, and evaluating the effectiveness of sensor-based functional and behavioral models and algorithms for just-in-time health assessment in actual living environments.
Cross domain activity recognition scenarios: the same set of activities can be performed by different persons and captured using a variety of smart devices across different body positions, introducing a wide range of heterogeneities e.g. device type/sampling rate, personal and positional and any combination of them.
CAREER: Scalable and Adaptable Cross-Domain Autonomous Health Assessment
This material is based upon work supported by the National Science Foundation under Grant Numbers: 1750936 (University of Maryland Baltimore County)
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
May 1, 2018 to April 30, 2023 (Estimated)
- Nirmalya Roy, University of Maryland, Baltimore County (Principal Investigator)
Graduated PhD Students
- H M Sajjad Hossain, Microsoft Research [Thesis: Active Activity Recognition with Context-Aware Annotator Selection][PDF][PPT]
- MD Abdullah Al Hafiz Khan, Philips Research [Thesis: Cross-Domain Scalable Activity Recognition Models in Smart Environments][PDF][PPT]
The goals of this project are to develop novel deep transfer learning techniques, optimization-based heuristics, opportunistic sensing architecture, and spatiotemporal dynamical systems-based approaches to address the diversity, adaptability, and reliability of activity and behavior recognition models across different users and technologies, while leveraging a human-in-the-loop control for improving the performance of the sensor systems.
The project specifically looks at cross-domain approaches that can accommodate variability in behavior, activity, and physiological health conditions across a large population and diverse set of smart home sensor systems. The inability to build scalable and adaptable activity and behavior monitoring models across domains such as multi-occupant homes with heterogeneous internet-of-things devices is a major impediment to adoption of smart home technologies for healthcare applications.
Summary of Current Results
Our current project research and development activities are helping to automate activity and physiological health monitoring at scale, and thereby improving the design and study of adaptive interventions for elderly people, their families, and professional caregivers.
Semi-supervised learning architecture for scalable human activity recognition introduced in AugToAct  leveraging unlabeled data to improve ADL and iADL classification
Accuracy comparison of semi-supervised and augmentation modules in AugToAct with different ratio of labeled and unlabeled samples on two datasets: DAR and HHAR.Note: The labeled data percentage is expressed in log scale on the x axis.
AugToAct  and HDCNN  architectures working in tandem in a semi-supervised transfer learning setup with limited labeled data in both source and target domain
Accuracy comparison of AugToAct and other transfer learning approaches on the DAR dataset.
- Faridee, Abu Zaher and Khan, Md Abdullah and Pathak, Nilavra and Roy, Nirmalya. "AugToAct: scaling complex human activity recognition with few labels," 16th ACM/EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous), 2019, 2019. doi:10.1145/3360774.3360831 Citation details
- Hossain, H M and Roy, Nirmalya. "Active Deep Learning for Activity Recognition with Context Aware Annotator Selection," Proceedings of the 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), Anchorage, Alaska, August 2019, 2019. doi:10.1145/3292500.3330688 Citation details
- Hossain, H. M. and Al Haiz Khan, MD Abdullah and Roy, Nirmalya. "DeActive: Scaling Activity Recognition with Active Deep Learning," Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, v.2, 2018. doi:10.1145/3214269 Citation details
- Ramasamy Ramamurthy, Sreenivasan and Roy, Nirmalya. "Recent trends in machine learning for human activity recognition-A survey," Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, v.8, 2018. doi:10.1002/widm.1254 Citation details
- Hossain, S and Roy, N.. "SocialAnnotator: Annotator Selection by Exploiting Social Relationships in Activity Recognition," Proceedings of the IEEE AAAI 2018 Fall Symposium, Oct 2018, 2018. Citation details
- Khan, Md Abdullah and Roy, Nirmalya and Misra, Archan. "Scaling Human Activity Recognition via Deep Learning-based Domain Adaptation," 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom), 2018. doi:10.1109/PERCOM.2018.8444585 Citation details
- Khan, Md Abdullah and Roy, Nirmalya. "UnTran: Recognizing Unseen Activities with Unlabeled Data Using Transfer Learning," 2018 IEEE/ACM Third International Conference on Internet-of-Things Design and Implementation (IoTDI), 2018. doi:10.1109/IoTDI.2018.00014 Citation details
- Dr Roy delivered a Keynote talk on Cross-Domain Machine Learning Framework for Pervasive Sensing at IEEE PerIoT workshop in conjunction with IEEE PerCom 2020 conference on March 27 in Austin, USA.
- Dr. Roy delivered an invited talk on Cross-Domain Machine Learning Framework for Scalable Human Activity and Behavior Recognition at 12th ACM/IEEE International Conference on COMmunication Systems & NETworkS (COMSNETS), Bangalore, India, January 2020.
- Graduate Student Abu Zaher Md Faridee attended 16th ACM/EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous) in Houston, Texas, Nov 10-14, 2019 and presented his work on Scaling Complex Human Activity Recognition with Few Labels
- Dr. Roy attended NSF CSR 2019 PI meeting, Nov 4-5, in Arlington, VA and presented a poster on Scalable and Adaptable Cross-Domain Autonomous Health Assessment.
To be added
To be added
To be added
The project is investigating various state-of-the-art machine learning techniques such as transfer learning, active learning, deep learning with ambient and wearable sensors, IoT technologies and clinical tools in practice to improve the quality-of-life of older adults living independently in retirement community centers, assisted living and smart home environments.
Work in progress.
Integration with the broader research community
The PI and his research team have been collaborating with the University of Maryland, School of Nursing for real deployment of smart home sensor systems and technologies at three retirement community centers and senior homes in the greater Baltimore area.
Dr. Roy has designed a new course on Smart Home Health Analytics and has been offering this course at UMBC every alternative semester. The course has generated many publications related with the broader application of the CAREER project to health and other research areas such as sports analytics, cyber-physical systems and smart city. Here are some publications which have been inspired by the various Smart Home Health Analytics course research projects.
David Welsh and Nirmalya Roy. Smartphone-based Mobile Gunshot Detection, In Proceedings of the 13th Workshop on Context and Activity Modeling and Recognition (CoMoRea’17), co-located with PerCom, March 2017. [pdf]
H M Sajjad Hossain, Md Abdullah Al Hafiz Khan, and Nirmalya Roy. SoccerMate: A Personal Soccer Attribute Profiler using Wearables, in Proceedings of the 1st IEEE PerCom International Workshop on Behavioral Implications of Contextual Analytics (BICA), co-located with PerCom, March 2017. [pdf]
Abu Zaher Md Faridee, Sreenivasan Ramasamy Ramamurthy, H M Sajjad Hossain, and Nirmalya Roy. HappyFeet: Recognizing and Assessing Dance on the Floor, in Proceedings of the ACM 19th International Workshop on Mobile Computing Systems and Applications (HotMobile), Feb. 2018 [pdf]
Varun Mandalapu, Lavanya Elluri and Nirmalya Roy. Developing Machine Learning based Predictive Models for Smart Policing, in Proceedings of the 1st IEEE International Students Workshop on Smart Computing (SmartStudents), co-located with SmartComp, pp. 1-6, Washington D.C., June 2019