Scalable and Adaptable Cross-Domain Autonomous Health Assessment

CAREER Project

The wide availability of commodity smart home sensor systems (Google Home, Amazon Echo, etc.) and internet-of-things (IoT) devices (Fitbit, Actigraph, etc.) is making it easier to continuously monitor individuals' health-related vital signals, activities, and behaviors to provide just-in-time health intervention to the aging population. This CAREER project seeks to design, implement, and evaluate heterogeneous sensor systems in smart homes that help ameliorate the progressive functional and behavioral health decline of older adults.

In order to realize autonomous health assessment methodologies in practice, it is necessary to build an activity and behavior recognition system across multiple inhabitants and various connected consumer devices that can select, adapt, and cope with device and user heterogeneities, privacy characteristics, resource constraints and scarcity of labeled data. To address the above-mentioned problems, this research project contributes to new methodology in four ways.

  • First, it is introducing deep transfer learning activity recognition model and multi-user multi-device optimization-based heuristics that automatically help adapt the inherent variations across different domains, including user/device-type/device-instance.

  • Second, it is designing a spatio-temporal dynamical system approach based on fractal dynamics to mitigate the variability in various sensor signals, and capture the self-similarity of human physiological health markers and establish the parametric task performance dependency between functional and behavioral health measurements.

  • Third, it posits an opportunistic sensing architecture and human-in-the loop activity model for real-time data sharing and annotation that help optimize the user interruption and system performance.

  • Fourth, it is designing a distributed implementation of tailored-computational techniques in actual smart home deployments, and evaluating the effectiveness of sensor-based functional and behavioral models and algorithms for just-in-time health assessment in actual living environments.

Cross domain activity recognition scenarios: the same set of activities can be performed by different persons and captured using a variety of smart devices across different body positions, introducing a wide range of heterogeneities e.g. device type/sampling rate, personal and positional and any combination of them.

Award Title

CAREER: Scalable and Adaptable Cross-Domain Autonomous Health Assessment

Acknowledgement

This material is based upon work supported by the National Science Foundation under Grant Numbers: 1750936 (University of Maryland Baltimore County)

Disclaimer

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Duration

May 1, 2018 to April 30, 2023 (Estimated)

Team

Investigators

  • Nirmalya Roy, University of Maryland, Baltimore County (Principal Investigator)

Graduate Students

Graduated PhD Students

  • H M Sajjad Hossain, Microsoft Research [Thesis: Active Activity Recognition with Context-Aware Annotator Selection][PDF][PPT]

  • MD Abdullah Al Hafiz Khan, Philips Research [Thesis: Cross-Domain Scalable Activity Recognition Models in Smart Environments][PDF][PPT]

Collaborators

Project Goals

The goals of this project are to develop novel deep transfer learning techniques, optimization-based heuristics, opportunistic sensing architecture, and spatiotemporal dynamical systems-based approaches to address the diversity, adaptability, and reliability of activity and behavior recognition models across different users and technologies, while leveraging a human-in-the-loop control for improving the performance of the sensor systems.

Research Challenges

The project specifically looks at cross-domain approaches that can accommodate variability in behavior, activity, and physiological health conditions across a large population and diverse set of smart home sensor systems. The inability to build scalable and adaptable activity and behavior monitoring models across domains such as multi-occupant homes with heterogeneous internet-of-things devices is a major impediment to adoption of smart home technologies for healthcare applications.

Summary of Current Results

Our current project research and development activities are helping to automate activity and physiological health monitoring at scale, and thereby improving the design and study of adaptive interventions for elderly people, their families, and professional caregivers.

AugToAct [13] At a Glance

Semi-supervised learning architecture for scalable human activity recognition introduced in AugToAct [13] leveraging unlabeled data to improve ADL and iADL classification

Accuracy comparison of semi-supervised and augmentation modules in AugToAct[13] with different ratio of labeled and unlabeled samples on two datasets: DAR and HHAR.

Note: The labeled data percentage is expressed in log scale on the x axis.

AugToAct [13] and HDCNN [18] architectures working in tandem in a semi-supervised transfer learning setup with limited labeled data in both source and target domain


Accuracy comparison of AugToAct[13] and other transfer learning approaches on the DAR dataset.

Publications

  1. Zahid Hasan, Emon Dey, Sreenivasan Ramasamy Ramamurthy, Nirmalya Roy, Archan Misra. RhythmEdge: Enabling Contactless Heart Rate Estimation on the Edge, in Proceedings of the 8th IEEE International Conference on Smart Computing (SmartComp), Espoo, Finland, June 2022

  2. Abu Zaher Md Faridee, Avijoy Chakma, Zahid Hasan, Nirmalya Roy, Archan Misra. CoDEm: Conditional Domain Embeddings for Scalable Human Activity Recognition, in Proceedings of the 8th IEEE International Conference on Smart Computing (SmartComp), Espoo, Finland, June 2022

  3. Md Abdullah Al Hafiz Khan and Nirmalya Roy. Cross-Domain Unseen Activity Recognition Using Transfer Learning, in Proceedings of IEEE Computers, Software, and Applications Conference (COMPSAC), June 2022

  4. Avijoy Chakma, Abu Zaher Md Faridee, Nirmalya Roy and Raghuveer Rao. Semi-supervised Multi-source Domain Adaptation in Wearable Activity Recognition, in Proceedings of 18th International Conference on Distributed Computing in Sensor Systems (DCOSS), Marina Del Rey, LA, California, June 2022

  5. Sreenivasan Ramasamy Ramamurthy, Soumyajit Chatterjee, Elizabeth Galik, Aryya Gangopadhyay, Nirmalya Roy, Bivas Mitra, Sandip Chakraborty. CogAx: Early Assessment of Cognitive and Functional Impairment from Accelerometry, IEEE PerCom 2022, Pisa, Italy, March 2022

  6. Avijoy Chakma, Abu Zaher Md Faridee, Md Abdullah Al Hafiz Khan, Nirmalya Roy. Activity Recognition in Wearables Using Adversarial Multi-Source Domain Adaptation, in Smart Health, December 2021

  7. Abu Zaher Md Faridee, Avijoy Chakma, Archan Misra, Nirmalya Roy. STranGAN: Adversarially-Learnt Spatial Transformer for Scalable Human Activity Recognition, in Proceedings of the IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE), Washington D.C. Dec 2021 Best Paper Award

  8. Zahid Hasan, Sreenivasan Ramasamy Ramamurthy, Nirmalya Roy. CamSense: A Camera-Based Contact-less Heart Activity Monitoring, in Proceedings of the IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE), Washington D.C. Dec 2021

  9. Nhan Nguyen, Avijoy Chakma, Nirmalya Roy. A Scalable and Domain Adaptive Respiratory Symptoms Detection Framework using Earables, in Proceedings of the National Symposium for NSF REU Research in Data Science, Systems, and Security in conjunction with IEEE Big Data Conference, Dec 2021

  10. Sreenivasan Ramasamy Ramamurthy, Indrajeet Ghosh, Aryya Gangopadhyay, Elizabeth Galik, Nirmalya Roy. STAR: A Scalable Self-taught Learning Framework for Older Adults’ Activity Recognition, in Proceedings of the 7th IEEE International Conference on Smart Computing (SmartComp), Irvine, USA, August 2021

  11. Soumyajit Chatterjee, Avijoy Chakma, Aryya Gangopadhyay, Nirmalya Roy, Bivas Mitra, Sandip Chakraborty. LASO: Exploiting Locomotive and Acoustic Signatures over the Edge to Annotate IMU Data for Human Activity Recognition, in Proceedings of the 22nd ACM International Conference on Multimodal Interaction (ICMI), Netherlands, October 2020

  12. Emon Dey and Nirmalya Roy. OMAD: On-device Mental Anomaly Detection for Substance and Non-Substance Users, in Proceedings of the 20th IEEE International Conference on Bioinformatics and Bioengineering (BIBE), October 2020

  13. Faridee, Abu Zaher and Khan, Md Abdullah and Pathak, Nilavra and Roy, Nirmalya. "AugToAct: scaling complex human activity recognition with few labels," 16th ACM/EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous), 2019. doi:10.1145/3360774.3360831 Citation details

  14. Hossain, H M and Roy, Nirmalya. "Active Deep Learning for Activity Recognition with Context Aware Annotator Selection," Proceedings of the 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), Anchorage, Alaska, August 2019, 2019. doi:10.1145/3292500.3330688 Citation details

  15. Hossain, H. M. and Al Haiz Khan, MD Abdullah and Roy, Nirmalya. "DeActive: Scaling Activity Recognition with Active Deep Learning," Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, v.2, 2018. doi:10.1145/3214269 Citation details

  16. Ramasamy Ramamurthy, Sreenivasan and Roy, Nirmalya. "Recent trends in machine learning for human activity recognition-A survey," Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, v.8, 2018. doi:10.1002/widm.1254 Citation details

  17. Hossain, S and Roy, N.. "SocialAnnotator: Annotator Selection by Exploiting Social Relationships in Activity Recognition," Proceedings of the IEEE AAAI 2018 Fall Symposium, Oct 2018, 2018. Citation details

  18. Khan, Md Abdullah and Roy, Nirmalya and Misra, Archan. "Scaling Human Activity Recognition via Deep Learning-based Domain Adaptation," 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom), 2018. doi:10.1109/PERCOM.2018.8444585 Citation details

  19. Khan, Md Abdullah and Roy, Nirmalya. "UnTran: Recognizing Unseen Activities with Unlabeled Data Using Transfer Learning," 2018 IEEE/ACM Third International Conference on Internet-of-Things Design and Implementation (IoTDI), 2018. doi:10.1109/IoTDI.2018.00014 Citation details

Presentations

  • Graduate Student Abu Zaher Md Faridee attended IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE) in Washington D.C. December 2021 in person and presented his work on Spatial Transformer for Scalable Human Activity Recognition and received Best Paper Award. Congratulations Zaher!

  • Dr. Roy discussed AI + IoT + ML for Innovative Smart Computing Applications at Colorado School of Mines, August 2021.

  • Dr. Roy presented on Multi-modal Sensing, Cross-Domain Modeling, and Community Deployments in Smart Environments at Army Research Lab, Adelphi, MD, July 2020.

  • Dr. Roy delivered a Keynote talk on Cross-Domain Machine Learning Framework for Pervasive Sensing at the IEEE PerIoT workshop in conjunction with IEEE PerCom 2020 conference on March 27 in Austin, USA.

  • Dr. Roy delivered an invited talk on Cross-Domain Machine Learning Framework for Scalable Human Activity and Behavior Recognition at the 12th ACM/IEEE International Conference on COMmunication Systems & NETworkS (COMSNETS), Bangalore, India, January 2020.

  • Graduate Student Abu Zaher Md Faridee attended 16th ACM/EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous) in Houston, Texas, Nov 10-14, 2019, and presented his work on Scaling Complex Human Activity Recognition with Few Labels

  • Dr. Roy attended NSF CSR 2019 PI meeting, Nov 4-5, in Arlington, VA, and presented a poster on Scalable and Adaptable Cross-Domain Autonomous Health Assessment.

  • Dr. Roy presented on IoT for Longitudinal Health Monitoring and Assessment at THINK GLOBALLY ACT LOCALLY UMBC-Italy Cooperation Meeting, UMBC, April 2018

  • Dr. Roy outlined the Journey of my CAREER Award at the UMBC CAREER workshop for Junior Faculty, March 2018

Vidoes

"STranGAN: Adversarially-Learnt Spatial Transformer for Scalable Human Activity Recognition" Presented at IEEE/ACM CHASE December 16, 2021, Washington D.C., USA.


Datasets

  1. Zahid Hasan, Sreenivasan Ramasamy Ramamurthy, Nirmalya Roy. (2021). MPSC-rPPG Dataset. IEEE Dataport. [URL]

  2. Indrajeet Ghosh, Sreenivasan Ramasamy Ramamurthy, Avijoy Chakma, Emon Dey, Zahid Hasan, Nirmalya Roy. (2020). Badminton Activity Recognition (BAR). IEEE Dataport. [URL]

Software Downloads

To be added

Patents

To be added

Broader Impacts

Algorithmic Advances

The project is investigating various state-of-the-art machine learning techniques such as transfer learning, active learning, deep learning with ambient and wearable sensors, IoT technologies and clinical tools in practice to improve the quality-of-life of older adults living independently in retirement community centers, assisted living and smart home environments.

Dataset Advances

Work in progress.

Integration with the broader research community

The PI and his research team have been collaborating with the University of Maryland, School of Nursing for real deployment of smart home sensor systems and technologies at three retirement community centers and senior homes in the greater Baltimore area.

Educational Material

Dr. Roy has designed a new course on Smart Home Health Analytics and has been offering this course at UMBC every alternative semester. The course has generated many publications related with the broader application of the CAREER project to health and other research areas such as sports analytics, cyber-physical systems and smart city. Here are some publications which have been inspired by the various Smart Home Health Analytics course research projects.

David Welsh and Nirmalya Roy. Smartphone-based Mobile Gunshot Detection, In Proceedings of the 13th Workshop on Context and Activity Modeling and Recognition (CoMoRea’17), co-located with PerCom, March 2017. [pdf]

H M Sajjad Hossain, Md Abdullah Al Hafiz Khan, and Nirmalya Roy. SoccerMate: A Personal Soccer Attribute Profiler using Wearables, in Proceedings of the 1st IEEE PerCom International Workshop on Behavioral Implications of Contextual Analytics (BICA), co-located with PerCom, March 2017. [pdf]

Abu Zaher Md Faridee, Sreenivasan Ramasamy Ramamurthy, H M Sajjad Hossain, and Nirmalya Roy. HappyFeet: Recognizing and Assessing Dance on the Floor, in Proceedings of the ACM 19th International Workshop on Mobile Computing Systems and Applications (HotMobile), Feb. 2018 [pdf]

Varun Mandalapu, Lavanya Elluri and Nirmalya Roy. Developing Machine Learning based Predictive Models for Smart Policing, in Proceedings of the 1st IEEE International Students Workshop on Smart Computing (SmartStudents), co-located with SmartComp, pp. 1-6, Washington D.C., June 2019

Point of Contact

  • Nirmalya Roy, University of Maryland, Baltimore County (Principal Investigator)

Last Updated

04/30/2022