PhD Dissertation Defense by Sougata SEN

Fusing Mobile, Wearable and Infrastructure Sensing for Immersive Daily Lifestyle Analytics

With the prevalence of sensors in public infrastructure as well as in personal devices, exploitation of data from these sensors to monitor and profile basic activities (e.g., locomotive states such as walking or running and gestural actions such as smoking) has gained popularity. Basic activities identified by these sensors will drive the next generation of lifestyle monitoring systems and services. To provide more advanced and personalized services, these next-generation systems will need to capture and understand increasingly finer-grained details of a variety of common daily life activities.

In this dissertation defense presentation, I shall demonstrate the possibility of building systems using off-the-shelf devices, that not only identify activities, but also provide fine-grained details about an individual's lifestyle, using a combination of multiple sensing modes. These systems utilise sensor data from personal as well as infrastructure devices to unobtrusively monitor the daily life activities. In this presentation, I will use eating and shopping as two examples of daily life activities and will show the possibility to monitor fine-grained details of these activities.

I shall present two systems that we have developed to monitor fine-grained details of activities: (1) Annapurna – a system which uses multiple sensor classes (specifically inertial and image sensing) on wearable devices to capture novel context about common gesture-driven activities, and (2) I4S – a system which combines sensing data from not just multiple personal devices (smartwatch and smartphone), but also by using inexpensive ambient sensors/IoT platforms to monitor an activity which involves both gestural interactions as well as physical movement. In addition to the two systems, I shall also describe a technique, CROSDAC, which captures of an individual's physical activities to infer higher-level, cognitive characteristics associated with daily life activities.