PhD Dissertation Defense by GOKARN Ila Nitin | Enabling Criticality-Aware Optimized Machine Perception at the Edge

ABOUT THE TALK

Cyber-physical systems and applications have fundamentally changed people and processes in the way they interact with the physical world, ushering in the fourth industrial revolution. Supported by a variety of sensors, hardware platforms, artificial intelligence and machine learning models, and systems frameworks, CPS applications aim to automate and ease the burden of repetitive, laborious, or unsafe tasks borne by humans. Machine visual perception, encompassing tasks such as object detection, object tracking and activity analysis, is a key technical enabler of such CPS applications. Efficient execution of such machine vision perception tasks on resource-constrained edge devices, especially in terms of ensuring both high fidelity and processing throughput, remains a formidable challenge. This is due to the continuing increase in resolution of sensor streams (e.g., video input streams generated by 4K/8K cameras and high-volume event streams generated by emerging neuromorphic event cameras) and the computational complexity of the Deep Neural Network (DNN) models that underpin such perception capabilities, which overwhelms edge platforms, adversely impacting machine perception efficiency. This challenge is even more severe when a perception pipeline operating on a single edge device must process multiple concurrent video streams for accurate sense-making of the physical world. 

This thesis introduces the paradigm of Canvas-based Processing and Criticality Awareness to tackle the challenge of multi-sensor machine perception pipelines on resource-constrained platforms. The proposed paradigm guides perception pipelines and systems on "what" to pay attention to in the sensing field and "when", across multiple camera streams, to significantly increase both perception fidelity under computational constraints and achievable system throughput on a single edge device. With multiple strategies for fine-tuning such a perception pipeline for real-world deployment characteristics, this thesis demonstrates that it is possible to achieve multiplicative gains in processing throughput with no cost to DNN task accuracy, across multiple concurrent RGB and event camera streams at the resource-constrained edge.

ABOUT THE SPEAKER

Ila is a fifth-year PhD candidate in Computer Science, advised by Prof. Archan Misra. She works primarily in the field of pervasive systems and sensing with a focus on edge computing paradigms for machine visual perception. During her PhD, she was accepted into the Rising Star Forum at Mobisys 2024, authored a book chapter and several research papers, and won two Best Research Demo awards at COMSNETS.