Periodic Reporting for period 4 - Re-SENSE (RESOURCE-EFFICIENT SENSING THROUGH DYNAMIC ATTENTION-SCALABILITY)
Reporting period: 2021-09-01 to 2022-02-28
Our society is facing grand challenges in the coming decades. The growing world population necessitates more efficient use of Earth’s resources, be efficient in mobility and manufacturing. Sensing technology is in a unique position to aid in tackling all aforementioned societal challenges.
By being able to continuously sense, monitor and analyze a large amount of parameters in real time, smart objects can gain more knowledge on their environment and of their impact on this environment. Yet, current sensing technology is fundamentally held back by the amount of information it can extract from the environment given the limited amount of resources sensors have available. My goal is to develop a disruptive class of multi‐modal sensor processing platforms towards real‐time embedded sensor fusion in resource‐scarce devices.
To achieve this, we take inspiration from human sensing. We, humans, are masters in sensor fusion, as we can seamlessly combine information coming from different senses to improve our judgements. In order to effectively exploit the many sensory streams available to us, we however do not devote the same level of attention, or mental effort, to each of them. This dynamic attention-scalability allows us to always extract the maximum amount of relevant information under our limited human computational bandwidth.
Re-SENSE ambitions to bring this capability to electronic devices with limited computational resources. Specifically, dynamic attention-scalability in embedded sensing will be pursued by innovating on resource-aware sensor fusion algorithms, together with dynamic, wide-range resource-scalable sensor fusion processors. This synergistic interaction between machine learning and processor design promises at least an order of magnitude efficiency improvement in resource-scarce sensory systems for e.g. robotics or health care.
By being able to continuously sense, monitor and analyze a large amount of parameters in real time, smart objects can gain more knowledge on their environment and of their impact on this environment. Yet, current sensing technology is fundamentally held back by the amount of information it can extract from the environment given the limited amount of resources sensors have available. My goal is to develop a disruptive class of multi‐modal sensor processing platforms towards real‐time embedded sensor fusion in resource‐scarce devices.
To achieve this, we take inspiration from human sensing. We, humans, are masters in sensor fusion, as we can seamlessly combine information coming from different senses to improve our judgements. In order to effectively exploit the many sensory streams available to us, we however do not devote the same level of attention, or mental effort, to each of them. This dynamic attention-scalability allows us to always extract the maximum amount of relevant information under our limited human computational bandwidth.
Re-SENSE ambitions to bring this capability to electronic devices with limited computational resources. Specifically, dynamic attention-scalability in embedded sensing will be pursued by innovating on resource-aware sensor fusion algorithms, together with dynamic, wide-range resource-scalable sensor fusion processors. This synergistic interaction between machine learning and processor design promises at least an order of magnitude efficiency improvement in resource-scarce sensory systems for e.g. robotics or health care.
To pursue dynamic attention scalability, work has been performed on three fronts:
a.) We developed resource-aware sensor fusion algorithms. These are based on probabilistic graphical models (PGM), which are compiled into sum-product networks for execution on embedded processors. We developed a scalable PGM-based sensor fusion framework, which assesses the execution cost of our sensor fusion algorithms, and performs several pruning steps on the PGM models to create Pareto-optimal configurations from a resource-accuracy trade-off point of view.
b.) We developed processing architectures for executing the resulting probabilistic models in an embedded processor. This involves the execution of highly irregular acyclic graphs, which do not map well on classical CPU’s, GPU’s or NPU’s. A new type of custom processor is created: the RPU, or Reasoning processing unit. This processor is taped out, and benchmarked against CPU, and GPU. It is published in ISSCC and JSSC. A second version is also developed (not taped out) and send for publication in MICRO2022.
c.) We applied to aforementioned developments onto practical applications. We first assessed the application of rapid cell sorting. It was however discovered the the database of this cell sorting (received from Imec) was not sufficiently labeled to be able to use it for this purpose. A new database was also not planned to be released any time soon. We therefore switched to human activity monitoring as our first application case. We applied the developed frameworks on open source human activity databases, containing inertial sensing data. Additionally, we started to collect our own database with inertial sensing information combined with acoustic sensing data of human activities, to perform sensor fusion between these two domains. A smartphone app was developed to record this data with multiple users.
a.) We developed resource-aware sensor fusion algorithms. These are based on probabilistic graphical models (PGM), which are compiled into sum-product networks for execution on embedded processors. We developed a scalable PGM-based sensor fusion framework, which assesses the execution cost of our sensor fusion algorithms, and performs several pruning steps on the PGM models to create Pareto-optimal configurations from a resource-accuracy trade-off point of view.
b.) We developed processing architectures for executing the resulting probabilistic models in an embedded processor. This involves the execution of highly irregular acyclic graphs, which do not map well on classical CPU’s, GPU’s or NPU’s. A new type of custom processor is created: the RPU, or Reasoning processing unit. This processor is taped out, and benchmarked against CPU, and GPU. It is published in ISSCC and JSSC. A second version is also developed (not taped out) and send for publication in MICRO2022.
c.) We applied to aforementioned developments onto practical applications. We first assessed the application of rapid cell sorting. It was however discovered the the database of this cell sorting (received from Imec) was not sufficiently labeled to be able to use it for this purpose. A new database was also not planned to be released any time soon. We therefore switched to human activity monitoring as our first application case. We applied the developed frameworks on open source human activity databases, containing inertial sensing data. Additionally, we started to collect our own database with inertial sensing information combined with acoustic sensing data of human activities, to perform sensor fusion between these two domains. A smartphone app was developed to record this data with multiple users.
a.) At the algorithmic side, this is the first time that actual hardware execution cost and execution energy is taken into account in the modeling and optimization of probabilistic reasoning. Previously, the notion of tractability was limited to the amount of nodes in the network, which do not correlate well with execution cost in energy constrained edge devices. Our new framework changes this, and shows benefits in execution cost of embedded reasoning tasks of up to a factor 10x for equal task accuracy. This was honored by a publication in the NeurIPS conference, one of the most prestigious conferences in machine learning.
b.) At the hardware side, no suitable processor for such tasks currently exists. We developed the architecture for a new type of processors, which achieve a speedup of a factor 20 over current high-end GPUs and CPUs. This architecture has been presented in a poster at the prestigious HotChips conference. It has been taped out, published at ISSCC and JSSC.
c.) Several application domains have been tackles, such as activity monitoring using intertial sensors as well as machine monitoring based on acoustics. Due to our self-adaptive systems, we were able to achieve energy benefits of 2x and more. this has been submitted for publication.
b.) At the hardware side, no suitable processor for such tasks currently exists. We developed the architecture for a new type of processors, which achieve a speedup of a factor 20 over current high-end GPUs and CPUs. This architecture has been presented in a poster at the prestigious HotChips conference. It has been taped out, published at ISSCC and JSSC.
c.) Several application domains have been tackles, such as activity monitoring using intertial sensors as well as machine monitoring based on acoustics. Due to our self-adaptive systems, we were able to achieve energy benefits of 2x and more. this has been submitted for publication.