Periodic Reporting for period 1 - V.I.P. (MassiVe MIMO Radio Channel- and Learning-based Indoor Passive Localization and Posture Identification for Multiple Humans)
Reporting period: 2021-11-01 to 2023-10-31
This project addresses the contact-free human sensing using advanced multi-antenna radio systems and advanced signal/data processing techniques. Going beyond the wearable solutions where the persons need to carry or wear devices on body, contact-free solutions have environment-embedded devices and add no stress to the persons of interest. Radio frequency contact-free devices, including radars, can operate regardless of the lighting conditions. As civilian radars become more accessible, it is envisioned that radar sensing will be integrated into communication infrastructure for 6G and next generation radio networks. V.I.P. project utilizes multi-antenna radio systems with mainly communication (and also radar) waveforms to perform human sensing. The sensing tasks include detection, localization and tracking, pose recognition and breathing estimation of one or more persons. The overall objectives is to optimize the multi-antenna system topology/configuration and devise algorithms for accurate contact-free (multi-)human sensing. The results can be utilized in remote health monitoring applications and provide proactive and continuous monitoring; this is socially important because our society faces tremendous challenges in securing the future wellness of a large segment of its population requiring special attentions, e.g. the elderly, the physically impaired and the children. Relevant results are expected to have a great economic impact on the EU's future advancement.
There are in total three coherent work packages (WP) in V.I.P.
WP1 is to build simulation and measurement platforms for site-specific massive multi-input multi-output (MIMO) optimization. I first worked on optimizing the focusing performance of non-ideal cell-free MIMO using generic algorithm. This work is based on simulation, and laid foundation for MIMO topology optimization and/or antenna selection for communication purposes. This work is also transferrable to sensing purposes by altering the goal functions to sensing performance metrics. Second, together with a PhD, we extend the MIMO work by taking into account the channel aging effect in dynamic scenarios, to satisfy the coverage and capacity requirement for football player's bodycam with cell-free massive MIMO. Third, zooming in from the mMIMO topology, I focused on single base station (BS) and multi-beam strategy for integrated sensing and communications. With dedicated beam profile for MIMO BS, we can predict in advance before a walking person could block the directive mmWave communication link. This is beneficial both for the communication quality and is also for the sake of the walking human (less exposure to mmWave links). Forth, I extend the work to the user equipment (UE) side, and I have a published work on how to utilize true-time-delay arrays at UE and exploit the 5G-NR synchronization blocks for bistatic sensing investigation, e.g. to detect potential human blocker around the UE.
WP2 is to measure, model and parameterize human form-factors. I started with modelling the human-radio interactions, by first considering human as a finite-length cylinder and using the modified diffraction model composing of specular reflected waves, creeping waves and edge diffracted waves. Later, together with a visiting PhD, we extended the model to sophisticated human model by considering ellipsoidal limbs or body parts. We also were able to verify the model via a set of measurement conducted in Japan while my visit in summer 2023. This human limb model and its interaction with radio waves from different link situations of MIMO are further used for sensing estimation purposes. Moreover, to study the concurrent scattering and blockage effects of human bodies in multi-link distributed MIMO scenarios, I designed measurement methodology of temporal-spatially synchronized point cloud, image and radio signals at dual bands (FR2 and FR3 of 5G NR), with extended collaboration with Japanese partners. I also coordinated a measurement campaign with co-existing communication and radar devices. The goal was to investigate if radar sensing of human activity (human forms) could be beneficial for communication reliability, or if communication could be beneficial for human sensing, how can radar and communication devices help each other.
WP3 is to design multi-human localization and posture identification algorithms. First, I participated in a measurement campaign of a visiting PhD. We did measurement with two persons walking in a dedicated controlled environment, with distributed MIMOs surrounding the area of interest. It is worth mentioning that the distributed MIMO systems are with OFDM waveforms whose parameters follow the cellular communication standard (so, not optimized for sensing purposes). But with dedicated algorithms, we were able to localize and track the persons with decent accuracy. With limited resources, the accuracy of tracking 1 person is better than that of 2 persons.
In addition to multi-person localization and tracking, I also initiated and led the work of breathing/vital activity recognition using MIMO systems. This first resulted in a master thesis, where we were the first using a 26 GHz multi-beam communication testbed with reconfigurable link configuration for 1-2 persons human breathing estimation.
I also explored the combination of posture and breathing estimation, where FMCW MIMO radar was used for capturing point clouds of standing breathing person. First, from the point cloud generated from multiple frames of the radio signal, we estimate the human posture, and then by beamforming towards the human chest, we can retrieve the breathing pattern more accurately than that without chest-beamfocusing. In addition to above model-drive approaches, I also dived into data-driven approaches using machine learning, via collaborated work on channel prediction using deep learning of point cloud and radio data.
WP1 is to build simulation and measurement platforms for site-specific massive multi-input multi-output (MIMO) optimization. I first worked on optimizing the focusing performance of non-ideal cell-free MIMO using generic algorithm. This work is based on simulation, and laid foundation for MIMO topology optimization and/or antenna selection for communication purposes. This work is also transferrable to sensing purposes by altering the goal functions to sensing performance metrics. Second, together with a PhD, we extend the MIMO work by taking into account the channel aging effect in dynamic scenarios, to satisfy the coverage and capacity requirement for football player's bodycam with cell-free massive MIMO. Third, zooming in from the mMIMO topology, I focused on single base station (BS) and multi-beam strategy for integrated sensing and communications. With dedicated beam profile for MIMO BS, we can predict in advance before a walking person could block the directive mmWave communication link. This is beneficial both for the communication quality and is also for the sake of the walking human (less exposure to mmWave links). Forth, I extend the work to the user equipment (UE) side, and I have a published work on how to utilize true-time-delay arrays at UE and exploit the 5G-NR synchronization blocks for bistatic sensing investigation, e.g. to detect potential human blocker around the UE.
WP2 is to measure, model and parameterize human form-factors. I started with modelling the human-radio interactions, by first considering human as a finite-length cylinder and using the modified diffraction model composing of specular reflected waves, creeping waves and edge diffracted waves. Later, together with a visiting PhD, we extended the model to sophisticated human model by considering ellipsoidal limbs or body parts. We also were able to verify the model via a set of measurement conducted in Japan while my visit in summer 2023. This human limb model and its interaction with radio waves from different link situations of MIMO are further used for sensing estimation purposes. Moreover, to study the concurrent scattering and blockage effects of human bodies in multi-link distributed MIMO scenarios, I designed measurement methodology of temporal-spatially synchronized point cloud, image and radio signals at dual bands (FR2 and FR3 of 5G NR), with extended collaboration with Japanese partners. I also coordinated a measurement campaign with co-existing communication and radar devices. The goal was to investigate if radar sensing of human activity (human forms) could be beneficial for communication reliability, or if communication could be beneficial for human sensing, how can radar and communication devices help each other.
WP3 is to design multi-human localization and posture identification algorithms. First, I participated in a measurement campaign of a visiting PhD. We did measurement with two persons walking in a dedicated controlled environment, with distributed MIMOs surrounding the area of interest. It is worth mentioning that the distributed MIMO systems are with OFDM waveforms whose parameters follow the cellular communication standard (so, not optimized for sensing purposes). But with dedicated algorithms, we were able to localize and track the persons with decent accuracy. With limited resources, the accuracy of tracking 1 person is better than that of 2 persons.
In addition to multi-person localization and tracking, I also initiated and led the work of breathing/vital activity recognition using MIMO systems. This first resulted in a master thesis, where we were the first using a 26 GHz multi-beam communication testbed with reconfigurable link configuration for 1-2 persons human breathing estimation.
I also explored the combination of posture and breathing estimation, where FMCW MIMO radar was used for capturing point clouds of standing breathing person. First, from the point cloud generated from multiple frames of the radio signal, we estimate the human posture, and then by beamforming towards the human chest, we can retrieve the breathing pattern more accurately than that without chest-beamfocusing. In addition to above model-drive approaches, I also dived into data-driven approaches using machine learning, via collaborated work on channel prediction using deep learning of point cloud and radio data.
Beyond the state of the art aspects in V.I.P. results are summarized below.
1) MIMO topology optimization and/or antenna selection using generic algorithms (a stochastic optimizer). The proposed framework can be used either for communication or for sensing performance optimization.
2) MIMO beam profile optimization at base station and user equipment sides for integrated sensing and communications.
3) finite cylinder diffraction model & ellipsoidal limb/torso model for mimicking radio-human interactions.
4) measuring concurrent scattering and blockage effects of human bodies in multi-link distributed MIMO scenarios at dual bands using point cloud image groundtruth.
5) trials with co-existing communication and radar devices with simultaneous communications and human sensing.
6) up to 2 persons localization and tracking using radar and using OFDM communication MIMO devices.
7) breathing/vital activity recognition using the 26 GHz multi-beam communication testbed, and using cell-free MIMO.
8) posture recognition and torso detection for improved vital sign estimation using FMCW MIMO radar.
9) data-driven approaches using deep learning of point cloud and radio data for channel prediction.
The greatest impact is that it becomes clearer that advanced communication devices could be used for accurate human sensing purposes. This is important as people are more acceptable towards existing infrastructure (cellular and Wi-Fi devices are everywhere).
1) MIMO topology optimization and/or antenna selection using generic algorithms (a stochastic optimizer). The proposed framework can be used either for communication or for sensing performance optimization.
2) MIMO beam profile optimization at base station and user equipment sides for integrated sensing and communications.
3) finite cylinder diffraction model & ellipsoidal limb/torso model for mimicking radio-human interactions.
4) measuring concurrent scattering and blockage effects of human bodies in multi-link distributed MIMO scenarios at dual bands using point cloud image groundtruth.
5) trials with co-existing communication and radar devices with simultaneous communications and human sensing.
6) up to 2 persons localization and tracking using radar and using OFDM communication MIMO devices.
7) breathing/vital activity recognition using the 26 GHz multi-beam communication testbed, and using cell-free MIMO.
8) posture recognition and torso detection for improved vital sign estimation using FMCW MIMO radar.
9) data-driven approaches using deep learning of point cloud and radio data for channel prediction.
The greatest impact is that it becomes clearer that advanced communication devices could be used for accurate human sensing purposes. This is important as people are more acceptable towards existing infrastructure (cellular and Wi-Fi devices are everywhere).