Final Report Summary - INVENTHI (Inventory-driven speech enhancement for binaural hearing instruments)
The INVENTHI project was a joint effort between the incoming international fellow assistant prof. Robert M. Nickel from Bucknell University, Lewisburg, Pensylvania, United States of America (USA) and the prof. Dr Ing. Rainer Martin, who is the director of the Institute of Communication Acoustics (IKA) of the Ruhr-Universität Bochum (RUB) in Germany. The goal of the project was the development and evaluation of advanced strategies for the improvement of hearing instruments for the hearing-impaired. A significant problem of state-of-the-art hearing instrument technology is that the removal of additive noise and reverberations alone can only partially restore the ability of a hearing-impaired person to exercise auditory focus and sound selectivity in complex acoustic scenes. The joint project was devoted to taking significant steps towards the development of technical solutions that provide the much needed auditory focus and sound selectivity by computational means.
The total length of the project was twelve month during which prof. Nickel was a guest of the IKA at the Ruhr-Universität Bochum. The technical work of the project was divided into seven subtasks. Tasks number (1), (2), and (3) consisted of the organisation and acquisition of the necessary research data, the development of an algorithmic test-bed, and the implementation of a suitable enhancement baseline system. Professor Nickel worked with a student, Simon Brandt, on the recording of the necessary acoustic datasets. The recording involved a manikin-head, hearing aid chassis with three microphones at each ear, multichannel recording equipment and a variety of reverberant environments including an anechoic chamber. The programming of the algorithmic test-bed and the development of the evaluation software were jointly accomplished by prof. Nickel and Mr Brandt. All datasets and software are available from the IKA (see contact information and web-site from above). Mr Brandt satisfied the thesis-requirement of his Dipl. Ing. Degree with his work on the project. Prof. Nickel served as his technical advisor. The title of Mr Brandt's thesis was (translated) 'Multichannel Dereverberation of Speech Signals via the Exploitation of Specific Signal Characteristics'.
Task number (4) of the project was concerned with the latency reduction of the considered algorithm. The latency problem was tackled with a sub-optimal search approach that was developed by prof. Nickel in earlier work. It turned out that an essential prerequisite to a successful implementation of the targeted multi-speaker enhancement system (originally proposed in tasks (5) and (7)) was an investigation on how to first reduce the memory and complexity of the considered single-speaker implementation. The memory and complexity reduction (originally scheduled in task (6)) proved to be significantly more difficult than anticipated. Nevertheless, a successful solution that enabled a dramatic reduction in both memory and complexity was developed and presented at the 19th European Signal Processing Conference (EUSIPCO 2012) in Barcelona, Spain. The title of the paper was 'Memory and Complexity Reduction for Inventory Style Speech Enhancement Systems'. The EUSIPCO conferences are considered premier events for the international dissemination of cutting-edge scientific work. Due to the additional time that had to be devoted to task (6) we were, unfortunately, unable to complete the work on the speaker independent approach. Nevertheless, a thorough performance evaluation of the developed procedures was performed (task (8)). A subset of the results will be presented at the upcoming 36th International Conference on Acoustics, Speech and Signal Processing (ICASSP 2012) in Kyoto, Japan. The title of the paper is 'Inventory-Style Speech Enhancement with Uncertainty-of-Observation Techniques'. The conference is scheduled for March 2012. The ICASSP conferences belong to the most prestigious international conferences in audio and speech signal processing. A significantly extended version of the ICASSP paper is currently in preparation and will be submitted to the IEEE Transactions on Audio, Speech, and Language Processing, which is, also, one of the most prominent technical journals in the field.
A transfer of knowledge from the incoming international fellow to the host institution of the project was accomplished via the many collaborative interactions between Prof. Nickel and the researchers and students of the Institute of Communication Acoustics (IKA) at the RUB. On 17 November 2010, prof. Nickel presented a two-part seminar at the IKA. The title of the first part was 'Inventory Based Speech Processing: Enhancement and Coding' and the topic of the second part was 'Number-Theoretic Transforms and Applications to Fast Algoritms'. On 14 April 2011, prof. Nickel was invited by prof. Dr Ing. Peter Vary to give a presentation about his work at the Institute of Communication Systems and Data Processing of the Rheinisch-Westfälische Technische Hochschule (RWTH) in Aachen, Germany. With prof. Vary he explored avenues for an exchange of electrical engineering students between Bucknell University and the RWTH. In September and October of 2010 prof. Nickel presented his work at the main conference of the International Speech Communication Association (InterSpeech 2010) in Makuhari, Japan, and at the 9th Speech Communication Conference of the Information Technology Society of the Association of German Engineers (VDE/ITG-Fachtagung) in Bochum, Germany. In February 2011, prof. Nickel participated in a meeting/workshop of the Marie Curie ITN AUDIS project which was specifically devoted to research and training in digital signal processing for applications in hearing instruments. He also attended the subsequent 38th Erlanger Kolloquium for Scientists and Developers in Audiology which was hosted by the Siemens Audiologische Technik GmbH Research Headquarters in Erlangen, Germany. In addition, Prof. Nickel attended the 35th International Conference on Acoustics, Speech and Signal Processing (ICASSP 2011) in Prague, Czech Republic, in May 2011. A potential for long-term collaborations was established with a few scientists and researchers at the host institution of the project. Prof. Nickel is currently collaborating with Prof. Dr Ing. Dorothea Kolossa, Dr Ing. Steffen Zeiler, and prof. Dr Ing. Rainer Martin to extend the algorithms and methods developed in the InventHI project to scenarios in which acoustic information is fused with visual facial information. Visual facial cues are important features that can help the hearing-impaired to regain auditory focus and sound selectivity. The goal of the collaboration is to explore how such visual cues could be exploited in the auditory enhancement process with computational means. In addition, prof. Nickel has begun to involve Dr Ing. Gerald Enzner (Akademischer Rat) and Dipl. Ing. Dominic Schmid, both from the IKA in Bochum, in talks about collaborative work on general multichannel extensions of the project work to solve problems of acoustic blind channel identification for speech signals.
As such, the project has not only resulted in the development of powerful new methods for the enhancement of distorted speech signals but also in a substantial exchange of specialty knowledge between the fellow, the host institution, and many other German and European researchers and scientists, as well as the creation of very promising new international research collaborations. The funding provided by the Community Research and Development Information Service (CORDIS) Marie Curie Programme was a key to the project, without which the project would not have been possible and successful. The fellow, prof. Nickel, as well as the host, Prof. Dr Ing. Rainer Martin, would both like to thank the EU and the Research Executive Agency for their dedicated support.
The total length of the project was twelve month during which prof. Nickel was a guest of the IKA at the Ruhr-Universität Bochum. The technical work of the project was divided into seven subtasks. Tasks number (1), (2), and (3) consisted of the organisation and acquisition of the necessary research data, the development of an algorithmic test-bed, and the implementation of a suitable enhancement baseline system. Professor Nickel worked with a student, Simon Brandt, on the recording of the necessary acoustic datasets. The recording involved a manikin-head, hearing aid chassis with three microphones at each ear, multichannel recording equipment and a variety of reverberant environments including an anechoic chamber. The programming of the algorithmic test-bed and the development of the evaluation software were jointly accomplished by prof. Nickel and Mr Brandt. All datasets and software are available from the IKA (see contact information and web-site from above). Mr Brandt satisfied the thesis-requirement of his Dipl. Ing. Degree with his work on the project. Prof. Nickel served as his technical advisor. The title of Mr Brandt's thesis was (translated) 'Multichannel Dereverberation of Speech Signals via the Exploitation of Specific Signal Characteristics'.
Task number (4) of the project was concerned with the latency reduction of the considered algorithm. The latency problem was tackled with a sub-optimal search approach that was developed by prof. Nickel in earlier work. It turned out that an essential prerequisite to a successful implementation of the targeted multi-speaker enhancement system (originally proposed in tasks (5) and (7)) was an investigation on how to first reduce the memory and complexity of the considered single-speaker implementation. The memory and complexity reduction (originally scheduled in task (6)) proved to be significantly more difficult than anticipated. Nevertheless, a successful solution that enabled a dramatic reduction in both memory and complexity was developed and presented at the 19th European Signal Processing Conference (EUSIPCO 2012) in Barcelona, Spain. The title of the paper was 'Memory and Complexity Reduction for Inventory Style Speech Enhancement Systems'. The EUSIPCO conferences are considered premier events for the international dissemination of cutting-edge scientific work. Due to the additional time that had to be devoted to task (6) we were, unfortunately, unable to complete the work on the speaker independent approach. Nevertheless, a thorough performance evaluation of the developed procedures was performed (task (8)). A subset of the results will be presented at the upcoming 36th International Conference on Acoustics, Speech and Signal Processing (ICASSP 2012) in Kyoto, Japan. The title of the paper is 'Inventory-Style Speech Enhancement with Uncertainty-of-Observation Techniques'. The conference is scheduled for March 2012. The ICASSP conferences belong to the most prestigious international conferences in audio and speech signal processing. A significantly extended version of the ICASSP paper is currently in preparation and will be submitted to the IEEE Transactions on Audio, Speech, and Language Processing, which is, also, one of the most prominent technical journals in the field.
A transfer of knowledge from the incoming international fellow to the host institution of the project was accomplished via the many collaborative interactions between Prof. Nickel and the researchers and students of the Institute of Communication Acoustics (IKA) at the RUB. On 17 November 2010, prof. Nickel presented a two-part seminar at the IKA. The title of the first part was 'Inventory Based Speech Processing: Enhancement and Coding' and the topic of the second part was 'Number-Theoretic Transforms and Applications to Fast Algoritms'. On 14 April 2011, prof. Nickel was invited by prof. Dr Ing. Peter Vary to give a presentation about his work at the Institute of Communication Systems and Data Processing of the Rheinisch-Westfälische Technische Hochschule (RWTH) in Aachen, Germany. With prof. Vary he explored avenues for an exchange of electrical engineering students between Bucknell University and the RWTH. In September and October of 2010 prof. Nickel presented his work at the main conference of the International Speech Communication Association (InterSpeech 2010) in Makuhari, Japan, and at the 9th Speech Communication Conference of the Information Technology Society of the Association of German Engineers (VDE/ITG-Fachtagung) in Bochum, Germany. In February 2011, prof. Nickel participated in a meeting/workshop of the Marie Curie ITN AUDIS project which was specifically devoted to research and training in digital signal processing for applications in hearing instruments. He also attended the subsequent 38th Erlanger Kolloquium for Scientists and Developers in Audiology which was hosted by the Siemens Audiologische Technik GmbH Research Headquarters in Erlangen, Germany. In addition, Prof. Nickel attended the 35th International Conference on Acoustics, Speech and Signal Processing (ICASSP 2011) in Prague, Czech Republic, in May 2011. A potential for long-term collaborations was established with a few scientists and researchers at the host institution of the project. Prof. Nickel is currently collaborating with Prof. Dr Ing. Dorothea Kolossa, Dr Ing. Steffen Zeiler, and prof. Dr Ing. Rainer Martin to extend the algorithms and methods developed in the InventHI project to scenarios in which acoustic information is fused with visual facial information. Visual facial cues are important features that can help the hearing-impaired to regain auditory focus and sound selectivity. The goal of the collaboration is to explore how such visual cues could be exploited in the auditory enhancement process with computational means. In addition, prof. Nickel has begun to involve Dr Ing. Gerald Enzner (Akademischer Rat) and Dipl. Ing. Dominic Schmid, both from the IKA in Bochum, in talks about collaborative work on general multichannel extensions of the project work to solve problems of acoustic blind channel identification for speech signals.
As such, the project has not only resulted in the development of powerful new methods for the enhancement of distorted speech signals but also in a substantial exchange of specialty knowledge between the fellow, the host institution, and many other German and European researchers and scientists, as well as the creation of very promising new international research collaborations. The funding provided by the Community Research and Development Information Service (CORDIS) Marie Curie Programme was a key to the project, without which the project would not have been possible and successful. The fellow, prof. Nickel, as well as the host, Prof. Dr Ing. Rainer Martin, would both like to thank the EU and the Research Executive Agency for their dedicated support.
