Goals, and technical approach of the project
Results on MPEG-4 tools and systems development, impact on MPEG-4 technology
Applications description and trials results
Foreseen industrial valorisation of the technology
EMPHASIS started in October 1995 and was successfully completed in October 1998. The partners were SGS-Thomson Microelectronics (UK) which was co-ordinator during the first two years of the project, SRF/PACT (UK), CSELT (Italy), HHI (Germany), University of Hannover (Germany), EPFL (Switzerland), FZI (Germany), ENST (France), CNET (France), Studer Professional Audio (Switzerland), Siemens (Germany), LEP-Philips (France) which was co-ordinator during the third year, Matra Nortel Communications (France) and TAO Group (UK).
The main goal of the EMPHASIS project was to firmly establish a European lead in software and silicon technology suitable for meeting the needs of MPEG-4. In so doing, EMPHASIS played a very active role in the definition of the MPEG-4 standard from an implementation realisation perspective, thereby bridging the gap between the MPEG-4 related hardware and software developments. More precisely, the objectives of EMPHASIS were to contribute to the following areas:
- Systems Architecture elements;
- Development of optimised MPEG-4 Audio and Video software decoders and implementation on a European processor;
- Audio & Video composition and rendering tools (2.5 & 3D);
- Specifications for processor and co-processor architectures that meet the processing demands of MPEG-4 applications;
- Integrated demonstration of a typical MPEG-4 application showing real time audio and video decoding, object based interactivity and multi users functionality.
The technology developed by EMPHASIS will permit the Multimedia, telecommunications and broadcast industries to rapidly exploit the benefits provided by the MPEG-4 standard at the earliest opportunity. At the time of writing (August 99), direct exploitation of the results is under way.
The approach taken was the following. In a first phase, using a C/C++ model of MPEG-4 developed within the project, the processing requirements induced by the MPEG-4 related functions have been analysed, taking into consideration the available processors being developed at that time by major European and non European manufacturers. With these results, the need for processor architecture improvements and/or co-processor assistance has been assessed and MPEG-4 relevant hardware elements have been synthesised. In parallel, optimised software for MPEG-4 audio and video real time decoding, composition and rendering have been developed. Two versions have been produced: generically optimised (i.e. platform independent) and platform (Philips processor TriMedia) optimised. Several successful demonstrations of real time MPEG-4 multimedia decoding have been organised during the duration of the project at MPEG-4 meetings (Stockholm, Fribourg), at ECMAST 98 and IBC98 where for the first time, real time streaming of MPEG-4 content was shown.
EMPHASIS has been a key contributor in the development and promotion of MPEG-4-related technologies both on the hardware and software aspects. It has also clearly demonstrated in several major events (MPEG meetings, ECMAST, IBC) the application areas of this standard.
EMPHASIS has produced an impressive list of key achievements:
- EMPHASIS was the first to demonstrate, at several MPEG meetings, real time decoding of MPEG-4 streams. Two decoder software modules were produced one running on general purpose platforms , the other running on the Philips VLIW Media processor TriMedia. The high quality of the produced images and the possibilities offered by MPEG-4 attracted the interest of the Philips Business Unit Digital Video Systems which for the first time demonstrated at IBC98 a real time streaming of MPEG-4 multimedia over the Internet. The figure shows the architecture of the video decoder and Table 1 the performance of the decoder (UltraSparc 2 workstation at 200MHz running Solaris 2.5).
Software architecture of the EMPHASIS video decoder
| Bitstream |
akiyo |
stefan |
stefan |
hall |
| Format |
QCIF |
CIF |
CIF |
QCIF |
| Shape |
Binary |
Binary |
Binary |
Rect. |
| Options |
none |
none |
OBMC |
none |
| Decoded Frame Rate (frame/s) |
367.3 |
150.5 |
123.2 |
536.7 |
Video Decoder performance for different reference streams
- EMPHASIS also produced excellent quality audio decoder and compositors that were presented at MPEG meetings and conferences with great success. The audio compositor developed in EMPHASIS is now included into a commercial product sold by Studer, one of the EMPHASIS partners, specialised in high end professional audio equipment.
- Hardware implementation and the definition of suitable hardware architecture for MPEG-4 has been a central aspect of EMPHASIS. Although a complete MPEG-4 processor architecture was not developed as anticipated in the original phases (the approach was probably not realistic at that time), EMPHASIS has studied different co processor architectures able to accelerate key steps of MPEG-4 decoding: video composition and rendering, padding, and 3D spatial synthetic audio. This work was carried on to the synthesised VHDL stage for the rendering co-processor and forms the basis of IPRs to be exploited in particular by Siemens since the co-processor can in fact be interfaced with different main processors. The figure shows the block diagram of the co-processor. Key elements are: host interface via PI-Bus, external local object memory access via DMA interface, on-chip "cache" memories for object data, polynomial address decoder, 2-D filter kernel and Video output formatter. Simulations showed the possibility of object rendering (warping, perspective transformation) of two CCIR objects at 25 Hz.
Block diagram of the renderer co-processor
- EMPHASIS developed, demonstrated and profiled a 3D Player based on the latest version of the IM1 core and employing the EMPHASIS VFCD-04 compliant Video Decoder as well as a multichannel AAC Decoder. The 3D Player was capable of displaying MPEG-4 bit streams containing natural audio, natural video and synthetic 3D objects. The player supported some 30 BiFS nodes as well as extra features (outside of the current standard) such as user navigation. The player was integrated with a portable 3D transformation engine specifically developed for cost sensitive applications such as Digital TV.
- Finally, the project members strongly contributed to the advancement of the standard (in particular through the Implementation Study group chaired by a member of EMPHASIS) and to several key conferences and exhibitions. Very useful contributions on complexity evaluation have been provided by the consortium members. Detailed complexity analysis of the generically optimised decoder has been presented at the MPEG San Jose Conference. An "instrumentation" tool developed in previous part of EMPHASIS has been completed by developing a graphical interface for the display of results. The new tool, useful in particular for the architectural analysis of MPEG-4 components, has been provided to the MPEG community. Profiling results submitted in various documents (see the figure) to the MPEG Implementation Study Group have contributed to a deep insight analysis of MPEG-4 video complexity and generated results for the definition of complexity levels based on decoding complexity (see for instance document M 3615).
Example of time profiling (relative CPU time) of key MPEG-4 processing steps for several MPEG-4 reference sequences
The integration of the developed technologies into a demonstrator was a key work package of the project. Main steps were:
- A first demonstration of integration at MPEG meeting in Bristol MPEG (April 1997);
- A successful intermediate demonstration at ECMAST 98 Berlin;
- Integration of the software tools coming from systems, video and audio work packages for the final demonstration at IBC98.
To produce a common framework for integration, a visible demonstration of the results of the different EMPHASIS work items and to highlight MPEG-4 functionality, the following components were successfully integrated: Video decoder and compositor software, Audio decoder and compositor software, System software, and specific object based content created for this purpose. The underlying system was MPEG-4 Phase 1 IM1 oriented, to be as close as possible to the standardisation activities; the additional EMPHASIS TriMedia Hardware integration gave the system the necessary performance gain to enable real time video decoding within the PC application system. The figure below shows the architecture of the demonstrator. Another major Systems related achievement of the project was the implementation of DMIF (client&server) within a scenario for remote access of objects-on-demand. As there are only few people involved in the DMIF activity within MPEG this implementation experience helped the DMIF developments. A Server (including DMIF and MPEG-4 Systems layer) for streaming MPEG-4 coded media over IP has been developed. Through the combination of the system work and the real-time scheduler implemented in the demonstrator, EMPHASIS could provide a content processing chain up to real-time serving of MPEG-4 coded audio-visual media over an IP based LAN.
Architecture of the EMPHASIS final demonstration
Exploitation and valorisation by Philips: MPEG-4 streaming
The Philips Digital Video Business Unit Systems demonstrated its involvement in MPEG-4 by giving the first live demonstration of MPEG-4 video streaming over IP on its stand at IBC98 (September 98). The demonstration involved the streaming of video content from a server located in Luxembourg, via satellite, in a video-on-demand application. The MPEG-4 video decoder used in this successful demonstration is the generically optimised decoder developed in EMPHASIS.
Among the many practical benefits of MPEG-4 Philips Digital Video Systems has identified the fact that it allows the transmission of video over all IP links at various bit rates, including telephone, ISDN, cable or satellite. This means that consumers can enjoy streaming video on their PCs. If PC users are receiving video via the normal telephone lines, it appears in a small window on the PC screen. However, if users are receiving video via satellite, the increased bandwidth means that streaming video extends over the full screen, as demonstrated at the IBC.
In this demonstration, the possibility offered by the MPEG-4 standard of multimedia streaming at various bit rates from very low ones (a few tens of kbit/s) to medium ones (a few hundreds of kbit/s) was exploited to deliver video and audio data (see the figure) over different platforms (satellite link, LAN, Internet).
MPEG-4 streaming demonstration platforms at the Philips Booth at IBC 98
Further investigations are now under way to exploit the other features of MPEG-4 , especially the object based interactivity. The TriMedia implementation will also be used.
Gerrit Niemeijer, General Manager of Nettec, a new line of Business of Philips Digital Video Systems' commented: "MPEG-4 will play a leading role in Internet broadcasting. This new standard in audio and video compression and encoding offers significant advantages over previous encoding protocols. It enables streaming video via the Internet independent of the access medium. The MPEG-4 technique offers an incredible degree of customisation of an image. For example, during the televising of sporting events, advertising boards in an arena could be made flexible to give a different message to different target audiences. The possibilities are enormous and with the availability of MPEG-4, we can expect a whole new exciting period in multimedia creativity to dawn."
A further demonstration of the technology took place at the last NAB exhibition (Las Vegas, April 1999).
Exploitation and valorisation by Siemens : hardware modules for MPEG-4 composition
Major achievement of the hardware work within EMPHASIS is a fully functional and completely synthesised VHDL description of a MPEG-4 rendering co-processor (called TANGRAM), supporting also perspective transformation. This architecture represents an intellectual property (IP) that will be exploited by the partners of the project. In addition the individual modules of the co-processor can be reused for other applications and will therefore become the basis of a building box for future MPEG video decoding systems.
Siemens ZT ME 3 will use the EMPHASIS TANGRAM co-processor for rendering of arbitrary shaped objects in natural video scenes for future projects in MPEG-4 video decoding. As the current co-processor implementation is targeted to one specific MPEG-4 profile and level, it might not be appropriate for all future MPEG-4 applications. Nevertheless the modules and the global architecture will be modified and reused in upcoming systems.
Some contacts inside Siemens have been established to transfer the knowledge from Siemens ZT ME 3 to the semiconductors division (now INFINEON Technologies). The architectural proposals have been presented to the development group for the next generation of Siemens microcontrollers in San Jose. The TriCore processor seems to be very appropriate for real-time applications in future MPEG-4 based multimedia systems. Some results on the extension of the instruction set of RISC cores as well as the co-processor approach will be seriously considered. Also the newly funded Siemens design centre in Singapore sets one focus on MPEG-4 decoders and is in contact with the group at Siemens ZT ME 3.
INFINEON Technologies is also verifying which hardware modules from the EMPHASIS project can be reused for the future product lines in the audio-visual domain. Digital TV is becoming a major application area and will be the driver for the strong exploitation of the EMPHASIS hardware results.
Exploitation and valorisation by Matra Nortel Communications:Optimised MPEG-4 audio decoder
During EMPHASIS, two important parts of the natural audio decoders included in the MPEG-4 standards have been generically optimized: the Advanced Audio Coding scheme (AAC) and the parametric audio tool "Harmonic and Individual Lines and Noise" (HILN). Excellent results were obtained on the parametric audio decoder which now runs nearly three times faster than the original Reference Software. Results of this optimisation have been presented at the 43
rd MPEG meeting (input document M3293). This document also provided meaningful figures for the complexity evaluation of MPEG-4 Audio since they represented the first results of a practical complexity evaluation based on a generically optimised decoder.
This work was completed by a platform dependent optimisation (namely on TRIMEDIA TM1000 from Philips) of the parametric MPEG-4 audio tool (HILN). Starting from the generically optimised software that used more than 15 % of the TriMedia CPU, the current software only uses about 6-7% of the CPU.
Exploitation and evaluation by Studer Professional Audio: 3D audio compositor
Within the EMPHASIS project, basic research has been conducted concerning binaural and multichannel 3D audio composition and rendering schemes. This led to the development of new software tools and algorithms. The results have been presented at international scientific conferences and trade shows. The software has been successfully integrated by Studer into a new large digital mixing console (Studer D950S) under the trademark „Virtual Surround Panning" (VSP). The product has been successfully introduced to the market and reviewed in several magazines specialized on professional audio business. Other stand-alone products are planned or already in preparation. The scheme has been applied for patent (US patent application P16148.S02: Process and Device for Mixing Sound Signals).
Moreover, the expertise and knowledge about the coming MPEG-4 standardthat was attained by participation at MPEG meetings and contacts with other partners within the EMPHASIS group and the MPEG community, will be exploited in order to advise potential clients. This will contribute to a fast acceptance of the new standard, in particular by the broadcasting market.
MPEG-4 related software tools will be applied workstations under development for the creation of audio/video content, for example DVD authoring software, which will be part of the business over the next years.
VIDAS >>
