Better technology for Big Data
Did you know that the typical ISP network transfers terabytes of data every day? That today data connects nearly two billion people worldwide, and that by 2020 the number of internet-connected devices is expected to surpass 50 billion? What these big numbers mean is that the already large amount of data being today will only continue to grow at an exceptional speed. The challenge is to accurately identify and categorise this network traffic according to application type – a challenge taken up by the EU-funded ONTIC project. Their objective: to develop new techniques for analysing and characterising the large amounts of data traffic happening across today’s modern computing networks. ‘Accurate identification and categorisation of network traffic according to application type is an important element of many network management and engineering tasks related to Quality of Service (QoS), capacity planning and detecting network attacks,’ says ONTIC lead researcher Alberto Mozo. Investigate, implement and test According to ONTIC researchers, proactive and dynamic QoS management means being able to detect network intrusions and congestion problems early. To do this requires an accurate and scalable mechanism for providing an online characterisation of the evolution in network traffic patterns. The problem, however, is that current approaches for online network traffic characterisation lack scalability and accuracy. Here, ONTIC researchers saw an opportunity to develop a new generation of scalable mechanisms and techniques capable of characterising online network traffic. ‘Our objective was to investigate, implement and test a novel architecture of mechanisms and techniques to characterise online network traffic data streams and to detect anomalies in real-time, when a large volume of packets per second are processed,’ says Mozo. ‘Our data analysis techniques are intended to identify the recurring regularities found in descriptive models.’ The project also aimed to develop a new set of offline data mining mechanisms and techniques to characterise network traffic, apply a Big Data analytics approach and use distributed computation paradigms in the cloud on large data sets. At the same time, researchers integrated online and offline mechanisms and techniques into autonomous supervised or unsupervised network traffic. Focus on scalable algorithms Not wanting to re-invent the wheel, the project adopted an already existing architectural framework, namely, the Big Data Lambda Architecture. ‘We decided to focus the main part of our efforts on developing massively scalable algorithms that could be applied in the context of network traffic classification,’ says Mozo. ‘In sum, we concentrated our efforts not on the architecture, but on the design of novel algorithms and their application to prototypes for anomaly detection, proactive congestion control and dynamic QoS/QoE management.’ As the ONTIC project aimed at producing the knowledge that technology companies need to keep their data safe, it focused on producing high-quality scientific papers and prototypes as a way disseminating its findings (as opposed to producing actual products or services). One of its key outcomes was making the code for all ONTIC algorithms freely available via a GitLab repository under an open source license agreement. By the time the project concluded in January 2017, it had produced a .5 petabyte publicly-available dataset containing anonymised packet headers that can be valuable for other researchers. Researchers also developed three prototypes that demonstrated the applicability of parallel machine learning to the telecom domain, along with three inventions relating to these prototypes and one patent application. Furthermore, various leading technology companies, including Ericsson, Satec and CNRS, all plan to carry some of ONTIC’s inventions forward towards marketisation.
Keywords
ONTIC, ICT, computing, Big Data, telecoms
