Combating cyber threats
Data mining is the process of discovering new patterns from large datasets. These patterns can be seen as a kind of summary of the data and they are used directly or in further analysis. One shortcoming of the current state of the art is that although there are well-understood techniques available for simple mining and summaries, computing more sophisticated stream data summaries that may differ in rate and volume remains a difficult problem. Addressing this complex issue was the objective of the 'Distributed multi-way analysis of stream data for detection of complex attacks' (DMASD4CA) project. This research considered ways to improve mining of complex data streams by studying data collection, analysis and knowledge extraction. The project showed that to obtain more accurate statistical and structural information, it was necessary to monitor and analyse data flows from multiple locations in a collaborative fashion. First, coordinated sampling was required in order to ensure independence among the agents — these are the programmes that collect the data. Secondly, online or near real-time versions of data analysis methods have to designed. The DMASD4CA project developed advanced techniques for analysing complex (multi-dimensional) data to determine the structure in the data. The resulting analysis method developed to detect malicious activity can be used to protect computer systems from attacks by hackers. It could also form the basis of an early-warning system to produce rapid responses to critical events, such as emerging disasters, epidemic outbreaks, or terrorist attacks.
