Periodic Reporting for period 2 - EVENFLOW (Robust Learning and Reasoning for Complex Event Forecasting)
Berichtszeitraum: 2024-04-01 bis 2025-12-31
EVENFLOW develops hybrid learning techniques for complex event forecasting, which combine deep learning with logic-based learning and reasoning into neuro-symbolic forecasting models. This approach combines neural representation learning techniques that construct event-driven features from streams of perception-level data with powerful symbolic learning and reasoning tools, which utilize such features to synthesize high-level, interpretable patterns for forecasting critical events.
To deal with the brittleness of neural predictors and the high volume/velocity of temporal data flows, the EVENFLOW techniques rely on novel, formal verification techniques for machine learning, in addition to a suite of scalability algorithms for training based on data synopsis, federated training and incremental model construction. The learnt forecasters will be interpretable and scalable, allowing for explainable and robust insights, delivered in a timely fashion and enabling proactive decision making.
EVENFLOW is evaluated on three use cases related to (1) oncological forecasting in healthcare, (2) safe and efficient behaviour of autonomous transportation robots in smart factories and (3) reliable life cycle assessment of critical infrastructure.
To deal with the brittleness of neural predictors and the high volume/velocity of temporal data flows, the EVENFLOW techniques rely on novel, formal verification techniques for machine learning, in addition to a suite of scalability algorithms for training based on data synopsis, federated training and incremental model construction. The learnt forecasters will be interpretable and scalable, allowing for explainable and robust insights, delivered in a timely fashion and enabling proactive decision making.
EVENFLOW is evaluated on three use cases related to (1) oncological forecasting in healthcare, (2) safe and efficient behaviour of autonomous transportation robots in smart factories and (3) reliable life cycle assessment of critical infrastructure.
(a) Neurosymbolic learning (NeSy):
- Temporal NeSy Learning and Reasoning: NeSyA, a novel NeSy technique for seamlessly integrating perception neural networks with symbolic temporal knowledge.
- NeSy Event Pattern Learning: two novel techniques for event pattern learning.
(i) NeurASAL, a method for learning the logical structure of event patterns, while simultaneously training a neural component to map percepts to symbols that these patterns use;
(ii) ∂SFA, a fully differentiable approach to learning symbolic automata-based complex event patterns via gradient-based techniques.
- Data Programming for NeSy Training with Weak Supervision: techniques for augmenting the indirect supervision signal in temporal NeSy training settings, to improve the quality of the learned models, enable faster convergence and combat Reasoning Shortcuts.
- NeSy Complex Event Forecasting:
(i) Mutual Information Markov Models: a novel NeSy method for discovering discrete latent states and transition dynamics from high-dimensional Markov data (e.g. image streams), without reconstructing the observations. It enables event forecasting by using probabilistic model checking to answer forecasting queries.
(ii) Forecasting as Forward Recognition: a novel approach where a generative model is trained to generate likely continuations of a signal (video, time series, multivariate sensor sequences) and a symbolic automaton is run over these continuations, with the matches amounting to the forecasts.
(iii) NeSy pipelines where a perception NN outputs symbols from raw data, which are subsequently passed to the Wayeb CEF system.
- Application to the EVENFLOW use cases: the aforementioned techniques were successfully applied to all three of the project's use cases.
(b) Formal Verification:
- Novel, expressive loss functions for verified training expressive loss functions (CC-IBP and MTL-IBP) that enable a smooth transition from adversarial training objectives to verified robustness objectives.
- Robustness Verification of NeSy Systems: novel techniques for formally verifying if perturbations in the input of a NeSy system affect the output of the reasoning layer.
- Verification Strategies for Complex NeSy and Probabilistic NeSy Systems: devised verification strategies for complex NeSy architectures using abstract verification techniques, including Interval Bound Propagation (IBP), CROWN-based methods and Linear relaxation techniques.
- Temporal and Sequence Model Verification: evaluated robustness verification techniques for temporal and sequence models, focusing on video classification systems.
- Exact Verification Techniques: improved the Venus verification toolkit for ReLU-based feed-forward neural networks by incorporating MILP-based verification enhancements and dependency cuts.
- Distributional Robustness: evaluated Probably Approximately Correct (PAC)-based bounds as an extended objective to provide robustness guarantees under assumptions about the underlying data distribution.
- Application to the project's use cases: evaluated the robustness of EVENFLOW NeSy models deployed in an Industry 4.0 use case scenarios.
(c) Scalable Training:
- Scalable synopsis-maintenance via parallel, tensor-compatible Synopsis Data Engine: the earlier framework (developed in RP1) was pushed toward end-to-end scalability by reimplementing synopsis derivation on Dask, to keep synopses maintenance parallel and tensor-compatible so that they are directly consumable by learning pipelines.
- Synopsis-driven training optimization (SuBiTO): a novel technique for making training adaptive to stream dynamics by jointly optimizing (i) synopsis/compression ratio, (ii) training effort (e.g. epochs), and (iii) neural architecture (via NAS).
- The distributed version of SuBiTO (Distribuito SuBiTO): a set of techniques for tuning the parallelism across the synopsis, learning and inference stages of the framework, scaling the computation to volatile statistical properties of the incoming streams.
- Scalable NeSy CER across IoT: NeuroFlinkCEP contributes scalable NeSy Complex Event Recognition by embedding neural simple-event inference into Flink CEP jobs.
- RATS+ (Resource Allocator for Tumor Simulations enriched with transfer-learning).
- SSTRESED (scalable semantic trajectory extraction and online SDE detection over movement streams) scales durative simple/derived-event detection over high-rate robot movement streams by splitting the workload into two independently scalable pipelines.
- Application to the EVENFLOW use cases: all these techniques were applied to the project's use cases.
(d) Use cases:
- Personalized Medicine:
(i) paediatric cancer and in particular medulloblastoma (MB): identification of new disease subgroups, demonstration of fairer patient stratification, in-silico validation using independent methods.
(ii) Kidney Renal Clear Cell Carcinoma (KIRC): synthetic trajectories of early-to-late-stage transition, early detection of disease stage transitions from partial longitudinal data, interpretable temporal patterns auditable by clinical experts.
- Industry 4.0:
(i) Proactive Deadlock Avoidance: developed and validated a prototype neuro-symbolic controller that achieves a 99% elimination rate of deadlocks and collisions in controlled scenarios.
(ii) Significant Efficiency Gains: demonstrated a 49% improvement in mission completion times and average execution time per robot compared to standard ROS2 navigation baselines.
(iii) Optimized Navigation Performance: evaluations show that robots followed smoother paths, resulting in 24% reduction in total distance travelled and 90% reduction in the need for recovery actions.
- Infrastructure Life Cycle Assessment:
(i) Application of neuro-symbolic forecasting (Mutual Information Markov Models) to forecast water pipe leakage events.
(ii) Leakage detection with high accuracy even with sensors 10x apart.
- Temporal NeSy Learning and Reasoning: NeSyA, a novel NeSy technique for seamlessly integrating perception neural networks with symbolic temporal knowledge.
- NeSy Event Pattern Learning: two novel techniques for event pattern learning.
(i) NeurASAL, a method for learning the logical structure of event patterns, while simultaneously training a neural component to map percepts to symbols that these patterns use;
(ii) ∂SFA, a fully differentiable approach to learning symbolic automata-based complex event patterns via gradient-based techniques.
- Data Programming for NeSy Training with Weak Supervision: techniques for augmenting the indirect supervision signal in temporal NeSy training settings, to improve the quality of the learned models, enable faster convergence and combat Reasoning Shortcuts.
- NeSy Complex Event Forecasting:
(i) Mutual Information Markov Models: a novel NeSy method for discovering discrete latent states and transition dynamics from high-dimensional Markov data (e.g. image streams), without reconstructing the observations. It enables event forecasting by using probabilistic model checking to answer forecasting queries.
(ii) Forecasting as Forward Recognition: a novel approach where a generative model is trained to generate likely continuations of a signal (video, time series, multivariate sensor sequences) and a symbolic automaton is run over these continuations, with the matches amounting to the forecasts.
(iii) NeSy pipelines where a perception NN outputs symbols from raw data, which are subsequently passed to the Wayeb CEF system.
- Application to the EVENFLOW use cases: the aforementioned techniques were successfully applied to all three of the project's use cases.
(b) Formal Verification:
- Novel, expressive loss functions for verified training expressive loss functions (CC-IBP and MTL-IBP) that enable a smooth transition from adversarial training objectives to verified robustness objectives.
- Robustness Verification of NeSy Systems: novel techniques for formally verifying if perturbations in the input of a NeSy system affect the output of the reasoning layer.
- Verification Strategies for Complex NeSy and Probabilistic NeSy Systems: devised verification strategies for complex NeSy architectures using abstract verification techniques, including Interval Bound Propagation (IBP), CROWN-based methods and Linear relaxation techniques.
- Temporal and Sequence Model Verification: evaluated robustness verification techniques for temporal and sequence models, focusing on video classification systems.
- Exact Verification Techniques: improved the Venus verification toolkit for ReLU-based feed-forward neural networks by incorporating MILP-based verification enhancements and dependency cuts.
- Distributional Robustness: evaluated Probably Approximately Correct (PAC)-based bounds as an extended objective to provide robustness guarantees under assumptions about the underlying data distribution.
- Application to the project's use cases: evaluated the robustness of EVENFLOW NeSy models deployed in an Industry 4.0 use case scenarios.
(c) Scalable Training:
- Scalable synopsis-maintenance via parallel, tensor-compatible Synopsis Data Engine: the earlier framework (developed in RP1) was pushed toward end-to-end scalability by reimplementing synopsis derivation on Dask, to keep synopses maintenance parallel and tensor-compatible so that they are directly consumable by learning pipelines.
- Synopsis-driven training optimization (SuBiTO): a novel technique for making training adaptive to stream dynamics by jointly optimizing (i) synopsis/compression ratio, (ii) training effort (e.g. epochs), and (iii) neural architecture (via NAS).
- The distributed version of SuBiTO (Distribuito SuBiTO): a set of techniques for tuning the parallelism across the synopsis, learning and inference stages of the framework, scaling the computation to volatile statistical properties of the incoming streams.
- Scalable NeSy CER across IoT: NeuroFlinkCEP contributes scalable NeSy Complex Event Recognition by embedding neural simple-event inference into Flink CEP jobs.
- RATS+ (Resource Allocator for Tumor Simulations enriched with transfer-learning).
- SSTRESED (scalable semantic trajectory extraction and online SDE detection over movement streams) scales durative simple/derived-event detection over high-rate robot movement streams by splitting the workload into two independently scalable pipelines.
- Application to the EVENFLOW use cases: all these techniques were applied to the project's use cases.
(d) Use cases:
- Personalized Medicine:
(i) paediatric cancer and in particular medulloblastoma (MB): identification of new disease subgroups, demonstration of fairer patient stratification, in-silico validation using independent methods.
(ii) Kidney Renal Clear Cell Carcinoma (KIRC): synthetic trajectories of early-to-late-stage transition, early detection of disease stage transitions from partial longitudinal data, interpretable temporal patterns auditable by clinical experts.
- Industry 4.0:
(i) Proactive Deadlock Avoidance: developed and validated a prototype neuro-symbolic controller that achieves a 99% elimination rate of deadlocks and collisions in controlled scenarios.
(ii) Significant Efficiency Gains: demonstrated a 49% improvement in mission completion times and average execution time per robot compared to standard ROS2 navigation baselines.
(iii) Optimized Navigation Performance: evaluations show that robots followed smoother paths, resulting in 24% reduction in total distance travelled and 90% reduction in the need for recovery actions.
- Infrastructure Life Cycle Assessment:
(i) Application of neuro-symbolic forecasting (Mutual Information Markov Models) to forecast water pipe leakage events.
(ii) Leakage detection with high accuracy even with sensors 10x apart.
All results from (a), (b) and (c) in the "Work Performed and Main Achievements" section are beyond state of the art in the domains of temporal neuro-symbolic learning, reasoning and forecasting, formal neural verification and scalable training.