(a) Neurosymbolic learning (NeSy):
- Temporal NeSy Learning and Reasoning: NeSyA, a novel NeSy technique for seamlessly integrating perception neural networks with symbolic temporal knowledge.
- NeSy Event Pattern Learning: two novel techniques for event pattern learning.
(i) NeurASAL, a method for learning the logical structure of event patterns, while simultaneously training a neural component to map percepts to symbols that these patterns use;
(ii) ∂SFA, a fully differentiable approach to learning symbolic automata-based complex event patterns via gradient-based techniques.
- Data Programming for NeSy Training with Weak Supervision: techniques for augmenting the indirect supervision signal in temporal NeSy training settings, to improve the quality of the learned models, enable faster convergence and combat Reasoning Shortcuts.
- NeSy Complex Event Forecasting:
(i) Mutual Information Markov Models: a novel NeSy method for discovering discrete latent states and transition dynamics from high-dimensional Markov data (e.g. image streams), without reconstructing the observations. It enables event forecasting by using probabilistic model checking to answer forecasting queries.
(ii) Forecasting as Forward Recognition: a novel approach where a generative model is trained to generate likely continuations of a signal (video, time series, multivariate sensor sequences) and a symbolic automaton is run over these continuations, with the matches amounting to the forecasts.
(iii) NeSy pipelines where a perception NN outputs symbols from raw data, which are subsequently passed to the Wayeb CEF system.
- Application to the EVENFLOW use cases: the aforementioned techniques were successfully applied to all three of the project's use cases.
(b) Formal Verification:
- Novel, expressive loss functions for verified training expressive loss functions (CC-IBP and MTL-IBP) that enable a smooth transition from adversarial training objectives to verified robustness objectives.
- Robustness Verification of NeSy Systems: novel techniques for formally verifying if perturbations in the input of a NeSy system affect the output of the reasoning layer.
- Verification Strategies for Complex NeSy and Probabilistic NeSy Systems: devised verification strategies for complex NeSy architectures using abstract verification techniques, including Interval Bound Propagation (IBP), CROWN-based methods and Linear relaxation techniques.
- Temporal and Sequence Model Verification: evaluated robustness verification techniques for temporal and sequence models, focusing on video classification systems.
- Exact Verification Techniques: improved the Venus verification toolkit for ReLU-based feed-forward neural networks by incorporating MILP-based verification enhancements and dependency cuts.
- Distributional Robustness: evaluated Probably Approximately Correct (PAC)-based bounds as an extended objective to provide robustness guarantees under assumptions about the underlying data distribution.
- Application to the project's use cases: evaluated the robustness of EVENFLOW NeSy models deployed in an Industry 4.0 use case scenarios.
(c) Scalable Training:
- Scalable synopsis-maintenance via parallel, tensor-compatible Synopsis Data Engine: the earlier framework (developed in RP1) was pushed toward end-to-end scalability by reimplementing synopsis derivation on Dask, to keep synopses maintenance parallel and tensor-compatible so that they are directly consumable by learning pipelines.
- Synopsis-driven training optimization (SuBiTO): a novel technique for making training adaptive to stream dynamics by jointly optimizing (i) synopsis/compression ratio, (ii) training effort (e.g. epochs), and (iii) neural architecture (via NAS).
- The distributed version of SuBiTO (Distribuito SuBiTO): a set of techniques for tuning the parallelism across the synopsis, learning and inference stages of the framework, scaling the computation to volatile statistical properties of the incoming streams.
- Scalable NeSy CER across IoT: NeuroFlinkCEP contributes scalable NeSy Complex Event Recognition by embedding neural simple-event inference into Flink CEP jobs.
- RATS+ (Resource Allocator for Tumor Simulations enriched with transfer-learning).
- SSTRESED (scalable semantic trajectory extraction and online SDE detection over movement streams) scales durative simple/derived-event detection over high-rate robot movement streams by splitting the workload into two independently scalable pipelines.
- Application to the EVENFLOW use cases: all these techniques were applied to the project's use cases.
(d) Use cases:
- Personalized Medicine:
(i) paediatric cancer and in particular medulloblastoma (MB): identification of new disease subgroups, demonstration of fairer patient stratification, in-silico validation using independent methods.
(ii) Kidney Renal Clear Cell Carcinoma (KIRC): synthetic trajectories of early-to-late-stage transition, early detection of disease stage transitions from partial longitudinal data, interpretable temporal patterns auditable by clinical experts.
- Industry 4.0:
(i) Proactive Deadlock Avoidance: developed and validated a prototype neuro-symbolic controller that achieves a 99% elimination rate of deadlocks and collisions in controlled scenarios.
(ii) Significant Efficiency Gains: demonstrated a 49% improvement in mission completion times and average execution time per robot compared to standard ROS2 navigation baselines.
(iii) Optimized Navigation Performance: evaluations show that robots followed smoother paths, resulting in 24% reduction in total distance travelled and 90% reduction in the need for recovery actions.
- Infrastructure Life Cycle Assessment:
(i) Application of neuro-symbolic forecasting (Mutual Information Markov Models) to forecast water pipe leakage events.
(ii) Leakage detection with high accuracy even with sensors 10x apart.
