Within Computer Science, Machine Learning (ML) and Artificial Intelligence (AI) are certainly among the most disruptive disciplines of the 21st century. Traditionally, the main focus of ML has been the so-called discriminative approach, which means that the goal is to predict outputs (e.g. a class label, 'face' vs 'no-face') from inputs (e.g. an image). Opposed to the discriminative approach is the so-called generative approach, which rather aims to capture the underlying data-generating process (e.g. how pixels in an image are related to each other, and the class label). The generative approach promises to overcome several key challenges of nowadays AI systems, such as ameliorating "catastrophic forgetting" (the phenomenon that ML models completely forget previously learned tasks, when trained on new tasks), overconfidence (predictions with highly exaggerated confidence), and noise and outlier robustness. Furthermore, generative approaches enable many techniques to improve human trust in AI systems, in particular techniques to improve interpretability, explainability, and fairness.
In this project, we addressed an important fundamental problem in generative modeling, namely the notorious hardness of inference. Formally, any generative model strives to capture the true data-generating probability distribution, which allows us to rigorously represent data-dependencies and uncertainty in a universal, unifying, and consistent framework: probability theory. Furthermore, probability theory provides us with tools to derive new insights from our models, to reason under uncertainty, and to derive optimal decisions. These tool, generally referred to as probabilistic inference, are formal and well-defined mathematical operations, which are amenable to automation. Unfortunately, however, most of these inference routines are NP-hard for most generative models (this means that most probably these problems cannot be solved efficiently using current computers).
The remedy for this dilemma, taken in this project, are so-called tractable models, i.e. a class of generative models, where inference can be done exactly and efficiently. One of the most prominent type of tractable model are sum-product networks, a special type of artificial neural network. There exists, however, a natural tension between tractability and expressiveness: when restricting the model class to tractable models, we naturally lose representational power, which means that a tractable model might not capture the data-generating process as faithfully as unrestricted models. The general picture before this project was that a practitioner had a forced choice: i) use unrestricted models and accept the downsides of approximate inference, or ii) use tractable models and accept the downside of restricted model power. Approaches which combined these complementary advantages were scarce at best.
In this project, we explored hybrid approaches, combining tractable models, in particular SPNs, with other approaches from the ML toolbox. The main objective of the project was to explore strategies to combine SPNs with other ML techniques in a meaningful manner, and to demonstrate the benefit of these hybrid learning systems, by establishing new state-the-art results on several ML/AI tasks.
