Skip to main content

Artificial Intelligence for Large-Scale Computer-Assisted Reasoning

Periodic Reporting for period 3 - AI4REASON (Artificial Intelligence for Large-Scale Computer-Assisted Reasoning)

Reporting period: 2018-09-01 to 2020-02-29

The AI4REASON project is targeting a very hard problem in AI and automation of reasoning, namely the problem of automatically proving theorems in large and complex theories.

Such complex formal theories arise in projects aimed at verification of today's advanced mathematics such as the Formal Proof of the Kepler Conjecture (Flyspeck), verification of software and hardware designs such as the seL4 operating system kernel, and verification of other advanced systems and technologies of today's information society.

It seems extremely complex and unlikely to design an explicitly programmed solution to the problem. However, we have recently shown that the performance of existing approaches can be multiplied by data-driven AI methods that learn reasoning guidance from large proof corpora. The AI4REASON project focuses on developing such novel AI methods.
The work proceeded as scheduled and good progress was made in the research areas covered by the five work packages:

WP1: High-Level Premise Selection
WP2: Internal Proof Guidance
WP3: Lemmatization, Conjecturing, and Concept Introduction
WP4: Self-Improving AI Systems Combining Deduction and Learning
WP5: Deployment and Cross-Corpora Reuse

The project has achieved several breakthroughs in these topics, leading to unusually high improvements in theorem proving performance over the state of the art. Combining learning and reasoning seems to be a very viable approach to building stronger AI and reasoning systems.The details will be given in the final Scientific report.

In WP1, we have worked on novel learning architectures for premise selection, applying deep neural networks to this task jointly with a new team at Google. We have also explored various boosting methods between the learning and proving systems. This has led to improvements in premise selection and consequently stronger theorem proving in large theories. The learning methods have been applied to several proof assistants, such as Isabelle, HOL4, and Coq.

In WP2,we have continued in implementation of improved internal guidance of automated theorem provers (ATP). We Implemented clauses selection mechanisms based on gradient boosting decision trees, improved representation of clauses by feature vectors, experiments with guidance based on neural networks. Experiments with relaxed term orderings are currently in print in the journal of Mathematics in Computer Science (MICS).

In WP3, we have further developed neural methods for the autoformalization task and combined them in feedback loops with symbolic methods such as formal type-checking

In WP4, the feedback loops between learning-based internal guidance of ATPs and learning has been developed in several ways. We have introduced reinforcement learning for theorem proving and obtained several strong results. The competitive and state-of-the-art tactical theorem prover TacticToe has been developed.

In WP5, the machine guidance system ENIGMA was, for the first time, employed in large scale and the results were published and presented at the ITP 2019.

Our ATP systems have won two divisions of the yearly competition of automated theorem provers (CADE) in 2018, also competed in the CASC competition /LTB division/ in 2019.

The researchers organized several conferences and workshops in the field, gave a number of invited talks, and successfully competed in several theorem-proving competitions.
Our project and domain is unique in connecting two major AI fields: Automated Reasoning and Machine Learning. This produces new methods in Automated Reasoning, as well as new tasks and issues in Machine Learning. Particularly interesting and important are combinations of learning and reasoning methods into larger AI metasystems where the learning and reasoning components inform and improve each other's work in various feedback loops.

In more detail, the major novel aspects and progress beyond state of the art include:

- equipping a number of theorem provers with a guiding component based on learning from previous proofs,
- application of deep learning and other advanced learning methods to theorem proving,
- defining the autoformalization task and building the first corpora and systems for autoformalization, and
- building several AI metasystems that combine learning and reasoning in various feedback loops.

These methods have led to a significant improvement of the performance of automated reasoning and autoformalization tools on several standard benchmarks as well as to new results in automatically assisted research-level mathematics.
The team of the Ai4REASON won the ATP System Competition (CASC- LTB Division) in 2018.