## Final Report Summary - SOSNA (Expressive Power of Tree Logics)

Buchi proved that over infinite words, monadic second-order logic has the same expressive power as a natural class of automata, which are currently called Buchi automata. This work was continued by Rabin, who extended to infinite trees the correspondence between monadic second-order logic and automata. Because of these results, monadic second-order logic is currently the established notion of regularity for infinite objects. In this project, we have tried to undermine this notion. Using a nonstandard quantifier, we defined new classes of languages, which properly extend Buchi's and Rabin's languages, and such that: a) the classes are closed under natural operations, including union, intersection and complementation; b) the classes have automata models; c) the classes have logic models (variants of weak monadic second-order logic); d) emptiness is decidable. We have done this both for infinite words and infinite trees (the notions do not make sense for finite objects). The new quantifier is close to the limits of decidability, both in the computational and set theoretic senses: we have proved that if full monadic second-order logic (and not weak monadic second-order logic) is extended with the new quantifier, then the logic becomes undecidable. Actually, the undecidability result has a weaker statement, which is very unusual for theoretical computer science and maybe even mathematics in general: there exist models of set theory were the logic is undecidable.

We developed a new approach to automata and languages over infinite alphabets. The new approach is to use a different set theory, namely sets with atoms, in which the notion of a ``finite'' object is more relaxed. In particular, some alphabets in the theory XML, which are infinite in the standard sense, become finite in the more relaxed sense. Our achievement is not discovering sets with atoms: these were discovered by Fraenkel in 1922 and also studied by Gabbay, Pitts and others in the last decade. Our achievement is defining the new notion of finiteness, called "orbit-finiteness", which only makes sense in the presence of atoms, and applying these ideas to theoretical computer science, where finiteness naturally plays a leading role. One of the main results is that in the presence of atoms, the notion of computability becomes more interesting than in the classical world: there exist languages which are recognised by nondeterministic Turing machines, but not by deterministic Turing machines. In particular, P ≠ NP in the presence of atoms; although this particular result is unlikely to shed any light on the classical P vs NP problem.

The algebraic theory of regular languages, in the case of words, is the approach to regular languages which uses monoids instead of automata. This, perhaps more mathematical, approach is the one that was taken by pioneers like Schutzenberger or Eilenberg. The algebraic theory has been successfully extended to infinite generalisations of words by Carton, Colcombet, Rispal, Wilke and others. In the project, we have proposed algebraic theories for infinite trees, infinite trees with countably many branches, languages over infinite alphabets, and word-to-word transducers. We have used these algebraic theories to get machine independent characterisations of monadic second-order logic, and algebraic and effective characterisations of fragments of monadic second-order logic.

We developed a new approach to automata and languages over infinite alphabets. The new approach is to use a different set theory, namely sets with atoms, in which the notion of a ``finite'' object is more relaxed. In particular, some alphabets in the theory XML, which are infinite in the standard sense, become finite in the more relaxed sense. Our achievement is not discovering sets with atoms: these were discovered by Fraenkel in 1922 and also studied by Gabbay, Pitts and others in the last decade. Our achievement is defining the new notion of finiteness, called "orbit-finiteness", which only makes sense in the presence of atoms, and applying these ideas to theoretical computer science, where finiteness naturally plays a leading role. One of the main results is that in the presence of atoms, the notion of computability becomes more interesting than in the classical world: there exist languages which are recognised by nondeterministic Turing machines, but not by deterministic Turing machines. In particular, P ≠ NP in the presence of atoms; although this particular result is unlikely to shed any light on the classical P vs NP problem.

The algebraic theory of regular languages, in the case of words, is the approach to regular languages which uses monoids instead of automata. This, perhaps more mathematical, approach is the one that was taken by pioneers like Schutzenberger or Eilenberg. The algebraic theory has been successfully extended to infinite generalisations of words by Carton, Colcombet, Rispal, Wilke and others. In the project, we have proposed algebraic theories for infinite trees, infinite trees with countably many branches, languages over infinite alphabets, and word-to-word transducers. We have used these algebraic theories to get machine independent characterisations of monadic second-order logic, and algebraic and effective characterisations of fragments of monadic second-order logic.