Periodic Reporting for period 5 - NLPRO (Natural Language Programming: Turning Text into Executable Code)
Periodo di rendicontazione: 2022-10-01 al 2023-07-31
Can we program computers in our native tongue? This idea, termed natural language programming, has attracted attention almost since the inception of computers themselves. From the point of view of natural language processing (NLP), current technology successfully extracts static information from NL texts. However, human-like NL understanding goes far beyond such extraction — it requires dynamic interpretation processes which affect, and are affected by, the environment, update states and lead to action. So, is it possible to endow computers with this kind of dynamic NL understanding? These questions are fundamental to SE and NLP, respectively, and addressing each requires a huge leap forward in the respective field. In this proposal I argue that the solutions to these seemingly separate challenges are actually closely intertwined, and that one community’s challenge is the other community’s stepping stone for a huge leap, and vice versa. Specifically, I propose to view executable programs in SE as semantic structures in NLP, and use them as the basis for broad-coverage dynamic semantic parsing. Our ambitious, cross-disciplinary goal is to develop a new NL compiler based on this novel approach to computational semantics. The NL compiler will accept an NL description as input and return an executable system as output. Moreover, it will continuously improve its NL understanding capacity via online learning that will feed on verification, simulation, synthesis or user feedback. Such dynamic, ever-improving, NL compilers will have vast applications in AI, SE, robotics and cognitive computing, and they will fundamentally change the way humans and computers interact.
Below please find key results by WP:
1. Linguistic representation for Language Technologies (WP1):
1a. The linguistic representation we assume for all natural language utterances is based on the Universal Dependencies Scheme (universaldependencies.org). We devised guidelines and developed automatic algorithms for parsing natural language texts into UD representations. We release the code of our parser (YAP - yet another parser) and applied it to the entire UD set (containing over 80 different languages). A neural version of the parser that is tailored for morphologically rich languages is being published at EACL 2024.
1b. Since the UD representation is morphological-syntactic, it is incomplete with respect to semantics of the sentences. We developed a software in python (called pyBART) that infers semantic information from the syntactic representation into meaningful semantic structures. We deliver code, demo, and a publication at a top tier conference (ACL 2020).
1c. Syntactic and Semantic representations are incomplete, because human understanding also incorporates the knowledge of elements missing in the text. We initiated the EMPTY ELEMENTS project, in which we annotate and infer empty elements that mimic natural language comprehension by humans. We delivered data annotation interface and annotated data sets with corresponding models. The project was presented in a European High Profile meeting (EurNLP2019)
1d. We collected over 2K paragraphs containing empty elements such as "bridging anaphora", and trained models on completing those empty elements. We published a respective article at the TACL journal of the MIT press (TACL 2022)
1e. We collected 1K instances of superlatives domain restrictions in 4 different domains (from wikipedia to dialogues). We trained models on these data and showed its benefits for downstream tasks (submitted to TACL 2024)
2. Models and Algorithms for Dynamic Semantic Parsing (WP2-WP3):
2a. We selected a challenging task for dynamic semantic parsing -- the task of natural language navigation -- automatically parsing free text instructions into executable path on a dense urban.
In doing so we addressed the problem of grounding (connecting elements in the world to elements in the text), generalization (interpreting instructions in unknown domains) and execution (reaching the desired end point).
2b. We delivered an annotated database, neural models, and a publication on Realistic natural language navigation (RUN) in a top tier publication venue (EMNLP 2019)
2c. Inspired by cognitive science studies that analyze spatial reasoning in natural language, we devise a new navigation task that involves three dimensions: landmark knowledge, route knowledge and survey knowledge. We collected over 10K instances, and delivered strong baseline models. This work will be presented at EACL 2024.
2d. To showcase the university of the natural language navigation task we collected a benchmark for Hebrew GeoLocation (HEGEL), published at ACL 2023. Extending this work for Arabic and other languages is now underway.
3. Methods and Tools for Natural Language Programming (WP4):
3a. We developed "The Hexagon App", a virtual creative board-game, where users use natural language to program complex abstract patterns on a hexahgons board (pic attached).
We developed a graphical interface, collected data for training and benchmarking, and trained models on this challenging task.
This work is published in the TACL Journal of the MIT Press
3b. We developed NoviCode, a novel task and benchmark wherein users specify in their own language procedures involving a novel API, and the model generates the respective code in python.
This work goes beyond the simple "text-to-code" paradigm as it aims to generate complete technical programs from non-technical novice texts.
The paper (along with oyr data, models, benchmark, and unit tests) is under review for the TACL journal of the MIT press.
4. Multilingual and Cross Lingual Applications (WP5):
4a. We deliver a joint neural parser for segmentation and parsing morphologically rich languages (EACL 2024)
4b. We deliver UniMorph 4.0 a universal representation for words in typologically diverse languages (LREC 2023)
4c. We deliver MightyMorph, a cross-linguistic benchmark and a novel scheme suitable for morphologically rich and low-resource languages. (TACL 2022)
1. Linguistic representation for Language Technologies (WP1):
1a. The linguistic representation we assume for all natural language utterances is based on the Universal Dependencies Scheme (universaldependencies.org). We devised guidelines and developed automatic algorithms for parsing natural language texts into UD representations. We release the code of our parser (YAP - yet another parser) and applied it to the entire UD set (containing over 80 different languages). A neural version of the parser that is tailored for morphologically rich languages is being published at EACL 2024.
1b. Since the UD representation is morphological-syntactic, it is incomplete with respect to semantics of the sentences. We developed a software in python (called pyBART) that infers semantic information from the syntactic representation into meaningful semantic structures. We deliver code, demo, and a publication at a top tier conference (ACL 2020).
1c. Syntactic and Semantic representations are incomplete, because human understanding also incorporates the knowledge of elements missing in the text. We initiated the EMPTY ELEMENTS project, in which we annotate and infer empty elements that mimic natural language comprehension by humans. We delivered data annotation interface and annotated data sets with corresponding models. The project was presented in a European High Profile meeting (EurNLP2019)
1d. We collected over 2K paragraphs containing empty elements such as "bridging anaphora", and trained models on completing those empty elements. We published a respective article at the TACL journal of the MIT press (TACL 2022)
1e. We collected 1K instances of superlatives domain restrictions in 4 different domains (from wikipedia to dialogues). We trained models on these data and showed its benefits for downstream tasks (submitted to TACL 2024)
2. Models and Algorithms for Dynamic Semantic Parsing (WP2-WP3):
2a. We selected a challenging task for dynamic semantic parsing -- the task of natural language navigation -- automatically parsing free text instructions into executable path on a dense urban.
In doing so we addressed the problem of grounding (connecting elements in the world to elements in the text), generalization (interpreting instructions in unknown domains) and execution (reaching the desired end point).
2b. We delivered an annotated database, neural models, and a publication on Realistic natural language navigation (RUN) in a top tier publication venue (EMNLP 2019)
2c. Inspired by cognitive science studies that analyze spatial reasoning in natural language, we devise a new navigation task that involves three dimensions: landmark knowledge, route knowledge and survey knowledge. We collected over 10K instances, and delivered strong baseline models. This work will be presented at EACL 2024.
2d. To showcase the university of the natural language navigation task we collected a benchmark for Hebrew GeoLocation (HEGEL), published at ACL 2023. Extending this work for Arabic and other languages is now underway.
3. Methods and Tools for Natural Language Programming (WP4):
3a. We developed "The Hexagon App", a virtual creative board-game, where users use natural language to program complex abstract patterns on a hexahgons board (pic attached).
We developed a graphical interface, collected data for training and benchmarking, and trained models on this challenging task.
This work is published in the TACL Journal of the MIT Press
3b. We developed NoviCode, a novel task and benchmark wherein users specify in their own language procedures involving a novel API, and the model generates the respective code in python.
This work goes beyond the simple "text-to-code" paradigm as it aims to generate complete technical programs from non-technical novice texts.
The paper (along with oyr data, models, benchmark, and unit tests) is under review for the TACL journal of the MIT press.
4. Multilingual and Cross Lingual Applications (WP5):
4a. We deliver a joint neural parser for segmentation and parsing morphologically rich languages (EACL 2024)
4b. We deliver UniMorph 4.0 a universal representation for words in typologically diverse languages (LREC 2023)
4c. We deliver MightyMorph, a cross-linguistic benchmark and a novel scheme suitable for morphologically rich and low-resource languages. (TACL 2022)
Progress and SOTA obtained in the course of the project:
1. Linguistic representation for Language Technologies (WP1):
- YAP (yet another parser) a universal morphosyntactic parser applied to over 80 languages (ACL 2019)
- TNE (text-based NP enrichment) benchmark and models for empty elements prediction (TACL 2022)
- Superlatives and domain restrictions: benchmark models and downstream applications (submitted to TACL)
2. Models and Algorithms for Dynamic Semantic Parsing (WP2-WP3):
- RUN through the streets: benchmark and models for natural language navigation (EMNLP 2019)
- RVS: a novel benchmark and models for spatial reasoning in natural language (EACL 2024)
- HEGEL: A novel benchmark and and models for Hebrew GeoLocation (ACL 2023)
3. Methods and Tools for Natural Language Programming (WP4):
- The Hexagons App, dataset and models for processing computational abstract structures in natural language (TACL 2022)
- NoviCode: Benchmark, models, and unit-tests for converting novice descriptions to executable programs (submitted to TACL).
4. Multilingual Aspects (WP5)
- A novel neural morphosyntactic parser making use of LLMs (EACL 2024)
- Unimorph 4.0 a novel dataset and scheme for multilingual word structures
- MightyMorph 1.0 a novel dataset and scheme for multilingual sentence structures
1. Linguistic representation for Language Technologies (WP1):
- YAP (yet another parser) a universal morphosyntactic parser applied to over 80 languages (ACL 2019)
- TNE (text-based NP enrichment) benchmark and models for empty elements prediction (TACL 2022)
- Superlatives and domain restrictions: benchmark models and downstream applications (submitted to TACL)
2. Models and Algorithms for Dynamic Semantic Parsing (WP2-WP3):
- RUN through the streets: benchmark and models for natural language navigation (EMNLP 2019)
- RVS: a novel benchmark and models for spatial reasoning in natural language (EACL 2024)
- HEGEL: A novel benchmark and and models for Hebrew GeoLocation (ACL 2023)
3. Methods and Tools for Natural Language Programming (WP4):
- The Hexagons App, dataset and models for processing computational abstract structures in natural language (TACL 2022)
- NoviCode: Benchmark, models, and unit-tests for converting novice descriptions to executable programs (submitted to TACL).
4. Multilingual Aspects (WP5)
- A novel neural morphosyntactic parser making use of LLMs (EACL 2024)
- Unimorph 4.0 a novel dataset and scheme for multilingual word structures
- MightyMorph 1.0 a novel dataset and scheme for multilingual sentence structures