The interdisciplinary nature of this project required it to be broken down into 6 work packages. WP1 entailed improvement of the theory and hypotheses to be tested, which guided the rest of the WPs. The goal of WP2 was data collection. Through web scraping of the IMF’s online archives, close to 40 thousand pages of EB meeting minutes were gathered. For the statistical analyses involving all IMF programs implemented between 1978-2014, data for numerous independent variables were collected. WP2 also involved data collection for case studies: Secondary data, through archival research of Romanian and Greek daily newspapers as well as newspapers and magazines with a special emphasis on business and economic news, and the IMF’s Independent Evaluation Office reports, were collected. The purpose of WP3 was NLP tool development. In this work package, an NLP tool for automatic extraction of EB views on IMF program design was developed, which is able to process the EB meeting minutes data collected in WP2. The output of the tool is the values of the specific EB-related variable defined in the theoretical model for the different IMF programs analyzed. The NLP tool was designed and implemented specific to the EB meeting minute domain. The model trained with the IMF dataset achieved above 95% accuracy, while the model that improves a popular publicly available BERT model without using any domain data achieved above 87% accuracy, which achieved the promise of the project. The tool was then applied to a different dataset to be able to test its power, which got published in the journal Sustainability. WP4 focused on statistical analyses. The degree of influence of EB’s sentiment on IMF loan amount and number of conditions was analyzed. The results indicate that the EB’s positive sentiments decreases the number of conditions, particularly in the external and financial sector. The intention of WP5 was ML model selection and development. In this work package, ML algorithms to most accurately model the IMF program design process based on the data collected in WP2 were selected. The selected models were implemented in software using an ML framework, and the accuracy of the models were evaluated with held-out test data for a subset of the IMF programs. The objective of WP6 was the analysis of all models and results. In order to compare the results of the regression models with those of the ML models created for estimating (a) the loan size (b) the total number of conditions (c) the number of condition categories, the estimates for each data point based on the obtained regression equations were categorized using the same cutoff values utilized in the ML models. Using a binary categorization scheme, it was observed that the regression models achieved around 72% accuracy in detecting the loan size, 20% accuracy in detecting the total number of conditions, and 55% accuracy in detecting the number of condition categories. The highest accuracies achieved by the developed ML models using the same independent variables are as follows: 85% for loan size, 60% for total number of conditionalities, and 63% for the number of conditionality categories. These results support the hypothesis of the research that it is possible to achieve significantly higher accuracy in program design estimation by utilizing state-of-the-art ML algorithms as compared to statistical regression models. A comprehensive report on all findings of the research, demonstrating the complementary aspects of the methods employed, has been accepted for presentation at the APSA Annual Meeting 2023.
