Development of a european multi-model ensemble system for seasonal to interannual prediction

A central aim of the project has been to evaluate the skill of the DEMETER multi-model prediction system, and its seven component CGCMs, in prediction of key atmospheric and oceanic climate variables on seasonal timescales. To this end, a central verification system (CVS) has been developed so that all models can be evaluated using common diagnostics and standardised display formats. The CVS, which is hosted at ECMWF, provides extensive results generated from the DEMETER retrospective forecast datasets, which allows for a comprehensive inter-comparison between models. The large forecast sample (up to 44 years, 1958 - 2001) provides greater robustness of results than previously available. Evaluation diagnostics and methods used include those recommended by the WMO for verification of long-range forecasts. This comprehensive set of evaluation diagnostics is available at online at http://www.ecmwf.int/research/demeter/verification/. This is a free access public site that facilitates the dissemination of these assessments to the research and applications communities worldwide. A key conclusion of the assessments is that predictions with the DEMETER multi-model have substantially improved skill and economic value than predictions from the individual component models. The DEMETER multi-model system comprises the seven European CGCMs (etip7271). In addition to results for the multi-model, the CVS provides results for each component model. Re-analysis data from the ERA40 project and also GPCP data for precipitation are used as "ground-truth" for verification. Most diagnostics are presented for 3-month-mean quantities at 1-month and 3-month leads. The CVS provides verification for both probabilistic and deterministic (ensemble-mean) forecasts. For probability forecasts the skill of both two category events (above or below the climate normal) and the outer two categories of three category (tercile) events is evaluated. The following types of performance assessment are included. - Graphical timeseries comparisons of ensemble predicted and observed quantities. For example, Nino3.4 SST, Southern Oscillation Index, North Atlantic Oscillation Index, and temperature and precipitation anomalies for various regions of the globe, including the Asian monsoon region, North America and Europe. - Global maps showing the geographical variation of prediction skill assessed over all hindcast years. For probability forecasts skill is assessed using Relative Operating Characteristics (ROC) and the Ranked Probability Skill Score (RPSS). Deterministic forecasts are assessed using the Mean Square Skill Score (MSSS) and Anomaly Correlation (AC). - Similar to the previous, but providing scores aggregated over various geographical regions; for example prediction skill for seasonal temperature and precipitation anomalies over Europe, North America, West Africa, and the tropics. Diagnostics used include reliability diagrams, Brier Skill Score, RPSS and Correlation Scores. - Assessment of the usefulness of the forecasts in terms of their potential economic value to a range of users with varying cost/loss sensitivities to specified seasonal weather conditions. - Assessment of the model systematic error for a number of surface and upper air parameters. - Specific comparison of the prediction skill of the DEMETER multi-model with the skill of the individual participant models. Conclusions from evaluation of the above diagnostics include the following: - There is positive skill at the seasonal timescale for a wide variety of regions and indices. Skill from the component DEMETER models is frequently complementary, i.e. the 'best' model generally varies with region, climate index and season. - The major ENSO events in the hindcast period are generally well captured at up to 6-months lead, with corresponding high overall correlation scores for tropical Pacific SST predictions. Skill for 2-metre temperature and precipitation is highest in tropical regions with positive skill scores available out to the maximum lead (3-months) considered. - Skill for the extratropics, though lower than in the tropics, is sufficient to indicate economic value in many regions/seasons such as Europe and North America - most notably for temperature predictions at 1-month lead. Best skill is generally found in the spring season. Significant skill is found for the Pacific North American oscillation index, one of the key larger scale modes of Northern Hemisphere variability. Correlations for the North Atlantic oscillation index are lower, but indicate skill in predicting the sign of the index. - The simple multi-model system provides improved skill and forecast reliability over the single-model ensembles for both deterministic and probabilistic predictions. - The source of increased multi-model skill is mainly due to the increase in the number of single-model contributions rather than to the increase in ensemble members.

In addition to the component parts of the DEMETER system and their validation, DEMETER demonstrates the potential of the "end-to-end" integrated modelling approach to produce potential economic benefits through the use of a probabilistic forecasting system with integrated application models. Further DEMETER has lead to the establishment of a European real-time multi-model (two model) seasonal forecasting system. This overall result, apart from the economic indicators, is more difficult to fully quantify as it was achieved towards the end of the project and further research and development are required to fully exploit the system. However, the DEMETER project has gathered and built a critical mass of knowledge and expertise to enable the first demonstration of the integration of application models running within a multi-model probabilistic forecasting system integrating out to seasonal time scales. This has allowed, for the first time, the assessment of the potential economic value of a seasonal forecasting system for the application disciplines of European crop yield forecasting and African malaria epidemic forecasting. One of the strengths of DEMETER was the involvement of the application partners in all aspects of the project development e.g. in discussions regarding the choice of variables within the archive, integration start dates and integration length as well as development of forecast products and post forecast processing in terms of bias correction and downscaling. This has enabled the project to produce the required "results" within the individual result components to allow the application groups to make pilot probabilistic forecasts within their own disciplines and ultimately investigate the DEMETER system's economic potential. Two DEMETER models are running as real-time systems enabling, for the first time, generation of multi-model seasonal forecasts for public and commercial interests in Europe. The integrated end-to-end system has shown the potential economic benefit of this approach, for certain regions and applications, by the application partners within DEMETER

Early in the project it was recognised that the coarse horizontal resolution of the DEMETER hindcasts would cause problems when used as input for application models. One of the ways to handle these problems was through the use of statistical downscaling of the DEMETER hindcasts. Several different statistical downscaling methods have been developed and investigated in the course of the DEMETER project. - A Model Output Statistics (MOS) approach where seasonal mean 2m-temperature and seasonal total precipitation were downscaled using a singular value decomposition of the cross-covariance between DEMETER model output and observations. The downscaled hindcasts were found to be skilful (using cross-validation) in most seasons for stations in Europe, North America and Australia (these three regions were chosen because they have the best historical coverage of observed data). - A stochastic weather generator approach has been developed that simulates downscaled daily weather (precipitation, max and min temperature and global radiation) required as input to crop yield models. The technique is to use the above MOS approach to downscale daily precipitation statistics, such as the probability of a wet day in a month or season, and to subsequently use the downscaled statistics to drive a stochastic weather generator. Output from the weather generator approach has been successfully used by crop yield models from JRC and ARPA. - Analogue methods based on close neighbours and on clustering techniques (k-means and neuronal Self-Organized Maps, SOMs) have been applied to direct ensemble outputs from global DEMETER models and, for the close neighbours method, to a few outputs from one regional model. For the close neighbour technique until 30 analogues for one specific daily situation are sought among the ECMWF Re-analysis ERA-15 dataset, and daily and seasonally accumulated precipitations over Spain estimated from the optimum percentile of the PDF of the observed values corresponding to the dates of the analogues found. - On the other hand, the SOM provides a convenient representation of the ensemble in terms of a Probability Density Function (PDF) on the space of atmospheric patterns defined by a reanalysis (so far, ERA-15 has been used, but an extension to ERA-40 is in progress). For instance, the spread of the ensemble is associated with the entropy of the PDF. A forecast to a local station is downscaled considering the observed distribution of the local variable in each of the clusters and combining the resulting information according to the ensemble PDF. The graphical visualization capabilities of SOM make the whole process intuitive and visually appealing. This technique is applied to the Iberian Pensinsula and the Northern Peru (Piura region). Promising results are obtained during the strong "El Nino" periods of 1982-83 and 1997-98 in Peru (with lead times ranging from one to four months ahead), whereas only weaker signals of downscaling success are detected on the Iberian peninsula.

This part of the DEMETER project has produced a unique database for evaluation or statistical adaptation of global seasonal forecasts. It covers the last 40 years, the 4 seasons of the year and 7 different numerical models representing the European state of the art. Each forecast consists of 9 ocean-atmosphere coupled integrations. Daily values of the most important meteorological parameters are saved in a common format, using the same archiving facility as the verification data (ERA40 reanalysis). More specifically, the database contains: CERFACS (ARPEGE+ORCA) for 1980-2001, ECMWF (IFS+HOPE) for 1958-2001, LODYC (IFS+ORCA) for 1974-2001, INGV (ECHAM+ORCA) for 1974-2001, CNRM (ARPEGE+OPA) for 1958-2001, UKMO (UK unified mod.) for 1959-2001, and MPI (ECHAM+HOPE) for 1969-2001. As one can see, the same atmospheric or oceanic model may have been used by different partners, which enables to investigate the relative contribution of each component to the forecast skill. The initial atmospheric situation is in most case the observed one (ERA40). For the ocean, various methods have been used, according to the observations available. The most general method consisted of forcing the ocean model by ERA40 surface fluxes during a long period with sea surface temperature relaxation. The results are expressed in term of deterministic as well as probabilistic skill. The HTML page of the project presents the scores for different years, regions, seasons, and parameters. The essential results of the previous European project PROVOST are confirmed. The multimodel (average of the different models in the deterministic case, aggregation of the members in the probabilistic case) is in most cases superior or equivalent to the best individual model (which varies amongst the cases). Predictive skill is found mainly in the tropics. In the midlatitudes, and in particular over Europe, skill is weaker but still statistically significant. Contrary to PROVOST, the level of skill is little dependent on the season. In fact, during the 15 years of PROVOST, a higher skill was found in winter. The reason for the very low skill in the 1960's and 1970's is still unclear. These results prove the feasibility of operational forecasts in the seasonal range, but the tiny skill, when measured with traditional tools like correlation, calls for multimodel ensembles (some models perform better in summer, some in winter), statistical adaptation, and probabilistic user-oriented approach. The database available has a potential to reveal many scientific results in the field of predictability.

At ARPA during the Demeter project we developed an integrated model using the existing soil water balance model Criteria, developed by us, and Wofost 7.1, a crop growth simulation model developed in Wageningen (NL) by Kees Van Diepen and used also at JRC in the framework of the Cgms crop yield forecasting system. The Criteria/Wofost integrated system was calibrated and tested using experimental data and expertise provided by ISA-MO (Istituto di Sperimentazione Agronomica of Modena). Some output variables from Criteria (leaf area index, rooting depth) were used to force the crop growth module in order to remove sources of high variability in the original Wofost model, which is very sensitive to initial condition settings. Other output variables from Criteria (potential and actual evapotranspiration) where used to force the growth model due to better accuracy of our soil water model compared with the one provided with the Dutch model. The calibrated and adapted model was then used to simulate wheat yields from 1977 to 1987 using combinations of gridded weather data and downscaled Demeter hindcasts (dDh), trying to establish the crop yield predictive power of the latter as a substitute for actual weather data. A combination consists of running crop growth simulations with observations up to a certain date (e.g. end of March, April or May) and with dDh up to harvest date (usually end of June in the plains of Northern Italy). The use of downscaled hindcasts in the last months of the wheat growing season resulted in an increase of crop yield predictive power. Median yields predicted with dDh were always performing equal or better in terms of determination coefficient (0.62, 0.73 and 0.97 for the March, April and May combinations resp.) than predictions obtained from linear regressions between partial simulation results at the above mentioned dates and final yields (0.63, 0.41, 0.90). However runs performed with climatology instead of dDh performed slightly better than dDh (0.64, 0.75, 0.91) with the exception of the May run. At the JRC, DEMETER (downscaled) hindcasts were used as input of the Crop Growth Monitoring System/Crop Yield Forecasting System, already existing and used to do real-time crop yield forecasting on Europe at the national level. Current system allows simulating crop development all along the growing season and is classically used with daily observed and interpolated ground station meteorological data. The DEMETER hindcasts from March to August were added to the usual data for January and February, and the annual yield estimations were made in beginning of March using simulated crop development and conditions until the crop maturity, that is to say 5 months in anticipate. Evaluation was made on 4 years, for winter wheat crop, and showed that DEMETER ensembles provide relatively accurate forecasts, notably in terms of precocity: in average DEMETER yield estimations accuracy is better than JRC estimations until the last weeks of the growing season. Globally only in end of July/August the JRC current (real-time) system proposes better results. An important issue is that the probabilistic aspect of the ensemble of simulated yield allows providing forecast of annual yield anomalies. For a crop reaching maturity in the end of summer, already in March it seems possible to depict eventual lower/higher yield than normal (expected) yield due to exceptional or extreme weather condition. Dissemination and use potential. No dissemination of the ARPA model is foreseen at the moment, because the crop yields it predicts are potential ones, not taking into account actual limiting factors like soil fertility, heat and cold stresses, pest and disease damage. The system though shows a high use potential in combination with expertise, field surveys and statistical post processing. Key innovative features of the result. For the first time the potential positive impact of multimodel ensemble downscaled seasonal forecasts was demonstrated for crop yield forecasting at both regional (Northern Italy, ARPA) and national scales (in Europe, JRC). Current status. The integrated agricultural models have been created, calibrated and tested, they allowed to use Demeter seasonal hindcasts for crop yield simulation, showing potential improvements in yield prediction compared with current methods. Use of the result. Results obtained are being used for publication. The combined CRITERIA/Wofost model is one of the supports to the crop yield forecasting activity yearly performed at ARPA. For JRC crop yield seasonal forecasting is a topic of major interest and it will continue working in these field. Expected benefits. Early and accurate yield forecasts are very important for the agricultural markets and also for local and central governments. The use of downscaled multimodel ensemble seasonal forecasts proved to have a good potential for yield forecast skill improvement. forecasts are very important for the agricultural markets and also for local and central governments. The use of downscaled multimodel ensemble seasonal forecasts proved to have a good potential for yield forecast skill improvement.

An important part of the project is the dissemination of the results, in particular making available the core DEMETER data set to the general interested public. To ensure a most user-friendly access to the data, an online data retrieval system has been developed and installed at ECMWF. A significant part of the DEMETER data set (monthly averages of a large subset of surface and upper-air fields) is now freely available for research purposes through a public website at ECMWF (http://data.ecmwf.int/data) The data available for downloading comprises a variety of gridded monthly mean fields from all ensemble members together with the corresponding verification from the reanalysis dataset. For example, geopotential height, temperature, wind and specific humidity are provided on three tropospheric pressure levels. Total precipitation, low-level wind, 2-metre temperature and mean sea level pressure are also available. A tool to plot these fields before retrieving them in gridded form is also provided. This data can be retrieved in both GRIB and NetCDF format. This dataset should prove useful for scientists and potential users of seasonal forecasts wishing to assess seasonal predictability using a truly state-of-the-art multi-model ensemble system, for regions and variables of interest. The dataset will also be valuable for training and education purposes.

One of the main goals of the project has been to develop a well-validated European seasonal forecasts system. Thus, the core activity of DEMETER has been the development and installation of the multi-model ensemble forecasts system. This system has been used to prove the superiority of the multi-model concept compared to single-model performance. The DEMETER system comprises seven state-of-the-art global coupled GCMs, six of them installed at the ECMWF computer facilities and one running on a remote computer. The partners and ocean-atmosphere model combinations contributing to the system are as follows: ECMWF (IFS + HOPE-E), LODYC (IFS + OPA-8.2), Meteo France (ARPEGE + OPA-8.0), CERFACS (ARPEGE + OPA-8.2), INGV (ECHAM-4 + OPA-8.1), MPI (ECHAM-5 + MPI-OM1), and UKMO (HadAM3 + GloSea). All models run with a similar set-up, i.e. they all sample uncertainties in initial conditions with a 9 member ensemble. Except for the MPI contribution, these 9 ensemble members are created by applying wind stress and SST perturbations to the ocean analyses. However, most importantly, all models are set-up with a common archiving strategy, i.e. all models store a set of agreed variables with common units in a common database (the ECMWF Meteorological Archival and Retrieval System, MARS). This similar set-up allows not only the combination of all individual model results to a multi-model forecast, but also a most efficient diagnosis of single- and multi-model performance. The installation of this unique multi-model ensemble forecast system has been the pre-requisite for all remaining work in the project, i.e. the production of the DEMETER data set and the assessment of the potential utility of seasonal forecast for end-users and application models.

Our aim is to assess the potential benefit of incorporating seasonal climate forecasts into malaria early warning systems in Africa through a retrospective analysis of the predictability of malaria epidemics based on seasonal climate hindcasts. In order to achieve our goal we have: - Developed a dynamical mathematical model of malaria transmission which can be driven by daily meteorological variables (rainfall and temperature); - Undertaken limited testing of the model against malaria case data from Africa; - Driven the model with ERA-40 data (rainfall and temperature); - Linked the model directly to the output of individual members of the seasonal climate forecast ensemble, which can then describe the future risk of malaria epidemics in probabilistic terms; - Linked the outputs to an economic model indicating cost and effectiveness of the timing of interventions given the probability of future epidemic risk. Although requiring further sensitivity analysis the malaria model driven by meteorological data appears to have significant skill in certain epidemic areas in Africa. However, when tested with ERA-40 data (as for example in Hwange district North Western Zimbabwe) this skill is lost. Comparison of ERA-40 data with meteorological data from the area suggests that this may be due to differences in the relationship between station data (most significantly temperature data) and ERA-40, which may be corrected, in subsequent analysis through downscaling. Malaria model skill levels not withstanding the results from the comparison of the model driven by ERA-40 and those predicted by DEMETER indicate the skilful nature of the forecast in Southern Africa and show that this skill is retained when the forecasts are processed through a dynamical malaria model. The potential economic value of the system, as compared to the control situation without malaria forecast, was calculated using a cost loss evaluation.

Numerous sensitivity studies have been performed as part of DEMETER. These were primarily aimed at assessing the benefits of: - Using fully coupled ocean/atmosphere models (CGCM) over atmospheric (AGCM) only setups, - Using ocean and satellite altimeter data in model initialisation schemes. To address these issues parallel sets of hindcasts were performed over extended periods with two of the coupled models. The Met-office model was used to perform AGCM persisted SST hindcasts, and the ECMWF model was used to carry out hindcasts with and without ocean data assimilation. Results indicate the overall performance of the coupled model is generally equivalent to that of the AGCM persisted SST setup, but with a few valuable benefits. Particular benefits include improved temperature and precipitation prediction at 3-month lead in the northern spring and winter seasons, both in the tropics and extratropics. The use of ocean data assimilation has also shown some important improvements. In addition to the above described experiments, sensitivity studies were performed to assess the benefits of increasing vertical and horizontal atmospheric resolution, the methods of ensemble member generation, and also to reduce the biases of the coupled models. Such experiments play an important role and are integral to improving the models and forecasts methods.

The DEMETER project used comprehensive coupled ocean-atmosphere models to produce retrospective climate hindcasts. However, there is no equivalent for the ocean of the atmosphere reanalysis ERA40, and thus no common data set of initial conditions and verification for the ocean. One major achievement of DEMETER has therefore been the production of such a set by each of the modelling partners. All modelling partners but one have used ERA40 daily wind and flux analyses to create four dimensional ocean analyses. These analyses have been interpolated on a common grid and stored on a common format on MARS for a scientific evaluation similar to that done for atmosphere. As the ocean memory is thought to be a major source of predictability, it was decided at the beginning of DEMETER that the ensemble generation strategy should be determined through ocean perturbation. Therefore most modelling partners have used similar strategies to obtain ensembles of slightly perturbed ocean initial conditions, making use of common sets of perturbed wind fields and Ocean Surface Temperature (made available by ECMWF). This practical approach to estimate uncertainties when starting climate forecasts was innovative and has been proven successful. Four modelling partners have produced sets of ocean analyses and initial conditions incorporating ocean observations through a data assimilation procedure. Most of them use in-situ temperature data. As a result, the unique data set stored on MARS gathers 6 sets of ocean analyses over three decades. It forms a state of the art representation of our knowledge of the global ocean state and variability over this period, and a quantified estimate of associated uncertainties.

One of the goals of the project has been to explore the ability to provide high regionalized detail to the seasonal forecast using statistical and dynamical methods (Work Package 7). Forecasts with the dynamical Rossby Center Atmospheric limited area model with 0.5 degrees resolution were undertaken covering a wide European Atlantic domain and, nested, a more restricted Southwest Europe domain surrounding Iberia with a 0,2 degrees resolution. Also a few integrations were undertaken over a wide domain over Africa with the purpose to provide data for the DEMETER application dealing with malaria. In the first phase a test under perfect model conditions using ERA-15 data as boundary conditions was undertaken. It was followed by some runs with real global model outputs from the ECMWF DEMETER model. Overall, 21 forecasts were undertaken to produce seven 3-member ensembles starting in November 1986 and May 1987 for the European-Atlantic domain, 3 forecast for one ensemble over Africa and 3 forecasts for the ensemble with higher resolution over Iberia. All this information is archived and available in the ECMWF MARS. An assessment of the feasibility of dynamical downscaling has been produced with some recommendations for a possible further experiment pursuing a detailed evaluation of the skill of dynamical downscaling from a set of global model outputs. However, for this period, the observed precipitation values over Spain were compared with those of this dynamical downscaling exercise and those obtained by the statistical downscaling method based in analogues referred under statistical downscaling.

Deliverables

Share this page

Download