1. OPENING OF THE MEETING 1.1 The meeting of the Expert Teams on Long-Range Forecasting (Infrastructure and Verification) of the CBS OPAG on DPFS chaired by Dr Chung-Kyu PARK was held at the WMO Secretariat Headquarters, in Geneva from 16 to 19 November 2004. Dr Chung-Kyu Park declared the meeting open and invited Mr Dieter Schiessl to address the Meeting.
1.2 On behalf of Mr Michel Jarraud, Secretary General of WMO, Mr Dieter Schiessl, Director World Weather Watch Department, welcomed the participants. Mr Schiessl recognized the accomplishment of major tasks of the Teams, in collaboration with other WMO bodies, and a workshop of potential global producing centres (GPCs), which were considered by CBS Extraordinary session in Cairns in December 2002 and adopted by the fourteenth Congress in May 2003. He noted that the results included a list of global products to be made available by global-scale production centres and revised standard verification procedures for operational use. He noted that EC in June 2004 recalled the need to establish a Lead Centre Web site and to develop an information system to help RCCs and NMHSs to assess the skill of long-range forecast products from verification data. As well, a meeting of the Lead Centres in December 2003 developed proposals that are submitted to this Meeting.
1.3 Mr Schiessl recalled that the major tasks of the Verification Team include development and implementation of verification schemes for Long-Range forecasts (LRF) in collaboration with CAS, while CCl would provide leadership in the development and implementation of post-processed products to end users, including their verification.
1.4 Mr Schiessl noted that the Meeting was expected to further develop the WWW aspect of the infrastructure for long range forecasting that assure all members actively participate in the arrangements as either users or producers of LRF information and to provide feed back, as users, to production centres to improve long-range forecasting skills.
2. ORGANIZATION OF THE MEETING
Adoption of the agenda
2.1.1 The Meeting adopted the agenda given in Appendix I.
2.2 Other organizational questions
2.2.1 The Meeting agreed on its working hours, mechanism and work schedule. The list of participants is given in Appendix II.
3. STATUS OF LRF PRODUCTION (FORECASTS AND SCORES) BY GPCs 3.1 The Meeting reviewed the status of generation by GPCs of products and scores for long range forecasts, seasonal and inter-annual outlooks, and related information especially for distribution to NMHSs and RCCs. It noted with satisfaction the considerable progress in implementation in these areas as follows:
Dynamical tools are widely used for operational LRF, while the statistical models are also used in parallel.
Multi-model ensemble techniques are being tested at several GPCs, and the preliminary results showed superiority to the single model performances.
There is a trend for GPCs towards developing ocean-atmosphere coupled global models for operational LRF, and some have already replaced their Tier-2 system.
LRF and their verification of the sub-seasonal variation are important for the detection of extreme climate signal, but their implementation is very challenging.
The limited availability of real-time verification data and soil moisture data to prescribe surface conditions hamper further improvement of the current LRF system.
The CBS SVSLRF has been operationally implemented in some GPCs, and produced valuable information, but more end-user oriented verification scores are required.
The implementation of SVSLRF involves a significant effort, however the outputs are worthwhile.
3.1.1 The Meeting received with satisfaction presentations of the status reports of LRF production (forecast and scores) by GPCs, a summary for each GPC is given in the annex to this paragraph.
3.2 The Meeting reviewed the real-time monitoring of forecasts and relevant exchange between participating centres and institutes, in relation to the guidelines listed by the Workshop on GPCs (Geneva, February 2003). It noted progress and shortcomings in implementation (see texts in italics below) in these areas as follows:
All Global Producers should implement the CBS adopted SVSLRF.
Several centres are implementing SVSLRF and all centres are encouraged to continue to do so.
List of products
All Global Producers should implement the list recommended by the ETILRF and noted by CBS-Ext.2002.
Several centres have implemented a significant number of products.
All Global Producers should define a fixed time schedule and inform their users accordingly.
Several GPCs have announced on their web sites their time schedule for availability of products. System changes and hindcast
All Global Producers should notify any important system change affecting model climate and make available their relevant hindcast data sets, with a minimum 3-month notice. Hindcast for system changes should be based on at least 15 years, otherwise periods given in the SVS should be used.
Global Producers should achieve hindcasts in a form as close as possible to the real-time operating mode in terms of resolution, ensemble size and parameters.
For many centres the system used to produce hindcasts are not identical to the systems used for real time forecasts (ensemble size, SSTs, etc.). Further studies are required to provide guidance on this matter. User feed back
NMHS and RCCs should provide user feedback to the global producing centres on their assessment of the use and utility of the products.
One present mechanism for provision of feedback is through climate fora. Other feedback mechanisms and procedures need to be developed. Distribution
The digital data for forecasts and hindcasts should be made available in both raw and calibrated forms for both actual and anomaly quantities.
Some centres are already making these data accessible on their web sites and other centres are encouraged to do the same.
Delivery of digital products should be made available in GRIB 2.
Some centres have already implemented and others are planning to implement this guideline. Multi-models ensembles
Global producers should continue to organise themselves on a regional and interregional basis with a view to developing operational multi-model seasonal forecasting systems.
These activities are becoming well organized, in Europe under through partnership between ECMWF, UKMETO and Meteo-France; in Asia-Pacific region by APCN; by the IRI; in Southern Africa activity is lead by the Global Forecasting Centre for Southern Africa (GFCSA). 3.3 The Meeting considered and agreed on the need to establish mechanisms to assist NMHSs and RCCs to use the global-scale products to provide end-user services, including development and provision of relevant software as capacity building measures to access information from producing centres and providing user friendly verification information.
3.4 The Meeting was informed that the process of designation of RCCs has started and would be made through Regional Associations. It was noted that the process was at various stages in the different Regional Associations. RA IV has started the process, RA VI was working on its proposals that involve 4 GPCs and RA II has made its proposals to the president of the region. RA V has done some work, while in RA III and RA I little has been done so far.
3.5 The Meeting considered and developed the following conclusions and recommendations to improve and increase the dissemination of information by GPCs: real time products, real time scores and verification scores:
3.5.1 The Meeting noted that significant progress has been made in Long Range Forecasting over the last few years. Global Producing centres are offering global forecasts products and services based under a regular schedule and are producing the SVSLRF. The Lead Centre on SVSLRF has developed a web site that will be fully functional in the first half of 2005.
3.5.2 It is the opinion of the joint Expert teams on Infrastructure and SVS on LRF that it is now time to formally designate the GPCs. This will allow institutions outside of the WWW system that have demonstrated capabilities in LRF production and services on an operational scale to be officially recognized as such. It will facilitate international cooperation and exchange of products within WMO and those institutions. It will also contribute to a more credible program in LRF, under the auspices of the WMO.
3.5.3 A Statement of user requirement has been endorsed at the CBS-Ext.2002 as contained in Appendix V of its report, and was used as the basis to determine the minimum list of products to be made available by GPCs in the annex to this paragraph. This list is recommended for adoption for inclusion in Appendix II-6 in the Manual on the GDPFS and should be used as a minimum requirement for designation purpose. This list will be reviewed on a regular basis to include additional products endorsed by CBS.Ext.2002 as capabilities of centres increase.
3.5.4 The procedures for broadening the functions of existing RSMCs and for designation of new RSMCs should be applied to the designation of Global Producing Centres. In order to be officially recognized as a GPC, organizations must as a minimum adhere to:
Fixed production cycles and time of issuance.
Provide a limited set of products as determined by the revised Appendix II-6 of the Manual on the GDPFS.
Provide verifications as per the WMO SVSLRF.
Provide up-to-date information on methodology used by the GPC.
Products will be accessible through the GPC web site or disseminated through the GTS or Internet.
GPCs are further committed to participate in development and research programmes, training users (RCCs and/or NMHSs) on LRF products.
CBS will monitor scientific progress with a view to improving LRF products and services.
RECOMMENDATIONS AND FUTURE WORK
3.6 The Meeting recommends:
that GPCs conduct studies to assess the optimum ensemble size.
that CCl provide guidance on the use of climatological normals as a basis for forecasting LRF anomalies, in consideration of both climatological shifts as well as availability of hindcast data sets.
that WCRP Working Group on Seasonal to Interannual Predictions and CAS Working Group on Numerical Experimentation consider conducting intercomparison studies on the relative performance of Tier-1 and Tier-2 LRF models and make available the results to GPCs.
3.7 The future work of the ET on Infrastructure of LRF should include, in addition to the current terms of reference:
To develop user interpretation guidance to facilitate the correct use of LRF anomaly forecast;
To further develop the infrastructure needed for provision and exchange of LRF Multi-Model Ensembles.
STATUS OF VERIFICATION FOR LRF INCLUDING LEAD CENTRE
4.1 The status of the implementation of the SVSLRF was reviewed. It was noted that APCN, IRI, CMC, JMA, ECMWF and UKMETO were computing SVSLRF verification results. It was found that JMA had the most comprehensive system. All centres were congratulated for their achievements in implementing the SVSLRF.
4.2 A presentation was made on the draft Lead Centre web site which has now been established. The Meeting was impressed with the work carried out by the co-lead centres, and the progress made since the meeting in December 2003. The web site is planned to be made operational in the first half of 2005.
4.3 The Meeting reviewed and agreed on the recommendations on the role and responsibilities of the Lead Centre, given in the annex to this paragraph, that would contribute to the further development of the activities of the Lead Centre Web site with links to the GPCs, and the development and provision of relevant software.
SVSLRF issues arising from the Lead Centre meeting
4.4 The Meeting reviewed the recommendations made at the meeting of the Lead Centres on long-range forecast verification held at the Canadian Meteorological Centre (CMC) in Montreal, Canada, 1-5 December 2003. Six experts (from CMC-Canada (2), Melbourne, NCEP (NOAA), IRI (USA) and ECMWF) and the Secretariat representative participated in the meeting. The main purpose of the meeting was to review the role of the Lead Centres in the exchange of long-range forecast verification results as specified in the Standardised Verification System (SVS) for Long-Range Forecasts (LRF) defined in the WMO Manual on the Global Data-Processing System.
4.5 The Meeting supported the recommendation that verification should be carried out on post-processed output and suggested that this be clarified in the documentation of the SVSLRF. However, the Meeting recommended that the team consider whether this should be applied to the MSSS and its three-term decomposition. The decomposition would be more meaningful on the raw data and of more use in model development.
4.6 The Meeting agreed that error bars are more informative than significance levels but noted the problems in map presentations. It agreed that there should be shading of significance areas for maps and error bars for line plots. It also agreed that error bar (and significance testing) calculations should take into account the spatial dependence of grid point values. The Team agreed to develop appropriate guidance material as soon as possible and make it available on the Lead Centre site. Statistics on the Lead Centre site should not include significance level information until this guidance had been prepared.
4.7 The Meeting also agreed that more information on significance testing should be developed by the Team and made available in the documentation.
4.8 The Meeting did not agree, at this stage, to recommend that ROC score (area under the ROC curve) should be calculated as the area under a fitted curve through the ROC points. The Meeting recognised that the accuracy of the ROC score could be improved with the use of a fitted curve instead of a straight line between the ROC points. However, it also noted that fitting implied more bin resolution than was available, and also that without additional processing there would be inconsistencies between the plots of raw data on the Lead Centre web site and the computed area. The Team should resolve the issue, but until then the current documentation would remain and software on the Lead Centre web site would use the Trapezium Rule.
4.9 The Meeting agreed that a list of official global producing centres should be maintained and that it was more appropriate for the Secretariat to perform this role.
4.10 The Meeting agreed that the real time verification is the role of the producing centres and that real time scores will not be posted on the SVSLRF web site. The documentation should be clarified on this point.
4.11 The Meeting agreed that more guidance should be prepared on the prescription of the cross-validation procedure (number of years skipped, etc.). It also felt that there should be separate procedures for dynamical and empirical schemes. The Meeting also felt that there was little gain and considerable extra effort in applying cross-validation to individual dynamical models, except where some form of MOS correction is used. It will be required for multi-model ensemble schemes that involve more than a simple averaging of individual model forecasts. The Meeting agreed that this was a topic that needed further consideration and advice.
Documentation of SVSLRF
4.12 The Meeting agreed with the recommendation that attachments II.8 and II.9 in the manual on GDPS be revised and unified and, in doing so, care should be taken to clearly specify the core verification system and what is additional. A chapter on the role of the Lead Centre should be added.
4.13 The Meeting recommended that the definition of ENSO years in paragraph 7 of attachment II.9 to the Manual on GDPS should be removed and that this table should be displayed on the SVSLRF web site. It acknowledged that there is debate at present on the specification of the ENSO years and would take advice from the appropriate expert group on the classification of years. In the meantime the current list would be used as the basis for ENSO stratification.
4.14 The Meeting also agreed that where possible the verification datasets be confined to one per parameter and the choice of datasets be periodically revised. However, the Meeting recognised that further advice was needed on the selection of the precipitation data set. The Meeting also recognised the practical problems that had been encountered in the use of the CRU surface temperature data set that has gaps in the data. For verification of 2m temperature the Meeting recommended the use of:
ERA40 Analysis fields for a globally complete data set
The CRU 0.5 degree data set for land areas (New et al., 1999; New et al., 2000)
4.15 The Meeting noted that other areas for clarification, addition and correction had been made in the draft revised and consolidated documentation.
Publicising the Lead Centre activity
4.16 The Meeting also agreed that the WMO Secretariat send a letter to GPCs to invite them to submit their verification results to the SVSLRF web site once the latter one is ready. Information also has to be sent to RCCs and NMHSs on the role of the Lead Centre.
4.17 The Meeting also recommended that the WMO Secretariat request RCCs and NMHSs for feedback one year after the launch of the SVSLRF web site. A questionnaire will be developed and processed by the Lead Centre.
4.18 It is recommended that long-range forecast verification and its importance be discussed and the Lead Centre activity be publicised at the Workshop of Global Producers of Seasonal to Inter-Annual Forecasts planned in 2005.
4.19 On terminology, the Meeting suggested that the term “Lead Centre” refer to both co-lead centres (Melbourne and Montreal) together as the functions are being distributed between the two. The web site is being jointly developed and maintained by the two co-lead centres and is designated as the “SVSLRF web site”.
Other verification issues related to Infrastructure for LRF
The Meeting noted that information on significance levels is required as part of SVSLRF. GPCs could consider how to use this information in the products they make available such as through masking areas of low skill.
4.21 It was also agreed that the Lead Centre site should retain any verification of forecast systems that are superseded for the ongoing information of users, rather than only holding statistics on the current operational system.
4.22 The Meeting also noted that the list of fields to be required in the formal designation of GPCs contained MSLP and upper level fields not included in the SVS. The SVS was deliberately confined to a small number of fields of most direct relevance to end users. SVS statistics for these fields is not required for publication on the Lead Centre site. However GPCs may decide to apply SVSLRF to these extra fields and make information available.
Remaining tasks for the Team 4.23 The Meeting recommended that the expert team on SVSLRF continue its work for the next period. The above paragraphs list tasks that need to be completed and issues to be resolved. Other areas that may need future consideration to augment the SVSLRF are:
Development of scores to measure skill in the ensemble spread
Assessment of multi-model ensembles
Standardising methods for defining terciles, etc.
Verification of extremes (such as the outlying quintiles)
Ongoing coordination and support of Lead Centre role
Clarification of issues arising from the broader use of SVSLRF
5. EXCHANGE OF ENSEMBLES PRODUCTS AND DEVELOPMENT OF MULTI-MODEL ENSEMBLE SYSTEM (MMES)
The Meeting reviewed the scientific benefits of the development of multi model ensemble for long-range forecasting and noted progress in APCN, European and IRI systems.
5.2 The APCN system: 5.2.1 The objective of APCN is to establish a well-validated multi-model ensemble system (MMES) for short-term climate prediction. The APCN MMES is based on the global models developed at different institutes of several APEC member economies, which have been partially validated in operational climate seasonal forecasts. At present, fifteen GCM models based on a 1 or 2-tier approach are participating in the real-time multi-model ensemble experiments to build up the infrastructure for the joint operational seasonal forecast. Currently, the hindcast data with the observed SST and sea ice for 21 years from 1979 to 1999 are used in training the models and cross-validation.
5.2.2 Participating institutes are requested to provide global hindcasts for more than twenty-years and real-time forecasts. The hind-casts, which will be used to develop the MMES, should be made, as far as practical, the same way that the real-time forecasts will be made. That is, they should not use any data that would be unavailable if the forecast was being made in real-time. The data format and other specifications will be, as far as practical, identical to the protocols established for SMIP2/HFP.
5.2.3 The APCN Secretariat located in KMA is responsible for the processing of dynamic ensemble prediction data and making it available to the participating members. The multi-model ensemble products are distributed through the APCN web site (http://www.apcn21.net). The APCN Secretariat is operating a Visiting Scientist Program starting from 2004 to employ eminent experts from all over the world in the areas of climate monitoring, prediction and application for the data processing and development of multi-model ensemble techniques.
5.2.4 The APCN developed various MME techniques for deterministic and probabilistic seasonal predictions. For deterministic forecast, three kinds of linear MME techniques are used, namely biased and unbiased simple composite, weighted combination of multi models based on SVD, and MME with statistical corrections. For probabilistic forecast, three tercile ranges are determined by ranking method based on the percentage of ensemble members from all the participating models in those three categories.
5.2.5 The practical importance of climate forecasts for the protection of life and property, together with concerns about environmental change, have led to the initiation of the APEC Climate Network (APCN) project. The project’s most important challenge is to provide accurate and reliable climate information for member economies in the Asian-Pacific region. The major aspect of APCN-MMES is the development of new techniques for providing forecast information to the user community.
5.2.6 The new techniques include the downscaling method to make a seasonal forecast at a target region. The range of seasonal forecasts and detailed climate information required and provided will be developed through dialogue between the scientists involved in research projects and the forecasters in NMHSs.
5.2.7 The results of the project will benefit both public and private meteorological and hydrological institutions in the Asia-Pacific region. Those institutions without the capacity to produce climate predictions will be able to access optimized, state-of-the-art global climate predictions. The predictions should enable national and international disaster prevention offices to respond more effectively to natural disasters and mitigate economic losses in the case of extreme climate events.
5.3 The European system
5.3.1 A European multi-model ensemble seasonal prediction system comprising 7 coupled ocean-atmosphere GCMs (CGCMs) from European centres has been developed and validated under the European project DEMETER. Three participants in DEMETER, ECMWF, UK Met Office and Meteo-France are developing a real-time MMES on the ECMWF computing facility. CGCMs from other European countries may join the multi-model in future.
5.3.2 Current specifications for the forecast and hindcasts of the three component models are as follows:
5.3.3 Of the three component models, the ECMWF seasonal model (system2) is fully operational. The Met Office model (GloSea) is running in real-time at ECMWF and will transfer to the ECMWF operational suite within the next few months. A schedule for the transition of the Meteo-France model from research to operations is yet to be defined. All partners have an upgrade scheduled for 2005 implying the construction of new hindcast sets.
5.3.4 It is difficult to indicate the optimal hind-cast ensemble size that could limit the computer demand and still provide enough information on the forecast probability distribution and its skill. Sub-sampling of DEMETER data shows a lack of stability in skill scores (only tropical precipitation seems to be robust). This indicates that the uncertainties associated with the insufficient length of the hind-cast period are the ones that are more critical for the forecast skill estimate.
5.3.5 The initial multi-model product will be based on a simple average of the 3 probabilities associated with each individual system. At a later stage Bayesian techniques will be extended to the multi-model. Raw (non calibrated) data will still be accessible.
5.3.6 Multi-model products will be available on the web. Estimates of the multi-model skill levels and of the skill of each individual component will be made available. Such estimates will be evaluated using methods that have been agreed by the SVSLRF. Documentation of the multi-model products and of each individual component will be also available.
5.3.7 ECMWF has implemented changes in its archiving system in order to handle the multi model data. A multi-model operational archive will contain all the members from each system.
5.3.8 Digital data from the real-time multi model will comply with the WMO recommendations.
5.3.9 A new project, ENSEMBLES, to develop provision of policy-relevant information on climate variability from seasonal through decadal to climate timescales has started under the European Union’s Framework VI Programme. ENSEMBLES will include further development of seasonal coupled ocean-atmosphere multi-model systems and their applications and will extend the forecast range to multi-annual and decadal timescales.
5.4 The IRI system 5.4.1 The IRI has been producing operational multi-model ensemble forecasts since 1997, and has been applying objective combination schemes since 1999. A published schedule for forecast release was implemented earlier in 2004, and is synchronized with the forecast release dates by NCEP. The IRI forecasts are based on an evolving suite of dynamical models, all of which involve two-tiered schemes. Currently these forecasts involve the combination of ensemble forecasts from six dynamical models, most of which are contributed by other centres. Different sea-surface temperature scenarios are used, and there are a range of different numbers of ensemble members per SST scenario and model.
5.4.2 Hindcasts for one of the models of the multi-model ensemble system run at the IRI are available from the IRI Data Library. For the other models, hindcasts based on observed SSTs only are available. Although most of the required scores of the SVSLRF have been computed, these have not yet all been made available. There are plans to advance the implementation of the SVS early in 2005.
5.4.3 Preliminary investigations by the IRI into possible methods for recalibrating probabilistic forecasts from dynamical models and combining them into a multi-model forecast have been conducted. The baseline against which more sophisticated techniques were compared consists of straight pooling of the raw ensemble probabilities from the individual models.
5.4.4 Several conclusions emerge:
Dynamical models are over-confident in their probability distributions, and dynamical predictions benefit in reliability from combining several models together by simple averaging;
Further improved reliability of seasonal climate forecasts can be achieved by recalibrating the forecasts;
Given the small sample sizes that typically are available for seasonal climate forecasts, best results may be achievable through simple combination schemes after recalibration of forecasts from individual models.
DATA NEEDS FOR LRF
6.1 The Meeting reviewed the data needs for LRF and suggested an updated SOG for inclusion in the rolling requirements review process conducted by the CBS/OPAG on IOS as given in the annex to this paragraph.
7. OPERATIONAL PROCEDURES AND PRACTICES 7.1 The Meeting considered necessary updates to operational practices (including the required verification system for LRF, which is included in the Manual on GDPS). These technical regulations provisions and practices are to be followed in terms of the information on validation results to be attached to the long-range forecasts products. The proposed updates arise in the light of operational experience and progress in research on verification activities and include the accepted recommendations as discussed under item 4.
7.2 The proposed updates, given in Appendix III to this report, will be presented for review by the ICT of DPFS with a view to their consideration and recommendation by the next CBS session.
8. CLOSURE OF MEETING 8.1 The meeting was closed at 16:00 on 19 November 2004.
The Australian Bureau of Meteorology currently maintains an operational (statistical) model for 3 month outlooks of Australian rainfall, maximum and minimum temperature, as well as a dynamical coupled (i.e., numerical climate prediction) model: the Predictive Ocean Atmosphere Model for Australia (POAMA).
Currently the POAMA-2 system is under development with a number of changes scheduled to be made to the ocean data assimilation and the atmospheric model. It is planned that POAMA-2 will be completed during 2005. POAMA-3, which will incorporate the new Australian Climate Ocean Model (AusCOM-1) being developed by the Australian ocean modelling community (BMRC, CSIRO, TPAC1, ACE-CRC2, Universities, etc.), will be available sometime between 2006 and 2008.
Seasonal and Longer Forecasts
The operational Seasonal Climate Outlook (SCO) model produces a seasonal forecast for Australian rainfall, minimum temperature and maximum temperature every month. The outlooks are available from the Bureau of Meteorology’s public seasonal outlook web site (http://www.bom.gov.au/climate/ahead/).
Public (freely available) forecasts are issued as the probability of exceeding the median. Subscribers to the SCO also receive tercile probability forecasts. The subscription service also contains subsidiary seasonal rainfall outlook information generated using stratified climatology techniques.
The POAMA system produces a global nine-month forecast each day using the latest ocean and atmospheric initial conditions. Products are based on the latest 30 daily forecasts, forming an effective 30-member ensemble. Products are available on the POAMA web site (http://www.bom.gov.au/bmrc/ocean/JAFOOS/POAMA/ ).
The SCO system has undergone a number of skill assessments, including LEPS, ROCS, Brier Scores, Value Scores and Percent Consistent rates (categorical hit rate). These have been based upon cross validated hindcasts from 1950 to 1999 (50 years). Currently most of these scores are not available externally but are used for internal decision making.
The exception is the Percent Consistent rates, which are available as maps, both for the 12 seasonal outlook periods. Raw hindcast data are also made publicly available.
A series of hindcasts were performed at the Canadian Meteorological Centre (CMC) using the Recherche en prévision numérique (RPN) Global Environmental Multiscale (GEM) model (Côté et al. 1998a and 1998b), in seasonal configuration, under the SMIP2 protocol. The GEM model is a grid point global model, which in that configuration was run at 1.875 horizontal resolution with 50 levels in the vertical. The physical parameterizations used were very similar to those used in the former hindcasts using the SEF model (Ritchie 1991). Main exceptions were the introduction of a blocking parameterization term that parameterizes the subgrid scale orography (Zadra et al. 2003) and a new surface scheme that can take into account four different surface types per tile.
Analysis of these hindcasts revealed that the level of skill in predicting seasonal surface temperature (T2m) anomalies is generally equivalent to the previous two models used in seasonal forecasting in Canada (Derome et al. 2001). However, the new model GEM demonstrates a superior skill for precipitation anomalies, although its skill remains modest. Over Canada, the overall behaviour is similar to previous models results, but not necessarily over the same seasons and regions. The model exhibits better skill in spring for the surface air temperature (SAT) anomalies, but less in fall and early winter. The skill of the accumulated precipitation anomalies demonstrates a systematic improvement, but is still far less than the SAT results.
The operational long-range forecasts (both monthly and seasonal) deterministic and probabilistic are available at: http://meteo.ec.gc.ca/saisons/index_e.html. Seasonal forecasts (deterministic and probabilistic) are always presented together with their expected skill calculated over the hindcast period. Since February 2003 the raw operational forecast data (monthly and seasonal means) can be accessed at http://www.cccma.bc.gc.ca/data/cmc/cmc.shtml and the hindcast data are available as well at http://www.cccma.bc.ec.gc.ca/data/hpf/hpf.shtml
A study of the impact on the skill of a multi-model ensemble forecast was made. We have a seasonal hindcasts for 3 models (RPN SEF, RPN GEM and CCCma AGCM2) covering 1969-1994. The skill increased with the number of members as well as with the number of models. Also, for the same number of members it is always better to use more models. For example, we can see that using 1 member from 3 different models is better than using 6 members from 1 model. The improvement from passing from 1 model to 2 is greater than to go from 2 models to 3. The rate of increase Therefore with our current system it is a better choice to increase the number of models than to increase the number of members.
This is why although the GEM model has replaced the SEF model in the operational suite in February 2004, this latter model will be re-introduced in the ensemble once a new hindcast on the new supercomputer is done.
The CMC is producing verification results for seasonal forecasts at levels one and two, as specified in the Standard Verification System for Long-range Forecasts (SVSLRF), attachments II.8 and II.9 to the Manual on the Global Data-Processing System, Volume I, and has agreed to make the results available. The scores were calculated for the normalized average of the seasonal anomalies of the RPN SEF and the CCCma GCM2. The verification data sets used are the CRUTS1.0 data sets (New et al. 1999 and 2000) both for temperature and precipitation anomaly forecasts. The SVSLRF verification results are available on the web. The login name is “svs”, password is “oui”: http://www.cmc.ec.gc.ca/~cmcdev/saisons/SVSLRF/SVSLRF_results.html.
An operational ensemble prediction system (EPS) for three-month outlook commenced in March 2003 using the T63L40 version of the JMA Global Spectral Model (GSM) with prescribed SST anomalies fixed to their initial conditions. The SST anomalies used as the lower boundary condition of the atmospheric model are provided by a combination of climatology, persistence and predicted SST anomalies by a coupled ocean-atmospheric model (JMA CGCM02). JMA has introduced numerical prediction techniques into all ranges of operational forecasting for one, three and six months.
The operational dissemination of GPVs (grid point values) and visualized results of EPS for the warm/cold season outlook to NMHSs (National Meteorological and Hydrological Services) was started in February 2004 on the website of the JMA’s Tokyo Climate Center (TCC) (http://cpd2.kishou.go.jp/tcc/), adding to those for one-month and three-month prediction. Precipitation, 2m-temperature, sea surface temperature (SST), 500hPa geopotential height, mean sea level pressure, 850hPa temperature, zonal and meridional wind components at 200hPa and 850hPa are included in the disseminated parameters at the present. The GPVs for all ensemble members are available for the EPS of the warm/cold season outlook, but not yet for that of the three month outlook.
The Standard Verification System (SVS) for Long-Range Forecasts (LRF) is mostly applied to the hindcast products of EPS for three-month and warm/cold season outlook. The scores for the SVS level-1 and level-2 and the contingent tables for SVS level-3 are open to NMHSs on TCC site.
The operational model results of 1-month and seasonal predictions are available on the KMA web page http://188.8.131.52/kmas/kma/english/ema09/ema0901.html .
KMA has developed an El-Nino/La Nina prediction model using the intermediate-ocean and statistical-atmosphere coupled model. The present ENSO prediction model is the modified version of the Lamont Model (Zebiak and Cane, 1987). The model predictability was improved by changing the ocean initialization method, by modifying the model dynamics on the parameterization of subsurface temperatures, by introducing statistical atmosphere model, and the use of NCEP reanalysis wind stresses
The KMA continues the improvement of operational LRF system in cooperation with universities and research institutes. The future LRF systems under research and development include 12-month forecast system, 3-month forecast system and seasonal/annual typhoon forecast system. The 12-month forecast system will be based on the global SST prediction & tier-2 LRF system, coupled GCM (tier-1 system) and statistical downscaling. The 3-month forecast system will be based on regional dynamical and statistical downscaling and multi-model ensemble. And, the seasonal/annual typhoon forecast system will be based on the dynamical and statistical methods. These systems are being developed for operational use in 2006.
SAWS, South Africa
Research and development during 2002/03 on long-range forecasting (LRF; from 30 days up to two years) at the South African Weather Service (SAWS) have primarily been on improving forecasting methods predicting 3-month seasonal rainfall anomalies. Additional research activities also include the implementation of a seasonal forecasting verification system, investigation into the use of multi-model ensembles, the sensitivity to domain choice in statistical downscaling, and the implementation of the RegCM3 regional climate model.
Implementing a multi-model MOS seasonal forecasting system using the GCM forecasts of the COLA T63 (SAWS), HadAM3 (UCT), Conformal Cubic Atmospheric Model (to be implemented at the University of Pretoria) and ECHAM4.5 (IRI);
Implementing a multi-regional model seasonal forecasting system using the RegCM3 and RSM;
Producing monthly rainfall and temperature probabilistic forecasts using MOS;
Expanding on the password protected web site from where large-scale GCM fields, archived and real-time, are available (http://www.gfcsa.net). Contributors to the web site are African scientific institutions that can produce global forecast fields operationally;
Verifying LRF operational forecasts for the years starting in 1998 until present, using the WMO Standard verification System for LRF products;
Expanding on research pertaining to the predictability of intra-seasonal rainfall characteristics;
Producing seasonal probabilistic forecasts of streamflow at the inlets of certain dams in South Africa using MOS;
Investigating the relevance of vegetation and soil moisture for operational LRF in southern Africa;
Investigating the predictability of extremely dry or wet rainfall seasons.
Forecast Verification Statistics
A verification system was installed at the SAWS by Dr. Simon Mason of the IRI. The SAWS has been issuing probabilistic LRF since 1998, and the implemented system will assess the operational forecast skill over South Africa from 1998 to present.
A coupled ocean-atmosphere GCM system (known as GloSea) has been used to produce real-time global seasonal forecasts since March 2003, replacing a two-tier system (AGCM plus persisted SST anomalies) used prior to that date. Comparisons of the GloSea and AGCM systems conducted as part of the EU DEMETER project show substantial skill benefits for GloSea, particularly in the tropics but also in some parts of the extratropics (Graham et al., 2005).
1.2 Recent operational implementations and model changes
An upgraded GloSea version (GloSea2) was implemented with the October 2004 forecast. With this upgrade, running of GloSea2 was moved into the ECMWF operational suite, with output parameters and formats adjusted to conform to an agreed multi-model framework. System changes included a minor change to the perturbation strategy, to enable perturbations in hindcast and forecast runs to be constructed in a consistent way (partly as a consequence of this change the ensemble size was increased from 40 to 41). Additionally, the ocean perturbations, initially considered too large, were reduced by adjustment of perturbations in the observed windstress, and by increasing the influence of observations on the analysis. A new set of calibration hindcasts with GloSea2 is currently underway.
2. Current availability of digital hindcast output
Hindcast output from GloSea2 is currently available, to registered WMO users, from the developing European seasonal multi-model archive (see www.ecmwt.int/services/archive ).
3. Seasonal forecast products for NMHSs and the public
Seasonal forecast products for NMHSs, RCCs and public users are available at www.metoffice.gov.uk/research/seasonal . Forecasts provided are for 3-month-mean temperature and precipitation at 1-, 2- and 3-months lead and forecasts of 1-month-mean SST anomaly for the Niño3, Niño3.4 and Niño4 regions to a range of 6-months
4. Seasonal forecast validation/verification
4.1 DEMETER verification
Extensive validation of the old (GloSea1) system has been performed over a 43-year (1959-2001) hindcast dataset as part of the EU project, DEMETER. Hindcasts were performed in a 9-member ensemble, for 4 start times each year (1st Feb, 1st May, 1st August and 1st November). Results for SVSLRF variables (and many others) are available at www.ecmwf.int/research/demeter/. Limited comparison of GloSea1 and the currently operational GloSea2 for atmospheric performance indicates similar skill for both model versions.
Currently available verification products for 2m temperature and precipitation forecasts
All products are level 2 (map format) and for the hindcast period 1987-2001.
Gerrity Skill Scores (GSS) for deterministic categorical forecasts. Cases when one category has probability > 40%, and the other two categories have probability < 33.3% are assessed as deterministic forecasts of the most likely category.
Mean Square Skill Score (MSSS) for the (calibrated) ensemble mean anomaly. Results are provided in the form of a skill mask which ‘blanks out’ regions where MSSS<0.10
ROC score for two-category (above/below) probability forecasts. Displayed in form of a skill mask which ‘blanks out’ regions with ROC area < 0.6
NB: the skill mask format will likely be replaced by full skill maps in new verification products.
5. Monthly forecasting
Met Office monthly forecasts products are currently generated for commercial purposes and are not provided ‘free to air’ on the website, though this is under review. Monthly forecasts are generated using a new coupled ocean-atmosphere, 51-member, monthly-range ensemble forecast system developed and run by ECMWF
Plans for future development include
Probability forecast products for outer-quintile ‘extremes’
Calibration of probability forecasts according to past performance
Implementation of multi-model combination techniques and multi-model products
Further compliance with WMO recommendations on infrastructure and verification.
Investigation of ocean state land-surface initialization methods, multi-annual and decadal forecasting (as part of the new EU project, ENSEMBLES).
Testing of the Hadley Centre’s next generation CGCM (HadGEM) for seasonal prediction
The U.S. seasonal outlooks generated by CPC incorporate forecast guidance from many different seasonal prediction tools that can be categorized as empirical or dynamical seasonal prediction methods. Empirical seasonal prediction methods include: Canonical Correlation Analysis (CCA); Optimum Climate Normal (OCN), a prediction tool based on long-term climate trend information; and observed composites based on Niño 3.4 SST index, among others. For each empirical prediction tool a long history of cross-validated hindcasts, together with an estimate skill is also available.
Guidance from dynamical seasonal prediction is based on a Coupled Forecast System (CFS) run operationally at the National Centers for Environmental Predictions (NCEP). CFS is a coupled ocean-atmosphere general circulation model and is developed by the scientists at the Environmental Modeling Center (EMC) at NCEP. In September 2004, CFS became operational at NCEP and replaced the Seasonal Forecast System (SFM). Since its operational inception, the real-time CFS forecasts are made once per day, and are of 10-month duration. Predicted SST and atmospheric anomalies are computed from the climatologies obtained from the hindcasts archives, and are provided to CPC forecasters. Together with the predicted seasonal anomalies, forecasters also have access to historical skill levels obtained from verifying hindcasts against the observations. http://www.cpc.ncep.noaa.gov/products/people/wwang/cfs_fcst4CPC/
To summarize, each month CPC forecasters are provided with multitude of seasonal forecast guidance from many different tools. This information is consolidated into a final forecast product that is disseminated to the user community. CPC has been issuing seasonal outlooks since 1995 in their current format. Further information about CPC seasonal outlooks can be found at:
http://www.magazine.noaa.gov/stories/mag143.htm The current status of verifications Seasonal climate outlooks issued by CPC are routinely verified using categorical and probabilistic verification procedures. Of more widely used is the categorical verification based on Heidke skill measure
Since CPC seasonal outlooks are in terms of anomalous probabilities, CPC has also started to verify their forecasts in terms of probabilistic measures, for example, Rank Probability Skill Score (RPSS). CPC also maintains a graphical web archive of its previous forecasts and verifying observations. http://www.cpc.ncep.noaa.gov/products/archives/long_lead/llarc.html
ECMWF 1. Introduction Since 1997, ECMWF issues global seasonal predictions routinely every month. In 2000 the seasonal forecasts became part of the operational products and by mid-2000 the seasonal forecast products became available to the all the WMO members.
The ECMWF seasonal forecast is a dynamical system consisting of a coupled atmosphere-ocean model and an ocean analysis. During 2002, a substantial upgrade was made to the seasonal forecasting system. A brief description of the operational system is given in section 2. In section 3 products and verification are discussed. Section 4 describes envisaged future implementations.
2. The current seasonal forecasting systems
The current seasonal forecasting system, was introduced into operational use at the beginning of 2002. It differs from the original system in a number of ways. The atmospheric component is CY23R4 of the IFS with a horizontal resolution of TL95 and 40 levels in the vertical. This is the same cycle of the IFS as was used in the ERA40 re-analysis. The ocean model resolution was increased to 0.3 degrees meridionally near the equator and to 1 degree x 1 degree at higher latitudes; the vertical resolution of the ocean increased from 20 to 29 levels. Changes were also made to the ocean model physics, mainly the parameterisation of vertical mixing.
As with all models, the seasonal forecast system is not perfect. One symptom of this is climate drift: the model climatology does not match the observed climatology. To account for this, the forecasts need to be referenced to the model climatology.
The estimate of the model climatology is based on an ensemble of 5 integrations spanning the years 1987- 2001. This 15-year climate gives a more stable basis for computing anomalies than the 6-year climate available in the original system. For a further description of the original and operational system, including an assessment of their different characteristics see Anderson et al 2003.
3. Seasonal forecast products and verification 3.1 Accessing data and products A selection of graphical products from the seasonal forecast system is displayed on the ECMWF webpages. All plots can be downloaded as postscript or pdf files, as well as being viewed on screen. Global spatial maps of 2 metre temperature, precipitation and mean sea level pressure are shown, in the form of probabilities for tercile and 15%ile categories as well as the ensemble mean anomaly and the probability of exceeding the climate median. The Nino SST indices include the Nino 3.4 and Nino 4 regions as well as Nino 3, and the ocean analysis plots include several meridional sections, as well as zonal and horizontal maps.
A large number of different model fields from seasonal forecast (both forecast and hind-cast) is archived although only a small subset of these are presently listed in the ‘ECMWF catalogue’ for commercial use. A full list of the output fields can be found in section 3 of the online Seasonal Forecast User Guide, at http://www.ecmwf.int/products/forecasts/seasonal/documentation
Ocean analysis data are also archived. For further details see:
http://www.ecmwf.int/products/forecasts/seasonal/documentation/ch3_2.html 3.2 Verification For a correct interpretation of seasonal predictions the user needs to complement the forecast products with knowledge of the forecast skill. The site at
provides a comprehensive documentation of skill levels, using methods that have been agreed at the international (WMO) level for the evaluation of long-range forecast systems.
4. Future developments The new seasonal forecasting system at ECMWF gives users access to a much wider range of products and data, and much better information on the performance characteristics of the system. Its ability to forecast El Nino type SST variability is well established, although the forecasts are not yet completely reliable. Based on a limited sample of ~15 years, the statistics suggest that there are many areas and parameters for which the atmospheric forecasts also have some skill, but the results are geographically variable and subject to sampling error.
Model error is a serious source of forecast error but this can be partly addressed by the use of several models. In the near future we plan to include the Met Office and Meteo-France models as part of the seasonal forecasting system and hopefully to include other models later. There is much work still to be done, but we are confident that we will continue both to improve our model forecasts, and to improve our ability to represent the forecast uncertainties.
The land surface is recognised as important on seasonal and sub-seasonal timescales, both in terms of modelling and initialization. Seasonal forecast skill could benefit from further work on this topic.
Annex to paragraph 3.5.3
Suggested amendments to the Manual on the GDPFS - Appendix II-6
Add at the appropriate place:
Minimum list of LRF products to be made available by global scale producing centres 1. Forecast Products Note: it is recognised that some centres may provide more information than the list below on the basis of the CBS-Ext.2002, Appendix V and may also include for example daily data, hindcast data.