Research overview
It is necessary to quantitatively test the accuracy of weather model simulations to estimate the probability of weather phenomena. For this purpose, we developed two verification methods and verified the degree to which the estimation accuracy was improved in the newly developed weather model in this project.
The first method was to conceptualize a virtual world that set ideal conditions to theoretically determine the nature of weather phenomena, such as cumulonimbus clouds that form, and to assume that complex physical phenomena did not occur. Then, we derived an actual probability distribution of the state of the phenomena, such as the location of cumulonimbus clouds and precipitation amount. Next, we performed simulation calculations based on physical laws in the virtual world to estimate the probability distribution. This method measured the closeness degree of the probability distribution predicted using a weather model to the actual probability distribution under the same conditions.
The second method tested whether or not the probability distribution predicted by the weather model was plausible based on observational data, such as the location and amount of precipitation of cumulonimbus clouds that caused wind and flood damage in the past.
By combining these verification methods, we comprehensively verified the probability estimation accuracy of the weather models.
Details of research
Our project aimed to develop a new weather model capable of accurately estimating the probability of weather phenomena, that is, whether the occurrence location and time, precipitation amount, and wind speed of weather phenomena, such as cumulonimbus clouds and tornadoes in certain areas under certain conditions, occur deterministically (inevitably) or randomly (coincidentally). For example, under certain geographical and weather conditions, there are various possibilities for the formation of cumulonimbus clouds: if a location with low probability is determined by chance, it may be possible to direct the cumulonimbus clouds to the sea with no humans present, by slightly changing the conditions. If the estimated probability is accurate, it is possible to select a weather control method suitable for the area and conditions, thereby preventing wind and flood damage.
How can the probability-estimation accuracy of the developed weather model be determined? As there are few studies on probability estimation, no method has been established to verify the estimation accuracy. Therefore, we began by developing a verification method.
To estimate the probability of a weather phenomenon occurring under certain conditions, it is necessary to quantify the probability distribution, which represents the extent to which occurrence location and time, precipitation amount, and wind speed of phenomena, such as cumulonimbus clouds, vary when conditions, such as air temperature and pressure distributions, are gradually changed.
For example, in the case of chemical experiments conducted in a laboratory, it is possible to measure the changes in a chemical reaction after the chemical concentration and temperature are altered. However, such experiments cannot be conducted on a global-to-regional scale as with natural weather phenomena, such as cumulonimbus clouds. In other words, the correct answer, that is, the actual probability value, is unknown.
It is difficult to gauge the closeness degree (the goodness-of-fit) between the probability estimated by the weather model and the actual probability observed when the actual probability is unknown.
We examine the accuracy of the probability estimation in two ways.
The first method considered ideal conditions and/or a simplified virtual world and determined the probability distribution of the occurrence location and precipitation amount of weather phenomena, such as cumulonimbus clouds. This method quantified the similarity of the predicted probability distributions of the new and old weather models to the actual probability.
For example, it is possible to theoretically determine the probability distribution of the phenomena by setting ideal conditions, which theoretically reveals the type of cumulonimbus clouds to form, such as flat and homogenous topography and uniform spatial conditions of temperature, wind speed, and amount of aerosol required to generate clouds.
In another example, by considering a simple virtual world that assumed that complex physical phenomena did not occur, such as water vapor transforming into water but not into ice, it is possible to obtain the probability distribution of a weather phenomenon from precise simulations based on physical laws at an extremely fine resolution.
This method measured the closeness of the predicted probability distributions of the new and old weather models to the actual probability under the same conditions.
In reality, terrain, temperature, wind speed, and aerosol amount vary spatially and temporally. In addition, complex physical phenomena are involved in weather events. Therefore, attention must be paid to the fact that the evaluation obtained by comparing the probability distribution of the weather models under ideal conditions and in a virtual world with a provisional correct answer does not necessarily reflect the real world. Nevertheless, this evaluation method is able to directly compare the predicted probability distribution of a weather model to that assumed to be correct.
This figure only shows the probability distribution of the location as an example, but it could include the occurrence time, precipitation amount, and wind speed.
Another method was to compare the observation data of past weather phenomena that caused wind and flood damage to the probability distributions calculated using the old and new weather models. The observation data in a given case are only one realization under this particular condition. Even if there is the possibility of various states under slightly different conditions, it is not possible to obtain such information. Therefore, to verify the probability-estimation accuracy of the weather model, a comparison to the observational data is necessary in many cases.
For example, in a certain case, suppose that a cumulonimbus cloud forms at Point A. If the new model yields a higher probability of the cumulonimbus cloud formation at point A than the old model, the prediction accuracy of the new model is considered to be high. In other cases, the old model may predict a higher probability, or both may predict an equal probability. By comparing the probability distributions estimated by the old and new models to the observation data for a large number of cases, if there are more cases where the new model closely predicted the observation data with higher probability, it can be concluded that the probability-estimation accuracy of the new model has improved.
By collecting information on where the observed values are located in the spread of the estimated probability distribution in many cases, we can measure the plausibility of the spread of the estimated distribution. When the estimated probability distribution is correct, the observed values are more likely to be located near the center of the distribution than toward the ends of the distribution. The estimation accuracy can be determined by considering the distribution of the positions of the observed values relative to the estimated probability distribution in many cases.
Data from a very large number of cases are required to perform a highly reliable verification based on observation data. A large amount of computation is required to simulate real cases, and even if a supercomputer is used, the number of cases that can be simulated is limited. Although it is difficult to verify whether the new or old model has higher estimation accuracy, as this verification method uses actual observation data, it has the advantage of being able to conduct verification under real conditions.
We strive for establishing a comprehensive and highly reliable verification method by combining the two different verification methods, with each having its own strengths and weaknesses as well as compensating for each other's weaknesses. Thus, not only do we have the leverage to confirm whether the weather model developed in this project can estimate the probability with higher accuracy than the old model, but also we improve on the new weather models by accounting for the probability-estimation accuracy necessary to realize weather control.