Evaluation of the QRF error-corrected K-fold and LRO Rainfall Rates

The validation of the QRF-error model is conducted with a K-fold and LRO cross-validation techniques using systematic and random error statistics by comparing its predicted error characteristics to the actual error calculated in between GR and actual GPROF V05 sensor rainfall retrievals. In this section evaluation of the two experiments is presented in an effort to understand if the QRF-error model has transferability in between regions.

Figure 5.5 presents the MRE values for the original GPROF V05 and CLIM-GPROF V05 retrievals (black bars), QRF error-corrected rainfall values of the K-fold (red circles), and the LRO cross-validation techniques (blue diamonds) for all sensors and all regions. The MRE results are

169 presented for different quantile ranges to understand the QRF-error model performance for different ranges of rainfall magnitudes. The presented rainfall quantile ranges are <Q20, Q20-Q40, Q40-Q60, Q60-Q80, Q80-Q95, and >Q95 which are calculated by taking the average of GR and QRF-error corrected rainfall values. Over all regions, MRE values were significantly lower for QRF error-corrected rainfall values than the original sensor rainfall values (Figure 5.5) for all quantile ranges with Olympic Mountain region lower quantile ranges being an exception. In general, all sensors over all regions shows very similar MRE performance, with lowest quantile ranges overestimating the GR values and underestimating the highest quantile ranges. In general, LRO cross-validation technique provides slightly lower MRE magnitudes compared to K-fold cross-validation technique. Over Olympic Mountain region for lower quantile ranges, the QRF- error model rainfall values for both cross-validation techniques provides higher MRE values compared to original GRPOF V05 MRE values, where LRO has lower MRE compared to K-fold. In general, each sensor performances are varying over all regions.

Figure 5.6 presents the CRMSE values for the original GPROF V05 and CLIM-GPROF V05 retrievals, QRF-error corrected rainfall values of the K-fold and the LRO cross-validation techniques for all sensors, regions and different quantile ranges. In general, LRO CRMSE values are the lowest compared to K-fold and original GPROF V05 values. Over North Italy, North Carolina and Canada CRMSE values are in general higher in middle quantile intervals and lower at lower and higher quantile intervals. Over Olympic Mountain region there is a decreasing CRMSE trend from lowest to highest quantile intervals. In general, QRF-error model is reducing the CRMSE values significantly for higher quantile intervals and slightly for lower quantile intervals. Similar to MRE results (Figure 5.5) each sensor has varying performance over each region, GMI in general with lower CRMSE values for all quantile intervals.

170 NMRV and NFASRV are calculated to understand the rain occurence performance of QRF- error model. Figure 5.7 presents the NMRV values for the original GPROF V05 and CLIM- GPROF V05 (black bars), the QRF error-corrected model of the K-fold (red bars), and the LRO cross-validation techniques (blue bars) for all sensors and all regions. The QRF error- corrected model significantly improved the rain occurrence for all sensors, and for both experiments it significantly reduced the NMRV values for all regions and sensors. LRO and K-fold cross- validation techniques provide similar amount of improvement for NMRV values over all regions and sensors.

Figure 5.8 presents NFASRV values for the original GPROF V05 and CLIM-GPROF V05 (black bars), the QRF error-corrected model of the K-fold (red bars), and the LRO cross-validation techniques (blue bars) for all sensors and all regions. QRF-error model slightly improves NFASRV values over North Italy and Olympic Mountains. On the other hand, there is a significant improvement of NFASRV values over North Carolina and Canada. Over North Carolina original GPROF V05 NFASRV values ranges in between 2-10, K-fold 1-3 and LRO 0.1-3. Over Canada original GPROF V05 NFASRV values ranges in between 2-5, K-fold 0.5-2 and LRO 0-1.5.

5.5. Conclusions

In this study, PMW precipitation retrieval error is modeled using a nonparametric statistical technique over complex terrain. The model is evaluated over eleven complex terrain regions (the northeastern Italian Alps, North Carolina, Olympic Mountain, the southern tip of Vancouver Island, the Rocky Mountains in Colorado, the Swiss Alps, Arizona, French Cevennes, Andes, Korea and Cyprus) by high temporal and spatial resolution X-band dual-polarization radar reference datasets. We retrieved the reference precipitation by using SCOP-ME algorithm, validated against in situ rain gauge, disdrometer, and MRR datasets. The PMW retrievals used in

171 this study comprised the GPROF V05 and CLIM-GPROF V05 algorithms for MHS, SSMIS, GMI, and AMSR2 sensors. The error model used was a nonparametric technique, tree-based QRF that we evaluated using two different cross-validation techniques: a K-fold experiment and a leave- one-region out experiment.

The study yielded the following results:

1. Both QRF-error model experiments significantly reduced systematic and random error as compared to the original GPROF V05 sensor rainfall retrieval values. In terms of systematic error, the K-fold and LRO experiment performances were equal (LRO reduces MRE values slightly more compared to K-fold), but the latter reduced random error slightly more than the former.

2. The general PMW retrieval trend of overestimating lower magnitude rainfall and underestimating higher magnitude rainfall is significantly reduced with QRF-error model for both cross-validation techniques.

3. Both QRF-error model experiments significantly improved the rainfall occurrence for all sensors and regions especially NMRV.

As mentioned, this study incorporated eleven different regions over the globe, all characterized as complex terrain. The QRF-error model LRO cross-validation technique results compared to those of the K-fold cross-validation technique indicated the transferability of the LRO error model among complex terrains; this is very important and can allow algorithm developers to integrate this error model to produce Level 3 products. Through collaboration with SPP developers, this model could be used to evaluate the impact of PMW error characteristics on selected integrated SPP products—such as NASA/Integrated Multi-satellitE Retrievals for Global Precipitation

172 Measurement (IMERG) and the NOAA/Climate Prediction Center Morphing Method (CMORPH)—in case studies over mountainous regions.

173 Table 5-1 X-band dual polarization radar properties

Ground Radar Location Period

XPOL Northeast Italian Alps 07/2014-09/2014 (21 events) NOXP Appalachian, North Carolina 05/2014-06/2014 (25 events) DOW Olympic Mountain, Washington 10/2015-02/2016 (39 events) CAX Vancouver, Canada 10/2015-02/2016 (39 events)

Table 5-2 Characteristics of satellites and sensors that provide microwave rainfall retrievals

Sensor Satellite Scan Type Microwave Frequencies (GHz) Sampling (Along-track x cross-track) SSMIS DMSP F16, F17, 18 Conical 19.35VH, 22.235V, 37.0VH, 50.3- 63.3VH, 91.65VH, 150H, 183.31H 13.2 km x 15.5 km MHS NOAA-18, NOAA-19, MetOp-A, MetOp-B

Cross track 89V, 157V, 183.31H(2),190.31V 15.88 km x variable

GMI GPM Conical 10.65VH, 18.70VH,23.80V, 36.5VH,

89.0VH, 165.6VH, 183.31V(2) 14.4 km x 8.6 km AMSR2 GCOM-W1 Conical 6.925/7.3VH, 10.65VH,

174 Figure 5-1 A schematic representation of the quantile regression forest (QRF) framework used

175 Figure 5-2 Variable importance plot, size and the color of each circle represents %IncMSE for (a) North Italy (b) North Carolina and (c) Olympic Mountain (d) Canada

176

Figure 5-3 UR of QRF error-corrected GPROFV05 and CLIM-GPROFV05 of SSMIS, MHS, GMI and AMSR2 sensors for K-fold (red bars) and LRO (blue bars) of (a) North Italy, (b) North Carolina, (c) Olympic Mountain and (d) Canada

177 Figure 5-4 EP of QRF error-corrected GPROFV05 and CLIM-GPROFV05 of SSMIS, MHS, GMI and AMSR2 sensors for K-fold (red bars) and LRO (blue bars) of (a) North Italy, (b) North Carolina, (c) Olympic Mountain and (d) Canada

178 Figure 5-5 MRE of QRF error-corrected K-fold (red circles), LRO (blue diamonds), GPROFV05 and CLIM-GPROFV05 (black bars) of SSMIS, MHS, GMI and AMSR2 rainfall rate for (a) North Italy, (b) North Carolina, (c) Olympic Mountain and (d) Canada.

179 Figure 5-6 CRMSE of QRF error-corrected K-fold (red circles), LRO (blue diamonds), GPROFV05 and CLIM-GPROFV05 (black bars) of SSMIS, MHS, GMI and AMSR2 rainfall rate for (a) North Italy, (b) North Carolina, (c) Olympic Mountain and (d) Canada.

180 Figure 5-7 NMRV of QRF error-corrected K-fold (red bars), LRO (blue bars), GPROFV05 and CLIM-GPROFV05 (black bars) of SSMIS, MHS, GMI and AMSR2 rainfall rate for (a) North Italy, (b) North Carolina, (c) Olympic Mountain and (d) Canada.

181 Figure 5-8 NFASRV of QRF error-corrected K-fold (red bars), LRO (blue bars), GPROFV05 and CLIM-GPROFV05 (black bars) of SSMIS, MHS, GMI and AMSR2 rainfall rate for (a) North Italy, (b) North Carolina, (c) Olympic Mountain and (d) Canada.

182