Thursday, August 13, 2020

AWDE Services Gathers Probabilistic TCF Feedback

The Aviation Weather Demonstration and Evaluation (AWDE) Services team provided support to the
Aviation Weather Testbed (AWT) during the two-week, all virtual, 2020 Summer Experiment. AWDE
conducted interviews with twenty-one participants to collect feedback concerning the capabilities
incorporated into the probabilistic TCF product. Participants included CWSU meteorologists, PERTI team
members, one MIC, and one NAM. All participants attended 30-60 minute virtual interviews and
provided valuable feedback concerning the operational suitability and usability of the probabilistic TCF

AWDE team members asked participants questions concerning the overall usability, how the products
capabilities would add value in an operational environment, and what would improve the product’s
suitability and usability. All participants gave feedback stating the product is a good first-look graphic
that provides an overview of convective regions nationwide for planning. This graphic would be used
alongside models to compare convective focus areas for planning. Additional overlays such as ARTCC
boundaries, jet routes, and airport locations would be useful. While the product provides a 24 hour
convective forecast, most participants would benefit from smaller time increments, such as three or six
hours, and be able to focus in on certain regions/areas. The ability to overlay MRMS and the final TCF
polygons as verification provided more confidence in using the probabilistic TCF product to identify
convective weather areas.

Overall, the consensus among participants is that the product would be used as an initial tool to identify
convective areas to focus on before analyzing more regionalized areas for day one planning.

Tuesday, August 11, 2020

Verification and Reliability of Probabilistic Information

When evaluating a product to determine it's usefulness, it is important to know how well it is performing. Verifying probabilistic information is a tricky problem. It's much easier to validate a deterministic product by looking at observations of what occurred. For this evaluation, we're offering two methods for users to utilize to look at how well the probabilistic information performed. 

The first option allows for overlaying “observed” TCF polygons on past runs. These polygons are generated from MRMS reflectivity and echo tops over the entire 24 hour period. This verification product is available on the AWC main website via the TCF page. To get a sense of what these polygons look like with the corresponding MRMS reflectivity, check out the graphic below. 

An example of the MRMS polygon verification product, showing how the polygons are generated around MRMS reflectivity that meets the TCF criteria.

The MRMS option is intended to serve as a subjective verification, of sorts, comparing how the guidance performed against observations. The polygons are color coded by valid times that occur during the 24hr period represented by the guidance probability contours. An example of what this looks like can be seen in the graphic below, with the MRMS generated polygons overlaid on top of a past HRRRe 24hr graphic.

An example of the TCF probabilistic HRRRe 24hr graphic overlaid with "observed" MRMS polygons from the same time period. 

Another way users can get a sense of how well the probabilistic product performed is by overlaying the final 4-hr TCF polygons generated by the AWC forecaster throughout the 24hr period. The Final 4-hr TCF option is intended to allow for comparison between the performance of the automated probabilities and the forecaster generated polygons, allowing users to see if there are areas that were captured by the forecasters, but not the probabilities, or vice versa. An example of this can be seen below.

An example of the TCF probabilistic HREF 24hr graphic overlaid with the Final 4-hr TCF polygons color coded by valid time.

In addition to overlaying observed data onto the probabilistic information, users can also look at the reliability of each guidance product (HREF & HRRRe) by model run. The reliability statistics are computed and plotted for each guidance run using TCF polygons generated from MRMS reflectivity and echo tops as observations. These reliability diagrams give an assessment of the performance of probabilistic guidance in terms of its reliability, or whether the probabilities tend to over-predict or under-predict the occurrence of a phenomenon. 

In the above example, both models performed fairly well for this run, staying close to the center diagonal line which would indicate a perfect forecast. In this case, the HREF slightly over-predicted the probability values while the HRRRE slightly under-predicted the probability of occurrence. While this capability is currently only available by run, AWC plans to combine full statistics to get a better sense of overall performance during the experiment and beyond. There has also been early feedback indicating users would also like to see this information broken down by region.

While probabilistic information continues to be a challenge to fully verify, users seem to appreciate the various methods AWC has explored to give them a sense of model guidance performance and reliability. Other potential means to discern this information has been a topic of discussion among participants and stakeholders throughout the experiment thus far.


Monday, August 10, 2020

Onto Week 2!

 We’re starting the second week of the 2020 Aviation Weather Testbed Summer Experiment today (8/10/2020). Just like last week, this week’s activities will be conducted 100% virtually. Nearly all of the AWT and FAA AWDE staff members are working from home, and most of our participants are either working from home or are considered essential employees who must report to their worksites. So far, we’ve noticed a few unexpected benefits as well as a few unexpected challenges with this year’s virtual format.

One of the unexpected benefits has been more flexibility for our participants. They’re able to evaluate an experimental aviation product without physically traveling to Kansas City, MO to spend a week in the testbed. This is possible thanks to the Aviation Weather Center's Testbed page, which can be used to stage and evaluate experimental products under password protection. This also allows our participants to utilize the product during their regular work hours, which may yield a fuller picture of how it could be incorporated into their existing workflows. 

Of course the virtual realm also introduces a few nuances that we've had to work though. Typically testbed activities would not be affected much by a tropical storm in the Atlantic Ocean. However, this year our AWDE team was directly impacted by this storm with some losing power due to the strong winds. Quick improvising allowed the interviews to continue, and we're grateful that our AWDE friends are safe.

After this evaluation concludes at the end of the week, we'll be taking a closer look at the benefits and drawbacks of a virtual format. The hope is that we can continue to utilize virtual capabilities moving forward even when we resume in-person testbed activities in the future.

Thursday, August 6, 2020

The 2020 Summer Experiment Has Gone Virtual!

Like so many other things in 2020, the Summer Experiment has gone virtual! While we sure do miss seeing and interacting with our partners and stakeholders in the testbed, we are excited to be able to keep the tradition going by continuing to evaluate and evolve products for the aviation community!

In order to facilitate an evaluation in the virtual realm, we are focusing on one product with a more targeted participant list. In keeping with one of the themes from recent experiments, we're focusing on convection in the next-day planning regime. For that reason our targeted participants include forecasters from the various Center Weather Service Units (CWSU) which are co-located within the FAA's Air Route Traffic Control Centers (ARTCC). These forecasters often collaborate with Aviation Weather Center (AWC) forecasters on the Traffic Flow Management Convective Forecast (TCF) creation and have a breadth of knowledge on how convection impacts planning in the National Airspace System (NAS). 

Map of CWSU coverage areas over the contiguous United States.

While we have looked at the idea of impact graphics for utilization in longer term planning for the NAS, this time we are bringing in probabilistic information to address uncertainty and start to better understand how this information can best be used in the planning environment.  For this evaluation, we are utilizing convective allowing ensemble guidance to diagnose the probability of convection exceeding TCF criteria. 

An example of the TCF Probability graphic highlighting the probability of sparse coverage of convection
An example of the TCF Probability graphic highlighting the probability of sparse coverage of convection

Guidance imagery, depicting the probability of exceeding sparse and medium coverage TCF thresholds, will be automatically generated from two different model sources twice per day, in the early morning and early afternoon. Evaluation of the guidance by participants will help us meet the following goals

  • Assess the usefulness of 24 hour summary information of TCF focused threats
  • Identify optimal ways to provide, interpret, and communicate probabilistic convective information
  • Assess the skillfulness of convection allowing ensemble systems for aviation focused convective forecasts
To help facilitate gathering feedback from our participants, we are once again partnering with the FAA's Aviation Weather Demonstration and Evaluation (AWDE) Services group.  Throughout the week they will help us assess the usability and utility of this product from the user's perspective via one-on-one interviews and a questionnaire. We will also have two large group discussions during which additional stakeholders will be invited to provide their perspective and input. 

Stay tuned for early results and key takeaways from the experiment!

Friday, August 23, 2019

Evaluating Extended Range Convection Products

AWT participants had the opportunity to look at new extended range convective forecast products this week with the goal of exploring various ways of depicting hazard information to support strategic planning for the next day and beyond. Evaluations included examining both automated and user generated forecast products.

Participants generated convective coverage forecasts over 6 hr windows (09-15z, 15-21z, 21-03z, 03-09z) during the Day 2 period. These forecasts depict areas of high confidence in convective coverage areas in the same manner as current standard TCF forecasts, communicating threat areas for strategic planning to focus on for the Day 2 period. Participants have provided positive feedback for these forecast products, indicating that they have potential to provide useful information for planning at extended ranges. During the week, emphasis has been placed on the importance of consistent forecast messaging between a Day 2 and a Day 1 product.

Participant generated convective coverage forecast graphic

In terms of automated guidance, participants evaluated two automated products currently produced at the Aviation Weather Center (AWC) for extended range convection. Comparisons were made between the Extended Convective Forecast Product (ECFP), which shows convective probabilities over a 6 hour window, and the Extended Traffic Flow Management Convective Forecast (eTCF), which displays a convective coverage forecast at 2 hour snapshots. Participants have found that the eTCF provided more useful guidance in the Day 2 period compared to the ECFP, as the inclusion of convection allowing model (CAM) guidance in the eTCF allows for greater detail in the evolution and timing of convection.

Comparison of operational and variant eTCF forecasts (upper left, upper right, lower left) with observations (lower right)

Additionally, participants evaluated two experimental variants of the operational eTCF, which is created through a blend of models which include convection parameterization and CAM guidance. Both variants were comprised of a blend of solely CAM guidance and have performed better than the operational eTCF in both subjective and objective evaluations. The improved performance of the variants can be attributed to the ability of the CAM guidance to better forecast the evolution and timing of convective processes. Based on these results, a potential upgrade to the operational eTCF will be explored in the near future.    

Thursday, August 22, 2019

AWDE Services Gathering GA Pilot Feedback

In addition to having members of the Aviation Weather Demonstration and Evaluation (AWDE) Services team in house at the AWT providing support during the week, they were also in full experimental mode at the FAA WJHTC in Atlantic City.  They conducted user evaluations, with general aviation pilot participants, to provide feedback concerning the new capabilities incorporated into the HEMS and GFA products.  Pilots attended the lab both in person and via Webex to provide their valuable feedback.
Overall, participants gave positive feedback concerning the new capabilities such as the expanded coverage and resolution over the expanded areas, the Map Options dialog, and additional MESONET observations.  The additional capabilities and information will help inform critical decision making.

The AWT also welcomed a few pilots that participated in one-on-one interviews with AWDE personnel.  AWDE staff asked them questions about the usability of the tools, how the new capabilities added value and what changes or requests they had for the tools in general.  Overall the pilots were pleased with the current products and proposed upgrades, but expressed the need for more user-friendly capabilities.  This includes making the products easier to navigate on a mobile device as well as the addition of contextual help on the tools.  Having capabilities and weather information explained in a way that users can clearly understand is extremely important as AWC continues to evolve their products to serve the greater aviation community.

Wednesday, August 21, 2019

Evaluating 3D Cloud Guidance

One of the focus areas for this year's experiment is evaluating three-dimensional cloud model guidance.  Participants are assessing current post-processing algorithms of cloud layers and the added benefits of experimental visualization capabilities.  In order to complete the evaluation and carefully assess the different guidance products available, the participants have been tasked with creating a detailed text forecast for five specific hours: 15Z, 18Z, 0Z, 9Z, and 12Z the next day.

In order to get a better visualization of the 3D cloud information, AWT developed a web interface that allows for better interrogation of the data.  The web interface displays the post-processed gridded output from various model guidance options including three cloud bases and corresponding sky coverage for each forecast hour from the the latest several model runs, as well as the model derived ceiling and visibility.  In addition to the single layer look at the information, the tool allows for creating point-and-click TAFs as well as cross-sections of the cloud layer information along a path.

Along with the descriptive cloud forecasts, the participants will be answering questions about the performance of the various guidance products, any notable strengths or weaknesses, and any biases identified as well.  The various guidance products available include experimental versions of the HRRR and RAP models, the NAM and NAM Nest, and the HIRES ARW, MEM2, and the NMMB.  The experimental 3D RTMA is also available for use as a cloud layer analysis for verification purposes. 

After day one, it was clear that the cross-section ability has several advantages and could be useful for multiple users. While it may not be the primary tool used to forecast ceilings, they could prove beneficial to bolster confidence in a cloud layer occurring or in seeing a thin low cloud deck or fog.  Forecasters may not have the time to look through multiple cross sections for multiple forecast hours, however they could use it to confirm a secondary layer or get a better idea of what the model is trying to tell them in terms of the cloud development.  However, pilots may find the tool extremely useful for planning along a flight path, especially if there could be a time element added in which conditions match the time in which they expect to be flying through that region.

In terms of model performance thus far, participants have been impressed with the HRRRx cloud placement in the vertical more accurately hitting the cloud bases and coverage.  This verified well on Day 1 with the stratus off the coast of California and cloud placement in the north east associated with late afternoon convection.  Overall, so far it seems the guidance tends to agree with cloud occurrence and conditions, but differ on timing and placement.

The experimental HRRR cloud base primary indicating low ceiling moving into the Sacramento valley via the delta breeze. 

Evaluations will continue through the week with specific areas of focus being determined each morning based on impacts for that day and the next.