Thursday, September 15, 2022

A Look at Developing Outlooks for General Aviation

One of the major themes, or desks, of this year's experiment is looking at the potential for creating Outlook Graphics for Days 1, 2, and 3 for AWC's General Aviation (GA) customers. Current AWC operational products do not go out beyond Day1, leaving an information gap (Fig.1) for those that are looking to plan for flights in the coming days. AWC has continuously received feedback from their pilot partners that they would like to have an easy to interpret, quick glance graphic that gives them an idea of what hazards to expect in the coming days.

Fig1. Current aviation weather forecast gap at the Aviation Weather Center for aviation hazards.

Experiment participants were tasked with developing Outlook Graphics that would be valid 12z-12z the following day and depict prolonged, impactful hazards which could include turbulence, IFR, icing, precipitation, smoke, and thunderstorms. The design would be similar to the current National Forecast Charts (Fig2) developed by the Weather Prediction Center, and maintain consistency with existing NWS products (ie. WPC, NHC, NDFD).  Participants used an interactive drawing GUI on the Testbed Website that allowed them to overlay fronts, modify the background map, and draw various polygons for each forecasted hazard.

Fig2. An example of the Weather Prediction Center's National Forecast Chart for Day1.

While ultimately user input will be critical for determining the utility of the Outlook graphics, participants were asked to fucus on things like the design, what guidance to use, what hazards were most impactful and should be included, and the overall workload that would be associated with creating such a graphic. It was clear from day one of the experiment that everyone brought different perspectives and ideas to the table, leading to an active and fun desk throughout the week.

Fig3. An example of one of the Outlook Graphics created during the experiment.

Throughout the week all participants agreed that determining the severity and impact of the graphics for the GA community will be the top priority before moving forward with operationalizing the product. Participants also indicated that a more detailed summary of the impacts with location and timing associated with them would be beneficial for Day1 for both GA fliers and fellow meteorologists. Other ideas that were brought up were the potential for multiple outlook 'tabs' for the various user types (GA, Low Altitude (LA), and National Air Space (NAS) planning), or even breaking up Day 1 into two separate Outlook graphics (morning and afternoon) to account for hazards that are only impactful during those periods.

In the end, the AWT staff recieved an abundance of perspective and ideas for moving the Outlook Graphic further toward operations. The next step will be to get the graphic in front of users to really assess the intuitiveness and utility of the product for its intended users.

Wednesday, September 14, 2022

AWT Summer Experiment Beta Website

Along with the evolution of the Traffic Flow Management Convective Forecast and creation of new prototype multi-day outlook graphics, the participants are evaluating the Aviation Weather Center's new experimental website, AWC developers are in the testbed this week to help answer questions, and learn more about how the website is used in operations. 

The AWC Testbed page has been updated to the new look and feel of the beta website and participants are testing new products and ideas using the website this week! Prototype outlook graphics and convection forecasts are drawn on the website and saved as part of the evaluations.

Participants using the new testbed page to create prototype multi-day outlooks

Welcome to the AWT 2022 Summer Experiment!

The 2022 AWT Summer Experiment kicked off on Tuesday with a full house of collaborators and stakeholders from multiple entities of the aviation weather enterprise. Participants include developers from NOAA’s Earth System Research Lab (ESRL), and NCEP’s Environmental Modeling Center (EMC); meteorologists from AWC, Center Weather Service Units (CWSUs) and Weather Forecast Offices (WFOs) from across the country, and Southwest Airlines; as well as researchers from the Weather Prediction Center (WPC) and the National Severe Storms Laboratory (NSSL).

After a few years of primarily virtual testbed activities, the Aviation Weather Testbed and AWC are excited to welcome participants back to Kansas City for this collaborative evaluation.

Participants discussing the Warn-on-Forecast System (WoFS)

There are two major themes this year: evolution of the Traffic Flow Management Convective Forecast and creation of new prototype multi-day outlook graphics. Participants will be utilizing experimental model guidance and tools this weeks, exploring ways to improve existing products and potentially create new ones.

Reviewing medium range guidance for the production of multi-day outlooks


Thursday, August 13, 2020

AWDE Services Gathers Probabilistic TCF Feedback

The Aviation Weather Demonstration and Evaluation (AWDE) Services team provided support to the
Aviation Weather Testbed (AWT) during the two-week, all virtual, 2020 Summer Experiment. AWDE
conducted interviews with twenty-one participants to collect feedback concerning the capabilities
incorporated into the probabilistic TCF product. Participants included CWSU meteorologists, PERTI team
members, one MIC, and one NAM. All participants attended 30-60 minute virtual interviews and
provided valuable feedback concerning the operational suitability and usability of the probabilistic TCF

AWDE team members asked participants questions concerning the overall usability, how the products
capabilities would add value in an operational environment, and what would improve the product’s
suitability and usability. All participants gave feedback stating the product is a good first-look graphic
that provides an overview of convective regions nationwide for planning. This graphic would be used
alongside models to compare convective focus areas for planning. Additional overlays such as ARTCC
boundaries, jet routes, and airport locations would be useful. While the product provides a 24 hour
convective forecast, most participants would benefit from smaller time increments, such as three or six
hours, and be able to focus in on certain regions/areas. The ability to overlay MRMS and the final TCF
polygons as verification provided more confidence in using the probabilistic TCF product to identify
convective weather areas.

Overall, the consensus among participants is that the product would be used as an initial tool to identify
convective areas to focus on before analyzing more regionalized areas for day one planning.

Tuesday, August 11, 2020

Verification and Reliability of Probabilistic Information

When evaluating a product to determine it's usefulness, it is important to know how well it is performing. Verifying probabilistic information is a tricky problem. It's much easier to validate a deterministic product by looking at observations of what occurred. For this evaluation, we're offering two methods for users to utilize to look at how well the probabilistic information performed. 

The first option allows for overlaying “observed” TCF polygons on past runs. These polygons are generated from MRMS reflectivity and echo tops over the entire 24 hour period. This verification product is available on the AWC main website via the TCF page. To get a sense of what these polygons look like with the corresponding MRMS reflectivity, check out the graphic below. 

An example of the MRMS polygon verification product, showing how the polygons are generated around MRMS reflectivity that meets the TCF criteria.

The MRMS option is intended to serve as a subjective verification, of sorts, comparing how the guidance performed against observations. The polygons are color coded by valid times that occur during the 24hr period represented by the guidance probability contours. An example of what this looks like can be seen in the graphic below, with the MRMS generated polygons overlaid on top of a past HRRRe 24hr graphic.

An example of the TCF probabilistic HRRRe 24hr graphic overlaid with "observed" MRMS polygons from the same time period. 

Another way users can get a sense of how well the probabilistic product performed is by overlaying the final 4-hr TCF polygons generated by the AWC forecaster throughout the 24hr period. The Final 4-hr TCF option is intended to allow for comparison between the performance of the automated probabilities and the forecaster generated polygons, allowing users to see if there are areas that were captured by the forecasters, but not the probabilities, or vice versa. An example of this can be seen below.

An example of the TCF probabilistic HREF 24hr graphic overlaid with the Final 4-hr TCF polygons color coded by valid time.

In addition to overlaying observed data onto the probabilistic information, users can also look at the reliability of each guidance product (HREF & HRRRe) by model run. The reliability statistics are computed and plotted for each guidance run using TCF polygons generated from MRMS reflectivity and echo tops as observations. These reliability diagrams give an assessment of the performance of probabilistic guidance in terms of its reliability, or whether the probabilities tend to over-predict or under-predict the occurrence of a phenomenon. 

In the above example, both models performed fairly well for this run, staying close to the center diagonal line which would indicate a perfect forecast. In this case, the HREF slightly over-predicted the probability values while the HRRRE slightly under-predicted the probability of occurrence. While this capability is currently only available by run, AWC plans to combine full statistics to get a better sense of overall performance during the experiment and beyond. There has also been early feedback indicating users would also like to see this information broken down by region.

While probabilistic information continues to be a challenge to fully verify, users seem to appreciate the various methods AWC has explored to give them a sense of model guidance performance and reliability. Other potential means to discern this information has been a topic of discussion among participants and stakeholders throughout the experiment thus far.


Monday, August 10, 2020

Onto Week 2!

 We’re starting the second week of the 2020 Aviation Weather Testbed Summer Experiment today (8/10/2020). Just like last week, this week’s activities will be conducted 100% virtually. Nearly all of the AWT and FAA AWDE staff members are working from home, and most of our participants are either working from home or are considered essential employees who must report to their worksites. So far, we’ve noticed a few unexpected benefits as well as a few unexpected challenges with this year’s virtual format.

One of the unexpected benefits has been more flexibility for our participants. They’re able to evaluate an experimental aviation product without physically traveling to Kansas City, MO to spend a week in the testbed. This is possible thanks to the Aviation Weather Center's Testbed page, which can be used to stage and evaluate experimental products under password protection. This also allows our participants to utilize the product during their regular work hours, which may yield a fuller picture of how it could be incorporated into their existing workflows. 

Of course the virtual realm also introduces a few nuances that we've had to work though. Typically testbed activities would not be affected much by a tropical storm in the Atlantic Ocean. However, this year our AWDE team was directly impacted by this storm with some losing power due to the strong winds. Quick improvising allowed the interviews to continue, and we're grateful that our AWDE friends are safe.

After this evaluation concludes at the end of the week, we'll be taking a closer look at the benefits and drawbacks of a virtual format. The hope is that we can continue to utilize virtual capabilities moving forward even when we resume in-person testbed activities in the future.

Thursday, August 6, 2020

The 2020 Summer Experiment Has Gone Virtual!

Like so many other things in 2020, the Summer Experiment has gone virtual! While we sure do miss seeing and interacting with our partners and stakeholders in the testbed, we are excited to be able to keep the tradition going by continuing to evaluate and evolve products for the aviation community!

In order to facilitate an evaluation in the virtual realm, we are focusing on one product with a more targeted participant list. In keeping with one of the themes from recent experiments, we're focusing on convection in the next-day planning regime. For that reason our targeted participants include forecasters from the various Center Weather Service Units (CWSU) which are co-located within the FAA's Air Route Traffic Control Centers (ARTCC). These forecasters often collaborate with Aviation Weather Center (AWC) forecasters on the Traffic Flow Management Convective Forecast (TCF) creation and have a breadth of knowledge on how convection impacts planning in the National Airspace System (NAS). 

Map of CWSU coverage areas over the contiguous United States.

While we have looked at the idea of impact graphics for utilization in longer term planning for the NAS, this time we are bringing in probabilistic information to address uncertainty and start to better understand how this information can best be used in the planning environment.  For this evaluation, we are utilizing convective allowing ensemble guidance to diagnose the probability of convection exceeding TCF criteria. 

An example of the TCF Probability graphic highlighting the probability of sparse coverage of convection
An example of the TCF Probability graphic highlighting the probability of sparse coverage of convection

Guidance imagery, depicting the probability of exceeding sparse and medium coverage TCF thresholds, will be automatically generated from two different model sources twice per day, in the early morning and early afternoon. Evaluation of the guidance by participants will help us meet the following goals

  • Assess the usefulness of 24 hour summary information of TCF focused threats
  • Identify optimal ways to provide, interpret, and communicate probabilistic convective information
  • Assess the skillfulness of convection allowing ensemble systems for aviation focused convective forecasts
To help facilitate gathering feedback from our participants, we are once again partnering with the FAA's Aviation Weather Demonstration and Evaluation (AWDE) Services group.  Throughout the week they will help us assess the usability and utility of this product from the user's perspective via one-on-one interviews and a questionnaire. We will also have two large group discussions during which additional stakeholders will be invited to provide their perspective and input. 

Stay tuned for early results and key takeaways from the experiment!