Tuesday, December 31, 2013

Meteorological Snake Oil Salesman: Ultra-Long Daily Forecasts

The overwhelming majority of meteorologists are upfront about their capabilities, not promising or suggesting that they can achieve more than the state-of-the-science.    But there is a growing group of forecasting entities that are promising daily or weekly forecasts for weeks, months or a year ahead of time, even though such forecasting is not possible for both theoretical and practical reasons.

They are embarrassments to my field and are the snake-oil salesmen of weather prediction.

Before I describe the various long-term forecasting services, let me note that groups that do rigorous verification find that weather prediction skill for day to day weather fades out around 7-9 days.   Theoretical studies suggest weather prediction (that is describing what is going to happen at some specific time) may not be possible past about 2 weeks.  The reason for this has to do with the chaotic nature of highly complex systems like the atmosphere.  We can never describe the atmosphere or its processes exactly and as a result forecast errors grow over time.   Eventually the errors swamp the forecast.  Although forecasting the exact weather at some point in time may not be possible more than a few weeks ahead, it may well be possible to produce useful forecasts of long-term averages over a region. The National Weather Service is trying to do this with their Climate Forecast System (CFS).

The first group to provide detailed forecasts up to a year ahead of time is the most harmless:  the Old Farmer's Almanac and the Farmer's Almanac (these are two distinct products).  The Old Farmer's Almanac will give you sub-weekly forecasts out about a year into the future.  Here is a sample:

Their forecast for the first week in January is clearly going to be completely in error (heavy rain is not going to happen to the southern NW coastal region).  This doesn't give you much confidence for their extended range predictions! My colleague Nick Bond, Washington State climatologist did a study of the accuracy of the Old Farmer's temperature and precipitation forecasts. The bottom line:  they are no more more accurate than flipping a fair coin.  Other researchers have found the same thing.

I don't think many people take almanac forecasts seriously, so I am not too concerned about it.

But what really bothers me is that several weather firms are seriously pushing long-term daily forecasts as real products.

For example, Accuweather, the well known weather forecasting firm, started to provide 45-day forecasts this summer (see graphic). 

You can view these 45-day forecasts for free on their web site or mobile app. Here are the daily  forecasts for the first two weeks of February for Seattle!  Not very daring forecasts:  high temperatures range from 43 to 45!   Their icons for clouds also look like fish...but that is another matter.


On their web site you will not find verifications, but it turns out some folks have done it for them. Jon Nese, an instructor at the well-known meteorology department at Penn. State, had his students verify the Accuweather forecasts over an extended period.  The results for temperature are summarized below.

A good way to tell whether you have any forecast skill is to compare your forecast errors to that of climatology...the average value for that day.  Accuweather does better than climatology for the first 9 days, but after ten days THEY ARE WORSE THAN CLIMATOLOGY!  In other words, their customers would be better off just using the climatological temperatures rather than trust the Accuweather forecast.   Any clams that Accuweather has long-term skill is uber snake oil salesmanship.   And a profound disservice to my profession.  Accuweather has a proud history as a national forecast firm:  why throw away their reputation in this way?


But as bad as Accuweather and the Old Farmer's Almanac are, they are not in the same league as a company that provides one-year DAILY forecasts: WeatherTrends360.com (see below)


Their products are inconsistent with our knowledge of how forecast errors grown in the atmosphere, but that does not stop them from pushing it on consumers and businesses.  To show you how problematic this forecast service is, let's check their predictions for the next month (see below).  Wow...on January 9th the high will be 22 and the low about 15F, with the cold wave starting next Tuesday.  I have checked reliable forecasting systems; nothing suggests this kind of extreme cold for Seattle. There will be cold air to the east of the coastal NW:  I suspect they are using low-resolution model output even for the short-term.


How about for the warm time of the year, late next July and August?  See below.  Only ONE DAY will get to 70F, making it perhaps the coldest August in Seattle history!  Pretty sad.


Folks such as WeatherTrends360.com and the rogue group at Accuweather are not only undermining their reputations but are diminishing the credibility of the weather forecasting profession.

Is providing forecasts you know to be inaccurate any different than selling magical elixirs that you know can't provide the promised cures?  I will let you decide.

17 comments:

Puffin said...

Ultra long-range forecasters do not publish confidence levels for a reason.

codetalker said...

Cliff you're just jealous of their chicken. The one they use to make the predictions, you know, the one that predicts warm weather if it pecks the corn or cold weather if it pecks the barley?

Cliff Mass said...

Codetalker,
I just want you to know I have a very nice caterpillar whose coat thickness is an excellent measure of the general nature of the upcoming winter! Far better than any pecking chicken....cliff

Wilbur JK said...

My meterology prof told us that the most acurate forecast is 30 seconds into the future then the accuracy starts to degrade.

I also remember this , (Cliff can you explain)
"When the flow at the 500mb level is from the south there will be a net divergence at that level, causing a "low" if you will. the effect is that lows traveling from the south to the north get more intense."

I cannot visualize why there would be the divergence at 500mb
"

Westside guy said...

I believe there was an episode of Green Acres where Arnold the Pig achieved fame for his uncannily accurate weather forecasts... including snow in the summer!

So obviously these long-term forecasters have simply managed to procure a magical pig.

tz said...

I've found 3 days is about the limit in certain places (I'm from southeast MI, but in the Seattle area right now), at least if I'm planning on doing something outdoors in the sun.

I've noticed on all the sites that 3+ days out, the icon for the weather will often change (sunny to cloudy, sometimes to rain).

It is less likely if there are to be several consecutive days of some weather, but worse if you are near an edge of where something might go to one side or another.

"50% chance of rain" is flipping a coin, but that means that for any particular place at a particular time, it might be dry, drizzly, or drenching.

"partly cloudy" can mean a checkerboard, or one half of the sky entirely covered.

They might know that 7 days from now the boundary between rain and shine will be somewhere over 10 degrees of latitude (or sweep in, so might arrive at 8am or 8pm), but if you are in the middle of the area, you can't plan to be outdoors.

richard583 said...

One thing that's not really focused on much where considering the idea of greater than 7 days out forecasts—or more specifically, this practice looked at together along with also, whatever more general prognostication that might be seen perhaps more climatological in nature—is what the larger patterning of the Jet might be expected to do more just and only, farther ranging. One, more realistic group, and approach to the idea of longer ranging forecasts that you have you haven't mentioned, professor—if more general, certainly—is the "US Farm Report", and their efforts where working to venture out past 7 days: more in earnest, and for both obvious and practical reasons. …

Jack Bloss said...

Thank you Cliff! I've been presonally trying to advertise this info to people for the past 6 months. I hate it when people will just say that two weeks from now it will snow based off accuweather. Whats your view on Weather Channel for outlying locations that aren't in a city area?

Rod said...

Indeed, Cliff.

I pay as much attention to long term weather forecasts as I do to energy analysts predicting the price of a barrel of oil two weeks out.

You meteorologists are getting very good, though....

Bob Triggs said...

In my experience, out on the Olympic Peninsula, hardest time of year to forecast the weather is from October through March. I work on the rivers on the coast and small changes in the weather can have big impacts on the rivers. Riddling the nuances of marine weather, mountain weather, river flows etc., is a daily task. I can only feel confident within a 72 hour window most of the winter. I have come to rely heavily on the U.W. programs, and thank God for the new coastal radar at Langley Hill . What a difference that has made. Cliff, I have learned so much from you. Thanks for all of your good work. Wishing you and yours a very Happy New Year!

Unknown said...

Joel Myers used to be my forecasting teacher in the early 70s when I was a Meteorology undergrad at Penn State. As I recall, he told us that his company Accuweather started in the business of forecasting the next night's low temperatures for New England ski resorts so the resorts could decide ahead of time whether or not to prep the snow-making equipment for that night. It sounded like an interesting consulting business. More recently, I saw these long range forecasts coming out of Accuweather which made absolutely no sense to me (mostly because, as you point out, their accuracy degrades a few days out). It seems contrary to his rigorous teaching at the time. Maybe he has nothing to do with the company now, but I can't imagine he -- or any of the forecasting professors with whom I studied -- would have endorsed something that has so little validity.

Unknown said...

I've noticed that even the regular forecasts produced by weather.com and weather underground if you have the BestForecast data active seem to be in error frequently.

About 3 years ago, weather.com forecast a -15 deg F low for Duvall, WA on a day in January. Also forecast heavy snow. Interestingly enough, the day was about 45 deg F and partly sunny. Low temperature was around 38.

Randy Kilmer said...

This is why I I trust Cliff and NOAA weather. I am not a meteorologist, just smart enough to recognize the snake oil.

GaryP. said...

A friend of mine who went to an accredited Meteorology school told me that the class assignment was to make a better prediction than "Tomorrows weather will be the same as todays.". Turns out that's pretty hard as in general things happen over a 3 day period, so the base statement is true 2/3rd's of the time.

So I've used that as my basis and it works pretty well, along with "look out the window." added to "look at the radar map.".

For my one month forecast, "pet the cat"... longer hair means rain is coming....

The Helpdesk said...

Unrelated, but a questions for you. When and why did the weather channel name name the latest storm across the east coast? Is this a twitter phenomenon?

David Jones said...

Hey Cliff...your posting is timely. I recently posted to my blog -

http://meteorologistdavidjones.wordpress.com/

- an essay I wrote a few years ago about seasonal forecasts. The complete essay is too long for a single comment box...so I made two comments!!

The Cold, Dry and Bitter Truth about Seasonal Forecasts

“Some suffer from an acute expert problem, producing cosmetic but fake knowledge, particularly in narrative disciplines…” Nassim Taleb, The Black Swan.

Recent headlines scream ‘Environment Canada Blows Winter Forecast’. The next day ‘Environment Canada Forecasts Warm Spring Ahead…’ The following day it’s ‘Groundhog Prophecies Mixed on spring’s Arrival’. The media seem stuck in a loop, rushing headlong every three months to create the seasonal weather story, often overselling it as Weather You Should Fear Today! In the rapid-fire nothing-is-too-inconsequential-to-be-made-consequential twitter-verse, the next weather horror is never far away. Once established, the hair-brained scare is impossible to unseat. In 2010 it was ‘The Worst Winter in 50 Years’. In 2011 it was ‘the Coldest Winter in 20 Years’. The oversell nicely sets up the inevitable end-of-season follow-up story about the ‘blowing’ of it. It’s really much ado about nothing but enough to make a meteorologist moan.

In my 28-year career, I’ve been asked for the seasonal ‘outlook’ more than any other forecast. The question assumes an answer is possible, that such a forecast exists, and that a meteorologist can provide it. Three cold, dry and bitter truths are: a useful answer is impossible, no such forecast exists and, you’re asking the wrong expert anyway...{truncated}

potvinj said...

I have been watching the Weather Channel (WC) blow their 2 week forecast during the last hurricane season (in the Atlantic Basin, in particular) and during the current (meteorological) winter. Thankfully WC does air its mea-culpa on those. Generally WC points to a bad call w/r to short term and mid-term movements of the blocking highs in the Atlantic and Pacific and/or of the MJO.

Is this THE problem with long-term forecasting? i.e. that we cant reliably forecast the tracks of those large scale structures beyond two weeks?

Any good references on blocking high movement simulation?