Climategate
Sunday, November 29, 2009 8:40:22 PM
Here is a link to some parts of it. I'm sure you can find the rest of the emails and code on your own unless Google restricts searches again (yes, it's true). I think there's a torrent with all of it somewhere.
I've heard plenty of people try to dismiss this. Not sure why. In case you haven't heard the story, the IPCC (the agency that advises the UN on climate change) had thousands of emails, code and data released a week ago. It is believed that either a hacker or someone on the inside released them. A BBC reporter found some of his own correspondence within the set of released emails and confirmed that the ones he was included in were authentic, but could not say anything about the others. Apparently, there were scientists at the University of East Anglia who were responsible for much of the data modeling. CRU has since admitted that the original raw data has been dumped and deleted.
Quoted from that article:
Professor Phil Jones, the activist-scientist who maintains the data set, has cited various reasons for refusing to release the raw data. Most famously, Jones told an Australian climate scientist in 2004:
Even if WMO agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it.
This is from the head guy at the IPCC back in 2004!
Within the emails that span 13 years are indications of coercions at all levels of the peer review process. Concerted efforts to remove dissenting opinions from being published. Efforts to remove editors that allow dissenting papers to be published. Deletion of emails and data so as to avoid Freedom of Information Act requests. Refusal to publish data on which their model is based. The list goes on and on. Not to mentioned they obtained millions of dollars from taxpayers. There are plenty of other blogs and articles that deals with this, so I won't bother quoting those parts here.
Normally, I wouldn't publish anything. But we know three things with 100% certainty.
1. BBC reporter validates some of the emails that he knew about.
2. CRU admits to data dumping.
3. The predicted climate change model is bogus.
I believe this is the tip of the iceberg. What I find amazing is that no personal info was found in those emails. It's almost as if the IPCC was getting ready to respond to a FOI request by weeding out "less favourable" emails and data files. But the emails that got weeded out were "released" by someone on the inside after probably several rounds of triage.
What's even more strange is the MSM's refusal to report on this. Complete and total silence.
About the programming notes, I've heard a lot of people comment on how they thought it was hoax or how no one would keep a separate txt file to document what they do. I do this all the time and have done so with almost all of my projects. I've also worked with messy and inconsistent data files. The big ball of mud that is found here should not be all that surprising. At the same time, if there is a case to be made for not commenting your code, this would be it.
My favourite part from the linked article and from the HARRY_READ_ME.txt file is this:
17. Inserted debug statements into anomdtb.f90, discovered that a sum-of-squared variable is becoming very, very negative! Key output from the debug statements: . OpEn= 16.00, OpTotSq= 4142182.00, OpTot= 7126.00 DataA val = 93, OpTotSq= 8649.00 DataA val = 172, OpTotSq= 38233.00 DataA val = 950, OpTotSq= 940733.00 DataA val = 797, OpTotSq= 1575942.00 DataA val = 293, OpTotSq= 1661791.00 DataA val = 83, OpTotSq= 1668680.00 DataA val = 860, OpTotSq= 2408280.00 DataA val = 222, OpTotSq= 2457564.00 DataA val = 452, OpTotSq= 2661868.00 DataA val = 561, OpTotSq= 2976589.00 DataA val = 49920, OpTotSq=-1799984256.00 DataA val = 547, OpTotSq=-1799684992.00 DataA val = 672, OpTotSq=-1799233408.00 DataA val = 710, OpTotSq=-1798729344.00 DataA val = 211, OpTotSq=-1798684800.00 DataA val = 403, OpTotSq=-1798522368.00 OpEn= 16.00, OpTotSq=-1798522368.00, OpTot=56946.00 forrtl: error (75): floating point exception IOT trap (core dumped) . ..so the data value is unbfeasibly large, but why does the sum-of-squares parameter OpTotSq go negative?!! . Probable answer: the high value is pushing beyond the single- precision default for Fortran reals?
His hunch would seem to be correct.
After 2147483648 is when 32bit signed numbers turn negative (and then get smaller). He's at -1799984256 when he reaches it first in his debug statements. Only a difference of 347.5 million with respect to values in the 2 billion range.
This isn't particularly odd or unheard of. It's a bug and one needs to fix it. But here's where I laughed out loud. This made my day. The following comment follows immediately the above quoted snippet.
Value located in pre.0312031600.dtb: -400002 3513 3672 309 HAMA SYRIA 1985 2002 -999 -999 6190 842 479 3485 339 170 135 106 0 9 243 387 737 1985 887 582 93 16 17 0 0 0 0 352 221 627 1986 899 252 172 527 173 30 0 0 0 84 496 570 1987 578 349 950 191 4 0 0 0 0 343 462 929 1988 1044 769 797 399 11 903 218 0 0 163 517 1181 1989 269 62 293 3 13 0 0 0 0 101 292 342 1990 328 276 83 135 224 0 0 0 0 87 343 230 1991 1297 292 860 320 70 0 0 0 0 206 298 835 1992 712 1130 222 39 339 301 0 0 0 0 909 351 1993 726 609 452 82 672 3 0 0 0 34 183 351 1994 625 661 561 41 155 0 0 0 22 345 953 1072 1995 488-9999-9999 182-9999 0-9999 0 0 0 754-9999 1996-9999 40949920-9999 82 0-9999 0 36 414 112 312 1997-9999 339 547-9999 561-9999 0 0 54 155 265 962 1998 1148 289 672 496-9999 0 0-9999 9 21-9999 1206 1999 343 379 710 111 0 0 0-9999-9999-9999 132 285 2000 1518 399 211 354 27 0-9999 0 27 269 316 1057 2001 370-9999-9999 273 452 0-9999-9999-9999 290 356-9999 2002 871 329 403 111 233-9999 0 0-9999-9999 377 1287 (value is for March 1996) Action: value replaced with -9999 and file renamed: pre.0312031600H.dtb (to indicate I've fixed it) .dts file also renamed for consistency. anomdtb then runs fine!! Producing the usual txt files.
See March 1996? It has a value of 49920. Replace with -9999 (a special value to omit this when processing). That fixed it. Remove the data that causes the program to crash and VOILA! Problem fixed. "anomdtb then runs fine!!" Dontcha know it!
I love "if (data[ i]==crash) data[ i]=-9999;" solutions. They rock!
If one value is suspect, what does that say about the rest?
Here, the expected 1990-2003 period is MISSING - so the correlations aren't so hot! Yet the WMO codes and station names /locations are identical (or close). What the hell is supposed to happen here? Oh yeah - there is no 'supposed', I can make it up. So I have :-) If an update station matches a 'master' station by WMO code, but the data is unpalatably inconsistent, the operator is given three choices: <BEGIN QUOTE> You have failed a match despite the WMO codes matching. This must be resolved!! Please choose one: 1. Match them after all. 2. Leave the existing station alone, and discard the update. 3. Give existing station a false code, and make the update the new WMO station. Enter 1,2 or 3: <END QUOTE> You can't imagine what this has cost me - to actually allow the operator to assign false WMO codes!! But what else is there in such situations? Especially when dealing with a 'Master' database of dubious provenance (which, er, they all are and always will be).
Read the original file for the rest of this comment. It keeps going. The two points here are "I can make it up. So I have :-)" and "dealing with a 'Master' database of dubious provenance (which, er, they all are and always will be)." I didn't even talk about the NUCLEAR option. It's figurative, but yes, there's more to this comment in the file if you're interested.
It's obvious he wants to get from point A (inconstant data) to point B (the predicted model published by the IPCC). Too bad he can't adapt the A* algorithm for his needs.
One more segment of that same comment.
You see? The leading zero's been lost (presumably through writing as i7) and then a zero has been added at the trailing end. So it's a 5-digi WMO code BUT NOT THE RIGHT ONE. Aaaarrrgghhhhhh!!!!!! I think this can only be fixed in one of two ways: 1. By hand. 2. By automatic comparison with other (more reliable) databases. As usual - I'm going with 2. Hold onto your hats.
This isn't unusual either. But for something that will affect the globe's governments, people, economies and infrastructure, it's kind of nonchalant.
Here's a comment from the code itself.
; ; Plots (1 at a time) yearly maps of calibrated (PCR-infilled or not) MXD ; reconstructions ; of growing season temperatures. Uses “corrected” MXD – but shouldn’t usually ; plot past 1960 because these will be artificially adjusted to look closer to ; the real temperatures.
There's been a lot of talk about corrected figures online. But what this is talking about are proxy measurements. A proxy measurement is one you use instead of a direct measurement. From what I've heard, they used tree ring measurements as an indication of the temperature in the past (a dubious practice). After 1960, they trended downwards, so they were "corrected". Other corrections involved the moving of stations, but these were not consistent either since the raw data before and after were relatively flat. But after correction, they trended upwards. Did the recording devices become uncalibrated over time that they would record lower temperatures over time? Seems a bit ridiculous.
; Computes regressions on full, high and low pass MEAN timeseries of MXD ; anomalies against full NH temperatures. ; THIS IS FOR THE AGE-BANDED (ALL BANDS) STUFF OF HARRY’S ; ; Specify period over which to compute the regressions (stop in 1940 to avoid ; the decline ;
STOP IN 1940 TO AVOID THE DECLINE!
There is other stuff about 1960 and from the last several years that have been omitted as well. Is this real? I realize this is more of a debug log as he's trying to fix things. But the fixes are appalling. I read up on science and other programming blogs how there's nothing to see here. But this should not surprise me coming from people who debate religiously about what language is best and how the scientific community is not corrupt.
The new tactic is to call anyone who says that the models for climate change are fake as "deniers". This is similar to "truthers" and "birthers". The deniers are tin foil wearing nut cases. To me, this is more proof that the books were cooked. If it's not ok to debate the issue without being called names... if dissent is treated with such hostility, then something is up.
Here is another analysis of the code and comments. For me, the going on's both with the programmer and with the scientist is business as usual. I know personally from having this blog what kind of responses one gets with unpopular or dissenting views by those who are religiously attached to their point of view. In today's world, it's science by proclamation and declaration.
On a science blog I will not link to said that it's ridiculous to say that certain programming languages aren't well suited for certain tasks. This is in response to the following quote from the previously linked article. (More here.)
Neither fortran nor IDL are the tools I would use for text processing - perl, awk, sed (all traditional unix tools available on the platforms the code runs on as far as I can tell) are all better at this. Indeed awk is used in a few spots making one wonder why it is not used elsewhere.
This is a valid assessment. If awk was used in one place, it's curious why it wasn't used more. I've used awk, sed and perl. They are VERY good at text processing. I used to write scripts all the time in University to monitor users and CPU time on all machines in the lab so that I could leech CPU time from free machines and be able to chat with other people in the room. You can parse output from commands very easily with these tools (though I used cshell, bourne shell and korn shell (didn't know about Perl back then)).
But no, this is crazy talk.
The problem isn't that these are scientists and not programmers or that he's trying to get a consistent database or any of that. It's that everything is unreliable and the raw data has been dumped. We know that the model(s) for climate change are bunk. They were made up. This isn't the only model that's bunk. The model for radiation energy "bounced" back into space has been debunked. And even if you believe the data COULD BE correct, it doesn't matter. It simply cannot be trusted.
It goes beyond this too. The amount of power and reach that the IPCC had over editors and research grants made it so they controlled every aspect of peer review. This is not science by any stretch of the imagination.
The IPCC has helped shape the way mankind makes decisions about global climate change.
Now, one must be careful. We don't know exactly what code was actually used or if the above mentioned code was the final version or not. If it was, then it can't possibly get much worse. The whole climate change theory needs to be reset.
I'm expecting that a lot of it will be denied or covered up. The MSM is obviously keeping quiet on the issue. I'm fully expecting that the above code will be dismissed as an earlier attempt to create a consistent database and that any correction was simply done with that goal in mind.
What I'm waiting for is how they will explain the dumping of the raw data and the refusal to release this data for years (they never released it at all BTW). If you didn't know this yet, it's true. The climate change models are based on data you're not allowed to see because someone might use it to discredit said model. This is according to Phil Jones at the IPCC.
The other thing is that billions of dollars (aside from the millions already obtained) are riding on the climate change theory. I wonder if people who are higher up will start asking questions? I'm not optimistic, but you never know. At the very least, there is very good indication of avoidance of FOI laws and even possibly fraud.
As one person said, I'd buy what the ShamWOW guy is selling before I buy what Phil Jones is selling.
In any case, as a conspiracy buff, I'm enjoying the popcorn.
---
Other snippets from the HARRY_READ_ME.txt file.
Who's who and their emails (youtube).
CRU FOI Avoidance (First hand account and a behind the scenes look revealed by the emails)
How CRU Peer Review actually works (youtube)
An apt quote that everyone should ask these CRU scientists.
-Charles BabbageOn two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.


Unregistered user # Monday, November 30, 2009 11:07:34 AM
Unregistered user # Monday, November 30, 2009 12:46:58 PM
Vorlath # Monday, November 30, 2009 5:30:11 PM
@Dan: About climategate, the fallout from this will really be that I agree we should reduce emissions and keep our planet clean. But I simply can't disregard that the data was made up and altered whichever way. I'm also hearing that individuals have no real effect on the total impact. It's the large corporations that control the natural resources. True or not, I'm not sure. And that's the sad thing is that this is a blow not just to climate change, but to everyone who wants real answers. A lot of credibility has been lost.
About Project V, it has unfortunately stalled in recent times because of other projects. But I started looking into it again. I'm not going to bother with undo right now. It's too involved and I have the infrastructure in place to implement it later. I'm now moving on to finishing building networks and running them, but I noticed I don't have a multi-line edit box written yet. So I have to complete that first. I want it because I'm going to use a command line for some of the custom component creation. Building networks will be done directly in the interface. So that's the plan and I have started to code this up again. I'm still not 100% into the swing of things though.
Unregistered user # Tuesday, December 1, 2009 11:41:01 AM
Vorlath # Tuesday, December 1, 2009 9:38:26 PM
That's complete nonsense. You're arguing that garbage input data can produce the correct output. Even if all the data was correct, you could still not get an accurate model because the margin of error is more than their predictions.
They didn't just remove ONE data item. The entirety of the input data is unreliable. On top of that, they specifically cherry picked what data they wanted. If that didn't do the job, they would skew the data at the end upwards. If that didn't work, they would skew the data at the front downward.
Yes, and this data cannot be used for any predictions. Even old stations that used to be outside of cities can find themselves inside a city over the course of time. But in many cases, they keep this data because it suits their model.
UNACCEPTABLE!
#2 is asinine. You can't have science if you hide the facts. #3 is even more absurd. This is science funded by taxpayer money.
That's not your decision to make. And it should be made available to anyone that wants to verify the models. I have to respectfully disagree that science by attrition is valid science.
Everything you say is unacceptable for proper behaviour in the scientific community. If you hide your data, why should I believe anything you have to say? This isn't American Idol. Science isn't done on the basis of how many scientists vote one way. Well, actually it is, but it shouldn't be. As Einstein said when the book 100 Authors Against Einstein disagreed with him: "If I were wrong, then one would have been enough!"
I can read any paper out there and learn the material if I don't already know it and see if anything is wrong. Note that I don't need to understand all details. But if I start seeing things that I know to be wrong, like faked data, coercing editors to not publish opposing views, deleting emails and data, etc., it's pretty obvious this is a scam. It's like Dijkstra said about debugging: "Testing shows the presence, not the absence of bugs." IOW, you don't need to know all the details to find flaws.
In this case though, I understand it all too well. I've seen it with my own eyes and this is the biggest scientific hoax the world has ever seen.
To say that you "see no malfeasance here" is not only dishonest, but harmful to having open and honest scientific discussion.
Vorlath # Tuesday, December 1, 2009 10:38:39 PM
Look at the graphs. These have already been discredited. The "hockey stick" figure is a hoax. And read through the article. They still defend everything, yet there's going to be an independent investigation? They're continuing the same tactics as before in stating their conclusions before obtaining the facts.
Look at where the data came from.
In red, Jones et al. (no longer credible source and never was.)
In blue, Mann et al. (creator of the hockey stick figure which has been discredited time and again).
I haven't had time to check Briffa yet. (edit: I have now and he's responsible for the Yamal fiasco. I can't believe how crooked this gets.)
From Phil Jones.
Same excuse as in the emails. Except he won't say WHICH data he used for his model. That's why the FOI requests were made in the first place.
Looks like it's business as usual. This is not the hoax you are looking for. Nothing to see here. Move along.
Vorlath # Tuesday, December 1, 2009 10:47:55 PM
They cherry picked what trees they used. From what I've seen, they used a handful of trees that gave results they wanted and ditched the rest. Tree rings as a proxy is dubious at best. They don't know why there is a divergance and cannot demonstrate that tree rings are an adequate proxy.
It's shoddy science like this that really hurts the scientific community. And this is just the beginning. The shifting of data downwards at the front and shifting it upwards at the end is still not explained.
(edit: I'll try to find the email where CRU admits that tree rings as a proxy is completely FUBAR. The CRU itself doesn't believe in it. If that's not faking the data, I don't know what is.)
Vorlath # Wednesday, December 2, 2009 12:14:00 AM
Email ID: 839858862.txt
Email ID: 845217169.txt
I've learned that Briffa has been doing research into tree rings as seen in the same message.
Search for the holy grail! Will they cheer when they find it? If you look long enough, you'll find it.
(edit: For those that want to know... If 1032 was the coldest in a thousand years, this would deny the existence of the Medieval Warm Period. The MWP is troublesome for climate researchers as it shows that warming can exist without human intervention and is not linked to CO2 levels. The MWP did exist as it was documented by people living at the time and it was also when the Vikings settled on Greenland.)
Read the whole thing. It's far from a "done deal".
(Emphasis mine throughout).
So will you believe the scientists themselves?
There are TONS of other emails like this. There was one that said it point blank that tree ring analysis was nothing more than fantasy.
Yeah, as long as it's not after 1960.
For those interested in the MWP debate.
This is still from the tree ring data.
This paragraph scares me.
WHAT. THE. FUCK?!
That's just pure insanity.
(edit: Readers should be aware that the above results from the Yamal tree rings have been debunked. Briffa is apparently still trying to defend the Yamal fiasco.)
Email ID: 981859677.txt
From John Daly specifying problems with tree rings.
Email ID: 990718382.txt
With respect to an ABC interview.
Email ID: 990718506.txt
About the hockey stick. The following quote is from Competitive Enterprise Institute sent from Mann (creator of the debunked hockey stick model) to Kevin Trenberth.
Kevin Trenberth responds.
And yet, the hockey stick model is now known to be a pure fabrication.
This entire exchange needs to be read. It really is a global conspiracy. This one is from someone at the University I went to. You'll note the hostility to any opposing views.
Get it?
So there.
There's just too much. This is a complete and utter joke.
I will end it with Phil Jones.
Email ID: 1120593115.txt
Read the rest of the email to find out what is rubbish.
At least right here, "your House subcommittees wanting Mike Mann and others and IPCC to respond on how they produced their reconstructions and how IPCC produced their report".
RUBBISH!
If ANYONE dares to say that this isn't a hoax, they are just deluding themselves or are flat out lying. There's MORE of this stuff. LOTS MORE. In fact, it seems to never end.
Unregistered user # Friday, December 11, 2009 9:28:57 PM