Friday, February 29, 2008

French Heart Attack Conclusions are Based on One Data Point; Stroke Admissions Increased After Smoke-Free Bar Law; Shoddy Science Exposed

It turns out that the truth about the report released last week which purported to show a 15% decline in heart attack and stroke admissions in France due to the bar and restaurant smoking ban is not good: the entire conclusion that the French bar/restaurant smoking ban led to a 15% decline in heart attacks and stroke admissions is based on a single data point - one month worth of data following the ban.

As I reported yesterday, a press release issued last week by the European Society of Cardiology stated that a new study had found a 15% decline in heart attack and stroke admissions attributable to the smoking ban in France, which was implemented in January 2007. These results were widely reported in the media (example 1; example 2; example 3; example 4; example 5; example 6; example 7; example 8).

Here's the rest of the story:

It turns out that the ban on smoking in bars, restaurants, cafes, hotels, and casinos in France went into effect just two months ago - in January 2008 (a workplace smoking ban that excluded these venues went into effect one year earlier). The reported decline in heart attacks and strokes in France was not a decline that was observed based on a year's worth of data; instead, it was based on a single data point: the heart attack and stroke admission rate in January 2008, the first month in which the smoke-free bar and restaurant legislation was in effect.

So the first thing we find out is that the press release was very deceptive. It suggested that there were observations over a full year to document a real decline in heart attack and stroke admission rates. We find out, however, that the entire conclusion - and all of the media publicity that followed the press release - is based on a single data point!

It also turns out, when one reads the entire report, which I was finally able to find yesterday, that stroke admissions were actually slightly higher in January 2008 than in January 2007. The report concluded that there was a 12% decline in stroke admissions associated with the bar/restaurant smoking ban only because it compared the January 2008 rate with the rate in January of 2005 and 2006, which was higher. But the rate had lowered in 2007 and the report fails to make the logical comparison of the stroke rate in January 2008 with the rate one year earlier, when the bar and restaurant smoking ban was not in effect.

Interestingly, in concluding that the heart attack and stroke rate dropped by 15% after the bar and restaurant smoking ban, the report compares January 2008 with January 2007. Had the report done the same comparison for the stroke rates alone, it would have found that there was essentially no difference. To me, this calls into question the objectivity of the report's conclusions.

The Rest of the Story

I would never draw a conclusion like this based on a single data point - one month of data. The report uses as a baseline 24 months worth of data prior to the smoking ban. Then, to determine the trend in admissions after the ban, it uses 1 (ONE) month of data. That is ridiculous. If the rate subsequently increases in another month or two, the entire conclusion will be wiped out. It is far too early to draw such a conclusion.

In other words, while I was yesterday criticizing the report for concluding that the observed 15% decline in heart attacks and strokes was attributable to the smoking ban, today it is not even clear that one can conclude that there was a true decline in heart attacks and strokes (by which I mean a decline that reflects a real change rather than just random variation).

If one looks at the graph of heart attack and stroke trends, one will see that between March and April of 2007, there was a "huge" decline in the heart attack and stroke rate, much greater than the decline observed from January 2007 to January 2008. Thus, wide variations in these data exist based purely on random variation.

The report is not concluding, for example, that the heart attack/stroke rate in April 2007 had declined due to the workplace smoking ban, even though it is much lower than it was in April 2006. But such a conclusion would follow from the same logic being used to conclude that there was a 15% decline associated with the bar and restaurant smoking ban. The report does not make the former conclusion because it is apparent, after watching the trend for another month (the rate went back up in May), that the decline in admissions in April 2007 was simply random variation.

When one looks at the entire year's worth of data for 2007 compared to 2006, it is quite clear that there was no significant change in the heart attack and stroke rate. In order to determine whether the rate in 2008 is significantly lower than in 2007, one clearly needs more than just one, or even a few, months of data.

Similarly, the stroke admission rate in January 2007 was much lower than in January 2006. But the report does not conclude that the workplace smoking ban caused the stroke rate to drop. Instead, it concludes that the bar/restaurant smoking ban caused the stroke rate to drop, even though the rate in January 2008 was actually slightly higher than in January 2007.

The bottom line is that the report's conclusions are not objective, nor are they based on enough data to make them scientifically supportable. There is no journal I know of which would publish a study with this conclusion based on these data. But even though this study, in its present form, will never sustain peer review nor make it to publication, its conclusion has already been widely disseminated throughout the world and will undoubtedly play a role in influencing public policy makers' decisions. The report will undoubtedly be hailed by anti-smoking groups as evidence that smoking bans result in an immediate (and I mean immediate) decline in heart attacks.

Now you can see why I am so leery of the idea of science by press release.

The most interesting aspect of this story is whether or not anti-smoking groups will tout this study or whether they will criticize it because of its unsupportable conclusions.

My guess: they will tout the study, because despite its non-scientifically supportable conclusions, it comes down on the "right" side.

No comments: