Guidelines Fall Short of Best Scientific Practices

Aug 6

The last two meetings of the expert committee reviewing the science for the 2020 Dietary Guidelines for Americans (DGA) have revealed many alarming problems. At a minimum, the process lacks transparency, does not use any verified methodology for reviewing the science, and excludes large numbers of rigorous studies while instead relying primarily on weak data for its conclusions.

Rejecting recommendations by the National Academy of Sciences

Remember that the USDA was urged, by the National Academies of Sciences, Engineering, and Medicine (NASEM), in two 2017 reports, to increase transparency and bring greater rigor to the DGA’s reviews of the science. However, USDA officials rejected most of NASEM’s recommendations.

The NASEM stated that
"To develop a trustworthy DGA, the process needs to be redesigned."

This has not happened (See our scorecard). Indeed, far from it.
We are therefore on track to have an untrustworthy, unreliable 2020 DGA.

Below are the major problems with the 2020 process. Now that the final public meetings of the Dietary Guidelines Advisory Committee (DGAC) have concluded, the public will receive no additional information until the committee submits its draft report, due May 11.

The 2020 DGA Process Relies on Flawed, Weak Science

Here’s what we’ve learned: Despite the NASEM recommendations and despite thousands of public comments on key issues, the DGA process is still:

excluding vast quantities of rigorous science on key topics, including on saturated fats and low-carb diets,
one-side and unbalanced in its reviews of disputed topics,
non-transparent in important ways, and
lacking any up-to-date method for reviewing the science.

Primary problems with the 2020 DGA process

At least a dozen USDA reviews are based on outdated science

By law, the DGA must reflect the “best and most current” science, yet the 2020 process does not meet that standard. At the DGAC meeting in January, committee member Katherine Dewey stated that the 13 reviews undertaken for the “B-24” population (birth-through-24-months) had looked at the scientific literature systematically only through 2016. For studies conducted after 2016, Dewey said the committee had done “an informal search to identify new evidence that has emerged since 2016” but “did not locate any studies that would have changed [their] conclusions.” Dewey did seem concerned, however, that these 13 reviews had perhaps missed some of the science from the past few years. She said,

“We would like to ask the public to please submit public comments if you know of any articles published since 2016 that meet the inclusion criteria and would also significantly affect these conclusions…we do appreciate any comment that the public would like to provide.”

It is, of course, completely unscientific to rely upon random submissions from the public for studies. Thus, the DGA’s B-24 reviews are self-evidently already out of date and violating the law before even being published.

Committee Relying on 2015 Reviews Deemed “Unsystematic” by the National Academies of Sciences, Engineering, and Medicine

For a number of reviews, the Committee used, as its starting point, the 2015 DGA reviews—even though NASEM concluded that the 2015 work was “non-systematic” and therefore not reliable. The 2020 reviews using this unreliable work from 2015 include, at a minimum, the reviews on saturated fats and those on the Dietary Patterns. Given the basic flaw in the foundational evidence, none of these 2020 reviews can be considered trustworthy.

Committee Issues “Strong” Recommendations Based on Weak Science

In at least one instance, the DGAC issued a “strong” recommendation based exclusively on a weak type of science, called epidemiology. The question was the following:

“What is the relationship between dietary patterns consumed and all-cause mortality?”

The committee considered the evidence to be “strong” that the USDA dietary patterns (Mediterranean, “US-Style,” and Vegetarian) could reduce all-cause mortality, despite the fact that not a single experiment (trial) was cited to support this claim. Instead, the committee cited exclusively observational—or epidemiological—studies, which can only show associations. Epidemiological data are useful for generating hypotheses but in the field of nutrition, have never reliably been able to ‘prove’ causality. The leap to assuming causation can only be made very rarely, when certain standards, called the “Bradford Hill Criteria” are satisfied[1]. The number one criteria is the size of the effect, or “strength of association.” In fact, Dr. Hill stated that this criteria was “First upon my list,” since it was the most important.

When the strength of association is large, such as the 20-30-times greater death rate from lung cancer seen among heavy smokers compared to never-smokers, causality can be considered, provided the rest of the Bradford Hill criteria are also satisfied. However, in nutritional epidemiology, this effect size rarely ever exceeds 2. Such a small number cannot be considered reliable (due to lack of precision from food-frequency questionnaires and residual confounding, among other things).

The DGAC committee, in discussing this weak evidence, said they found it convincing, because it was so “consistent.” However, “consistency” is Bradford Hill’s second criteria. Without strong effects, consistency is not enough. Indeed, this type of consistency could easily reflect bias in the field—as so many governments worldwide have followed the U.S. lead in adopting the same type of diet, emphasizing fruits, vegetables, whole grains, lean meat, low-fat dairy, nuts and seeds; And researchers relying on government grants have every incentive of finding in favor of the diet promoted by their funders.

Moreover, committee members noted numerous problems regarding inconsistencies among the 152 epidemiological studies they were lumping together, including some four-to-fifteen different methods for analyzing the data (0:22:58 – 0:25:59) and “different definitions of food and beverages” among the studies. Analyses have found such dramatic heterogeneity among definitions of a dietary pattern ultimately mean that such studies cannot be combined with “any degree of reliability.”

Findings from nutritional epidemiology have a track record of being incorrect. When properly tested in clinical trials, they are confirmed only 0-20% of the time. These are of course very low odds on which to bet the public health.

USDA not following any recognized, verified methodology for reviewing the science

Perhaps the most alarming part of the USDA’s treatment of the evidence is that it makes no attempt to prioritize data of different strengths. Every recognized methodology for reviewing scientific data prioritizes more rigorous types of evidence (clinical trials) over weaker data. The failure for the USDA to follow this kind of internationally recognized practice is why NASEM repeatedly recommended that USDA upgrade its process to one of the “state of the art” methodologies. USDA rejected NASEM’s recommendations and has continued with its unverified, unrecognized method of reviewing the science. The only possible outcome, as more than one expert has commented, is that the outcomes of the USDA reviews will be unreliable and untrustworthy.

In sum, it is a serious problem that the USDA has no methodology to distinguish between high- and low-quality evidence. “This distinction between high- and low-quality evidence lies at the core of any rigorous evaluation of science,” wrote Gordon Guyatt, in a public comment to USDA. Guyatt is a co-founder of the GRADE methodology, and Distinguished Professor in the Department of Health Research Methods at McMaster University.

Another methodology expert, Bradley Johnston, wrote in a public comment to USDA, that the agency’s “proposed methodology…deviate[s] significantly from basic scientific precepts in a number of important ways. Taken together, these deviations…from international standards for systematic review methodology will result in a non-systematic approach that would seriously undermine the reliability of these reviews.” [Emphasis added]

No opportunity for public comment on reviews of low-carb studies

At the final public meeting on March 12-13, the committee said that it has not completed its reviews on low-carb diets[2]. This means that there will be no opportunity for the public to comment on the results of these scientific reviews when they come out.

Will the DGAC complete its reviews on low-carb and/or vegan diets at all?

Barbara Schneeman, DGAC committee chair, talking about low-carb diets suggested that the Dietary Guidelines was not the right place to discuss low-carb diets at all. (0:54:07) She says, “…high/low- carbohydrate, keto, we certainly have seen that a lot of comments have addressed those types of diets, and oftentimes, they are being presented in the context of treatment for chronic disease, such as type 2 diabetes, or as a part of a weight loss program…” but apparently, Schneeman does not see a place for these diets in the current DGA reviews. She asked, “If it’s not the Dietary Guidelines, where else can it be dealt with?” Where, indeed, if not the nation’s nutrition policy in charge of reviewing all studies that might have a bearing on the public health.

It’s also worth noting that the committee seems to have dropped the question of studying vegan diets altogether, despite this diet being on the initial list of questions mandated for review.

Lack of Transparency

The NASEM also strongly recommended that members of the committee disclose their conflicts of interest. Yet the USDA rejected this advice as well. On the 2020 committee is a Medical Director for Nestle, the former science director of Merck Pharmaceuticals, and others with multiple other serious conflicts of interest, but none of this information has been disclosed to the public. See The Nutrition Coalition’s preliminary coverage of these conflicts here and here.

Committee’s Review on Saturated Fats Ignores Most Recent, Most Rigorous Science

The DGAC has ignored a massive body of science showing that the longstanding caps on saturated fats are not supported by the science. The Nutrition Coalition has a separate post on saturated fats here.

Conclusion

In sum, there are numerous problems with the 2020 DGA process that ultimately reflect a failure to respond to the National Academies’ recommendations. In the absence of dramatic changes, the next set of guidelines are unfortunately on track to be unreliable and untrustworthy.

The question now is not if we can encourage better science in the DGA, since thousands of public comments and even the NASEM—the top scientific body in the land—have proven unable to promote reform. The process is clearly resistant to change, whether due to the influence of Big Food, Big Pharma, or simply the tendency to adhere to and replicate the status quo. For those who recognize reform is necessary, what is the next step?

One has to consider if we should continue to have Dietary Guidelines if they are so opposed to good data, so bound to weak, unreliable science, and so self-evidently unable to curb the epidemics of diet-related diseases in America.

[1] Hill AB. The environment and disease: association or causation? Proc R Soc Med. 1965;58:295–300.

[2] The subcommittee on Dietary Patterns said repeatedly, regarding outcomes on obesity, diabetes, and cardiovascular disease, “For diets based on macronutrient distribution and this outcome, the full body of evidence is still under review.” See transcript, March 12, 2020, at 38:03, 45:24, and 51:12.