How to Validate an Industry Study
Confirming that an industry study is involves the application of rigorous, systematic, and objective procedures is one of the most critical yet overlooked elements of report analysis. For what is the benefit of basing your future business model—or personal purchasing choices—on a study that is not truly valid or is slanted to a particular point of view?
To best determine if a study is grounded in scientifically based research, answers to the following nine questions should be thoughtfully assessed.
- Was the study conducted by a reputable research firm? Since organizations will occasionally hire a research firm that will focus on data that supports their point of view, confirming that a research firm has a reliable reputation for unbiased work is the first step in determining a study’s validity. An easy way to do this is to see what other studies the firm has conducted and if there has been any online “chatter” about the validity of the findings.
- Where was the research conducted? Since international standards and practices vary widely, data should optimally be culled from sources that are geographically aligned with the report, for example, U.S. sources for data relating to U.S. manufacturing practices.
When comparing data amongst industries (such as plastics, paperboard, glass, etc.), are the dates of the studies comparable? Industries and trends can change dramatically in a relatively short period of time so a good study should clearly disclose the date of when the source data was generated.
- Have the results been vetted by a third party? Whenever a company or organization conducts a study within its own industry—especially if its findings support their industry—a qualified third party should vet the results to ensure that the data was properly collected and the results correctly interpreted and presented. An unvetted report is an indication that the study may be biased.
- How was the data collected and from how many sources? Since life cycle studies are often generated by firms using different methodologies, to ensure credible findings, researchers should identify and disclose differences in the scope and methodologies used to compile the source data.
- Is the scope of the study fully disclosed? For cost or timing reasons, an organization will sometimes limit the scope of a life cycle study, so disclosure of the scope is critical to properly assessing study results. A good study will also consider a cradle-to-cradle life cycle rather than a partial one that positions the data in such a way as to support potentially shaky conclusions.
- Are the products’ primary, secondary, and/or tertiary packaging equitably compared? In a comparative study, changes in primary packaging will often impact both secondary and transport packaging, making it necessary to study the complete packaging system. In a real way, it is cheating to focus on primary packaging and not reflect required changes in secondary or transport packaging.
- Is the study ISO 14040-compliant? ISO 14040 is an environmental standard that provides the framework for life cycle assessment analysis (LCA) and life cycle inventory studies (LCI). If a study is compliant, it should be readily apparent in the introduction of the report. Even so, compliance does not mean that all the measureable variables (or quality indicators used to determine the degree of adherence to a standard) are being fully represented. So if a report only addresses some of the indicators (such as the product’s impact on soil and water depletion but not on aquatic eco-toxicity and carbon sequestration), this may suggest that the organization is selectively publishing only those indicators that support their claims. If a comparative study claims to be ISO 14040-compliant, then full disclosure of results is required. So make sure to ask the how many indicators were originally measured and how many were actually published.
Is the issue of carbon sequestration properly addressed? Carbon sequestration refers to how much carbon is locked up (i.e., not being released into the atmosphere) by any given product or process. According to a 2007 study by the National Council for Air and Stream Improvement, greenhouse gas emissions from the forest products industry’s value chain are largely offset by the sequestration of carbon in sustainably managed forest products (such as paper and paperboard). So in the case of a study that compares paperboard to other packaging substrates, it is important for studies to offset the environmental impact of substrates such as paperboard because of its innate ability to sequester carbon. If such an offset is not considered, then the full life cycle of the substrate is not being fully represented.
If you find that the answers to the above questions show a marked bias towards a particular perspective—or you cannot confirm that the researchers followed rigorous analytic methodologies—then it is advisable not to take the results seriously or base any important policy decisions on the findings.