Tag: methodological problems

From Climate Etc.

by Nic Lewis

The determination of equilibrium climate sensitivity (ECS)—the long-term warming response to doubled atmospheric CO₂ concentrations—remains one of the most crucial yet challenging problems in climate science. Recent exchanges in the literature have highlighted both the complexity of this endeavor and the importance of maintaining rigorous methodological standards in the pursuit of reliable estimates.

Background and Context

In 2020, Steven Sherwood and twenty four co-authors published a comprehensive assessment of Earth’s climate sensitivity (S20) that claimed to substantially narrow the ‘likely’ (66% probability) uncertainty range to 2.6–3.9°C, with a best estimate of 3.1°C. Their assessment was undertaken under the auspices of the World Climate Research Programme’s Grand Science Challenge on Clouds, Circulation and Climate Sensitivity, following a 2015 workshop that I participated in at Ringberg Castle in Germany.

S20 has been an exceptionally influential study. Its observationally-driven ECS approach and range was approximately adopted in the 2021 IPCC sixth assessment report (AR6). This represented a significant departure from the broader ranges that had persisted in IPCC assessments since the late 1970s: previous IPCC best estimates had usually been 3°C, but their uncertainty ranges had remained much wider, almost always spanning 1.5°C to 4.5°C.  Moreover, AR6 gave a 90% probability (‘very likely’) range for the first time, of 2.0–5.0°C, widened slightly from the S20 estimate of 2.3–4.7°C.

In 2022, I published a detailed peer-reviewed examination (Lewis22) of S20’s methodology, identifying what I believed to be several significant issues with their analysis. My revised assessment, using their basic framework but with corrected and improved methodologies and carefully justified updatings and other revisions to some of their input data, suggested a substantially lower and narrower likely range for ECS of 1.75–2.7°C, with a median estimate of 2.16°C, and a 90% probability range of 1.55–3.2°C .

As in S20, ECS estimation in Lewis22 was based on combining historical period evidence with that from the Last Glacial Maximum (LGM) and mid-Pliocene Warm Period (mPWP) paleoclimate periods, and from process understanding (estimates of individual climate feedbacks). Like S20, Lewis22 actually estimated S, the usual proxy for ECS in GCMs, which is generally slightly lower than their estimated true ECS. The two terms are only distinguished here when discussing their relationship.

In 2024, Sherwood, together with another scientist, Chris Forest,[1] published an opinion piece in Atmospheric Chemistry and Physics journal (ACP) questioning whether climate sensitivity uncertainty had really been narrowed since S20, and specifically challenging several aspects of my 2022 analysis. I considered that their article contained fundamental mischaracterizations of my work that warranted clarification, along with broader methodological concerns that merited discussion. My endeavor to do so has now been published in ACP as ‘Comment on “Opinion: Can uncertainty in climate sensitivity be narrowed further?” by Sherwood and Forest (2024)‘.

Methodological Errors and Inconsistencies in the original Sherwood et al. study

My 2022 study identified several methodological problems in the S20 analysis, some of which significantly affected their results. Sherwood and Forest claimed, incorrectly, that these merely represented “differences in opinion on methodological choices and priors”, not errors, and that “they moreover were acknowledged to have little effect on the outcome”.

The most fundamental error involved was the use in S20 of an invalid likelihood estimation methodology. The likelihood function—which quantifies how well different values of the parameter being estimated match observational evidence—is fundamental to Bayesian estimation, indeed to all statistical estimation of uncertain parameters. Lewis22 used three quite different likelihood estimation methods,[2] which all produced the same results. Moreover, S20 used an uncertainty estimate that was a factor of ten lower than stated for Paleocene-Eocene Thermal Maximum (PETM) CO₂ forcing, due to a coding error[3].

The resulting errors in S20’s likelihood estimates are shown in Figure 1.The estimates of historical and PETM evidence likelihoods at high climate sensitivity values were particularly affected. Although S20 did not use PETM evidence for its main results, and the underestimation of historical likelihood at high ECS values had only a small effect on the combined-evidence likelihood, unsound derivation of likelihoods is a very serious statistical error.  Sherwood has admitted, in a detailed comment (here: CC1) during the peer review process prior to my Comment on their 2024 opinion piece being accepted, that the sampling method S20 used to derive its likelihoods was “probably not optimal”, but has not published any related correction to S20.

Additionally, Lewis22 identified a mathematically incorrect treatment in S20 of CO₂ forcing estimates used to estimate ECS process and historical evidence. When deriving ECS from climate feedback estimates, they used an effective radiative forcing (ERF) value for a doubling of CO₂ concentration (F_2×CO2) based on fixed sea surface temperature (SST) simulations, while they should have used regression-based forcing estimates, which are lower. This unjustifiable choice biased S20’s process and historical evidence-derived estimates upward by approximately 16%. This issue is illustrated in Figure 2, using data for a typical GCM.

Also, S20 converted their paleoclimate based estimates of true equilibrium climate sensitivity to estimates of S using an ECS to S ratio (1+ζ) estimated by comparing ECS derived from long GCM CO₂ doubling simulations, with S derived from 150 year CO₂ quadrupling simulations by the same eight GCMs. They scaled the CO₂ quadrupling simulation based estimates down by a factor of two, rather than by the true ratio of ERF from a quadrupling of CO₂ (F_4×CO2) to F_2×CO2 of about 2.1. This resulted in an inconsistency between S20’s paleoclimate estimates of S and those from its process and historical evidence (which were based on estimates of F_2×CO2, not of F_4×CO2/2). Lewis22 avoided this inconsistency by basing its estimates of the ECS to S ratio on comparison of estimated true ECS with S estimated from regression over the first 150 years, separately within each of sixteen long CO₂ doubling or CO₂ quadrupling GCM simulations, avoiding the need for any F_4×CO2 to F_2×CO2 scaling factor[4].  

These issues represent more than methodological preferences—they constitute conceptual errors and inconsistencies that materially affected the final results. When I corrected these problems while retaining all other aspects of S20 analysis, the climate sensitivity estimates shifted substantially downward.

Clarifying Misrepresentations

Sherwood and Forest’s article contains several significant mischaracterizations of my work. Most importantly, they claim that by rejecting the possibility of a large historical ‘pattern effect’ and downwardly revising estimated historical aerosol cooling, Lewis22 had concluded that “the historical record rules out a high ECS level.” This characterization is entirely incorrect.

My analysis of historical evidence alone yielded a 90% uncertainty range of 1.2–7.6°C, which clearly does not rule out high sensitivity values. Even with the addition of a reasonable prior constraint (0 <  ECS < 20 °C), based on the Earth not having experienced runaway warming or cooling, the range was 1.15–6.1°C. Both ranges substantially exceed the 4.7°C upper (95%) bound from S20’s combined-evidence estimate.

The narrowing in my final estimates resulted from combining multiple independent lines of evidence—process understanding, historical observations, paleoclimate reconstructions—using appropriate statistical methods, with a prior distribution designed to have minimal influence on the results. This is precisely how one expects Bayesian analysis should work when evidence is combined: no single line of evidence may rule out particular values, but their combination may provide strong constraints.

The Challenge of Aerosol Forcing Uncertainty

One reason historical evidence alone cannot definitively constrain high ECS values lies in the substantial uncertainty surrounding historical aerosol forcing. Aerosols from fossil fuel burning and other anthropogenic sources have provided uncertain amounts of cooling that have partially masked greenhouse gas warming, creating a fundamental difficulty in interpreting the historical temperature record.

In Lewis 2022 I revised the aerosol forcing distribution used in the original S20 analysis, reducing the probability assigned to very strong cooling, based on observational constraints. However, this revision followed evidence from other researchers against very strong aerosol cooling, and importantly, had minimal impact on my final combined-evidence ECS estimates. When I reverted to S20’s original aerosol assumptions while maintaining all other revisions, the median Lewis22 ECS estimate changed by less than 0.05°C. Lewis22 did not test using the AR6 aerosol distribution, however its median value is almost identical to that of S20’s aerosol distribution.   

This demonstrates that my study’s lower ECS estimates were not driven by its revision to aerosol assumptions, contrary to the assertions in Sherwood and Forest’s critique of Lewis22.

Reassessing the Pattern Effect

The “pattern effect”—how the geographical distribution of SST warming affects climate feedbacks—represents another area where Sherwood and Forest’s 2024 opinion piece challenged my analysis. While I had adopted a smaller estimate based on my evaluation of the available evidence, they argued that recent studies strongly support a large historical pattern effect. Sherwood doubled down on this in his comment (CC1) on my 2025 manuscript, writing that Lewis22 argued in particular that “the aerosol forcing and historical pattern effect were each smaller and better known than in either S20 or AR6”.

In fact, more recent work by Modak and Mauritsen (2023) supports smaller pattern effect estimates, obtaining a slightly lower estimate than per Lewis22 when averaged across multiple SST datasets. Notably, they found that the most commonly used, AMIPII, SST dataset, which produces the largest pattern effect estimates, appears to be an outlier among available datasets.

Moreover, upon examining the three studies Sherwood and Forest cited, I found that two of them focused on recent decades (post-1980 and post-2000) rather than the full historical period relevant to climate sensitivity estimation in S20 and Lewis22. The third study, when its data from an alternative SST dataset to AMIPII is analyzed using approaches that minimize bias from interannual variability and account for model structural similarities, yields an estimate closely in line with my adopted value.

As regards how well the historical pattern effect is known, in L22 I adopted the same large estimate of the degree of uncertainty involved as Sherwood et al. used in S20 and the IPCC assessed in AR6.

Statistical methodology and Prior selection

Beyond specific technical disputes lies a more fundamental disagreement about statistical methodology. For scientific inference to be reliable, the statistical methods employed must be calibrated to produce uncertainty ranges that approximate true confidence intervals. Where the data are weak, this requires either objective Bayesian methods with noninformative priors or frequentist approaches—both of which are designed with this goal in mind.

The original Sherwood et al. study employed what statisticians term a “subjective Bayesian” approach, incorporating a prior distribution for the parameter being estimated based on expert judgment. Such an approach may produce very ill-calibrated uncertainty ranges, although in S20’s case the chosen prior (a uniform prior in λ) was close enough to a noninformative one that the mis-calibration was minor.

In Lewis22, I adopted an “objective Bayesian” methodology using a computed Jeffreys’ prior for the combined evidence, designed to minimize the influence of the prior on the resulting estimate. Note that this is different from using an objective (noninformative) prior for one line of evidence and using the resulting posterior pdf as the prior for estimation from the likelihood from the next line of evidence analyzed, and so on. Although such ‘Bayesian updating’ is standard statistical practice, contrary to general belief it is not soundly based mathematically and it may not result in well-calibrated estimation. See here.

The distinction between subjective and objective Bayesian estimation matters significantly when dealing with highly uncertain parameters like ECS. Subjective Bayesian methods are not designed to produce well-calibrated confidence intervals, and their uncertainty ranges can be severely biased when data are insufficient to dominate prior assumptions—a common situation in climate sensitivity estimation.

I demonstrated this problem in my Comment article by showing that a seemingly reasonable uniform prior in ECS (spanning 0–20°C), as used in the IPCC AR4 report, produced unreasonable results when applied to S20’s historical evidence alone, yielding a median estimate of 8.5°C with a 95% bound of 18.6°C, compared to a median estimate of 4.2°C with a 95% bound of 13.7°C when using a noninformative computed prior. The mathematical properties of a uniform prior in ECS makes it highly informative rather than neutral in this case, concentrating probability at extremely high values.

Steven Sherwood claimed in his comment (CC1) that “The main impact of L22’s objective prior is to narrow the pdf—i.e., claim ECS to be known more confidently.” The truth is the opposite. Adopting L22’s objective prior in place of S20’s prior actually widened and slightly raised S20’s ECS range. I provided a full response (AC2) to all the points in Sherwood’s comment.

Structural Model Uncertainties

Both Sherwood and Forest and I agree that structural uncertainties in the models used may affect ECS estimates. They focused on assumptions in ‘forward’ models used to predict what would be observed given a particular ECS, however these typically include assumptions based on GCM behavior.

The most significant structural uncertainty may concern tropical warming patterns. Climate models consistently predict that greenhouse warming should weaken the east-west temperature gradient across the tropical Pacific, with the eastern regions warming faster than the western regions, contrary to what appears to have happened during most of the historical period warming. This predicted pattern change underlies the weakening of climate feedbacks over time in GCM simulations, which contributes to their higher ECS estimates and underlies the pattern effect based upwards adjustment to climate sensitivity estimates based directly on historical warming and forcing (S_hist).

However, several recent studies suggest that this predicted pattern change may be unrealistic, with western Pacific sea surface temperatures actually being more sensitive to greenhouse gas forcing than eastern Pacific temperatures, contrary to model predictions. If correct, this would imply that the feedback weakening simulated by models over 150-year timescales following abrupt CO₂ increases is also unrealistic.

This structural uncertainty affects not only historical ECS estimates but also those based on process understanding and emergent constraints, since these typically rely on model behavior that incorporates the potentially erroneous tropical warming patterns. If the GCM-predicted weakening of the tropical Pacific east-west temperature gradient is not realistic, all these types of ECS estimates would likely be biased toward overestimating ECS. Even if the Pacific east-west temperature gradient does eventually weaken to the extent simulated by GCMs, a multidecadal-to-centennial delay in that weakening could imply a significantly lower warming this century than implied by GCM behavior.

Summary

My 2022 analysis systematically addressed multiple aspects of the original Sherwood et al. study, correcting erroneous likelihood computation, replacing unsatisfactory methodological choices that resulted in biased and/or inconsistent estimation with more appropriate ones, and revising certain input data assumptions–mainly updating them based on more recent evidence. The resulting ECS estimates were lower and more tightly constrained.

My published Comment includes a detailed sensitivity analysis that I conducted, which shows how different classes of revisions contributed to the differences between the S20 results (after correcting S20’s likelihood errors and adopting computed Jeffreys’ priors, which slightly raised the S20 ECS estimate) and the final Lewis22 results. The largest contribution (55% of the total reduction in median ECS) came from remedying the F_2×CO2 and the ECS-to-S ratio estimation to avoid bias and inconsistencies, and updating the S20 estimates of non-aerosol forcing and of the ratio of ocean surface air temperature to SST warming, using IPCC AR6 values. These changes  should be uncontroversial. Most of the remaining reduction came for reappraising LGM cooling and forcing estimates, and using more recent and appropriate estimates for two mPWP-specific ratios. The justification for each of these revisions is discussed in my published Comment on Sherwood and Forest’s article. In the light of conflicting evidence and resulting large uncertainty concerning cloud feedback and aerosol forcing, the Lewis22 revisions to those items, and possibly also to pattern effect estimates, are more uncertain. However, even without adopting any changes to the S20 estimates of those three items, over 80% of the reduction in median ECS in Lewis22 is retained.

Implications for Climate Science

The exchange with Sherwood and Forest highlights several important issues for the climate sensitivity estimation field.

First, the reluctance to abandon statistical methods that can produce systematically biased results is concerning. The continued use of subjective Bayesian approaches for ECS estimation, despite their known limitations when data are weak, compromises the reliability of ECS estimation.

Second, the possibility that fundamental aspects of climate model behavior—particularly tropical warming patterns—may be incorrect has broad implications. If models systematically overestimate the weakening of climate feedbacks over time, this could affect not only ECS estimates but also projections of near-term warming rates and regional climate changes. Resolving this issue should be a key objective.

Third, careful methodological scrutiny is important. Climate sensitivity estimates inform policy decisions with enormous economic and social consequences. Ensuring that these estimates are based on rigorous and unbiased analysis is essential for maintaining public trust in climate science.

Moving Forward

My intention in my published Comment and this article has been to contribute to a more accurate understanding of climate sensitivity estimation, in particular by correcting misunderstandings of the causes of the differences between estimates in S20 and in Lewis22. The poor understanding displayed in Sherwood and Forest’s article of the effects of the revisions made to historical aerosol forcing and pattern effect estimates, and by their claim that as a result in Lewis22 the historical record rules out a high ECS level, are worrying. It is clear from L22’s results tables that the historical record does not rules out a high ECS level−it is paleoclimate and process evidence that do so. Steven Sherwood’s incorrect assertions in his comment (CC1) on my 2025 manuscript that the main impact of L22’s objective prior is to narrow the pdf [for ECS], when it in fact Lewis22 showed that it slightly widens S20’s pdf, and that L22’s multiplication of fixed-SST F2×CO2 estimates by 0.86 when estimating S from process and historical evidence introduces n inconsistency between forcing and feedback, when it actually avoids such an inconsistency, are also concerning.

The ongoing debate over climate sensitivity reflects the inherent difficulty of the problem, and some methodological weaknesses–particularly regarding statistical issues–in published research, rather than any fundamental disagreement about the reality of human-caused climate change. While climate sensitivity research has progressed substantially over the last decade or two, significant uncertainties remain. Debate and disagreements are healthy, but should centre on evidence and its interpretation, not on baseless claims.

---

[1] Ironically, Forest was  lead author of a key ECS study in the IPCC’s fourth assessment report (AR4) that Lewis had shown in a peer reviewed 2013 paper to be riddled with errors in its likelihood estimation. Surprisingly, although Forest (with his joint author) corrected those errors in a later study using the same methodology, he never corrected the same errors in his AR4 study – which has been cited nearly fifty times since my 2013 paper showed it was erroneous.

[2] Two for historical evidence, as the third method was defeated by S20’s highly asymmetrical aerosol forcing distribution. See Lewis22 Supporting Information Figs. S1, S2, and S3

[3] I pointed this coding error out to the S20 authors in September 2022. They have now corrected this error in the online version of S20, albeit without acknowledgement of my having notified them of  it

[4] Even when using their method, S20 would have estimated the ECS to S ratio at a value very close to that derived in Lewis22 had they not included an outlier CO₂ quadrupling simulation result from a GCM in which it exhibited near runaway warming.