This whitepaper discusses the many complexities related to CECL and examines the importance of institutions taking a proactive monitoring and backtesting approach. The steps detailed in this paper are critical in driving the accuracy of a CECL model, required in a sound model risk management plan, and are a practical approach for management to fully understand the assumptions used within the CECL model.
Introduction
The U.S. financial crisis of 2007-2009 had a profound effect on the banking industry as it uncovered the near-term view of risk that banks, and their examiners, took when analyzing their loan portfolio and the economy. The financial crisis had plenty of causes but two (2) of the primary causes were the commercial real estate and residential real estate asset value bubbles and subsequent three-to-five-year asset value decline. Many bank internal loss models were unable to properly address the sustained value freefall that occurred in key U.S. markets, followed by sustained levels of judicial foreclosures that occurred across the country. This left banks scrambling to predict the bottom of a powerful recession without any historical reference points or functional models offering valid insight into a recovery.
In July 2010, following the end of the recession, Congress passed the Dodd-Frank Act which introduced a host of banking regulations and consumer protections. The Act also introduced the concept of a forward-looking view of a bank’s balance sheet by creating what ultimately became the Comprehensive Capital Analysis and Review (“CCAR”) and Dodd-Frank Act Stress Test (“DFAST”) for larger national and regional banks and their holding companies. These stress tests generate impact to the capital structure and balance sheet by requiring firms to measure the impact by applying several pre-determined economic scenarios. The tests require banks to consider a forward view of broader economic variables rather than the near-term review of delinquencies, bankruptcies, and nonaccruals that most institutions reviewed before the recession.
In 2013, The Financial Accounting Standards Board (“FASB”) in conjunction with Federal examiners introduced the forecast to the allowance process with the issuance of the Current Expected Credit Loss (“CECL”).
CECL was initially released in 2016 in ASU 2016-13 with a 2019 compliance date for the largest institutions, but this was later pushed back to Q1 2020 for the largest institutions. CECL follows the Allowance for Loan and Lease Losses (“ALLL”), which was a rolling one-year estimate of uncollectible amounts. The ALLL was typically based on an historic average loss rate combined with a qualitative adjustment for current conditions. CECL contrasted the ALLL by adding a “reasonable and supportable forecast” to the historic loss rate and current adjustment for a life of loan loss rate estimate of expected losses.
FASB identified appropriate methodologies firms may use to calculate the CECL historic loss rate. The examiners have been reticent about endorsing one method over the others as well as prescribing how to appropriately forecast expected losses. This reticence of the examiners left banks with a tremendous amount of uncertainty regarding the development of their CECL models, the appropriateness of their calculations and, more specifically, the development of their forecast. The examiners believe that banks should be able to determine an expected loss based on their knowledge of the portfolio, and economic conditions, while considering the framework of the CECL standard. Essentially the examiners are asking firms themselves to determine what they believe is a reasonable, and documented, process for determining a CECL reserve.
A CECL calculation is a loan level, data intensive exercise, that requires either the development of a new model internally or the purchase of sophisticated software from a vendor. The sophistication of a CECL model and the importance of a CECL reserve require firms to increase their development and oversight of their model risk management program. Model risk management should not be just an audit function that checks the box, especially as it relates to CECL. It should be an integral part of the CECL process that allows firms to continually refine their model so expected losses ultimately forecast actual losses.
What is Model Risk Management?
Model risk management is the framework that firms need to develop to identify and mitigate the risks concerning the software, inputs, assumptions and results associated with a quantitative process. The Federal Reserve and the OCC jointly issued Supervisory Guidance on Model Risk Management (SR 11-7, 2011-12) in 2011 because of the proliferation of sophisticated models in the decision-making process of banks. The guidance defined a model as “a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates”. The guidance further defined the two primary types of model risk:
- The model may have fundamental errors and may produce inaccurate outputs when viewed against the design objective and intended business uses
- The model may be used incorrectly or inappropriately
Examiners have begun taking Covid-related interest in the model risk management environment for key strategic models due to the complexity of the models, and in CECL’s case, the assumptions used for the methodologies and the forecast. The development of a CECL model presents several risk management challenges to a bank, ironically, due to the largely subjective nature of the CECL guidance. CECL guidance has identified six (6) methodologies, each with differing degrees of complexity, while the guidance, and examiners, have not been prescriptive regarding a preferred methodology.
The guidance concerning the forecast component of CECL is also non-prescriptive as well, leaving it to the banks to determine a “reasonable and supportable forecast”. Essentially leaving it to the banks to determine a well-documented and thoughtful approach to determining a CECL calculation.
A key component of model risk management is the concept of effective challenge, or the “critical analysis by objective, informed parties who can identify model limitations and assumptions and produce appropriate changes” according to Model Risk Management guidance. Essentially effective challenge is the concept that there is at least equal or greater incentive to challenge the validity of the champion model as there was building the champion model. Laying the groundwork for a process to validate a CECL model should be a critical part of the development of that model. The focus of Model risk management for CECL should not be to tear down a CECL model but to improve the model and act as a risk mitigation tool. To do this the program must test and challenge each of the following items in a CECL model:
Governance
A model risk governance program is defined by the Board of Directors and establishes the policies and procedures as well as the resources necessary to define the mission of the program. The governance program is delegated to senior management to implement as part of a wider risk management mandate. Management is responsible for all aspects of the CECL model development, including allocating development and testing resources, vendor management, reviewing results and findings as well developing an effective model risk management program that tests the results.
A strong governance program is key to developing a CECL model because it will define the framework to develop, operate and ultimately test the model. Examiners are reviewing the models, but they are also critically reviewing the process of how it was developed and the overall governance structure.
There are three key components to a successful governance program:
Policies and procedures are critical for CECL given the importance of a CECL calculation to examiners, the Bank, investors, and key stakeholders. The policies should show key aspects of the CECL process including model development, vendor selection as well key items such as data validation, methodology selection and forecasting. Policies should also identify members of the CECL committee, with their roles and responsibilities defined. These policies must address the concept of critical testing to promote effective model risk management for the CECL model. This testing includes using external third-party providers to validate the CECL model, its inputs, and its results. Critical testing should also include backtesting, the critical view of results and the measurement of forecasted versus actual results.
As with any policy, these should be reviewed no less than annually by the Board of Directors. The CECL documentation generally will include, but not be limited to, the following: model document, accounting policy, allowance policy, data flow diagram, and internal controls. Model risk protocols are typically defined in the policies and procedures but to address the development, implementation and use of the CECL model. These protocols are the actual process and means of addressing risk in the CECL model. They include model development or vendor selection, methodology testing, assumptions as well as the type or process of testing the results for sufficiency and accuracy. Finally, oversight is critical in the CECL process. Oversight falls to the Board of Directors, in close association with management, but as it relates to CECL this means being able to challenge the assumptions and the results. CECL at a minimum was a largely subjective process.
Adding COVID into the mix has added a level of complexity and uncertainty not seen before. Management has the responsibility to understand the results and communicate them to the Board, shareholders, and stakeholders but to ensure that the CECL model has considered all relevant inputs and assumptions. Only by carefully considering the inputs and assumptions can management accurately provide an estimate of losses.
Model Validation
CECL model validation is increasingly being looked upon by examiners and auditors as a key item that documents the potential accuracy, reliability, and sufficiency of a CECL model. It can also identify any weakness or deterioration in model performance. The validation is typically done by a knowledgeable external third party that tests all aspects of the model and delivers a final report.
It is important to note the importance of a model validation as a risk management tool. The results will highlight areas where the model has deviated from policy as well as whether the model meets the terms of the CECL guidance. Having this external third-party validation should be a critical document to address any issues with the model.
Guidance from examiners suggest that some form of validation be done no less than annually but preferably periodically with dedicated internal employees not involved in the CECL development process.
This ensures continuous testing of a model subject to increased levels of complexity and assumptions. Validations should also be considered prior to implementation of the model. Using a third-party firm to perform a CECL validation will require that the firm has access to the model and its inputs and assumptions. It is important to be upfront with any validator with assumptions and thoughts for them to be an effective challenger. CECL is a new calculation with an added layer of uncertainty due to COVID.
An effective validation will review the logic and soundness of the model and determine if the model is returning the desired results. A formal validation will address several key items from a CECL model. First, it will address if the model meets GAAP (“Generally Accepted Accounting Principles”) standards by correctly reserving for pooled loans, individual loans, unfunded commitments, Purchase Credit Deteriorated (“PCD”) loans, and their specific inputs. This includes methodology selection, use of amortized cost, individual loan selection among others.
Second, the validation will determine if the model itself is returning the results as intended. This may involve a detailed scrub of the model processes in the case of an internal model. Models sourced from vendors typically rely on a SOC 1/SOC 2 report prepared by a third party on the vendor.
The next step in the validation process concerns an analysis of the overall soundness and sufficiency of the model and its corresponding inputs and assumptions. Having a conversation with the validator prior to any review about each step in the model will ensure the logic and assumptions are understood. Initially the model and its assumptions will be compared to policy documents followed by an analysis of any narratives or documentation. The next, and maybe most critical, part of the process is testing the methodologies and forecast for each properly segmented pool. This may involve reviewing multiple scenarios. The validator is reviewing each primary component of the CECL calculation individually to identify any risk associated with the calculation. As a whole, the validator is working to determine if the model is generating a reserve sufficient to cover expected losses.
Backtesting
Backtesting is a critical tool that should be considered both during the testing process as well as after CECL implementation. Backtesting is the continuous comparison of forecasted results and actual results to drive the refinement of the CECL model. Backtesting can take many forms but its purpose in regard to CECL concerns the ultimate refinement of the CECL model so that it accurately reflects expected losses over a specific timeframe. An example that CECL adopters can relate to is how COVID has impacted the CECL ACL (“Allowance for Credit Losses”) process. Many institutions struggle to get loan level data back to 2006 or earlier, which would be needed to cover one full economic cycle. If loan level data was available from 2000 to December 31, 2019, the time of adoption for many large institutions, an institution could likely build statistically acceptable models that could predict future losses with a reasonable level of accuracy.
However, the COVID pandemic has triggered heavy provision expense in the first quarter of adoption of CECL for two main reasons. First, a global pandemic of this kind is nearly impossible to predict, and second, very few quantitative models were calibrated with the extreme economic factors present in 2020. This can be easily illustrated in the following graph from the FRED database that simply charts the unemployment rate and net charge off rate for all banks since 2000.
Until 2020, there is reasonable correlation between the unemployment rate and charge off rate. An effective backtest will not only show that charge offs did not correlate with the unemployment rate, or at least not yet, but it should also give some insight into where the breakdown(s) are.
For example, what economic variables were considered? There has been unprecedented stimulus from the government including payment protection program, employee retention credits, main street lending program, expanded unemployment benefits, and others that impacted the quantitative calculations. There were also loan modification programs that got relief on things like TDR accounting. Under these modifications, the contractual terms of the loans prevented defaults from occurring as there were no payments due for extended periods of time. As such, in order to accurately model these, internal models and vendor models would have to be modified to more accurately include model payment holidays.
In some instances, it is not reasonable to expect a quantitative model to account for all of these things, but it is important to understand exactly what is considered in a quantitative model so that qualitative adjustments can be properly recorded.
Although backtesting should be leveraged for all loss methodologies, there are certain methodologies that are less challenging to backtest. Specifically, methods that use explicit assumptions, such as Probability of Default (“PD”), Loss Given Default (“LGD”), and prepayment speed assumptions. Furthermore, if the method natively segments the default, losses, or prepayments into a distinct time series, it allows for more direct comparisons. Generally, this includes DCF/cash flow methods and vintage methods to a certain degree.
Finally, even when performed on collectively assessed loans, if the underlying calculations are loan level, a more complete analysis can be performed.
Final Notes
As discussed in this paper, there are many complexities related to CECL including vendor management, governance, financial reporting, regulatory review, and external audit. In order to ensure success, institutions need to make sure they have a proactive monitoring and backtesting plan. These steps are critical in driving the accuracy of a CECL model, required in a sound model risk management plan, and are a pro-active approach so that management can fully understand the assumptions used within the CECL model.
About the Authors
Rob Ashbaugh, Managing Consultant
Rob is a former banker and portfolio risk manager with more than 20 years of commercial banking and capital markets experience at both the large bank and community bank levels. He has managed the ALLL and portfolio stress testing process and been an invited speaker at many national and regional conferences. Rob is a past holder of the Series 7, 52, and 63 licenses and received his Bachelor’s degree in both economics and international business from Temple University. When Rob is not working, he enjoys skiing, fishing, and spending time on the Lake.
Craig Engle, Director of Product Management
Craig is a co-founder of Valuant has more than twelve years of public accounting, consulting, and software development experience with an emphasis in financial institutions and SEC registrants. He serves as Director of Product Management working with both the Product Team and Client Service Team. On the product side, he provides technical expertise on existing features and new enhancements. On the Client Service side, he assists financial institutions on Purchase Accounting (Day 1 and Day 2), Allowance for Loan Loss (“ALLL”) consulting and software implementation, and Current Expected Credit Loss (“CECL”) consulting and software implementation. He played a key role in ValuCastTM, a proprietary suite of software tools that assists companies with the various services outlined above. When Craig is not working at Valuant, he enjoys spending time with his wife and kids, spending time outdoors, and serves on the Leadership Board at Emmaus Church.