A Loan Decision That Takes Four Seconds Still Has to Explain Itself

What Underwriting Has to Determine

subscription It's the decision of whether to extend credit and on what terms. If you eliminate the paperwork it all comes down to an estimate: the probability that this specific borrower will pay back the loan combined with what the lender would recover if he didn't

The traditional tool for that estimate is a credit score built from payment history how much is owed relative to available credit how long the file has existed and a handful of similar factors. It's a narrow model on purpose. Both regulators and lenders like it to be transparent. Anyone can look at the inputs and roughly reconstruct the output

What Automated Underwriting Changes

A machine learning model can weight many more variables than a five-factor scorecard and can capture interactions that a linear model cannot. Two factors mean that nothing on its own can combine into a strong signal and a flexible model finds that combination on its own rather than waiting for an analyst to manually code it

One real use of this is to score applicants with poor credit history since a thin file is not the same as a risky file it's just an unmeasured one. I'm not going to dwell on that side of the story here it deserves its own treatment. What matters in this article is more limited: automation tends to become more precise specifically as it becomes more complicated and that creates a problem that the old scorecard never had to face

The Law Requires a Reason

Consumer loans are governed by rules that don't care how the decision was made. Two of them are the most important here

The first is the adverse action notice required by the Equal Credit Opportunity Act and its implementing regulations. A rejected applicant has the right to know the main and specific reasons for that decision not a form letter. That single requirement is the lynchpin around this entire article. It means that whoever produced the decision has to return a short accurate human-readable list of the reasons at all times and for each applicant

The second is the prohibition of discriminatory results including disparate impact: A practice that at first glance appears neutral may still be illegal if it produces substantially worse outcomes for a protected group regardless of whether anyone intended that outcome

What a Reason Code Actually Is

Let's say a lender turns you down. The letter can't just say "the model said no." It has to include something like "the amount owed on revolving accounts is too high" or "the length of credit history is too short." Those short phrases are reason codes and aren't picked by a compliance officer after the fact. They come out of the template itself

In a classic points-based scorecard this is almost mechanical. Each factor has a maximum number of points your record earns an amount less than that maximum on each and the difference between what you earned and the maximum is the "cost" of that factor. Sort the gaps from largest to smallest and you'll have your reason codes in order: largest gap first second largest next and so on. I'll explain this with real numbers in a minute because building one once makes the idea fit

The more difficult question is what a reason code means when there is no points table. A gradient-driven tree or neural network does not calculate a "maximum gap" for each variable because it is not built from independent additive factors in the first place. Your variables interact their effects are modified and reversed depending on the context and there is usually no clear way to ask you how many points your income cost you because the model was never organized around points. Lenders respond to this with methods ofa posteriori approximation most commonly a technique called SHAP short for Shapley Additive Explanations which borrows an idea from cooperative game theory to estimate how much each input variable drove a specific prediction up or down relative to an average applicant

The Accuracy You Give Up

Here is the real tension that regulation creates and it is real not a compliance technicality. The kinds of models that are easiest to explain are also structurally the least flexible

A logistic regression with a dozen hand-picked monotonic variables (more income is always better more delinquencies is always worse no factor can reverse direction depending on the context) gives you the reason code for free because the coefficient on each variable is the explanation. There is nothing to reconstruct after the fact. But that same monotonic constraint means that the model will never be able to learn that say a high utilization rate matters a lot to someone with a short credit history and almost nothing to someone with a long one. You have to deal with the utilization ofthe same way for everyone because letting the effect double would break precisely what makes it explainable

A model without that constraint free to allow effects to double reverse and interact will generally better fit the true relationship between financial behavior and pay because that relationship is really that fuzzy in real life. Therefore the model that best predicts default is usually the most difficult model to translate into four short sentences that someone without a financial background can actually read and understand. That's not a coincidence and it's not something a better engineering team will fix. It's closer to a true one.trade-off like the one between a car's fuel economy and its power. You can move along the curve. You can't fool it completely

Call it as an example a six-point gap in AUC (a common way to measure how well a model classifies risky versus safe applicants where 50 is a coin flip and 100 is a perfect separation) between a fully constrained monotone model and an unconstrained one trained on the same data. Suppose the constrained model with a fixed target default rate of 4 percent approves 60 out of every 100 applicants who apply.Unrestricted model with that same 4 percent default rate approves 68 out of every 100. That's 8 more approvals per 100 applicants or about 13 percent more credit extended with identical risk (8 divided by 60 is a little more than 0.13). Those 8 people are not approved under the model for which a lender can put their name in a reason code. That gap repeated throughout an entire portfolio is theactual price of explainability. It is not paid by the lender's compliance department. It is paid by the applicant within the range that they would have qualified based on the most accurate model and do not qualify based on the one that the lender can actually explain

A Worked Example: Building a Reason Code by Hand

Let me build the scorecard version from start to finish with made-up round numbers since that's the version simple enough to verify completely by hand

Suppose a simplified scorecard divides 1,000 total points into five factors: payment history worth up to 350 points amounts owed worth up to 250 length of credit history worth up to 150 new credit worth up to 100 and credit mix worth up to 150. Verify that these add up to 1,000: 350 plus 250 is600 plus 150 is 750 plus 100 is 850 plus 150 is 1000. Good

Now let's say an applicant scores 300 out of 350 on payment history 150 out of 250 on amounts owed 105 out of 150 on length of history 60 out of 100 on new credit and 130 out of 150 on credit mix. Add them up: 300 plus 150 is 450 plus 105 is 555 plus 60is 615 plus 130 is 745. This applicant's total score is 745

Let's say the lender's limit for this product is 760. The applicant is 15 points short since 760 minus 745 is 15 and they reject it. Now for the reason codes: Take the gap between the maximum of each factor and what the applicant actually got. The payment history 350 minus 300 is 50 points apart. The amounts owed 250 minus 150 are a difference of100 points. The length of the story 150 minus 105 is a difference of 45 points. The new credit 100 minus 60 is a gap of 40 points. The combination of credits 150 minus 130 is a gap of 20 points. Check the sum of the spaces correctly with the total: 50 plus 100 plus 45 plus 40 plus 20 is 255 and745 plus 255 is 1000 which matches the maximum. The arithmetic closes

Rank those gaps from largest to smallest and you have the adverse action letter: amounts owed first with a gap of 100 payment history second with a gap of 50 length of credit history third with a gap of 45 and new credit fourth with a gap of 40. Credit mix with the smallest gap at 20 doesn't make the cut if the lender discloses four reasons which is a practicecommon. Each number in that card traces back to an arithmetic difference that the applicant could in principle check for himself. That's the whole appeal of the scorecard approach and it's exactly the property that becomes harder to preserve once the underlying model stops being additive

Case Study: The Apple Card Investigation

In November 2019 a software developer publicly posted that the Apple Card issued by Goldman Sachs had given him a credit limit twenty times the limit he gave his wife even though the two filed taxes together and his wife had the longest credit history and by his account the highest credit score. Apple co-founder Steve Wozniak said something similar had happened to him and his wife. The story spread quickly for one obvious reason: This is precisely whatthe scenario that adverse action rules are meant to prevent a credit decision that no one could adequately explain at the time it mattered

The New York financial regulator opened an investigation into whether the underlying algorithm violated the fair lending law. Goldman's public position was that sex was never an input to the lending decision and that its underwriting did not use gender as a variable at all. That defense is a version of the "remove the variable" solution from a few sections back and it runs right into the same limitation: excluding a protected characteristic as explicit input does not prove that the result was free of its influence since a sufficiently flexible model can reconstruct much of whatthat variable would have told you from everything else in the file especially anything related to the joint financial life of a shared household

As I understand it and I would like anyone relying on this for more than the general shape of the story to check the primary source the regulator's review ultimately did not find a fair lending violation. Regardless the episode still really worked. It became the gold standard case for what happens when a lender can't give a specific convincing answer to the "why" in public under pressure and in real time. Whatever the final legal conclusion Goldman devoted months of public trust to a question that a well-structured explanation should haveable to respond at the time the client first asked it

Proxies, Feedback Loops, and Why Deleting a Column Isn't Enough

The intuitive solution to the risk of discrimination is to simply exclude protected characteristics from the model. Necessary but not sufficient because other variables act as proxies

A sufficiently flexible model can reconstruct a protected characteristic from combinations of allowed variables without anyone trying to do so. This is exactly why the previous Apple Card defense was weaker than it seemed. In practice compliance means testing results between groups after the fact rather than relying on an audit of the list of inputs and when a disparity arises looking for a less discriminatory alternative model that works comparably well

There is a second quieter problem behind the proxy problem and it is about where the training data comes from in the first place. These models learn from historical credit results and historical credit results reflect historical credit decisions some of which were themselves discriminatory. If a group was historically denied credit at a higher rate there is less payment data behind that group and a model trained on the results data learns from an absence rather than from evidence of actual risk. It then rejectsthat group more frequently resulting in no new payout data reinforcing the original pattern in the next training run. The model doesn't perform poorly in this scenario. It's about accurately learning a story that no one wants to keep playing and it's much harder to test than a single correlated column

Where This Breaks

I've been describing the explainability requirements as a clear and sensible check of a black box. Let me defend the other side because I think the honest version of this article has to do it

The first problem is that a SHAP-style explanation is an approximation not a reading. It depends on a choice of background data a choice of how missing features are handled and a handful of other modeling decisions that reasonable engineers might disagree with and two defensible SHAP configurations on exactly the same underlying model can classify the same applicant's main reasons differently. If the "reason" itself is somewhat arbitrary depending on the defensible method that produced it the promise that adverse action noticesprovide applicants with something specific and precise becomes much more unstable than the plain language of the regulation suggests. You may satisfy the letter of "produce a reason" while the reason you present is one of several equally defensible reasons that the same model could have generated instead

The second problem goes the other way: the explainability requirement may push lenders toward simpler models specifically because they are easier to defend not because they are fairer or safer. A monotonous scorecard is not automatically less discriminatory than a complex model. It is simply discriminatory in a way that is easier to see and defend in a hearing. Steelmanning focuses on the black box side for a second: a more flexible model adequately tested for disparate impacts on theproduction could in principle be more precise and no more discriminatory than the simple model and it is filed anyway because no one can write their adverse action letter with full confidence. Applicants who would have been approved under that model pay a real cost for a compliance requirement that was intended to protect people exactly like them

The third condition on which the entire framework is strained is speed. A four-second automated decision has to run all of its disparate impact testing drift monitoring and reason code generation within those four seconds in production for every application not just in a quarterly compliance review. The faster lenders push the decision the more of this will have to be pre-integrated into the model architecture rather than checked after the fact and that constraint shapes what is built long before any regulator analyzesthe result

How I Actually Use This

When I read about a lender implementing a new underwriting model the first question I ask isn't "how accurate is it?" It's "what does the adverse action letter look like and who built the pipeline that generates it." If a company's marketing leads with predictive lift and doesn't say anything about the explanation methodology I treat it as a gap not an oversight because in this industry the second part is not optional and a team that fails to mention it either hasn't resolved it yet or doesn't believe it.worth mentioning. It's not a great sign either

My reading is that the practical checklist for any lender using one of these models has four parts: an explanation method that produces genuine reasons rather than merely plausible reasons periodic testing of results across groups rather than a one-time audit of the input list a documented search for a less discriminatory alternative whenever a disparity appears and continuous monitoring of drift as the applicant population changes over time. If you skip any of the four you won't actually have a compatible model. You have a model thathas not yet been captured

I admit that the SHAP approximation problem seemed really awkward to me the first time I understood it properly. Going in I assumed that "the model can explain itself" was a solved technical problem and that the remaining fight was purely about whether lenders wanted to bother. It's not solved. It's a real unresolved tension between two things that everyone claims to want at the same time: a model that predicts as well as possible and a model that can give a stranger a short true stable answer to the "forwhat." I don't think you can have the maximum amount of both at the same time. You're picking a point on a curve whether you say it out loud or not

The way I would actually use this if I were evaluating a lender or fintech from the outside: I wouldn't be impressed with just a headline accuracy number and I'd like to know if the explanation channel was built alongside the model or coupled with it after the model already existed. Added after that is a red flag as it usually means that the explanation was never part of the design constraint and might be the least reliable part of the entire system. This is not investment advice or a recommendation for or againstfrom any lender or company mentioned here. It's a way of reading a disclosure that I've found really helpful and it's the lens I wish a hiring manager knew they already had before they walked into a room to defend it

The Bottom Line

Automated underwriting is typically more precise than a traditional scorecard and the reason is structural: it is allowed to use variables and interactions that a linear model cannot touch. That same flexibility is what makes it difficult to explain and the law does not consider "difficult to explain" as an acceptable response. A rejected applicant is entitled to specific reasons discriminatory results are prohibited regardless of intent and a reason code must survive contact with both requirements at once. The scorecard version of a reason code isThis makes the explanation process as much of an engineering problem as the prediction itself and in consumer lending it's the part you really have to put up with when a real person asks a real question about a decision you took four seconds to make