e-KYC: Demystifying FAR and FRR
Updated: Dec 4, 2020
Appraisals - a dirty word in the world of managing people. Human beings hate having to sit through having their progress evaluated, slip-ups pointed out, and goals recalibrated for the coming cycle. Managers equally hate the mental and administrative burden that goes with preparing for the entire cycle. Even in a high performing team destined for stellar reviews - everyone still seems to approach that time of the year with a steady amount of dread.
Well, now the machines have finally taken over (does that evoke any 80s Terminator vibes?). Everything is automated, you have ‘artificial intelligence’, ‘machine learning’, and ‘big data’ on your side. Hurrah! You exclaim to yourself, no more appraisals, right?
Not so fast, buddy!
Enter the appraisal process for the machine.
If you work in anything remotely close to the disciplines of compliance, customer due diligence, or anti-money laundering, you'd have noticed terms like FAR and FRR being thrown around a lot lately. It's not an entirely new concept, as back in the old days when humans were processing applications, we just called it "someone made a mistake" or occasionally tossed around the term quality metrics.
But now we have to measure the same for the machine, and at scale. Why would you do this? Simply because organizations need metrics to ensure they're on the right track. Your business's longevity is on the line, and measuring the effectiveness of your electronic KYC (e-KYC) program should be the primary goal of your Quality Assurance (QA) function.
In some industries your regulators will require that you formally have this QA function set up and routinely report the results back to them - that is, if you plan to stay both in business and in their good books. But mostly, the fact is that regulators love FAR and FRR as metrics for e-KYC, and if you were to have to drill down your QA reporting to only one metric, FAR should be it.
So what exactly is FAR and FRR?
False Acceptance Rate (FAR) is a measurement of how many times your system messed up big time. A false acceptance happens when the system should have declined a user's application but instead approved it. It means that now either a malicious criminal or a genuine customer tagged to a false/fictitious identity is loose in your system. While some error is inevitable in any process, neither of these situations is a good thing. Any business involved with moving money, be it eMoney, investments, or remittance should try to keep it's FAR as low as possible. Regulators seldom provide specific guidance, but when they do the number seems to hover below 5%.
The False Rejection Rate (FRR) on the other hand is more of an opportunity cost to the business rather than a regulatory issue. The system has simply declined to a customer that actually should have had their application approved. Sometimes when fine-tuning the sensitivity of the e-KYC tool, it's necessary to sacrifice FRR for the sake of preventing a FAR catastrophe. Falsely rejected customers can, after all, just get an automatic do-over that often fixes the problem.
You calculate the FAR and FRR of your system using the formula below, expressed as a percentage:
FAR = number of false positives ÷ (number of false positives + number of true negatives) x 100
FRR = number of false negatives ÷ (number of false negatives + number of true positives) x 100
It would be impossible to scale a process if you test every application, plus this defeats the purpose of automating it in the first place. Therefore, it is best practice to calculate FAR and FRR off a sample. The main consideration when sampling is ensuring, as the regulators often say, that the samples are "random, unbiased and representative". In short, ask your data team to pull up a stratified random sample based on your most recent customer portfolio.
So I've started measuring, now what?
The goal is to bring those numbers down to as close to zero as possible. No system is ever perfect and if you did somehow get it down to zero, you should now instead be worried about how much you're choking your sales funnel. A constant endeavor to bring the number to an acceptable level and keep it consistently there should be good enough to convince all your stakeholders that you know what you're doing. Here are some ways to do just that.
Use multiple data points in your checks
There are three basic authentication categories you can use in these checks:
A unique identifier (e.g. identity document)
A biometric characteristic (e.g. thumbprint or face)
Some personal information (e.g. personal information marching, or challenge questions)
Ceteris paribus, a solution that is built on multiple factors will always be more robust than any single factor system. Most modern e-KYC systems take care of the first two points, but only the best can make use of "personal information" because it's the hardest to get a hold of reliably. Examples of these sources of data are government maintained national digital Identities programs like India's Aadhaar or Singapore's SingPass. Also, some verifications run through industry populated databases like Thailand's NDID or Malaysia's CTOS. The system can make comparisons between data input or use challenge questions to ensure that the user is exactly who they say they are.
Put in a little "Human Learning"
Machine learning is great and all, but not all of us have access to such tools. But that's okay because before there was Artificial Intelligence, there was Actual Intelligence and hopefully your team has lots of that to spare. Put a little time into finding patterns in the data. Specific errors that only occur for a certain segment of users or perhaps during specific document checks. Once highlighted, you can have your team document findings to check for consistency.
Adjust parameters and thresholds
Parameters for each test can be set, for example, the spelling of names or usage of nationalities vs country of residence. When dealing with names and dates, an important concept is fuzziness or the variation allowed between Steven and Stephen - Check how many percent you're using to start with. Perhaps you're casting too wide a net. After analyzing the breakdown of the cases caught within the False Acceptance and Rejection cases, make adjustments before running the tests again.
Run, and then re-run your tests
While time-consuming, you should always run all your tests multiple times. Once for each setting, and then another two repeat tests for good measure once you think you've found the version that gives you the best balance. Remember that no testing script is perfect, but the key to a responsible program is getting the best possible results and demonstrating continuous improvement.
Have effective controls and document them well
A QA program ensures that you are continuously trying to improve what you've designed. You're looking at ensuring you catch the small percentage of errors when they happen and not just during the design and testing phase. You also need to ensure that you have a remediation program in place to ensure that when you detect something wrong you aren't just keeping them as statistics to deal with 'one fine day' but have a process to weed out the bad actors the moment you find them. Documenting this from both a journey perspective as well as the individual cases funneled into your remediation process will help you prove that your shiny new e-KYC process will stand the test of time, and survive a 'kicking of the tires' when your regulator comes knocking on your door every other year.
Still not sure how to proceed?
All this can be a lot to process, and even more difficult to implement. Dicorm helps companies achieve these kinds of optimization through a structured approach, leaving you with more time to concentrate on making your product more attractive to your customers. Contact us today to have a no-obligation chat about your screening setup and how we can optimize your e-KYC setup.