Test your fraud risk strategy
Last updated: October 2, 2024
Testing is a critical part of maintaining and improving your fraud risk strategy for payments. There are several reasons why you should test fraud risk strategies regularly:
- Evolving fraud techniques – Fraudsters are constantly developing new techniques to exploit vulnerabilities in payment systems.
- Regulatory compliance – Many industries are subject to regulations and compliance standards related to payment security. Regular testing of fraud risk strategies helps you to meet these requirements and avoid potential penalties or legal consequences.
- Revenue optimization – An effective fraud risk strategy both reduces fraud losses as much as possible, and reduces the number of false positives that can disrupt legitimate transactions. Regular testing enables organizations to balance these needs.
- Adjustable number of manual reviews – You can add, remove, or change your flag rules to adjust the number of payments that are flagged, which your operations team needs to review.
Checkout.com’s Fraud Detection solution offers two different ways you can test your fraud risk strategy:
When you carry out shadow testing, you run the test strategy at the same time as the existing strategy. However, you do not take any action based on its results.
You can implement and iterate shadow tests regularly, which enables you to:
- Observe a fraud risk strategy's performance in real-time, without interfering with actual transactions (whether you're deploying a completely new strategy or changing an existing strategy).
- Identify any potential issues.
- Fine-tune rules.
- Assess the strategy's effectiveness before full implementation.
Information
A shadow test relies on real transaction data, so it can take between 24 hours and three months for patterns to become clear. For example, if you had extra sales in April, then you might want to consider this when analyzing the backtest results, as it may not be representative of future traffic.
The Forward play outcome comparison tab shows the difference between the live and test outcomes for each processing decision. The change in reported fraud numbers displays in red for each outcome. You can also hover your cursor over each part of the bar to display a breakdown of the numbers.
To see this data:
- Sign in to the Dashboard.
- Go to the Payments > Fraud > Strategy section.
- Scroll down to the Outcome comparison section.
- Open the Shadow testing tab.
Pro only
Information
If you have any feedback about Checkout.com's backtesting functionality, email us at [email protected].
Backtesting enables you to assess the performance of a new fraud risk strategy by simulating how it would have performed in comparison to your current live strategy, using your historical transactions. This analysis helps you measure the strategy's effectiveness, detection rates, false positive rates, and potential impact on historical fraud cases. Backtests are much faster to perform than shadow tests — usually, it only takes minutes to simulate the data, whereas a shadow test can take between 24 hours and three months.
Backtesting analytics include:
- Key performance indicators (KPIs), which display the difference in key metrics between your live and test strategy
- An overview of the Pre-Auth and Post-Auth processing decision changes
- The Amount and number of Transactions associated with each processing decision (Decline, Void, 3DS, Flag, Accept, Capture)
You can use these analytics to determine whether your new risk strategy will ultimately improve profitability. You may also wish to consider these analytics alongside what you already know about your business. For example, how much a fraudulently obtained chargeback costs you, your existing business patterns, and so on.
Backtesting takes all transactions in the date range you have selected, and runs them against both your test strategy and your current live strategy. Checkout.com doesn’t compare results with the historical outcome, as your live strategy could have changed multiple times over the selected date range.
There are also some other factors you should consider when implementing backtesting.
Decline lists, trust lists, and rule-leveraging custom lists all use the list values as of the current date on which the test occurs. They do not use the lists as they originally existed. This ensures your test strategy and your live strategy are compared as fairly as possible.
Velocities set frequency thresholds for certain attributes that appear in your patterns over a period of time. All velocities are treated as starting from 0, from the beginning of the date range — not their actual historic values as of the transaction occurring.
Machine learning scores are not recalculated at the time of the backtesting. Instead, historical scores are reused.
In some situations, Checkout.com may not know whether a transaction that was historically declined, or that bypassed 3D Secure (3DS), will pass or fail 3DS. If we know the outcome, we’ll apply it. If we don’t, we’ll then assume that the transaction will pass successfully if sent to 3DS.
| Historical pre-3DS outcome | Passes 3DS? |
|---|---|
Transaction attempted | |
Transaction passed 3DS | |
Transaction failed 3DS | |
Transaction declined |
Similarly, a transaction that had historically been declined by the issuer (that is, failed at authorization) would be declined between pre-auth and post-auth.
However, in cases where we don’t know, we’ll assume the authorization would succeed. For example, for a transaction that was historically declined but is now accepted. With backtesting, we assume that the transaction wouldn’t have been blocked, and therefore we need to make assumptions about what happened to it.
Both backtesting and shadow testing can currently only be started from the client level account.
Backtesting will also consider your live and draft strategies at the entity level when evaluating outcomes and displaying results. However, for shadow testing, only the live results will take into account entity-level rules and strategies.
Comparing historical backtests enables you to gain insights to inform your strategy development.
To see your historical backtests:
- Sign in to the Dashboard.
- Go to the Payments > Fraud > Strategy section.
- Scroll down to the Outcome comparison section.
- Open the Backtesting tab.