The bug fix for the XRP Ledger’s Automated Market Maker (AMM feature) has been successfully integrated into the XRPL Mainnet as of April 11. This marks the resolution of an issue discovered on March 23 regarding an unexpected number of LPTokens in an AMM pool. Thanks to the collaboration with XRPL validators and community participants, we were able to work together to identify, diagnose and resolve the issue swiftly.
In my previous blog post, I provided an overview of the issue, immediate response, and next steps. As promised, this follow-up post will dive deeper into the details of the bug, its impact, the lessons learned, and the ongoing work to prevent similar issues from occurring in the future.
Background – Payment Engine and Liquidity
A major benefit of the XRPL is how features are composable, which enables flexibility and efficiency in payment processing. Through the XRPL’s payment engine, the AMM and built-in decentralized exchange (DEX) can both interact to provide liquidity for payments. Some payments, such as transferring XRP from one account to another, are straightforward. Others are more complex. For example, consider a scenario where a user wants to send USD and have the destination account receive EUR. In this case, the payment engine would use the USD to EUR order book to convert the funds. The process of specifying how currencies are meant to be converted is called a “payment path.”
The payment engine can use one or more paths to complete a payment. In the example, in addition to the USD to EUR order book, the sender could also specify a second path that uses an order book to convert from USD to XRP; then a second order book from XRP to EUR. Both payment paths may be used in the same payment, and the payment engine will always consume liquidity from the path that offers the most cost-effective option for the sender.
AMMs provide a new way to convert between assets that complements the DEX’s order books. Unlike offers, which have fixed exchange rates, the AMM exchange rate varies continuously as liquidity is consumed from the AMM pool according to a constant product formula (see XLS-30 for details). The payment engine needs a way to compare the discrete liquidity from an offer with the continuous liquidity from an AMM to provide the most cost-efficient path for payments.
To address this challenge, the payment engine creates synthetic offers backed by the AMM. From the payment engine’s point of view, these synthetic offers act the same way as users’ offers. In particular, consuming liquidity from these synthetic offers does not change the exchange rate. The size of synthetic offers depends on the specific payment paths. When there is only a single payment path, the engine can use the constant-product formula to infer the right-sized exchange rate and offer size. In the multi-path case, that size depends on the liquidity from other paths, so the payment engine iteratively generates synthetic offers of increasing size to utilize liquidity from the AMM pool.
The Bug
The bug occurred when a multi-path payment interacted with an AMM synthetic offer, and the synthetic offer required to fulfill the payment exceeded the AMM pool size. In this case, the payment engine attempted to use the single-path strategy to back-out the maximum liquidity available to swap against the pool. Although this condition was properly detected, there was an error on how the payment engine resized the synthetic offer to consume liquidity only up to the AMM size. Instead of using the actual swap rate, which represents the cost of fully swapping one side of an AMM pool, the new synthetic offer’s exchange rate was incorrectly set to the spot exchange rate of the AMM (that is, the rate it takes to execute an infinitesimally sized order). Consequently, an AMM operation that should have been expensive became relatively cheap, effectively violating the constant-product invariant of the pool. In this state, another user could deposit funds into the drained side of the pool, generating a large number of LP tokens and taking ownership of the pool. Now that the fix amendment is active, the team has opened a PR with a unit test demonstrating this behavior.
Impact
At the time of the bug report, there was only a single impacted AMM pool holding USD.rvYA and USD.rhub. Prior to the bug, the pool had $273 of USD.rhub and $3 of USD.rvYA. After a subsequent payment that caused the constant product rule to be violated, the pool had nearly $0 of USD.rhub and $16 of USD.rvYA. The payment was instructed to swap out $272.455 of USD.rhub by spending up to $50 of USD.rvYA, but was fully executed by spending only $13 of USD.rvYA. This left the pool in an unbalanced state, where a subsequent deposit in the pool enabled a single depositor to own nearly all the pool’s LP tokens. Since that time, a large AMMBid transaction burned enough LP tokens to restore the constant product invariant of the pool.
During the time after the release of the bug fix but before the amendment became enabled, there appears to only be one other AMM pool impacted by this bug, this time holding EUR.rhub and SeagullCash.rNHe. For this pool, an OfferCreate transaction, which automatically leverages auto-bridging to execute against available liquidity sources, was able to withdraw the entire €6.103 from the pool in exchange for 1 billion SeagullCash.rNHe. Due to the bug that fully drained one side, the pool is in a stuck state where assets cannot be added to or removed from the pool.
Timeline & Response
As a brief recap of events:
Bug Discovery: Early morning March 23, the Ripple dev team received a report from Tequ, a member of the XRPL community, regarding an unexpected number of LP Tokens in an AMM pool.
Issue Diagnosis: By the evening of March 23, the Ripple dev team, alongside community members Tequ and Orchestra, swiftly investigated the issue to identify the root cause and assessed the impact on AMM pools.
Fix Development and Testing: Once the bug was reproduced, the team quickly had a fix prepared, but worked through March 24 to add an additional invariant check to enforce the properties of AMM pools during transaction processing. This check took extended time to develop in order to validate the numerical sensitivity of the invariant across a broad range of liquidity scenarios.
Community Communication: We promptly communicated the issue to the XRPL community through X (formerly Twitter) and direct outreach to community members including AMM UIs, ensuring transparency and collaboration throughout the process.
Hotfix Release: On March 26, a hotfix PR was opened. It was merged on the morning of March 27.
Amendment Voting: The amendment for the bug fix was supported by XRPL validators and reached 80% consensus on March 28 and activated on April 11, showcasing the community’s commitment to the XRPL’s stability and security.
Lessons Learned and Fortifying the Future
The initial focus was on fixing the bug and mitigating the impact. Since releasing the fix, the team has spent significant time reflecting on how to prevent a similar issue from occurring in the future, and on how the community can best respond and manage incidents like this.
Invariants are key. Invariant checking is a safety feature of the XRP Ledger. It consists of a set of checks, separate from normal transaction processing, that guarantee that certain invariants hold true across all transactions and helps eliminate any adverse edge conditions or error case mishandling. The AMM feature has an obvious invariant from the constant product formula — the number of issued LP tokens should always be less than or equal to the product of the square root of the assets in the pool. Although the team had built an invariant check during development, it was inadvertently removed before release. This was a huge miss.
The team is taking several corrective actions – first, the missing invariant check was restored as part of the fix. Second, the team is also working to add additional invariants for other AMM transactions, and will initiate a workstream to review existing XRPL features and propose additional invariants that would make those features more robust. Going forward, the team will also propose updating the XLS and PR process with an explicit section on invariants, as well as a section to address any numerical sensitivity decisions (which was a proximate issue for this AMM bug).
Composability comes with complexity. As mentioned above, a major benefit of the XRPL is the way features like the DEX, AMM and payments compose and work together; there aren’t any other protocols that seamlessly integrate these types of features. But this also creates a large set of interactions and edge cases to test. Although the XRPL has a strong track record of performance and stability due to the existing software development and testing processes, there are always ways to improve. The team is finalizing a PR that significantly increases unit test coverage of the AMM feature, with a goal of 100% coverage for any new features added to the Ledger going forward. Leveraging the added invariant checks, the team will also start a workstream to build out context-aware fuzz testing to thoroughly stress the protocol in a controlled manner. We are excited to learn more about community proposals for a Canary Network, and are also iterating on the existing Bug Bounty program to ensure it has the right visibility and support for extensive testing in the test and dev networks in addition to the main network.
It takes a village. A huge thank you to the community members who responsibly disclosed the bug, the projects who quickly responded to limit further deposits into AMMs, the devs who worked around the clock on the fix, validator operators who voted in favor of the amendment fix, and the broader community for the support and patience during this process. These last few weeks should serve as a starting point for a playbook for future incidents, a foundation that the XRPL community can collectively develop and improve.
A major challenge in responding quickly was the 2 week window for approving amendments. The two week period is meant to give sufficient time for node operators to upgrade to the version of rippled that supports that amendment, for validator operators to review the changes to decide their vote, and also change their vote based on feedback from other operators and the community. For new features, this is an appropriate mechanism for the community to decide the best path forward for the ledger. For urgent bug fixes, 2 weeks is an eternity. The team will work on XLS proposals for alternative ways to deploy fixes, such as fast-track amendments that have an accelerated voting process. We look forward to hearing similar ideas and innovations from the community on ways to respond quickly when necessary.
As we move forward, Ripple will continue to monitor the AMM feature closely alongside the XRPL community to ensure its stability and reliability. As with any new functionality, we encourage the community to test and educate themselves on how it works. Together, we could work towards building a stronger, more resilient XRP Ledger that empowers innovation and drives the future of on-chain finance.
Discover more from Coursity
Subscribe to get the latest posts sent to your email.