Incident update: Protocol paused out of an abundance of caution; unpaused after investigation determined no ETH at risk.

• Severity: 0
• Status: Resolved, with monitoring in place to ensure ongoing stability
• Risks:
  • No risk to user ETH
  • Reduced protocol efficiency for a limited scope of time while the risk was determined, during which time a percentage of staked ETH was not actively receiving ETH network rewards
• Remediations:
  • The corrective action plan for remediating the root cause of this incident includes bug patching, an audit of the protocol’s offchain software components, and an improvement plan for the protocol’s monitoring and alert systems.
  • The Liquid Foundation reimbursed the protocol for the estimated potential ETH network rewards that may have been missed during the time validators were exited, in addition to reimbursing the protocol’s participants (e.g. Platforms, Node Operators, etc.) for the estimated missed Protocol Service Fees that would have been minted on those ETH network rewards.
  • An accounting error in the calculation of Node Operator Service Fee distributions, identified by the exiting of validator keys, was remediated.

• During an investigation of unexpected validator exits (related to unexpected exit daemon behavior) on January 30, 2024, the Liquid Collective protocol was paused out of an abundance of caution.
• While assessing the cause of the incident, the protection of user funds was prioritized over protocol liveness in a conservative approach.
• Following an investigation, it was determined there was no security vulnerability, slashing risk, or user ETH at risk of loss. The protocol's infrastructure and Node Operator infrastructure was not compromised. The protocol was then unpaused.
• A root cause analysis of the incident found that the protocol’s exit daemon, which is responsible for listening to onchain events and, if instructed, requests exits from Node Operators, experienced two bugs.
• All Liquid Collective Node Operators successfully deployed v0.17.0 of the exit daemon software, including two patches to remediate the cause of this incident.
• As of April 12, 2024, this incident and the bugs identified in it have been fully remediated, including reimbursing the protocol’s participants for potential ETH network rewards that were not received during the time that the protocol was paused.

Severity classification

This protocol incident was classified with a severity of 0 as no ETH was at risk of loss, and there was no slashing incident.

Incident

A Liquid Collective Node Operator unexpectedly exited validators from Ethereum, triggering an investigation. It was determined that the cause was due to two bugs, isolated to the offchain operations of Liquid Collective’s exit daemon software.

Out of an abundance of caution, the protocol was paused while investigation determined that no user ETH was at risk, and that no malicious activity had taken place. The protocol was then unpaused via a coordinated action of The Liquid Foundation’s executive governor multisig.

Response

As of January 31, 2024:

• A temporary increase of the protocol’s variation safeguard was prepared as a cautionary step to ensure the exited ETH could be seamlessly absorbed by the Liquid Collective protocol and programmatically staked again. After confirming new onchain data on January 31, and running simulations, the temporary increase of the protocol’s variation safeguard was found to potentially not be necessary for the efficient re-entry of the exited ETH to the Liquid Collective protocol. The temporary increase never became necessary to deploy.
• Investigations into the bugs in the exit daemon kicked off in collaboration with Liquid Collective’s Node Operators and Service Providers, to ensure there would be no further impact to the protocol’s expected function.

As of February 9, 2024:

• All Liquid Collective Node Operators successfully deployed v0.17.0 of the exit daemon software, including two patches to remediate the cause of this incident. The protocol was fully functioning as expected at this time.
• Active monitoring of the protocol continued to ensure ongoing stability while a root cause analysis was conducted and further remediations considered.

As of April 12, 2024:

• In addition to the successful deployment of v0.17.0 of the exit daemon software patching the bugs that caused this incident, a complete corrective action plan for remediating this incident, including an audit of the protocol’s offchain software components and an improvement plan for the protocol’s monitoring and alert systems, was put in place.
• The Liquid Foundation reimbursed the protocol for 5.8 ETH, the estimated amount of ETH network rewards which may have been missed during the time that validators were exited due to this incident.
• The Liquid Foundation reimbursed the protocol participants (including Platforms and Node Operators) for the estimated 0.84 LsETH in missed Protocol Service Fees that would have been minted on those estimated missed ETH network rewards.
• A previous accounting error calculating the Node Operator funded keys count was identified: the error was remediated and the distribution of the Node Operator Protocol Service Fees was revised.

Root cause overview

The Liquid Collective protocol’s exit daemon, an offchain software component responsible for listening to onchain events and, if instructed, requesting validator exits from Liquid Collective’s Node Operators, experienced two bugs that resulted in validators being exited from the protocol that should not have been.

• The caching bug: The exit daemon did not correctly maintain the order of its validators between runs in its cache, which resulted in incorrectly requesting the exit of too many validators.
• Exit condition bug: The above caching bug had the effect of making a loop in code, responsible for building a list of validator keys to send to a Node Operator to request their exit, which couldn't meet its exit condition, resulting in the computed list including the entire list of available validator keys.

Corrective actions for these bugs included:

• Patched bug: Node Operators pushed fixes to the software components as part of v0.17.0
• An audit of Liquid Collective's offchain components has been scheduled
• A workstream is underway to improve the Liquid Collective protocol's observability and to improve service provider response time

Liquid Collective is dedicated to maintaining high security standards. We recognize the importance of protecting participants' security when using the protocol, and understand that security is paramount to maintaining participants' trust.

You can view the protocol’s security resources on Liquid Collective’s Diligence page.