Polygon recently faced a network disruption due to a software bug that temporarily disrupted milestone finality and caused some nodes to go offline. The incident began when a bug affecting Bor, Polygon’s block producer, and Erigon, its data access layer, caused disruptions across the ecosystem. While the blockchain continued to produce blocks, some Remote Procedure Call (RPC) services and validators halted, leading providers to rewind to the last finalized block and resynchronize.
To address the issue, Polygon engineers executed an emergency hard fork that resolved the finality bug. The bug was preventing node progress under certain configurations, but restarting affected nodes helped resolve the issue for several validators and RPC providers. Polygonscan, the network’s block explorer, also experienced issues due to a faulty milestone produced by Heimdall, the project’s consensus engine. However, Polygon worked with Polygonscan to switch to functioning nodes that had not halted, ensuring that block production remained live.
The root cause of the problem was traced back to Heimdall’s finality gadget, which was responsible for producing milestones every few seconds. Checkpoints to Ethereum were still being posted every 20 minutes, but milestones for faster, deterministic finality were not being processed, causing delays in local fast finality. To fix the issue, Polygon rolled out emergency updates for Bor and Heimdall, including a hard fork that deleted the faulty milestone and purged it from node databases.
After successfully completing the hard fork, Polygon confirmed that milestones and checkpoints were once again finalizing normally. Sandeep Nailwal, Polygon Foundation co-founder and CEO, explained that the disruption originated from a faulty milestone proposal that pushed some Bor nodes onto divergent forks, causing short-term instability. He emphasized that while checkpoint finality on Ethereum remained active, local milestone finality was delayed.
Despite the temporary setback, the network has resumed normal operations, with blocks, milestones, and checkpoints processing in real-time. However, the incident raised questions about Polygon’s reliability, especially as the total value locked on the network has decreased significantly from its peak in 2021. The chain’s native token, POL, has also experienced a decline in value.
In conclusion, the recent network disruption on Polygon highlights the challenges of scaling proof-of-stake chains and the importance of continuous monitoring and upgrades to ensure network stability. Polygon’s response to the incident demonstrates its commitment to resolving issues promptly and improving the resilience of its network.