Infura ‘hard fork’ causes debate about update mechanism

  • Ethereum Core developers have agreed to improve the update release mechanism.
  • The Optimistic Ethereum team has caused the failure of Infura by triggering a critical synchronization bug on Geth client.

On November 11, an “unannounced hard fork” on Ethereum impacted the operation of Infura and most exchanges that relied on its back-end services. Although ETH’s price was not affected and reached a new annual high of $476 on November 12, the event was met with harsh criticism from the community.

The chief developer of Blockchair, Nikita Zhavoronkov, offered one of the first explanations for the failure and declared that it “should not be underestimated”. Zhavoronkov believes that the incident is of great importance to Ethereum and one of the most important that it has faced since “the DAO debacle four years ago”.

A later security report, published via Twitter by Ethereum Core developer, Péter Szilágyi, gave a more detailed view on the bug that specifically affected the Geth client. According to the report, on October 24th the Geth team was notified of a Denial of Service (DoS) bug found in the Go programming language library:

The DoS issue can be used to crash all Geth nodes during block processing, the effects of which would be that a major part of the Ethereum network went offline.

The bug was fixed with an update, but versions prior to Geth 1.9.19 remained vulnerable. Infura and other users were not notified and continued to use vulnerable versions of the Geth client. Thus, when on November 10th the Optimistic Ethereum (OE) team decided to “test a bug” they found in the Ethereum Virtual Machine resulted in 30 blocks to be lost in a chain split, as an OE member stated:

I would like to apologize to the community for inadvertently causing some nodes to fall out of sync on Ethereum last night (…). we decided to test the bug and see what would happen. This was boneheaded in hindsight – we didn’t realize the impact of the few nodes that were not upgraded.

Ethereum’s developers discuss Geth’s bug

As mentioned earlier, the unannounced hard fork has caused great controversy in the crypto community. Users and developers are discussing the responsibilities of both parties. In a recent call by the Ethereum core developers, the issue was also raised. Tim Beiko reported on the call and started with Szilágyi’s comments:

(…) this is not the first time geth has fixed a consensus issue silently. They decide whether to advertise the fix based on how likely it is for someone to exploit the bug. In this case, they decided to keep it hidden.

According to the developer, exploiting the vulnerability would have been “too easy” to make public. However, he agreed that after the update, the developers of Geth “should have reported that an earlier version had a problem”. The call therefore discussed the possibility of reporting future problems at least one month after they have been fixed, in order to bring about an improvement. Szilágyi added:

Everytime they disclose a bug, it creates a risk for the network and everyone who doesn’t upgrade.

On his part, James Prestwich has come up with the possibility of creating a “private list” to notify members of the list of similar bugs. However, creating such a mechanism presents several challenges: increasing centralization on Ethereum, determining who should be on the list and giving “competitive advantage” to added members.

In that sense, Szilágyi maintained its position that the priority should be the security of Ethereum. Therefore, he asked the community for suggestions on how to improve the release of updates to the public.