Network Performance Report: July 2023

Network highlights since the October 2022 report

Network upgrades to better handle high traffic and demand: In 2022, the Solana network experienced several slowdowns or outages during times of high activity, such as during large NFT mints. In response, core developers rolled out network upgrades including QUIC, Stake Weighted QoS, and localized fee markets. Since the rollout of these upgrades, the network has performed incredibly well in times of high stress (such as the MadLads NFT mint). Developers are also continuing to test and develop other network upgrades and initiatives, including increasing maximum transaction sizes (currently limited to 1232 bytes) and simplifying the voting logic, which reduces the overall amount of data needed to be transmitted and stored.

• QUIC TPU: QUIC is a network protocol designed by Google for fast asynchronous communication but with sessions and flow control. QUIC can be used to limit traffic from any one actor, allowing the network to focus on genuine valid transactions. QUIC has been added to the port(s) that ingest user transactions. 
• Stake-weighted QoS: Stake-weighting is used to efficiently allocate leader network bandwidth. Instead of indiscriminately accepting transactions on a first-come-first-served basis, it gives access to transit information, so that a node with 0.5% stake will have the right to transmit at least 0.5% of the packets to the leader. This can make the network more resilient by disincentivizing spam or malicious actors. Stake-weighted QoS has been added to QUIC TPU ports. 
• Localized fee markets: Fee markets provide a way for a user to add an additional fee to their transactions to express urgency in comparison to other transactions. Priority fees are calculated based on the amount of computing resources that a transaction is expected to require. For example, a simple token transfer would require a lower total priority fee than an NFT mint that expresses the same level of urgency. This process makes markets more efficient by allowing users to express urgency and prioritize transactions.

Improved upgrade process: In response to the Feb. 25 outage (more on this below), Solana Labs CEO Anatoly Yakovenko published a blog post suggesting new processes for software release rollouts in order to make the upgrade process more resilient in the face of new upgrades as the network scales. These include bringing additional external developers and auditors into the release process to test and find exploits, improving the server restart process, emphasizing focus on network stability, and forming an adversarial team. Several of these proposals have since gone into effect, including:

• Comprehensive upgrade process on testnet: In order to simulate an upgrade process, testnet is now first downgraded to the previous software version before simulating the software upgrade process that will be rolled out on mainnet. Before this, testnet was a more minor version of mainnet at the time of a mainnet upgrade. 
• Optimizations to tooling used for manual restarts: For example, an initial design is in place that will automate some of the more mechanical steps (SIMD0046). 
• Adversarial tests are being run against both private clusters and in testnet: These tests are running in an adhoc fashion today as core engineers work to build a more formal testing schedule. Relatedly, the Solana Labs validator reference client now has an adversarial port, and core labs engineers are adding the ability to configure scenarios that can be used to build adversarial tests. 

Smooth release of 1.14.17: Version 1.14 was considered officially adopted by May 22, the date when over 66.6% of total stake had upgraded to 1.14. This was the first significant network upgrade since the Feb. 25 outage and the rollout of the improved upgrade progress, described above. As of May 30, version 1.14 of the Solana validator client was adopted by 97.4% of stake on the Solana network. 

Compressed NFTs and state compression: After many phases of development, adoption, and rollout, compressed NFTs are now live on Solana mainnet-beta. Compressed NFTs are 2,400-24,000x cheaper than their uncompressed counterparts. Minting 100 million NFTs on Solana now costs as little as 50 SOL. This new technical structure for NFTs, called “state compression”, unlocks a host of new use cases and scales the potential impact of the blockchain by making it accessible to mint NFTs for use cases such as sending messages or enhancing customer loyalty programs. State compression was developed for NFTs, but can be used for other applications. Put simply, state compression involves storing some data on the blockchain that would otherwise be stored locally on every machine. 

Smooth Helium migration: In April 2023, the Helium network successfully completed migration to the Solana network, in a process that was relatively seamless for Helium network participants and caused no serious disruption to the Solana network. Helium’s migration took advantage of state compression to mint NFTs more efficiently and cheaper than would have been possible otherwise. More details here.

Additional Firedancer progress: Firedancer, a Solana validator client being developed by Jump Crypto, is adding and testing more components. 

• QUIC implementation demo: Firedancer did a recent demo of high performance QUIC implementation, which should lead to more efficient data transactions. 
• Reed-Solomon Erasure Coding released: Turbine is a mechanism in a validator that is responsible for distributing “shreds” (pieces of a block) to other validators in the cluster. Reed-Solomon is an error correction coding scheme, which helps accurately encode data in these shreds before they are transmitted to other validators. 

Tinydancer: Tinydancer, the first light client on Solana, was developed as part of the Solana Foundation’s Grizzlython Hackathon. Light clients are software clients that can run locally on simple devices like laptops, connecting to full nodes in the network to verify the state of the ledger and check for invalid state transitions.

Real-time emissions monitoring: In April 2023, Solana became the first major smart-contracts blockchain to have its carbon footprint measured in real time. TryCarbonara, an independent startup, launched www.solanaclimate.com to track the network’s impact. 

Network challenges since October 2022 

The Solana network experienced one outage since the last network performance report in October 2022. No user funds were lost in this incident. 

Below, we review the outage and what is being done to address the outage and its causes. 

February 2023 outage: On Feb. 25, Mainnet Beta began to experience long block finalization times. After investigation, the cause was traced back to several services on the network running custom block-forwarding software that inadvertently transmitted a huge amount of data, equivalent to several orders of magnitude larger than a normal block. The network’s de-duplication logic was unable to cope with this, overwhelming the Turbine protocol and significantly degrading the network. After diagnosing this issue, core engineers created enhancements to the de-duplication logic, which are now in place as of Solana Labs validators client v1.13.7 and v1.14.17. This will also be mitigated by longer term Solana protocol design to replace all UDP-based networking protocols with QUIC, which will be better at enforcing the constraints in Turbine. Read the full outage report here. 

Several proposed network upgrades have been implemented or are being rolled out to address the causes of this outage, specifically the implementation of QUIC, fee markets, and stake-weighted Quality of Service (QoS).

Upcoming initiatives 

Solana core developers have been working on a number of new network upgrades aimed at strengthening the network in the face of massive user growth and adoption. 

These Solana Improvement Documents, or SIMD(s), are proposed design documents for how to make changes to the network that require coordination across multiple core development teams. One recently accepted SIMD is SIMD-33: Timely Vote Credits, which are awarded based on the number of slots voted on and awarded more for lower latency votes. This is intended to reduce block finalization times and disincentivize intentionally delayed voting.

There’s some network upgrades that are included with the latest release, but the features are not activated just yet. Turbine improvements is a 1.14 feature that greatly improves latency on the network. There’s a new Deactivate Delinquent Stake instruction that allows anyone to undelegate stake on the network that is delinquent for five or more epochs, lowering skip rate and improving performance. (Note: the undelegated stake accounts remain in the original staker’s control and are free to redelegate to an activate validator or withdraw their funds). 

Users can continue to monitor the performance of the Solana network and use reports like this one to help the community track its development over time.