At TokenAnalyst we are experts at hosting, running, and extracting valuable information from blockchain nodes. Through our time working with Ethereum, we encountered some interesting quirks that wed love to share with the broader community. We work with the two top most-used Ethereum clients, namely Geth (go-ethereum), built mainly by the Ethereum Foundation, and Parity, built by Parity Technologies.

At TokenAnalyst we are experts at hosting, running, and extracting valuable information from blockchain nodes. Through our time working with Ethereum, we encountered some interesting quirks that we'd love to share with the broader community. We work with the two top most-used Ethereum clients, namely Geth (go-ethereum), built mainly by the Ethereum Foundation, and Parity, built by Parity Technologies.

The following quirks have been found by my coworkers Ankit, Sid and myself. If you have any questions regarding the following findings, we would be happy to help. There are not too many people trying to get data from nodes, but if they do, they'll inevitably encounter these quirks at some point. The following list is anecdotal, for further references links are supplied. The first two points are particularly important to know when you set up your own node.

1. Run SSD

Magnetic hard-drives are not optimal for running nodes; they are just too slow. SSD is a much better option. We started our full nodes on AWS t3.xlarge instances, having EBS storage attached. However, it turned out, that NVMe instance storage in RAID-0 is much more cost-effective, with a 30x improvement on I/O throughput when writing files.

2. Responsiveness

When Geth is not in sync with the latest block, the RPC interface is unresponsive, meaning you cannot query for e.g. blocks, transactions, receipts etc. This works in Parity.

3. No Error Response Standards

Even though there is a standard what kind of RPC methods and parameters exist. There are no standard error responses across nodes. Hence, if you want to write software that integrates with different clients you have to parse responses dependent on the client.

4. Response Fields not Consistent

The fields, r, s, v of a transaction are not consistent on different clients. They're different on Geth, Parity and Infura, see reference.

5. Parity vs. Geth

Running the latest versions of Geth and Parity on the same AWS t3.xlarge instance types with EBS storage, in the same regions, Geth is more out-of-sync than Parity. This is particularly unfortunate since Geth is unresponsive while syncing. The comparison was pre-1.8.19. After we updated to 1.8.19 we experienced better performance.

6. Typesafe

The JSON-RPC interface is not type-safe. That means, for data pipelines building on top of this data source, post-processing e.g. validation and typecasting is required. The "standard" on the wiki is not very enforcing.

7. Traces and Statediffs

For data, like internal transactions, state diffs, there is not an RPC standard yet. Geth is not supporting them yet, Parity has implemented them: https://wiki.parity.io/JSONRPC-trace-module.html. The state diff result format is a bit weird, and it's not fully consistent: see.

8. Data Bugs

There are two ways of retrieving the latest blocks. Through HTTP polling or a kept alive web socket connection. For both ways block re-organization (a wrong block is forwarded and later discarded) is not correctly implemented on Geth and Parity, see.

9. Smaller Differences

Multiple smaller differences between the interfaces of Geth and Parity: Uncle size is null in Parity, but does exist in Geth. Filter IDs on Geth and Parity are different. Geth assigns random IDs, and Parity uses an incremental counter.

10. In General

The consensus rules among clients have to match in order to make Ethereum work, but everything else, e.g. the exposed API, or RPC interface can be totally different. Take each client with a grain of salt.

To sum it up.

Through the process of setting up our infrastructure. We learned that when building fault-tolerant, highly reliable, and mission-critical application on top of Ethereum, it is not enough to rely on the functionality of specific Ethereum clients. An extremely deep understanding of how nodes work, what they need, and how to orchestrate them is required to ensure effectively zero downtime along the whole data pipeline.