When CPUs Hit 100%: Hard-Earned Lessons from a Multichain Indexer Outage

At Fystack, where we build blockchain wallet infrastructure, our transaction indexer needs to process over 100 million transactions per day across multiple blockchains, including Ethereum, BNB, Polygon, Tron, and Solana.

Our initial indexing logic was straightforward:
We maintain a database of wallet addresses generated for our users. For each block on the blockchain, we parse the raw transaction data and check if the destination address exists in our database. If it does, we process the transaction; otherwise, we ignore it.

Handling EVM chains alone was manageable, but everything changed when we integrated Solana, one of the most active chains, with huge transaction volumes driven by meme coin crazes and projects like Bump.fun, averaging around 42 million transactions per day (~3,000 TPS).

Our servers, particularly the CPU, were constantly under stress, with utilization hovering between 80–90%. On peak days, CPU usage reached 100%, causing the server to become unresponsive and our customers repeatedly encounter 504 Gateway Timeout errors and started complaining.

The team had to temporarily suspend Solana processing and restart the server just to restore normal operations.

We knew it was time to optimize.
Our team quickly identified that Consul where we persist user wallet addresses, was becoming a bottleneck. Storing and querying every address there doesn’t scale well when transaction volume is high.

To handle this efficiently, we adopted a Bloom filter, a well-known probabilistic data structure also used by the Ethereum blockchain for log filtering. A Bloom filter answers a simple question: Could this item be in the set?

• If it says no, the item is definitely absent and can be skipped.
• If it says yes, the item might be present, and we perform a full check against our source of truth.

Because most transactions are unrelated to our platform, this lets us skip the vast majority of checks. Properly tuned, a Bloom filter can store millions of wallet addresses while consuming only a few MBs of memory.

Bloom Filter Process

To ensure efficiency and scalability, we utilize the Redis Bloom filter module. When a transaction arrives, we check if the destination address exists in the Bloom filter. If it does not, we skip the transaction; if it does, we proceed to verify it in persistent storage.

The team successfully refactored the system, added new logic, tested, and deployed the solution in under 2 days, allowing operations on Solana to resume as usual.

func (s *publicKeyStore) Exist(addressType enum.AddressType, publicKey string) bool {
	// First check the bloom filter.
	// If the bloom filter returns false, the key definitely doesn't exist.
	if !s.bloomFilter.Contains(publicKey, addressType) {
		return false
	}

	// Since bloom filters may have false positives, check the underlying KV store.
	v, err := s.kvstore.GetWithOptions(composeKey(addressType, publicKey), &infra.DefaultCacheOptions)
	if v == "" || err != nil {
		return false
	}

	return true
}

To make it works seamlessly, we periodically synchronize addresses from our PostgreSQL database to a Redis Bloom filter through a simple and efficient process.

Recent optimizations have significantly enhanced our indexer’s performance and resilience. It now maintains CPU utilization between 40-60%, even under high load, rarely reaching 100%.

We’ve open-sourced our multichain indexer, supporting Ethereum, EVM chains, and Tron. Check it out at github.com/fystack/multichain-indexer. Bitcoin and Solana support are planned for future development.

At Fystack, we are building open source scallable wallet infrastructure that users can self host and doens't need to rely on expensive custodians.

Check out our Github : https://github.com/fystack/