Failed to recover missing trie node

[NoahStahl]

Description
Encountered the below exception just now. Seems to have resolved after restarting Nethermind, but wanted to check if this is expected or potentially a bug. I happened to notice a few minutes after it began, so I'm not sure if this would have recovered on its own?

Time UTC: 2025-01-05 23:47:24.4789

Exception repeated for ~6 minutes until Nethermind was restarted.

Logs of exception (can provide more as needed)

2025-01-05 17:47:24.4789|INFO|Merge.Plugin.Handlers.NewPayloadHandler|165|Received New Block:  21561646 (0x9f5881...87249b), Extra Data: Titan (titanbuilder.xyz) 
2025-01-05 17:47:24.5068|WARN|Synchronization.Trie.TrieNodeRecovery`1[[System.Collections.Generic.IReadOnlyList`1[[Core.Crypto.Hash256, Core, Version=1.30.2.0, Culture=neutral, PublicKeyToken=null]], System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]|35|Missing trie node 0x214d1131649b1f3ab5ca773533aa5e52c9954ded1e9641aef842d5a8babe6e7b, trying to recover from network 
2025-01-05 17:47:24.5068|WARN|Synchronization.Trie.TrieNodeRecovery`1[[System.Collections.Generic.IReadOnlyList`1[[Core.Crypto.Hash256, Core, Version=1.30.2.0, Culture=neutral, PublicKeyToken=null]], System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]|35|Failed to recover missing trie node 
2025-01-05 17:47:24.5068|WARN|Consensus.Processing.BlockProcessor|35|Encountered exception Nethermind.Trie.MissingTrieNodeException: Node 0x214d1131649b1f3ab5ca773533aa5e52c9954ded1e9641aef842d5a8babe6e7b is missing from the DB
 ---> Nethermind.Trie.TrieNodeException: Failed to load root hash 0x214d1131649b1f3ab5ca773533aa5e52c9954ded1e9641aef842d5a8babe6e7b while loading key 036b6384b5eca791c62761152d0c79bb0604c104a5fb6f4eb0703f3154bb3db0.
Node 0x214d1131649b1f3ab5ca773533aa5e52c9954ded1e9641aef842d5a8babe6e7b is missing from the DB
   at Nethermind.Synchronization.Trie.HealingTrieStore.LoadRlp(Hash256 address, TreePath& path, Hash256 keccak, ReadFlags readFlags)
   at Nethermind.Trie.PreCachedTrieStore.LoadRlp(Hash256 address, TreePath& path, Hash256 hash, ReadFlags flags)
   at Nethermind.Trie.PatriciaTree.ResolveNode(TrieNode node, TraverseContext& traverseContext, TreePath& path)
   --- End of inner exception stack trace ---
   at Nethermind.Trie.PatriciaTree.ResolveNode(TrieNode node, TraverseContext& traverseContext, TreePath& path)
   at Nethermind.Trie.PatriciaTree.Get(ReadOnlySpan`1 rawKey, Hash256 rootHash)
   at Nethermind.State.PersistentStorageProvider.LoadFromTreeStorage(StorageCell storageCell)
   at Nethermind.State.PersistentStorageProvider.LoadFromTreeReadPreWarmCache(StorageCell& storageCell)
   at Nethermind.Evm.VirtualMachine`1.InstructionSLoad[TTracingInstructions,TTracingStorage](EvmState vmState, EvmStack`1& stack, Int64& gasAvailable, IReleaseSpec spec)
   at Nethermind.Evm.VirtualMachine`1.ExecuteCode[TTracingInstructions,TTracingRefunds,TTracingStorage](EvmState vmState, EvmStack`1& stack, Int64 gasAvailable, IReleaseSpec spec)
   at Nethermind.Evm.VirtualMachine`1.Run[TTracingActions](EvmState state, IWorldState worldState, ITxTracer txTracer)
   at Nethermind.Evm.TransactionProcessing.TransactionProcessorBase.ExecuteEvmCall(Transaction tx, BlockHeader header, IReleaseSpec spec, ITxTracer tracer, ExecutionOptions opts, Int32 delegationRefunds, StackAccessTracker& accessedItems, Int64& gasAvailable, ExecutionEnvironment& env, TransactionSubstate& substate, Int64& spentGas, Byte& statusCode)
   at Nethermind.Evm.TransactionProcessing.TransactionProcessorBase.Execute(Transaction tx, BlockExecutionContext& blCtx, ITxTracer tracer, ExecutionOptions opts)
   at Nethermind.Consensus.Processing.TransactionProcessorAdapterExtensions.ProcessTransaction(ITransactionProcessorAdapter transactionProcessor, BlockExecutionContext& blkCtx, Transaction currentTx, BlockReceiptsTracer receiptsTracer, ProcessingOptions processingOptions, IWorldState stateProvider)
   at Nethermind.Consensus.Processing.BlockProcessor.ProcessBlock(Block block, IBlockTracer blockTracer, ProcessingOptions options)
   at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) while processing blocks.

Steps to Reproduce
One-off occurrence so far, no repro as such.

Screenshots
If applicable, please include screenshots to help illustrate the problem.

Desktop (please complete the following information):
Please provide the following information regarding your setup:

• Operating System: Windows 10
• Version: 1.30.2
• Installation Method: GitHub release windows-x64 binaries
• Consensus Client: Teku 24.12.1

[kamilchodola]

Hi, thanks for report!

It is definitely a bug - can you tell me if this was a freshly synced DB on this version or upgrade from any previous one?

[NoahStahl]

It was upgraded from 1.29.1 on January 3rd (2 days before this incident).