[filebeat] Critical memory leak of StandardMeter in broker.Close(), the element in the global arbiter map not released

[panbooks]

• Version: 8.17.0
• Operating System: CentOS Linux 7

https://github.com/elastic/beats/blob/v8.17.0/libbeat/outputs/kafka/config.go#L309-L315

If adding adapter.GoMetricsNilify in newSaramaConfig when creating a broker, the object of StandardMeter in arbiter map will not be released when calling broker.Close().
In Broker.Open(), registerMeter() will create a new StandardMeter, and save it to the global arbiter map (in go-metric lib).
But the adapter.GoMetricsNilify option changes the StandardMeter to NilMeter and save it to MetricRegistry, while the StandardMeter object has already created and saved in the global map arbiter. So the original StandardMeter object will not be deleted when calling MetricRegister.Unregister or UnregisterAll.

In our production environment, through mem profile, the inuse objects and allocate objects are the same which reaches highly to 1.4 million which takes nearly 400M space, resulting in OOM when the memory is limited.

Here is the benchmark to show the difference.
This code can add to here https://github.com/elastic/elastic-agent-libs/blob/v0.17.3/monitoring/adapter/go-metrics_test.go

// filters is nil
func BenchmarkGoMetricsAdapter(b *testing.B) {
 var filters []MetricFilter
 monReg := monitoring.NewRegistry()
 var reg metrics.Registry = GetGoMetrics(monReg, "test", filters...)

 for i := 0; i < b.N; i++ {
  meter := reg.GetOrRegisterAdapter("meter", func() interface{} {
   return metrics.NewMeter()
  }).(metrics.Meter)
  meter.Count()
  
  reg.UnregisterAll()
 }
}

// filters include GoMetricsNilify
func BenchmarkGoMetricsAdapterNilify(b *testing.B) {
 filters := []MetricFilter{
  GoMetricsNilify,
 }
 monReg := monitoring.NewRegistry()
 var reg metrics.Registry = GetGoMetrics(monReg, "test", filters...)

 for i := 0; i < b.N; i++ {
  meter := reg.GetOrRegister("meter", func() interface{} {
   return metrics.NewMeter()
  }).(metrics.Meter)
  meter.Count()
  
  reg.UnregisterAll()
 }
}

The BenchmarkGoMetricsAdapterNilify's result shows that the inuse objects will remain the same with allocated objects.
But the BenchmarkGoMetricsAdapter's result shows that the inuse objects remain zero.

[botelastic]

This issue doesn't have a Team:<team> label.