Optimizing client-go Informer Cache for Large‑Scale Kubernetes Clusters

Background: In large‑scale Kubernetes clusters, a custom controller that list‑watches many resources can consume a lot of memory if the cached objects are not trimmed. client-go provides a TransformFunc that can modify objects before they are stored in the cache.

client-go defines the type type TransformFunc func(interface{}) (interface{}, error). The controller‑runtime library exposes a type ByObject struct { Transform toolscache.TransformFunc } struct that can be used to attach a transform to a specific object type.

Usage: In a cluster with more than 500 Secrets, watching all resources with the default informer consumes significant memory. An experiment using the client‑go demo to watch 500 Secrets implements a filtered informer:

func (f *filteredSecretInformer) Informer() Informer {
  typedInformer := f.typedInformerFactory.InformerFor(&corev1.Secret{}, f.newTyped)
  metadataInformer := f.metadataInformerFactory.ForResource(secretsGVR).Informer()
  return &informer{typedInformer: typedInformer, metadataInformer: metadataInformer}
}

func (f *filteredSecretInformer) Lister() SecretLister {
  typedLister := corev1listers.NewSecretLister(f.typedInformerFactory.InformerFor(&corev1.Secret{}, f.newTyped).GetIndexer())
  metadataLister := metadatalister.New(f.metadataInformerFactory.ForResource(secretsGVR).Informer().GetIndexer(), secretsGVR)
  return &secretLister{typedClient: f.typedClient, namespace: f.namespace, typedLister: typedLister, partialMetadataLister: metadataLister, ctx: f.ctx}
}

Resulting apiserver metrics show a peak memory usage of about 80 MiB, which drops to roughly 70 MiB after five minutes, and a CPU peak of 0.06 % of a core.

Adding a trimming function to client‑go:

func metadataRemoveTransform(obj interface{}) (interface{}, error) {
  partialMeta, ok := obj.(*metav1.PartialObjectMetadata)
  if !ok {
    return nil, fmt.Errorf("internal error: cannot cast object %v to PartialObjectMetadata", obj)
  }
  partialMeta.Annotations = nil
  partialMeta.ManagedFields = nil
  partialMeta.Labels = nil
  return partialMeta, nil
}

func (f *filteredSecretInformer) Informer() Informer {
  typedInformer := f.typedInformerFactory.InformerFor(&corev1.Secret{}, f.newTyped)
  metadataInformer := f.metadataInformerFactory.ForResource(secretsGVR).Informer()
  // Set transform to trim metadata
  if err := metadataInformer.SetTransform(metadataRemoveTransform); err != nil {
    panic(fmt.Sprintf("internal error: error setting transformer on the metadata informer: %v", err))
  }
  return &informer{typedInformer: typedInformer, metadataInformer: metadataInformer}
}

Adding a trimming function via controller‑runtime:

controllerOptions := controllerruntime.Options{
  Scheme:                 gclient.NewSchema(),
  LeaderElection:         opts.LeaderElection.LeaderElect,
  LeaseDuration:         &opts.LeaderElection.LeaseDuration.Duration,
  RenewDeadline:         &opts.LeaderElection.RenewDeadline.Duration,
  RetryPeriod:           &opts.LeaderElection.RetryPeriod.Duration,
  LeaderElectionID:      opts.LeaderElection.ResourceName,
  LeaderElectionNamespace: opts.LeaderElection.ResourceNamespace,
  LeaderElectionResourceLock: opts.LeaderElection.ResourceLock,
  HealthProbeBindAddress: net.JoinHostPort(opts.BindAddress, strconv.Itoa(opts.SecurePort)),
  LivenessEndpointName:   "/healthz",
  ReadinessEndpointName:  "/readyz",
  Cache: cache.Options{ByObject: map[client.Object]cache.ByObject{
    &corev1.Secret{}: {Transform: func(obj interface{}) (interface{}, error) {
      partialMeta, ok := obj.(*metav1.PartialObjectMetadata)
      if !ok { return nil, fmt.Errorf("internal error: cannot cast object %v to PartialObjectMetadata", obj) }
      partialMeta.Annotations = nil
      partialMeta.ManagedFields = nil
      partialMeta.Labels = nil
      return partialMeta, nil
    }},
    &corev1.ConfigMap{}: {Transform: func(obj interface{}) (interface{}, error) {
      partialMeta, ok := obj.(*metav1.PartialObjectMetadata)
      if !ok { return nil, fmt.Errorf("internal error: cannot cast object %v to PartialObjectMetadata", obj) }
      partialMeta.Annotations = nil
      partialMeta.ManagedFields = nil
      partialMeta.Labels = nil
      return partialMeta, nil
    }},
  }},
}

After applying the controller‑runtime transform, apiserver metrics improve: peak memory usage drops to about 37 MiB and settles near 20 MiB after five minutes; CPU peaks at 0.15 % of a core and falls to roughly 0.05 % after the same period.

Result: In heavy read/write scenarios for Secrets and ConfigMaps, using a custom transform to strip annotations, managed fields, and labels reduces apiserver memory consumption by roughly 60 %.