bigtable: potential memory leak from xDS runner goroutine when DirectPath is unavailable, introduced in v1.46.0

Client

v1.46.0+

Environment

go 1.26.3

Code and Dependencies

// initialize bigtable client
bigtableClientClientConf := gcpBigtable.ClientConfig{
		AppProfile:      appProfile,
		MetricsProvider: gcpBigtable.NoopMetricsProvider{},
	}
bigtableClient, btErr := gcpBigtable.NewClientWithConfig(ctx, projectID, instanceID, bigtableClientClientConf, option.WithGRPCConnectionPool(cfg.BigtableConnectionPoolSize))
...

go.mod

module modname

go 1.25.0

require (
   cloud.google.com/go/bigtable v1.47.0
)

Expected behavior

Memory usage is stable, consistent with v1.43.0–v1.45.0 behavior

Actual behavior

Heap grows continuously after upgrading to v1.46.0. Heap profile shows (*adsStreamImpl).runner from google.golang.org/grpc@v1.80.0/internal/xds/clients/xdsclient/ads_stream.go accumulating memory. Which caused OOM in one of our pods.

Screenshots

Memory Usage Trends

Heap Live Size

Additional context

We are suspecting the bug is the unconditional DirectPath probe spawning xDS goroutines that can't clean up on non-DirectPath infrastructure.
The trigger is v1.46.0 enabling BigtableChannelPool by default, which exposed this to all users.

Our temporary workaround :
After setting CBT_ENABLE_DIRECTPATH=false, the memory usage returns to stable:

@sushanb

Reassigning to @sushanb who works on the data client.

@mnjnwang

@mnjnwang to clarify, this is about the step increase in the overall memory utilisation when upgrading to 1.46.0 client, right?

I just want to make sure you are seeing the memory increase and stay constant but the constant memory accumulation through the lifetime of the pod.

@mnjnwang to clarify, this is about the step increase in the overall memory utilisation when upgrading to 1.46.0 client, right?

I just want to make sure you are seeing the memory increase and stay constant but the constant memory accumulation through the lifetime of the pod.

It is not a stable step increase. We are seeing constant memory accumulation over the lifetime of the pod. It steadily climbs and looks likely a memory leak introduced in 1.46.0.

Thanks. We are looking into it.
One more question, do you set the connection pool size or use the default one?

@sushanb

@sushanb Yes we set the pool size to 2 by adding option.WithGRPCConnectionPool(2)) in the opts during client initialization.

I believe we're seeing the same thing. We were using direct path at 1.44.0 via setting CBT_DIRECT_PATH=true with no obvious memory leak. We then accidentally disabled it. Yesterday we realised, upgraded to 1.48 where it is on by default (and I believe now uses a new bigtable connection pool by default too?). Today we see evidence of steady memory growth since the upgrade.

We use option.WithGRPCConnectionPool(6)

Note that in our case Direct Path is available. It certainly was working with 1.44, and CheckDirectAccessSupported still returns true.

edit: I should say the wavy nature of the line is probably because in this environment the only thing that's running is a bunch of system tests that run about once every 10 minutes

The leak appears to be related to XDS endpoints

 go tool pprof -diff_base mem.prof mem6.prof 
File: psp-tagging-service
Build ID: 561ab4f049bf7cf58b3c063e01d942597e1e969f
Type: inuse_space
Time: 2026-06-03 12:33:18 BST
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) drop_negative
(pprof) top
Showing nodes accounting for 5643.37kB, 11.28% of 50008.79kB total
Dropped 6 nodes (cum <= 250.04kB)
Showing top 10 nodes out of 50
      flat  flat%   sum%        cum   cum%
 1536.09kB  3.07%  3.07%  1536.09kB  3.07%  google.golang.org/grpc/internal/xds/xdsclient/xdsresource.parseEndpoints
  520.04kB  1.04%  4.11%   520.04kB  1.04%  google.golang.org/grpc/internal/mem.(*sizedBufferPool).Get
  514.38kB  1.03%  5.14%   514.38kB  1.03%  google.golang.org/grpc/balancer/pickfirst.deDupAddresses
  512.69kB  1.03%  6.17%  1024.70kB  2.05%  google.golang.org/grpc/internal/xds/resolver.(*xdsResolver).sendNewServiceConfig
  512.10kB  1.02%  7.19%   512.10kB  1.02%  google.golang.org/grpc/internal/xds/resolver.(*xdsResolver).addOrGetActiveClusterInfo
  512.02kB  1.02%  8.21%   512.02kB  1.02%  google.golang.org/grpc/internal/buffer.NewUnbounded
  512.02kB  1.02%  9.24%  1024.05kB  2.05%  google.golang.org/grpc/internal/xds.SetLocalityIDInEndpoint
  512.02kB  1.02% 10.26%   512.02kB  1.02%  google.golang.org/protobuf/internal/encoding/json.(*Decoder).parseString
  512.01kB  1.02% 11.28%   512.01kB  1.02%  google.golang.org/grpc/internal/buffer.(*Unbounded).Put
         0     0% 11.28%   512.02kB  1.02%  encoding/json.(*decodeState).array

Inuse-object counts perhaps points to XDS endpoint updates getting left in the unbounded buffer? It's not at all clear why that might happen.

(pprof) sample_index=inuse_objects
(pprof) top          
Showing nodes accounting for 98469, 20.43% of 482033 total
Dropped 6 nodes (cum <= 2410)
Showing top 10 nodes out of 54
      flat  flat%   sum%        cum   cum%
     32768  6.80%  6.80%      32768  6.80%  google.golang.org/grpc/internal/buffer.(*Unbounded).Put
     24577  5.10% 11.90%      24577  5.10%  google.golang.org/grpc/internal/xds/xdsclient/xdsresource.parseEndpoints
     16384  3.40% 15.30%      16384  3.40%  google.golang.org/protobuf/internal/encoding/json.(*Decoder).parseString
     10923  2.27% 17.56%      10923  2.27%  google.golang.org/grpc/internal/buffer.NewUnbounded
     10923  2.27% 19.83%      21846  4.53%  google.golang.org/grpc/internal/xds.SetLocalityIDInEndpoint
      2521  0.52% 20.35%       2521  0.52%  google.golang.org/grpc/internal/xds/resolver.(*xdsResolver).addOrGetActiveClusterInfo
       373 0.077% 20.43%      16757  3.48%  google.golang.org/grpc/internal/xds/resolver.(*xdsResolver).sendNewServiceConfig
         0     0% 20.43%      16384  3.40%  encoding/json.(*decodeState).array
         0     0% 20.43%      16384  3.40%  encoding/json.(*decodeState).object
         0     0% 20.43%      16384  3.40%  encoding/json.(*decodeState).unmarshal

I'm trying CBT_ENABLE_DIRECTPATH=true & CBT_BIGTABLE_CONN_POOL=false. Memory usage looks flat, but it takes a while to accumulate so I'll try to remember to report back later.

Memory growth is definitely slower like this, and perhaps is rising slowly to a maximum (asymptotic). The graph starts on 1.44 without direct path, the middle section is 1.48, the final section is 1.48 with CBT_ENABLE_DIRECTPATH=true & CBT_BIGTABLE_CONN_POOL=false

I let the original leak run overnight. This is with the default 1.48 behaviour, not disabling the new conn pool.

Showing nodes accounting for 21539.34kB, 43.07% of 50008.79kB total
Dropped 11 nodes (cum <= 250.04kB)
Showing top 10 nodes out of 163
      flat  flat%   sum%        cum   cum%
10758.60kB 21.51% 21.51% 13318.65kB 26.63%  google.golang.org/grpc/internal/xds/xdsclient/xdsresource.parseEndpoints
 2560.05kB  5.12% 26.63%  2560.05kB  5.12%  net.JoinHostPort (inline)
 2057.01kB  4.11% 30.75%  2057.01kB  4.11%  google.golang.org/protobuf/internal/impl.consumeBytesNoZero
 1025.38kB  2.05% 32.80%  3586.17kB  7.17%  google.golang.org/grpc/internal/xds/resolver.(*xdsResolver).sendNewServiceConfig
 1024.59kB  2.05% 34.85%  1024.59kB  2.05%  google.golang.org/grpc/balancer/rls/internal/keys.MakeBuilderMap
 1024.20kB  2.05% 36.89%  1024.20kB  2.05%  google.golang.org/grpc/internal/xds/resolver.(*xdsResolver).addOrGetActiveClusterInfo
 1024.14kB  2.05% 38.94%  1536.15kB  3.07%  fmt.Sprintf
 1024.08kB  2.05% 40.99%  1024.08kB  2.05%  google.golang.org/grpc/internal/buffer.NewUnbounded
  528.17kB  1.06% 42.05%   528.17kB  1.06%  google.golang.org/grpc/internal/mem.(*sizedBufferPool).Get
  513.12kB  1.03% 43.07%   513.12kB  1.03%  google.golang.org/grpc/balancer/pickfirst.deDupAddresses


Showing nodes accounting for 359164, 74.51% of 482033 total
Dropped 53 nodes (cum <= 2410)
Showing top 10 nodes out of 111
      flat  flat%   sum%        cum   cum%
    131074 27.19% 27.19%     131074 27.19%  net.JoinHostPort (inline)
     94218 19.55% 46.74%     225292 46.74%  google.golang.org/grpc/internal/xds/xdsclient/xdsresource.parseEndpoints
     32768  6.80% 53.54%      52865 10.97%  encoding/json.(*decodeState).object
     32768  6.80% 60.33%      32768  6.80%  google.golang.org/grpc/attributes.(*Attributes).String-range1 (inline)
     18432  3.82% 64.16%      51200 10.62%  fmt.Sprintf
     16384  3.40% 67.56%      16384  3.40%  google.golang.org/protobuf/internal/encoding/json.(*Decoder).parseString
     15605  3.24% 70.79%      15605  3.24%  google.golang.org/grpc/internal/buffer.NewUnbounded
      8192  1.70% 72.49%       8192  1.70%  google.golang.org/grpc/internal/resolver/dns.(*dnsResolver).lookup
      5042  1.05% 73.54%       5042  1.05%  google.golang.org/grpc/internal/xds/resolver.(*xdsResolver).addOrGetActiveClusterInfo
      4681  0.97% 74.51%       4681  0.97%  context.(*cancelCtx).Done

@philpearl

@philpearl thanks for the pprof. i am trying to run the tests with CBT_ENABLE_CONN_POOL=true and false and see if i can repro the memory leak on my side. thanks.

We have a mitigation by disabling the connection recycler. the bug seems to be in the xds library so what we can do is

set the conn recycler to a high value (O(weeks)) to avoid conn recycler which causes the buildup.
Ask grpc go to land the fix
Enable the conn recycler.

fix(bigtable): raise default connection-recycler MaxAge to 7 days (#1…

i also disabled dynamic channel pool to be safe(it does not cause bloat compared to regular connection recycling). #19945.

For workaround in the affected version, 1.46.0, 1.47.0 and 1.48.0, customers can set
configs.DisableConnectionRecycler = true

google-cloud-go/bigtable/client.go

Line 76 in a7959ef

DisableConnectionRecycler bool

Updated known issue in

https://github.com/googleapis/google-cloud-go/releases/tag/bigtable%2Fv1.46.0
https://github.com/googleapis/google-cloud-go/releases/tag/bigtable%2Fv1.47.0
https://github.com/googleapis/google-cloud-go/releases/tag/bigtable%2Fv1.48.0

fix(bigtable): disable dynamic channel pool by default and raise defa…

@mnjnwang

@mnjnwang @philpearl bigtable: v1.49.0 Latest this should remove the memory leak with direct access.

Please give it a try without any flags and let us know if you encounter any issue.

@sushanb

Thanks @sushanb! I've got 1.49.0 running with a couple of services in our staging environment. It all looks good so far, but I'll check again tomorrow to give it a chance to leak.

@philpearl

any updates @philpearl

I have put the code into production for a couple of services, but I do think we're still seeing a slight rise in memory over time that we didn't see before. I'll see if I can get some data on what's happening, but it's slight enough that I'm not 100% that it's a leak. I think it would be reasonable to count this as fixed!

docs: add example 1-hour run with pprof bookends

@philpearl

@philpearl we just released a bigtable client with bidi support in 1.52.0 (instead per rpc, we used a long lived bidi stream and use that for sending and receiving requests). This should provide even better performance in both p50 and p99.

It is gated (basically the 1.52.0 client routes every method via original path). Would you be interested in trying and seeing the performance in staging environment? If so, we could enable the new session path for your staging instances.

@sushanb

@sushanb yes, that would be very interesting. I'll get us rolled up to 1.52.0 (I think we're currently on 1.51)

I should let you know that we think we are still seeing a memory leak, perhaps only in lower-volume services and environments. We're still trying to characterise it.

To avoid PII leaking here, could you send me the instance and app profile for staging in sushanb@google.com?

There will be a incremental memory stepup (cost of 10 grpc channels), in our benchmark it showed around addition of 400 goroutines and 15MB of memory. For this client, we ran multiple tests for our long lived bidi client and there is no memory leak.

I should let you know that we think we are still seeing a memory leak, perhaps only in lower-volume services and environments. We're still trying to characterise it.
-please let me know if you characterise it.

xds/cdsbalancer/e2e_test: drop cc.Connect and issue link per review

Navigation Menu

Repository navigation

bigtable: potential memory leak from xDS runner goroutine when DirectPath is unavailable, introduced in v1.46.0 #14582

Client

Environment

Code and Dependencies

Expected behavior

Actual behavior

Screenshots

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

bigtable: potential memory leak from xDS runner goroutine when DirectPath is unavailable, introduced in v1.46.0 #14582

Description

Client

Environment

Code and Dependencies

Expected behavior

Actual behavior

Screenshots

Additional context

Activity

bhshkh commented on May 15, 2026

sushanb commented on May 19, 2026

mnjnwang commented on May 20, 2026

sushanb commented on May 20, 2026

mnjnwang commented on May 20, 2026

philpearl commented on Jun 3, 2026

philpearl commented on Jun 3, 2026

philpearl commented on Jun 3, 2026

philpearl commented on Jun 4, 2026

philpearl commented on Jun 4, 2026

sushanb commented on Jun 10, 2026

sushanb commented on Jun 10, 2026

sushanb commented on Jun 10, 2026

sushanb commented on Jun 10, 2026

sushanb commented on Jun 10, 2026

philpearl commented on Jun 11, 2026

sushanb commented on Jun 12, 2026

philpearl commented on Jun 15, 2026

sushanb commented on Aug 4, 2026

philpearl commented on Aug 5, 2026

sushanb commented on Aug 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions