Repository navigation
bigtable: Slow memory leak with Direct Access enabled #20331
Description
Client
cloud.google.com/go/bigtable v1.52.0
google.golang.org/grpc v1.83.0
Environment
Go 1.26, scratch containers on GKE
Code and Dependencies
Code is too big and ugly to list here. But it's just doing normal reads and writes to a Bigtable table
Expected behavior
Memory use doesn't keep growing
Actual behavior
Memory use does keep growing.
Screenshots
Additional context
This is very similar to github.com//issues/14582, but slower. Profiles point to similar objects being leaked. The leak is more apparent in services with low traffic - but that's perhaps because their memory usage is very small compared to the size of the leak.
Here's some profile output showing inuse_objects and inuse_space compared to a baseline profile taken about 18 hours before
Showing nodes accounting for 369747, 51.91% of 712220 total
Dropped 68 nodes (cum <= 3561)
Showing top 20 nodes out of 213
flat flat% sum% cum cum%
120152 16.87% 16.87% 120152 16.87% net.JoinHostPort (inline)
52025 7.30% 24.17% 172177 24.17% google.golang.org/grpc/internal/xds/xdsclient/xdsresource.parseEndpoints
32768 4.60% 28.78% 32768 4.60% google.golang.org/grpc/internal/xds/balancer/outlierdetection.newCallCounter
32768 4.60% 33.38% 32768 4.60% google.golang.org/protobuf/internal/filedesc.(*Builder).optionsUnmarshaler
21845 3.07% 36.44% -10924 1.53% google.golang.org/grpc/internal/hierarchy.SetInEndpoint
16384 2.30% 38.74% 16384 2.30% google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).switchTo
16384 2.30% 41.04% 16384 2.30% google.golang.org/grpc/internal/xds/xdsdepmgr.(*DependencyManager).populateAggregateClusterLocked
16384 2.30% 43.34% 16384 2.30% google.golang.org/grpc/internal/xds/xdsdepmgr.newClusterWatcher
10923 1.53% 44.88% 10923 1.53% fmt.Sprintf
10923 1.53% 46.41% 10923 1.53% google.golang.org/grpc/balancer/rls/internal/keys.MakeBuilderMap
6554 0.92% 47.33% 6554 0.92% google.golang.org/grpc/internal/xds/clients/xdsclient.(*AnyProto).ToAny
4681 0.66% 47.99% 4681 0.66% reflect.unsafe_New
4096 0.58% 48.56% 4096 0.58% github.com/modern-go/reflect2.newUnsafeStructField
4096 0.58% 49.14% 4096 0.58% github.com/unravelin/core/feature-extraction-service/proto.buildFlattenStruct
4096 0.58% 49.71% 4096 0.58% google.golang.org/protobuf/internal/encoding/json.(*Decoder).parseString
3641 0.51% 50.23% 3641 0.51% encoding/json.(*decodeState).literalStore
3641 0.51% 50.74% 24822 3.49% encoding/json.(*decodeState).object
3344 0.47% 51.21% 3344 0.47% runtime.mallocgc
2521 0.35% 51.56% -2160 0.3% context.(*cancelCtx).propagateCancel
2521 0.35% 51.91% 2521 0.35% google.golang.org/grpc/balancer/weightedtarget/weightedaggregator.(*Aggregator).Add
Showing nodes accounting for 19.54MB, 27.57% of 70.88MB total
Dropped 35 nodes (cum <= 0.35MB)
Showing top 20 nodes out of 225
flat flat% sum% cum cum%
6.50MB 9.18% 9.18% 9MB 12.70% google.golang.org/grpc/internal/xds/xdsclient/xdsresource.parseEndpoints
2.50MB 3.53% 12.70% 2.50MB 3.53% net.JoinHostPort (inline)
1.50MB 2.12% 14.83% 1.50MB 2.12% google.golang.org/protobuf/internal/impl.consumeBytesNoZero
1MB 1.41% 16.24% 1MB 1.41% google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).switchTo
0.52MB 0.73% 16.97% 0.52MB 0.73% google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile
0.51MB 0.72% 17.69% 0.51MB 0.72% github.com/unravelin/core/lib/lang/internal/names.map.init.0
0.50MB 0.71% 18.40% 0.50MB 0.71% math/rand.newSource
0.50MB 0.71% 19.11% 0.50MB 0.71% github.com/unravelin/core/pspdomain.map.init.1
0.50MB 0.71% 19.81% 0.50MB 0.71% cloud.google.com/go/bigtable/internal/metrics.NewFactory
0.50MB 0.71% 20.52% 0.50MB 0.71% google.golang.org/grpc/balancer/weightedtarget/weightedaggregator.(*Aggregator).Add
0.50MB 0.71% 21.22% 0.50MB 0.71% google.golang.org/grpc/internal/xds/resolver.(*xdsResolver).addOrGetActiveClusterInfo
0.50MB 0.71% 21.93% 1MB 1.41% google.golang.org/grpc/internal/xds/xdsdepmgr.(*DependencyManager).applyRouteConfigUpdateLocked
0.50MB 0.71% 22.63% 1.50MB 2.12% google.golang.org/grpc/internal/xds/xdsdepmgr.(*DependencyManager).populateClusterConfigLocked
0.50MB 0.71% 23.34% 0.50MB 0.71% google.golang.org/protobuf/internal/impl.(*MessageInfo).makeFieldTypes
0.50MB 0.71% 24.05% 0.50MB 0.71% reflect.mapassign_faststr0
0.50MB 0.71% 24.75% 0.50MB 0.71% encoding/json.(*decodeState).literalStore
0.50MB 0.71% 25.46% 2.50MB 3.53% encoding/json.(*decodeState).object
0.50MB 0.71% 26.16% 0.50MB 0.71% github.com/modern-go/reflect2.newUnsafeStructField
0.50MB 0.71% 26.87% 0.50MB 0.71% github.com/unravelin/core/feature-extraction-service/proto.buildFlattenStruct
0.50MB 0.71% 27.57% 0.50MB 0.71% google.golang.org/protobuf/internal/encoding/json.(*Decoder).parseString
Reacted by Adam Jones, ig-ravelin and Wally S
Activity
sushanb commentedon Aug 19, 2026
i sent this one. grpc/grpc-go#9301 which should be in grpc 1.84.0
adamroyjones commentedon Aug 19, 2026
@sushanb: Thanks! Once a new version of google.golang.org/grpc has been released that contains the changes, we'll roll it out to a staging cluster and provide an update.
wshirey commentedon Sep 17, 2026
@adamroyjones @philpearl Looks like grpc-go v1.8.4 was released. Eager to see your findings.
adamroyjones commentedon Sep 18, 2026
It doesn't look like grpc/grpc-go#9301 made it into v1.84.0, sadly (diff of v1.83.2 and v1.84.0).
We'll continue to eagerly await its release.
wshirey commentedon Oct 27, 2026
@adamroyjones @philpearl FYI it looks like grpc-go v1.8.5 was released today and included the fix.
adamroyjones commentedon Oct 27, 2026
@wshirey: Thanks! v1.85.0 doesn't mention the PR but I can see the changes went out (e.g., this region is from the PR).
I'll look to test it from tomorrow. I'll hopefully have an update by Monday at the latest.
adamroyjones commentedon Oct 28, 2026
The initial signs are promising with
and with Direct Access enabled.
A 12-hour period used to look like this; this is the same (internal) service that Phil was showing graphs for.
google.golang.org/grpc@v1.83.0
The new deployment looks like this over a 2.5-hour period.
google.golang.org/grpc@v1.85.0
I don't see an obvious leak in this, though it's possible that the rate of growth is so low that it'll take a few days to properly manifest itself.
I'll provide an update on Monday. (I also intend to roll our use of the Bigtable client library forward from v1.51.0, but I'll wait before doing that.)
wshirey commentedon Oct 30, 2026
@adamroyjones thanks for sharing your results, I agree, they do look promising.