Occasionally when writing to Aerospike, a service crashes in the Aerospike client when attempting to get a node from the current partition. The length of the array is always 0 when the crash occurs.
The code here could be modified to ensure it's not getting exceeding the bounds of replicas, but I'm trying to understand why this is happening. Any thoughts?
Client version: 5.7.0
Example stack trace (excluding service specific code):
github.com/aerospike/aerospike-client-go/v5.(*Partition).getSequenceNode(0x46d5d3, 0x61f9685e)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.7.0/partition.go:195 +0xfc
github.com/aerospike/aerospike-client-go/v5.(*Partition).GetNodeWrite(0xc01b79eef8, 0x48c6d7)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.7.0/partition.go:165 +0x53
github.com/aerospike/aerospike-client-go/v5.(*operateCommand).getNode(0xc07677f782a295c3, {0x4e94324576a4, 0x1d0a1c0})
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.7.0/operate_command.go:43 +0x2d
github.com/aerospike/aerospike-client-go/v5.(*baseCommand).executeAt(0xc03018ae10, {0x13db778, 0xc03018ae10}, 0xc001fabb90, 0x80, {0xc001fabb90, 0x0, 0x1d0a1c0}, 0x0, 0x0)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.7.0/command.go:2058 +0x532
github.com/aerospike/aerospike-client-go/v5.(*baseCommand).execute(0xc001fabb90, {0x13db778, 0xc03018ae10}, 0x4)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.7.0/command.go:1991 +0x8a
github.com/aerospike/aerospike-client-go/v5.(*operateCommand).Execute(...)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.7.0/operate_command.go:60
github.com/aerospike/aerospike-client-go/v5.(*Client).Operate(0xc0001fc840, 0x0, 0x0, {0xc01cf073e0, 0xfc4500, 0xc01cf13eb0})
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.7.0/client.go:515 +0x3beActivity
3
asecondo4y ago@khaf We are monitoring our clusters, but I wasn't able to find any direct indicators that a partition or major network issue happened during the time this error popped up. Do you have any suggestions for anything to specifically look for?
For a bit more context, this has only happened ~75 times in the past 30 days, and 1 time in the past 2 weeks. It's likely that this is related to some partition or network issue as you described since there's no clear pattern that I've been able to identify yet.
ghost1y agoAny updated on this issue? we are getting the same crash with version 6.
I think it happens if we are closing client from one go-routine, while sending commend from another go-routine
This is very surprising. Are you monitoring your cluster? Does a partition or major network issue happen at the same time as these errors?