#581
Client v8: Out-of-Range Panics in BatchGet and GetObject
Our company is currently upgrading our clusters from older releases (such as v6.3) to v8.1. According to the compatibility matrix, the client must first be updated to at least v7.0.0 before the server upgrade. Since we’re already performing an upgrade, we’ve chosen to move directly to the latest client version, v8.5.0. During testing, we observed two bounds out-of-range panics triggered by different commands: the first with GetObject (a reflection-based call) and the second with BatchGet. The other commands that we use in our code, e.g. Get and Query, seem to function correctly.
These panics occur infrequently -- approximately once every 15 minutes per client machine. The second case is particularly problematic because it cannot be recovered from at the application level: the panicking goroutine is created within the library itself, leaving the application unable to recover and ultimately causing the entire application to crash. As a side note, I wonder whether this behavior is intentional -- specifically, the choice not to recover from the panic and instead return a client internal failure error to the caller. At least in that scenario, the application using the library wouldn’t crash. For read operations, I see no reason not to recover from the panic. For write operations, I can imagine recovery leaving the server or client in an inconsistent state, making a crash preferable -- but even then, I’m not entirely convinced that’s the right approach.
Here are two different stack traces for the BatchGet panic:
panic: runtime error: slice bounds out of range [44571:7]
goroutine 14176947 [running]:
github.com/aerospike/aerospike-client-go/v8.bytesToParticle(0xc211942dc8?, {0xc05004a028, 0xa?, 0xffd8}, 0x0, 0x7)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/value.go:1240 +0x689
github.com/aerospike/aerospike-client-go/v8.(*batchCommandGet).parseRecord(0xc211942dc8, 0xc2119212c0, 0x1, 0x19, 0x4ebc63)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/batch_command_get.go:205 +0x17c
github.com/aerospike/aerospike-client-go/v8.(*batchCommandGet).parseRecordResults(0xc211942dc8, {0x27b55f0, 0xc211942dc8}, 0x4b6)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/batch_command_get.go:167 +0x307
github.com/aerospike/aerospike-client-go/v8.(*baseMultiCommand).parseResult(0xc211942dc8, {0x27b55f0, 0xc211942dc8}, 0x10000?)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/multi_command.go:184 +0x35d
github.com/aerospike/aerospike-client-go/v8.(*baseCommand).executeAt(0xc211942dc8, {0x27b55f0, 0xc211942dc8}, 0xc22d29ef80, {0xc1daa3df18?, 0xc1daa3df18?, 0x3d00420?}, 0xc329ffb590?)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/command.go:3891 +0xcca
github.com/aerospike/aerospike-client-go/v8.(*baseCommand).execute(0xc211942dc8, {0x27b55f0, 0xc211942dc8})
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/command.go:3659 +0x65
github.com/aerospike/aerospike-client-go/v8.(*baseMultiCommand).execute(...)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/multi_command.go:509
github.com/aerospike/aerospike-client-go/v8.(*batchCommandGet).Execute(0xc358a1d9e0?)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/batch_command_get.go:272 +0x35
github.com/aerospike/aerospike-client-go/v8.(*werrGroup).execute.func1()
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/werrgroup.go:62 +0xb1
created by github.com/aerospike/aerospike-client-go/v8.(*werrGroup).execute in goroutine 14157668
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/werrgroup.go:55 +0x130panic: runtime error: index out of range [5] with length 4
goroutine 38904 [running]:
github.com/aerospike/aerospike-client-go/v8.(*unpacker).unpackObject(0xc00193a9f8?, 0x31?)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/unpacker.go:271 +0xa9a
github.com/aerospike/aerospike-client-go/v8.(*unpacker).unpackList(0xc00193aa98, 0xd85e7100)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/unpacker.go:75 +0x49
github.com/aerospike/aerospike-client-go/v8.(*unpacker).unpackObject(0x30?, 0x20?)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/unpacker.go:368 +0x7b6
github.com/aerospike/aerospike-client-go/v8.(*unpacker).unpackObjects(...)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/unpacker.go:208
github.com/aerospike/aerospike-client-go/v8.bytesToParticle(0xc0d694c588?, {0xc000e6e02b, 0xd?, 0xffd5}, 0x0, 0x4)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/value.go:1253 +0x5ce
github.com/aerospike/aerospike-client-go/v8.(*batchCommandGet).parseRecord(0xc0d694c588, 0xc0d6093200, 0x1, 0x30c, 0x149dc)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/batch_command_get.go:205 +0x17c
github.com/aerospike/aerospike-client-go/v8.(*batchCommandGet).parseRecordResults(0xc0d694c588, {0x27b56f0, 0xc0d694c588}, 0xb3)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/batch_command_get.go:167 +0x307
github.com/aerospike/aerospike-client-go/v8.(*baseMultiCommand).parseResult(0xc0d694c588, {0x27b56f0, 0xc0d694c588}, 0x10000?)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/multi_command.go:184 +0x35d
github.com/aerospike/aerospike-client-go/v8.(*baseCommand).executeAt(0xc0d694c588, {0x27b56f0, 0xc0d694c588}, 0xc14e0e9380, {0xc18fb0de20?, 0xc002d5d7a0?, 0x3d00420?}, 0xc18fb0ddc0?)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/command.go:3891 +0xcca
github.com/aerospike/aerospike-client-go/v8.(*baseCommand).execute(0xc0d694c588, {0x27b56f0, 0xc0d694c588})
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/command.go:3659 +0x65
github.com/aerospike/aerospike-client-go/v8.(*baseMultiCommand).execute(...)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/multi_command.go:509
github.com/aerospike/aerospike-client-go/v8.(*batchCommandGet).Execute(0xc0bb3e1890?)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/batch_command_get.go:272 +0x35
github.com/aerospike/aerospike-client-go/v8.(*werrGroup).execute.func1()
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/werrgroup.go:62 +0xb1
created by github.com/aerospike/aerospike-client-go/v8.(*werrGroup).execute in goroutine 28863
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/werrgroup.go:55 +0x130And here's the stack trace for the GetObject panic:
runtime error: slice bounds out of range [:369098930] with capacity 65536
goroutine 5153597 [running]:
... REDACTED...
github.com/aerospike/aerospike-client-go/v8/utils/buffer.BytesToUint32(...)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/utils/buffer/buffer.go:141
github.com/aerospike/aerospike-client-go/v8.parseObject(0xc0cd07d0e0?, 0x3cff400?, 0x3cff400?, 0x701a3550?, 0xc2?)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/read_command_reflect.go:50 +0x5ad
github.com/aerospike/aerospike-client-go/v8.(*readCommand).parseResult(0xc0cd07d0e0, {0x27b41e0, 0xc0cd07d0e0}, 0x10000?)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/read_command.go:90 +0x27f
github.com/aerospike/aerospike-client-go/v8.(*baseCommand).executeAt(0xc0cd07d0e0, {0x27b41e0, 0xc0cd07d0e0}, 0xc0018f2f50, {0x0?, 0x0?, 0x3cff400?}, 0x0?)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/command.go:3891 +0xcca
github.com/aerospike/aerospike-client-go/v8.(*baseCommand).execute(0xc0cd07d0e0, {0x27b41e0, 0xc0cd07d0e0})
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/command.go:3659 +0x65
github.com/aerospike/aerospike-client-go/v8.(*readCommand).Execute(...)
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/read_command.go:99
github.com/aerospike/aerospike-client-go/v8.(*Client).GetObject(0xc0018f8000, 0x4?, 0xc0ce49c9c0, {0x1f39d60, 0xc276e13320})
/go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/client_reflect.go:86 +0x31d
... REDACTED...Thanks!
@matthewd98 support team will create a ticket for this issue. Further communication for this issue will be conducted via support channel.