#492
Issues/aerospike/aerospike-client-go/#492

v8 client BatchGetComplex panics with index out of range

Openyizhaopened 1y ago

v8 BatchGetComplex panics with batch policy (v7 is good).

policy *aerospike.BatchPolicy := aerospike.NewBatchPolicy()
policy.TotalTimeout = timeout // 8ms
policy.ConcurrentNodes = 0
policy.MaxRetries = 0
policy.AllowPartialResults = true
panic: runtime error: index out of range [12] with length 0

goroutine 3413 gp=0x4002d9d180 m=14 mp=0x4000d44808 [running]:
panic({0x10dffe0?, 0x4001d60a50?})
        runtime/panic.go:811 +0x154 fp=0x400397cae0 sp=0x400397ca30 pc=0x87fd4
runtime.goPanicIndex(0xc, 0x0)
        runtime/panic.go:115 +0x74 fp=0x400397cb20 sp=0x400397cae0 pc=0x4d984
github.com/aerospike/aerospike-client-go/v8.newBatchOperateNodeListIfcRetry(0x4001e58008, 0x400acdeb60, {0x0, 0x0, 0xbe?}, 0x1, 0x1, 0x4009b83aa0)
        github.com/aerospike/aerospike-client-go/v8@v8.2.2/batch_node_list.go:214 +0x568 fp=0x400397cc30 sp=0x400397cb20 pc=0xb59848
github.com/aerospike/aerospike-client-go/v8.(*batchCommandOperate).generateBatchNodes(0x1?, 0x0?)
        github.com/aerospike/aerospike-client-go/v8@v8.2.2/batch_command_operate.go:302 +0x40 fp=0x400397cc80 sp=0x400397cc30 pc=0xb54ff0
github.com/aerospike/aerospike-client-go/v8.(*batchCommand).retryBatch(0x400ace01a0, {0x1447fd0, 0x400ace01a0}, 0xc205634729e78aa1?, 0x1)
        github.com/aerospike/aerospike-client-go/v8@v8.2.2/batch_command.go:77 +0x38 fp=0x400397cd00 sp=0x400397cc80 pc=0xb4ff08
github.com/aerospike/aerospike-client-go/v8.(*baseCommand).executeAt(0x400ace01a0, {0x1443320, 0x400ace01a0}, 0x400acdeb60, {0x0?, 0x11b86a2?, 0x1f17f80?}, 0x40018bfa40?)
        github.com/aerospike/aerospike-client-go/v8@v8.2.2/command.go:3650 +0x1b98 fp=0x400397cea0 sp=0x400397cd00 pc=0xb82808
github.com/aerospike/aerospike-client-go/v8.(*baseCommand).execute(0x400ace01a0, {0x1443320, 0x400ace01a0})
        github.com/aerospike/aerospike-client-go/v8@v8.2.2/command.go:3591 +0x68 fp=0x400397cf00 sp=0x400397cea0 pc=0xb80b88
github.com/aerospike/aerospike-client-go/v8.(*baseMultiCommand).execute(...)
        github.com/aerospike/aerospike-client-go/v8@v8.2.2/multi_command.go:495
github.com/aerospike/aerospike-client-go/v8.(*batchIndexCommandGet).Execute(0x4002f4c708?)
        github.com/aerospike/aerospike-client-go/v8@v8.2.2/batch_index_command_get.go:65 +0x4c fp=0x400397cf30 sp=0x400397cf00 pc=0xb56ffc
github.com/aerospike/aerospike-client-go/v8.(*werrGroup).execute.func1()
        github.com/aerospike/aerospike-client-go/v8@v8.2.2/werrgroup.go:62 +0xb0 fp=0x400397cfd0 sp=0x400397cf30 pc=0xbd0460
runtime.goexit({})
        runtime/asm_arm64.s:1223 +0x4 fp=0x400397cfd0 sp=0x400397cfd0 pc=0x905b4
created by github.com/aerospike/aerospike-client-go/v8.(*werrGroup).execute in goroutine 3133
        github.com/aerospike/aerospike-client-go/v8@v8.2.2/werrgroup.go:55 +0x128

Activity

5
khaf1y ago

Thank you for your report. Do you have a gist that could reproduce this issue?

yizha1y ago

No I don't. It happened when we upgraded the aerospike client version to v8 and tested it in our prd env.

One strange thing I noticed is that looks like the panic happens when it tries to retry the batch query, from the stack trace github.com/aerospike/aerospike-client-go/v8.(*batchCommand).retryBatch but we set MaxRetries to 0.

matthewd9810mo ago

@khaf My company appears to be experiencing a similar issue, though with the BatchGet call. We’re running a v6.3 cluster and using the v8.5.0 client, and the panic occurs infrequently -- about once every 15 minutes per client machine.

Here are two stack traces:

panic: runtime error: slice bounds out of range [44571:7]
 
goroutine 14176947 [running]:
github.com/aerospike/aerospike-client-go/v8.bytesToParticle(0xc211942dc8?, {0xc05004a028, 0xa?, 0xffd8}, 0x0, 0x7)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/value.go:1240 +0x689
github.com/aerospike/aerospike-client-go/v8.(*batchCommandGet).parseRecord(0xc211942dc8, 0xc2119212c0, 0x1, 0x19, 0x4ebc63)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/batch_command_get.go:205 +0x17c
github.com/aerospike/aerospike-client-go/v8.(*batchCommandGet).parseRecordResults(0xc211942dc8, {0x27b55f0, 0xc211942dc8}, 0x4b6)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/batch_command_get.go:167 +0x307
github.com/aerospike/aerospike-client-go/v8.(*baseMultiCommand).parseResult(0xc211942dc8, {0x27b55f0, 0xc211942dc8}, 0x10000?)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/multi_command.go:184 +0x35d
github.com/aerospike/aerospike-client-go/v8.(*baseCommand).executeAt(0xc211942dc8, {0x27b55f0, 0xc211942dc8}, 0xc22d29ef80, {0xc1daa3df18?, 0xc1daa3df18?, 0x3d00420?}, 0xc329ffb590?)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/command.go:3891 +0xcca
github.com/aerospike/aerospike-client-go/v8.(*baseCommand).execute(0xc211942dc8, {0x27b55f0, 0xc211942dc8})
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/command.go:3659 +0x65
github.com/aerospike/aerospike-client-go/v8.(*baseMultiCommand).execute(...)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/multi_command.go:509
github.com/aerospike/aerospike-client-go/v8.(*batchCommandGet).Execute(0xc358a1d9e0?)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/batch_command_get.go:272 +0x35
github.com/aerospike/aerospike-client-go/v8.(*werrGroup).execute.func1()
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/werrgroup.go:62 +0xb1
created by github.com/aerospike/aerospike-client-go/v8.(*werrGroup).execute in goroutine 14157668
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/werrgroup.go:55 +0x130
panic: runtime error: index out of range [5] with length 4

goroutine 38904 [running]:
github.com/aerospike/aerospike-client-go/v8.(*unpacker).unpackObject(0xc00193a9f8?, 0x31?)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/unpacker.go:271 +0xa9a
github.com/aerospike/aerospike-client-go/v8.(*unpacker).unpackList(0xc00193aa98, 0xd85e7100)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/unpacker.go:75 +0x49
github.com/aerospike/aerospike-client-go/v8.(*unpacker).unpackObject(0x30?, 0x20?)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/unpacker.go:368 +0x7b6
github.com/aerospike/aerospike-client-go/v8.(*unpacker).unpackObjects(...)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/unpacker.go:208
github.com/aerospike/aerospike-client-go/v8.bytesToParticle(0xc0d694c588?, {0xc000e6e02b, 0xd?, 0xffd5}, 0x0, 0x4)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/value.go:1253 +0x5ce
github.com/aerospike/aerospike-client-go/v8.(*batchCommandGet).parseRecord(0xc0d694c588, 0xc0d6093200, 0x1, 0x30c, 0x149dc)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/batch_command_get.go:205 +0x17c
github.com/aerospike/aerospike-client-go/v8.(*batchCommandGet).parseRecordResults(0xc0d694c588, {0x27b56f0, 0xc0d694c588}, 0xb3)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/batch_command_get.go:167 +0x307
github.com/aerospike/aerospike-client-go/v8.(*baseMultiCommand).parseResult(0xc0d694c588, {0x27b56f0, 0xc0d694c588}, 0x10000?)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/multi_command.go:184 +0x35d
github.com/aerospike/aerospike-client-go/v8.(*baseCommand).executeAt(0xc0d694c588, {0x27b56f0, 0xc0d694c588}, 0xc14e0e9380, {0xc18fb0de20?, 0xc002d5d7a0?, 0x3d00420?}, 0xc18fb0ddc0?)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/command.go:3891 +0xcca
github.com/aerospike/aerospike-client-go/v8.(*baseCommand).execute(0xc0d694c588, {0x27b56f0, 0xc0d694c588})
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/command.go:3659 +0x65
github.com/aerospike/aerospike-client-go/v8.(*baseMultiCommand).execute(...)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/multi_command.go:509
github.com/aerospike/aerospike-client-go/v8.(*batchCommandGet).Execute(0xc0bb3e1890?)
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/batch_command_get.go:272 +0x35
github.com/aerospike/aerospike-client-go/v8.(*werrGroup).execute.func1()
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/werrgroup.go:62 +0xb1
created by github.com/aerospike/aerospike-client-go/v8.(*werrGroup).execute in goroutine 28863
        /go/pkg/mod/github.com/aerospike/aerospike-client-go/v8@v8.5.0/werrgroup.go:55 +0x130

As you can see, the bottom of the stack is identical to @yizha’s, which makes sense since both are batch operations.

I’m also curious whether it’s intentional not to recover from the panic and instead return a client-internal failure error to the caller. In that case, at least, the application using the library wouldn’t crash... For read operations in particular, I can’t see a reason not to recover from the panic. For write operations, it’s possible that recovery could leave the server or library in an inconsistent state, making a crash preferable -- but even then, I’m not entirely convinced that’s the case.

khaf10mo ago

@mirzakaracic Do you think this is related to https://github.com/aerospike/aerospike-client-go/pull/568 ?

mirzakaracic10mo ago

@khaf the trace that is reported by @yizha is fixed with https://github.com/aerospike/aerospike-client-go/pull/568. The issue reported by @matthewd98 is something else. @matthewd98 has filed a separate issue here for the above errors.