sidecar: ICA GPU profile with TurboOCR and CUDA NER

• Add TurboOCR framebuffer OCR and CUDA ONNX NER configuration without changing CPU ICA defaults.
• Build a versioned GPU runtime on a trusted GPU runner for image and native-bundle delivery.
• Gate GPU releases on explicit asset inputs and add offline validation, examples, and installation
 guidance.

graph TD
  runner["GPU runner"] --> builder["Runtime builder"] --> packer["Asset verifier"] --> asset[("Versioned runtime")] --> targets["Docker targets"] --> image["GPU image"] --> sidecar["ICA sidecar"]
  targets --> bundle["Native bundle"] --> sidecar

The following are alternative approaches to this PR:

1. Build the runtime during each release

• ➕ Removes the separately published runtime input.
• ➖ Requires target GPU capacity in the release pipeline.
• ➖ Couples GPU-specific engine validation to every opted-in release and reduces runtime reuse.

2. Ship only a GPU container

• ➕ Avoids the native installer and bundle lifecycle.
• ➖ Does not support hosts using the existing Docker-less ICA installation path.

Recommendation: Keep the separate, GPU-tested runtime asset and explicit release opt-in. It lets CPU releases proceed without GPU infrastructure while giving the image and native bundle the same verified input; neither alternative preserves both properties.

Enhancement (5) +354 / -32

Dockerfile.sidecarAdd checksum-verified ICA GPU image and bundle targets +55/-0

Add checksum-verified ICA GPU image and bundle targets

• Adds a native-asset download stage, an amd64 GPU sidecar image, and an exportable native bundle. The GPU targets require explicit runtime URL and SHA256 inputs; existing CPU targets remain in place.

Dockerfile.sidecar

install-gpu.shInstall and preflight the native ICA GPU bundle +156/-0

Install and preflight the native ICA GPU bundle

• Verifies bundle contents, stages an immutable release, and checks native dependencies, GPU smoke, and configuration as the service user before switching the installation. Optional startup waits for readiness, including CUDA NER initialization.

deploy/ica-sidecar/install-gpu.sh

install.shRoute GPU bundles to their dedicated installer +5/-0

Route GPU bundles to their dedicated installer

• Dispatches bundles marked 'flavour=ica-gpu' to the GPU installer while leaving the CPU installation path intact.

deploy/ica-sidecar/install.sh

ica.goExpose and validate GPU ICA configuration fields +63/-22

Expose and validate GPU ICA configuration fields

• Adds capability-gated TurboOCR paths and OCR, NER, and masking options, with enum and device-range checks. Forwards those fields to the resolved ICA runtime and includes GPU paths in host validation.

sidecar/daemon/ica.go

ica.goResolve TurboOCR and CUDA NER settings in the ICA engine +75/-10

Resolve TurboOCR and CUDA NER settings in the ICA engine

• Applies TurboOCR-specific defaults and operator overrides, adds resolved GPU settings to lane notes, and discovers GPU bundles by manifest instead of requiring CPU OCR assets. Discovery checks required files; serving remains responsible for inference initialization.

sidecar/ica/ica.go

Tests (6) +551 / -0

pullrequest.ymlRun GPU tooling and example-config tests in PR CI +13/-0

Run GPU tooling and example-config tests in PR CI

• Runs offline Python tests for the GPU asset tools and validates the GPU YAML examples in an ORT-tagged Go build.

.github/workflows/pullrequest.yml

test_build_gpu_runtime.pyTest the runtime producer with synthetic inputs +170/-0

Test the runtime producer with synthetic inputs

• Covers dependency collection, relocation checks, source pins, successful archive creation, and suppression of an archive after smoke failure. Command doubles do not constitute live GPU verification.

deploy/ica-sidecar/test_build_gpu_runtime.py

test_gpu_assets.pyTest GPU archive integrity and extraction rules +105/-0

Test GPU archive integrity and extraction rules

• Exercises archive round trips and engine matching, plus rejection of missing files, links, traversal, duplicate entries, and inventory or hash tampering.

deploy/ica-sidecar/test_gpu_assets.py

ica_gpu_test.goValidate both GPU YAML examples offline +84/-0

Validate both GPU YAML examples offline

• Checks example parsing and GPU backend settings in an ORT build, then validates host-path handling with temporary dummy assets without loading native libraries.

sidecar/cmd/ica_gpu_test.go

ica_gpu_test.goTest GPU option validation and forwarding +44/-0

Test GPU option validation and forwarding

• Confirms GPU fields survive configuration resolution and invalid backend, device, and entity values are rejected. Also checks that GPU paths remain subject to serving-host validation.

sidecar/daemon/ica_gpu_test.go

gpu_test.goTest GPU defaults, conflicts, and host discovery +135/-0

Test GPU defaults, conflicts, and host discovery

• Checks that TurboOCR relaxes implicit CPU settings but rejects explicit conflicts, while CPU defaults remain unchanged. Covers ORT requirements and manifest-based GPU asset discovery without CPU assets.

sidecar/ica/gpu_test.go

Documentation (5) +431 / -0

GPU-RUNTIME.mdDocument GPU runtime production and publication +127/-0

Document GPU runtime production and publication

• Describes builder prerequisites, source and SDK pins, device and smoke gates, archive contents, and the manual publication workflow.

deploy/ica-sidecar/GPU-RUNTIME.md

GPU.mdDocument ICA GPU configuration and delivery +167/-0

Document ICA GPU configuration and delivery

• Explains GPU options, asset compatibility, image and bundle builds, native installation, and the limits of match-only masking and host checks.

deploy/ica-sidecar/GPU.md

config.gpu.cidr.yamlAdd a transparent-mode GPU configuration example +65/-0

Add a transparent-mode GPU configuration example

• Provides a TPROXY listener example using TurboOCR, CUDA NER, bundle-owned paths, and deployment-specific network placeholders.

deploy/ica-sidecar/examples/config.gpu.cidr.yaml

config.gpu.single.yamlAdd a fixed-upstream GPU configuration example +63/-0

Add a fixed-upstream GPU configuration example

• Provides a single-VDA example using TurboOCR, CUDA NER, bundle-owned paths, and placeholders for the upstream and admitted peer.

deploy/ica-sidecar/examples/config.gpu.single.yaml

README.mdLink ICA GPU delivery guidance from the sidecar README +9/-0

Link ICA GPU delivery guidance from the sidecar README

• Introduces the GPU image and native bundle and points readers to their configuration, asset, and installation documentation.

sidecar/README.md

Other (6) +756 / -0

ica-gpu-runtime.ymlAdd a manual GPU runtime build and publication workflow +128/-0

Add a manual GPU runtime build and publication workflow

• Builds and tests the pinned native runtime on a self-hosted GPU runner, retaining logs and uploading the verified archive. Optional production publication checks its checksum and refuses an existing version prefix.

.github/workflows/ica-gpu-runtime.yml

pullrequest-release.ymlAdd opt-in GPU artifacts to PR release previews +52/-0

Add opt-in GPU artifacts to PR release previews

• When both GPU asset variables are set, builds and pushes a preview GPU image and stages the native GPU bundle alongside the existing PR assets.

.github/workflows/pullrequest-release.yml

release.ymlGate GPU release outputs on a URL and checksum +53/-0

Gate GPU release outputs on a URL and checksum

• Rejects partially configured GPU inputs. When both variables are present, publishes the GPU image and stages a native bundle from the same runtime asset.

.github/workflows/release.yml

build-gpu-runtime.pyBuild and smoke-test a pinned native GPU runtime +342/-0

Build and smoke-test a pinned native GPU runtime

• Checks source revisions and installed SDK versions, builds TurboOCR and the framebuffer bridge, and collects relocatable runtime dependencies. Device tests and two smoke modes must pass before it creates a versioned, verified archive.

deploy/ica-sidecar/build-gpu-runtime.py

gpu-assets.pyDefine the GPU runtime archive contract +158/-0

Define the GPU runtime archive contract

• Packs, verifies, and unpacks archives with compatibility metadata and a SHA256 file inventory. Rejects unsafe paths, links, duplicate entries, missing assets, and mismatched contents.

deploy/ica-sidecar/gpu-assets.py

gpu-builder.example.jsonProvide a GPU builder profile template +23/-0

Provide a GPU builder profile template

• Shows the required source revision, SDK and model locations, device selection, and compatibility metadata for a prepared GPU build host.

deploy/ica-sidecar/gpu-builder.example.json

Description

install-gpu.sh installs the staged executable as /usr/local/bin/hoop-ica and configures the
service to run that name instead of preserving hoop-inspect. Every native GPU installation and
subsequent service restart therefore uses the alternate executable name and violates the component
naming contract.

Code

deploy/ica-sidecar/install-gpu.sh[R44-45]

+install -m 0755 "$here/hoop-ica" "$rel/bin/hoop-ica"
+install -m 0644 "$here/release.txt" "$rel/release.txt"

Relevance

●●● Strong

The installer directly violates the explicit hoop-inspect naming rule; installer safety findings are
consistently accepted.

PR-#1994

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The changed installer copies the staged executable into the bundle as hoop-ica and installs it
under that name, while the checklist requires the produced executable to retain the hoop-inspect
name.

Rule 3062565: Preserve hoop-inspect naming for binary, config directory, and environment variable deploy/ica-sidecar/install-gpu.sh[44-45]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

Issue description
The GPU installer renames the sidecar executable to `hoop-ica`, but the component contract requires the executable to remain named `hoop-inspect`.

Fix Focus Areas
- deploy/ica-sidecar/install-gpu.sh[44-45]
- deploy/ica-sidecar/install-gpu.sh[118-133]

Recommended Fix
Install the staged binary as `/usr/local/bin/hoop-inspect`, update the systemd `ExecStart` and related installer messages to use that name, and keep `/etc/hoop-inspect` as the configuration directory.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

Description

The new GPU preview steps inherit the ica-assets job's steps.changed.outputs.build gate, but
that gate's file list omits sidecar/daemon, where the new GPU configuration fields are defined. A
pull request that changes only those fields skips the GPU image and bundle builds, so those
artifacts are not checked against the configuration change before release.

Code

.github/workflows/pullrequest-release.yml[R810-811]

+      - name: Build and push (ica-gpu)
+        if: ${{ steps.changed.outputs.build == 'true' && vars.ICA_GPU_ASSETS_URL != '' && vars.ICA_GPU_ASSETS_SHA256 != '' }}

Relevance

●●● Strong

Recent workflow precedents accept safeguards preventing CI from silently skipping changed artifacts.

PR-#2073 PR-#1920

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The change detector checks Dockerfile.sidecar, deploy/ica-sidecar, sidecar/ica, sidecar/cmd and the
workflow, but not sidecar/daemon. The new GPU build step requires that detector's build == 'true'
output, while the new GPU option fields reside in sidecar/daemon/ica.go.

.github/workflows/pullrequest-release.yml[759-771] .github/workflows/pullrequest-release.yml[810-811] sidecar/daemon/ica.go[128-137]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
GPU image and bundle preview steps are skipped for pull requests that change only the daemon's GPU configuration fields because their shared change gate excludes `sidecar/daemon`.

## Fix Focus Areas
- .github/workflows/pullrequest-release.yml[759-771]
- .github/workflows/pullrequest-release.yml[810-811]

## Recommended Fix
Add `sidecar/daemon` to the file paths checked by the `Decide whether the bundle changed` step, so GPU configuration changes trigger the existing preview builds.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

Description

The new Build and push (ica-gpu), Export the ica GPU assets bundle and `Stage the ica GPU assets
bundle steps run inside the same job as the CPU bundle, before its Stage the ica assets bundle`
and upload-artifact steps. If anything in the GPU path fails while the two variables are set, the
job stops before dist-artifacts-ica-assets is uploaded, so the unchanged CPU bundle never reaches
publish-release. Causes include an unreachable or rotated runtime URL, a digest mismatch, or
gpu-assets.py unpack --engine rejecting the runtime because the libhoop-ica pin in go.mod
changed (even a patch bump). The CPU -ica image has already been pushed by then, and the PR
preview job in pullrequest-release.yml is coupled the same way.

Code

.github/workflows/release.yml[R1051-1054]

+      - name: Build and push (ica-gpu)
+        if: ${{ vars.ICA_GPU_ASSETS_URL != '' && vars.ICA_GPU_ASSETS_SHA256 != '' }}
+        uses: docker/build-push-action@v7
+        with:

Relevance

●● Moderate

Failure isolation is a legitimate release concern, but history lacks a close precedent and includes
rejected workflow hardening.

PR-#1731 PR-#1920

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The GPU steps come before the CPU staging and upload steps in the same job, and none of them sets
continue-on-error. The Dockerfile's native stage fails the build whenever the runtime manifest's
libhoop_ica differs from the engine version go list resolves from go.mod, so a routine engine
bump fails the whole release job until someone builds a new GPU runtime on the self-hosted GPU host.
In pullrequest-release.yml the GPU steps (lines 804-850) also sit before the job's
upload-artifact step (line 851).

.github/workflows/release.yml[1051-1101] Dockerfile.sidecar[448-454] deploy/ica-sidecar/gpu-assets.py[90-91] .github/workflows/pullrequest-release.yml[804-856]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The ica-gpu image and bundle steps share a job with the CPU ica-assets bundle and run before its staging and upload. Any GPU failure stops the CPU bundle from being uploaded and blocks the release. Possible failures: the runtime URL is down, the digest does not match, or the engine pin moved past the runtime's `libhoop_ica`.

## Fix Focus Areas
- .github/workflows/release.yml[1040-1101]
- .github/workflows/pullrequest-release.yml[804-856]

## Recommended Fix
Move the GPU image and bundle into a separate job that `needs` the same prerequisites. Have that job upload its own artifact, for example `dist-artifacts-ica-gpu-assets`, and make publish-release consume it only when the variables are set. Alternatively, upload the CPU `stage/` artifact before the GPU steps run. In either case, also check before the build that the runtime's engine pin matches `sidecar/ica/go.mod`, so a mismatch produces a clear message.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

Description

The step that checks ICA_GPU_ASSETS_URL/ICA_GPU_ASSETS_SHA256 runs only after `Build and push
(ica) has already pushed hoophq/hoopsidecar:<tag>-ica`. When only one variable is set the job
fails after that push, so the image is published but the matching assets artifact is never uploaded.

Code

.github/workflows/release.yml[R1042-1050]

+      - name: Check ICA GPU release inputs
+        env:
+          GPU_URL: ${{ vars.ICA_GPU_ASSETS_URL }}
+          GPU_SHA: ${{ vars.ICA_GPU_ASSETS_SHA256 }}
+        run: |
+          if { [ -n "$GPU_URL" ] && [ -z "$GPU_SHA" ]; } || { [ -z "$GPU_URL" ] && [ -n "$GPU_SHA" ]; }; then
+            echo "Set both ICA_GPU_ASSETS_URL and ICA_GPU_ASSETS_SHA256, or neither" >&2
+            exit 1
+          fi

Relevance

●●● Strong

Recent release precedent accepts preventing late failures after publishing artifacts.

PR-#1920 PR-#1595

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

In this job the -ica image push (around line 1002) comes before the new check at 1042. The
upload-artifact that publish-release consumes comes after it, so a failed check leaves a pushed
image with no artifact.

.github/workflows/release.yml[1002-1050] .github/workflows/release.yml[1093-1101]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The ICA GPU variable consistency check runs after the CPU `-ica` image push, so a misconfigured variable fails the release only after that image is already published.

## Fix Focus Areas
- .github/workflows/release.yml[1042-1050]

## Recommended Fix
Move the 'Check ICA GPU release inputs' step to the start of the job, before 'Build and push (ica)', or into an earlier gating job, so a variable mismatch fails before anything is pushed.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

💡 Did you know, you can turn on the rule miner and Qodo learns your standards from review history

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Review updated until commit 7f59fd8 🧠 Deep

Results up to commit 8eeb693 🧠 Deep

🐞 Bugs (5) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. A failed install can switch live assets 🐞 Bug ☼ Reliability

Description

install-gpu.sh repoints /opt/libhoop-ica/current before installing the executable and service
configuration. If either subsequent installation fails, the script exits with the live asset path
already pointing to the new GPU release while the executable or configuration remains from the
previous installation.

Code

deploy/ica-sidecar/install-gpu.sh[R116-119]

+ln -sfn "releases/${version}-gpu" /opt/libhoop-ica/current
+# Unlink before installing to allow replacement of a running ELF binary.
+install -m 0755 "$rel/bin/hoop-ica" /usr/local/bin/hoop-ica
+install -o root -g hoop -m 0640 "$rel/config.yaml" /etc/hoop-inspect/config.yaml

Relevance

●●● Strong

Recent installer precedent explicitly requires switching live paths only after all checks succeed.

PR-#1994

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Lines 116–119 switch the shared current path before two fallible install calls; set -e exits
on either failure without restoring the previous target. The generated service and GPU configuration
consume paths below current.

deploy/ica-sidecar/install-gpu.sh[5-5] deploy/ica-sidecar/install-gpu.sh[113-119] deploy/ica-sidecar/install-gpu.sh[123-137]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The installer changes the live asset symlink before all remaining installation steps have succeeded.
## Fix Focus Areas
- deploy/ica-sidecar/install-gpu.sh[113-141]
## Recommended Fix
Stage the executable, configuration, and unit changes first. Switch `current` only after those writes succeed, and retain or restore the previous symlink if any later activation step fails.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

---

2. A failed GPU install blocks every retry 🐞 Bug ☼ Reliability

Description

install-gpu.sh creates and fills $rel=/opt/libhoop-ica/releases/${version}-gpu before the YAML
preflight, the native dependency check, the smoke test and -validate run, and never removes it
when one of those fails. A rerun of the same bundle stops at the [[ ! -e $rel ]] guard because the
version comes from the bundle's release.txt, so an operator who fixes a typo in the config or a
GPU permission has to find and delete the partial directory by hand.

Code

deploy/ica-sidecar/install-gpu.sh[R39-42]

+rel=/opt/libhoop-ica/releases/${version}-gpu
+[[ ! -e $rel ]] || die "$rel already exists; use a new release version (existing assets are never overwritten)"
+install -d -m 0755 "$rel/bin" "$rel/models"
+cp -a "$here/gpu" "$rel/gpu"

Relevance

●●● Strong

Recent installer precedent accepts staged validation and cleanup to prevent failed upgrades blocking
retries.

PR-#1994

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Line 40 refuses an existing $rel. Lines 41-48 create it and copy assets into it, and every check
after that (52-76 preflight, 94-106 ldd/smoke/validate) exits through die or a non-zero status
under set -e. No trap or cleanup ever removes $rel. The version is read from the bundle at line
23, so the operator cannot choose a new one.

deploy/ica-sidecar/install-gpu.sh[23-48] deploy/ica-sidecar/install-gpu.sh[94-106]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
install-gpu.sh copies assets into the final versioned release directory before running its checks. A failed check leaves that directory behind, and the 'already exists' guard then blocks every retry of the same bundle.

## Fix Focus Areas
- deploy/ica-sidecar/install-gpu.sh[39-106]

## Recommended Fix
Stage into a temporary sibling such as `/opt/libhoop-ica/releases/.${version}-gpu.staging.$$`, and register `trap 'rm -rf "$staging"' EXIT` to remove it. Run the preflight, ldd, smoke and validate checks against the staged paths. Only after every check passes, `mv` the staging directory to `$rel`, clear the trap, and then switch `current`. Alternatively, put an `rm -rf "$rel"` in an ERR/EXIT trap that is disarmed once validation succeeds.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

---

3. GPU runtime archive stores large libraries several times 🐞 Bug ➹ Performance

Description

collect() adds every *.so* file matching GPU_LIBRARY from the SDK directories to the queue
under its own file name, and copy_file follows symlinks, so libcudnn.so, libcudnn.so.9 and
libcudnn.so.9.x.y each become a separate full copy. The same happens to the cuBLAS, TensorRT and
other CUDA libraries, many of which are hundreds of MB to over 1 GB, so the runtime tarball and
every installed release grow by several GB.

Code

deploy/ica-sidecar/build-gpu-runtime.py[R135-138]

+    for directory in search_dirs:
+        for path in sorted(directory.glob("*.so*")):
+            if GPU_LIBRARY.match(path.name) and not HOST_LIBRARY.match(path.name):
+                queue.append((path.name, path))

Relevance

●●● Strong

The sweep dereferences library aliases, causing avoidable multi-gigabyte archive duplication.

PR-#1994

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The sweep queues every glob match by file name. copy_file resolves symlinks and writes a separate
regular file for each destination name (lines 115-127). gpu-assets.py pack forbids symlinks, so
the unversioned link, the SONAME link and the fully versioned file are all stored as full copies.
install-gpu.sh then copies the whole gpu tree into every release directory.

deploy/ica-sidecar/build-gpu-runtime.py[130-154] deploy/ica-sidecar/build-gpu-runtime.py[114-127] deploy/ica-sidecar/gpu-assets.py[55-61]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The GPU-library sweep in collect() copies every symlink alias of each CUDA/cuDNN/TensorRT library as a separate full file, which multiplies the archive size.

## Fix Focus Areas
- deploy/ica-sidecar/build-gpu-runtime.py[130-154]

## Recommended Fix
In the sweep, skip any path that is a symlink and is not the library's SONAME. Read the SONAME with `patchelf --print-soname` or `objdump -p`, and queue each real file once under its SONAME, which is the name the loader and dlopen use. Drop the unversioned development names (`libfoo.so`) and the fully versioned file names, and track resolved paths so each real file is copied once.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

---

View medium (2)

4. GPU asset override stays out of sample ⊘ Outdated

Description

HostCheck reads HOOP_ICA_GPU_ASSETS directly, but the repository sample omits the key. Operators
configuring the GPU asset root therefore have no sample entry to discover or set this override.

Code

sidecar/ica/ica.go[R47-50]

+	gpuRoot := os.Getenv("HOOP_ICA_GPU_ASSETS")
+	if gpuRoot == "" {
+		gpuRoot = defaultGPUAssets
+	}

Relevance

●●● Strong

The repository recently accepted mirroring newly introduced environment variables in .env.sample.

PR-#2008

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The changed ICA host-check code introduces a direct read of HOOP_ICA_GPU_ASSETS, while the root
environment sample contains no matching key.

Rule 1800927: New env vars must be mirrored in .env.sample sidecar/ica/ica.go[47-54] .env.sample[1-80]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

Issue description
`HostCheck` introduces the `HOOP_ICA_GPU_ASSETS` environment variable, but the root `.env.sample` does not list it.

Fix Focus Areas
- sidecar/ica/ica.go[47-50]
- .env.sample[1-179]

Recommended Fix
Add `HOOP_ICA_GPU_ASSETS=` to `.env.sample`, with a concise comment explaining that it overrides the default GPU asset root for ICA GPU deployments.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

5. GPU builds reject valid custom libraries 🐞 Bug ≡ Correctness

Description

build() calls ort_version() before configuring the SDK and custom runtime library directories,
and that probe loads ONNX Runtime in the current Python process. When the library depends on shared
objects available only through configured runtime_library_dirs, the build fails at version
detection before the later subprocess environment can use those directories.

Code

deploy/ica-sidecar/build-gpu-runtime.py[223]

+        "ort": ort_version(ortlib / "libonnxruntime.so"),

Relevance

●● Moderate

Documented custom dependency directories are configured after ctypes loading; fixing this requires
process-loading changes.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

ort_version() uses ctypes.CDLL in the builder process. The configured SDK and
runtime_library_dirs paths are assembled into a subprocess-only environment after the call,
although custom dependency directories are a documented builder input.

deploy/ica-sidecar/build-gpu-runtime.py[86-94] deploy/ica-sidecar/build-gpu-runtime.py[217-234] deploy/ica-sidecar/GPU-RUNTIME.md[26-32]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The ONNX Runtime version probe runs before configured dependency directories are available to its loader.
## Fix Focus Areas
- deploy/ica-sidecar/build-gpu-runtime.py[86-94]
- deploy/ica-sidecar/build-gpu-runtime.py[217-234]
## Recommended Fix
Construct the library search environment before probing ORT, and perform the version probe in a subprocess started with that environment. Add a test using an ORT library with a dependency in a configured custom directory.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

---

6. A half-configured release still pushes the CPU image 🐞 Bug ☼ Reliability

Description

The step that checks ICA_GPU_ASSETS_URL/ICA_GPU_ASSETS_SHA256 runs only after `Build and push
(ica) has already pushed hoophq/hoopsidecar:<tag>-ica`. When only one variable is set the job
fails after that push, so the image is published but the matching assets artifact is never uploaded.

Code

.github/workflows/release.yml[R1042-1050]

+      - name: Check ICA GPU release inputs
+        env:
+          GPU_URL: ${{ vars.ICA_GPU_ASSETS_URL }}
+          GPU_SHA: ${{ vars.ICA_GPU_ASSETS_SHA256 }}
+        run: |
+          if { [ -n "$GPU_URL" ] && [ -z "$GPU_SHA" ]; } || { [ -z "$GPU_URL" ] && [ -n "$GPU_SHA" ]; }; then
+            echo "Set both ICA_GPU_ASSETS_URL and ICA_GPU_ASSETS_SHA256, or neither" >&2
+            exit 1
+          fi

Relevance

●●● Strong

Recent release precedent accepts preventing late failures after publishing artifacts.

PR-#1920 PR-#1595

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

In this job the -ica image push (around line 1002) comes before the new check at 1042. The
upload-artifact that publish-release consumes comes after it, so a failed check leaves a pushed
image with no artifact.

.github/workflows/release.yml[1002-1050] .github/workflows/release.yml[1093-1101]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The ICA GPU variable consistency check runs after the CPU `-ica` image push, so a misconfigured variable fails the release only after that image is already published.

## Fix Focus Areas
- .github/workflows/release.yml[1042-1050]

## Recommended Fix
Move the 'Check ICA GPU release inputs' step to the start of the job, before 'Build and push (ica)', or into an earlier gating job, so a variable mismatch fails before anything is pushed.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

---