ENG-817: sidecar http lane reads the SQL in Databricks JSON requests

Open

rogefmwants to merge intofrom

IMPORTANT

Every PR MUST have exactly one of these labels, the merge is blocked otherwise:

major / minor / patch publishes a new release on merge with the corresponding semver bump.

skip-release merges without publishing a release (use for docs, CI changes, refactors, etc.).

📝 Description

An http.sql block makes an http lane read the SQL inside JSON request bodies: Databricks' Statement Execution API (statement) and the workspace SQL MCP server (tools/call of execute_sql). Each SQL string becomes a statement of its own, so operation and table rules, the analyzer and the audit trail judge it as SQL. Before, a DROP passed a lane written to refuse it.

It also fixes two defects on the same path. A JSON response too large to mask is now refused; before, it was forwarded unmasked or it crashed the process. An operation rule naming unknown now matches client statements only; before, it refused every response.

📣 User-facing impact

The sidecar can block destructive SQL that agents send to Databricks through its SQL API and MCP server, and it refuses result downloads that would skip masking. This release also ships the Spark SQL dialect (ENG-823, #2012).

On an http lane with masking, a result the Sidecar cannot mask, such as a ClickHouse FORMAT JSON result over 8 MiB, now gets a 403 instead of crashing the Sidecar. A gzip result over 16 MiB once inflated is refused instead of passing unmasked.

🔗 Related Issue

ENG-817. It absorbs ENG-822, ENG-819 and ENG-893, and fixes the first half of ENG-545.

🚀 Type of Change

📋 Changes Made

sidecar/jsonsql (new, protocol-neutral): reads the SQL a JSON document carries, at a dotted path or in a JSON-RPC tools/call, and classifies it in the dialect. It knows no protocol, so another protocol that carries JSON can reuse it.
sidecar/codec/http/sqlbody.go (new): the HTTP seam over jsonsql. It reads the SQL for configured routes. The route matches the resource that cleanResource (ENG-895) already resolved, because Databricks routes ., .., // and \ like the canonical path; the SQL route also drops a trailing slash. An MCP tools/call carries SQL only for the named tools, and a JSON-RPC batch gives one statement per call. refuse rejects field=value pairs, and a non-string value at such a field, before forwarding. A body with no readable SQL is unknown. Only the bodies of the configured routes are kept.
sidecar/daemon/httpsql.go, analyzer.go, config.go, httpcodec.go, schema.json: the http.sql config block (cap:"http_sql", UI tags) and its validation. It is refused together with an analyzer that holds for approval (see Additional Notes).
sidecar/analyzer: the SQL renders as SQL, at one model call per query. A resource trigger matches the SQL by the route that carried it (http.sql_resource).
sidecar/policy/policy.go: an operation rule naming unknown never matches a response, on every protocol (ENG-893, ENG-545). The operations scope of other rule types is unchanged.
libhoop pinned to fd5e0b4 on libhoop main (hoophq/libhoop#141, merged): RefuseUnmaskable, and Flush(nil) discards (the ENG-822 crash). sidecar/codec/http/http.go turns it on.
sidecar/gate, sidecar/proxy: a response refused after part of it reached the client is cut, without a 403 inside it (Decision.NoFrame). A close-delimited body refused at connection end gets the 403 (FlushResponseChecked). Both came from the Qodo review of libhoop#141. The gate asks the codec through two optional interfaces (responseStarter, checkedFlusher) and holds no protocol rule; a codec without them keeps the deny frame.
sidecar/README.md: "SQL in request bodies: Databricks", the Spark dialect's config key, the masking paragraph, and unknown on responses.

🧪 Testing

Result: a DROP sent through the Statement Execution API or the workspace SQL MCP server is refused before Databricks runs it. Emails stay masked, and a request for results that would skip the sidecar is refused. Tested on acdce66c9 against origin/main 2c2f6ba45, against a real Databricks trial workspace, with no license (1 guardrail, 1 mask).

1. A DROP through the Statement Execution API and through MCP

Bottom of each panel. On main both DROPs return 200 and tables after: []. On the branch both return 403, and the tables remain.
SELECT, EXTERNAL_LINKS. Main answers with a link to cloud storage, where the rows are not masked. The branch refuses it and names INLINE.
The SELECT lines. Emails are redacted on both sides.

2. A JSON result too large to mask

Same config and stand-in upstream on both sides, only the binary changes. On main, a 17 MiB gzip result goes out with 186,843 emails unmasked, and a 9 MiB plain result gets 403 and then the process dies (nil-pointer panic). The branch refuses both with 403 and masks the next response. A real workspace sends inline results in chunks of about 5.5 MB, so it does not reach this path.

3. The fail-closed unknown rule on an http lane

Both sides run the same rule. Main refuses every response, a SELECT included, but still runs the EXECUTE IMMEDIATE DROP. The branch lets reads through and refuses the hidden DROP before it leaves.

What it solves: Databricks customers who run the sidecar alone, on the free tier, get a deterministic read-only guardrail on the SQL their agents send through the Statement Execution API and the workspace SQL MCP server. Setup: hoophq/documentation#244.

Definition of done (independent QA, PASS on round 3):

Tests that guard it (each new behavior fails on main):

proxy: TestHTTPSQLBodyIsJudgedBeforeItIsForwarded, TestHTTPSQLNonCanonicalPathIsJudged, TestHTTPSQLRefusesAPaddedBody, TestHTTPSQLRefusesExternalLinks, TestHTTPSQLLaneMasksDatabricksResults, TestHTTPUnknownRuleJudgesRequestsNotResponses, TestOversizedJSONResponseIsRefusedAndTheRelayKeepsServing, TestGzipJSONInflatingPastTheMaskLimitIsRefused, TestChunkedStreamRefusedMidResponseIsCut, TestCloseDelimitedGzipTooLargeToMaskGets403.
jsonsql: TestReadFindsTheSQL, TestRefuse, TestToolArgument. codec/http: TestSQLBody* (including TestSQLBodyIgnoresWebSocketMessages). analyzer: TestHTTPSQLRequestIsOneModelCall, TestHTTPBuilderRendersBodySQLAsSQL. daemon: TestHTTPSQL*. policy: TestUnknownRuleDoesNotMatchResponses, TestUnknownScopeStillReachesResponses, TestValidResourcePattern. gate: TestARefusedResponseKeepsTheOneBeforeIt.

Re-run:

make test-sidecar && make test-sidecar-compat
cd sidecar && go test ./proxy ./codec/http ./analyzer ./daemon ./policy -run 'SQL|Unknown|Oversized|Gzip'

Against a workspace: follow the docs page (hoophq/documentation#244). Its config and Verify commands are the ones this test used.

Test Configuration:

Browser(s): n/a
OS: macOS 26 (sidecar binaries built with CGO_ENABLED=0)

Tests performed:

Unit tests pass
Integration tests pass
Manual testing completed

📸 Screenshots (if applicable)

See Testing.

✅ Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing unit tests pass locally with my changes
I have checked my code and corrected any misspellings

📄 Additional Notes

Rebased 2026-10-29 on main 03b28d94b, then on 2e9756300: only the libhoop lines of client/go.sum conflicted with #2090, and no other commit changed (git range-diff). The first rebase's only conflicts were the libhoop pins; dadd269b5 re-pins libhoop and makes the SQL route reuse ENG-895's resource cleaning instead of its own. make test-sidecar and make test-sidecar-compat pass on it.
After QA: 53e938e65 and 9b2e3ba33 fix the two Qodo findings on libhoop#141, and cfc624081 moves the JSON reading into jsonsql with no change in behavior (the existing tests pass unchanged). All three came after the round-3 PASS. make test-sidecar passes on cfc624081 and make test-sidecar-compat on 9b2e3ba33 (rebased on main 99eecdb8d); the new tests fail on acdce66c9.
Qodo review on this PR: 7838b17b4. A jsonrpc member hid a body's SQL (a real bypass: the Statement Execution API runs such a body); now it counts only where the path is under params.arguments, which needs tools. Tied patterns naming two paths refuse the request. Only the operation rule exempts responses from unknown. A later Qodo pass found that a WebSocket message on a SQL route was read as a request body; 2956e59f1 skips it. A third pass, d1482774a: responses finished before a refused one go out ahead of the deny frame (on main they were dropped with br/zstd too), a route pattern that matches nothing is refused at load, and a trigger on the carrying request's method selects its SQL. Through the sidecar on the workspace, the DROP with jsonrpc now gets 403. The new tests fail on dadd269b5.
Merge order: hoophq/libhoop#141 is merged; 3a140b17c pins its main commit fd5e0b4 in the 9 go.mod files, the same tree this PR tested. Merge hoophq/documentation#244 only after use.hoop.dev runs the release.
Feature flag: none. http.sql is opt-in, and a lane without it behaves as today. The two behavior changes fix a leak and a crash; a flag would keep both.
Sidecar ↔ Control Plane: the block carries cap:"http_sql". An older sidecar refuses the key (unknown field "sql"), and the plane does not serve it to 1.267.x (checked by QA). The other way: a sidecar with http.sql and a token does not start on a plane without this change (refused the imported config: ... unknown field "sql", seen on use.hoop.dev). Upgrade the plane first (ADR-0022). On a plane built from this branch, the token flow runs the lane on the real workspace. No gateway ↔ agent change.
Departures from the approved design note:
- a flat block, because the listener form cannot render a list of objects;
- the request stays a statement of its own, which identity and http_resource rules need;
- an approval-holding analyzer is refused with http.sql for now, because an approval on body SQL would never be spent (noted on ENG-818);
- route paths are cleaned.
Risks: an operation rule naming unknown now never matches a response on any protocol. A Postgres or MySQL lane with an unknown rule stops refusing its responses, which is the ENG-545 fix. A JSON body too large to mask now gets a 403.
Follow-ups: ENG-818 (the Control Plane form). ENG-895 merged in #2058; this branch reuses its resource cleaning.