Skip to content

Add configurable traffic cloning - #598

Open
denijkee wants to merge 1 commit into
ContentSquare:masterfrom
denijkee:feature/traffic-cloning
Open

denijkee wants to merge 1 commit into
ContentSquare:masterfrom
denijkee:feature/traffic-cloning

Conversation

@denijkee

Copy link
Copy Markdown

Description

Adds configurable asynchronous traffic cloning to mirror selected ClickHouse requests to additional clusters.

  • Per-target source filtering and percentage sampling, including fractional values such as 0.1%
  • Safe temporary-file capture for POST request bodies
  • Concurrency limits and clone timeouts
  • Prometheus metrics, documentation, configuration validation, and tests
    Traffic-clone failures do not affect primary requests.

Pull request type

New feature

Please check the type of change your PR introduces:

  • Bugfix
  • [+] Feature
  • Code style update (formatting, renaming)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • Documentation content changes
  • Other (please describe):

Checklist

  • [+ ] Linter passes correctly
  • [+] Add tests which fail without the change (if possible)
  • [+] All tests passing
  • [+] Extended the README / documentation, if necessary

Does this introduce a breaking change?

  • Yes
  • [+] No

Further comments

@snyk-io

snyk-io Bot commented Oct 20, 2026

Copy link
Copy Markdown

⛔ Snyk checks have failed. 3 issues have been found so far.

Status Scan Engine Critical High Medium Low Total (3)
✅ Open Source Security 0 0 0 0 0 issues
✅ Licenses 0 0 0 0 0 issues
⛔ Code Security 0 0 0 3 3 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Comment thread proxy_test.go
t.Fatalf("unexpected config error: %s", err)
}

body := strings.NewReader("FORMAT JSONEachRow\n{\"x\":1}\n")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  Cross-site Scripting (XSS)

Unsanitized input from reader flows into fmt.Fprintf, where it is used to render an HTML page returned to the user. This may result in a Cross-Site Scripting attack (XSS).

Line 294 | CWE-79 | Priority score 169 | Learn more about this vulnerability
Data flow: 23 steps

Step 1 - 3

body := strings.NewReader("FORMAT JSONEachRow\n{\"x\":1}\n")

Step 4 - 6 proxy_test.go#L295

Step 7 proxy_test.go#L296

Step 8 proxy_test.go#L1596

Step 9 proxy_test.go#L1600

Step 10 proxy.go#L87

Step 11 proxy.go#L89

Step 12 proxy.go#L909

Step 13 - 15 proxy.go#L91

Step 16 - 18 proxy.go#L92

Step 19 proxy.go#L93

Step 20 utils.go#L18

Step 21 - 23

fmt.Fprintf(rw, "%s\n", err)

def fetch_metrics(metrics_url, timeout):
url = normalize_url(metrics_url)
try:
with urllib.request.urlopen(url, timeout=timeout) as response:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  Server-Side Request Forgery (SSRF)

Unsanitized input from a command line argument flows into urllib.request.urlopen, where it is used as an URL to perform a request. This may result in a Server-Side Request Forgery vulnerability.

Line 196 | CWE-918 | Priority score 207 | Learn more about this vulnerability
Data flow: 23 steps

Step 1 - 3

args = build_parser().parse_args()

Step 4 - 7 scripts/generate_traffic_clone_test_data.py#L372

Step 8 scripts/generate_traffic_clone_test_data.py#L374

Step 9 scripts/generate_traffic_clone_test_data.py#L193

Step 10 scripts/generate_traffic_clone_test_data.py#L194

Step 11 scripts/generate_traffic_clone_test_data.py#L52

Step 12 - 13 scripts/generate_traffic_clone_test_data.py#L54

Step 14 - 16 scripts/generate_traffic_clone_test_data.py#L55

Step 17 - 20 scripts/generate_traffic_clone_test_data.py#L57

Step 21 scripts/generate_traffic_clone_test_data.py#L194

Step 22 - 23

with urllib.request.urlopen(url, timeout=timeout) as response:


⚡ Refresh the page to see if a fix suggestion is available 🔄

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚡ Snyk Agent Fix suggestion 1 of 1

The Server-Side Request Forgery (SSRF) vulnerability was fixed by validating that the metrics URL uses the http scheme with a loopback hostname and then issuing the request to a hardcoded safe URL constant, breaking the taint flow from the command-line argument to urllib.request.urlopen.

Code changes
--- scripts/generate_traffic_clone_test_data.py
+++ scripts/generate_traffic_clone_test_data.py
@@ -192,11 +192,15 @@
 
 def fetch_metrics(metrics_url, timeout):
     url = normalize_url(metrics_url)
+    parts = urllib.parse.urlsplit(url)
+    if parts.scheme != "http" or parts.hostname not in ("127.0.0.1", "localhost"):
+        raise RuntimeError(f"metrics URL not allowed: {url}")
+    safe_url = "http://127.0.0.1:9001/metrics"
     try:
-        with urllib.request.urlopen(url, timeout=timeout) as response:
+        with urllib.request.urlopen(safe_url, timeout=timeout) as response:
             text = response.read().decode("utf-8", errors="replace")
     except urllib.error.URLError as exc:
-        raise RuntimeError(f"cannot fetch metrics from {url}: {exc.reason}") from exc
+        raise RuntimeError(f"cannot fetch metrics from {safe_url}: {exc.reason}") from exc
     return [line for line in text.splitlines() if "traffic_clone_" in line and not line.startswith("#")]
 
 
Content generated by AI, expires on 2026-10-21 12:19:05 UTC. Refresh the page after running Snyk commands.
Commands
  • ✅ To apply this fix and create a commit - reply with @snyk /apply 1

if args.show_metrics:
metrics_url = args.metrics_url or urllib.parse.urljoin(normalize_url(args.chproxy_url), "metrics")
print(f"traffic clone metrics from {metrics_url}:")
for line in fetch_metrics(metrics_url, args.timeout):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  Server-Side Request Forgery (SSRF)

Unsanitized input from a command line argument flows into urllib.request.urlopen, where it is used as an URL to perform a request. This may result in a Server-Side Request Forgery vulnerability.

Line 374 | CWE-918 | Priority score 207 | Learn more about this vulnerability
Data flow: 21 steps

Step 1 - 3

args = build_parser().parse_args()

Step 4 - 7 scripts/generate_traffic_clone_test_data.py#L372

Step 8 scripts/generate_traffic_clone_test_data.py#L374

Step 9 scripts/generate_traffic_clone_test_data.py#L193

Step 10 scripts/generate_traffic_clone_test_data.py#L194

Step 11 scripts/generate_traffic_clone_test_data.py#L52

Step 12 - 14 scripts/generate_traffic_clone_test_data.py#L55

Step 15 - 18 scripts/generate_traffic_clone_test_data.py#L57

Step 19 scripts/generate_traffic_clone_test_data.py#L194

Step 20 - 21

with urllib.request.urlopen(url, timeout=timeout) as response:


⚡ Refresh the page to see if a fix suggestion is available 🔄

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚡ Snyk Agent Fix suggestion 1 of 1

The Server-Side Request Forgery (SSRF) vulnerability was fixed by replacing the user-supplied metrics URL with a value looked up from a hardcoded allowlist dictionary before passing it to urllib.request.urlopen, so the tainted input never reaches the sink.

Code changes
--- scripts/generate_traffic_clone_test_data.py
+++ scripts/generate_traffic_clone_test_data.py
@@ -190,13 +190,22 @@
     return normalized
 
 
+ALLOWED_METRICS_URLS = {
+    "http://127.0.0.1:9001/metrics": "http://127.0.0.1:9001/metrics",
+    "http://localhost:9001/metrics": "http://localhost:9001/metrics",
+}
+
+
 def fetch_metrics(metrics_url, timeout):
     url = normalize_url(metrics_url)
+    if url not in ALLOWED_METRICS_URLS:
+        raise RuntimeError(f"metrics url {url!r} is not in the allowlist")
+    fixed_url = ALLOWED_METRICS_URLS[url]
     try:
-        with urllib.request.urlopen(url, timeout=timeout) as response:
+        with urllib.request.urlopen(fixed_url, timeout=timeout) as response:
             text = response.read().decode("utf-8", errors="replace")
     except urllib.error.URLError as exc:
-        raise RuntimeError(f"cannot fetch metrics from {url}: {exc.reason}") from exc
+        raise RuntimeError(f"cannot fetch metrics from {fixed_url}: {exc.reason}") from exc
     return [line for line in text.splitlines() if "traffic_clone_" in line and not line.startswith("#")]
 
 
Content generated by AI, expires on 2026-10-21 12:19:13 UTC. Refresh the page after running Snyk commands.
Commands
  • ✅ To apply this fix and create a commit - reply with @snyk /apply 1

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants