Skip to content

feat: add HTTP/HTTPS forward proxy support with HTTP CONNECT tunneling - #21

Merged
sni merged 1 commit into
ConSol-Monitoring:masterfrom
gpineda-dev:add-http-proxy-support
Nov 20, 2026
Merged

sni merged 1 commit into
ConSol-Monitoring:masterfrom
gpineda-dev:add-http-proxy-support

Conversation

@gpineda-dev

@gpineda-dev gpineda-dev commented Nov 5, 2026 •

Copy link
Copy Markdown

Hello,
First, thank you for SNClient and check_snclient -- it's a fantastic monitoring ecosystem that we rely on daily.

Also, it is truly great and reassuring to see the recent momentum around OMD-Labs (website revamp, renewed commit activity)! This direction is very exciting, especially as it paves the way for modern hybrid monitoring architectures bridging Naemon with Prometheus (check_prometheus), VictoriaMetrics, and Nagflux (eg. long term metrics vs metric lake).

Context & Motivation

In our monitoring setup with Naemon / OMD-Labs, we regularly need to monitor isolated servers located in private subnets or DMZs that cannot be contacted directly from the monitoring core.

While SNClient allows configuring cascaded checks (e.g. Naemon -> SNClient on Gateway A -> check_snclient to Server B), we found that routing checks through a standard L4 Forward Proxy (HTTP CONNECT) brings major architectural, security, and operational benefits:

  1. Zero Secret Leakage (Principle of Least Privilege): Gateway A never sees or stores Server B's password. Only Naemon holds the credentials.
  2. End-to-End TLS / mTLS Integrity: HTTP CONNECT creates a raw Layer-4 TCP tunnel. The proxy does not terminate or decrypt TLS; certificate validation and mTLS happen directly between Naemon and the target agent.
  3. Auditability & Security Compliance: The forward proxy generates standard access.log records (source IP, target IP:port, HTTP CONNECT status, bytes transferred) suitable for SIEM auditing, and proxy ACLs can restrict traffic strictly to port 8443.
  4. Clean Naemon Network Topology (parents): Gateway A is configured as the native parents host for Server B. If the gateway goes down, Naemon marks Server B as UNREACHABLE instead of DOWN, naturally preventing alert storms.
  5. Zero Maintenance on the Gateway: No need to maintain snclient.ini configs or wrapped command definitions on intermediate gateways; all monitoring logic remains centralized in Naemon.

Implementation: Pure Go Standard Library

I wanted to keep this change as minimal, non-intrusive, and robust as possible.
No external dependencies were added. We simply leveraged Go's native net/http and crypto/tls transport capabilities:

  • Direct by default (Zero side-effects): If -proxy is omitted, check_snclient connects directly without reading ambient environment variables, ensuring existing checks are never affected if HTTP_PROXY is set on the system.
  • Explicit Proxy URL: -proxy <url> supports http://, https://, and socks5:// (with auto-prepending of http:// if omitted), as well as Basic Authentication (http://user:pass@proxy:port).
  • Opt-in Environment Proxy: Passing -proxy env (or CHECK_SNCLIENT_PROXY=env) explicitly activates Go's standard http.ProxyFromEnvironment (HTTP_PROXY, HTTPS_PROXY, NO_PROXY).
  • Explicit Bypass: -proxy none or -proxy direct forces direct connections even if proxy environment variables exist.

Verification: E2E Testbed Built with Gemini Flash 3.8

Beyond Go unit tests (go test ./...), I have been seeking for real-world guarantees that HTTP CONNECT tunneling and 407 Proxy Authentication Required challenges work seamlessly against real daemons (not synthetic in-memory mocks).

To achieve reproducible black-box testing, I designed an E2E test forge pair-programming with Gemini Flash 3.8. It orchestrates a live SNClient daemon and Tinyproxy forward proxy via Ansible playbooks, running Testinfra assertions with live command tracing:

gpineda@thinkpad-e15g2:~/Documents/gpineda-dev/check_snclient/forge$ uv run forge up --backend docker
==> Starting Docker testbed container...
33ddaa7826b94afb66e0a0dd08a0ddf3e61e1911fbf136f54949a046955c2202

PLAY [Synchronize binary slot] **************************************************************************************************************************************************************************************************************************************************************************

TASK [Copy binary to container slot] ********************************************************************************************************************************************************************************************************************************************************************
[WARNING]: Platform linux on host snclient-testbed is using the discovered Python interpreter at /usr/bin/python3.11, but future installation of another Python interpreter could change the meaning of that path. See https://docs.ansible.com/ansible-
core/2.17/reference_appendices/interpreter_discovery.html for more information.
changed: [snclient-testbed]

TASK [Update active container symlink] ******************************************************************************************************************************************************************************************************************************************************************
changed: [snclient-testbed]

TASK [Update active host symlink] ***********************************************************************************************************************************************************************************************************************************************************************
skipping: [snclient-testbed]

PLAY RECAP **********************************************************************************************************************************************************************************************************************************************************************************************
snclient-testbed           : ok=2    changed=2    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   

                                 Testbed Components Status                                  
┏━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Component ┃ State   ┃ Port  ┃ Socket Health ┃ URL                                        ┃
┡━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ SNClient  │ RUNNING │ 18443 │ LISTENING     │ https://127.0.0.1:18443                    │
│ Tinyproxy │ RUNNING │ 18888 │ LISTENING     │ http://proxyuser:proxypass@127.0.0.1:18888 │
└───────────┴─────────┴───────┴───────────────┴────────────────────────────────────────────┘

              Testbed Topology & Placement Matrix              
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Component       ┃ Placement     ┃ Endpoint / Scope          ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ SNClient Agent  │ docker        │ docker://snclient-testbed │
│ Tinyproxy Proxy │ docker        │ docker://snclient-testbed │
│ Check Caller    │ docker (auto) │ docker://snclient-testbed │
└─────────────────┴───────────────┴───────────────────────────┘
gpineda@thinkpad-e15g2:~/Documents/gpineda-dev/check_snclient/forge$ uv run forge attach /home/gpineda/Documents/gpineda-dev/tmp/check_snclient/check_snclient

PLAY [Synchronize binary slot] **************************************************************************************************************************************************************************************************************************************************************************

TASK [Copy binary to container slot] ********************************************************************************************************************************************************************************************************************************************************************
[WARNING]: Platform linux on host snclient-testbed is using the discovered Python interpreter at /usr/bin/python3.11, but future installation of another Python interpreter could change the meaning of that path. See https://docs.ansible.com/ansible-
core/2.17/reference_appendices/interpreter_discovery.html for more information.
ok: [snclient-testbed]

TASK [Update active container symlink] ******************************************************************************************************************************************************************************************************************************************************************
ok: [snclient-testbed]

TASK [Update active host symlink] ***********************************************************************************************************************************************************************************************************************************************************************
skipping: [snclient-testbed]

PLAY RECAP **********************************************************************************************************************************************************************************************************************************************************************************************
snclient-testbed           : ok=2    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   

✔ Attached check_snclient -> /home/gpineda/Documents/gpineda-dev/tmp/check_snclient/check_snclient (+proxy)
gpineda@thinkpad-e15g2:~/Documents/gpineda-dev/check_snclient/forge$ uv run forge test
==> Running Testinfra integration tests (caller: auto)...
========================================================================================================================================== test session starts ==========================================================================================================================================
platform linux -- Python 3.10.19, pytest-9.2.0, pluggy-1.7.0 -- /home/gpineda/Documents/gpineda-dev/check_snclient/forge/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/gpineda/Documents/gpineda-dev/check_snclient/forge
configfile: pyproject.toml
plugins: testinfra-10.2.2
collected 6 items                                                                                                                                                                                                                                                                                       

tests/test_checks.py::TestSNClientIntegration::test_alive_check[docker://snclient-testbed] 
  ▶ Exec : /usr/bin/check_snclient -k -u https://127.0.0.1:18443 -p secret_forge_pass
  ↳ [RC=0] OK - REST API reachable on https://127.0.0.1:18443
PASSED
tests/test_checks.py::TestSNClientIntegration::test_check_cpu[docker://snclient-testbed] 
  ▶ Exec : /usr/bin/check_snclient -k -u https://127.0.0.1:18443 -p secret_forge_pass check_cpu
  ↳ [RC=0] OK - CPU load is ok. 5% on 8 cores |'total 5s'=7.847%;80;90;; 'total 1m'=5.975%;80;90;; 'total 5m'=5.975%;80;90;;
PASSED
tests/test_checks.py::TestSNClientIntegration::test_check_uptime[docker://snclient-testbed] 
  ▶ Exec : /usr/bin/check_snclient -k -u https://127.0.0.1:18443 -p secret_forge_pass check_uptime
  ↳ [RC=1] WARNING - uptime: 1d 12:34h, boot: 2026-11-04 10:09:19 UTC |'uptime'=131642s;172800:;86400:;;
PASSED
tests/test_checks.py::TestSNClientIntegration::test_invalid_password[docker://snclient-testbed] 
  ▶ Exec : /usr/bin/check_snclient -k -u https://127.0.0.1:18443 -p bad_password check_cpu
  ↳ [RC=3] RESPONSE-ERROR: http request failed: 403 Forbidden
PASSED
tests/test_checks.py::TestSNClientIntegration::test_proxy_traversal_success[docker://snclient-testbed] 
  ▶ Exec : /usr/bin/check_snclient -k -u https://127.0.0.1:18443 -p secret_forge_pass -proxy http://proxyuser:proxypass@127.0.0.1:18888 check_uptime
  ↳ [RC=1] WARNING - uptime: 1d 12:34h, boot: 2026-11-04 10:09:19 UTC |'uptime'=131642s;172800:;86400:;;
PASSED
tests/test_checks.py::TestSNClientIntegration::test_proxy_traversal_invalid_auth[docker://snclient-testbed] 
  ▶ Exec : /usr/bin/check_snclient -k -u https://127.0.0.1:18443 -p secret_forge_pass -proxy http://wronguser:wrongpass@127.0.0.1:18888 check_uptime
  ↳ [RC=3] UNKNOWN - Get "https://127.0.0.1:18443/...: Unauthorized
PASSED

=========================================================================================================================================== 6 passed in 0.78s ===========================================================================================================================================

Note: The test forge lives on a separate branch forge/testinfra to keep this PR strictly Go-focused. If the team is interested in this testbed -- which supports local runs, dynamic ephemeral ports, and an immutable Docker image baked with Packer. I'd be very happy to discuss contributing it in a separate PR!

Looking forward to your feedback and thoughts!

Comment thread README.md
check_snclient is a REST client for the NSClient++/SNClient webserver for querying
and receiving check information over HTTP(S).
Version:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this for?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gpineda@thinkpad-e15g2:~/Documents/gpineda-dev/check_snclient$ ./check_snclient --help
Usage:
  check_snclient [options] [query parameters]

Description:
  check_snclient is a REST client for the NSClient++/SNClient webserver for querying
  and receiving check information over HTTP(S).

Version:
  check_snclient v0.7.6

It was an initiative to realign the cli usage described in the readme with the actual output.
By copying the output directly in the documentation (maybe a to lazy approach to include the new proxy options.
Feel free to remove it since this depends on the compiled version.

@sni

sni commented Nov 20, 2026

Copy link
Copy Markdown

that's a lot of text... Looks good in general, only a small question about the version in the readme.

@sni
sni merged commit e933af5 into ConSol-Monitoring:master Nov 20, 2026
@sni

sni commented Nov 20, 2026

Copy link
Copy Markdown

thanks for your contribution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants