[alexluong]

When ClickHouse restarts behind a proxy or load balancer, pooled connections can go stale in ways clickhouse-go can't detect on checkout. The proxy either closes the client side only when it next writes, so each stale connection fails one call with failed to read first block packet: EOF, or it keeps the client side open with nothing behind it, so each stale connection hangs until the caller's deadline or the read timeout. The driver doesn't retry and the other stale connections stay pooled, so the failure repeats once per connection. A process with little ClickHouse traffic keeps its stale connections for up to an hour (the driver's connection lifetime).

The idea: drop connections nobody is using, and replace the whole pool as soon as one connection turns out dead. clickhouse.New now returns a wrapper around the driver pool, so callers don't change.

1. Close idle pools. After CLICKHOUSE_IDLE_CLOSE_SECONDS (default 300) with no ClickHouse call, the pool is closed and the next call dials fresh. A timer drives it, so calls don't pay for a check, and a process making calls never hits it.
2. Replace the pool on connection errors. On EOF, unexpected EOF, connection reset or broken pipe, the pool is swapped for a fresh one; concurrent failures on the same pool cause one swap. Connections still in use on the old pool finish their work and close when released. A timeout (read timeout or the caller's deadline) replaces the pool only if no call on that pool succeeded since the timed-out call started, so a busy server that answers some calls slowly doesn't cause pool churn. Server exceptions, dial errors and caller cancellation don't trigger it.
3. Retry once where it's safe. Query, QueryRow, Select, Ping, QueryFormat and PrepareBatch (no rows are sent before Send) run once more on the new pool after a dropped connection. Exec, AsyncInsert, InsertFormat and batch Send reset the pool and return the error without a retry, so rows are never sent twice; callers keep their own retry. A timeout is never retried: it can't tell a dead connection from a slow query.

Other changes:

• chlogstore closes its insert batches when an append fails, so the connection goes back to the pool.

The first call that hits a half-open connection still fails, at its deadline or the read timeout. The pool is replaced then, so the next call works; closing idle pools keeps this rare.

🤖 Generated with Claude Code