From bef9b6fa2c1ae29808fcbfe197c6eb8baf452f39 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Thu, 29 Oct 2026 08:51:40 -0500 Subject: [PATCH] Documented cf-reactor watch methods --- .../reference/components/cf-reactor.markdown | 55 ++++++++++++++++++- .../reference/promise-types/events.markdown | 44 +++++++++++++++ 2 files changed, 98 insertions(+), 1 deletion(-) diff --git a/content/reference/components/cf-reactor.markdown b/content/reference/components/cf-reactor.markdown index 76d44f55b..cc0fbf2ea 100644 --- a/content/reference/components/cf-reactor.markdown +++ b/content/reference/components/cf-reactor.markdown @@ -88,11 +88,64 @@ The daemon is then signaled to handle the event and run the bundles of the `then **Notes:** -- Event detection is based on polling, but `cf-reactor` does not use busy-waiting. +- Event detection is based on polling, sped up by [watch methods](#watch-methods), but `cf-reactor` does not use busy-waiting. Instead, the polling thread sleeps between polling cycles and remains dormant when no events are registered. - `cf-reactor` does not guarantee that an event will be handled immediately after it occurs. The delay depends on factors such as the polling interval, policy reloads, and slow code execution. However, `cf-reactor` guarantees that changes will be detected as fast as possible and the corresponding code executed, provided that execution does not fail. +- After a watch method wakes it up, `cf-reactor` waits 200 milliseconds before checking, so that a burst of changes is checked once. +- An event that fires more than 10 times in 60 seconds is delayed, starting at 10 seconds and doubling up to 5 minutes, until it calms down. + The changes it holds back are handled in one run when the delay ends. + Meanwhile, and while its bundles run, the event is not checked again: it is checked once when it can run. + +### Watch methods + +A watch method wakes an event's watcher up as soon as the operating system reports a change to the watched file or its directory, so that `cf-reactor` checks the event right away instead of at its next [`interval`][events#interval]. +Each event uses the first available method of its [`watch_methods`][events#watch_methods] attribute. +Methods only decide _when_ an event is checked, never _whether_ it happened, and polling continues underneath all of them. +Changing the method never changes which events fire, only how soon. + +- `ebpf` (Linux, Enterprise) loads a small program into the kernel that matches changes against the watched files' inodes before telling `cf-reactor`. + It needs kernel 5.8 or later (or the RHEL 8 kernel) with BTF, and `CAP_SYS_ADMIN` (or `CAP_BPF` and `CAP_PERFMON`). + It has no per-directory watch limits, and watches up to 65,536 files; further events use their next watch method, and `cf-reactor` logs a warning. +- `fanotify` and `inotify` (Linux, Enterprise) watch each file for writes and changes of its permissions or owner, and its parent directory for files being created, deleted and renamed. + Files created and deleted in that directory are reported to `cf-reactor`, which ignores the ones for other files. + `inotify` is limited by `fs.inotify.max_user_watches` and `fs.inotify.max_user_instances`, and `fanotify` by `fs.fanotify.max_user_marks`. + When out of watches or marks, they watch files through their directories only, and `cf-reactor` logs a warning; a write through a file descriptor opened before the file was renamed into place is then only seen at the next interval. + Raise the limit to watch more than about half as many files as it allows. +- `usn` (Windows, Enterprise) reads the NTFS change journal, and `rdcw` (Windows, Enterprise) uses `ReadDirectoryChangesW`. +- `poll` only checks at the interval. It is the only method in CFEngine Community. + +**Cost:** away from watched files, no method adds measurable CPU time to the rest of the system. +In a directory that holds a watched file, each file operation costs about 1 µs more with `ebpf`, `fanotify` or `inotify` (about 10%), counting the process making the change and `cf-reactor`. +At 1,000 file operations per second in one watched directory this is about 0.1% of a CPU core, so the choice of method makes no practical difference to CPU time. +Where the methods differ: + +| Method | Wakes the watcher after a change | Kernel resources | +| ---------- | -------------------------------- | ------------------------------------------------------------------------ | +| `ebpf` | under 2 ms | up to 65,536 files, about 0.7 MB of locked memory for 1,000 | +| `fanotify` | about 20 ms | one mark per file and one per directory (`fs.fanotify.max_user_marks`) | +| `inotify` | about 20 ms | one watch per file and one per directory (`fs.inotify.max_user_watches`) | +| `poll` | at the next interval | none | + +`cf-reactor` then waits 200 milliseconds before checking, so in practice the native methods react equally fast. +`fanotify` and `inotify` gather the changes of 20 milliseconds into one wakeup. +Measured on Ubuntu 24.04 (kernel 6.8) in a virtual machine. + +`file_content_modified` hashes the whole file when its size or modification time changed. +For a large file that is written to all the time, such as a log, this costs about as much with every method, since the event is limited to 10 runs a minute (see above) and checked once per run: hashing a 200 MB file appended to 20 times a second took about 20% of a CPU core. +Use `file_modified` for such files, which only compares the modification time. + +**Changes only seen at the next interval:** + +- writes through a memory mapping (`mmap`) that stays open, with every method +- with `ebpf`, writes on file systems that don't report them to the kernel's generic write path (for example some network file systems) until the file is closed +- with every method, changes on network file systems made by other hosts + +**Notes:** + +- On CFEngine Enterprise, the method each event uses is reported to the hub, in the `reactor_watchers` table of the reporting database. +- SELinux or other mandatory access control may deny `cf-reactor` a method; it then falls back to the next one in the list. ## Enterprise hub tasks diff --git a/content/reference/promise-types/events.markdown b/content/reference/promise-types/events.markdown index f9c34b9d8..77c0f348f 100644 --- a/content/reference/promise-types/events.markdown +++ b/content/reference/promise-types/events.markdown @@ -119,6 +119,7 @@ bundle reactor main Each event has its own interval. A lower value means the event is noticed sooner but checked more often. +A [watch method](#watch_methods) can wake the check up earlier, so with one the interval mostly bounds how late a missed change is noticed. For how polling works, see [Under the hood][cf-reactor#Under the hood]. ```cf3 @@ -135,6 +136,49 @@ body when deleted_slow(file) **Default value:** `1` +#### watch_methods + +**Description:** The watch methods that may wake this event's watcher up before its next interval, in order of preference. + +A watch method is a notification from the operating system that the watched file, or its directory, changed. +It makes `cf-reactor` check the event right away instead of at the next interval. +It never decides whether the event happened: the check does, so the same events fire whatever the method. +Polling at the [`interval`](#interval) continues underneath every method, and catches what a method misses. + +`cf-reactor` uses the first method in the list that is available on the host and can watch the event's target. +Without `watch_methods`, it tries `ebpf`, `fanotify`, `inotify`, `usn`, `rdcw` and `poll`, in that order. +CFEngine Community only has `poll`. +If none of the listed methods is available, the event is not watched and `cf-reactor` logs an error; list `poll` last to fall back to polling instead. + +| Method | Platform | Edition | +| ---------- | -------- | ---------- | +| `ebpf` | Linux | Enterprise | +| `fanotify` | Linux | Enterprise | +| `inotify` | Linux | Enterprise | +| `usn` | Windows | Enterprise | +| `rdcw` | Windows | Enterprise | +| `poll` | All | All | + +For how the methods differ, see [Watch methods][cf-reactor#Watch methods]. + +```cf3 +body when passwd_modified +{ + file_content_modified => "/etc/passwd"; + interval => 300; + watch_methods => { "inotify", "poll" }; +} +``` + +**Type:** `slist` + +**Allowed input range:** `poll`, `inotify`, `fanotify`, `ebpf`, `usn`, `rdcw` + +**Notes:** + +- changing `watch_methods` does not reset the event: on policy reload, a watcher keeps its state when only its watch methods change. +- with `watch_methods => { "poll" }` the event is only checked every `interval` seconds. + #### file_deleted **Description:** Creates an event when the file at the given path is deleted.