Documented cf-reactor watch methods

content/reference/components/cf-reactor.markdown

@@ -88,11 +88,64 @@ The daemon is then signaled to handle the event and run the bundles of the `then
 
 **Notes:**
 
-- Event detection is based on polling, but `cf-reactor` does not use busy-waiting.
+- Event detection is based on polling, sped up by [watch methods](#watch-methods), but `cf-reactor` does not use busy-waiting.
   Instead, the polling thread sleeps between polling cycles and remains dormant when no events are registered.
 - `cf-reactor` does not guarantee that an event will be handled immediately after it occurs.
   The delay depends on factors such as the polling interval, policy reloads, and slow code execution.
   However, `cf-reactor` guarantees that changes will be detected as fast as possible and the corresponding code executed, provided that execution does not fail.
+- After a watch method wakes it up, `cf-reactor` waits 200 milliseconds before checking, so that a burst of changes is checked once.
+- An event that fires more than 10 times in 60 seconds is delayed, starting at 10 seconds and doubling up to 5 minutes, until it calms down.
+  The changes it holds back are handled in one run when the delay ends.
+  Meanwhile, and while its bundles run, the event is not checked again: it is checked once when it can run.
+
+### Watch methods
+
+A watch method wakes an event's watcher up as soon as the operating system reports a change to the watched file or its directory, so that `cf-reactor` checks the event right away instead of at its next [`interval`][events#interval].
+Each event uses the first available method of its [`watch_methods`][events#watch_methods] attribute.
+Methods only decide _when_ an event is checked, never _whether_ it happened, and polling continues underneath all of them.
+Changing the method never changes which events fire, only how soon.
+
+- `ebpf` (Linux, Enterprise) loads a small program into the kernel that matches changes against the watched files' inodes before telling `cf-reactor`.
+  It needs kernel 5.8 or later (or the RHEL 8 kernel) with BTF, and `CAP_SYS_ADMIN` (or `CAP_BPF` and `CAP_PERFMON`).
+  It has no per-directory watch limits, and watches up to 65,536 files; further events use their next watch method, and `cf-reactor` logs a warning.
+- `fanotify` and `inotify` (Linux, Enterprise) watch each file for writes and changes of its permissions or owner, and its parent directory for files being created, deleted and renamed.
+  Files created and deleted in that directory are reported to `cf-reactor`, which ignores the ones for other files.
+  `inotify` is limited by `fs.inotify.max_user_watches` and `fs.inotify.max_user_instances`, and `fanotify` by `fs.fanotify.max_user_marks`.
+  When out of watches or marks, they watch files through their directories only, and `cf-reactor` logs a warning; a write through a file descriptor opened before the file was renamed into place is then only seen at the next interval.
+  Raise the limit to watch more than about half as many files as it allows.
+- `usn` (Windows, Enterprise) reads the NTFS change journal, and `rdcw` (Windows, Enterprise) uses `ReadDirectoryChangesW`.
+- `poll` only checks at the interval. It is the only method in CFEngine Community.
+
+**Cost:** away from watched files, no method adds measurable CPU time to the rest of the system.
+In a directory that holds a watched file, each file operation costs about 1 µs more with `ebpf`, `fanotify` or `inotify` (about 10%), counting the process making the change and `cf-reactor`.
+At 1,000 file operations per second in one watched directory this is about 0.1% of a CPU core, so the choice of method makes no practical difference to CPU time.
+Where the methods differ:
+
+| Method     | Wakes the watcher after a change | Kernel resources                                                         |
+| ---------- | -------------------------------- | ------------------------------------------------------------------------ |
+| `ebpf`     | under 2 ms                       | up to 65,536 files, about 0.7 MB of locked memory for 1,000              |
+| `fanotify` | about 20 ms                      | one mark per file and one per directory (`fs.fanotify.max_user_marks`)   |
+| `inotify`  | about 20 ms                      | one watch per file and one per directory (`fs.inotify.max_user_watches`) |
+| `poll`     | at the next interval             | none                                                                     |
+
+`cf-reactor` then waits 200 milliseconds before checking, so in practice the native methods react equally fast.
+`fanotify` and `inotify` gather the changes of 20 milliseconds into one wakeup.
+Measured on Ubuntu 24.04 (kernel 6.8) in a virtual machine.
+
+`file_content_modified` hashes the whole file when its size or modification time changed.
+For a large file that is written to all the time, such as a log, this costs about as much with every method, since the event is limited to 10 runs a minute (see above) and checked once per run: hashing a 200 MB file appended to 20 times a second took about 20% of a CPU core.
+Use `file_modified` for such files, which only compares the modification time.
+
+**Changes only seen at the next interval:**
+
+- writes through a memory mapping (`mmap`) that stays open, with every method
+- with `ebpf`, writes on file systems that don't report them to the kernel's generic write path (for example some network file systems) until the file is closed
+- with every method, changes on network file systems made by other hosts
+
+**Notes:**
+
+- On CFEngine Enterprise, the method each event uses is reported to the hub, in the `reactor_watchers` table of the reporting database.
+- SELinux or other mandatory access control may deny `cf-reactor` a method; it then falls back to the next one in the list.
 
 ## Enterprise hub tasks
 

content/reference/promise-types/events.markdown

@@ -119,6 +119,7 @@ bundle reactor main
 
 Each event has its own interval.
 A lower value means the event is noticed sooner but checked more often.
+A [watch method](#watch_methods) can wake the check up earlier, so with one the interval mostly bounds how late a missed change is noticed.
 For how polling works, see [Under the hood][cf-reactor#Under the hood].
 
 ```cf3
@@ -135,6 +136,49 @@ body when deleted_slow(file)
 
 **Default value:** `1`
 
+#### watch_methods
+
+**Description:** The watch methods that may wake this event's watcher up before its next interval, in order of preference.
+
+A watch method is a notification from the operating system that the watched file, or its directory, changed.
+It makes `cf-reactor` check the event right away instead of at the next interval.
+It never decides whether the event happened: the check does, so the same events fire whatever the method.
+Polling at the [`interval`](#interval) continues underneath every method, and catches what a method misses.
+
+`cf-reactor` uses the first method in the list that is available on the host and can watch the event's target.
+Without `watch_methods`, it tries `ebpf`, `fanotify`, `inotify`, `usn`, `rdcw` and `poll`, in that order.
+CFEngine Community only has `poll`.
+If none of the listed methods is available, the event is not watched and `cf-reactor` logs an error; list `poll` last to fall back to polling instead.
+
+| Method     | Platform | Edition    |
+| ---------- | -------- | ---------- |
+| `ebpf`     | Linux    | Enterprise |
+| `fanotify` | Linux    | Enterprise |
+| `inotify`  | Linux    | Enterprise |
+| `usn`      | Windows  | Enterprise |
+| `rdcw`     | Windows  | Enterprise |
+| `poll`     | All      | All        |
+
+For how the methods differ, see [Watch methods][cf-reactor#Watch methods].
+
+```cf3
+body when passwd_modified
+{
+  file_content_modified => "/etc/passwd";
+  interval => 300;
+  watch_methods => { "inotify", "poll" };
+}
+```
+
+**Type:** `slist`
+
+**Allowed input range:** `poll`, `inotify`, `fanotify`, `ebpf`, `usn`, `rdcw`
+
+**Notes:**
+
+- changing `watch_methods` does not reset the event: on policy reload, a watcher keeps its state when only its watch methods change.
+- with `watch_methods => { "poll" }` the event is only checked every `interval` seconds.
+
 #### file_deleted
 
 **Description:** Creates an event when the file at the given path is deleted.