Repository navigation
Conversation
357c6d8 to
754aaef
Compare
saba8814
added
ci:main
wkz
left a comment
wkz
left a comment
There was a problem hiding this comment.
I really like the trajectory of this! 💪
As we said in the internal discussion, I think there are synergies with #1750 , and that we should figure out what the relationships between the Env, the Test and the devices should be to make adding these kinds of features easier.
| [ -f /mnt/aux/test-mode ] || exit 0 | ||
|
|
||
| # Qemu settings (for testing) take precedence over the test-mode file | ||
| file=/sys/firmware/qemu_fw_cfg/by_name/opt/test-mode/raw |
There was a problem hiding this comment.
fw_cfg is a very neat way to import the test mode config. Is there any reason to also look for the marker?
Could we not use the existence of either the on-disk marker, or the fw_cfg file as an indication to enable test-mode? Then we could also get rid of the test-mode-injection juggling in test/env.
| done = set() | ||
| threads = [threading.Thread(target=log, args=(dev,), daemon=True) for dev in devs] | ||
| for t in threads: | ||
| t.start() | ||
| deadline = time.monotonic() + 1 | ||
| for t in threads: | ||
| t.join(max(0, deadline - time.monotonic())) |
There was a problem hiding this comment.
Could we not reuse util.parallel() here?
There was a problem hiding this comment.
Currently util.parallel() does not have timeout and wee need that for the markers. I could add optional timeout parameter to parallel() if we decide to?
When a test fails in CI the DUTs are gone, and with them their logs. In test mode, a DUT with a management interface in its test mode settings now sends all its logs to ff02::161 on that port, tagged with its name in the topology, since tests change the hostname. This is set up at boot, outside the configuration, so it keeps working when a test configures syslog or reboots the DUT. Signed-off-by: Ejub Sabic <ejub1946@outlook.com>
All virtual DUTs share one image, and with it one test-mode file, so each node gets its management port and topology name from Qeneth in opt/test-mode instead. Signed-off-by: Ejub Sabic <ejub1946@outlook.com>
test/env starts a sysklogd listening to ff02::161 on each of the controller's management ports, saving the logs of all DUTs in one file per session, named in $TEST_SYSLOG. Infamy logs test-start on attach, step at the start of each step, and test-stop on exit, with the test name and the DUT's role as structured data. Markers are best effort: a DUT that does not answer is skipped until it is attached again. syslog_split.py uses the markers to filter the collected log on a test or a device, or to split it into one file per test and device. After make test the split logs are packed in .log/last/logs.tar.gz, which CI uploads as the dut-syslog artifact. Resolves: #1608 Signed-off-by: Ejub Sabic <ejub1946@outlook.com>
Signed-off-by: Ejub Sabic <ejub1946@outlook.com>
754aaef to
645f0c3
Compare
|
Putting this as draft until we take an offline meeting. FYI: @wkz @troglobit |
Description
This PR introduces collection of DUT logs during tests, so failures in
CI can be debugged after the run.
In test mode, each DUT forwards all its logs to the test controller
from boot, tagged with its name in the topology. The test environment
always runs a syslog sink that saves them in one file, and Infamy marks
where each test and step starts and stops. After
make test, the logsare split per test and DUT and uploaded as the
dut-syslogartifact inCI.
It also lifts the RESTCONF rate limit for the test controller in test
mode, and bumps the test container to 2.18 for sysklogd 2.8.0. That
image needs to be published before merge.
Checklist
Tick relevant boxes, this PR is-a or has-a: