feat: Add ability to generate more realistic learner journey data by bmtcril · Pull Request #276 · openedx/xapi-db-load

bmtcril · 2026-10-09T17:27:50Z

Adds a journeys command that generates learner data with known expected results, so Aspects report numbers can be checked against a known answer instead of only being load-tested.

The existing backends pick each event independently. This new mode simulates each enrolled learner moving through a properly nested course (sections > subsections > units > problems and videos) in time-ordered sessions. The simulator knows exactly what each learner did, so it also writes the expected engagement results.

What it generates (gzipped CSV, same layouts as the csv backend):

Event sink rows: courses, blocks (several publishes, with deleted units only in earlier publishes), external ids and user profiles.
xAPI events matching event-routing-backends shapes:
- navigation events whose object is the unit being left, as in the Learning MFE;
- problem attempted / evaluated;
- video initialized / played / paused / seeked / completed / terminated, with completed fired once at 95%;
- optional mailto: actors.
expected_engagement: per learner and section / subsection, pages / problems / videos done out of total, with the Aspects status label.
expected_video_seconds: watched seconds per learner and video. "Observable" is what the events can show (each play paired with the next video event). "Truth" is what the learner actually watched, including tab closes.
manifest.json: counts, plus the most active learner per course for the learner dashboard.

Edge cases (all on in journeys_oracle.yaml): mailto: actors, deleted units, duplicate events, pause and resume at the same instant, recent activity inside a refreshable-view lookback window, and still-running courses.

Example configs: journeys_oracle.yaml (~9K events, for correctness), journeys_m.yaml (~15M) and journeys_l.yaml (~75M) for benchmarks. The same config, seed and --now produce identical files.

Testing

18 tests in test_journeys.py. The main check rebuilds every expected page / problem row and every observable watched-seconds value from the generated xAPI events and block structure alone, so the simulator's bookkeeping can't drift from the events it emits. The others cover reproducibility, ordering, nesting, edge-case presence, video pairing edge cases, the CLI and config errors.
Used to benchmark and check correctness of aspects-dbt's engagement models. The oracle and M datasets load into a ClickHouse 25.8 / Aspects stack and parse through the full MV chain.

Usage

xapi-db-load journeys --config_file example_configs/journeys_oracle.yaml --now "2026-10-08 12:00:00"

Loading instructions are in the README.

🤖 Generated with Claude Code

codecov · 2026-10-09T17:29:41Z

Codecov Report

❌ Patch coverage is 96.27474% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.78%. Comparing base (8aca276) to head (0468a1c).

Files with missing lines	Patch %	Lines
xapi_db_load/journeys/simulate.py	92.75%	9 Missing and 11 partials ⚠️
xapi_db_load/journeys/generate.py	96.02%	3 Missing and 4 partials ⚠️
xapi_db_load/journeys/load.py	97.52%	2 Missing and 1 partial ⚠️
xapi_db_load/journeys/structure.py	97.97%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #276      +/-   ##
==========================================
+ Coverage   85.54%   88.78%   +3.23%     
==========================================
  Files          29       36       +7     
  Lines        1986     2845     +859     
  Branches      175      308     +133     
==========================================
+ Hits         1699     2526     +827     
- Misses        260      275      +15     
- Partials       27       44      +17

Flag	Coverage Δ
unittests	`88.78% <96.27%> (+3.23%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

felipemontoya

I used this branch of the project when creating the test data for aspects-dbt#176. It worked beautifully and let me correctly analyze the interplay of the video pairing events.

I was experimenting with a review from OpenWeights model GLM5.3 so I had it upload a bunch of nits into the specific files. Feel free to ignore them or have your agent respond to them in any way. The overall code is working and has been tested to successful results.

felipemontoya · 2026-10-14T15:56:54Z

+    def run(self, output_dir: str) -> Dict:
+        """Generate everything into ``output_dir`` and return the manifest."""
+        self.setup()
+        os.makedirs(output_dir, exist_ok=True)


Overwrite manifest.json with {"complete": false} here before anything else: a crash mid-rerun (Ctrl-C, bad config value) truncates the CSVs but leaves the previous run's manifest in place — a complete-looking dir whose counts lie.

Good catch. Fixed in 7f61c4e: run() now deletes manifest.json before writing anything, so a run that stops partway leaves no manifest. Added a test that makes a re-run fail partway and checks the old manifest is gone.

felipemontoya · 2026-10-14T15:56:54Z

+    except ValueError as e:
+        raise click.UsageError(str(e)) from e
+    start = datetime.datetime.now()
+    manifest = generator.run(out)


Move run() inside the try and type-check required config keys in __init__/from_config: wrong-typed values (e.g. days_between_subsections: "2.0", window_days: "300") escape the except ValueError net and die as raw TypeErrors — some only mid-run, after the CSVs are already truncated.

Fixed in e960e27. The generator now checks every top-level, course template, course and behavior setting when it's created, before any file is opened. Errors name the setting (for example behavior.days_between_subsections must be a number >= 0.001, got '2.0'), and the CLI shows them as a usage error. Because nothing can fail on config after that point, run() stays outside the try. 14 new test cases cover this.

felipemontoya · 2026-10-14T15:56:54Z

+    verticals_per_sequential: Tuple[int, int]
+    # Each vertical gets a random count of these, drawn from the given weights
+    # (index = count), e.g. [0.4, 0.4, 0.2] -> 40% none, 40% one, 20% two.
+    problems_per_vertical: List[float]


These hold probability weights indexed by count, not counts — rename to problem_count_weights/video_count_weights before the schema ships, or a config author writing problems_per_vertical: [0, 4] (like every sibling range key) silently gets a 2-weight distribution instead.

Agreed. Renamed to problems_per_vertical_weights / videos_per_vertical_weights in d6d95db, and the README now explains them next to the [low, high] ranges. e960e27 also rejects an empty list and a list of all zeros.

felipemontoya · 2026-10-14T15:56:54Z

+                writers["xapi"].writerow((e["event_id"], _fmt(e["emission_time"]), e["event"]))
+
+        return {"num_enrollments": n_enrollments, "num_xapi_events": n_events,
+                "heavy_learners": heavy}


heavy_learners is one most-active learner per course run, not a ranking — and when nobody viewed pages the argmax still picks the first enrollment (pages_viewed_fraction: 0.0), so skip empty courses; consider most_active_learner_per_course while the manifest schema is still free.

Both done. Renamed to most_active_learner_per_course in d6d95db, and 7f61c4e skips courses where nobody viewed a page, so a pages_viewed_fraction: 0.0 entry can no longer appear. There's a test for each.

felipemontoya · 2026-10-14T15:56:54Z

+max_insert_threads = 1, max_threads = 1``.
+
+``window_days`` must stay under 365, because Aspects drops xAPI events older than
+a year. Behavior probabilities (``behavior`` section) are documented in


This pointer is a dead end: Behavior's docstring sends readers back to the YAMLs, which document ~5 of 16 keys — the dataclass fields are the accepted behavior: schema, so per-field docstrings there would make it true.

Fixed in e960e27. Every Behavior field now has a comment saying what it controls, the class docstring says it is the behavior: schema, and the README points there.

felipemontoya · 2026-10-14T15:56:54Z

+
+    def __init__(self, rng: random.Random, behavior: Behavior, now: datetime.datetime):
+        self.rng = rng
+        self.b = behavior


self.b → self.behavior (25 dereferences, and it collides with the b loop variables in structure/generate).

Renamed to self.behavior in d6d95db.

felipemontoya · 2026-10-14T15:56:54Z

+    return json.dumps(e)
+
+
+def video(  # pylint: disable=too-many-positional-arguments


The only builder not named for its verb — video_statement(), or per-verb builders for symmetry with registered/navigated/problem_evaluated.

Renamed to video_statement() in d6d95db. I kept it as one builder because the video verbs share every field except the result extensions.

felipemontoya · 2026-10-14T15:56:54Z

+            self.events.append(dict(row))
+        return event_id
+
+    def _is_past(self, when: datetime.datetime) -> bool:


Returns True at now, not before it — _has_happened would match the inclusive boundary the docstring already states.

Renamed to _has_happened in d6d95db.

felipemontoya · 2026-10-14T15:56:54Z

+  running_course_fraction: 0.4
+  course_length_days: 120
+  num_organizations: 5
+  num_actors: 100000


[NIT]: the surface splits one person into two words — config says learners, data says actor_id: either rename the schema-level keys toward one term or add one README sentence declaring the split deliberate (keep actor_id; renaming it breaks the aspects-dbt consumers).

Leaving as is. The config already uses num_actors for people, and learners in a course entry is the number of enrollments, so the two words name different things. actor_id matches Aspects, as you say.

felipemontoya · 2026-10-14T15:56:54Z

+instead simulates each enrolled learner moving through a properly nested course
+(sections > subsections > units > problems and videos) in time-ordered
+sessions, and writes the expected engagement results alongside the data, so
+report numbers can be checked against a known answer:


Use the word "oracle" once here (or rename oracle.py → expected_results.py): today the term exists only in the module name and the journeys_oracle.yaml filename, so a reader going README → package never connects the two.

Done in e960e27: the README now says the expected results are the "oracle", computed in xapi_db_load/journeys/oracle.py.

- Stop counting a video's actual watched seconds past "now" for sessions still playing. - Report journeys config problems as CLI usage errors. - Check course_length_days fits in window_days. - Add tests for the CLI, video pairing edge cases and unsorted output. - Document the journeys command in the README; fix dataset size comments. - Lint fixes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- problems/videos_per_vertical -> *_per_vertical_weights: they weight each count, unlike the [low, high] ranges beside them. - manifest heavy_learners -> most_active_learner_per_course. - Simulator.b -> behavior, _is_past -> _has_happened, statements.video -> video_statement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ge views - A run that stopped partway left the previous run's manifest.json beside the new, partial CSVs. The manifest is now removed before any file is written. - A course where nobody viewed a page listed its first learner as the most active one, with pages_viewed_fraction 0. Such courses are now skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A wrong-typed value, e.g. days_between_subsections: "2.0", used to fail as a raw TypeError partway through a run, after the CSVs were already truncated. Every top-level, course template, course and behavior setting is now checked when the generator is built, and errors name the setting. Each behavior setting is also documented on the Behavior class, and the README explains the weight lists and the "oracle". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bmtcril · 2026-10-14T17:07:17Z

@felipemontoya Thanks for the review! I went through all 12 comments: 9 are addressed in d6d95db, 7f61c4e and e960e27, and I've left 3 as they are, with reasons inline. The generated oracle data is byte-identical to before, so existing known answers still hold.

bmtcril · 2026-10-14T17:28:28Z

Reopening to try to kick CLA check

Inserts the event sink rows and expected results straight into ClickHouse and sends the xAPI events through the configured backend: clickhouse, ralph (without duplicate events, which an LRS rejects) or vector. Ralph and Vector insert asynchronously, so it waits until every event id has arrived. A new journeys setting, org_prefix, keeps journeys organizations apart from load-db's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

expected_video_seconds gains observable_max_views, the count the video watch charts show as total views. The event-derived check in the tests covers it too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Reload ClickHouse dictionaries after inserting the event sink rows, so course names and users are found without waiting for the next dictionary refresh. - New journeys setting first_user_id (default 1, which keeps the output unchanged) so journeys users don't share user ids, usernames or emails with load-db's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bmtcril · 2026-10-15T13:59:17Z

@felipemontoya @saraburns1 sorry to add on here, but I wanted to be able to use the new journey features to be able to check exact row counts in tutor-contrib-aspects CI. This should allow us to have a full integration test that makes sure we haven't broken anything in aspects-dbt, the data pipeline, or xapi-db-load as part of each TCA run.

I'm working out the kinks on the current TCA PR for performance: openedx/tutor-contrib-aspects#1355

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Printing every event in a few seconds is far faster than an LMS logs, and one log line was lost on its way to Vector at that rate in CI. Pause after every vector_batch_size (default 100) events. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bmtcril mentioned this pull request Oct 9, 2026

Video segment pairing fix openedx/aspects-dbt#177

Merged

bmtcril marked this pull request as draft October 9, 2026 17:36

bmtcril marked this pull request as ready for review October 9, 2026 17:55

bmtcril requested review from felipemontoya and saraburns1 October 9, 2026 17:58

This was referenced Oct 10, 2026

perf: fix engagement chart numbers and improve video chart performance openedx/aspects-dbt#180

Open

Improve video watch performance openedx/tutor-contrib-aspects#1355

Open

felipemontoya approved these changes Oct 14, 2026

View reviewed changes

Tycho Hob and others added 5 commits October 14, 2026 12:35

feat: Add ability to generate more realistic learner journey data

ce28a04

bmtcril force-pushed the perf/engagement-harness branch from 457b3c4 to e960e27 Compare October 14, 2026 17:03

bmtcril closed this Oct 14, 2026

bmtcril reopened this Oct 14, 2026

Tycho Hob and others added 3 commits October 15, 2026 09:07

feat: add the most views of any second to the expected video results

da59230

expected_video_seconds gains observable_max_views, the count the video watch charts show as total views. The event-derived check in the tests covers it too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Tycho Hob and others added 2 commits October 15, 2026 13:00

feat: name the events that never arrived when load-journeys times out

5fa34a7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

		return json.dumps(e)


		def video( # pylint: disable=too-many-positional-arguments

Conversation

bmtcril commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

felipemontoya left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bmtcril commented Oct 14, 2026

Uh oh!

bmtcril commented Oct 14, 2026

Uh oh!

bmtcril commented Oct 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bmtcril commented Oct 9, 2026 •

edited

Loading

codecov Bot commented Oct 9, 2026 •

edited

Loading