Repository navigation
fix(sync): don't touch url/last_modified on non-datastore resources - #29
Merged
Merged
Conversation
After a write, the CKAN sync patched `last_modified` (and `url` on create) on every resource. On an upload or link resource those describe the source file, and CKAN flags a change to either as `url_changed`, so ckanext-aircan re-submitted the resource to the ingestion pipeline. A fields-only `datastore_create` with `force` (updating the data dictionary) re-ran a full ingestion and replaced the upload's file URL with the dump URL. Only patch `url` / `last_modified` when `url_type == "datastore"`; otherwise sync the schema alone, or skip the patch when there is nothing to sync. Co-Authored-By: Claude <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
UAT report (neso-nextgen-shared#12): calling
datastore_createwithforce: trueand onlyfieldson an ingested upload, to update its data dictionary, re-ran the whole ingestion pipeline (about 7 minutes for a 428 MB file).Cause
After every write,
_sync_resource_to_ckansends aresource_patchwithlast_modified=now, and on create alsourl=<dump URL>. CKAN core setsurl_changedwhen either field changes. ckanext-aircan'snotifyhook reads that as a new upload and submits the DAG. The hook only skips resources withurl_type="datastore".On an upload, the patch also replaces the file URL with the dump URL. CKAN strips upload URLs to their last path segment, so the resource's download link can break.
Fix
_sync_resource_to_ckanonly writesurlandlast_modifiedwhenurl_type == "datastore". For upload and link resources it syncs onlyschema(anddatastore_activeon a table drop). If there is nothing to sync, it skips theresource_patch.datastore_createkeeps the resource'surl_typebeforedata_dict["resource"]is overwritten. The dict-form create always counts as datastore-owned. Upsert and delete pass theurl_typethey already have.url_typecounts as not owned, because CKAN link resources also have an emptyurl_type.Behaviour change
Upserts and deletes on upload or link resources no longer update the resource's
last_modified("updated" date), and no longer add a CKAN activity entry when there's no schema change. On those resources,last_modifieddescribes the source file.Testing
pytest: 542 passed.nhs_dev). Steps: upload a CSV, ingest it, then send a fields-only forceddatastore_create.last_modifiedchanged and a newpipeline_ckan_to_bigqueryrun started 145 ms later.urlandlast_modifiedwere unchanged and no DAG run started.resource.schemaanddatastore_infoboth showed the new descriptions, and the 3 rows were kept.urlandlast_modified.