Skip to content

feat: async pipeline - #177

Draft
ilan-gold wants to merge 7 commits into
mainfrom
ig/async_double_memory
Draft

ilan-gold wants to merge 7 commits into
mainfrom
ig/async_double_memory

Conversation

@ilan-gold

Copy link
Copy Markdown
Collaborator

This is a vibe-coded solution to #44 (comment) - instead of writing directly into the shared output buffer within the async tasks (which AFAICT is impossible), we just use double the memory. This is what zarrs does anyway internally. I will hopefully have time to clean this up at some point. Definitely open to high-level points, but not sure there is much point in code review now. This is more of a PoC

@ilan-gold

Copy link
Copy Markdown
Collaborator Author

Investigating this more, we are making a quite fundamental tradeoff here beyond that already enumerated - by writing to the output buffer in rust, we are not allowing the python to call the rust asynchronously (basically smashing up the stack the lifetime problem that motivated the doubling of memory in the first place).

@flying-sheep

Copy link
Copy Markdown
Collaborator

One interesting thing I learned about safety is that numpy doesn’t actually make guarantees against data races – so any memory shared with numpy can not be converted into a rust reference (read or write), but can be written through with raw pointers (* or NonNull or so).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants