Skip to content

async

Async Planning

I've been thinking about this a bit more and this post is to progress detailed planning.

Note

Within this post I refer sometimes to "wrapped" method or function or task. What I mean by that is the (synchronous) function that is "wrapped" by django_q.tasks.async_task().

Issues

There are 5 issues, each of which need to be considered at the level of powercrud as well as of downstream projects:

  1. Conflict detection: maintain records of object ids that are subject to currently running async tasks. Prevent initiation of update (synchronous or asynchronous) if there is a clash. The conflict management framework needs to be shared across both levels. Need to use either (redis) cache or custom database model.
  2. Dashboard: I'd prefer to keep this at the downstream project level if possible, rather than bogging powercrud down with extraneous models required to support this stuff. But if a dashboard (and supporting model) is needed, then we need some way of getting powercrud bulk update async tasks into the dashboard. So need hooks or override methods or something.
  3. Progress Updates: the "easy" pattern is to always write from within the "wrapped" task (ie the synchronous task that is called using async_task()) to the redis cache (or a custom model) and then use htmx polling on the front end where needed. We need front end progress updates probably in powercrud on the modals that pop-up to indicate conflict with existing async task. If we use redis then there's no need to worry about extraneous models.
  4. Launching async task: this is more about having a common pattern for launching, since at both levels there is a need to share common conflict management framework, and possibly also the dashboard and progress frameworks.
  5. Error handling cascades: we need a way to ensure cleanup of problematic task instances, including cleaning up associated progress and conflict and custom dashboard data.

Django Async Task Conflict Prevention for PowerCRUD

By introducing async processing we introduce potential conflicts. Also want to allow downstream project developers to use the same base mechanism for conflict detection, in case they want to run async processes independent of powercrud (eg save() method that updates child and descendant objects) but want overall conflict detection.

Problems to Solve

  1. Lock conflicts - preventing simultaneous updates to same objects
  2. User state confusion - users making decisions based on stale data while async tasks are running
  3. Race conditions - timing-dependent bugs in concurrent operations
  4. Complex dependencies - bulk operations, single saves with descendant updates, and async tasks can all affect overlapping sets of objects
  5. Downstream flexibility - powercrud needs to work with any downstream project's specific object relationships