Announced 2022-10-11 by the Graphile Team

Graphile are pleased to announce the release of Graphile Worker 0.14, which includes many new significant changes in preparation for moving towards version 1.0.

Batch Jobs

Normally a job's payload is an object; however jobs are also allowed to have a payload which is an array of objects. Jobs of this type are called "batch jobs" and they have a few special behaviours:

• when you use job_key in replace or preserve_run_at mode, when a job is replaced/updated, instead of overwriting the payload, the existing and new payloads will be merged into a larger array (this only occurs when the existing and new payloads are both arrays, otherwise the payload is simply replaced).
• when a task executes a batch job, it may return a list of promises that is the same length as the payload array. If any of these promises reject, then the job is said to have 'partial success', the result of which is it being sent back to the queue for a retry, but with the successful objects removed from the payload so only the failed objects will be retried.

Batch jobs can be useful where you need to aggregate multiple tasks together over time for efficiency; for example if you have a notification system you might schedule a notification to be sent to a user in 2 minutes time that they received a DM. Over the next 2 minutes if any other DMs are received, these can be appended to the job payload such that when the job executes it can inform the user of all of these DMs, not just the latest one.

Other Features

• Significantly improved 'large jobs table' performance (e.g. when a large queue is locked, or there's a lot of jobs queued for task identifiers your worker instance doesn't support, or a lot of failed jobs). There's around 20x improvement in this 'worst case' performance for real user workloads.
• Added new (experimental) much faster add_jobs batch API.
• Fix error handling of cron issues in 'run' method.
• CronItem.match can now accept either a pattern string or a matcher function.

See the full update in the project's release notes.

Sponsor the project

Please support development of this project via sponsorship. With your support we can improve performance, usability and documentation at a greater rate, leading to reduced running and engineering costs for your organisation, leading to a net ROI. Find out how to sponsor Graphile Worker here.