Similarly to Action Mailbox, these tests didn't pass because the skip
patch was included before ActiveSupport::TestCase was defined. Moving
the patch to the bottom of the file fixes the issue.
Once the skip patch was fixed, all of the skips were due to differences
in adapters which aren't really test skips as much as tests that should
not ever run against those adapters.
It's possible since Rails 6 (3ea2857943dc294d7809930b4cc5b318b9c39577) to let the framework create Event objects, but the guides and docs weren't updated to lead with this example.
Manually instantiating an Event doesn't record CPU time and allocations, I've seen it more than once that people copy-pasting the example code get confused about these stats returning 0. The tests here show that - just like the apps I've worked on - the old pattern keeps getting copy-pasted.
Sidekiq has a useful optimisation called `push_bulk` that enqueues many jobs at
once, eliminating the repeated Redis roundtrips. However, this feature is not
exposed through Active Job, so it only works for `Sidekiq::Worker` jobs. This
adds a barrier to Active Job adoption for apps that rely on this feature. It
also makes it harder for other queue adapters to implement similar
functionality, as they then have to take care of serialization, callbacks, etc.
themselves.
This commit adds `ActiveJob.perform_all_later(<job1>, <job2>)`, backed by
Sidekiq's `push_bulk` and with a fallback to enqueuing serially if the queue
adapter does not support bulk enqueue.
The performance benefit for 1000 jobs can be more than an order of magnitude:
| Enqueue type | Serial time (ms) | Bulk time (ms) | Speedup |
| ------------------ | ---------------- | -------------- | ------- |
| Raw Sidekiq | 2661 | 119 | 22x |
| Active Job Sidekiq | 2853 | 208 | 14x |
(Measured in a simple test app in our production environment.)
Instrumentation for perform_all_later uses a new event `enqueue_all.active_job`
There is presently no clean way of telling a caller of `perform_later`
the reason why a job failed to enqueue. When the job is enqueued
successfully, the job object itself is returned, but when the job can
not be enqueued, only `false` is returned. This does not allow callers
to distinguish between classes of failures.
One important class of failures is when the job backend experiences a
network partition when communicating with its underlying datastore. It
is entirely possible for that network partition to recover and as such,
code attempting to enqueue a job may wish to take action to reenqueue
that job after a brief delay. This is distinguished from the class of
failures where due a business rule defined in a callback in the
application, a job fails to enqueue and should not be retried.
This PR changes the following:
- Allows a block to be passed to the `perform_later` method. After the
`enqueue` method is executed, but before the result is returned, the
job will be yielded to the block. This allows the code invoking the
`perform_later` method to inspect the job object, even in failure
scenarios.
- Adds an exception `EnqueueError` which job adapters can raise if they
detect a problem specific to their underlying implementation or
infrastructure during the enqueue process.
- Adds two properties to the job base class: `successfully_enqueued` and
`enqueue_error`. `enqueue_error` will be populated by the `enqueue`
method if it rescues an `EnqueueError` raised by the job backend.
`successfully_enqueued` will be true if the job is not rejected by
callbacks and does not cause the job backend to raise an
`EnqueueError` and will be `false` otherwise.
This will allow developers to do something like the following:
MyJob.perform_later do |job|
unless job.successfully_enqueued?
if job.enqueue_error&.message == "Redis was unavailable"
# invoke some code that will retry the job after a delay
end
end
end
