Durations that were round-tripped through ActiveJob::Arguments.serialize
would appear fine at a first glance, but trying to perform
duration-math on them would fail:
```
irb(main):001:0> d = ActiveJob::Arguments.deserialize(ActiveJob::Arguments.serialize([1.year]))[0]
=> 1 year
irb(main):002:0> d + 1.day
activesupport-6.1.4.4/lib/active_support/duration.rb:242:in `+': undefined method `merge' for [[:years, 1]]:Array (NoMethodError)
```
This is an internal fix, not user facing. I noticed it while working on https://github.com/rails/rails/pull/48585.
The `async` adapter has an `immediate` option, which should only be used in tests. This option should tell the adapter to run jobs inline. This works correctly with `perform_later`, but it does not work with `enqueue_at`, which is what other internal mechanisms such as `retry_job` use.
This PR fixes this bug.
This is an internal fix, not user facing.
Currently the `LoggingJob` does not accept more than one argument. But there's [a few tests](f46d3452ae/activejob/test/cases/test_helper_test.rb (L518-L537)) that call it with multiple arguments and assert that it is queued correctly. Those tests pass because the job is not performed, but if the job was performed, they'd fail.
This PR just fixes `LoggingJob` to accept a splat of arguments, and adds a test to ensure that it works correctly.
The testing guide for Active Job currently implies that when you queue a job it will be performed.
This isn't true; by default jobs are enqueued, not performed.
This PR fleshes out the docs a bit to show both examples, and adds a test to confirm the default behaviour.
This has utility for gems/modules included on jobs, which can tie into this behaviour and run something when a job fails.
after_discard respects the existing retry behaviour, but will run even if a retried exception is handled in a block.
In this test case we only check if the third child's queue adpter is nil or not. We should instead check if the queue adapter is the same as base adapter to ensure that it did not change when we changed qdapters for other jobs.
50f0c0f064/activejob/test/cases/queue_adapter_test.rb (L44-L52)
This gives queue adapters more freedom to name and organize their code.
For example, if `FancyQueue` wants to have their adapter at
`FancyQueue::ActiveJobAdapter`, the name would be `active_job` before this
change. After this change, they can implement `queue_adapter_name` to return
`fancy_queue`.
Before this change, setting a module as queue adapter would result in
`queue_adapter_name` being `"module"`. This PR fixes the name extraction logic
to handle module or class queue adapters.
The display_name method is used by delayed job to log information
about a certain job, including failure messages. Whenever a job class
is moved or deleted, the instances still scheduled cannot be
constantized anymore, causing display_name and hence the log method to
raise an exception. In certain cases, e.g. when logging happens in a
rescue block, this may terminate the entire delayed job worker. With
the failsafe method, the worker handles failed jobs gracefully and
continues work, all with appropriate log output.
Sidekiq has a useful optimisation called `push_bulk` that enqueues many jobs at
once, eliminating the repeated Redis roundtrips. However, this feature is not
exposed through Active Job, so it only works for `Sidekiq::Worker` jobs. This
adds a barrier to Active Job adoption for apps that rely on this feature. It
also makes it harder for other queue adapters to implement similar
functionality, as they then have to take care of serialization, callbacks, etc.
themselves.
This commit adds `ActiveJob.perform_all_later(<job1>, <job2>)`, backed by
Sidekiq's `push_bulk` and with a fallback to enqueuing serially if the queue
adapter does not support bulk enqueue.
The performance benefit for 1000 jobs can be more than an order of magnitude:
| Enqueue type | Serial time (ms) | Bulk time (ms) | Speedup |
| ------------------ | ---------------- | -------------- | ------- |
| Raw Sidekiq | 2661 | 119 | 22x |
| Active Job Sidekiq | 2853 | 208 | 14x |
(Measured in a simple test app in our production environment.)
Instrumentation for perform_all_later uses a new event `enqueue_all.active_job`
Since Sidekiq 7.0 requires Ruby 2.7, older versions of Rails that
support older Rubies are still tested against Sidekiq 6.x.
Sidekiq::MAJOR was added in 7.0 so it can't be used for version testing,
see 862dc5b
The config change is due to changes in Sidekiq 6.5. These were accounted
for in 7a069dc but removed in 6d31993.
Sidekiq::MAJOR was added in mperham/sidekiq@b4092e3, which is only
included in 7.0.0+ so we can't use it to check the version for
Sidekiq 6.
Since the test code is written to support both, the condition should
also support both.
This commit adds `ActiveJob.deprecator` and replaces all usages of
`ActiveSupport::Deprecation.warn` in `activejob/lib` with
`ActiveJob.deprecator`.
Additionally, this commit adds `ActiveJob.deprecator` to
`Rails.application.deprecators` so that it can be configured via
settings such as `config.active_support.report_deprecations`.
This commit also removes a defunct `ActiveSupport::Deprecation.silence`
call that was added in 9eb4b4ed01f893c2fc7ff3ee4f99301dec7ea3c2 but not
removed when the deprecation was completed in
10bd5e59c3291bb2034869e97cca30634a559afb.
I noticed a difference in behavior when inspecting `ActiveJob::Core#scheduled_at` in non-test environments vs when inspecting in while executing tests.
Prior to this change, `ActiveJob::TestHelper` would set `ActiveJob::Core#scheduled_at` to a `Time` object in tests,
however this attribute was set to a `Float` objects in non-test environments:
<d1aa6af5cc/activejob/lib/active_job/core.rb (L161)>
This change ensures parity in all runtime environments.
We recently let a few very easy to avoid warnings get merged.
The root cause is that locally the test suite doesn't run in
verbose mode unless you explictly pass `-w`.
On CI warnings are enabled, but there is no reason to look at the
build output unless something is failing. And even if one wanted
to do that, that would be particularly work intensive since warnings
may be specific to a Ruby version etc.
Because of this I believe we should:
- Always run the test suite with warnings enabled.
- Raise an error if a warning is unexpected.
We've been using this pattern for a long time at Shopify both in private
and public repositories.
Previously, BigDecimal was listed as not needing a serializer. However,
when used with an adapter storing the job arguments as JSON, it would get
serialized as a simple String, resulting in deserialization also producing
a String (instead of a BigDecimal).
By using a serializer, we ensure the round trip is safe.
During upgrade deployments of applications with multiple replicas making use of
BigDecimal job arguments with a queue adapter serializing to JSON, there exists
a possible race condition, whereby a "new" replica enqueues a job with an
argument serialized using `BigDecimalSerializer`, and an "old" replica fails to
deserialize it (as it does not have `BigDecimalSerializer`).
Therefore, to ensure safe upgrades, serialization will not use
`BigDecimalSerializer` until `config.active_job.use_big_decimal_serializer` is
enabled, which can be done safely after successful deployment of Rails 7.1.
This option will be removed in Rails 7.2, when it will become the default.
The Active Job integration tests for Que and Queue Classic assume a
connection to a PostgreSQL database running on localhost:5432 since the
psql command to create the databases does not pass the -h and -p options.
In other log messages like perform/performed and ActiveJob error logging, the job ID and exception message were already included, eg:
Error performing TestFailureJob (Job ID: d70ad13e-e58b-409c-a8cc-e0447fc792b5) from Resque(default) in 1446.56ms: RuntimeError (Error Message):
But log message related to retry/discard behavior from ActiveJob::Exceptions did not include the Job ID or exception message. We now include them, in a consistent format with other existing messages.
Job ID is especially useful because, while other ActiveJob-related log messages get the Job ID via tagged logging, retry-related log messages do not currently end up tagged with Job ID. And it's really useful to use Job ID to be able to collect all the log lines related to a particular job, put the error messages together with the retry and subsequent perform messages.
We also include job.executions in the enqueue_retry log message because it was available and useful, to know if this is the 1st retry or 2nd or whatever. Previously job.executions was included in retry_stopped logging, but not enqueue_retry.
Replace Que.* settings that were removed in Que 1.0 with options passed
to Que::Locker. The Locker class creates its own thread for managing
work distribution, so the Thread created in the QueJobsManager was also
removed.
Using `assert` by itself without an error message leads to test output
like the following:
```
Failure:
QueuingTest#test_current_timezone_is_kept_while_running_perform_later [.../rails/activejob/test/integration/queuing_test.rb:118]:
Expected false to be truthy.
```
These added helpers provide a more specific error message so that the
reasons for test failures become more clear:
```
Failure:
QueuingTest#test_current_timezone_is_kept_while_running_perform_later [.../rails/activejob/test/integration/queuing_test.rb:118]:
Job AJ-63fe8bf6-7062-4a56-a10a-97c37fcaa6e6 was not executed
```
Fixing the synchronous setting uncovered some more errors with Que 1.0:
exception tests with retry_on started failing due to an assertion in
Que's Active Job integration that running jobs won't nest. However, this
is not the case when running retriable jobs synchronously.
This change overrides Que's Active Job wrapper to not make this
assertion.
I also opened an issue in the que repo: https://github.com/que-rb/que/issues/329
Que.mode was removed in 6783fdd68a
and replaced by Que.run_synchronously = true per changelog:
fa4988ee33/CHANGELOG.md (L120)
It looks like it started failing because Que 1.0 was released yesterday.
Error messages from assert_enqueued_with and assert_performed_with
shows which other jobs are enqueued or performed when you get an
error. We can improve these messages to make it clearer why the job
didn't match. This one error messages now have three different formats
based on why the job didn't match
1. Didn't match because no jobs are queued at all. This now reports,
```
No performed job found with {:job=>HelloJob, :args=>[]}
No jobs where performed
```
2. Didn't match because that job class was not queued. This now reports,
```
No performed job found with {:job=>MultipleKwargsJob, :args=>[#<Person:0x00007fe38f9f8100 @id=11>]}
No jobs of class MultipleKwargsJob where performed, job classes performed: HelloJob
```
3. Doesn't match due to arguments, queue, priority or other reason. This now reports
```
No performed job found with {:job=>HelloJob, :args=>[#<Person:0x00007fe3a89fc2c8 @id=11>]}
Potential matches: {"job_class"=>"HelloJob", "job_id"=>"f8636fd9-c7a0-4565-9198-17e175f30f0e", "provider_job_id"=>nil, "queue_name"=>"default", "priority"=>nil, "arguments"=>[{"_aj_globalid"=>"gid://aj/Person/9"}], "executions"=>0, "exception_executions"=>{}, "locale"=>"en", "timezone"=>nil, "enqueued_at"=>"2021-10-27T23:19:54Z", :job=>HelloJob, :args=>[#<Person:0x00007fe3a89dce00 @id="9">], :queue=>"default", :priority=>nil}
```
Which matches the old error message, but only reports jobs that where
queued of the same class as the job you are asserting was queued.
The ModuleSerializer does not support serializing anonymous classes
because when we try to deserialize the anonymous class, it wouldn't
know which class to use (since class name is nil).
For this reason, ModuleSerialzier now raises an error if the class
name is nil. Previously, ModuleSerializer would raise an `undefined
method `constantize' for nil:NilClass` error during deserialization.
It's not clear why the deserialization failed from the error.
In this commit, we raise an explicit error when trying to serialize
an anonymous class indicating this behaviour is not supported.
Previously in `perform_enqueued_jobs`, `deserialize_arguments_if_needed`
was called before calling `perform_now`. When a record no longer exists
and is serialized using GlobalID this led to raising
an `ActiveJob::DeserializationError` before reaching `perform_now` call.
This behaviour makes difficult testing the job `discard_on/retry_on` logic.
Now `deserialize_arguments_if_needed` call is postponed to when `perform_now`
is called.
Example:
```ruby
class UpdateUserJob < ActiveJob::Base
discard_on ActiveJob::DeserializationError
def perform(user)
# ...
end
end
User.destroy_all
assert_nothing_raised do
perform_enqueued_jobs only: UpdateUserJob
end
assert_no_enqueued_jobs
```
Before this changes the test will fail, now it passes.
This commit restores the test deleted in
cd22ecbfc2
Active Job should not test things about autoloading, this would
belong to the railties test suite probably. However, there, it feels
a bit too distant from here.
Imperfect, but on a second thought I believe this trade-off is better.
I found an unexpected use of assertion in the block of `assert_raise`
when I implemented https://github.com/rubocop/rubocop-minitest/pull/137.
It is expected to be asserted after an exception is raised in
`assert_raise` block, but in actually it is not asserted after an
exception is raised. Therefore, this PR removes or updates assertions
that have not been asserted after an exception has raised.
This PR will add `rubocop-minitest` and enable
`Minitest/UnreachableAssertion` cop to able similar auto-detection,
but will remove `rubocop-minitest` from this PR if you don't like it.
