This pull request workarounds Active Job integration test with `resque` reported at https://buildkite.com/rails/rails/builds/72277#410533b8-0676-4ad2-900f-fa8131f9833f/1826
This fix is similar to 7cf8e30 and rails/rails#40451 which is due to https://bugs.ruby-lang.org/issues/17220
- Steps to reproduce
```
- Install Docker
$ git clone https://github.com/rails/rails
$ cd rails
$ git clone https://github.com/rails/buildkite-config .buildkite/
$ RUBY_IMAGE=rubylang/ruby:master-nightly-bionic docker-compose -f .buildkite/docker-compose.yml build base && CI=1 docker-compose -f .buildkite/docker-compose.yml run activejob runner activejob 'AJ_ADAPTER=resque AJ_INTEGRATION_TESTS=true bin/test test/integration/queuing_test.rb --seed 36344'
```
- Without this commit
```
$ RUBY_IMAGE=rubylang/ruby:master-nightly-bionic docker-compose -f .buildkite/docker-compose.yml build base && CI=1 docker-compose -f .buildkite/docker-compose.yml run activejob runner activejob 'AJ_ADAPTER=resque AJ_INTEGRATION_TESTS=true bin/test test/integration/queuing_test.rb --seed 36344'
... snip ...
+++ activejob: AJ_ADAPTER=resque AJ_INTEGRATION_TESTS=true bin/test test/integration/queuing_test.rb --seed 36344
Using resque
/usr/local/lib/ruby/gems/3.0.0/gems/resque-scheduler-4.4.0/lib/resque/scheduler/lock/base.rb:50: warning: Socket.gethostbyname is deprecated; use Addrinfo.getaddrinfo instead.
Run options: --seed 36344
.F
Failure:
QueuingTest#test_should_run_jobs_enqueued_on_a_listening_queue [/rails/activejob/test/integration/queuing_test.rb:14]:
Expected false to be truthy.
bin/test test/integration/queuing_test.rb:11
F
Failure:
QueuingTest#test_current_timezone_is_kept_while_running_perform_later [/rails/activejob/test/integration/queuing_test.rb:119]:
Expected false to be truthy.
bin/test test/integration/queuing_test.rb:110
.SSF
Failure:
QueuingTest#test_current_locale_is_kept_while_running_perform_later [/rails/activejob/test/integration/queuing_test.rb:102]:
Expected false to be truthy.
bin/test test/integration/queuing_test.rb:93
.F
Failure:
QueuingTest#test_should_run_job_enqueued_in_the_future_at_the_specified_time [/rails/activejob/test/integration/queuing_test.rb:76]:
Expected false to be truthy.
bin/test test/integration/queuing_test.rb:71
SSSSSS
Finished in 34.122641s, 0.4396 runs/s, 0.2344 assertions/s.
15 runs, 8 assertions, 4 failures, 0 errors, 8 skips
You have skipped tests. Run with --verbose for details.
$
```
- With this commit
```ruby
$ RUBY_IMAGE=rubylang/ruby:master-nightly-bionic docker-compose -f .buildkite/docker-compose.yml build base && CI=1 docker-compose -f .buildkite/docker-compose.yml run activejob runner activejob 'AJ_ADAPTER=resque AJ_INTEGRATION_TESTS=true bin/test test/integration/queuing_test.rb --seed 36344'
... snip ...
+++ activejob: AJ_ADAPTER=resque AJ_INTEGRATION_TESTS=true bin/test test/integration/queuing_test.rb --seed 36344
Using resque
/usr/local/lib/ruby/gems/3.0.0/gems/resque-scheduler-4.4.0/lib/resque/scheduler/lock/base.rb:50: warning: Socket.gethostbyname is deprecated; use Addrinfo.getaddrinfo instead.
Run options: --seed 36344
....SS...SSSSSS
Finished in 13.850658s, 1.0830 runs/s, 0.7220 assertions/s.
15 runs, 10 assertions, 0 failures, 0 errors, 8 skips
You have skipped tests. Run with --verbose for details.
$
```
This pull request addresses Active Job integration test with `sneakers` reported at https://buildkite.com/rails/rails/builds/72257#20014e66-6e08-47ae-a827-b71de7148306/2017
This fix is similar to 7cf8e30902 which is due to https://bugs.ruby-lang.org/issues/17220
- Steps to reproduce
Install ruby 3.0.0dev using ruby-build or whatever then execute these steps.
```ruby
$ git clone https://github.com/rails/rails
$ cd rails/activejob
$ bundle install
$ bundle exec rake test:integration:sneakers
```
- Result without this commit
```ruby
$ ruby -v
ruby 3.0.0dev (2020-10-24T13:53:53Z master 148961adcd) [x86_64-linux]
$ cd activejob
$ bundle exec rake test:integration:sneakers
... snip ...
/home/yahonda/src/github.com/rails/rails/activejob/test/support/integration/adapters/sneakers.rb:49:in `rescue in start_workers': Failed to start sneakers worker (RuntimeError)
from /home/yahonda/src/github.com/rails/rails/activejob/test/support/integration/adapters/sneakers.rb:41:in `start_workers'
from /home/yahonda/src/github.com/rails/rails/activejob/test/support/integration/helper.rb:27:in `<top (required)>'
from /home/yahonda/src/github.com/rails/rails/activejob/test/helper.rb:12:in `require'
from /home/yahonda/src/github.com/rails/rails/activejob/test/helper.rb:12:in `<top (required)>'
from /home/yahonda/src/github.com/rails/rails/activejob/test/integration/queuing_test.rb:3:in `require'
from /home/yahonda/src/github.com/rails/rails/activejob/test/integration/queuing_test.rb:3:in `<top (required)>'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:17:in `require'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:17:in `block in <main>'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:5:in `select'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:5:in `<main>'
/home/yahonda/src/github.com/rails/rails/activejob/test/support/integration/adapters/sneakers.rb:44:in `sleep': execution expired (Timeout::Error)
from /home/yahonda/src/github.com/rails/rails/activejob/test/support/integration/adapters/sneakers.rb:44:in `block in start_workers'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/3.0.0/timeout.rb:112:in `timeout'
from /home/yahonda/src/github.com/rails/rails/activejob/test/support/integration/adapters/sneakers.rb:42:in `start_workers'
from /home/yahonda/src/github.com/rails/rails/activejob/test/support/integration/helper.rb:27:in `<top (required)>'
from /home/yahonda/src/github.com/rails/rails/activejob/test/helper.rb:12:in `require'
from /home/yahonda/src/github.com/rails/rails/activejob/test/helper.rb:12:in `<top (required)>'
from /home/yahonda/src/github.com/rails/rails/activejob/test/integration/queuing_test.rb:3:in `require'
from /home/yahonda/src/github.com/rails/rails/activejob/test/integration/queuing_test.rb:3:in `<top (required)>'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:17:in `require'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:17:in `block in <main>'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:5:in `select'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:5:in `<main>'
rake aborted!
Command failed with status (1): [ruby -w -I"lib:test" -I"/home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib" "/home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb" "test/integration/queuing_test.rb" ]
/home/yahonda/.rbenv/versions/3.0.0-dev/bin/bundle:23:in `load'
/home/yahonda/.rbenv/versions/3.0.0-dev/bin/bundle:23:in `<main>'
Tasks: TOP => test:integration:sneakers
(See full trace by running task with --trace)
$
```
- Result with this commit
```ruby
$ bundle exec rake test:integration:sneakers
... snip ...
15 runs, 6 assertions, 0 failures, 0 errors, 11 skips
```
The `Sidekiq::Logging` module was removed in Sidekiq 6.0 in favour of
`Sidekiq.logger=`.
https://github.com/mperham/sidekiq/wiki/Logging#api-changes
> The old Sidekiq::Logging class was removed in Sidekiq 6.0. You should
> not be accessing any Sidekiq internal classes to use the logger. Use
> `logger` directly in your Worker or use `Sidekiq.logger` elsewhere.
Since `@mutex.synchronize` is enforced in the `ensure_connection!` method,
there is no need to do so on the caller side.
c1b47f9c5d/lib/sneakers/publisher.rb (L22-L26)
Due to this, `ThreadError(deadlock; recursive locking)` has occurred.
psqlrc files can affect the execution of commands in ways that can hold
up execution by blocking or otherwise cause unexpected side effects and
should best be ignored when using psql programmatically.
Currently when executing `bundle exec rake test:integration` under `activejob/`
derectory, it prints helpful info like:
```
(snip)
*** rake aj:integration:async ***
(snip)
*** rake aj:integration:delayed_job ***
(snip)
```
but there is no defined `:aj` scope in `activejob/Rakefile`,
so I think output should be like:
```
(snip)
*** rake test:integration:async ***
(snip)
*** rake test:integration:delayed_job ***
(snip)
```
By the way `rake test:integration` doesn't work if execute it without
prepending `bundle exec` to that command. It is probably what we should
fix too.
Reasons are that the Qu gem wasn't compatible since Rails 5.1,
gem development was stopped in 2014 and maintainers have
confirmed its demise. See issue #32273
Record what was the current timezone in effect when the job was
enqueued and then restore when the job is executed in same way
that the current locale is recorded and restored.
Currently, sidekiq integration test + Ruby 2.5.0-rc1 show exception as follows.
```
#<Thread:0x000000000670bec0@/home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:23 run> terminated with exception (report_on_exception is true):
/home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.1.7/lib/bootsnap/load_path_cache/core_ext/active_support.rb:53:in `block in load_missing_constant': uninitialized constant Sidekiq::CLI (NameError)
from /home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.1.7/lib/bootsnap/load_path_cache/core_ext/active_support.rb:8:in `without_bootsnap_cache'
from /home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.1.7/lib/bootsnap/load_path_cache/core_ext/active_support.rb:53:in `rescue in load_missing_constant'
from /home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.1.7/lib/bootsnap/load_path_cache/core_ext/active_support.rb:42:in `load_missing_constant'
from /home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.0.5/lib/sidekiq/launcher.rb:65:in `heartbeat'
from /home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.0.5/lib/sidekiq/launcher.rb:123:in `start_heartbeat'
```
https://travis-ci.org/rails/rails/jobs/317187279#L2152
The reason for this is that `Sidekiq::CLI` has not been loaded.
Sidekiq integration test launches a Sidekiq instance within
another Ruby process. In such a case, need to require 'sidekiq/cli'
in that launch code.
Ref: https://github.com/mperham/sidekiq/pull/3692#issuecomment-352032251
Omit `rails activestorage:install` for jdbcmysql, jdbc and shebang tests
AppGeneratorTest#test_config_jdbcmysql_database
rails aborted!
LoadError: Could not load 'active_record/connection_adapters/mysql_adapter'.
Make sure that the adapter in config/database.yml is valid.
If you use an adapter other than 'mysql2', 'postgresql' or 'sqlite3' add
the necessary adapter gem to the Gemfile.
(compressed)
bin/rails:4:in `<main>'
Tasks: TOP => activestorage:install => environment
(See full trace by running task with --trace)
AppGeneratorTest#test_config_jdbc_database
rails aborted!
LoadError: Could not load 'active_record/connection_adapters/jdbc_adapter'.
Make sure that the adapter in config/database.yml is valid.
If you use an adapter other than 'mysql2', 'postgresql' or 'sqlite3' add
the necessary adapter gem to the Gemfile.
(compressed)
bin/rails:4:in `<main>'
Tasks: TOP => activestorage:install => environment
(See full trace by running task with --trace)
AppGeneratorTest#test_shebang_is_added_to_rails_file
/home/ubuntu/.rbenv/versions/2.4.1/bin/ruby: no Ruby script found in input (LoadError)
Prevent PendingMigrationError in tests
* Run `bin/rails db:migrate RAILS_ENV=test` in test_cases before start tests to prevent PendingMigrationError
* FileUtils.rm_r("db/migrate")
* --skip-active-storage
Fix failed tests in `railties/test/railties/engine_test.rb`
Related to #30111
Imporve `SharedGeneratorTests#test_default_frameworks_are_required_when_others_are_removed`
- Explicitly skip active_storage
- Ensure that skipped frameworks are commented
- Ensure that default frameworks are not commented
Fix error `Errno::ENOSPC: No space left on device - sendfile`
Since `rails new` runs `rails active_storage:install`
that boots an app.
Since adding Bootsnap 0312a5c67e35b960e33677b5358c539f1047e4e1
during booting an app, it creates the cache:
264K tmp/cache/bootsnap-load-path-cache
27M tmp/cache/bootsnap-compile-cache
* teardown_app must remove app
* Use `gem 'redis', '~> 4.0'` for new app Gemfiles
* Loosen Action Cable redis-rb dep to `>= 3.3, < 5`
* Bump redis-namespace for looser Redis version dep
* Avoid using the underlying `redis.client` directly
* Use `Redis.new` instead of `Redis.connect`
Since f55ecc6, the integration test of sidekiq and resque is not working
in CI.
https://travis-ci.org/rails/rails/jobs/251783876
Because f55ecc6 required a password to access redis.
Therefore, handling by passing passwords when connecting to redis.
Currently, backburner integration test is not running on CI.
https://travis-ci.org/rails/rails/jobs/196005322#L610
Using `Backburner::Worker.connection` to check whether beanstalkd is
running. But `Backburner::Worker.connection` was removed in backburner
1.2.0.
81fde499c2
Therefore, this check process always becomes false, so the test is no
longer done. I fixed it so that check processing is done correctly.
On most filesystems file ctime is limited to 1 second granularity, which means that on
faster computers multiple simple jobs (for instance dummy TestJob) can finish within the
same second.
The execution order test in ActiveJob integration tests relies on multiple TestJobs
writing files then comparing the ctime. As a result integration tests would sometimes
fail as the ctime of the files written by these TestJobs could have coincidental ctimes
making the comparison for job order fail.
This commit adds a far more precise execution time (to the extent that the Ruby Time
class allows) to the file created by TestJob, and updates the execution order assertion
to use it, removing the race condition.
