Right now we are using both to test the Rails applications we generate
and to test Rails itself. Let's keep CI for the app and BUILDKITE to
the framework.
This is an internal fix, not user facing. I noticed it while working on https://github.com/rails/rails/pull/48585.
The `async` adapter has an `immediate` option, which should only be used in tests. This option should tell the adapter to run jobs inline. This works correctly with `perform_later`, but it does not work with `enqueue_at`, which is what other internal mechanisms such as `retry_job` use.
This PR fixes this bug.
Since Sidekiq 7.0 requires Ruby 2.7, older versions of Rails that
support older Rubies are still tested against Sidekiq 6.x.
Sidekiq::MAJOR was added in 7.0 so it can't be used for version testing,
see 862dc5b
The config change is due to changes in Sidekiq 6.5. These were accounted
for in 7a069dc but removed in 6d31993.
Sidekiq::MAJOR was added in mperham/sidekiq@b4092e3, which is only
included in 7.0.0+ so we can't use it to check the version for
Sidekiq 6.
Since the test code is written to support both, the condition should
also support both.
The Active Job integration tests for Que and Queue Classic assume a
connection to a PostgreSQL database running on localhost:5432 since the
psql command to create the databases does not pass the -h and -p options.
Replace Que.* settings that were removed in Que 1.0 with options passed
to Que::Locker. The Locker class creates its own thread for managing
work distribution, so the Thread created in the QueJobsManager was also
removed.
Fixing the synchronous setting uncovered some more errors with Que 1.0:
exception tests with retry_on started failing due to an assertion in
Que's Active Job integration that running jobs won't nest. However, this
is not the case when running retriable jobs synchronously.
This change overrides Que's Active Job wrapper to not make this
assertion.
I also opened an issue in the que repo: https://github.com/que-rb/que/issues/329
* Remove spring as a default installation option
Faster computers have meant that most apps won't see a big benefit from Spring on small to moderate size apps. Thus the pain of dealing with the occasional spring issue is no longer warranted by default.
* Errant end
* No longer an option
* Additional spring removals
* Pointer to docs is enough
This pull request workarounds Active Job integration test with `resque` reported at https://buildkite.com/rails/rails/builds/72277#410533b8-0676-4ad2-900f-fa8131f9833f/1826
This fix is similar to 7cf8e30 and rails/rails#40451 which is due to https://bugs.ruby-lang.org/issues/17220
- Steps to reproduce
```
- Install Docker
$ git clone https://github.com/rails/rails
$ cd rails
$ git clone https://github.com/rails/buildkite-config .buildkite/
$ RUBY_IMAGE=rubylang/ruby:master-nightly-bionic docker-compose -f .buildkite/docker-compose.yml build base && CI=1 docker-compose -f .buildkite/docker-compose.yml run activejob runner activejob 'AJ_ADAPTER=resque AJ_INTEGRATION_TESTS=true bin/test test/integration/queuing_test.rb --seed 36344'
```
- Without this commit
```
$ RUBY_IMAGE=rubylang/ruby:master-nightly-bionic docker-compose -f .buildkite/docker-compose.yml build base && CI=1 docker-compose -f .buildkite/docker-compose.yml run activejob runner activejob 'AJ_ADAPTER=resque AJ_INTEGRATION_TESTS=true bin/test test/integration/queuing_test.rb --seed 36344'
... snip ...
+++ activejob: AJ_ADAPTER=resque AJ_INTEGRATION_TESTS=true bin/test test/integration/queuing_test.rb --seed 36344
Using resque
/usr/local/lib/ruby/gems/3.0.0/gems/resque-scheduler-4.4.0/lib/resque/scheduler/lock/base.rb:50: warning: Socket.gethostbyname is deprecated; use Addrinfo.getaddrinfo instead.
Run options: --seed 36344
.F
Failure:
QueuingTest#test_should_run_jobs_enqueued_on_a_listening_queue [/rails/activejob/test/integration/queuing_test.rb:14]:
Expected false to be truthy.
bin/test test/integration/queuing_test.rb:11
F
Failure:
QueuingTest#test_current_timezone_is_kept_while_running_perform_later [/rails/activejob/test/integration/queuing_test.rb:119]:
Expected false to be truthy.
bin/test test/integration/queuing_test.rb:110
.SSF
Failure:
QueuingTest#test_current_locale_is_kept_while_running_perform_later [/rails/activejob/test/integration/queuing_test.rb:102]:
Expected false to be truthy.
bin/test test/integration/queuing_test.rb:93
.F
Failure:
QueuingTest#test_should_run_job_enqueued_in_the_future_at_the_specified_time [/rails/activejob/test/integration/queuing_test.rb:76]:
Expected false to be truthy.
bin/test test/integration/queuing_test.rb:71
SSSSSS
Finished in 34.122641s, 0.4396 runs/s, 0.2344 assertions/s.
15 runs, 8 assertions, 4 failures, 0 errors, 8 skips
You have skipped tests. Run with --verbose for details.
$
```
- With this commit
```ruby
$ RUBY_IMAGE=rubylang/ruby:master-nightly-bionic docker-compose -f .buildkite/docker-compose.yml build base && CI=1 docker-compose -f .buildkite/docker-compose.yml run activejob runner activejob 'AJ_ADAPTER=resque AJ_INTEGRATION_TESTS=true bin/test test/integration/queuing_test.rb --seed 36344'
... snip ...
+++ activejob: AJ_ADAPTER=resque AJ_INTEGRATION_TESTS=true bin/test test/integration/queuing_test.rb --seed 36344
Using resque
/usr/local/lib/ruby/gems/3.0.0/gems/resque-scheduler-4.4.0/lib/resque/scheduler/lock/base.rb:50: warning: Socket.gethostbyname is deprecated; use Addrinfo.getaddrinfo instead.
Run options: --seed 36344
....SS...SSSSSS
Finished in 13.850658s, 1.0830 runs/s, 0.7220 assertions/s.
15 runs, 10 assertions, 0 failures, 0 errors, 8 skips
You have skipped tests. Run with --verbose for details.
$
```
This pull request addresses Active Job integration test with `sneakers` reported at https://buildkite.com/rails/rails/builds/72257#20014e66-6e08-47ae-a827-b71de7148306/2017
This fix is similar to 7cf8e30902 which is due to https://bugs.ruby-lang.org/issues/17220
- Steps to reproduce
Install ruby 3.0.0dev using ruby-build or whatever then execute these steps.
```ruby
$ git clone https://github.com/rails/rails
$ cd rails/activejob
$ bundle install
$ bundle exec rake test:integration:sneakers
```
- Result without this commit
```ruby
$ ruby -v
ruby 3.0.0dev (2020-10-24T13:53:53Z master 148961adcd) [x86_64-linux]
$ cd activejob
$ bundle exec rake test:integration:sneakers
... snip ...
/home/yahonda/src/github.com/rails/rails/activejob/test/support/integration/adapters/sneakers.rb:49:in `rescue in start_workers': Failed to start sneakers worker (RuntimeError)
from /home/yahonda/src/github.com/rails/rails/activejob/test/support/integration/adapters/sneakers.rb:41:in `start_workers'
from /home/yahonda/src/github.com/rails/rails/activejob/test/support/integration/helper.rb:27:in `<top (required)>'
from /home/yahonda/src/github.com/rails/rails/activejob/test/helper.rb:12:in `require'
from /home/yahonda/src/github.com/rails/rails/activejob/test/helper.rb:12:in `<top (required)>'
from /home/yahonda/src/github.com/rails/rails/activejob/test/integration/queuing_test.rb:3:in `require'
from /home/yahonda/src/github.com/rails/rails/activejob/test/integration/queuing_test.rb:3:in `<top (required)>'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:17:in `require'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:17:in `block in <main>'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:5:in `select'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:5:in `<main>'
/home/yahonda/src/github.com/rails/rails/activejob/test/support/integration/adapters/sneakers.rb:44:in `sleep': execution expired (Timeout::Error)
from /home/yahonda/src/github.com/rails/rails/activejob/test/support/integration/adapters/sneakers.rb:44:in `block in start_workers'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/3.0.0/timeout.rb:112:in `timeout'
from /home/yahonda/src/github.com/rails/rails/activejob/test/support/integration/adapters/sneakers.rb:42:in `start_workers'
from /home/yahonda/src/github.com/rails/rails/activejob/test/support/integration/helper.rb:27:in `<top (required)>'
from /home/yahonda/src/github.com/rails/rails/activejob/test/helper.rb:12:in `require'
from /home/yahonda/src/github.com/rails/rails/activejob/test/helper.rb:12:in `<top (required)>'
from /home/yahonda/src/github.com/rails/rails/activejob/test/integration/queuing_test.rb:3:in `require'
from /home/yahonda/src/github.com/rails/rails/activejob/test/integration/queuing_test.rb:3:in `<top (required)>'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:17:in `require'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:17:in `block in <main>'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:5:in `select'
from /home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:5:in `<main>'
rake aborted!
Command failed with status (1): [ruby -w -I"lib:test" -I"/home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib" "/home/yahonda/.rbenv/versions/3.0.0-dev/lib/ruby/gems/3.0.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb" "test/integration/queuing_test.rb" ]
/home/yahonda/.rbenv/versions/3.0.0-dev/bin/bundle:23:in `load'
/home/yahonda/.rbenv/versions/3.0.0-dev/bin/bundle:23:in `<main>'
Tasks: TOP => test:integration:sneakers
(See full trace by running task with --trace)
$
```
- Result with this commit
```ruby
$ bundle exec rake test:integration:sneakers
... snip ...
15 runs, 6 assertions, 0 failures, 0 errors, 11 skips
```
The `Sidekiq::Logging` module was removed in Sidekiq 6.0 in favour of
`Sidekiq.logger=`.
https://github.com/mperham/sidekiq/wiki/Logging#api-changes
> The old Sidekiq::Logging class was removed in Sidekiq 6.0. You should
> not be accessing any Sidekiq internal classes to use the logger. Use
> `logger` directly in your Worker or use `Sidekiq.logger` elsewhere.
Since `@mutex.synchronize` is enforced in the `ensure_connection!` method,
there is no need to do so on the caller side.
c1b47f9c5d/lib/sneakers/publisher.rb (L22-L26)
Due to this, `ThreadError(deadlock; recursive locking)` has occurred.
psqlrc files can affect the execution of commands in ways that can hold
up execution by blocking or otherwise cause unexpected side effects and
should best be ignored when using psql programmatically.
Currently when executing `bundle exec rake test:integration` under `activejob/`
derectory, it prints helpful info like:
```
(snip)
*** rake aj:integration:async ***
(snip)
*** rake aj:integration:delayed_job ***
(snip)
```
but there is no defined `:aj` scope in `activejob/Rakefile`,
so I think output should be like:
```
(snip)
*** rake test:integration:async ***
(snip)
*** rake test:integration:delayed_job ***
(snip)
```
By the way `rake test:integration` doesn't work if execute it without
prepending `bundle exec` to that command. It is probably what we should
fix too.
Reasons are that the Qu gem wasn't compatible since Rails 5.1,
gem development was stopped in 2014 and maintainers have
confirmed its demise. See issue #32273
Record what was the current timezone in effect when the job was
enqueued and then restore when the job is executed in same way
that the current locale is recorded and restored.
Currently, sidekiq integration test + Ruby 2.5.0-rc1 show exception as follows.
```
#<Thread:0x000000000670bec0@/home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:23 run> terminated with exception (report_on_exception is true):
/home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.1.7/lib/bootsnap/load_path_cache/core_ext/active_support.rb:53:in `block in load_missing_constant': uninitialized constant Sidekiq::CLI (NameError)
from /home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.1.7/lib/bootsnap/load_path_cache/core_ext/active_support.rb:8:in `without_bootsnap_cache'
from /home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.1.7/lib/bootsnap/load_path_cache/core_ext/active_support.rb:53:in `rescue in load_missing_constant'
from /home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.1.7/lib/bootsnap/load_path_cache/core_ext/active_support.rb:42:in `load_missing_constant'
from /home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.0.5/lib/sidekiq/launcher.rb:65:in `heartbeat'
from /home/travis/build/rails/rails/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.0.5/lib/sidekiq/launcher.rb:123:in `start_heartbeat'
```
https://travis-ci.org/rails/rails/jobs/317187279#L2152
The reason for this is that `Sidekiq::CLI` has not been loaded.
Sidekiq integration test launches a Sidekiq instance within
another Ruby process. In such a case, need to require 'sidekiq/cli'
in that launch code.
Ref: https://github.com/mperham/sidekiq/pull/3692#issuecomment-352032251
Omit `rails activestorage:install` for jdbcmysql, jdbc and shebang tests
AppGeneratorTest#test_config_jdbcmysql_database
rails aborted!
LoadError: Could not load 'active_record/connection_adapters/mysql_adapter'.
Make sure that the adapter in config/database.yml is valid.
If you use an adapter other than 'mysql2', 'postgresql' or 'sqlite3' add
the necessary adapter gem to the Gemfile.
(compressed)
bin/rails:4:in `<main>'
Tasks: TOP => activestorage:install => environment
(See full trace by running task with --trace)
AppGeneratorTest#test_config_jdbc_database
rails aborted!
LoadError: Could not load 'active_record/connection_adapters/jdbc_adapter'.
Make sure that the adapter in config/database.yml is valid.
If you use an adapter other than 'mysql2', 'postgresql' or 'sqlite3' add
the necessary adapter gem to the Gemfile.
(compressed)
bin/rails:4:in `<main>'
Tasks: TOP => activestorage:install => environment
(See full trace by running task with --trace)
AppGeneratorTest#test_shebang_is_added_to_rails_file
/home/ubuntu/.rbenv/versions/2.4.1/bin/ruby: no Ruby script found in input (LoadError)
Prevent PendingMigrationError in tests
* Run `bin/rails db:migrate RAILS_ENV=test` in test_cases before start tests to prevent PendingMigrationError
* FileUtils.rm_r("db/migrate")
* --skip-active-storage
Fix failed tests in `railties/test/railties/engine_test.rb`
Related to #30111
Imporve `SharedGeneratorTests#test_default_frameworks_are_required_when_others_are_removed`
- Explicitly skip active_storage
- Ensure that skipped frameworks are commented
- Ensure that default frameworks are not commented
Fix error `Errno::ENOSPC: No space left on device - sendfile`
Since `rails new` runs `rails active_storage:install`
that boots an app.
Since adding Bootsnap 0312a5c67e35b960e33677b5358c539f1047e4e1
during booting an app, it creates the cache:
264K tmp/cache/bootsnap-load-path-cache
27M tmp/cache/bootsnap-compile-cache
* teardown_app must remove app
* Use `gem 'redis', '~> 4.0'` for new app Gemfiles
* Loosen Action Cable redis-rb dep to `>= 3.3, < 5`
* Bump redis-namespace for looser Redis version dep
* Avoid using the underlying `redis.client` directly
* Use `Redis.new` instead of `Redis.connect`
