This adds an additional method to configure the parallelization
threshold. Before this, the only way of configuring the threshold was
via an option:
```
config.active_support.test_parallelization_minimum_number_of_tests
```
Parallelizing tests has a cost in terms of database setup and fixture
loading. This change makes Rails disable parallelization when the number
of tests is below a configurable threshold.
When running tests in parallel each process gets its own database
instance. On each execution, each process will update each database
schema (if needed) and load all the fixtures. This can be very expensive
for non trivial datasets.
As an example, for HEY, when running a single file with 18 tests,
running tests in parallel in my box adds an overhead of 13 seconds
versus not parallelizing them. Of course parallelizing is totally worthy
when there are many tests to run, but not when running just a few tests.
The threshold is configurable via
config.active_support.test_parallelization_minimum_number_of_tests,
which is 30 50 by default.
This also adds some tracing to know how tests are being executed:
When in parallel:
```
Running 2829 tests in parallel in 8 processes
```
When not in parallel:
```
Running 15 tests in a single process (parallelization threshold is 30)
```
Reverts a change from
2327ebfdc6
which can overwrite `test_order` that may have been manually set in
config. This can cause a situation where the user is depending on a
particular `test_order` but is unknowingly forced into another.
Setting up the parallel workers could be an overhead when running
individual files.
This patch disables that process in case the number of files to run
is less than one.
Results running a sample file:
Before:
```
actionpack $ bin/test test/controller/parameters/accessors_test.rb
Run options: --seed 48261
........................................................................
Finished in 0.211923s, 339.7460 runs/s, 552.0873 assertions/s.
72 runs, 117 assertions, 0 failures, 0 errors, 0 skips
```
After
```
actionpack $ bin/test test/controller/parameters/accessors_test.rb
Run options: --seed 5461
........................................................................
Finished in 0.008411s, 8560.2189 runs/s, 13910.3557 assertions/s.
72 runs, 117 assertions, 0 failures, 0 errors, 0 skips
```
This hack prevails everywhere in the codebase by being copy & pasted, and it's actually not a negative thing but a necessary thing for framework implementors,
so it should better have a name and be a thing.
And with this commit, activesupport/test/abstract_unit.rb now doesn't silently autoload AS::TestCase,
so we're ready to establish clearner environment for running AS tests (probably in later commits)
* Add option to set parallel test worker count to the physical core count of the machine
Also, use the physical core count of the machine as
the default number of workers, and generate the `test_helper.rb` file
with `parallelize(workers: :number_of_processors)`
Closes#34734
* Ensure that we always test parallel testing
Since #34734 we decided to use the physical core count of the machine as
the default number of workers in the parallel testing, we need to
ensure that some tests use at least 2 workers because we could
run those tests on VM that has only 1 physical core.
It also fixes tests failures on the CI since Travis server we are using
has only one physical core.
See https://travis-ci.org/rails/rails/jobs/469281088#L2352
The name of the minitest library is spelled that way: regular font, and
lowercase. Lowercase is used even at the beginning of sentences, see
http://docs.seattlerb.org/minitest/
I double-checked this with @zenspider too (thanks!).
- In #32472 I introduced a fix in order for all `after_teardown` method provided by libraries and Rails to run, even if the application's `teardown` method raised an error (That's the default minitest behavior). However this change wasn't enough and doesn't take in consideration the ancestors chain.
If a library's module containing an `after_teardown` method get included after the `SetupAndTeardown` module (one example is the [ActiveRecord::TestFixtures module](7d2400ab61/activerecord/lib/active_record/fixtures.rb (L855-L856)), then the ancestors of the test class would look something like
```ruby
class MyTest < ActiveSupport::TestCase
end
puts MyTest.ancestors # [MyTest, ActiveSupport::TestCase, ActiveRecord::TestFixtures, ActiveSupport::Testing::SetupAndTeardown]
```
Any class/module in the ancestors chain that are **before** the `ActiveSupport::Testing::SetupAndTeardown` will behave incorrectly:
- Their `before_setup` method will get called **after** all regular setup method
- Their `after_teardown` method won't even get called in case an exception is raised inside a regular's test `teardown`
A simple reproduction script of the problem here https://gist.github.com/Edouard-chin/70705542a59a8593f619b02e1c0a188c
- One solution to this problem is to have the `AS::SetupAndTeardown` module be the very first in the ancestors chain. By doing that we ensure that no `before_setup` / `after_teardown` get executed prior to running the teardown callbacks
Provides both a forked process and threaded parallelization options. To
use add `parallelize` to your test suite.
Takes a `workers` argument that controls how many times the process
is forked. For each process a new database will be created suffixed
with the worker number; test-database-0 and test-database-1
respectively.
If `ENV["PARALLEL_WORKERS"]` is set the workers argument will be ignored
and the environment variable will be used instead. This is useful for CI
environments, or other environments where you may need more workers than
you do for local testing.
If the number of workers is set to `1` or fewer, the tests will not be
parallelized.
The default parallelization method is to fork processes. If you'd like to
use threads instead you can pass `with: :threads` to the `parallelize`
method. Note the threaded parallelization does not create multiple
database and will not work with system tests at this time.
parallelize(workers: 2, with: :threads)
The threaded parallelization uses Minitest's parallel exector directly.
The processes paralleliztion uses a Ruby Drb server.
For parallelization via threads a setup hook and cleanup hook are
provided.
```
class ActiveSupport::TestCase
parallelize_setup do |worker|
# setup databases
end
parallelize_teardown do |worker|
# cleanup database
end
parallelize(workers: 2)
end
```
[Eileen M. Uchitelle, Aaron Patterson]
ActiveSupport::Testing::Assertions.
We have a separate module in which have defined Rails' own custom
assertions. So it would be good to keep all custom Rails' assertions in
one place i.e. in this module.
Before:
```
DEPRECATION WARNING: Passing arguments to assert_nothing_raised
is deprecated and will be removed in Rails 5.1.
```
After:
```
DEPRECATION WARNING: Passing arguments to assert_nothing_raised is deprecated and will be removed in Rails 5.1.
```
The line filter parsing added to ActiveSupport::TestCase is only half the story
to enable line filtering. The other half, of adding the patterns to the options,
is done in the Minitest plugin that Railties has.
Thus it makes more sense to have the filter in Railties with the other half and
all the line filtering tests.
Move the filter and extend Active Support in an initializer, so that when users
or `rails/all.rb` require `rails/test_unit/railtie` we can still filter by line.
This also adds free mix and matching of directories, files and lines filters.
Like so:
bin/rails test models/post_test.rb test/integration models/person_test.rb:26
You can also mix in a traditional Minitest filter:
bin/rails test test/integration -n /check_it_out/
It's a thin layer to provide easy access to sample files throughout
test-cases. This adds the directory `test/fixtures/files` to newly
generated applications.
This patch uniformizes warning messages. I used the most common style
already present in the code base:
* Capitalize the first word.
* End the message with a full stop.
* "Rails 5" instead of "Rails 5.0".
* Backticks for method names and inline code.
Also, converted a few long strings into the new heredoc convention.
Goals:
1. Default to :random for newly generated applications
2. Default to :sorted for existing applications with a warning
3. Only show the warning once
4. Only show the warning if the app actually uses AS::TestCase
Fixes#16769
Calling ActiveSupport::TestCase.i_suck_and_my_tests_are_order_dependent! in AS::TestCase makes
everyone's tests order dependent, which should never be done by the framework.
We are planning to remove mocha from our test suite because of
performance problems. To make this possible we should stop require mocha
on ActionSupport::TestCase.
This should not affect applications since users still need to add mocha
to Gemfile and this already load mocha.
Added FIXME notes to place that still need mocha removal
Add `ActiveSupport::Testing::TimeHelpers#travel` and `#travel_to`. These
methods change current time to the given time or time difference by
stubbing `Time.now` and `Date.today` to return the time or date after
the difference calculation, or the time or date that got passed into the
method respectively. These methods also accept a block, which will
return current time back to its original state at the end of the block.
Example for `#travel`:
Time.now # => 2013-11-09 15:34:49 -05:00
travel 1.day
Time.now # => 2013-11-10 15:34:49 -05:00
Date.today # => Sun, 10 Nov 2013
Example for `#travel_to`:
Time.now # => 2013-11-09 15:34:49 -05:00
travel_to Time.new(2004, 11, 24, 01, 04, 44)
Time.now # => 2004-11-24 01:04:44 -05:00
Date.today # => Wed, 24 Nov 2004
Both of these methods also accept a block, which will return the current
time back to its original state at the end of the block:
Time.now # => 2013-11-09 15:34:49 -05:00
travel 1.day do
User.create.created_at # => Sun, 10 Nov 2013 15:34:49 EST -05:00
end
travel_to Time.new(2004, 11, 24, 01, 04, 44) do
User.create.created_at # => Wed, 24 Nov 2004 01:04:44 EST -05:00
end
Time.now # => 2013-11-09 15:34:49 -05:00
This module is included in `ActiveSupport::TestCase` automatically.
