This is too much complexity for something low value. Let's just always
warn when a test doesn't have any assertions.
If people want to raise an error or ignore them, they can do so by
overriding `Warning.warn`.
This changes the default number of workers when parallelizing tests to
be based on logical core count instead of physical core count. This may
provide a performance boost for parallelized tests.
This also reverts #50545 in favor of using the default worker count.
In the case that one single test file can't run with more than 1
parallel workers, but the base class has parallelization enabled, we
should still allow the user to set the number of workers to 1.
These methods are only available when requiring the railties
"rails/test_help" file, which is included by default in the
test_helper.rb for newly generated Rails applications.
Since requiring that file is the only way, and some applications may
have removed it or not used test-unit when generating their application,
I think it's worth calling out explicitly here.
Previously, these could be found in the Testing guide, but not in the
api documentation.
The assert_raise alias was moved to ActiveSupport::Testing::Assertions
because that is where assert_raises is defined (and rdoc handles that
case).
The Minitest aliases are not easily handled by rdoc, so they are written
by hand. We could copy the documentation for the aliased methods here,
but linking to them seems sufficient.
This fixes some issues I found while working on https://github.com/rails/rails/pull/46718
- Some of the tests in `test_runner_test.rb` generate a new test file, then run it. In some cases, the generated tests were failing. The calling test didn't expect this, but didn't fail despite that. So in this PR I have fixed the generated tests to always pass, and added `allow_failure: false` to enforce this.
- Added a missing test for when `PARALLEL_WORKERS=1`.
- Fixed a typo and removed some unnecessary wording in the docs.
This patch just changes the inspect method on test case instances.
Seeing test instance internals probably isn't helpful when an exception
is raised (for example a `NoMethodError`).
This isn't as good as #45122, but should fix#45121
This adds an additional method to configure the parallelization
threshold. Before this, the only way of configuring the threshold was
via an option:
```
config.active_support.test_parallelization_minimum_number_of_tests
```
Parallelizing tests has a cost in terms of database setup and fixture
loading. This change makes Rails disable parallelization when the number
of tests is below a configurable threshold.
When running tests in parallel each process gets its own database
instance. On each execution, each process will update each database
schema (if needed) and load all the fixtures. This can be very expensive
for non trivial datasets.
As an example, for HEY, when running a single file with 18 tests,
running tests in parallel in my box adds an overhead of 13 seconds
versus not parallelizing them. Of course parallelizing is totally worthy
when there are many tests to run, but not when running just a few tests.
The threshold is configurable via
config.active_support.test_parallelization_minimum_number_of_tests,
which is 30 50 by default.
This also adds some tracing to know how tests are being executed:
When in parallel:
```
Running 2829 tests in parallel in 8 processes
```
When not in parallel:
```
Running 15 tests in a single process (parallelization threshold is 30)
```
Reverts a change from
2327ebfdc6
which can overwrite `test_order` that may have been manually set in
config. This can cause a situation where the user is depending on a
particular `test_order` but is unknowingly forced into another.
Setting up the parallel workers could be an overhead when running
individual files.
This patch disables that process in case the number of files to run
is less than one.
Results running a sample file:
Before:
```
actionpack $ bin/test test/controller/parameters/accessors_test.rb
Run options: --seed 48261
........................................................................
Finished in 0.211923s, 339.7460 runs/s, 552.0873 assertions/s.
72 runs, 117 assertions, 0 failures, 0 errors, 0 skips
```
After
```
actionpack $ bin/test test/controller/parameters/accessors_test.rb
Run options: --seed 5461
........................................................................
Finished in 0.008411s, 8560.2189 runs/s, 13910.3557 assertions/s.
72 runs, 117 assertions, 0 failures, 0 errors, 0 skips
```
This hack prevails everywhere in the codebase by being copy & pasted, and it's actually not a negative thing but a necessary thing for framework implementors,
so it should better have a name and be a thing.
And with this commit, activesupport/test/abstract_unit.rb now doesn't silently autoload AS::TestCase,
so we're ready to establish clearner environment for running AS tests (probably in later commits)
* Add option to set parallel test worker count to the physical core count of the machine
Also, use the physical core count of the machine as
the default number of workers, and generate the `test_helper.rb` file
with `parallelize(workers: :number_of_processors)`
Closes#34734
* Ensure that we always test parallel testing
Since #34734 we decided to use the physical core count of the machine as
the default number of workers in the parallel testing, we need to
ensure that some tests use at least 2 workers because we could
run those tests on VM that has only 1 physical core.
It also fixes tests failures on the CI since Travis server we are using
has only one physical core.
See https://travis-ci.org/rails/rails/jobs/469281088#L2352
The name of the minitest library is spelled that way: regular font, and
lowercase. Lowercase is used even at the beginning of sentences, see
http://docs.seattlerb.org/minitest/
I double-checked this with @zenspider too (thanks!).
- In #32472 I introduced a fix in order for all `after_teardown` method provided by libraries and Rails to run, even if the application's `teardown` method raised an error (That's the default minitest behavior). However this change wasn't enough and doesn't take in consideration the ancestors chain.
If a library's module containing an `after_teardown` method get included after the `SetupAndTeardown` module (one example is the [ActiveRecord::TestFixtures module](7d2400ab61/activerecord/lib/active_record/fixtures.rb (L855-L856)), then the ancestors of the test class would look something like
```ruby
class MyTest < ActiveSupport::TestCase
end
puts MyTest.ancestors # [MyTest, ActiveSupport::TestCase, ActiveRecord::TestFixtures, ActiveSupport::Testing::SetupAndTeardown]
```
Any class/module in the ancestors chain that are **before** the `ActiveSupport::Testing::SetupAndTeardown` will behave incorrectly:
- Their `before_setup` method will get called **after** all regular setup method
- Their `after_teardown` method won't even get called in case an exception is raised inside a regular's test `teardown`
A simple reproduction script of the problem here https://gist.github.com/Edouard-chin/70705542a59a8593f619b02e1c0a188c
- One solution to this problem is to have the `AS::SetupAndTeardown` module be the very first in the ancestors chain. By doing that we ensure that no `before_setup` / `after_teardown` get executed prior to running the teardown callbacks
Provides both a forked process and threaded parallelization options. To
use add `parallelize` to your test suite.
Takes a `workers` argument that controls how many times the process
is forked. For each process a new database will be created suffixed
with the worker number; test-database-0 and test-database-1
respectively.
If `ENV["PARALLEL_WORKERS"]` is set the workers argument will be ignored
and the environment variable will be used instead. This is useful for CI
environments, or other environments where you may need more workers than
you do for local testing.
If the number of workers is set to `1` or fewer, the tests will not be
parallelized.
The default parallelization method is to fork processes. If you'd like to
use threads instead you can pass `with: :threads` to the `parallelize`
method. Note the threaded parallelization does not create multiple
database and will not work with system tests at this time.
parallelize(workers: 2, with: :threads)
The threaded parallelization uses Minitest's parallel exector directly.
The processes paralleliztion uses a Ruby Drb server.
For parallelization via threads a setup hook and cleanup hook are
provided.
```
class ActiveSupport::TestCase
parallelize_setup do |worker|
# setup databases
end
parallelize_teardown do |worker|
# cleanup database
end
parallelize(workers: 2)
end
```
[Eileen M. Uchitelle, Aaron Patterson]
ActiveSupport::Testing::Assertions.
We have a separate module in which have defined Rails' own custom
assertions. So it would be good to keep all custom Rails' assertions in
one place i.e. in this module.
Before:
```
DEPRECATION WARNING: Passing arguments to assert_nothing_raised
is deprecated and will be removed in Rails 5.1.
```
After:
```
DEPRECATION WARNING: Passing arguments to assert_nothing_raised is deprecated and will be removed in Rails 5.1.
```
The line filter parsing added to ActiveSupport::TestCase is only half the story
to enable line filtering. The other half, of adding the patterns to the options,
is done in the Minitest plugin that Railties has.
Thus it makes more sense to have the filter in Railties with the other half and
all the line filtering tests.
Move the filter and extend Active Support in an initializer, so that when users
or `rails/all.rb` require `rails/test_unit/railtie` we can still filter by line.
This also adds free mix and matching of directories, files and lines filters.
Like so:
bin/rails test models/post_test.rb test/integration models/person_test.rb:26
You can also mix in a traditional Minitest filter:
bin/rails test test/integration -n /check_it_out/
It's a thin layer to provide easy access to sample files throughout
test-cases. This adds the directory `test/fixtures/files` to newly
generated applications.
