* Add option to set parallel test worker count to the physical core count of the machine
Also, use the physical core count of the machine as
the default number of workers, and generate the `test_helper.rb` file
with `parallelize(workers: :number_of_processors)`
Closes#34734
* Ensure that we always test parallel testing
Since #34734 we decided to use the physical core count of the machine as
the default number of workers in the parallel testing, we need to
ensure that some tests use at least 2 workers because we could
run those tests on VM that has only 1 physical core.
It also fixes tests failures on the CI since Travis server we are using
has only one physical core.
See https://travis-ci.org/rails/rails/jobs/469281088#L2352
The ActionDispatch::HostAuthorization is a new middleware that prevent
against DNS rebinding and other Host header attacks. By default it is
included only in the development environment with the following
configuration:
Rails.application.config.hosts = [
IPAddr.new("0.0.0.0/0"), # All IPv4 addresses.
IPAddr.new("::/0"), # All IPv6 addresses.
"localhost" # The localhost reserved domain.
]
In other environments, `Rails.application.config.hosts` is empty and no
Host header checks will be done. If you want to guard against header
attacks on production, you have to manually permit the allowed hosts
with:
Rails.application.config.hosts << "product.com"
The host of a request is checked against the hosts entries with the case
operator (#===), which lets hosts support entries of type RegExp,
Proc and IPAddr to name a few. Here is an example with a regexp.
# Allow requests from subdomains like `www.product.com` and
# `beta1.product.com`.
Rails.application.config.hosts << /.*\.product\.com/
A special case is supported that allows you to permit all sub-domains:
# Allow requests from subdomains like `www.product.com` and
# `beta1.product.com`.
Rails.application.config.hosts << ".product.com"
Sometimes `test_scaffold_tests_pass_by_default` test fails in CI.
https://travis-ci.org/rails/rails/jobs/457621750#L2095-L2120
It seems `manifest.json` was broken.
`webpacker` will compile automatically if packs is not compiled.
If parallel test is enabled, it seems that this compilation process is
executed simultaneously in multiple processes, and it may become an
inconsistent state.
In order to avoid this, compile before running the test.
DRb wraps in `DRbUnknown` if the data contains a type that can not be
resolved locally. This can happen if an error occurs in the test and the
error class can not be resolved on the server side.
When this happens, an instance of `DRbUnknown` is passed to the `result`
of `Server#record`. This causes another error(undefined method assertions
for #<DRb::DRbUnknown:> (NoMethodError)) in `reporter.record`.
This can confirm by the following steps.
```
$ rails new app -B --dev; cd app
$ rails g scaffold user name:string
$ edit `config.action_controller.allow_forgery_protection = true` in environments/test.rb
$ bin/rails t
```
In the case of `DRbUnknown` occurs, can't resolve error exception. So wrap
exception with `DRbRemoteError` in the same way as an unmarshalled object.
Addressing warnings are important but it should be out of this test scope.
https://travis-ci.org/rails/rails/jobs/454145524#L4122-L4131
```
.F
Failure:
ApplicationTests::BinSetupTest#test_bin_setup_output [test/application/bin_setup_test.rb:49]:
--- expected
+++ actual
@@ -1,4 +1,5 @@
"== Installing dependencies ==
+warning: Passing permitted_classes with the 2nd argument of Psych.safe_load is deprecated. Use keyword argument like Psych.safe_load(yaml, permitted_classes: ...) instead.
The Gemfile's dependencies are satisfied
== Preparing database ==
rails test test/application/bin_setup_test.rb:38
```
AR instance support `filter_parameters` since #33756.
Though Regex or Proc is valid as `filter_parameters`,
they are not supported as AR#inspect.
I also add :mask option and #filter_params to
`ActiveSupport::ParameterFilter#new` to implement this.
* Use Webpacker by default on new apps
* Stop including coffee-rails by default
* Drop using a js_compressor by default
* Drop extra test for coffeescript inclusion by default
* Stick with skip_javascript to signify skipping webpack
* Don't install a JS runtime by default any more
* app/javascript will be the new default directory for JS
* Make it clear that this is just for configuring the default Webpack framework setup now
* Start using the Webpack tag in the default layout
* Irrelevant test
* jQuery is long gone
* Stop having asset pipeline compile default application.js
* Add rails-ujs by default to the Webpack setup
* Add Active Storage JavaScript to application.js pack by default
* Consistent quoting
* Add Turbolinks to default pack
* Add Action Cable to default pack
Need some work on how to set the global consumer that channels will
work with. @javan?
* Require all channels by default and use a separate consumer stub
* Channel generator now targets Webpack style
* Update task docs to match new generator style
* Use uniform import style
* Drop the JS assets generator
It was barely helpful as it was. It’s no longer helpful in a Webpacked
world. Sayonara!
* Add app/javascript to the stats directories
* Simpler import style
Which match the other imports.
* Address test failures from dropping JS compilation (and compression)
* webpacker-default: Modify `AssetsGeneratorTest`
Before:
```
$ bin/test test/generators/assets_generator_test.rb
Run options: --seed 46201
F
Failure:
AssetsGeneratorTest#test_assets [/Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/assets_generator_test.rb:12]:
Expected file "app/assets/javascripts/posts.js" to exist, but does not
bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/assets_generator_test.rb:10
.
Finished in 0.031343s, 63.8101 runs/s, 95.7152 assertions/s.
2 runs, 3 assertions, 1 failures, 0 errors, 0 skips
```
After:
```
$ bin/test test/generators/assets_generator_test.rb
Run options: --seed 43571
..
Finished in 0.030370s, 65.8545 runs/s, 65.8545 assertions/s.
2 runs, 2 assertions, 0 failures, 0 errors, 0 skips
```
* webpacker-default: Modify `ChannelGeneratorTest`
Before:
```
$ bin/test test/generators/channel_generator_test.rb
Run options: --seed 8986
.F
Failure:
ChannelGeneratorTest#test_channel_with_multiple_actions_is_created [/Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:43]:
Expected file "app/assets/javascripts/channels/chat.js" to exist, but does not
bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:34
.F
Failure:
ChannelGeneratorTest#test_channel_is_created [/Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:29]:
Expected file "app/assets/javascripts/channels/chat.js" to exist, but does not
bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:22
E
Error:
ChannelGeneratorTest#test_cable_js_is_created_if_not_present_already:
Errno::ENOENT: No such file or directory @ apply2files - /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/fixtures/tmp/app/assets/javascripts/cable.js
bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:60
F
Failure:
ChannelGeneratorTest#test_channel_suffix_is_not_duplicated [/Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:87]:
Expected file "app/assets/javascripts/channels/chat.js" to exist, but does not
bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:80
F
Failure:
ChannelGeneratorTest#test_channel_on_revoke [/Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:77]:
Expected file "app/assets/javascripts/cable.js" to exist, but does not
bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:68
Finished in 0.064384s, 108.7227 runs/s, 481.4861 assertions/s.
7 runs, 31 assertions, 4 failures, 1 errors, 0 skips
```
After:
```
$ bin/test test/generators/channel_generator_test.rb
Run options: --seed 44857
.......
Finished in 0.060243s, 116.1961 runs/s, 697.1764 assertions/s.
7 runs, 42 assertions, 0 failures, 0 errors, 0 skips
```
* Fix shared generator tests.
* webpacker-default: Modify `ControllerGeneratorTest`
The JS assets generator was dropped. ref. 46215b1794
* Revert "Simpler import style". It's currently failing with an error of "TypeError: undefined is not an object (evaluating '__WEBPACK_IMPORTED_MODULE_2_activestorage___default.a.start')". Waiting for @javan to have a look.
This reverts commit 5d3ebb71059f635d3756cbda4ab9752027e09256.
* require webpacker in test app
* Add webpacker without making the build hang/timeout. (#33640)
* use yarn workspaces to allow for installing unreleased packages and only generate js/bootsnap when required
* no longer need to have webpacker in env templates as webpacker moved this config to yml file
* Fix rubocop violation
* Got the test passing for the running scaffold
* update expected lines of code
* update middleware tests to account for webpacker
* disable js in plugins be default to get the tests passing (#34009)
* clear codeclimate report issues
* Anything newer than currently released is good
* Use Webpacker development version during development of Rails
* Edge should get development webpacker as well
* Add changelog entry for Webpacker change
The attribute methods for a model are currently defined lazily the first
time that model is instantiated, even when `config.eager_load` is true.
This means the first request to use each model incurs the cost, which
usually involves a database round trip to fetch the schema definition.
By defining the attribute methods for all models while the application
is booting, we move that work out of any individual request. When using
a forking web server, this also reduces the number of times the schema
definition is queried by doing it once in the parent process instead of
from each forked process during their first request.
Before this change this test was passing even if we revert #31135. The
reason for that is that `app 'development'` will load the environment in
the test process and it is happening before db_create_and_drop is
called.
This was not asserting that the environment was loaded in the db:create
task itself.
To test it we enhance the db:create task with a block that writes to a
tmp file the value of the config. If the environment is loaded before
that task enhancement runs the content of the file will have "true"
insteand of "false".
If you are using the "in cache versioning" also known as "recyclable cache keys" the cache store must be aware of this scheme, otherwise you will generate cache entries that never invalidate.
This PR adds a check to the initialization process to ensure that if recyclable cache keys are being used via
```
config.active_record.cache_versioning = true
```
Then the cache store needs to show that it supports this versioning scheme. Cache stores can let Rails know that they support this scheme by adding a method `supports_in_cache_versioning?` and returning true.
In production the query cache was already being loaded before the first
request even without #33856, so added a test to make sure of it.
This new test is passing even if #33856 is reverted.
It would allow `filter_attributes` to be reused across multiple
calls to `#inspect` or `#pretty_print`.
- Add `require "set"`
- Remove `filter_attributes` instance reader. I think there is no need
to keep it.
In a test app we observed that the query cache was not enabled on the
first request. This was because the query cache hooks are installed on
load and active record is loaded in the middle of the first request.
If we remove the `on_load` from the railtie the query cache hooks will
be installed before the first request, allowing the cache to be enabled
on that first request.
This is ok because query cache doesn't load anything else, only itself
so we're not eager loading all of active record before the first
request, just the query cache hooks.
[Eileen M. Uchitelle & Matthew Draper]
* Why
Some sensitive data will be exposed in log accidentally by calling `#inspect`, e.g.
```ruby
@account = Account.find params[:id]
payload = { account: @account }
logger.info "payload will be #{ payload }"
```
All the information of `@account` will be exposed in log.
* Solution
Add a class attribute filter_attributes to specify which values of columns shouldn't be exposed.
This attribute equals to `Rails.application.config.filter_parameters` by default.
```ruby
Rails.application.config.filter_parameters += [:credit_card_number]
Account.last.insepct # => #<Account id: 123, credit_card_number: [FILTERED] ...>
```
The rake tasks which became deprecate now does not load the environment.
Therefore, even if the application specifies the behavior of deprecating,
the message is output to stderr ignoring the specification.
It seems that this is not the expected behavior.
We should respect the setting even in the rake tasks.
Changes the `configs_for` method from using traditional arguments to
using kwargs. This is so I can add the `include_replicas` kwarg without
having to always include `env_name` and `spec_name` in the method call.
`include_replicas` defaults to false because everywhere internally in
Rails we don't want replicas. `configs_for` is for iterating over
configurations to create / run rake tasks, so we really don't ever need
replicas in that case.
While the three-tier config makes it easier to define databases for
multiple database applications, it quickly became clear to offer full
support for multiple databases we need to change the way the connections
hash was handled.
A three-tier config means that when Rails needed to choose a default
configuration (in the case a user doesn't ask for a specific
configuration) it wasn't clear to Rails which the default was. I
[bandaid fixed this so the rake tasks could work](#32271) but that fix
wasn't correct because it actually doubled up the configuration hashes.
Instead of attemping to manipulate the hashes @tenderlove and I decided
that it made more sense if we converted the hashes to objects so we can
easily ask those object questions. In a three tier config like this:
```
development:
primary:
database: "my_primary_db"
animals:
database; "my_animals_db"
```
We end up with an object like this:
```
@configurations=[
#<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10
@env_name="development",@spec_name="primary",
@config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>,
#<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90
@env_name="development",@spec_name="animals",
@config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>
]>
```
The configurations setter takes the database configuration set by your
application and turns them into an
`ActiveRecord::DatabaseConfigurations` object that has one getter -
`@configurations` which is an array of all the database objects.
The configurations getter returns this object by default since it acts
like a hash in most of the cases we need. For example if you need to
access the default `development` database we can simply request it as we
did before:
```
ActiveRecord::Base.configurations["development"]
```
This will return primary development database configuration hash:
```
{ "database" => "my_primary_db" }
```
Internally all of Active Record has been converted to use the new
objects. I've built this to be backwards compatible but allow for
accessing the hash if needed for a deprecation period. To get the
original hash instead of the object you can either add `to_h` on the
configurations call or pass `legacy: true` to `configurations.
```
ActiveRecord::Base.configurations.to_h
=> { "development => { "database" => "my_primary_db" } }
ActiveRecord::Base.configurations(legacy: true)
=> { "development => { "database" => "my_primary_db" } }
```
The new configurations object allows us to iterate over the Active
Record configurations without losing the known environment or
specification name for that configuration. You can also select all the
configs for an env or env and spec. With this we can always ask
any object what environment it belongs to:
```
db_configs = ActiveRecord::Base.configurations.configurations_for("development")
=> #<ActiveRecord::DatabaseConfigurations:0x00007fd1acbdf800
@configurations=[
#<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10
@env_name="development",@spec_name="primary",
@config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>,
#<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90
@env_name="development",@spec_name="animals",
@config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>
]>
db_config.env_name
=> "development"
db_config.spec_name
=> "primary"
db_config.config
=> { "adapter"=>"sqlite3", "database"=>"db/development.sqlite3" }
```
The configurations object is more flexible than the configurations hash
and will allow us to build on top of the connection management in order
to add support for primary/replica connections, sharding, and
constructing queries for associations that live in multiple databases.
We originally did the whole `load_database_yaml` thing because this test
wasn't cooperating and we needed to finish the namespaced rake tasks for
multiple databases.
However, it turns out that YAML can't eval ERB if you don't tell it it's
ERB so you get Pysch parse errors if you're using multi-line ERB or
ERB with conditionals. It's a hot mess.
After trying a few things and thinking it over we decided that it wasn't
worth bandaiding over, the test needed to be improved. The test was
added in #31135 to test that the env is loaded in these tasks. But it
was blowing up because we were trying to read a database name out of the
configuration - however that's not the purpose of this change. We want
to read environment files in the rake tasks, but not in the config
file.
In this PR we changed the test to test what the PR was actually fixing.
We've also deleted the `load_database_yaml` because it caused more
problems than it was worth. This should fix the issues described in
https://github.com/rails/rails/pull/32274#issuecomment-384161057. We
also had these problems at GitHub.
Co-authored-by: alimi <aibrahim2k2@gmail.com>
Issue #27852 reports that when `rails db:create` fails, it causes
leaking of user's DB credentials to $stderr.
We print a DB's configuration hash in order to help users more quickly
to figure out what could be wrong with his configuration.
This commit changes message from
"Couldn't create database for #{configuration.inspect}" to
"Couldn't create '#{configuration['database']}' database. Please check your configuration.".
There are two PRs that fixing it #27878, #27879, but they need a bit more work.
I decided help to finish this and added Author of those PRs credit in this commit.
Since it is a security issue, I think we should backport it to
`5-2-stable`, and `5-1-stable`.
Guided by https://edgeguides.rubyonrails.org/maintenance_policy.html#security-issuesFixes#27852Closes#27879
Related to #27878
[Alexander Marrs & bogdanvlviv]
Purpose metadata prevents cookie values from being
copy-pasted and ensures that the cookie is used only
for its originally intended purpose.
The Purpose and Expiry metadata are embedded inside signed/encrypted
cookies and will not be readable on previous versions of Rails.
We can switch off purpose and expiry metadata embedded in
signed and encrypted cookies using
config.action_dispatch.use_cookies_with_metadata = false
if you want your cookies to be readable on older versions of Rails.
