Commit Graph

84137 Commits

Author SHA1 Message Date
Jean Boussier
4491822643
Merge pull request #45536 from byroot/foreign-key-flaky-test
Fix flaky foreign key test
2022-07-06 16:16:08 +02:00
Jean Boussier
939d4ee7ca
Merge pull request #45531 from djmb/reduce-encrypted-properties-allocations
Fewer object allocations in Encryption::Properties
2022-07-06 16:15:42 +02:00
Jean Boussier
c112287974 Fix flaky foreign key test
Fix: https://github.com/rails/rails/pull/45491
2022-07-06 16:04:43 +02:00
Donal McBreen
0279c033e2 Fewer object allocations in Encryption::Properties
`delegate_missing_to` and `Enumerable#find` both allocate objects. When
selecting a large number of encrypted values with can lead to a
significant number of allocations.

```

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"

  git_source(:github) { |repo| "https://github.com/#{repo}.git" }

  # Activate the gem you are reporting the issue against.
  gem "activerecord", "~> 7.0.0"
  gem "sqlite3"
  gem "benchmark-ips"
end

require "active_record"
require 'benchmark/ips'

ActiveRecord::Base.establish_connection(adapter: "sqlite3", database: ":memory:")

ActiveRecord::Encryption.configure \
  primary_key: "test master key",
  deterministic_key: "test deterministic key",
  key_derivation_salt: "testing key derivation salt",
  support_unencrypted_data: true

ActiveRecord::Schema.define do
  create_table :comments, force: true do |t|
    t.string :message, length: 1000
  end
end

class Comment < ActiveRecord::Base
  encrypts :message
end

srand(123456)
1000.times { Comment.create!(message: 100.times.map { ("A".."Z").to_a.sample }.join) }

if ENV['OPTIMIZED']
  module ActiveRecord
    module EncryptionPropertiesAvoidAllocations
      extend ActiveSupport::Concern

      ALLOWED_VALUE_CLASSES = [String, ActiveRecord::Encryption::Message, Numeric, TrueClass, FalseClass, Symbol, NilClass, Float, Integer]

      if %w{delegation all}.include?(ENV['OPTIMIZED'])
        delegate :each, :[], :key?, to: :data
      end

      if %w{find all}.include?(ENV['OPTIMIZED'])
        # find also involves allocations, so we can cache the results which classes
        # are valid to avoid the call
        def validate_value_type(value)
          unless ALLOWED_VALUE_CLASSES.include?(value.class) || ALLOWED_VALUE_CLASSES.any? { |klass| value.is_a?(klass) }
            raise ActiveRecord::Encryption::Errors::ForbiddenClass, "Can't store a #{value.class}, only properties of type #{ALLOWED_VALUE_CLASSES.inspect} are allowed"
          end
        end
      end
    end
  end

  ActiveRecord::Encryption::Properties.prepend(ActiveRecord::EncryptionPropertiesAvoidAllocations);
end

def allocation_count
  before = GC.stat(:total_allocated_objects)
  yield
  GC.stat(:total_allocated_objects) - before
end

allocated_objects = allocation_count { Comment.pluck(:message) }
puts "Optimization: #{ENV['OPTIMIZED'] || "none"}, allocations: #{allocated_objects}"

Benchmark.ips do |x|
  x.config(time: 30)
  x.report("default")                 { Comment.pluck(:message) }
  x.report("delegation_optimization") { Comment.pluck(:message) }
  x.report("find_optimization")       { Comment.pluck(:message) }
  x.report("all_optimization")        { Comment.pluck(:message) }
  x.hold! "temp_results"
  x.compare!
end
```

Results:
```
$ ruby encryption_properties_benchmark.rb; OPTIMIZED=delegation ruby encryption_properties_benchmark.rb; OPTIMIZED=find ruby encryption_properties_benchmark.rb; OPTIMIZED=all ruby encryption_properties_benchmark.rb
...snip...
Optimization: none, allocations: 72238
Warming up --------------------------------------
             default     6.000  i/100ms
Calculating -------------------------------------
             default     70.643  (± 4.2%) i/s -      2.118k in  30.052046s

...snip...
Optimization: delegation, allocations: 62313
Warming up --------------------------------------
delegation_optimization
                         7.000  i/100ms
Calculating -------------------------------------
delegation_optimization
                         75.785  (± 4.0%) i/s -      2.275k in  30.086061s

Pausing here -- run Ruby again to measure the next benchmark...

Comparison:
delegation_optimization:       75.8 i/s
             default:       70.6 i/s - same-ish: difference falls within error

...snip...
Optimization: find, allocations: 60306
Warming up --------------------------------------
   find_optimization     7.000  i/100ms
Calculating -------------------------------------
   find_optimization     74.179  (± 4.0%) i/s -      2.226k in  30.082991s

Pausing here -- run Ruby again to measure the next benchmark...

Comparison:
delegation_optimization:       75.8 i/s
   find_optimization:       74.2 i/s - same-ish: difference falls within error
             default:       70.6 i/s - same-ish: difference falls within error

...snip...
Optimization: all, allocations: 50315
Warming up --------------------------------------
    all_optimization     8.000  i/100ms
Calculating -------------------------------------
    all_optimization     84.689  (± 5.9%) i/s -      2.528k in  30.008722s

Comparison:
    all_optimization:       84.7 i/s
delegation_optimization:       75.8 i/s - 1.12x  (± 0.00) slower
   find_optimization:       74.2 i/s - 1.14x  (± 0.00) slower
             default:       70.6 i/s - 1.20x  (± 0.00) slower

```

Overall it's about 15% faster with a 30% reduction in allocations.
2022-07-06 14:56:18 +01:00
Jean Boussier
2dfefb5104
Merge pull request #45534 from etiennebarrie/fix-actionpack-test-with-rack-test-2
Fix actionpack test with rack test 2
2022-07-06 15:25:39 +02:00
Étienne Barrié
889886d6ad Allow rack-test >= 2 2022-07-06 14:54:39 +02:00
Aaron Patterson
07be723bc1 Ensure Rails is green with Rack-Test main branch
This commit just ensures we're green with the main branch of rack test.
The changes are things we should have done anyway, and are backwards
compatible with older versions of rack test
2022-07-06 14:54:01 +02:00
Jean Boussier
fd1196bd51
Merge pull request #45414 from fatkodima/whole-table-batching
Optimize Active Record batching for whole table iterations
2022-07-06 10:10:10 +02:00
Gannon McGibbon
02444d9267 Add --parent option to job generator to specify parent class of job.
Example:

`bin/rails g job process_payment --parent=payment_job` generates:

```ruby
class ProcessPaymentJob < PaymentJob
  # ...
end
```
2022-07-05 20:55:38 -04:00
Ryo Nakamura
6f9bb2cc0f Fix NoMethodError on custom ActiveSupport::Deprecation behavior
with some additional changes made later:

1. Flip the condition for backward compatibility
   Co-authored-by: Jonathan Hefner <jonathan@hefner.pro>

2. Improve custom behavior test
   Co-authored-by: Jonathan Hefner <jonathan@hefner.pro>

3. Fix indentation
2022-07-06 06:41:41 +09:00
fatkodima
620f247829 Optimize Active Record batching for whole table iterations 2022-07-06 00:21:39 +03:00
Andrew Porterfield
d0f07d7abd Update index in migration at bottom of Active Record Basic guide 2022-07-05 17:06:14 -04:00
Jonathan Hefner
4c67975c9c Improve generator implied option handling
This commit improves handling of implied options for `AppGenerator` and
`PluginGenerator` in a few different ways.

Implied options are now reported:

```console
$ rails new my_cool_app --skip-active-job
Based on the specified options, the following options will also be activated:

  --skip-action-mailer [due to --skip-active-job]
  --skip-active-storage [due to --skip-active-job]
  --skip-action-mailbox [due to --skip-active-storage]
  --skip-action-text [due to --skip-active-storage]

...
```

Option conflicts raise an error:

```console
$ rails new my_cool_app --skip-active-job --no-skip-active-storage
Based on the specified options, the following options will also be activated:

  --skip-action-mailer [due to --skip-active-job]
  --skip-active-storage [due to --skip-active-job]
    ERROR: Conflicts with --no-skip-active-storage
  --skip-action-mailbox [due to --skip-active-storage]
  --skip-action-text [due to --skip-active-storage]

railties/lib/rails/generators/app_base.rb:206:in `report_implied_options': Cannot proceed due to conflicting options (RuntimeError)
```

Meta option (e.g. `--minimal`) implications are also reported:

```console
$ rails new my_cool_app --minimal
Based on the specified options, the following options will also be activated:

  --skip-active-job [due to --minimal]
  --skip-action-mailer [due to --skip-active-job, --minimal]
  --skip-active-storage [due to --skip-active-job, --minimal]
  --skip-action-mailbox [due to --skip-active-storage, --minimal]
  --skip-action-text [due to --skip-active-storage, --minimal]
  --skip-javascript [due to --minimal]
  --skip-hotwire [due to --skip-javascript, --minimal]
  --skip-action-cable [due to --minimal]
  --skip-bootsnap [due to --minimal]
  --skip-dev-gems [due to --minimal]
  --skip-system-test [due to --minimal]

...
```

`--no-*` options now work with meta options, and are both comprehensive
and precise:

```console
$ rails new my_cool_app --minimal --no-skip-active-storage
Based on the specified options, the following options will also be activated:

  --skip-action-mailer [due to --minimal]
  --skip-action-mailbox [due to --minimal]
  --skip-action-text [due to --minimal]
  --skip-javascript [due to --minimal]
  --skip-hotwire [due to --skip-javascript, --minimal]
  --skip-action-cable [due to --minimal]
  --skip-bootsnap [due to --minimal]
  --skip-dev-gems [due to --minimal]
  --skip-system-test [due to --minimal]

...
```

Closes #45223.
Closes #45205.

Co-authored-by: Brad Trick <btrick@appacademy.io>
2022-07-05 15:16:28 -05:00
Eileen M. Uchitelle
19f9922523
Merge pull request #45461 from fatkodima/sqlite3-index-trailing-comment
Fix retrieving sqlite3 expression indexes created by sql annotated with trailing comment
2022-07-05 14:59:45 -04:00
Eileen M. Uchitelle
45eb2958a7
Merge pull request #45526 from eileencodes/remove-db-config-owner-name
Remove owner_name from db_config
2022-07-05 14:05:11 -04:00
eileencodes
4a95e9aa5f
Remove owner_name from db_config
The majority of where this functionality was removed in #38672.
`owner_name` was never used publicaly in the db_config and was at a time
used for identifying connection owners. Now the connection owner is
always stored in the PoolConfig (same place as the db_config) so the
db_config doesn't need this information.
2022-07-05 13:33:32 -04:00
Jonathan Hefner
c0f71dac70
Merge pull request #45501 from ghiculescu/same-site-false
Allow opting out of the `SameSite` cookie attribute when setting a cookie
2022-07-05 11:06:45 -05:00
Eileen M. Uchitelle
b70947bcf6
Merge pull request #45525 from sushantmittal/fix-flaky-test-cases
Fix flaky test cases in file 'activerecord/test/cases/relation/merging_test.rb'.
2022-07-05 09:13:58 -04:00
Sushant Mittal
c265974538 Fixed flaky test cases in 'test/cases/relation/merging_test.rb'. 2022-07-05 16:54:50 +05:30
Jean Boussier
b3175b953c
Merge pull request #37944 from vlado/with_cte
Common Table Expression support added "out-of-the-box"
2022-07-05 08:39:29 +02:00
Rafael Mendonça França
dadf171db8
Merge pull request #45508 from malis/patch-3
Reading form's method by getAttribute instead of form.method
2022-07-04 15:33:28 -04:00
Rafael Mendonça França
0f7b27cded
Merge pull request #45518 from fatkodima/pg-schemas-improvements
Improve working with PostgreSQL schemas
2022-07-04 14:10:26 -04:00
Rafael Mendonça França
77f97d8eef
Merge pull request #45515 from fatkodima/remove_check_constraint-symbol
Fix remove_check_constraint by symbol name
2022-07-04 14:00:19 -04:00
Rafael Mendonça França
e445f48cbb
Merge pull request #45513 from skipkayhil/rm-legacy-connection-guide
Remove docs for legacy_connection_handling [ci skip]
2022-07-04 13:57:43 -04:00
Rafael Mendonça França
9ef41cd548
Merge pull request #45512 from skipkayhil/rm-aj-false-on-aborted-enqueue
Remove return_false_on_aborted_enqueue again
2022-07-04 13:56:06 -04:00
fatkodima
a4d1aceeba Improve working with PostgreSQL schemas 2022-07-04 20:35:23 +03:00
fatkodima
8103d611f4 Fix remove_check_constraint by symbol name 2022-07-04 15:38:52 +03:00
Hartley McGuire
0348b219cf
Remove docs for legacy_connection_handling
The config was fully removed in ad52c0a but this was missed
2022-07-03 20:12:03 -04:00
Hartley McGuire
a0030bf44f
Remove return_false_on_aborted_enqueue again
This was supposed to be removed in 7c788e91 but only the deprecation was
actually removed.
2022-07-03 16:23:04 -04:00
Alex Ghiculescu
d29e755aea Allow opting out of the SameSite cookie attribute when setting a cookie.
Since 7ccaa125ba it's not been possible to not include `SameSite` on your cookies. `SameSite` is recommended, but it's not a required field, and you should be able to opt out of it.

This PR introduces that ability: you can opt out of `SameSite` by passing `same_site: false`.

```ruby
cookies[:foo] = { value: "bar", same_site: false }
```

Previously, this incorrectly set the `SameSite` attribute to the value of the `cookies_same_site_protection` setting. https://github.com/rails/rails/pull/44934 added docs saying that you could pass `nil` as a value, but that also would fall back to the default (`:lax`).
2022-07-03 10:40:51 -05:00
Dominik Malis
646bff6091
Reading form's method by getAttribute
Accessing form's method by getAttribute('method') instead of form.method as this property may return named form element if there is any named "method".
2022-07-02 22:14:32 +02:00
Eileen M. Uchitelle
31b1403919
Merge pull request #45504 from fatkodima/mysql-optimize-data-sources-query
Optimize data sources sql for MySQL
2022-07-01 15:00:57 -04:00
Vlado Cingel
098b0eb5db .with query method added.
Construct common table expressions with ease and get `ActiveRecord::Relation` back.
2022-07-01 17:44:51 +02:00
fatkodima
20a58fdd91 Optimize data sources sql for MySQL 2022-07-01 13:29:57 +03:00
Eileen M. Uchitelle
c704da66de
Merge pull request #45497 from eileencodes/refactor-pool-manager-in-est-conn
Refactor connection handler's establish_connection
2022-06-30 12:09:26 -04:00
Eileen M. Uchitelle
5cf939ff18
Merge pull request #45439 from etiennebarrie/pass-test-options
Pass options to the test runner for test:{all,system,generators}
2022-06-30 11:01:35 -04:00
eileencodes
0396fe918d
Refactor connection handler's establish_connection
In the prior code we were getting the pool manager a bunch of times -
once in the retrieve_connection_pool, once in remove_connection_pool,
and once in the else conditional after setting it.

This change ensures we're only getting the pool manager once. If there
is no pool manager for the given key then we create a new one in the new
`set_pool_manager` method and use that.

The refactor also has a change in that we no longer instrument if a new
connection was not established. This instrumentation is private though
(denoted by the !) so it's safe to make this change.

Followup to #45450
2022-06-30 10:39:13 -04:00
Rafael Mendonça França
92fa470aaa
Copy edit the CHANGELOG
Active Record names has space between the words.
2022-06-29 22:14:48 +00:00
Rafael Mendonça França
4755259a65
Merge pull request #45385 from skipkayhil/refine-configuring-docs-2
Add docs for all Application::Configuration attr [ci skip]
2022-06-29 18:07:27 -04:00
Rafael Mendonça França
2d2fd947d4
Merge pull request #45484 from codergeek121/fix-npm-release-tags
Fix the latest tag for npm packages
2022-06-29 18:02:41 -04:00
Jonathan Hefner
50402fc2ce
Merge pull request #45473 from jonathanhefner/message_encryptor-urlsafe-option
Support `:urlsafe` option for `MessageEncryptor`
2022-06-29 16:21:12 -05:00
Jonathan Hefner
932dd97018
Merge pull request #45482 from jonathanhefner/message_verifier-replace-urlsafe-tests
Replace `MessageVerifier` `:urlsafe` option tests
2022-06-29 16:20:44 -05:00
Jonathan Hefner
d9823326e8 Support :urlsafe option for MessageEncryptor
This adds a `:urlsafe` option to the `MessageEncryptor` constructor.
When enabled, this option ensures that messages use a URL-safe encoding.
This matches the `MessageVerifier` `:urlsafe` option added in #45419.
2022-06-29 16:00:16 -05:00
Rafael Mendonça França
8648ac7915
Merge pull request #45490 from ghiculescu/action-cable-anchorr
Anchor the Action Cable server's route
2022-06-29 14:44:14 -04:00
Alex Ghiculescu
1ee984dfe4 Anchor the Action Cable server's route
Fixes https://github.com/rails/rails/issues/45489

- Adds `anchor: true` to the Action Cable server mount, so that it only strictly matches `/cable` rather than anything that starts with that.
- Uses `reverse_merge` instead of `merge` in `Mapper#mount`, so that you can override these options if you need to.
2022-06-29 13:26:49 -05:00
Eileen M. Uchitelle
462545c0a5
Merge pull request #45450 from eileencodes/only-remove-connection-if-config-is-different
Only remove connection for an existing pool if the config is different
2022-06-29 13:37:46 -04:00
eileencodes
adb64db43d
Only remove connection for an existing pool if the config is different
Previously Rails would always remove the connection if it found a
matching class in the pool manager. Therefore if
`ActiveRecord::Base.establish_connection` was called with the same
config, each time it was called it would be clobbered, even though the
config hasn't changed and the existing connection is prefectly fine. As
far as I can tell from conversations and reading the history this
functionality was added for ActiveRecord tests to be able to clobber the
connection and use a new config, then re-establish the old connection.
Essentially outside Rake tasks and AR tests, this functionality doesn't
have a ton of value.

On top of not adding a ton of value, this has resulted in a few bugs. In
Rails 6.0 I made it so that if you established a connection on
`ApplicationRecord` Rails would treat that connection the same as
`ActiveRecord::Base.` The reason for this is that the Railtie
establishes a connection on boot to the first database, but then if
you're using multiple databases you're calling `connects_to` in your
`ApplicationRecord` or primary abstract class which essentially doubles
your connections to the same database. To avoid opening 2 connections to
the same database, Rails treats them the same.

However, because we have this code that removes existing connections,
when an application boots, `ApplicationRecord` will clobber the
connection that the Railtie established even though the connection
configs are the same.

This removal of the connection caused bugs in migrations that load up a
model connected to `ApplicationRecord` (ex `Post.first`) and then calls
`execute("SELECT 1")` (obviously a simplified example). When `execute`
runs the connection is different from the one opened to run the
migration and essentially it is lost when the `remove_connection` code
is called.

To fix this I've updated the code to only remove the connection if the
database config is different. Ultimately I'd like to remove this code
altogether but to do that we first need to stop using
`Base.establish_connection` in the rake tasks and tests. This will fix
the major bugs until I come up with a solution for the areas that
currently need to call `establish_connection` on Base.

The added benefit of this change is that if your app is calling
`establish_connection` multiple times with the same config, it is now
3x faster than the previous implementation because we can return the
found pool instead of setting it up again. To benchmark this I
duplicated the `establish_connection` method to use the new behavior
with a new name.

Benchmark script:

```ruby
require "active_record"
require "logger"
require "benchmark/ips"

config_hash = { "development" => { "primary" => { "adapter" => "mysql2", "username" => "rails", "database" => "activerecord_unittest"}}}
ActiveRecord::Base.configurations = config_hash

db_config = ActiveRecord::Base.configurations.configs_for(env_name: "development", name: "primary")

p "Same model same config"
ActiveRecord::Base.connected_to(role: :writing, prevent_writes: true) do
  Benchmark.ips do |x|
    x.report "establish_connection with remove" do
      ActiveRecord::Base.establish_connection(db_config)
    end

    x.report "establish_connection without remove" do
      ActiveRecord::Base.establish_connection_no_remove(db_config)
    end

    x.compare!
  end
end
```

Benchmark results:

```
Warming up --------------------------------------
establish_connection with remove
                         4.677k i/100ms
establish_connection without remove
                        19.501k i/100ms
Calculating -------------------------------------
establish_connection with remove
                         41.252k (±11.3%) i/s -    205.788k in   5.075525s
establish_connection without remove
                        179.205k (± 6.9%) i/s -    897.046k in   5.029742s

Comparison:
establish_connection without remove:   179205.1 i/s
establish_connection with remove:    41252.3 i/s - 4.34x  (± 0.00) slower
```

Other changes:

1) sqlite3 now disconnects and reconnects the connection when `purge` is
called. This is necessary now that a new connection isn't created
everyt time `establish_connection` is called. Without this change to
purge the new database is left in an inaccessible state causing a
readonly error from the sqlite3 client. This wasn't happening in mysql
or postgres because they were already reconnecting the db connection.
2) I added `remove_connection` to tests that use `ApplicationRecord`.
This is required because `ApplicationRecord` or any class that is a
`primary_abstract_class` will be treated the same as
`ActiveRecord::Base`. This is fine in applications because they are
shared connections, but in the AR test environment, we don't want those
connnections to stick around (we want AR::Base back).
3) In the async tests I removed 2 calls to `establish_connection`. These
were causing sqlite3 tests to leak the state of async_executor because
it's stored on the connection. I'm not sure why these were calling
`establish_connection` but it's not necessary and was leaking state when
now that we are no longer removing the connection.

Fixes: #41855
Fixes: #41876
Fixes: #42873
Fixes: #43004
2022-06-29 11:25:17 -04:00
Rafael Mendonça França
c3c747547a
Merge pull request #45485 from santib/fix-open-redirects
Fix vulnerability in open redirects
2022-06-28 18:27:35 -04:00
Santiago Bartesaghi
708bb9d314 Fix vulnerability on open redirects 2022-06-28 18:31:58 -03:00
Niklas Haeusele
7a555e460f Update the npm package release task to not add the latest package for every version. 2022-06-28 23:16:09 +02:00