This will allow to enable/disable strict_loading mode by default for a model.
The configuration's value is inheritable by subclasses, but they can override that value and
it will not impact parent class:
```ruby
class Developer < ApplicationRecord
self.strict_loading_by_default = true
has_many :projects
end
dev = Developer.first
dev.projects.first
\# => ActiveRecord::StrictLoadingViolationError Exception: Developer is marked as strict_loading and Project cannot be lazily loaded.
```
What is great about this feature that it could help users to nip N+1 queries in
the bud, especially for fresh applications, by setting
`ActiveRecord::Base.strict_loading_by_default = true` / `config.active_record.strict_loading_by_default = true`.
That is also a great way to prevent new N+1 queries in the existing applications
after all the N+1 queries are eliminated.
(See https://guides.rubyonrails.org/v6.0/active_record_querying.html#eager-loading-associations,
https://github.com/seejohnrun/prelude for details on how to fight against N+1 queries).
Related to https://github.com/rails/rails/pull/37400, https://github.com/rails/rails/pull/38541
Follow up to #27962.
#27962 only deprecated `quoted_id`, but still conservatively allowed
passing an Active Record object.
Since the quoting methods on a `connection` are low-level API and
querying API does not rely on that ability, so people should pass casted
value instead of an Active Record object if using the quoting methods
directly.
5 years ago, I made dumping full table options at #17569, especially to
dump `ENGINE=InnoDB ROW_FORMAT=DYNAMIC` to use utf8mb4 with large key
prefix.
In that time, omitting the default engine `ENGINE=InnoDB` was not useful
since `ROW_FORMAT=DYNAMIC` always remains as long as using utf8mb4 with
large key prefix.
But now, MySQL 5.7.9 has finally changed the default row format to
DYNAMIC, utf8mb4 with large key prefix can be used without dumping the
default engine and the row format explicitly.
So now is a good time to make the default engine is omitted.
Before:
```ruby
create_table "accounts", options: "ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci", force: :cascade do |t|
end
```
After:
```ruby
create_table "accounts", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
end
```
To entirely omit `:options` option to make schema agnostic, I've added
`:charset` and `:collation` table options to exclude `CHARSET` and
`COLLATE` from `:options`.
Fixes#26209.
Closes#29472.
See also #33608, #33853, and #34742.
We've learned that `merge` causes duplicated multiple values easily, so
if we missed to deduplicate the values, it will cause weird behavior
like #38052, #39171.
I've investigated the deduplication for the values, at least that had
existed since Rails 3.0.
bed9179aa1
Aggregations with group by multiple fields was introduced at Rails 3.1,
but we missed the deduplication for the aggregation result, unlike the
generated SQL.
a5cdf0b9eb
While the investigation, I've found that `annotate` is also missed the
deduplication.
I don't suppose this weird behavior is intended for both.
So I'd like to deprecate the duplicated behavior in Rails 6.1, and will
be deduplicated all multiple values in Rails 6.2.
To migrate to Rails 6.2's behavior, use `uniq!(:group)` to deduplicate
group fields.
```ruby
accounts = Account.group(:firm_id)
# duplicated group fields, deprecated.
accounts.merge(accounts.where.not(credit_limit: nil)).sum(:credit_limit)
# => {
# [1, 1] => 50,
# [2, 2] => 60
# }
# use `uniq!(:group)` to deduplicate group fields.
accounts.merge(accounts.where.not(credit_limit: nil)).uniq!(:group).sum(:credit_limit)
# => {
# 1 => 50,
# 2 => 60
# }
```
Bump an Active Record instance's lock version after updating its counter
cache. This avoids raising an unnecessary ActiveRecord::StaleObjectError
upon subsequent transactions by maintaining parity with the
corresponding database record's lock_version column.
Related #39236.
`relation.merge` method sometimes replaces mergee side condition, but
sometimes maintain both conditions unless `relation.rewhere` is used.
It is very hard to predict merging result whether mergee side condition
will be replaced or not.
One existing way is to use `relation.rewhere` for merger side relation,
but it is also hard to predict a relation will be used for `merge` in
advance, except one-time relation for `merge`.
To address that issue, I propose to support merging option `:rewhere`,
to allow mergee side condition to be replaced exactly.
That option will allow non-`rewhere` relation behaves as `rewhere`d
relation.
```ruby
david_and_mary = Author.where(id: david.id..mary.id)
# both conflict conditions exists
david_and_mary.merge(Author.where(id: bob)) # => []
# mergee side condition is replaced by rewhere
david_and_mary.merge(Author.rewhere(id: bob)) # => [bob]
# mergee side condition is replaced by rewhere option
david_and_mary.merge(Author.where(id: bob), rewhere: true) # => [bob]
```
Add support for finding records based on signed ids, which are tamper-proof, verified ids that can be set to expire and scoped with a purpose. This is particularly useful for things like password reset or email verification, where you want the bearer of the signed id to be able to interact with the underlying record, but usually only within a certain time period.
This is the opposite direction of #39039.
#39111 fixes `minimum` and `maximum` on date columns with type casting
by column type on the database. But column type has no information for
time zone aware attributes, it means that attribute type should always
be precedence over column type. I've realized that fact in the related
issue report #39248.
I've reverted the expectation of #39039, to make time zone aware
attributes works.
```ruby
steve = Person.find_by(name: "Steve")
david = Author.find_by(name: "David")
relation = Essay.where(writer: steve)
# Before
relation.rewhere(writer: david).to_a # => []
# After
relation.rewhere(writer: david).to_a # => [david]
```
For now `rewhere` only works for truly column names, doesn't work for
alias attributes, nested conditions, associations.
To fix that, need to build new where clause first, and then get
attribute names from new where clause.
The type information for type casting is entirely separated to type
object, so if anyone does passing a column to `type_cast` in Rails 6,
they are likely doing something wrong. See the comment for more details:
28d815b894/activerecord/lib/active_record/connection_adapters/abstract/quoting.rb (L33-L42)
This also deprecates passing legacy binds (an array of `[column, value]`
which is 4.2 style format) to query methods on connection. That legacy
format was kept for backward compatibility, instead of that, I've
supported casted binds format (an array of casted values), it is easier
to construct binds than existing two binds format.
`allowed_index_name_length` was used for internal temporary operations
in SQLite3, since index name in SQLite3 must be globally unique and
SQLite3 doesn't have ALTER TABLE feature (so it is emulated by creating
temporary table with prefix).
`allowed_index_name_length` was to reserve the margin for the prefix,
but actually SQLite3 doesn't have a limitation for identifier name
length, so the margin has removed at 36901e6.
Now `allowed_index_name_length` is no longer relied on by any adapter,
so I'd like to remove the internal specific method which is no longer
used.
`in_clause_length` was added at c5a284f to address to Oracle IN clause
length limitation.
Now `in_clause_length` is entirely integrated in Arel visitor since
#35838 and #36074.
Since Oracle visitors are the only code that rely on `in_clause_length`.
so I'd like to remove that from Rails code base, like has removed Oracle
visitors (#38946).
Currently, `count` and `average` always returns numeric value, but
`sum`, `maximum`, and `minimum` not always return numeric value if
aggregated on custom attribute type.
I think that inconsistent behavior is surprising:
```ruby
# All adapters except postgresql adapter are affected
# by custom type casting.
Book.group(:status).sum(:status)
# => { "proposed" => "proposed", "published" => nil }
```
That is caused by fallback looking up cast type to `type_for(column)`.
Now all supported adapters can return numeric value without that
fallback, so I think we can remove that, it will also fix aggregate
functions to return numeric value consistently.
This PR allows for passing `if_exists` options to the `remove_index`
method so that we can ignore already removed indexes. This work follows
column `if/if_not_exists` from #38352 and `:if_not_exists` on `add_index`
from #38555.
We've found this useful at GitHub, there are migrations where we don't
want to raise if an index was already removed. This will allow us to
remove a monkey patch on `remove_index`.
I considered raising after the `index_name_for_remove` method is called
but that method will raise if the index doesn't exist before we get to
execute. I have a commit that refactors this but after much
consideration this change is cleaner and more straightforward than other
ways of implementing this.
This change also adds a little extra validation to the `add_index` test.
Fix `nodoc` on edited methods.
This removes ibm_db, informix, mssql, oracle, and oracle12 Arel visitors
which are not used in the code base.
Actually oracle and oracle12 visitors are used at oracle-enhanced
adapter, but now I think that those visitors should be in the adapter's
repo like sqlserver adapter and the dedicated Arel visitor
(https://github.com/rails-sqlserver/activerecord-sqlserver-adapter/blob/master/lib/arel/visitors/sqlserver.rb),
otherwise it is hard to find a bug and review PRs for the oracle
visitors (e.g. #35838, #37646), since we don't have knowledge and
environment enough for Oracle.
Previously, if `build_association` was called multiple times for a `has_one` association but never committed to the database, the first newly-associated record would trigger `touch` during the attempted removal of the record.
For example:
class Post < ActiveRecord::Base
has_one :comment, inverse_of: :post, dependent: :destroy
end
class Comment < ActiveRecord::Base
belongs_to :post, inverse_of: :comment, touch: true
end
post = Post.create!
comment_1 = post.build_comment
comment_2 = post.build_comment
When `comment_2` is initialized, the `has_one` would attempt to destroy `comment_1`, triggering a `touch` on `post` from an association record that hasn't been committed to the database.
This removes the attempt to delete an associated `has_one` unless it’s persisted.
When we try to create a table which already exists which also adds
indexes, then the `if_not_exists` option passed to `create_table` is
not extended to indexes. So the migration results into an error if the
table and indexes already exist.
This change extends the `if_not_exists` support to `add_index` so that
if the migration tries to create a table which also has existing
indexes, error won't be raised.
Also as a side-effect individual `add_index` calls will also accept
`if_not_exists` option and respect it henceforth.
[Prathamesh Sonpatki, Justin George]
This reverts commit f265e0ddb1139a91635b7905aae1be76b22c6db1, reversing
changes made to 08dfa9212df4a6bf332a4c49b7e8a7d876a69331.
Reverted due to surprising behavior for applications. We need to
deprecate this behavior first instead of raising by default.
fix insert_all enum test
fix rubocop, change test to double quotes
Update activerecord/test/cases/insert_all_test.rb
fix inser_all_enum_values test: double quotes and order relation before pluck
Co-Authored-By: Ryuta Kamizono <kamipo@gmail.com>
change insert_all_enum_values test to not skip duplicates so it works across adapters
With a multiple database application `db:rollback` becomes problematic.
We can't rollback just the primary, that doesn't match the behavior in
the other tasks. We can't rollback a migration for every database, that
is unexpected.
To solve this I handled `db:rollback` the same way I handled `:up` and
`:down`. If `db:rollback` is called for a multi-db application then it
will raise an error recommending you use `db:rollback:[NAME]` instead.
Calling `db:rollback:primary` or `db:rollback:animals` will rollback
the migration for the number of steps specified.
Closes: #38513
Follow-up to: #34078
If a transaction is wrapped in a Timeout.timeout(duration) block, then the
transaction will be committed when the transaction block is exited from the
timeout, since it uses `throw`. Ruby code doesn't have a way to distinguish
between a block being exited from a `return`, `break` or `throw`, so
fixing this problem for the case of `throw` would require a backwards
incompatible change for block exited with `return` or `break`. As such,
the current behaviour so it can be changed in the future.
This behaviour is in
rails/activerecord/lib/active_record/enum.rb #serialize(value) line no 143
if value is not present in mapping we are sending the value back ,
which in mysql returns unrelated record.
I have changed to return nil is value is not present in mapping
Implemented code review changes
Improved test case coverage
[ci skip] - cosmetic changes for better readibility of change log
Signed-off-by: ak <atulkanswal@gmail.com>
Behavior has not changed here but the previous API could be
misleading to people who thought it would switch connections for only
that class. `connected_to` switches the context from which we are
getting connections, not the connections themselves.
Co-authored-by: John Crepezzi <john.crepezzi@gmail.com>
Applications can now connect to multiple shards and switch between
their shards in an application. Note that the shard swapping is
still a manual process as this change does not include an API for
automatic shard swapping.
Usage:
Given the following configuration:
```yaml
production:
primary:
database: my_database
primary_shard_one:
database: my_database_shard_one
```
Connect to multiple shards:
```ruby
class ApplicationRecord < ActiveRecord::Base
self.abstract_class = true
connects_to shards: {
default: { writing: :primary },
shard_one: { writing: :primary_shard_one }
}
```
Swap between shards in your controller / model code:
```ruby
ActiveRecord::Base.connected_to(shard: :shard_one) do
# Read from shard one
end
```
The horizontal sharding API also supports read replicas. See
guides for more details.
This PR also moves some no-doc'd methods into the private namespace as
they were unnecessarily public. We've updated some error messages and
documentation.
Co-authored-by: John Crepezzi <john.crepezzi@gmail.com>
I have so. many. regrets. about using `spec_name` for database
configurations and now I'm finally putting this mistake to an end.
Back when I started multi-db work I assumed that eventually
`connection_specification_name` (sometimes called `spec_name`) and
`spec_name` for configurations would one day be the same thing. After
2 years I no longer believe they will ever be the same thing.
This PR deprecates `spec_name` on database configurations in favor of
`name`. It's the same behavior, just a better name, or at least a
less confusing name.
`connection_specification_name` refers to the parent class name (ie
ActiveRecord::Base, AnimalsBase, etc) that holds the connection for it's
models. In some places like ConnectionHandler it shortens this to
`spec_name`, hence the major confusion.
Recently I've been working with some new folks on database stuff and
connection management and realize how confusing it was to explain that
`db_config.spec_name` was not `spec_name` and
`connection_specification_name`. Worse than that one is a symbole while
the other is a class name. This was made even more complicated by the
fact that `ActiveRecord::Base` used `primary` as the
`connection_specification_name` until #38190.
After spending 2 years with connection management I don't believe that
we can ever use the symbols from the database configs as a way to
connect the database without the class name being _somewhere_ because
a db_config does not know who it's owner class is until it's been
connected and a model has no idea what db_config belongs to it until
it's connected. The model is the only way to tie a primary/writer config
to a replica/reader config. This could change in the future but I don't
see value in adding a class name to the db_configs before connection or
telling a model what config belongs to it before connection. That would
probably break a lot of application assumptions. If we do ever end up in
that world, we can use name, because tbh `spec_name` and
`connection_specification_name` were always confusing to me.
Add `#strict_loading` to any record to prevent lazy loading of associations.
`strict_loading` will cascade down from the parent record to all the
associations to help you catch any places where you may want to use
`preload` instead of lazy loading. This is useful for preventing N+1's.
Co-authored-by: Aaron Patterson <aaron.patterson@gmail.com>
indexes in a table.
Currently the pg_class catalog is filtered out to retrieve the indexes in a
table by its relkind value. Which in versions lower than 11 of PostgreSQL
is always `i` (lower case). But since version 11, PostgreSQL
supports partitioned indexes referenced with a relkind value of `I`
(upper case). This makes any feature within the current code base to exclude those
partitioned indexes.
The solution proposed is to make use of the `IN` clause to filter those
relkind values of `i` and/or `I` when retrieving a table indexes.
This PR adds support for `if_exists` on `remove_column` and
`if_not_exists` on `add_column` to support silently ignoring migrations
if the remove tries to remove a non-existent column or an add tries to
add an already existing column.
We (GitHub) have custom monkey-patched support for these features and
would like to upstream this behavior.
This matches the same behavior that is supported for `create_table` and
`drop_table`. The behavior for sqlite is different from mysql/postgres
and sqlite for remove column and that is reflected in the tests.
This PR moves advisory lock to it's own connection instead of
`ActiveRecord::Base` to fix#37748. As a note the issue is present on
both mysql and postgres. We don't see it on sqlite3 because sqlite3
doesn't support advisory locks.
The underlying problem only appears if:
1) the app is using multiple databases, and therefore establishing a new
connetion in the abstract models
2) the app has a migration that loads a model (ex `Post.update_all`)
which causes that new connection to get established.
This is because when Rails runs migrations the default connections are
established, the lock is taken out on the `ActiveRecord::Base`
connection. When the migration that calls a model is loaded, a new
connection will be established and the lock will automatically be
released.
When Rails goes to release the lock in the ensure block it will find
that the connection has been closed. Even if the connection wasn't
closed the lock would no longer exist on that connection.
We originally considered checking if the connection was active, but
ultimately that would hide that the advisory locks weren't working
correctly because there'd be no lock to release.
We also considered making the lock more granular - that it only blocked
on each migration individually instead of all the migrations for that
connection. This might be the right move going forward, but right now
multi-db migrations that load models are very broken in Rails 6.0 and
master.
John and I don't love this fix, it requires a bit too much knowledge of
internals and how Rails picks up connections. However, it does fix the
issue, makes the lock more global, and makes the lock more resilient to
changing connections.
Co-authored-by: John Crepezzi <john.crepezzi@gmail.com>
This updates the database tasks for dumping the Active Record schema cache as
well as clearing the schema cache file, allowing the path to be defined in the
database configuration YAML file.
As before, the value can also be defined in an ENV variable, though this would
not work for a multi-db application. If the value is specified neither in the
DB config, nor in the ENV, then the path will continue to be derived from the
DB config spec_name.
Note that in order to make this change cleaner I also moved a bit of logic
out of a rake task and into the DatabaseTasks class, for symmetry.
We have two rake tasks for the schema cache:
$ rake db:schema:cache:dump
$ rake db:schema:cache:clear
The cache:dump task was implemented in DatabaseTasks, but the
cache:clear one was not.
I also added some tests for the behavior that I was changing, since some of
the code paths weren't tested.
Calling `#remove_connection` on the handler is deprecated in favor of
`#remove_connection_pool`. This change was made to support changing the
return value from a hash to a `DatabaseConfig` object.
`#remove_connection` will be removed in 6.2.
NOTE: `#remove_connection` on `ActiveRecord::Base` will also now return
a `DatabaseConfig` object. We didn't use a deprecation here since
it's not documented that this method used to return a `Hash`.
Co-authored-by: John Crepezzi <john.crepezzi@gmail.com>
Database configurations are now objects almost everywhere, so we don't
need to fake access to a hash with `#default_hash` or it's alias `#[]`.
Applications should `configs_for` and pass `env_name` and `spec_name` to
get the database config object. If you're looking for the default for
the test environment you can pass `configs_for(env_name: "test", spec_name:
"primary")`. Change test to developement to get the dev config, etc.
`#default_hash` and `#[]` will be removed in 6.2.
Co-authored-by: John Crepezzi <john.crepezzi@gmail.com>
Since test fixtures share connections (due to transactional tests) we
end up overwriting the reading configuration so Rails doesn't recognize
it as a replica connection.
This change ensures that if we're using the `reading` role that
connections will always have prevent writes turned on.
If you need a replica connection that does not block writes, you should
use a different role name other than `:reading`.
The db selector test and connection handlers test have been updated to
test for these changes. In the db selector test we don't always have a
writing handler so I updated test fixtures to return if that's nil.
Lastly one test needed to be updated to use a different handler name due
to it needing to write to successfully test what it needs to test.
Fixes#37765
As multiple databases have evolved it's becoming more and more
confusing that we have a `connection_specification_name` that defaults
to "primary" and a `spec_name` on the database objects that defaults to
"primary" (my bad).
Even more confusing is that we use the class name for all
non-ActiveRecord::Base abstract classes that establish connections. For
example connections established on `class MyOtherDatabaseModel <
ApplicationRecord` will use `"MyOtherDatabaseModel"` as it's connection
specification name while `ActiveRecord::Base` uses `"primary"`.
This PR deprecates the use of the name `"primary"` as the
`connection_specification_name` for `ActiveRecord::Base` in favor of
using `"ActiveRecord::Base"`.
In this PR the following is true:
* If `handler.establish_connection(:primary)` is called, `"primary"`
will not throw a deprecation warning and can still be used for the
`connection_specification_name`. This also fixes a bug where using this
method to establish a connection could accidentally overwrite the actual
`ActiveRecord::Base` connection IF that connection was not using a
configuration named `:primary`.
* Calling `handler.retrieve_connection "primary"` when
`handler.establish_connection :primary` has never been called will
return the connection for `ActiveRecord::Base` and throw a deprecation
warning.
* Calling `handler.remove_connection "primary"` when
`handler.establish_connection :primary` has never been called will
remove the connection for `ActiveRecord::Base` and throw a deprecation
warning.
See #38179 for details on more motivations for this change.
Co-authored-by: John Crepezzi <john.crepezzi@gmail.com>
Add ActiveRecord::Relation#cache_key_with_version. This method will be
used by ActionController::ConditionalGet to ensure that when collection
cache versioning is enabled, requests using ConditionalGet don't return
the same ETag header after a collection is modified.
Prior to the introduction of collection cache versioning in
4f2ac80d4cdb01c4d3c1765637bed76cc91c1e35, all collection cache keys
included a version. However, with cache versioning enabled, collection
cache keys remain constant. In turn, ETag headers remain constant,
rendering them ineffective.
This commit takes the cache_key_with_version method used for individual
Active Record objects (from aa8749eb52d7919a438940c9218cad98d892f9ad),
and adds it to collections.
Fixes https://github.com/rails/rails/issues/28827.
The steps to reproduce are as follows:
git clone git@github.com:bbuchalter/rails-issue-28827.git
cd rails-issue-28827
bundle install
bin/rails db:create
Observe that we create two databases when invoking db:create: development and test. Now observe what happens when we invoke our drop command while using DATABASE_URL.
DATABASE_URL=sqlite3://$(pwd)/db/database_url.sqlite3 bin/rails db:create
As expected, the development environment now uses the DATABASE_URL. What is unexpected is that the test environment does not.
It's unclear what the expected behavior should be in this case, but the cause of it is this: 9f2c74eda0/activerecord/lib/active_record/tasks/database_tasks.rb (L494)
Because of each_local_configuration, there seems to be no way invoke these database rake on only the development environment to ensure DATABASE_URL is respected.
The smallest scope of change I can think to make would be to conditionalize this behavior so it does not get applied when DATABASE_URL is present.
`name` is used by Rails to find the configuration by connection
specification name, but database adapters don't need to use `name` in
order to establish a connection. This is part of our work to separate
what the database needs to connect (the configuration hash) and the
what Rails needs to find connections (everything else).
Co-authored-by: John Crepezzi <john.crepezzi@gmail.com>
- I'm hoping to get this change accepted even though this flag was
introduced years ago in 6576f7354e50afb79881aaf3a6f50f4e81dfab70
My use case is the following:
We were never enforcing belongs to association and we have a lot
of models that implicitely declare an association as optional.
We are now changing all our models to make associations required
by default.
Since we have a lot of models (more than 1000), I'd like to
progressively enable them to use the `belongs_to_required_by_default`
flag.
The problem is that this flag is a mattr_accessor and doesn't to be
set per model. We basically need to modify all our models (which
could take years) before being able to modify the global flag.
I'd like to change this flag to a class_attribute to solve the
issue.
The `connection_config` method returns a `Hash`, but since we're moving
toward a place where we're using `DatabaseConfiguration::DatabaseConfig`
objects everywhere, we're introducing a new method here to replace it
called `connection_db_config`.
Co-authored-by: eileencodes <eileencodes@gmail.com>
A querystring value should be allowed to include an equal sign `=`.
This is necessary to support passing `options` for a PostgresSQL connection.
```
development:
url: postgresql://localhost/railsdevapp_development?options=-cmysetting.debug=on
```
Before this PR, attempting to start the rails process with that configuration would result in an error:
```
> bundle exec rails console
Traceback (most recent call last):
49: from bin/rails:4:in `<main>'
48: from bin/rails:4:in `require'
...
1: from /rails/activerecord/lib/active_record/database_configurations/connection_url_resolver.rb:58:in `query_hash'
/rails/activerecord/lib/active_record/database_configurations/connection_url_resolver.rb:58:in `[]': invalid number of elements (3 for 1..2) (ArgumentError)
```
After this PR, rails can properly parse the configuration:
```
> bundle exec rails console
Loading development environment (Rails 6.1.0.alpha)
2.6.5 :001 > ActiveRecord::Base.connection.select_all("show mysetting.debug").to_a
(0.4ms) show mysetting.debug
=> [{"mysetting.debug"=>"on"}]
```
Applying `includes` and `joins` to a relation that selected additional
database fields would cause those additional fields not to be included
in the results even though they were queried from the database:
posts = Post.select('1 as other').includes(:tbl).joins(:tbl)
posts.to_sql.include?('1 as other') #=> true
posts.first.attributes.include?('other') #=> false
This commit includes these additionally selected fields in the
instantiated results.
The `database` kwarg in `connected_to` has resulted in a lot of bug
reports that are trying to use it for sharding when that's not the
intent of the key. After considering where the database kwarg is used in
tests and thinking about usecases for it, we've determined it should be
removed.
There are plans to add sharding support and in the mean time the
database kwarg isn't the right solution for that. Applications that need
to create new connections can use establish_connection or connects_to.
Since the database key causes new connections to be established on every
call, that causes bugs if connected_to with a database kwarg is used
during a request or for any connection that's not a one-off.
Co-authored-by: John Crepezzi <john.crepezzi@gmail.com>
We have these two objects, `ConnectionAdapters::Resolver` and
`DatabaseConfiguratons` that implement a lot of the same logic. One of
them is used for configurations defined in `config/database.yml` and the
other is used when passing raw configurations `String` or `Hash` objects
into methods like `establish_connection`.
Over time these two have diverged a bit. In the interest of less code
complexity, and more consistency for users this commit brings them back
together.
* Remove `Resolver` altogether and replace its primary method with
`DatabaseConfigurations#resolve`.
* Move `resolve_pool_config` over to the `ConnectionPool` alongside the code
that uses it.
Previously in some places we used symbol keys, and in some places we used
string keys. That made it pretty confusing to figure out in a particular
place what type of configuration object you were working with.
Now internally, all configuration hashes are keyed by symbols and
converted to such on the way in.
A few exceptions:
- `DatabaseConfigurations#to_h` still returns strings for backward compatibility
- Same for `legacy_hash`
- `default_hash` previously could return strings, but the associated
comment mentions it returns symbol-key `Hash` and now it always does
Because this is a change in behavior, a few method renames have happened:
- `DatabaseConfig#config` is now `DatabaseConfig#configuration_hash` and returns a symbol-key `Hash`
- `ConnectionSpecification#config` is now `ConnectionSpecification#underlying_configuration_hash` and returns the `Hash` of the underlying `DatabaseConfig`
- `DatabaseConfig#config` was added back, returns `String`-keys for backward compatibility, and is deprecated in favor of the new `configuration_hash`
Co-authored-by: eileencodes <eileencodes@gmail.com>
Before this patch, column names could only be passed as a positional
argument when no other options were supplied:
remove_index :reports, :report_id
Passing column names positionally along with other options would fail:
remove_index :reports, :report_id, unique: true
# => ArgumentError: wrong number of arguments (given 3, expected 1..2)
In 3809c80cd55ac2838f050346800889b6f8e041ef, adding an index with a
name that's already in use was changed from an error to a warning, to
allow other statements in the same migration to complete successfully.
In 55d0d57bfc72c0bdbc81ae5d95c99729f16899af this decision was reversed,
but instead of allowing the statement to execute and raise an adapter-
specific error as it did before, an `ArgumentError` was raised instead.
This interferes with a legitimate use case: on MySQL, it's possible to
drop an index and add another one with the same name in a single `ALTER`
statement. Right now an `ArgumentError` is raised when trying to do so,
even though the resulting statement would execute successfully.
There's no corresponding `ArgumentError` raised when attempting to add a
duplicate column, so I think we can safely remove the check and allow
the adapter to raise its own error about duplicate indexes again.
The `InsertAll` class currently calls `exec_query`, which doesn't give
the query cache enough information to know that it needs to be cleared.
By adding an `exec_insert_all` method that calls `exec_query` internally
we can configure the query cache to clear when that method is called.
If a user is using the middleware for swapping database connections and
manually calling `connected_to` in a controller/model/etc without
calling `while_preventing_writes(false)` there is potential for a race
condition where writes will be blocked.
While the user could _just_ call `while_preventing_writes` in the same
place they call `connected_to` this would mean that all cases need to
call two methods.
This PR changes `connected_to` to call `while_preventing_writes`
directly. By default we'll assume you don't want to prevent writes, but
if called with `connected_to(role: :writing, prevent_writes: true)` or
from the middleware (which calls `connected_to` this way) the writes
will be blocked.
For replicas, apps should use readonly users to enforce not writing
rather than `while_preventing_writes` directly.
Should fix the remaining issues in
https://github.com/rails/rails/issues/36830
Currently, string joins are always applied as last joins part, and Arel
join nodes are always applied as leading joins part (since #36304), it
makes people struggled to preserve user supplied joins order.
To mitigate this problem, preserve the order of string joins and Arel
join nodes either before or after of association joins.
Fixes#36761.
Fixes#34328.
Fixes#24281.
Fixes#12953.
Previously matches_regex was only availble on PostgreSql, this will enable it for MySql
Usage example:
users = User.arel_table;
users = User.arel_table; User.where(users[:email].matches_regexp('(.*)\@gmail.com'))
Update activerecord/test/cases/arel/visitors/mysql_test.rb
Co-Authored-By: Ryuta Kamizono <kamipo@gmail.com>
Allow specifying what fixtures can be ignored by setting
`ignore` in fixtures YAML file:
# users.yml
_fixture:
ignore:
- base
base: &base
admin: false
introduction: "This is a default description"
admin:
<<: *base
admin: true
visitor:
<<: *base
In the above example, "base" fixture will be ignored when creating
users fixture. This is helpful when you want to inherit attributes
and it makes your fixtures more "DRY".
This commit adds a feature which allows separate database ENV variables
to be defined for each spec in a 3-tier config. The names for the
environment variables will be `#{name.upcase}_DATABASE_URL`
This commit also introduces a change in behavior around handling of
`DATABASE_URL`. Instead of using `DATABASE_URL` to change _all_ specs
in a multi-database configuration, it will now only affect the `primary`
connection.
Ruby 2.7 introduces beginless ranges (..value and ...value) and as with
endless ranges we can turn these into inequalities, enabling expressions
such as
Order.where(created_at: ..1.year.ago)
User.where(karma: ...0)
When a enum element contains the prefix 'not_'. I warns to users
to be aware of this new feature.
Example code:
class Foo < ActiveRecord::Base
enum status: [:sent, :not_sent]
end
When a record does not have a table name, as in the case for a record
with `self.abstract_class = true` and no `self.table_name` set the error
message raises a cryptic:
"ActiveRecord::StatementInvalid: Could not find table ''" this patch now
raises a new `TableNotSpecified Error`
Fixes: #36274
Co-Authored-By: Eugene Kenny <elkenny@gmail.com>
When SQLite connects it will silently create a database if the database does not
exist. This behaviour causes different issues because of inconsistent behaviour
between adapters: #36383, #32914. This commit adds a `database_exists?` method
as a way to check the database without creating it. This is a stepping stone to
fully resolving the above issues.
GROUP BY with virtual count attribute is invalid for almost all
databases, but it is valid for PostgreSQL, and it had worked until Rails
5.2.2, so it is a regression for Rails 5.2.3 (caused by 311f001).
I can't find perfectly solution for fixing this for now, but I would not
like to break existing apps, so I decided to allow referencing virtual
count attribute in ORDER BY clause when GROUP BY aggrigation (it partly
revert the effect of 311f001) to fix the regression #36022.
Fixes#36022.
* Make ActiveRecord `ConnectionPool.connections` thread-safe.
ConnectionPool documentation is clear on the need to synchronize
access to @connections but also states that public methods do not
require synchronization. Existing code exposed @connections
directly via attr_reader. The fix uses synchronize() to lock
@connections then returns a copy to the caller using Array.dup().
Includes comments on the connections method that thread-safe access
to the connections array does not imply thread-safety of accessing
methods on the actual connections.
Adds a test-case that modifies the pool using a supported method
in one thread while a second thread accesses pool.connections.
The test fails without this patch.
Fixes#36465.
* Update activerecord/test/cases/connection_pool_test.rb
[jeffdoering + Rafael Mendonça França]
If an sqlite3 table contains a decimal column behind columns with a collation
definition, then parsing the collation of all preceeding columns will fail --
the collation will be missed without notice.
Before this fix it would either generate an invalid schema, passing `comment` option twice to `create_table`, or it move the comment from primary key column to the table if table had no comment when the dump was generated.
The situation now is that a comment on primary key will be ignored (not present on schema).
Fixes#29966
`where.not` with polymorphic association is partly fixed incidentally at
213796f (refer #33493, #26207, #17010, #16983, #14161), and I've added
test case e9ba12f to avoid lose that fix accidentally in the future.
In Rails 5.2, `where.not(polymorphic: object)` works as expected as
NAND, but `where.not(polymorphic_type: object.class.polymorphic_name,
polymorphic_id: object.id)` still unexpectedly works as NOR.
To will make `where.not` working desiredly as NAND in Rails 6.1, this
deprecates `where.not` working as NOR. If people want to continue NOR
conditions, we'd encourage to them to `where.not` each conditions
manually.
```ruby
all = [treasures(:diamond), treasures(:sapphire), cars(:honda), treasures(:sapphire)]
assert_equal all, PriceEstimate.all.map(&:estimate_of)
```
In Rails 6.0:
```ruby
sapphire = treasures(:sapphire)
nor = all.reject { |e|
e.estimate_of_type == sapphire.class.polymorphic_name
}.reject { |e|
e.estimate_of_id == sapphire.id
}
assert_equal [cars(:honda)], nor
without_sapphire = PriceEstimate.where.not(
estimate_of_type: sapphire.class.polymorphic_name, estimate_of_id: sapphire.id
)
assert_equal nor, without_sapphire.map(&:estimate_of)
```
In Rails 6.1:
```ruby
sapphire = treasures(:sapphire)
nand = all - [sapphire]
assert_equal [treasures(:diamond), cars(:honda)], nand
without_sapphire = PriceEstimate.where.not(
estimate_of_type: sapphire.class.polymorphic_name, estimate_of_id: sapphire.id
)
assert_equal nand, without_sapphire.map(&:estimate_of)
```
Resolves#31209.
Cache versioning enables the same cache key to be reused when the object
being cached changes by moving the volatile part of the cache key out of
the cache key and into a version that is embedded in the cache entry.
This is already occurring when the object being cached is an
`ActiveRecord::Base`, but when caching an `ActiveRecord::Relation`
we are currently still putting the volatile information (max updated at
and count) as part of the cache key.
This PR moves the volatile part of the relations `cache_key` into the
`cache_version` to support recycling cache keys for
`ActiveRecord::Relation`s.
Currently the rollback only restores primary key value, `new_record?`,
`destroyed?`, and `frozen?`. Since the `save` clears current dirty
attribute states, retrying save after rollback will causes no change
saved if partial writes is enabled (by default).
This makes `remember_transaction_record_state` remembers original values
then restores dirty attribute states after rollback.
Fixes#15018.
Fixes#30167.
Fixes#33868.
Fixes#33443.
Closes#33444.
Closes#34504.
We can revert migrations using `change_column_comment` or
`change_table_comment` at current master.
However, results are not what we expect: comments are remained in new
status.
This change tells previous comment to these methods in a way like
`change_column_default`.
Regardless of a record isn't saved (e.g. validation is failed),
`after_commit` / `after_rollback` callbacks are invoked for now.
To fix the issue, this adds a record to the current transaction only
when a record is actually saved.
Fixes#29747.
Closes#29833.
The minimum token length is set to 24 due to security constraints. We
can now specify a longer length through the length: parameter. This is
especially useful for cases when your data storage is case-insensitive
and you want to increase your entropy.
Follow-up of 5c71000, #29834, and #30271.
Currently, preloading and eager loading are not to be affected by
scoping, with the exception of `unscoped`.
But non eager loaded association access is still affected by scoping.
Although this is a breaking change, the association loading will work
consistently whether preloaded / eager loaded or not.
Before:
```ruby
Post.where("1=0").scoping do
Comment.find(1).post # => nil
Comment.preload(:post).find(1).post # => #<Post id: 1, ...>
Comment.eager_load(:post).find(1).post # => #<Post id: 1, ...>
end
```
After:
```ruby
Post.where("1=0").scoping do
Comment.find(1).post # => #<Post id: 1, ...>
Comment.preload(:post).find(1).post # => #<Post id: 1, ...>
Comment.eager_load(:post).find(1).post # => #<Post id: 1, ...>
end
```
Fixes#34638.
Fixes#35398.
* use backticks instead of `+`
* and more (e.g. missed replacing `Array#excluding` and
`Enumerable#excluding` in b89a3e7e638a50c648a17d09c48b49b707e1d90d)
When adding a child record via a has_many :through association,
build_through_record would previously build the join record, and then
assign the child record and source_type option to it. Because the
before_add and after_add callbacks are called as part of build, however,
this caused the callbacks to receive incomplete records, specifically
without the other end of the has_many :through association. Collecting
all attributes before building the join record ensures the callbacks
receive the fully constructed record.
This patch has two main portions:
1. Add SQL comment support to Arel via Arel::Nodes::Comment.
2. Implement a Relation#annotate method on top of that.
== Adding SQL comment support
Adds a new Arel::Nodes::Comment node that represents an optional SQL
comment and teachers the relevant visitors how to handle it.
Comment nodes may be added to the basic CRUD statement nodes and set
through any of the four (Select|Insert|Update|Delete)Manager objects.
For example:
manager = Arel::UpdateManager.new
manager.table table
manager.comment("annotation")
manager.to_sql # UPDATE "users" /* annotation */
This new node type will be used by ActiveRecord::Relation to enable
query annotation via SQL comments.
== Implementing the Relation#annotate method
Implements `ActiveRecord::Relation#annotate`, which accepts a comment
string that will be appeneded to any queries generated by the relation.
Some examples:
relation = Post.where(id: 123).annotate("metadata string")
relation.first
# SELECT "posts".* FROM "posts" WHERE "posts"."id" = 123
# LIMIT 1 /* metadata string */
class Tag < ActiveRecord::Base
scope :foo_annotated, -> { annotate("foo") }
end
Tag.foo_annotated.annotate("bar").first
# SELECT "tags".* FROM "tags" LIMIT 1 /* foo */ /* bar */
Also wires up the plumbing so this works with `#update_all` and
`#delete_all` as well.
This feature is useful for instrumentation and general analysis of
queries generated at runtime.
We as Arm Treasure Data are using Optimizer Hints with a monkey patch
(https://gist.github.com/kamipo/4c8539f0ce4acf85075cf5a6b0d9712e),
especially in order to use `MAX_EXECUTION_TIME` (refer #31129).
Example:
```ruby
class Job < ApplicationRecord
default_scope { optimizer_hints("MAX_EXECUTION_TIME(50000) NO_INDEX_MERGE(jobs)") }
end
```
Optimizer Hints is supported not only for MySQL but also for most
databases (PostgreSQL on RDS, Oracle, SQL Server, etc), it is really
helpful to turn heavy queries for large scale applications.
Adds a method to ActiveRecord allowing records to be inserted in bulk without instantiating ActiveRecord models. This method supports options for handling uniqueness violations by skipping duplicate records or overwriting them in an UPSERT operation.
ActiveRecord already supports bulk-update and bulk-destroy actions that execute SQL UPDATE and DELETE commands directly. It also supports bulk-read actions through `pluck`. It makes sense for it also to support bulk-creation.
* Add `ActiveRecord::Base.connection.truncate` for SQLite3 adapter.
SQLite doesn't support `TRUNCATE TABLE`, but SQLite3 adapter can support
`ActiveRecord::Base.connection.truncate` by using `DELETE FROM`.
`DELETE` without `WHERE` uses "The Truncate Optimization",
see https://www.sqlite.org/lang_delete.html.
* Add `rails db:seed:replant` that truncates database tables and loads the seeds
Closes#34765
When using `select` with `'DISTINCT( ... )'` if you use method `size` on a non loaded relation it overrides the column selected by passing `:all` so it returns different value than count.
This fixes#35214
Related cbcdecd, 2a56b2d.
This is a regression caused by cbcdecd.
If query caching is enabled, prepared statement handles are never
re-used, since we missed that a query is preprocessed when query caching
is enabled, but doesn't keep the `preparable` flag.
We should care about that case.
Incrementing the lock version invalidates any other process's optimistic
lock, which is the desired outcome: the record no longer looks the same
as it did when they loaded it.
This is a follow-up of #35310.
Currently `Topic.find_by(id: "not-a-number")` matches to a `id = 0`
record. That is considered as silently leaking information.
If non numeric string is given to find by an integer column, it should
not be matched to any record.
Related #12793.
That is considered as silently leaking information.
If type casting doesn't return any actual value, it should not be
matched to any record.
Fixes#33624.
Closes#33946.
This deprecates using class level querying methods if the receiver scope
regarded as leaked, since #32380 and #35186 may cause that silently
leaking information when people upgrade the app.
We need deprecation first before making those.
This reverts commit b67d5c6dedbf033515a96a95d24d085bf99a0d07, reversing
changes made to 2e018361c7c51e36d1d98bf770b7456d78dee68b.
Reason: #35186 may cause that silently leaking information when people
upgrade the app.
We need deprecation first before making this.
Fixes GH#28706. Now rails g migration create_users and rails g model User have the same behavior for timestamps since they implement the same migration template. The expected behavior is that this create table migration will create the table with timestamps unless you pass --no-timestamps or --skip-timestamps to the generator. The expected migration should match what you get when you use the model generator. Using the migration generator, which doesn't have a class_option for timestamps would cause them to not be added to the migration file. Now the migration behavior of the migration generator, create_table only, is aligned with the migration behavior of the model generator. Also modified relevant example of ActiveRecord Migrations Guide.
`relation.create` populates scope attributes to new record by `scoping`,
it is necessary to assign the scope attributes to the record and to find
STI subclass from the scope attributes.
But the effect of `scoping` is class global, it was caused undesired
behavior that pollute all class level querying methods in initialization
block and callbacks (`after_initialize`, `before_validation`,
`before_save`, etc), which are user provided code.
To avoid the leaking scope issue, restore the original current scope
before initialization block and callbacks are invoked.
Fixes#9894.
Fixes#17577.
Closes#31526.
Active Record uses `scoping` to delegate to named scopes from relations
for propagating the chaining source scope. It was needed to restore the
source scope in named scopes, but it was caused undesired behavior that
pollute all class level querying methods.
Example:
```ruby
class Topic < ActiveRecord::Base
scope :toplevel, -> { where(parent_id: nil) }
scope :children, -> { where.not(parent_id: nil) }
scope :has_children, -> { where(id: Topic.children.select(:parent_id)) }
end
# Works as expected.
Topic.toplevel.where(id: Topic.children.select(:parent_id))
# Doesn't work due to leaking `toplevel` to `Topic.children`.
Topic.toplevel.has_children
```
Since #29301, the receiver in named scopes has changed from the model
class to the chaining source scope, so the polluting class level
querying methods is no longer required for that purpose.
Fixes#14003.
The following PR adds behavior to Rails to allow an application to
automatically switch it's connection from the primary to the replica.
A request will be sent to the replica if:
* The request is a read request (`GET` or `HEAD`)
* AND It's been 2 seconds since the last write to the database (because
we don't want to send a user to a replica if the write hasn't made it
to the replica yet)
A request will be sent to the primary if:
* It's not a GET/HEAD request (ie is a POST, PATCH, etc)
* Has been less than 2 seconds since the last write to the database
The implementation that decides when to switch reads (the 2 seconds) is
"safe" to use in production but not recommended without adequate testing
with your infrastructure. At GitHub in addition to the a 5 second delay
we have a curcuit breaker that checks the replication delay
and will send the query to a replica before the 5 seconds has passed.
This is specific to our application and therefore not something Rails
should be doing for you. You'll need to test and implement more robust
handling of when to switch based on your infrastructure. The auto
switcher in Rails is meant to be a basic implementation / API that acts
as a guide for how to implement autoswitching.
The impementation here is meant to be strict enough that you know how to
implement your own resolver and operations classes but flexible enough
that we're not telling you how to do it.
The middleware is not included automatically and can be installed in
your application with the classes you want to use for the resolver and
operations passed in. If you don't pass any classes into the middleware
the Rails default Resolver and Session classes will be used.
The Resolver decides what parameters define when to
switch, Operations sets timestamps for the Resolver to read from. For
example you may want to use cookies instead of a session so you'd
implement a Resolver::Cookies class and pass that into the middleware
via configuration options.
```
config.active_record.database_selector = { delay: 2.seconds }
config.active_record.database_resolver = MyResolver
config.active_record.database_operations = MyResolver::MyCookies
```
Your classes can inherit from the existing classes and reimplment the
methods (or implement more methods) that you need to do the switching.
You only need to implement methods that you want to change. For example
if you wanted to set the session token for the last read from a replica
you would reimplement the `read_from_replica` method in your resolver
class and implement a method that updates a new timestamp in your
operations class.
In MySQL, the text column size is 65,535 bytes by default (1 GiB in
PostgreSQL). It is sometimes too short when people want to use a text
column, so they sometimes change the text size to mediumtext (16 MiB) or
longtext (4 GiB) by giving the `limit` option.
Unlike MySQL, PostgreSQL doesn't allow the `limit` option for a text
column (raises ERROR: type modifier is not allowed for type "text").
So `limit: 4294967295` (longtext) couldn't be used in Action Text.
I've allowed changing text and blob size without giving the `limit`
option, it prevents that migration failure on PostgreSQL.
This PR addresses the issue described in #28025. On `dependent: :nullify` strategy only the foreign key of the relation is nullified. However on polymorphic associations the `*_type` column is not nullified leaving the record with a NULL `*_id` but the `*_type` column is present.
This commit adds support for endless ranges, e.g. (1..), that were added
in Ruby 2.6. They're functionally equivalent to explicitly specifying
Float::INFINITY as the end of the range.
Generally followed the pattern for https://github.com/rails/rails/pull/32034
* Removes needless CI configs for 2.4
* Targets 2.5 in rubocop
* Updates existing CHANGELOG entries for fewer merge conflicts
* Removes Hash#slice extension as that's inlined on Ruby 2.5.
* Removes the need for send on define_method in MethodCallAssertions.
Since MySQL 5.7.9, the `innodb_default_row_format` option defines the
default row format for InnoDB tables. The default setting is `DYNAMIC`.
The row format is required for indexing on `varchar(255)` with `utf8mb4`
columns.
As long as using MySQL 5.6, CI won't be passed even if MySQL server
setting is properly configured the same as MySQL 5.7
(`innodb_file_per_table = 1`, `innodb_file_format = 'Barracuda'`, and
`innodb_large_prefix = 1`) since InnoDB table is created as the row
format `COMPACT` by default on MySQL 5.6, therefore indexing on string
with `utf8mb4` columns aren't succeeded.
Making `ROW_FORMAT=DYNAMIC` create table option by default for legacy
MySQL version would mitigate the indexing issue on the user side, and it
makes CI would be passed on MySQL 5.6 which is configured properly.
