With the introduction of composite primary keys, a common usecase is querying for records with tuples representing the composite key. This change introduces new syntax to the where clause that allows specifying an array of columns mapped to a list of corresponding tuples. It converts this to an OR-joined list of separate queries, similar to previous implementations that rely on grouping queries.
In https://github.com/rails/rails/pull/46690 the `db_warnings_action` and `db_warnings_ignore` configs where added. The
`db_warnings_ignore` can take a list of warning messages to match against.
At GitHub we have a subscriber that that does something like this but also filters out error codes. There might also be
other applications that filter via error codes and this could be something they can use instead of just the explicit
messages.
This also refactors the adapter tests in order for mysql2 and postgresql adapters to share the same helper when setting
the db_warnings_action and db_warnings_ignore configs
By default, exclude constraints in PostgreSQL are checked after each statement.
This works for most use cases, but becomes a major limitation when replacing
records with overlapping ranges by using multiple statements.
```ruby
exclusion_constraint :users, "daterange(valid_from, valid_to) WITH &&", deferrable: :immediate
```
Passing `deferrable: :immediate` checks constraint after each statement,
but allows manually deferring the check using `SET CONSTRAINTS ALL DEFERRED`
within a transaction. This will cause the excludes to be checked after the transaction.
It's also possible to change the default behavior from an immediate check
(after the statement), to a deferred check (after the transaction):
```ruby
exclusion_constraint :users, "daterange(valid_from, valid_to) WITH &&", deferrable: :deferred
```
*Hiroyuki Ishii*
```ruby
add_unique_key :sections, [:position], deferrable: :deferred, name: "unique_section_position"
remove_unique_key :sections, name: "unique_section_position"
```
See PostgreSQL's [Unique Constraints](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-UNIQUE-CONSTRAINTS) documentation for more on unique constraints.
By default, unique constraints in PostgreSQL are checked after each statement.
This works for most use cases, but becomes a major limitation when replacing
records with unique column by using multiple statements.
An example of swapping unique columns between records.
```ruby
old_item = Item.create!(position: 1)
new_item = Item.create!(position: 2)
Item.transaction do
old_item.update!(position: 2)
new_item.update!(position: 1)
end
```
Using the default behavior, the transaction would fail when executing the
first `UPDATE` statement.
By passing the `:deferrable` option to the `add_unique_key` statement in
migrations, it's possible to defer this check.
```ruby
add_unique_key :items, [:position], deferrable: :immediate
```
Passing `deferrable: :immediate` does not change the behaviour of the previous example,
but allows manually deferring the check using `SET CONSTRAINTS ALL DEFERRED` within a transaction.
This will cause the unique constraints to be checked after the transaction.
It's also possible to adjust the default behavior from an immediate
check (after the statement), to a deferred check (after the transaction):
```ruby
add_unique_key :items, [:position], deferrable: :deferred
```
PostgreSQL allows users to create a unique constraints on top of the unique
index that cannot be deferred. In this case, even if users creates deferrable
unique constraint, the existing unique index does not allow users to violate uniqueness
within the transaction. If you want to change existing unique index to deferrable,
you need execute `remove_index` before creating deferrable unique constraints.
*Hiroyuki Ishii*
Now that we support a way to register custom configurations we need to
allow applications to find those configurations. This change adds a
`config_key` option to `configs_for` to find db configs where the
configuration_hash contains a particular key.
I have also removed the deprecation for `include_replicas` while I was
in here to make the method signature cleaner. I've updated the upgrade
guide with the removal.
Previously, applications could only have two types of database
configuration objects, `HashConfig` and `UrlConfig`. This meant that if
you wanted your config to implement custom methods you had to monkey
patch `DatabaseConfigurations` to take a custom class into account. This
PR allows applications to register a custom db_config handler so that
custom configs can respond to needed methods. This is especially useful
for tools like Vitess where we may want to indicate it's sharded, but
not give Rails direct access to that knowledge.
Using the following database.yml as an example:
```yaml
development:
primary:
database: my_db
animals:
database: my_animals_db
vitess:
sharded: 1
```
We can register a custom handler that will generate `VitessConfig`
objects instead of a `HashConfig` object in an initializer:
```ruby
ActiveRecord::DatabaseConfigurations.register_db_config_handler do |env_name, name, url, config|
next unless config.key?(:vitess)
VitessConfig.new(env_name, name, config)
end
```
and create the `VitessConfig` class:
```ruby
class VitessConfig < ActiveRecord::DatabaseConfigurations::UrlConfig
def sharded?
vitess_config.fetch("sharded", false)
end
private
def vitess_config
configuration_hash.fetch(:vitess_config)
end
end
```
Now when the application is booted, the config with the `vitess` key
will generate a `VitessConfig` object where all others will generate a
`HashConfig`.
Things to keep in mind:
1) It is recommended but not required that these custom configs inherit
from Rails so you don't need to reimplement all the existing methods.
2) Applications must implement the configuration in which their config
should be used, otherwise first config wins (so all their configs
will be the custom one.)
3) The procs must support 4 arguments to accommodate `UrlConfig`. I am
thinking of deprecating this and forcing the URL parsing to happen in
the `UrlConfig` directly.
4) There is one tiny behavior change where when we have a nil url key in
the config hash we no longer merge it back into the configuration hash.
We also end up with a `HashConfig` instead of a `UrlConfig`. I think
this is fine because a `nil` URL is...useless.
Co-authored-by: John Crepezzi <john.crepezzi@gmail.com>
Add support for including non-key columns in
btree indexes for PostgreSQL with the INCLUDE
parameter.
Example:
def change
add_index :users, :email, include: [:id, :created_at]
end
Will result in:
CREATE INDEX index_users_on_email USING btree (email) INCLUDE (id,
created_at)
The INCLUDE parameter is described in the PostgreSQL docs:
https://www.postgresql.org/docs/current/sql-createindex.html
One particularly annoying thing with YAMLColumn type restriction
is that it is only checked on load.
Which means if your code insert data with unsupported types, the
insert will work, but now you'll be unable to read the record, which
makes it hard to fix etc.
That's the reason why I implemented `YAML.safe_dump` (https://github.com/ruby/psych/pull/495).
It applies exactly the same restrictions than `safe_load`, which means
if you attempt to store non-permitted fields, it will fail on insertion
and not on further reads, so you won't create an invalid record in your
database.
It can sometimes happen that `sql` is encoded in UTF-8 but contains
some invalid binary data of some sort.
When this happens `strip` end up raising an EncodingError.
Overall I think this strip is quite wasteful, so we might as well
just skip it.
For databases and adapters which support them (currently PostgreSQL
and MySQL), options can be passed to `explain` to provide more
detailed query plan analysis.
`ActiveRecord.db_warnings_action` can be used to configure the
action to take when a query produces a warning. The warning can be
logged, raised, or trigger custom behaviour provided via a proc.
`ActiveRecord.db_warnings_ignore` allows applications to set an
allowlist of SQL warnings that should always be ignored, regardless
of the configured action.
Co-authored-by: Paarth Madan <paarth.madan@shopify.com>
This patch tries to solve Heroku's new [PostgreSQL extension policy](https://devcenter.heroku.com/changelog-items/2446)
while keeping the migration and schema code idiomatic.
PostgreSQL adapter method `enable_extension` now allows to add an schema in its name.
The extension must be installed on another schema.
Usage:
`enable_extension('other_schema.hstore')`
The `enable_extension` can work with `schema` only if the given schema
already exists in the database.
`ActiveRecord::Base::normalizes` declares a normalization for one or
more attributes. The normalization is applied when the attribute is
assigned or updated, and the normalized value will be persisted to the
database. The normalization is also applied to the corresponding
keyword argument of finder methods. This allows a record to be created
and later queried using unnormalized values. For example:
```ruby
class User < ActiveRecord::Base
normalizes :email, with: -> email { email.strip.downcase }
end
user = User.create(email: " CRUISE-CONTROL@EXAMPLE.COM\n")
user.email # => "cruise-control@example.com"
user = User.find_by(email: "\tCRUISE-CONTROL@EXAMPLE.COM ")
user.email # => "cruise-control@example.com"
user.email_before_type_cast # => "cruise-control@example.com"
User.exists?(email: "\tCRUISE-CONTROL@EXAMPLE.COM ") # => true
User.exists?(["email = ?", "\tCRUISE-CONTROL@EXAMPLE.COM "]) # => false
```
As of https://github.com/rails/rails/pull/46525, the behaviour around
before_committed! callbacks has changed: callbacks are run on every
enrolled record in a transaction, even multiple copies of the same record.
This is a significant change that apps should be able to opt into in order
to avoid unexpected issues.
Currently if you do this:
```ruby
config.active_record.query_log_tags = [:namespaced_controller]
```
A request that's processed by the `NameSpaced::UsersController` will log as `namespaced_controller='NameSpaced%3A%3AUsersController'`.
By contrast if you set the tag to `:controller` it would log as `controller='user'`, much nicer.
This PR makes the `:namespaced_controller` formatting more similar to `:controller` - it will now log as `namespaced_controller='name_spaced/users'`.
* Use storage/ instead of db/ for sqlite3 db files
db/ should be for configuration only, not data. This will make it easier to mount a single volume into a container for testing, development, and even sqlite3 in production.
In case when an index is present, using `lower()` prevents from using
the index.
The index is typically present for columns with uniqueness, and
`lower()` is added for `validates_uniqueness_of ..., case_sensitive: false`.
However, if the index is defined with `lower()`, the query without
`lower()` wouldn't use the index either.
Setup:
```
CREATE EXTENSION citext;
CREATE TABLE citexts (cival citext);
INSERT INTO citexts (SELECT MD5(random()::text) FROM generate_series(1,1000000));
```
Without index:
```
EXPLAIN ANALYZE SELECT * from citexts WHERE cival = 'f00';
Gather (cost=1000.00..14542.43 rows=1 width=33) (actual time=165.923..169.065 rows=0 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Parallel Seq Scan on citexts (cost=0.00..13542.33 rows=1 width=33) (actual time=158.218..158.218 rows=0 loops=3)
Filter: (cival = 'f00'::citext)
Rows Removed by Filter: 333333
Planning Time: 0.070 ms
Execution Time: 169.089 ms
Time: 169.466 ms
EXPLAIN ANALYZE SELECT * from citexts WHERE lower(cival) = lower('f00');
Gather (cost=1000.00..16084.00 rows=5000 width=33) (actual time=166.896..169.881 rows=0 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Parallel Seq Scan on citexts (cost=0.00..14584.00 rows=2083 width=33) (actual time=157.348..157.349 rows=0 loops=3)
Filter: (lower((cival)::text) = 'f00'::text)
Rows Removed by Filter: 333333
Planning Time: 0.084 ms
Execution Time: 169.905 ms
Time: 170.338 ms
```
With index:
```
CREATE INDEX val_citexts ON citexts (cival);
EXPLAIN ANALYZE SELECT * from citexts WHERE cival = 'f00';
Index Only Scan using val_citexts on citexts (cost=0.42..4.44 rows=1 width=33) (actual time=0.051..0.052 rows=0 loops=1)
Index Cond: (cival = 'f00'::citext)
Heap Fetches: 0
Planning Time: 0.118 ms
Execution Time: 0.082 ms
Time: 0.616 ms
EXPLAIN ANALYZE SELECT * from citexts WHERE lower(cival) = lower('f00');
Gather (cost=1000.00..16084.00 rows=5000 width=33) (actual time=167.029..170.401 rows=0 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Parallel Seq Scan on citexts (cost=0.00..14584.00 rows=2083 width=33) (actual time=157.180..157.181 rows=0 loops=3)
Filter: (lower((cival)::text) = 'f00'::text)
Rows Removed by Filter: 333333
Planning Time: 0.132 ms
Execution Time: 170.427 ms
Time: 170.946 ms
DROP INDEX val_citexts;
```
With an index with `lower()` has a reverse effect, a query with
`lower()` performs better:
```
CREATE INDEX val_citexts ON citexts (lower(cival));
EXPLAIN ANALYZE SELECT * from citexts WHERE cival = 'f00';
Gather (cost=1000.00..14542.43 rows=1 width=33) (actual time=174.138..177.311 rows=0 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Parallel Seq Scan on citexts (cost=0.00..13542.33 rows=1 width=33) (actual time=165.983..165.984 rows=0 loops=3)
Filter: (cival = 'f00'::citext)
Rows Removed by Filter: 333333
Planning Time: 0.080 ms
Execution Time: 177.333 ms
Time: 177.701 ms
EXPLAIN ANALYZE SELECT * from citexts WHERE lower(cival) = lower('f00');
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on citexts (cost=187.18..7809.06 rows=5000 width=33) (actual time=0.021..0.022 rows=0 loops=1)
Recheck Cond: (lower((cival)::text) = 'f00'::text)
-> Bitmap Index Scan on lower_val_on_citexts (cost=0.00..185.93 rows=5000 width=0) (actual time=0.018..0.018 rows=0 loops=1)
Index Cond: (lower((cival)::text) = 'f00'::text)
Planning Time: 0.102 ms
Execution Time: 0.048 ms
(6 rows)
Time: 0.491 ms
```
This enables subclasses to sync database timezone changes without overriding `#raw_execute`,
which removes the need to redefine `#raw_execute` in the `Mysql2Adapter` and other
adapters subclassing `AbstractMysqlAdapter`.
Co-authored-by: Paarth Madan <paarth.madan@shopify.com>
composed_of values should be automatically frozen by Active Record.
This worked correctly when assigning a new value object via the writer,
but objects instantiated based on database columns were NOT frozen. The
fix consists of calling #dup and then #freeze on the cached value
object when it's added to the aggregation cache in #reader_method.
Additionally, values assigned via the accessor are duplicated and then
frozen to avoid caller confusion.
Previously, assignment would succeed but silently not write to the
database.
The changes to counter_cache are necessary because incrementing the
counter cache for a column calls []=. I investigated an approach to use
_write_attribute instead, however counter caches are expected to resolve
attribute aliases so write_attribute/[]= seems more correct.
Similarly, []= was replaced with _write_attribute in merge_target_lists
to skip the overriden []= and the primary key check. attribute_names
will already return custom primary keys so the primary_key check in
write_attribute is not needed.
Co-authored-by: Alex Ghiculescu <alex@tanda.co>
