c9e4c848 has one performance optimization for `aggregate_alias` to early
returning by `aggregate_alias.match?(/\A\w+\z/)`, but it is caused a
regression that failing deduplication for non word chars #36867.
I've quited the optimization and add a test to prevent a future
regression.
Fixes#36867.
GROUP BY with virtual count attribute is invalid for almost all
databases, but it is valid for PostgreSQL, and it had worked until Rails
5.2.2, so it is a regression for Rails 5.2.3 (caused by 311f001).
I can't find perfectly solution for fixing this for now, but I would not
like to break existing apps, so I decided to allow referencing virtual
count attribute in ORDER BY clause when GROUP BY aggrigation (it partly
revert the effect of 311f001) to fix the regression #36022.
Fixes#36022.
```ruby
$ bundle exec rake test_postgresql
... snip ...
Failure:
CalculationsTest#test_pluck_columns_with_same_name [/home/yahonda/git/rails/activerecord/test/cases/calculations_test.rb:842]:
--- expected
+++ actual
@@ -1 +1 @@
-[["The First Topic", "The Second Topic of the day"], ["The Third Topic of the day", "The Fourth Topic of the day"]]
+[["The Third Topic of the day", "The Fourth Topic of the day"], ["The First Topic", "The Second Topic of the day"]]
```
This follows up ebc09ed9ad9a04338138739226a1a92c7a2707ee.
We've still experienced a regression for `size` (`count(:all)`) with
eager loading and explicit select and order when upgrading Rails to 5.1.
In that case, the eager loading enforces `distinct` to subselect but
still keep the custom select, it would cause the ORDER BY with DISTINCT
issue.
```
% ARCONN=postgresql bundle exec ruby -w -Itest test/cases/relations_test.rb -n test_size_with_eager_loading_and_custom_select_and_order
Using postgresql
Run options: -n test_size_with_eager_loading_and_custom_select_and_order --seed 8356
# Running:
E
Error:
RelationTest#test_size_with_eager_loading_and_custom_select_and_order:
ActiveRecord::StatementInvalid: PG::InvalidColumnReference: ERROR: for SELECT DISTINCT, ORDER BY expressions must appear in select list
LINE 1: ..." ON "comments"."post_id" = "posts"."id" ORDER BY comments.i...
^
```
As another problem on `distinct` is enforced, the result of `count`
becomes fewer than expected if `select` is given explicitly.
e.g.
```ruby
Post.select(:type).count
# => 11
Post.select(:type).distinct.count
# => 3
```
As long as `distinct` is enforced, we need to care to keep the result of
`count`.
This fixes both the `count` with eager loading problems.
When using `select` with `'DISTINCT( ... )'` if you use method `size` on a non loaded relation it overrides the column selected by passing `:all` so it returns different value than count.
This fixes#35214
Currently custom attributes are always qualified by the table name in
the generated SQL wrongly even if the table doesn't have the named
column, it would cause an invalid SQL error.
Custom attributes should only be qualified if the table has the same
named column.
Currently several queries cannot return correct result due to incorrect
`RangeError` handling.
First example:
```ruby
assert_equal true, Topic.where(id: [1, 9223372036854775808]).exists?
assert_equal true, Topic.where.not(id: 9223372036854775808).exists?
```
The first example is obviously to be true, but currently it returns
false.
Second example:
```ruby
assert_equal topics(:first), Topic.where(id: 1..9223372036854775808).find(1)
```
The second example also should return the object, but currently it
raises `RecordNotFound`.
It can be seen from the examples, the queries including large number
assuming empty result is not always correct.
Therefore, This change handles `RangeError` to generate executable SQL
instead of raising `RangeError` to users to always return correct
result. By this change, it is no longer raised `RangeError` to users.
This reverts 27c6c07 since `arel_attr.to_s` is not right way to avoid
the type error.
That to_s returns `"#<struct Arel::Attributes::Attribute ...>"`, there
is no reason to match the regex to the inspect form.
And also, the regex path is not covered by our test cases. I've tweaked
the regex for redundant part and added assertions for the regex path.
If `eager_loading` is true, `apply_join_dependency` force applies
LIMIT/OFFSET before JOINs by `limited_ids_for` to keep parent records
count. But for aggregation queries, LIMIT/OFFSET should be applied after
aggregations the same as SQL semantics.
And also, we could not replace SELECT list by `limited_ids_for` when a
query has a GROUP BY clause. It had never been worked since it will
causes generating invalid SQL for MySQL, PostgreSQL, and probably most
backends.
```
% ARCONN=postgresql be ruby -w -Itest test/cases/calculations_test.rb -n test_group_by_with_limit
Using postgresql
Run options: -n test_group_by_with_limit --seed 20925
# Running:
E
Error:
CalculationsTest#test_group_by_with_limit:
ActiveRecord::StatementInvalid: PG::GroupingError: ERROR: column "posts.id" must appear in the GROUP BY clause or be used in an aggregate function
LINE 1: SELECT DISTINCT "posts"."id", "posts"."type" AS alias_0 FRO... ^
: SELECT DISTINCT "posts"."id", "posts"."type" AS alias_0 FROM "posts" LEFT OUTER JOIN "comments" ON "comments"."post_id" = "posts"."id" GROUP BY "posts"."type" ORDER BY "posts"."type" ASC LIMIT $1
```
Fixes#8103.
Closes#27249.
This is a regression caused by 6beb4de.
In PostgreSQL, ORDER BY expressions must appear in SELECT list when
using DISTINCT.
When using `count(:all)` with eager loading, Active Record enforces
DISTINCT to count the driving table records only. 6beb4de was caused the
regression because `count(:all)` with DISTINCT path no longer removes
ORDER BY.
We need to ignore ORDER BY when DISTINCT is enforced, otherwise not
always generated valid SQL for PostgreSQL.
Fixes#31783.
Currently `count(:all)` with `distinct` doesn't work correctly because
SELECT list is always replaced to `*` or primary key in that case even
if having custom SELECT list.
And also, PostgreSQL has a limitation that ORDER BY expressions must
appear in select list for SELECT DISTINCT.
Therefore, we should not replace custom SELECT list when using
`count(:all)` with `distinct`.
Closes#31277.
This is the fix for the regression of #29848.
In #29848, I've kept existing select list in the subquery for the count
if ORDER BY is given. But it had accidentally affect to GROUP BY
queries also. It should keep the previous behavior in that case.
Fixes#30886.
`to_i` was added for SQLite3 adapter which did not handle number
but sqlite3 gem already supports it then `to_i` is unnecessary.
else condition is kept for adapters which return string,
i.e. mysql(not mysql2) and sqlserver.
Renamed `test_cache_does_not_wrap_string_results_in_arrays`
to `test_cache_does_not_wrap_results_in_arrays` to explain the
current behavior. most of adapters return integer, not only string.
* Refer these commits:
"future proofing the sqlite3 adapter code"
beda2d43d6
"Refactor calculation test to remove unneeded SQLite special case."
47d568ed3f
"no need to to_i, sqlite does that for us"
6cf44a1bd6
Since #26972, `ORDER BY` is kept if `LIMIT` is presented for
performance. But in most SQL servers (e.g. PostgreSQL, SQL Server, etc),
`ORDER BY` expressions must appear in select list for `SELECT DISTINCT`.
We should not replace existing select list in that case.
The `find_each`, `find_in_batches` and `in_batches` APIs usually operate
on large numbers of records, where it's preferable not to load them all
into memory at once.
If the query cache is enabled, it will hold onto the query results until
the end of the execution context (request/job), which means the memory
used is still proportional to the total number of records. These queries
are typically not repeated, so the query cache isn't desirable here.
Without this, test fails because the load schema when pluck is executed.
Steps to reproduce:
```
bin/test -a postgresql -w --seed 61689 test/cases/*test.rb -n "/^(?:InheritanceComputeTypeTest#(?:test_inheritance_new_with_subclass_as_default)|CalculationsTest#(?:test_pluck_loaded_relation))$/"
# Running:
.F
Failure:
CalculationsTest#test_pluck_loaded_relation [/home/yaginuma/program/rails/master_y_yagi/rails/activerecord/test/cases/calculations_test.rb:722]:
1 instead of 0 queries were executed.
Queries:
SELECT c.relname FROM pg_class c LEFT JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = ANY (current_schemas(false)) AND c.relname = 'companies' AND c.relkind IN ('r','v','m').
Expected: 0
Actual: 1
bin/test test/cases/calculations_test.rb:7
```
`select`, `count`, and `sum` in `Relation` are also `Enumerable` method
that can be passed block. `select` with block already doesn't take
arguments since 4fc3366. This is follow up of that.
