It's possible since Rails 6 (3ea2857943dc294d7809930b4cc5b318b9c39577) to let the framework create Event objects, but the guides and docs weren't updated to lead with this example.
Manually instantiating an Event doesn't record CPU time and allocations, I've seen it more than once that people copy-pasting the example code get confused about these stats returning 0. The tests here show that - just like the apps I've worked on - the old pattern keeps getting copy-pasted.
- An oversight of #48615 is that it changes the `Rails.logger` to be
a broadcast logger after the app is booted. Anything referencing
`Rails.logger` during the boot process will get a simple logger and
ultimately resulting in logs not being broadcasted.
For example `ActionController::Base.logger.info("abc")` would
just output logs in the `development.log` file, not on STDOUT.
----
The only solution I could think of is to create a BroadcastLogger
earlier at boot, and add logger to that broadcast when needed (instead
of modiyfing the `Rails.logger` variable).
- ## Context
While working on https://github.com/rails/rails/pull/44695, I
realised that Broadcasting was still a private API, although it’s
commonly used. Rafael mentioned that making it public would require
some refactor because of the original implementation which was hard
to understand and maintain.
### Changing how broadcasting works:
Broadcasting in a nutshell worked by “transforming” an existing
logger into a broadcasted one.
The logger would then be responsible to log and format its own
messages as well as passing the message along to other logger it
broadcasts to.
The problem with this approach was the following:
- Heavy use of metaprogramming.
- Accessing the loggers in the broadcast wasn’t possible.
Removing a logger from the broadcast either.
- More importantly, modifying the main logger (the one that broadcasts
logs to the others) wasn’t possible and the main source of
misunderstanding.
```ruby
logger = Logger.new(STDOUT)
stderr_logger = Logger.new(STDER))
logger.extend(AS::Logger.broadcast(stderr_logger))
logger.level = DEBUG # This modifies the level on all other loggers
logger.formatter = … # Modified the formatter on all other loggers
```
To keep the contract unchanged on what Rails.logger returns, the new
BroadcastLogger class implement duck typing with all methods
that has the vanilla Ruby Logger class.
It's a simple and boring PORO that keeps an array of all the loggers
that are part of the broadcast and iterate over whenever a log is
sent.
Now, users can access all loggers inside the broadcast and modify
them on the fly. They can also remove any logger from the broadcast
at any time.
```ruby
# Before
stdout_logger = Logger.new(STDOUT)
stderr_logger = Logger.new(STDER)
file_logger = Logger.new(“development.log”)
stdout_logger.extend(AS::Logger.broadcast(stderr_logger))
stdout_logger.extend(AS::Logger.broadcast(file_logger))
# After
broadcast = BroadcastLogger.new(stdout_logger, stderr_logger, file_logger)
```
I also think that now, it should be more clear for users that the
broadcast sole job is to pass everything to the whole loggers in
the broadcast. So there should be no surprise that all loggers in
the broadcast get their level modified when they call
`broadcast.level = DEBUG` .
It’s also easier to wrap your head around more complex setup such
as broadcasting logs to another broadcast:
`broadcast.broadcast_to(stdout_logger, other_broadcast)`
The `concurrent` require was [added][1] previously because a
`Concurrent::Map` was added to hold all of the log levels by `Thread` id.
However, the `Map` was later [removed][2] by storing the log levels in the
`Thread` locals (and later in `IsolatedExecutionState`) instead. The new
implementation additionally removed the `cattr_accessor` (removing the
need to require the "attribute_accessors" core_ext) and also replaced
the [usage][3] of `Fiber` with `Thread` (so the require for `Fiber` is also no
longer necessary).
Since `Concurrent`, `Fiber`, and `cattr_accessor` are no longer used here, we
can remove the requires. Since `Logger` is used here (but was previously
required by `concurrent`), a require was added for it.
[1]: 629efb605728b31ad9644f6f0acaf3760b641a29
[2]: 2379bc5d2a7d9580f270eebfde87d9f94b3da6c9
[3]: 56ec504db6c130d448ffc1d68c9fdd95fdfc1130
Previously, #overlap? would incorrectly return true when one of the
ranges is effectively "empty":
```ruby
(2...2).overlap? 1..2 # => true
(1..2).overlap? 2...2 # => true
```
This is fixed in the Ruby 3.3 implementation of Range#overlap?, so this
commit fixes it for Ruby < 3.3 as well.
The tests added are from the Ruby repository and the implementation is
effectively a Ruby version of the fix in C.
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Shouichi Kamiya <shouichi.kamiya@gmail.com>
This commit uses Ruby 3.3 `Range#overlap?` that has been added to Ruby via https://github.com/ruby/ruby/pull/8242 .
Rails 7.1 renames `Range#overlaps?` to `Range#overlap?` via https://github.com/rails/rails/pull/48565 ,
This commit is not feasible to backport because there is no `Range#overlap?` in Rails 7.0.z
This commit addresses the CI faiilure at https://buildkite.com/rails/rails/builds/99745#018a9ea8-82f0-40a6-90c3-cdaa6dabebab/1092-1095
because without this commit, it shows `warning: method redefined; discarding old overlap?`.
```ruby
$ ruby -v
ruby 3.3.0dev (2023-09-16T05:57:19Z master e9b503f1bb) [x86_64-linux]
$ RAILS_STRICT_WARNING=true bundle exec ruby -w -Itest test/core_ext/range_ext_test.rb
/home/yahonda/src/github.com/rails/rails/activesupport/lib/active_support/core_ext/range/overlap.rb:7: warning: method redefined; discarding old overlap?
Running 46 tests in a single process (parallelization threshold is 50)
Run options: --seed 583
\# Running:
..............................................
Finished in 0.011672s, 3940.9670 runs/s, 4883.3722 assertions/s.
46 runs, 57 assertions, 0 failures, 0 errors, 0 skips
```
When `use_message_serializer_for_metadata` is false, the message
format should be enveloped in the same way it was in Rails 7.0
to make sure we don't have inconsistent formats.
Followup: https://github.com/rails/rails/pull/49108
DescendantsTracker need to work whether the `Class#descendants` core
ext is loaded or not. I missed that in the previous PR.
Either we are on a modern Ruby with `Class#subclass`, in which case
`DescendantsTracker#subclass` isn't defined, so we don't need the
filtering module.
Or we're on an old Ruby, and `DescendantsTracker.subclass` already does
the filtering.
`Kernel#caller` has linear performance based on how deep the
stack is. While this is a development only feature, it can end
up being quite slow.
Ruby 3.2 introduced `Thread.each_caller_location`, which lazily
yield `Backtrace::Location` objects.
Ref: https://bugs.ruby-lang.org/issues/16663
This is perfect for this use case as we are searching for the
closest frame that matches a pattern, saving us from collecting
the entire backtrace.
XmlMini is currently being tested against libxml-ruby 4.0.0[^1], but the
doc comment for XmlMini says:
```
To use the much faster libxml parser:
gem 'libxml-ruby', '=0.9.7'
```
This comment seems to be very much out of date and could mislead users.
Fix by removing the version specifier so that the documentation simply
recommends:
```
To use the much faster libxml parser:
gem 'libxml-ruby'
XmlMini.backend = 'LibXML'
```
[^1]: 621bc68548/Gemfile.lock (L310)
When we're editing the contents of encrypted files, we should use the
`Tempfile` class because it creates temporary files with restrictive
permissions. This prevents other users on the same system from reading
the contents of those files while the user is editing them.
[CVE-2023-38037]
Since Rails 6.1 (via c4845aa7791839fcdf723dc77e3df258e7274496), it has
been possible to specify `coder: nil` to allow the store to handle cache
entries directly.
This commit adds documentation and regression tests for the behavior.
In the case that one single test file can't run with more than 1
parallel workers, but the base class has parallelization enabled, we
should still allow the user to set the number of workers to 1.
This adds a cache optimization such that expired and version-mismatched
cache entries can be detected without deserializing their values. This
optimization is enabled when using cache format version >= 7.1 or a
custom serializer.
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
`Rails.cache.delete('key')` is supposed to return `true` if an entry
exists and `false` otherwise. This is how most stores behave.
However, the `RedisCacheStore` would return a `1` when deleting an entry
that does exist and a `0` otherwise.
As `0` is truthy this is unexpected behaviour.
`RedisCacheStore` now returns true if the entry exists and false
otherwise, making it consistent with the other cache stores.
Similarly the `FileCacheStore` now returns `false` instead of `nil` if
the entry doesn't exist.
A test is added to make sure this behaviour applies to all stores.
The documentation for `delete` has been updated to make the behaviour
explicit.
This commit adds support for replacing the compressor used for
serialized cache entries. Custom compressors must respond to `deflate`
and `inflate`. For example:
```ruby
module MyCompressor
def self.deflate(string)
# compression logic...
end
def self.inflate(compressed)
# decompression logic...
end
end
config.cache_store = :redis_cache_store, { compressor: MyCompressor }
```
As part of this work, cache stores now also support a `:serializer`
option. Similar to the `:coder` option, serializers must respond to
`dump` and `load`. However, serializers are only responsible for
serializing a cached value, whereas coders are responsible for
serializing the entire `ActiveSupport::Cache::Entry` instance.
Additionally, the output from serializers can be automatically
compressed, whereas coders are responsible for their own compression.
Specifying a serializer instead of a coder also enables performance
optimizations, including the bare string optimization introduced by cache
format version 7.1.
`ActiveSupport::Cache::UNIVERSAL_OPTIONS` already defines the list of
base options, and it includes option aliases such as `:expire_in` and
`:expired_in`. Thus, using `UNIVERSAL_OPTIONS` allows `RedisCacheStore`
to support these aliases.
Follow-up to #48449.
Since #48449 changed the list of accepted cache format versions back to
just `6.1`, `7.0`, and `7.1`, we can raise a more specific error.
Using [] or the dynamic accessors don't result in the same value because
`[]` is delegated to `config` (the decrypted deserialized YAML), whereas
`[]=` and the dynamic accessors are delegated to `options`, an
ActiveSupport::OrderedOptions instance.
When development tools try to load Rails components, they sometimes end up loading files that will error out since a dependency is missing. In these cases, the tooling can catch the error and change its behaviour.
However, since the warning is printed directly to `$stderr`, the tooling cannot catch and suppress it easily, which ends up causing noise in the output of the tool.
This change makes Rails print these warnings using `Kernel#warn` instead, which can be suppressed by the tooling.
To use a binary-encoded string as a byte buffer, appended strings should
be force-encoded as binary. Otherwise, appending a non-ASCII-only
string will raise `Encoding::CompatibilityError`.
Fixes#48748.
This help treating caches entries as expandable.
Because Marshal will hapily serialize almost everything, It's not uncommon to
inadvertently cache a class that's not particularly stable, and cause deserialization
errors on rollout when the implementation changes.
E.g. https://github.com/sporkmonger/addressable/pull/508
With this change, in case of such event, the hit rate will suffer for a
bit, but the application won't return 500s responses.
`activesupport/lib/active_support/cache.rb` is already rather long, and
having a separate file allows subclasses of `Cache::Entry` to be defined
without first requiring `activesupport/lib/active_support/cache.rb`.
`NullCoder` is functionally equivalent to the `:passthrough` coder.
Additionally, `NullCoder` was added in c4845aa7791839fcdf723dc77e3df258e7274496
to serve as the default coder for `MemCacheStore`, but `MemCacheStore`
now uses `:passthrough` as its (legacy) default coder.
Running gsub! 5 times with string arguments seems to be faster than
running it once with a regex and Hash.
When there are matches to the regex (there are characters to escape)
this is faster in part because CRuby will allocate a new match object
and string as a key to lookup in the map hash provided. It's possible
that could be optimized upstream, but at the moment this avoids those
allocations.
Surprisingly (at least to me) this is still much faster when there is no
replacement needed: in my test ~3x faster on a short ~200 byte string,
and ~5x faster on a pre-escaped ~600k twitter.json.
While working on fixing some deprecation warnings introduced when the
6.1 cache_format deprecation was [moved][1] to be on usage, I found that
the MemCacheStore actually has its own `default_coder` method.
This adds the warning to MemCacheStore's `default_coder` method so that
every cache store will warn on using the 6.1 format.
[1]: 088551c802bb4005a667aaa33814cfbb1feb3927
An issue was recently opened with the following error message:
```
.../activesupport-7.0.6/lib/active_support/evented_file_update_checker.rb:120:in `start': undefined method `wait_for_state' for #<Listen::Listener ...>
```
The issue is that the user was using Listen 3.0.8, however the
`wait_for_state` method was [added][1] in Listen 3.3.0
We can make the error message a little better by defining a lower bound
on Listen 3.3.0 (like we do for other optional dependencies):
```
.../bundler/rubygems_integration.rb:280:in `block (2 levels) in replace_gem': can't activate listen (~> 3.3), already activated listen-3.0.8. Make sure all dependencies are added to Gemfile. (Gem::LoadError)
```
There is definitely still room for improvement here, but this should
be much more helpful in figuring out that the issue is a bad Listen
version and not a bug in Rails.
[1]: 12b4fc54a9
Previously jsonify would call `.as_json` for Integer, nil, true, and
false, even though those types are considered "JSON-ready". Technically
a user could have overridden `.as_json` for these types but I can't
imagine a use case and I don't think we should support that.
I left the same behaviour of calling `.as_json` for generic "Numeric" as
that can have user subclasses where one may have implemented as_json.
This behaviour is also used for Float (which coerces
NaN/Infinity/-Infinity into nil).
This also adds Symbol to the list of "JSON-ready" types, to avoid
unnecessarily casting them to strings (possible as we no longer perform
escaping on input). The output of jsonify should never be user visible
before it is passed through JSON.generate, so I don't think this should
be a user facing
change.
This also corrects our handling of Hash to call to_s on all keys,
matching the behaviour of `.as_json` and JSON's requirement that keys
are Strings (Symbols are also permitted as JSON knows to convert them to
a String).
Rails performs additional escaping of strings compared to the JSON gem,
and will escape U+2028, U+2029, <, >, and &.
In JSON, the only places those characters are valid is inside of a
JSON string, so it should be equivalent (and faster) to perform the
escaping on the output.
JSONGemEncoder.encode previously would always perform two passes. First
it would call `.as_json(options)`, but then would perform a second pass
"jsonify" to recursively call `.as_json` (this time without options)
until the data converges into a "JSON-ready" representation.
When options are not given, the second pass should be equivalent to the
first, so we can detect that, and only perform the "jsonify" step.
The only user-visible effect of this should be that we will pass no
options to `as_json` instead of an empty Hash, but implementations of
`as_json` should already be expected to accept that.
A deprecation warning was [added][1] to ensure that applications
manually setting `config.active_support.cache_format_version` to `6.1`
will be aware that they need to migrate. However, if an app is not using
a `config.load_defaults` of `7.0` or greater, this warning will never be
triggered.
This commit moves the deprecation warning to where the `cache_format`
value gets used to cover both cases.
[1]: 2ba3ac29c36f4c6d23b1dd302584f15739ff2aff
Previously, `EncryptedConfiguration` was [updated][1] to use
`InheritableOptions` so that keys could be called like methods. It was
later [updated again][2] to ensure that it behaved both like a `Hash` and
`OrderedOptions`. In this second change, the `InheritableOptions`
instance was accidentally nested with another `InheritableOptions`
instance.
This continued to mostly work as expected because `InheritableOptions`
will fall back to the inner `InheritableOptions` when the outer one
doesn't have a key. However, any methods that try to treat the outer
`InheritableOptions` like it should know about all of its keys will fail
(for example, `#keys`, `#to_h`, `#to_json`, etc.)
This commit fixes the issue by removing the extraneous outer
`InheritableOptions` instance.
[1]: a6a9fed1719a3cfe47eb1566ae4d6034374fd809
[2]: 80585daf2def5bf94854be69f81e24a16ce14c55
These methods are only available when requiring the railties
"rails/test_help" file, which is included by default in the
test_helper.rb for newly generated Rails applications.
Since requiring that file is the only way, and some applications may
have removed it or not used test-unit when generating their application,
I think it's worth calling out explicitly here.
If anyone calls a message encryptor in the console it will
show the secret of the encryptor.
By overriding the `inspect` method to only show the class name we can
avoid accidentally outputting sensitive information.
Before:
```ruby
ActiveSupport::MessageEncryptor.new(secret, cipher: "aes-256-gcm").inspect
"#<ActiveSupport::MessageEncryptor:0x0000000104888038 ... @secret=\"\\xAF\\bFh]LV}q\\nl\\xB2U\\xB3 ... >"
```
After:
```ruby
ActiveSupport::MessageEncryptor.new(secret, cipher: "aes-256-gcm").inspect
"#<ActiveSupport::MessageEncryptor:0x0000000104888038>"
```
If anyone calls `Rails.application.credentials` in the console it will
show the unencrypted contents of the credentials.
By overriding the `inspect` method to only show the class name we can
avoid accidentally outputting sensitive information.
Before:
```ruby
Rails.application.credentials.inspect
"#<ActiveSupport::EncryptedConfiguration:0x000000010d2b38e8 ... @config={:secret=>\"something secret\"} ... @key_file_contents=\"915e4ea054e011022398dc242\" ...>"
```
After:
```ruby
Rails.application.credentials.inspect
"#<ActiveSupport::EncryptedConfiguration:0x000000010d2b38e8>"
```
In #48104, `:message_pack` was added as a supported value for the cache
format version because the format version essentially specifies the
default coder. However, as an API, the format version could potentially
be used to affect other aspects of serialization, such as the default
compression format.
This commit removes `:message_pack` as a supported value for the format
version, and, as a replacement, adds support for specifying
`coder: :message_pack`:
```ruby
# BEFORE
config.active_support.cache_format_version = :message_pack
# AFTER
config.cache_store = :redis_cache_store, { coder: :message_pack }
```
This changed `many?` to yield with `element, *args`. The causes elements that are arrays to be destructured which can lead to defective behaviour. Since the introduction of `element, *args` was for readability purposes, we can just use `*args` instead and get the same behaviour without the defect.
Followup: https://github.com/rails/rails/pull/47354
It does a bit more than just giving you a `.instance` method
it also change the behavior of dup and clone, we don't need
any of that, and `.instance` is deprecated anyway.
Fix: https://github.com/rails/rails/issues/48352
While we should ensure instantiating the store doesn't immediately
attempt to connect, we should eagerly process arguments so that
if they are somehow invalid we fail early during boot rather than at
runtime.
Additionally, since it's common to get pool parameters from environment
variable, we can use `Integer` and `Float` so that string representations
are valid.
Prior to this commit, `MemoryStore::DupCoder` called `Entry#dup_value!`
in both `dump` and `load` to guard against external mutation of cached
values. `Entry#dup_value!` calls `Marshal.dump` then `Marshal.load` to
dup complex objects. However, to prevent external mutation, we only
need to call `Marshal.dump` once for `DupCoder.dump` and `Marshal.load`
once for `DupCoder.load`.
This commit changes `DupCoder` to call `Marshal.dump` and `Marshal.load`
directly instead of relying on `Entry#dup_value!`, thus halving the work
done by `Marshal` when writing and reading complex objects.
__Benchmark__
```ruby
# frozen_string_literal: true
require "benchmark/ips"
require "active_support"
require "active_support/cache"
LARGE_OBJECT = 100.times.map { "x" * 100 }
LARGE_STRING = LARGE_OBJECT.join
cache = ActiveSupport::Cache.lookup_store(:memory_store)
Benchmark.ips do |x|
x.report("large string") do
cache.write("x", LARGE_STRING)
cache.read("x")
end
x.report("large object") do
cache.write("x", LARGE_OBJECT)
cache.read("x")
end
end
```
__Before__
```
Warming up --------------------------------------
large string 2.667k i/100ms
large object 332.000 i/100ms
Calculating -------------------------------------
large string 26.539k (± 1.8%) i/s - 133.350k in 5.026373s
large object 3.336k (± 0.9%) i/s - 16.932k in 5.076458s
```
__After__
```
Warming up --------------------------------------
large string 2.541k i/100ms
large object 715.000 i/100ms
Calculating -------------------------------------
large string 25.117k (± 1.7%) i/s - 127.050k in 5.059945s
large object 7.111k (± 1.2%) i/s - 35.750k in 5.028267s
```
Closes#46403.
Co-authored-by: Breno Gazzola <breno.gazzola@gmail.com>
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
This method previously maintained the `html_safe?` property of a string on the return
value. Because this string has been escaped, however, not marking it as `html_safe` causes
entities to be double-escaped.
As an example, take this view snippet:
```html
<p><%= html_escape_once("this & that & the other") %></p>
```
Before this change, that would be double-escaped and render as:
```html
<p>this &amp; that &amp; the other</p>
```
After this change, it renders correctly as:
```html
<p>this & that & the other</p>
```
[Fix#48256]
Most of these are redundant because rdoc handles these itself, but
`titlecase` on `ActiveSupport::Inflector` does not exist so that one is
just incorrect.
Because `ActiveSupport::MessagePack::Serializer::SIGNATURE` includes a
non-ASCII-only byte (`"\xCC"`), it raises `Encoding::CompatibilityError`
when compared with another string that is not encoded with
`Encoding::BINARY` and also includes a non-ASCII-only byte.
To prevent that, this commit changes `AS::MessagePack#signature?` to
directly compare the first two bytes of both strings.
Fixes#48196.
Updated the ActiveSupport::InheritableOptions documentation to exemplify
how to use a parent hash that contains string keys.
parent = { "foo" => true }
child = ActiveSupport::InheritableOptions.new(parent.symbolize_keys)
child.foo # => true
This example is helpful because ActiveSupport::OrderedOptions only deals
with symbol keys and this implementation detail is hidden from the API
docs.
Co-authored-by: Jonathan Hefner <jonathan@hefner.pro>
This commit adds support for `:message_pack` as a serializer for
`MessageEncryptor` and `MessageVerifier`, and, consequently, as an
option for `config.active_support.message_serializer`.
The `:message_pack` serializer is implemented via
`ActiveSupport::Messages::SerializerWithFallback` and can fall back to
deserializing with `AS::JSON`. Additionally, the `:marshal`, `:json`,
and `:json_allow_marshal` serializers can now fall back to deserializing
with `AS::MessagePack`.
This commit also adds support for `:message_pack_allow_marshal` as a
serializer, which can fall back to deserializing with `Marshal` as well
as `AS::JSON`.
Prior to this commit, `config.load_defaults 7.1` would cause old
messages (or messages from older apps) to become unreadable, and
developers were encouraged to manually set
`config.active_support.message_serializer = :json_allow_marshal` in
order to prevent this.
This commit changes the default message serializer set by
`config.load_defaults 7.1` from `:json` to `:json_allow_marshal` so that
upgraded apps can continue to read old messages without additional
configuration.
The intention is to eventually change the default to `:json` (with no
`Marshal` fallback) in Rails 7.2, after some time has passed with apps
generating JSON-serialized messages.
Apps can opt in to JSON-only serialization before Rails 7.2 by manually
setting `config.active_support.message_serializer = :json`.
Fixes#48118.
In #42843 and #42846, several config settings were added to control the
default serializer for `MessageEncryptor` and `MessageVerifier`, and to
provide a migration path from a default `Marshal` serializer to a
default `JSON` serializer:
* `config.active_support.default_message_encryptor_serializer`
* Supports `:marshal`, `:hybrid`, or `:json`.
* `config.active_support.default_message_verifier_serializer`
* Supports `:marshal`, `:hybrid`, or `:json`.
* `config.active_support.fallback_to_marshal_deserialization`
* Affects `:hybrid` for both `MessageEncryptor` and `MessageVerifier`.
* `config.active_support.use_marshal_serialization`
* Affects `:hybrid` for both `MessageEncryptor` and `MessageVerifier`.
This commit unifies those config settings into a single setting,
`config.active_support.message_serializer`, which supports `:marshal`,
`:json_allow_marshal`, and `:json` values. So, for example,
```ruby
config.active_support.default_message_encryptor_serializer = :hybrid
config.active_support.default_message_verifier_serializer = :hybrid
config.active_support.fallback_to_marshal_deserialization = true
config.active_support.use_marshal_serialization = false
```
becomes
```ruby
config.active_support.message_serializer = :json_allow_marshal
```
and
```ruby
config.active_support.default_message_encryptor_serializer = :hybrid
config.active_support.default_message_verifier_serializer = :hybrid
config.active_support.fallback_to_marshal_deserialization = false
config.active_support.use_marshal_serialization = false
```
becomes
```ruby
config.active_support.message_serializer = :json
```
This commit also replaces `ActiveSupport::JsonWithMarshalFallback` with
`ActiveSupport::Messages::SerializerWithFallback`, which implements a
generic mechanism for serializer fallback. The `:marshal` serializer
uses this mechanism too, so
```ruby
config.active_support.default_message_encryptor_serializer = :hybrid
config.active_support.default_message_verifier_serializer = :hybrid
config.active_support.fallback_to_marshal_deserialization = false
config.active_support.use_marshal_serialization = true
```
becomes
```ruby
config.active_support.message_serializer = :marshal
```
Additionally, the logging behavior of `JsonWithMarshalFallback` has been
replaced with notifications which include the names of the intended and
actual serializers, as well as the serialized and deserialized message
data. This provides a more targeted means of tracking serializer
fallback events. It also allows the user to "silence" such events, if
desired, without an additional config setting.
All of these changes make it easier to add migration paths for new
serializers such as `ActiveSupport::MessagePack`.
This updates two instances of custom initializer examples in the docs to
use the standard "library.name" format. Previously, one used just "name"
which can lead to hard to debug "cyclic dependency" errors, and the
other used "name.library" which is not the suggested format.
Follow-up to #48154.
This adds short-circuiting behavior to `delete_multi` when an empty key
list is specified in order to prevent an `ArgumentError` from being
raised, similar to `read_multi`.
This commit addresses a few problems:
1. `read_multi` (and `fetch_multi` and `delete_multi`) logs multiple
keys as if they were a single composite key. For example,
`read_multi("posts/1", "posts/2")` will log "Cache read_multi:
posts/1/posts/2". This can make the log confusing or indecipherable
when keys contain more slashes, such as with view fragments.
2. `write_multi` logs its entire argument as a single composite key.
For example, `write_multi("p1" => post1, "p2" => post2)` will log
"Cache write_multi: p1=#<Post:0x...>/p2=#<Post:0x...>".
3. `MemoryStore#cleanup` logs its instrumentation payload instead of
setting it on the event. For example, when 10 entries are in the
cache, `cleanup` will log "Cache cleanup: size=10" instead of
merging `{ size: 10 }` into the event payload.
Multi-key logging was first added in ca6aba7f30ad9910f17e4c5b39667889d9518794,
then reverted in c4a46fa781d39f1b4607cb1613a1c67fa044ce54 due to being
unwieldy, and then re-added in 2b96d5822bfe407be7589e293f3265c0c7a6726c
(for `write_multi`) and 62023884f76c108127c8966f4d67bb717338dd66 (for
`read_multi`) but without any handling or formatting.
This commit changes the way multi-key operations are logged in order to
prevent these problems. For example, `read_multi("posts/1", "posts/2")`
will now log "Cache read_multi: 2 key(s) specified", and
`write_multi("p1" => post1, "p2" => post2)` will now log "Cache
write_multi: 2 key(s) specified".
Fix: https://github.com/rails/rails/pull/48145
`read_multi`, `write_multi` and `fetch multi` should all
bail out early if somehow called with an empty list.
Co-Authored-By: Joshua Young <djry1999@gmail.com>
This commit introduces a performance optimization for cache entries with
bare string values such as view fragments.
A new `7.1` cache format has been added which includes the optimization,
and the `:message_pack` cache format now includes the optimization as
well. (A new cache format is necessary because, during a rolling
deploy, unupgraded servers must be able to read cache entries from
upgraded servers, which means the optimization cannot be enabled for
existing apps by default.)
New apps will use the `7.1` cache format by default, and existing apps
can enable the format by setting `config.load_defaults 7.1`. Cache
entries written using the `6.1` or `7.0` cache formats can be read when
using the `7.1` format.
**Benchmark**
```ruby
# frozen_string_literal: true
require "benchmark/ips"
serializer_7_0 = ActiveSupport::Cache::SerializerWithFallback[:marshal_7_0]
serializer_7_1 = ActiveSupport::Cache::SerializerWithFallback[:marshal_7_1]
entry = ActiveSupport::Cache::Entry.new(Random.bytes(10_000), version: "123")
Benchmark.ips do |x|
x.report("dump 7.0") do
$dumped_7_0 = serializer_7_0.dump(entry)
end
x.report("dump 7.1") do
$dumped_7_1 = serializer_7_1.dump(entry)
end
x.compare!
end
Benchmark.ips do |x|
x.report("load 7.0") do
serializer_7_0.load($dumped_7_0)
end
x.report("load 7.1") do
serializer_7_1.load($dumped_7_1)
end
x.compare!
end
```
```
Warming up --------------------------------------
dump 7.0 5.482k i/100ms
dump 7.1 10.987k i/100ms
Calculating -------------------------------------
dump 7.0 73.966k (± 6.9%) i/s - 367.294k in 5.005176s
dump 7.1 127.193k (±17.8%) i/s - 615.272k in 5.081387s
Comparison:
dump 7.1: 127192.9 i/s
dump 7.0: 73966.5 i/s - 1.72x (± 0.00) slower
Warming up --------------------------------------
load 7.0 7.425k i/100ms
load 7.1 26.237k i/100ms
Calculating -------------------------------------
load 7.0 85.574k (± 1.7%) i/s - 430.650k in 5.034065s
load 7.1 264.877k (± 1.6%) i/s - 1.338M in 5.052976s
Comparison:
load 7.1: 264876.7 i/s
load 7.0: 85573.7 i/s - 3.10x (± 0.00) slower
```
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
This silences a "ActiveSupport::MessagePack requires the msgpack gem"
warning that can occur when using a non-`:message_pack` serializer and
trying to detect the format of a serialized cache entry.
`silence_warnings` is a blunt instrument, but this `require` is only for
decoding legacy cache entries that were encoded using `:message_pack`,
if any. (i.e. If `:message_pack` is currently being used as a
serializer, then `SerializerWithFallback::[]` should have already loaded
`active_support/message_pack`.) Therefore, it seems less hazardous to
inadvertently silence other warnings that may occur when loading the
file.
Co-authored-by: Alex Ghiculescu <alex@tanda.co>
This commit adds support for `:message_pack` as an option for
`config.active_support.cache_format_version`.
Cache entries written using the `6.1` or `7.0` formats can be read when
using the `:message_pack` format. Additionally, cache entries written
using the `:message_pack` format can now be read when using the `6.1` or
`7.0` format. These behaviors makes it easy to migrate between formats
without invalidating the entire cache.
attach_to was previously [improved][1] to allow defining methods after
attaching instead of having to attach after all methods have been
defined. The docs for Subscriber were [updated][2], but LogSubscriber
docs were not.
This commit copies the attach_to doc changes from Subscriber to
LogSubscriber. In addition, there are other improvements to linking and
applying the monospace formatting that were present in one of Subscriber
or LogSubscriber and are now applied to both. The final change is
updating the description of how flush_all! is used, because its usage
has [changed][3] since this doc was written.
[1]: 34088572270a1dd5a2164b6aa5fc3642cb0479cb
[2]: 25b3f738e4f5b09e4d6a66e1454e697defcdda2c
[3]: 378464a2e47bb849f3351cb8c87366554b7ce74d
`ActiveSupport::MessagePack` is a serializer that integrates with the
`msgpack` gem to serialize a variety of Ruby objects. `AS::MessagePack`
supports several types beyond the base types that `msgpack` supports,
including `Time` and `Range`, as well as Active Support types such as
`AS::TimeWithZone` and `AS::HashWithIndifferentAccess`.
Compared to `JSON` and `Marshal`, `AS::MessagePack` can provide a
performance improvement and message size reduction. For example, when
used with `MessageVerifier`:
```ruby
# frozen_string_literal: true
require "benchmark/ips"
require "active_support/all"
require "active_support/message_pack"
marshal_verifier = ActiveSupport::MessageVerifier.new("secret", serializer: Marshal)
json_verifier = ActiveSupport::MessageVerifier.new("secret", serializer: JSON)
asjson_verifier = ActiveSupport::MessageVerifier.new("secret", serializer: ActiveSupport::JSON)
msgpack_verifier = ActiveSupport::MessageVerifier.new("secret", serializer: ActiveSupport::MessagePack)
ActiveSupport::Messages::Metadata.use_message_serializer_for_metadata = true
expiry = 1.year.from_now
data = { bool: true, num: 123456789, string: "x" * 50 }
Benchmark.ips do |x|
x.report("Marshal") do
marshal_verifier.verify(marshal_verifier.generate(data, expires_at: expiry))
end
x.report("JSON") do
json_verifier.verify(json_verifier.generate(data, expires_at: expiry))
end
x.report("AS::JSON") do
asjson_verifier.verify(asjson_verifier.generate(data, expires_at: expiry))
end
x.report("MessagePack") do
msgpack_verifier.verify(msgpack_verifier.generate(data, expires_at: expiry))
end
x.compare!
end
puts "Marshal size: #{marshal_verifier.generate(data, expires_at: expiry).bytesize}"
puts "JSON size: #{json_verifier.generate(data, expires_at: expiry).bytesize}"
puts "MessagePack size: #{msgpack_verifier.generate(data, expires_at: expiry).bytesize}"
```
```
Warming up --------------------------------------
Marshal 1.206k i/100ms
JSON 1.165k i/100ms
AS::JSON 790.000 i/100ms
MessagePack 1.798k i/100ms
Calculating -------------------------------------
Marshal 11.748k (± 1.3%) i/s - 59.094k in 5.031071s
JSON 11.498k (± 1.4%) i/s - 58.250k in 5.066957s
AS::JSON 7.867k (± 2.4%) i/s - 39.500k in 5.024055s
MessagePack 17.865k (± 0.8%) i/s - 89.900k in 5.032592s
Comparison:
MessagePack: 17864.9 i/s
Marshal: 11747.8 i/s - 1.52x (± 0.00) slower
JSON: 11498.4 i/s - 1.55x (± 0.00) slower
AS::JSON: 7866.9 i/s - 2.27x (± 0.00) slower
Marshal size: 254
JSON size: 234
MessagePack size: 194
```
Additionally, `ActiveSupport::MessagePack::CacheSerializer` is a
serializer that is suitable for use as an `ActiveSupport::Cache` coder.
`AS::MessagePack::CacheSerializer` can serialize `ActiveRecord::Base`
instances, including loaded associations. Like `AS::MessagePack`, it
provides a performance improvement and payload size reduction:
```ruby
# frozen_string_literal: true
require "benchmark/ips"
require "active_support/message_pack"
ActiveRecord::Base.establish_connection(adapter: "sqlite3", database: ":memory:")
ActiveRecord::Schema.define do
create_table :posts, force: true do |t|
t.string :body
t.timestamps
end
create_table :comments, force: true do |t|
t.integer :post_id
t.string :body
t.timestamps
end
end
class Post < ActiveRecord::Base
has_many :comments
end
class Comment < ActiveRecord::Base
belongs_to :post
end
post = Post.create!(body: "x" * 100)
2.times { post.comments.create!(body: "x" * 100) }
post.comments.load
cache_entry = ActiveSupport::Cache::Entry.new(post)
Rails70Coder = ActiveSupport::Cache::Coders::Rails70Coder
CacheSerializer = ActiveSupport::MessagePack::CacheSerializer
Benchmark.ips do |x|
x.report("Rails70Coder") do
Rails70Coder.load(Rails70Coder.dump(cache_entry))
end
x.report("CacheSerializer") do
CacheSerializer.load(CacheSerializer.dump(cache_entry))
end
x.compare!
end
puts "Rails70Coder size: #{Rails70Coder.dump(cache_entry).bytesize}"
puts "CacheSerializer size: #{CacheSerializer.dump(cache_entry).bytesize}"
```
```
Warming up --------------------------------------
Rails70Coder 329.000 i/100ms
CacheSerializer 492.000 i/100ms
Calculating -------------------------------------
Rails70Coder 3.285k (± 1.7%) i/s - 16.450k in 5.008447s
CacheSerializer 4.895k (± 2.4%) i/s - 24.600k in 5.028803s
Comparison:
CacheSerializer: 4894.7 i/s
Rails70Coder: 3285.4 i/s - 1.49x slower
Rails70Coder size: 808
CacheSerializer size: 593
```
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
Previously, these could be found in the Testing guide, but not in the
api documentation.
The assert_raise alias was moved to ActiveSupport::Testing::Assertions
because that is where assert_raises is defined (and rdoc handles that
case).
The Minitest aliases are not easily handled by rdoc, so they are written
by hand. We could copy the documentation for the aliased methods here,
but linking to them seems sufficient.
When 8b2f57dc6f was
introduced it broke delegation of a method with implicit block like this:
```ruby
class Foo
def self.bar
yield
end
end
class Bar
delegate :bar, to: Foo
end
Bar.new.bar { } # raises LocalJumpError: no block given (yield)
```
It happens because it's impossible to detect a method with implicit block and
we rely on method's parameters when we generate delegated methods.
I've found only one solution to this issue: we always generate a method that explicitly accepts
block. It kind of makes sense since evey Ruby method accepts block implicitly anyway.
Fixes https://github.com/rails/rails/issues/47624
In Rails 7, if you do `Rails.cache.write(key, value, expires_in: 1.minute.from_now)`, it will work. The actual expiration will be much more than a minute away, but it won't raise. (The correct code is `expires_in: 1.minute` or `expires_at: 1.minute.from_now`.)
Since https://github.com/rails/rails/pull/45892 the same code will error with:
```
NoMethodError: undefined method `negative?' for 2008-04-24 00:01:00 -0600:Time
/Users/alex/Code/rails/activesupport/lib/active_support/cache.rb:743:in `merged_options'
/Users/alex/Code/rails/activesupport/lib/active_support/cache.rb:551:in `write'
```
To make it a bit easier to upgrade to Rails 7.1, this PR introduces a better error if you pass a `Time` object to `expires_in:`
```
ArgumentError: expires_in parameter should not be a Time. Did you mean to use expires_at? Got: 2023-04-07 14:47:45 -0600
/Users/alex/Code/rails/activesupport/lib/active_support/cache.rb:765:in `handle_invalid_expires_in'
/Users/alex/Code/rails/activesupport/lib/active_support/cache.rb:745:in `merged_options'
/Users/alex/Code/rails/activesupport/lib/active_support/cache.rb:551:in `write'
```
3d00c8b97f introduced a regression:
```ruby
ActiveSupport::Inflector.inflections(:en) do |inflect|
inflect.acronym("API")
end
```
If you have a controller called `APIController`, you may get a crash backtrace that looks like this:
```
FrozenError: can't modify frozen String: "API"
rails (b8a399cd6ffe) actionpack/lib/action_dispatch/http/request.rb:89:in `controller_class_for'
rails (b8a399cd6ffe) actionpack/lib/action_dispatch/request/utils.rb💯in `action_encoding_template'
rails (b8a399cd6ffe) actionpack/lib/action_dispatch/request/utils.rb:85:in `encode'
rails (b8a399cd6ffe) actionpack/lib/action_dispatch/request/utils.rb:45:in `set_binary_encoding'
rails (b8a399cd6ffe) actionpack/lib/action_dispatch/http/parameters.rb:68:in `path_parameters='
```
And you can see the issue in action like this:
```ruby
Tanda @ Rails 7.1.0.alpha (test) :004 > "API".freeze.underscore.frozen?
false # this is expected
Tanda @ Rails 7.1.0.alpha (test) :004 > "API".freeze.underscore.camelize.frozen?
false # this is *not* expected, and is what causes the above code to crash
```
I think the correct behaviour is for the inflector to always return non-frozen strings, even if it's using a cached frozen string as is the case in 3d00c8b97f
Other semi-related PRs: https://github.com/rails/rails/pull/41174, https://github.com/rails/rails/pull/41881
cc @amatsuda @byroot
`Listen.to` starts a bunch of background threads that need to perform
some work before they are able to receive events, but it doesn't block
until they are ready, which expose us to a race condition.
With `wait_for_state(:processing_events)` we can ensure that it's ready on
Linux, however on macOS, the Darwin backend has a second background thread
we can't wait on.
As a workaround we wait a bit after the fork to allow that thread to
reach it's listning state.
Fanout is a singleton shared among threads, so all of its hash maps
should be concurrent hash maps otherwise we'll see errors like this:
```
RuntimeError: can't add a new key into hash during iteration
```
Currently when opening the main framework pages there is no introduction
to the framework. Instead we only see a whole lot of modules and the
`gem_version` and `version` methods.
By including the READMEs using the `:include:` directive each frameworks
has a nice introduction.
For markdown READMEs we need to add the :markup: directive.
[ci-skip]
Co-authored-by: zzak <zzakscott@gmail.com>
All methods called on ActiveSupport::Deprecation are delegated to the
singleton instance, but now that each framework has its own Deprecation
instance, it's no longer necessary for internal deprecations.
For external code, instead of using ActiveSupport::Deprecation.warn,
they should create their own instance of the class and call warn on it.
Use case
A very common pattern in Ruby, especially in testing is to save the value of an attribute, set a new value, and then restore the old value in an `ensure` clause.
e.g. in unit tests
```ruby
def test_something_when_enabled
enabled_was, SomeLibrary.enabled = SomeLibrary.enabled, true
# test things
ensure
SomeLibrary.enabled = enabled_was
end
```
Or sometime in actual APIs:
```ruby
def with_something_enabled
enabled_was = @enabled
@enabled = true
yield
ensure
@enabled = enabled_was
end
```
There is no inherent problem with this pattern, but it can be easy to make a mistake, for instance the unit test example:
```ruby
def test_something_when_enabled
some_call_that_may_raise
enabled_was, SomeLibrary.enabled = SomeLibrary.enabled, true
# test things
ensure
SomeLibrary.enabled = enabled_was
end
```
In the above if `some_call_that_may_raise` actually raises, `SomeLibrary.enabled` is set back to `nil` rather than its original value. I've seen this mistake quite frequently.
Object#with
I think it would be very useful to have a method on Object to implement this pattern in a correct and easy to use way.
NB: `public_send` is used because I don't think such method should be usable if the accessors are private.
With usage:
```ruby
def test_something_when_enabled
SomeLibrary.with(enabled: true) do
# test things
end
end
```
```ruby
GC.with(measure_total_time: true, auto_compact: false) do
# do something
end
```
Lots of tests in Rails's codebase could be simplified, e.g.:
- Changing `Thread.report_on_exception`: 2d2fdc941e/activerecord/test/cases/connection_pool_test.rb (L583-L595)
- Changing a class attribute: 2d2fdc941e/activerecord/test/cases/associations/belongs_to_associations_test.rb (L136-L150)
Why:
----
Following up on [#47323](https://github.com/rails/rails/issues/47323).
Many options are not forwarded to the Dalli client when it is
initialized from the `ActiveSupport::Cache::MemCacheStore`. This is to
support a broader set of features powered by the implementation. When an
instance of a client is passed on the initializer, it takes precedence,
and we have no control over which attributes will be overridden or
re-processed on the client side; this is by design and should remain as
such to allow both projects to progress independently. Having this
option introduces several potential bugs that are difficult to pinpoint
and get multiplied by which version of the tool is used and how each
evolves. During the conversation on the issue, the `Dalli` client
maintainer supports [deprecating](https://github.com/rails/rails/issues/47323#issuecomment-1424292456)
this option.
How:
----
Removing this implicit dependency will ensure each library can evolve
separately and cements the usage of `Dalli::Client` as an [implementation
detail](https://github.com/rails/rails/issues/21595#issuecomment-139815433)
We can not remove a supported feature overnight, so I propose we add a
deprecation warning for the next minor release(7.2 at this time).
There was a constant on the `Cache` namespace only used to restrict
options passed to the `Dalli::Client` initializer that now lives on the
`MemCacheStore` class.
Co-authored-by: Eileen M. Uchitelle <eileencodes@users.noreply.github.com>
Previously, requiring any of the ragel generated parsers in mail would
output tons of warnings in tests, making output much harder to read
(especially in Railties).
This commit extends the RaiseWarnings patch to suppress output from
these mail parsers.
The suppression can be removed once mikel/mail#1557 or mikel/mail#1551
or any other PR fixes the issue
* Remove Copyright years
* Basecamp is now 37signals... again
Co-authored-by: David Heinemeier Hansson <dhh@hey.com>
---------
Co-authored-by: David Heinemeier Hansson <dhh@hey.com>
Fix: https://github.com/rails/rails/issues/47418
We should trust the `Process._fork` callback, as it can't really
be bypassed except throught `Process.daemonize` but I don't think
it's a problem.
We initially kept the PID check even if the callback was present,
but it turns out that on modern GLIBC the getpid() syscall isn't
cached and it causes a significant overhead.
Add test coverage for existing `Object#with_options` support for
`Hash`-like objects.
The current implementation expects "Hash-like" to mean that the argument
implements both `Hash#deep_merge` and `#to_hash` (to be called
explicitly with `#to_hash` and implicitly with `**`).
There are multiple points of failure when processing a message with
`MessageEncryptor` or `MessageVerifier`, and there several ways we might
want to handle those failures. For example, swallowing a failure with
`MessageVerifier#verified`, or raising a specific exception with
`MessageVerifier#verify`, or conditionally ignoring a failure when
rotations are configured.
Prior to this commit, the _internal_ logic of handling failures was
implemented using a mix of `nil` return values and raised exceptions.
This commit reimplements the internal logic using `throw` and a few
precisely targeted `rescue`s. This accomplishes several things:
* Allow rotation of serializers for `MessageVerifier`. Previously,
errors from a `MessageVerifier`'s initial serializer were never
rescued. Thus, the serializer could not be rotated:
```ruby
old_verifier = ActiveSupport::MessageVerifier.new("secret", serializer: Marshal)
new_verifier = ActiveSupport::MessageVerifier.new("secret", serializer: JSON)
new_verifier.rotate(serializer: Marshal)
message = old_verifier.generate("message")
new_verifier.verify(message)
# BEFORE:
# => raises JSON::ParserError
# AFTER:
# => "message"
```
* Allow rotation of serializers for `MessageEncryptor` when using a
non-standard initial serializer. Similar to `MessageVerifier`, the
serializer could not be rotated when the initial serializer raised an
error other than `TypeError` or `JSON::ParserError`, such as
`Psych::SyntaxError` or a custom error.
* Raise `MessageEncryptor::InvalidMessage` from `decrypt_and_verify`
regardless of cipher. Previously, when a `MessageEncryptor` was using
a non-AEAD cipher such as AES-256-CBC, a corrupt or tampered message
would raise `MessageVerifier::InvalidSignature` due to reliance on
`MessageVerifier` for verification. Now, the verification mechanism
is transparent to the user:
```ruby
encryptor = ActiveSupport::MessageEncryptor.new("x" * 32, cipher: "aes-256-gcm")
message = encryptor.encrypt_and_sign("message")
encryptor.decrypt_and_verify(message.next)
# => raises ActiveSupport::MessageEncryptor::InvalidMessage
encryptor = ActiveSupport::MessageEncryptor.new("x" * 32, cipher: "aes-256-cbc")
message = encryptor.encrypt_and_sign("message")
encryptor.decrypt_and_verify(message.next)
# BEFORE:
# => raises ActiveSupport::MessageVerifier::InvalidSignature
# AFTER:
# => raises ActiveSupport::MessageEncryptor::InvalidMessage
```
* Support `nil` original value when using `MessageVerifier#verify`.
Previously, `MessageVerifier#verify` did not work with `nil` original
values, though both `MessageVerifier#verified` and
`MessageEncryptor#decrypt_and_verify` do:
```ruby
encryptor = ActiveSupport::MessageEncryptor.new("x" * 32)
message = encryptor.encrypt_and_sign(nil)
encryptor.decrypt_and_verify(message)
# => nil
verifier = ActiveSupport::MessageVerifier.new("secret")
message = verifier.generate(nil)
verifier.verified(message)
# => nil
verifier.verify(message)
# BEFORE:
# => raises ActiveSupport::MessageVerifier::InvalidSignature
# AFTER:
# => nil
```
* Improve performance of verifying a message when it has expired and one
or more rotations have been configured:
```ruby
# frozen_string_literal: true
require "benchmark/ips"
require "active_support/all"
verifier = ActiveSupport::MessageVerifier.new("new secret")
verifier.rotate("old secret")
message = verifier.generate({ "data" => "x" * 100 }, expires_at: 1.day.ago)
Benchmark.ips do |x|
x.report("expired message") do
verifier.verified(message)
end
end
```
__Before__
```
Warming up --------------------------------------
expired message 1.442k i/100ms
Calculating -------------------------------------
expired message 14.403k (± 1.7%) i/s - 72.100k in 5.007382s
```
__After__
```
Warming up --------------------------------------
expired message 1.995k i/100ms
Calculating -------------------------------------
expired message 19.992k (± 2.0%) i/s - 101.745k in 5.091421s
```
Fixes#47185.
This factors common methods into a `ActiveSupport::Messages::Codec` base
class. This also disentangles serialization (and deserialization) from
encryption (and decryption) in `MessageEncryptor`.
Prior to this commit, messages with metadata were always serialized in
the following way:
```ruby
Base64.strict_encode64(
ActiveSupport::JSON.encode({
"_rails" => {
"message" => Base64.strict_encode64(
serializer.dump(data)
),
"pur" => "the purpose",
"exp" => "the expiration"
},
})
)
```
in which the message data is serialized and URL-encoded twice.
This commit changes message serialization such that, when possible, the
data is serialized and URL-encoded only once:
```ruby
Base64.strict_encode64(
serializer.dump({
"_rails" => {
"data" => data,
"pur" => "the purpose",
"exp" => "the expiration"
},
})
)
```
This improves performance in proportion to the size of the data:
**Benchmark**
```ruby
# frozen_string_literal: true
require "benchmark/ips"
require "active_support/all"
verifier = ActiveSupport::MessageVerifier.new("secret", serializer: JSON)
payloads = [
{ "content" => "x" * 100 },
{ "content" => "x" * 2000 },
{ "content" => "x" * 1_000_000 },
]
if ActiveSupport::Messages::Metadata.respond_to?(:use_message_serializer_for_metadata)
ActiveSupport::Messages::Metadata.use_message_serializer_for_metadata = true
end
Benchmark.ips do |x|
payloads.each do |payload|
x.report("generate ~#{payload["content"].size}B") do
$generated_message = verifier.generate(payload, purpose: "x")
end
x.report("verify ~#{payload["content"].size}B") do
verifier.verify($generated_message, purpose: "x")
end
end
end
puts
puts "Message size:"
payloads.each do |payload|
puts " ~#{payload["content"].size} bytes of data => " \
"#{verifier.generate(payload, purpose: "x").size} byte message"
end
```
**Before**
```
Warming up --------------------------------------
generate ~100B 1.578k i/100ms
verify ~100B 2.506k i/100ms
generate ~2000B 447.000 i/100ms
verify ~2000B 1.409k i/100ms
generate ~1000000B 1.000 i/100ms
verify ~1000000B 6.000 i/100ms
Calculating -------------------------------------
generate ~100B 15.807k (± 1.8%) i/s - 80.478k in 5.093161s
verify ~100B 25.240k (± 2.1%) i/s - 127.806k in 5.066096s
generate ~2000B 4.530k (± 2.4%) i/s - 22.797k in 5.035398s
verify ~2000B 14.136k (± 2.3%) i/s - 71.859k in 5.086267s
generate ~1000000B 11.673 (± 0.0%) i/s - 59.000 in 5.060598s
verify ~1000000B 64.372 (± 6.2%) i/s - 324.000 in 5.053304s
Message size:
~100 bytes of data => 306 byte message
~2000 bytes of data => 3690 byte message
~1000000 bytes of data => 1777906 byte message
```
**After**
```
Warming up --------------------------------------
generate ~100B 4.689k i/100ms
verify ~100B 3.183k i/100ms
generate ~2000B 2.722k i/100ms
verify ~2000B 2.066k i/100ms
generate ~1000000B 12.000 i/100ms
verify ~1000000B 11.000 i/100ms
Calculating -------------------------------------
generate ~100B 46.984k (± 1.2%) i/s - 239.139k in 5.090540s
verify ~100B 32.043k (± 1.2%) i/s - 162.333k in 5.066903s
generate ~2000B 27.163k (± 1.2%) i/s - 136.100k in 5.011254s
verify ~2000B 20.726k (± 1.7%) i/s - 105.366k in 5.085442s
generate ~1000000B 125.600 (± 1.6%) i/s - 636.000 in 5.064607s
verify ~1000000B 122.039 (± 4.1%) i/s - 616.000 in 5.058386s
Message size:
~100 bytes of data => 234 byte message
~2000 bytes of data => 2770 byte message
~1000000 bytes of data => 1333434 byte message
```
This optimization is only applied for recognized serializers that are
capable of serializing a `Hash`.
Additionally, because the optimization changes the message format, a
`config.active_support.use_message_serializer_for_metadata` option has
been added to disable it. The optimization is disabled by default, but
enabled with `config.load_defaults 7.1`.
Regardless of whether the optimization is enabled, messages using either
format can still be read.
In the case of a rolling deploy of a Rails upgrade, wherein servers that
have not yet been upgraded must be able to read messages from upgraded
servers, the optimization can be disabled on first deploy, then safely
enabled on a subsequent deploy.
Prior to this commit, `ActiveSupport::SecureCompareRotator` used
`ActiveSupport::Messages::Rotator` for part of its rotation logic, even
though `SecureCompareRotator` is entirely unrelated to messages. This
made it precarious to alter `Messages::Rotator`, especially because
`Messages::Rotator` was `prepend`ed to `SecureCompareRotator` rather
than `include`d.
This commit reimplements `SecureCompareRotator` without
`Messages::Rotator`, which simplifies the logic and, as a bonus,
improves performance:
```ruby
# frozen_string_literal: true
require "benchmark/ips"
require "active_support/all"
comparer = ActiveSupport::SecureCompareRotator.new("new secret")
comparer.rotate("old secret")
Benchmark.ips do |x|
x.report("compare old") do
comparer.secure_compare!("old secret")
end
end
```
__Before__
```
Warming up --------------------------------------
compare old 72.073k i/100ms
Calculating -------------------------------------
compare old 719.844k (± 1.0%) i/s - 3.604M in 5.006682s
```
__After__
```
Warming up --------------------------------------
compare old 147.486k i/100ms
Calculating -------------------------------------
compare old 1.473M (± 0.9%) i/s - 7.374M in 5.006655s
```
Add as_json method for the new Data object introduced in Ruby 3.2 so
Data objects can be encoded/decoded.
data = Data.define(:name).new(:test)
data.as_json
We calculate idle time by subtracting two separately-obtained values,
and apparently it's possible for the CPU-time clock to tick a
millisecond ahead of the monotonic clock.
I think it's safe to ignore that potential difference in general, but
even at the extreme of CPU activity, I'm reasonably confident time
doesn't move backwards.
Follow-up to #42846.
Prior to this commit, `ActiveSupport::EncryptedFile` used the
`ActiveSupport::MessageEncryptor` default serializer. #42846 changed
the default serializer to `JSON` when using `config.load_defaults 7.1`.
Thus, encrypted files that were written with the previous default
serializer (`Marshal`) could not be read by `Rails.application.encrypted`.
That included files which were written with `bin/rails encrypted:edit`
even when using the new default, because the `bin/rails encrypted:edit`
command does not run the initializer that applies the configuration to
`MessageEncryptor`:
```console
$ bin/rails r 'puts ActiveSupport::MessageEncryptor.default_message_encryptor_serializer'
json
$ bin/rails encrypted:edit config/my_encrypted.yml.enc
...
$ bin/rails encrypted:show config/my_encrypted.yml.enc
foo: bar
$ bin/rails r 'Rails.application.encrypted("config/my_encrypted.yml.enc").read'
rails/activesupport/lib/active_support/message_encryptor.rb:241:in `rescue in _decrypt': ActiveSupport::MessageEncryptor::InvalidMessage (ActiveSupport::MessageEncryptor::InvalidMessage)
...
ruby-3.2.0/lib/ruby/3.2.0/json/common.rb:216:in `parse': unexpected token at "foo: bar (JSON::ParserError)
```
This commit hard codes the serializer for `EncryptedFile` to `Marshal`.
Ruby 3.2 significantly changed how instance variables are store.
It now use shapes, and in short, it's important for performance
to define instance variables in a consistent order to limit the
amount of shapes.
Otherwise, the number of shapes will increase past a point where
MRI won't be able to cache instance variable access. The impact
is even more important when YJIT is enabled.
This PR is data driven. I dump the list of Shapes from Shopify's
monolith production environment, and Rails is very present among
the top offenders:
```
Shape Edges Report
-----------------------------------
770 @default_graphql_name
697 @own_fields
661 @to_non_null_type
555 @own_interface_type_memberships
472 @description
389 @errors
348 @oseid
316 @_view_runtime
310 @_db_runtime
292 @visibility
286 @shop
271 @attribute_method_patterns_cache
264 @namespace_for_serializer
254 @locking_column
254 @primary_key
253 @validation_context
244 @quoted_primary_key
238 @access_controls
234 @_trigger_destroy_callback
226 @_trigger_update_callback
224 @finder_needs_type_condition
215 @_committed_already_called
214 @api_type
203 @mutations_before_last_save
202 @access_controls_overrides
201 @options
198 @mutations_from_database
190 @_already_called
183 @name
179 @_request
176 @own_arguments
175 @_assigns
175 @virtual_path
174 @context
173 @_controller
173 @output_buffer
173 @view_flow
172 @_default_form_builder
169 @cache
159 @_touch_record
151 @attribute_names
151 @default_attributes
150 @columns_hash
149 @attribute_types
148 @columns
147 @marked_for_same_origin_verification
146 @schema_loaded
143 @_config
143 @type
141 @column_names
```
All the changes are of similar nature, the goal is to preset the instance
variable to nil when objects are allocated, or when classes are created.
For classes I leverage the `inherited` hook. If the patern becomes common enough
it might make sense to add a helper for this in `ActiveSupport::Concern`.
Attributes such as `set` and `reset` should not be used as they clash with the `CurrentAttributes` public API. This PR raises an `ArgumentError` if a restricted attribute name is used.
A user reported seeing the following error when trying to use the
EventedFileUpdateChecker in their app:
```shell
$ rails server
=> Booting Puma
=> Rails 7.0.4 application starting in development
=> Run `bin/rails server --help` for more startup options
Exiting
/usr/local/lib/ruby/gems/3.0.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require': cannot load such file -- listen (LoadError)
from /usr/local/lib/ruby/gems/3.0.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
from /usr/local/lib/ruby/gems/3.0.0/gems/activesupport-7.0.4/lib/active_support/evented_file_update_checker.rb:6:in `<top (required)>'
from /usr/local/lib/ruby/gems/3.0.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
from /usr/local/lib/ruby/gems/3.0.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
from /home/ross/Data/EmySystem/newentry/Entry2/config/environments/development.rb:84:in `block in <top (required)>'
from /usr/local/lib/ruby/gems/3.0.0/gems/railties-7.0.4/lib/rails/railtie.rb:257:in `instance_eval'
from /usr/local/lib/ruby/gems/3.0.0/gems/railties-7.0.4/lib/rails/railtie.rb:257:in `configure'
from /home/ross/Data/EmySystem/newentry/Entry2/config/environments/development.rb:3:in `<top (required)>'
...
```
While we were able to direct them to the proper fix (add listen to
their app's Gemfile), the error message does not really point them in
that direction on its own.
This commit improves the error message by using Kernel#gem to indicate
that the gem should be in the user's bundle. This is the same approach
used for other not-specified dependencies in Rails, such as the various
database drivers in their Active Record adapters.
New error message:
```shell
$ bin/rails r "ActiveSupport::EventedFileUpdateChecker"
/home/hartley/.cache/asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/bundler/rubygems_integration.rb:276:in `block (2 levels) in replace_gem': listen is not part of the bundle. Add it to your Gemfile. (Gem::LoadError)
from /home/hartley/src/github.com/skipkayhil/rails/activesupport/lib/active_support/evented_file_update_checker.rb:3:in `<top (required)>'
from /home/hartley/.cache/asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
from /home/hartley/.cache/asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
...
```
Ref: ad39d6b
Ensure that all evals include file and line number to identify their
source.
Two of the evals reported by this cop were unneccesary and replaced with
non-eval alternatives: xml is set as a local variable in
Template::Handlers::Builder#call, and instance_eval isn't needed to get
the current binding.
There are additionally 27 offenses in test directories, but since it
seems less important to fix those they are currently ignored.
Active Support's original Hash#transform_keys used not to take any argument, and
that version has been transported to Ruby 2.5
https://github.com/ruby/ruby/commit/1405111722
Then since Ruby 3.0, that method in Ruby has been expanded to take a Hash
argument https://github.com/ruby/ruby/commit/b25e27277d
Hence, we'd better update our own in-house Hash-like class
HashWithIndifferentAccess to follow the Hash API change.
Co-authored-by: Hartley McGuire <skipkayhil@gmail.com>
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
Method delegation with `...` argument is known to be slow because it allocates
redundant Array and Hash objects each time when being called.
see: https://bugs.ruby-lang.org/issues/19165
Current implementation of `delegate` defines all delegation methods in this
slow form regardless of the original method's arity, but in case the delegation
target is a Class or a Module, we can investigate the arity of the original
method in the definition timing, then we can define the delegation in proper
minimal arity.
This makes 3.5 times faster zero arity method delegation as follows:
Warming up --------------------------------------
old 811.534k i/100ms
new 1.807M i/100ms
Calculating -------------------------------------
old 9.809M (± 3.4%) i/s - 49.504M in 5.053355s
new 34.360M (± 0.8%) i/s - 173.465M in 5.048692s
Comparison:
new: 34360408.4 i/s
old: 9809157.4 i/s - 3.50x (± 0.00) slower
This avoids an extra allocation from `map`.
__Benchmark__
```ruby
# frozen_string_literal: true
require "benchmark/ips"
require "active_support/all"
Hash.alias_method(:old_values_at, :values_at)
Hash.alias_method(:new_values_at, :values_at)
class ActiveSupport::HashWithIndifferentAccess
def old_values_at(*keys)
super(*keys.map { |key| convert_key(key) })
end
def new_values_at(*keys)
keys.map! { |key| convert_key(key) }
super
end
end
hwia = { foo: 1, bar: 2, baz: 3, qux: 4 }.with_indifferent_access
splat_keys = [:bar, :baz]
Benchmark.ips do |x|
x.report("old values_at 1") do
hwia.old_values_at(:bar)
end
x.report("new values_at 1") do
hwia.new_values_at(:bar)
end
x.compare!
end
Benchmark.ips do |x|
x.report("old values_at splat") do
hwia.old_values_at(*splat_keys)
end
x.report("new values_at splat") do
hwia.new_values_at(*splat_keys)
end
x.compare!
end
```
__Results__
```
Warming up --------------------------------------
old values_at 1 150.881k i/100ms
new values_at 1 163.731k i/100ms
Calculating -------------------------------------
old values_at 1 1.509M (± 1.3%) i/s - 7.695M in 5.099286s
new values_at 1 1.646M (± 1.1%) i/s - 8.350M in 5.072959s
Comparison:
new values_at 1: 1646260.9 i/s
old values_at 1: 1509283.6 i/s - 1.09x (± 0.00) slower
Warming up --------------------------------------
old values_at splat 110.815k i/100ms
new values_at splat 118.871k i/100ms
Calculating -------------------------------------
old values_at splat 1.118M (± 1.3%) i/s - 5.652M in 5.057480s
new values_at splat 1.194M (± 0.9%) i/s - 6.062M in 5.077104s
Comparison:
new values_at splat: 1194171.4 i/s
old values_at splat: 1117670.4 i/s - 1.07x (± 0.00) slower
```
This clarifies the `ActiveSupport::ParameterFilter` documentation, and
tweaks the example code to be more friendly to the syntax highlighter
(similar to the tweaks made for `ActionDispatch::Http::FilterParameters`
in 782bed5d450363b302e0e6aa28b7ea0aef306d9f).
This also trims the `ActionDispatch::Http::FilterParameters`
documentation, and links it to `ActiveSupport::ParameterFilter`, since
`ActiveSupport::ParameterFilter` is responsible for filter behavior.
This is in line with transitioning away from the global
`ActiveSupport::Deprecation` instance, towards individual
`ActiveSupport::Deprecation` instances.
This matches the indentation used in generated code, such as code from
`railties/lib/rails/generators/rails/scaffold_controller/templates/controller.rb.tt`.
Unless you're very good at math, this test fail message is not the easiest to debug:
```
"User.count" didn't change by 32.
Expected: 1611
Actual: 1579
```
It's not obvious from the error, but in this case, it actually changed by 0. This is a pretty strong clue as to what went wrong, but the message doesn't tell us that.
This PR improves the message to make debugging easier:
```
"User.count" didn't change by 32 (changed by 0).
Expected: 1611
Actual: 1579
```
because splatting the argument allocates an extra Array object.
benchmark
```ruby
s = 'a'.html_safe
Benchmark.ips do |x|
x.report('') { s * 1 }
end
```
result
```
before
Warming up --------------------------------------
216.816k i/100ms
Calculating -------------------------------------
2.341M (± 2.0%) i/s - 11.708M in 5.002555s
after
Warming up --------------------------------------
315.118k i/100ms
Calculating -------------------------------------
3.704M (± 1.5%) i/s - 18.592M in 5.020261s
```
