The documentation on ActiveSupport::MessageVerifier used the “sensitive data” string as an example; that wording might induce the developer to think we’re dealing with encryption, while the payload is actually only Base64 encoded and is not protected at all.
We also improve the documentation on ActiveRecord::SignedId, which uses MessageVerifier and thereby will also expose the ID as encoded cleartext, making explicit that it’s not encryption, only signing.
Lastly, we refer the developer to MessageEncryptor if the payload needs to be encrypted.
This commit adds support for `:message_pack` as a serializer for
`MessageEncryptor` and `MessageVerifier`, and, consequently, as an
option for `config.active_support.message_serializer`.
The `:message_pack` serializer is implemented via
`ActiveSupport::Messages::SerializerWithFallback` and can fall back to
deserializing with `AS::JSON`. Additionally, the `:marshal`, `:json`,
and `:json_allow_marshal` serializers can now fall back to deserializing
with `AS::MessagePack`.
This commit also adds support for `:message_pack_allow_marshal` as a
serializer, which can fall back to deserializing with `Marshal` as well
as `AS::JSON`.
Prior to this commit, `config.load_defaults 7.1` would cause old
messages (or messages from older apps) to become unreadable, and
developers were encouraged to manually set
`config.active_support.message_serializer = :json_allow_marshal` in
order to prevent this.
This commit changes the default message serializer set by
`config.load_defaults 7.1` from `:json` to `:json_allow_marshal` so that
upgraded apps can continue to read old messages without additional
configuration.
The intention is to eventually change the default to `:json` (with no
`Marshal` fallback) in Rails 7.2, after some time has passed with apps
generating JSON-serialized messages.
Apps can opt in to JSON-only serialization before Rails 7.2 by manually
setting `config.active_support.message_serializer = :json`.
Fixes#48118.
In #42843 and #42846, several config settings were added to control the
default serializer for `MessageEncryptor` and `MessageVerifier`, and to
provide a migration path from a default `Marshal` serializer to a
default `JSON` serializer:
* `config.active_support.default_message_encryptor_serializer`
* Supports `:marshal`, `:hybrid`, or `:json`.
* `config.active_support.default_message_verifier_serializer`
* Supports `:marshal`, `:hybrid`, or `:json`.
* `config.active_support.fallback_to_marshal_deserialization`
* Affects `:hybrid` for both `MessageEncryptor` and `MessageVerifier`.
* `config.active_support.use_marshal_serialization`
* Affects `:hybrid` for both `MessageEncryptor` and `MessageVerifier`.
This commit unifies those config settings into a single setting,
`config.active_support.message_serializer`, which supports `:marshal`,
`:json_allow_marshal`, and `:json` values. So, for example,
```ruby
config.active_support.default_message_encryptor_serializer = :hybrid
config.active_support.default_message_verifier_serializer = :hybrid
config.active_support.fallback_to_marshal_deserialization = true
config.active_support.use_marshal_serialization = false
```
becomes
```ruby
config.active_support.message_serializer = :json_allow_marshal
```
and
```ruby
config.active_support.default_message_encryptor_serializer = :hybrid
config.active_support.default_message_verifier_serializer = :hybrid
config.active_support.fallback_to_marshal_deserialization = false
config.active_support.use_marshal_serialization = false
```
becomes
```ruby
config.active_support.message_serializer = :json
```
This commit also replaces `ActiveSupport::JsonWithMarshalFallback` with
`ActiveSupport::Messages::SerializerWithFallback`, which implements a
generic mechanism for serializer fallback. The `:marshal` serializer
uses this mechanism too, so
```ruby
config.active_support.default_message_encryptor_serializer = :hybrid
config.active_support.default_message_verifier_serializer = :hybrid
config.active_support.fallback_to_marshal_deserialization = false
config.active_support.use_marshal_serialization = true
```
becomes
```ruby
config.active_support.message_serializer = :marshal
```
Additionally, the logging behavior of `JsonWithMarshalFallback` has been
replaced with notifications which include the names of the intended and
actual serializers, as well as the serialized and deserialized message
data. This provides a more targeted means of tracking serializer
fallback events. It also allows the user to "silence" such events, if
desired, without an additional config setting.
All of these changes make it easier to add migration paths for new
serializers such as `ActiveSupport::MessagePack`.
There are multiple points of failure when processing a message with
`MessageEncryptor` or `MessageVerifier`, and there several ways we might
want to handle those failures. For example, swallowing a failure with
`MessageVerifier#verified`, or raising a specific exception with
`MessageVerifier#verify`, or conditionally ignoring a failure when
rotations are configured.
Prior to this commit, the _internal_ logic of handling failures was
implemented using a mix of `nil` return values and raised exceptions.
This commit reimplements the internal logic using `throw` and a few
precisely targeted `rescue`s. This accomplishes several things:
* Allow rotation of serializers for `MessageVerifier`. Previously,
errors from a `MessageVerifier`'s initial serializer were never
rescued. Thus, the serializer could not be rotated:
```ruby
old_verifier = ActiveSupport::MessageVerifier.new("secret", serializer: Marshal)
new_verifier = ActiveSupport::MessageVerifier.new("secret", serializer: JSON)
new_verifier.rotate(serializer: Marshal)
message = old_verifier.generate("message")
new_verifier.verify(message)
# BEFORE:
# => raises JSON::ParserError
# AFTER:
# => "message"
```
* Allow rotation of serializers for `MessageEncryptor` when using a
non-standard initial serializer. Similar to `MessageVerifier`, the
serializer could not be rotated when the initial serializer raised an
error other than `TypeError` or `JSON::ParserError`, such as
`Psych::SyntaxError` or a custom error.
* Raise `MessageEncryptor::InvalidMessage` from `decrypt_and_verify`
regardless of cipher. Previously, when a `MessageEncryptor` was using
a non-AEAD cipher such as AES-256-CBC, a corrupt or tampered message
would raise `MessageVerifier::InvalidSignature` due to reliance on
`MessageVerifier` for verification. Now, the verification mechanism
is transparent to the user:
```ruby
encryptor = ActiveSupport::MessageEncryptor.new("x" * 32, cipher: "aes-256-gcm")
message = encryptor.encrypt_and_sign("message")
encryptor.decrypt_and_verify(message.next)
# => raises ActiveSupport::MessageEncryptor::InvalidMessage
encryptor = ActiveSupport::MessageEncryptor.new("x" * 32, cipher: "aes-256-cbc")
message = encryptor.encrypt_and_sign("message")
encryptor.decrypt_and_verify(message.next)
# BEFORE:
# => raises ActiveSupport::MessageVerifier::InvalidSignature
# AFTER:
# => raises ActiveSupport::MessageEncryptor::InvalidMessage
```
* Support `nil` original value when using `MessageVerifier#verify`.
Previously, `MessageVerifier#verify` did not work with `nil` original
values, though both `MessageVerifier#verified` and
`MessageEncryptor#decrypt_and_verify` do:
```ruby
encryptor = ActiveSupport::MessageEncryptor.new("x" * 32)
message = encryptor.encrypt_and_sign(nil)
encryptor.decrypt_and_verify(message)
# => nil
verifier = ActiveSupport::MessageVerifier.new("secret")
message = verifier.generate(nil)
verifier.verified(message)
# => nil
verifier.verify(message)
# BEFORE:
# => raises ActiveSupport::MessageVerifier::InvalidSignature
# AFTER:
# => nil
```
* Improve performance of verifying a message when it has expired and one
or more rotations have been configured:
```ruby
# frozen_string_literal: true
require "benchmark/ips"
require "active_support/all"
verifier = ActiveSupport::MessageVerifier.new("new secret")
verifier.rotate("old secret")
message = verifier.generate({ "data" => "x" * 100 }, expires_at: 1.day.ago)
Benchmark.ips do |x|
x.report("expired message") do
verifier.verified(message)
end
end
```
__Before__
```
Warming up --------------------------------------
expired message 1.442k i/100ms
Calculating -------------------------------------
expired message 14.403k (± 1.7%) i/s - 72.100k in 5.007382s
```
__After__
```
Warming up --------------------------------------
expired message 1.995k i/100ms
Calculating -------------------------------------
expired message 19.992k (± 2.0%) i/s - 101.745k in 5.091421s
```
Fixes#47185.
This factors common methods into a `ActiveSupport::Messages::Codec` base
class. This also disentangles serialization (and deserialization) from
encryption (and decryption) in `MessageEncryptor`.
Prior to this commit, messages with metadata were always serialized in
the following way:
```ruby
Base64.strict_encode64(
ActiveSupport::JSON.encode({
"_rails" => {
"message" => Base64.strict_encode64(
serializer.dump(data)
),
"pur" => "the purpose",
"exp" => "the expiration"
},
})
)
```
in which the message data is serialized and URL-encoded twice.
This commit changes message serialization such that, when possible, the
data is serialized and URL-encoded only once:
```ruby
Base64.strict_encode64(
serializer.dump({
"_rails" => {
"data" => data,
"pur" => "the purpose",
"exp" => "the expiration"
},
})
)
```
This improves performance in proportion to the size of the data:
**Benchmark**
```ruby
# frozen_string_literal: true
require "benchmark/ips"
require "active_support/all"
verifier = ActiveSupport::MessageVerifier.new("secret", serializer: JSON)
payloads = [
{ "content" => "x" * 100 },
{ "content" => "x" * 2000 },
{ "content" => "x" * 1_000_000 },
]
if ActiveSupport::Messages::Metadata.respond_to?(:use_message_serializer_for_metadata)
ActiveSupport::Messages::Metadata.use_message_serializer_for_metadata = true
end
Benchmark.ips do |x|
payloads.each do |payload|
x.report("generate ~#{payload["content"].size}B") do
$generated_message = verifier.generate(payload, purpose: "x")
end
x.report("verify ~#{payload["content"].size}B") do
verifier.verify($generated_message, purpose: "x")
end
end
end
puts
puts "Message size:"
payloads.each do |payload|
puts " ~#{payload["content"].size} bytes of data => " \
"#{verifier.generate(payload, purpose: "x").size} byte message"
end
```
**Before**
```
Warming up --------------------------------------
generate ~100B 1.578k i/100ms
verify ~100B 2.506k i/100ms
generate ~2000B 447.000 i/100ms
verify ~2000B 1.409k i/100ms
generate ~1000000B 1.000 i/100ms
verify ~1000000B 6.000 i/100ms
Calculating -------------------------------------
generate ~100B 15.807k (± 1.8%) i/s - 80.478k in 5.093161s
verify ~100B 25.240k (± 2.1%) i/s - 127.806k in 5.066096s
generate ~2000B 4.530k (± 2.4%) i/s - 22.797k in 5.035398s
verify ~2000B 14.136k (± 2.3%) i/s - 71.859k in 5.086267s
generate ~1000000B 11.673 (± 0.0%) i/s - 59.000 in 5.060598s
verify ~1000000B 64.372 (± 6.2%) i/s - 324.000 in 5.053304s
Message size:
~100 bytes of data => 306 byte message
~2000 bytes of data => 3690 byte message
~1000000 bytes of data => 1777906 byte message
```
**After**
```
Warming up --------------------------------------
generate ~100B 4.689k i/100ms
verify ~100B 3.183k i/100ms
generate ~2000B 2.722k i/100ms
verify ~2000B 2.066k i/100ms
generate ~1000000B 12.000 i/100ms
verify ~1000000B 11.000 i/100ms
Calculating -------------------------------------
generate ~100B 46.984k (± 1.2%) i/s - 239.139k in 5.090540s
verify ~100B 32.043k (± 1.2%) i/s - 162.333k in 5.066903s
generate ~2000B 27.163k (± 1.2%) i/s - 136.100k in 5.011254s
verify ~2000B 20.726k (± 1.7%) i/s - 105.366k in 5.085442s
generate ~1000000B 125.600 (± 1.6%) i/s - 636.000 in 5.064607s
verify ~1000000B 122.039 (± 4.1%) i/s - 616.000 in 5.058386s
Message size:
~100 bytes of data => 234 byte message
~2000 bytes of data => 2770 byte message
~1000000 bytes of data => 1333434 byte message
```
This optimization is only applied for recognized serializers that are
capable of serializing a `Hash`.
Additionally, because the optimization changes the message format, a
`config.active_support.use_message_serializer_for_metadata` option has
been added to disable it. The optimization is disabled by default, but
enabled with `config.load_defaults 7.1`.
Regardless of whether the optimization is enabled, messages using either
format can still be read.
In the case of a rolling deploy of a Rails upgrade, wherein servers that
have not yet been upgraded must be able to read messages from upgraded
servers, the optimization can be disabled on first deploy, then safely
enabled on a subsequent deploy.
Although Ruby provides `Base64.urlsafe_encode64` and
`Base64.urlsafe_decode64` methods, the technical term is "URL-safe",
with "URL" and "safe" as separate words.
For better readability, this commit renames the `:urlsafe` option for
`MessageEncryptor` and `MessageVerifier` to `:url_safe`.
urlsafe option was introduced to MessageVerifier in
09c3f36a962a7ffd350dfda643d2f980734cb5c9 but it can generate strings
containing padding character ("=") which is not urlsafe.
Fix not to pad when base64 encode.
MessageVerifier uses Base64.strict_encode64 and generated strings are
not urlsafe. Though the goal is to make MessageVerifier generated
strings urlsafe, We can not simply switch to Base64.urlsafe_encode64
because it will be a breaking change. Thus, as a first step, urlsafe
option is added to the MessageVerifier initializer.
* [ci skip] Refer to Rails.application.message_verifier in MessageVerifier docs
* [ci skip] Clarify authentication example
* [ci skip] Use meaningful example data for message verifier docs
* [ci skip] Link to Rails.application.message_verifier docs
* [ci skip] Clarify order of message signing and verification
* [ci skip] Re-order sections in order of expected use
* [ci skip] Recommend using a purpose to reduce risks
* [ci skip] More consistent parentheses
We can avoid using String#split by calculating the indexes of the
encrypted data, IV, and auth tag in the payload. This increases the
resistance of the solution against ill-formed payloads that don't
include the separator.
This is a follow up to the work in PR #42919.
ActiveSupport::MessageVerifier#verified is causing signed_message to be
split twice: first inside #valid_message? and then inside #verified.
This is ultimately unnecessary.
We can avoid String#split all together by calculating the indexes of the
data and the digest in the payload. This increases the resistance of the
solution against ill-formed payloads that don't include the separator.
In the docs [here](https://apidock.com/rails/v6.0.0/ActiveSupport/MessageVerifier) under the `Making messages expire` section, it was a little unclear what is `doowad` & `parcel` which ideally be a +string+ or a variable but in the complete documentation, we haven't used the reference of any variables except +token+, so these 2 should be a string.
Since Rails 6.0 will support Ruby 2.4.1 or higher
`# frozen_string_literal: true` magic comment is enough to make string object frozen.
This magic comment is enabled by `Style/FrozenStringLiteralComment` cop.
* Exclude these files not to auto correct false positive `Regexp#freeze`
- 'actionpack/lib/action_dispatch/journey/router/utils.rb'
- 'activerecord/lib/active_record/connection_adapters/sqlite3_adapter.rb'
It has been fixed by https://github.com/rubocop-hq/rubocop/pull/6333
Once the newer version of RuboCop released and available at Code Climate these exclude entries should be removed.
* Replace `String#freeze` with `String#-@` manually if explicit frozen string objects are required
- 'actionpack/test/controller/test_case_test.rb'
- 'activemodel/test/cases/type/string_test.rb'
- 'activesupport/lib/active_support/core_ext/string/strip.rb'
- 'activesupport/test/core_ext/string_ext_test.rb'
- 'railties/test/generators/actions_test.rb'
It's become clear to me that the use case is still a bit muddy
and the upgrade path is going to be tough for people to figure
out.
This attempts at understanding it better through documentation,
but still needs follow up work.
[ Michael Coyne & Kasper Timm Hansen ]
Both classes now have a rotate method where new instances are added for
each call. When decryption or verification fails the next rotation
instance is tried.
Corrects the time comparison to be `Time.now < time` which allows the user to
be set only when the current time is less than the 2 week window given in the
example.
I wrote a utility that helps find areas where you could optimize your program using a frozen string instead of a string literal, it's called [let_it_go](https://github.com/schneems/let_it_go). After going through the output and adding `.freeze` I was able to eliminate the creation of 1,114 string objects on EVERY request to [codetriage](codetriage.com). How does this impact execution?
To look at memory:
```ruby
require 'get_process_mem'
mem = GetProcessMem.new
GC.start
GC.disable
1_114.times { " " }
before = mem.mb
after = mem.mb
GC.enable
puts "Diff: #{after - before} mb"
```
Creating 1,114 string objects results in `Diff: 0.03125 mb` of RAM allocated on every request. Or 1mb every 32 requests.
To look at raw speed:
```ruby
require 'benchmark/ips'
number_of_objects_reduced = 1_114
Benchmark.ips do |x|
x.report("freeze") { number_of_objects_reduced.times { " ".freeze } }
x.report("no-freeze") { number_of_objects_reduced.times { " " } }
end
```
We get the results
```
Calculating -------------------------------------
freeze 1.428k i/100ms
no-freeze 609.000 i/100ms
-------------------------------------------------
freeze 14.363k (± 8.5%) i/s - 71.400k
no-freeze 6.084k (± 8.1%) i/s - 30.450k
```
Now we can do some maths:
```ruby
ips = 6_226k # iterations / 1 second
call_time_before = 1.0 / ips # seconds per iteration
ips = 15_254 # iterations / 1 second
call_time_after = 1.0 / ips # seconds per iteration
diff = call_time_before - call_time_after
number_of_objects_reduced * diff * 100
# => 0.4530373333993266 miliseconds saved per request
```
So we're shaving off 1 second of execution time for every 220 requests.
Is this going to be an insane speed boost to any Rails app: nope. Should we merge it: yep.
p.s. If you know of a method call that doesn't modify a string input such as [String#gsub](b0e2da69f0/lib/let_it_go/core_ext/string.rb (L37)) please [give me a pull request to the appropriate file](b0e2da69f0/lib/let_it_go/core_ext/string.rb (L37)), or open an issue in LetItGo so we can track and freeze more strings.
Keep those strings Frozen
