74275accb8
13 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Jacopo
|
cebdf71a00 |
Fix decoding data encoded using a non-String purpose
Data encoded using a non-String purpose and `use_message_serializer_for_metadata == false` was incorrectly decoded, triggering a "mismatched purpose" error during decode. Fix this by ensuring that we compare both sides as a String. |
||
Rafael Mendonça França
|
fea9ad3337
|
Make sure the message format doesn't change
When `use_message_serializer_for_metadata` is false, the message format should be enveloped in the same way it was in Rails 7.0 to make sure we don't have inconsistent formats. |
||
Jonathan Hefner
|
af6d83521c |
Support :message_pack as message serializer
This commit adds support for `:message_pack` as a serializer for `MessageEncryptor` and `MessageVerifier`, and, consequently, as an option for `config.active_support.message_serializer`. The `:message_pack` serializer is implemented via `ActiveSupport::Messages::SerializerWithFallback` and can fall back to deserializing with `AS::JSON`. Additionally, the `:marshal`, `:json`, and `:json_allow_marshal` serializers can now fall back to deserializing with `AS::MessagePack`. This commit also adds support for `:message_pack_allow_marshal` as a serializer, which can fall back to deserializing with `Marshal` as well as `AS::JSON`. |
||
|
Jonathan Hefner
|
9fbfd8100b |
Unify Message{Encryptor,Verifier} serializer config
In #42843 and #42846, several config settings were added to control the default serializer for `MessageEncryptor` and `MessageVerifier`, and to provide a migration path from a default `Marshal` serializer to a default `JSON` serializer: * `config.active_support.default_message_encryptor_serializer` * Supports `:marshal`, `:hybrid`, or `:json`. * `config.active_support.default_message_verifier_serializer` * Supports `:marshal`, `:hybrid`, or `:json`. * `config.active_support.fallback_to_marshal_deserialization` * Affects `:hybrid` for both `MessageEncryptor` and `MessageVerifier`. * `config.active_support.use_marshal_serialization` * Affects `:hybrid` for both `MessageEncryptor` and `MessageVerifier`. This commit unifies those config settings into a single setting, `config.active_support.message_serializer`, which supports `:marshal`, `:json_allow_marshal`, and `:json` values. So, for example, ```ruby config.active_support.default_message_encryptor_serializer = :hybrid config.active_support.default_message_verifier_serializer = :hybrid config.active_support.fallback_to_marshal_deserialization = true config.active_support.use_marshal_serialization = false ``` becomes ```ruby config.active_support.message_serializer = :json_allow_marshal ``` and ```ruby config.active_support.default_message_encryptor_serializer = :hybrid config.active_support.default_message_verifier_serializer = :hybrid config.active_support.fallback_to_marshal_deserialization = false config.active_support.use_marshal_serialization = false ``` becomes ```ruby config.active_support.message_serializer = :json ``` This commit also replaces `ActiveSupport::JsonWithMarshalFallback` with `ActiveSupport::Messages::SerializerWithFallback`, which implements a generic mechanism for serializer fallback. The `:marshal` serializer uses this mechanism too, so ```ruby config.active_support.default_message_encryptor_serializer = :hybrid config.active_support.default_message_verifier_serializer = :hybrid config.active_support.fallback_to_marshal_deserialization = false config.active_support.use_marshal_serialization = true ``` becomes ```ruby config.active_support.message_serializer = :marshal ``` Additionally, the logging behavior of `JsonWithMarshalFallback` has been replaced with notifications which include the names of the intended and actual serializers, as well as the serialized and deserialized message data. This provides a more targeted means of tracking serializer fallback events. It also allows the user to "silence" such events, if desired, without an additional config setting. All of these changes make it easier to add migration paths for new serializers such as `ActiveSupport::MessagePack`. |
||
|
Jonathan Hefner
|
a2a6331451 |
Add ActiveSupport::MessagePack
`ActiveSupport::MessagePack` is a serializer that integrates with the
`msgpack` gem to serialize a variety of Ruby objects. `AS::MessagePack`
supports several types beyond the base types that `msgpack` supports,
including `Time` and `Range`, as well as Active Support types such as
`AS::TimeWithZone` and `AS::HashWithIndifferentAccess`.
Compared to `JSON` and `Marshal`, `AS::MessagePack` can provide a
performance improvement and message size reduction. For example, when
used with `MessageVerifier`:
```ruby
# frozen_string_literal: true
require "benchmark/ips"
require "active_support/all"
require "active_support/message_pack"
marshal_verifier = ActiveSupport::MessageVerifier.new("secret", serializer: Marshal)
json_verifier = ActiveSupport::MessageVerifier.new("secret", serializer: JSON)
asjson_verifier = ActiveSupport::MessageVerifier.new("secret", serializer: ActiveSupport::JSON)
msgpack_verifier = ActiveSupport::MessageVerifier.new("secret", serializer: ActiveSupport::MessagePack)
ActiveSupport::Messages::Metadata.use_message_serializer_for_metadata = true
expiry = 1.year.from_now
data = { bool: true, num: 123456789, string: "x" * 50 }
Benchmark.ips do |x|
x.report("Marshal") do
marshal_verifier.verify(marshal_verifier.generate(data, expires_at: expiry))
end
x.report("JSON") do
json_verifier.verify(json_verifier.generate(data, expires_at: expiry))
end
x.report("AS::JSON") do
asjson_verifier.verify(asjson_verifier.generate(data, expires_at: expiry))
end
x.report("MessagePack") do
msgpack_verifier.verify(msgpack_verifier.generate(data, expires_at: expiry))
end
x.compare!
end
puts "Marshal size: #{marshal_verifier.generate(data, expires_at: expiry).bytesize}"
puts "JSON size: #{json_verifier.generate(data, expires_at: expiry).bytesize}"
puts "MessagePack size: #{msgpack_verifier.generate(data, expires_at: expiry).bytesize}"
```
```
Warming up --------------------------------------
Marshal 1.206k i/100ms
JSON 1.165k i/100ms
AS::JSON 790.000 i/100ms
MessagePack 1.798k i/100ms
Calculating -------------------------------------
Marshal 11.748k (± 1.3%) i/s - 59.094k in 5.031071s
JSON 11.498k (± 1.4%) i/s - 58.250k in 5.066957s
AS::JSON 7.867k (± 2.4%) i/s - 39.500k in 5.024055s
MessagePack 17.865k (± 0.8%) i/s - 89.900k in 5.032592s
Comparison:
MessagePack: 17864.9 i/s
Marshal: 11747.8 i/s - 1.52x (± 0.00) slower
JSON: 11498.4 i/s - 1.55x (± 0.00) slower
AS::JSON: 7866.9 i/s - 2.27x (± 0.00) slower
Marshal size: 254
JSON size: 234
MessagePack size: 194
```
Additionally, `ActiveSupport::MessagePack::CacheSerializer` is a
serializer that is suitable for use as an `ActiveSupport::Cache` coder.
`AS::MessagePack::CacheSerializer` can serialize `ActiveRecord::Base`
instances, including loaded associations. Like `AS::MessagePack`, it
provides a performance improvement and payload size reduction:
```ruby
# frozen_string_literal: true
require "benchmark/ips"
require "active_support/message_pack"
ActiveRecord::Base.establish_connection(adapter: "sqlite3", database: ":memory:")
ActiveRecord::Schema.define do
create_table :posts, force: true do |t|
t.string :body
t.timestamps
end
create_table :comments, force: true do |t|
t.integer :post_id
t.string :body
t.timestamps
end
end
class Post < ActiveRecord::Base
has_many :comments
end
class Comment < ActiveRecord::Base
belongs_to :post
end
post = Post.create!(body: "x" * 100)
2.times { post.comments.create!(body: "x" * 100) }
post.comments.load
cache_entry = ActiveSupport::Cache::Entry.new(post)
Rails70Coder = ActiveSupport::Cache::Coders::Rails70Coder
CacheSerializer = ActiveSupport::MessagePack::CacheSerializer
Benchmark.ips do |x|
x.report("Rails70Coder") do
Rails70Coder.load(Rails70Coder.dump(cache_entry))
end
x.report("CacheSerializer") do
CacheSerializer.load(CacheSerializer.dump(cache_entry))
end
x.compare!
end
puts "Rails70Coder size: #{Rails70Coder.dump(cache_entry).bytesize}"
puts "CacheSerializer size: #{CacheSerializer.dump(cache_entry).bytesize}"
```
```
Warming up --------------------------------------
Rails70Coder 329.000 i/100ms
CacheSerializer 492.000 i/100ms
Calculating -------------------------------------
Rails70Coder 3.285k (± 1.7%) i/s - 16.450k in 5.008447s
CacheSerializer 4.895k (± 2.4%) i/s - 24.600k in 5.028803s
Comparison:
CacheSerializer: 4894.7 i/s
Rails70Coder: 3285.4 i/s - 1.49x slower
Rails70Coder size: 808
CacheSerializer size: 593
```
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
|
||
Étienne Barrié
|
b3c3bb6792 | Configure serialization of metadata per MessageVerifier object | ||
|
Jonathan Hefner
|
d3917f5fdd |
Use throw for message error handling control flow
There are multiple points of failure when processing a message with
`MessageEncryptor` or `MessageVerifier`, and there several ways we might
want to handle those failures. For example, swallowing a failure with
`MessageVerifier#verified`, or raising a specific exception with
`MessageVerifier#verify`, or conditionally ignoring a failure when
rotations are configured.
Prior to this commit, the _internal_ logic of handling failures was
implemented using a mix of `nil` return values and raised exceptions.
This commit reimplements the internal logic using `throw` and a few
precisely targeted `rescue`s. This accomplishes several things:
* Allow rotation of serializers for `MessageVerifier`. Previously,
errors from a `MessageVerifier`'s initial serializer were never
rescued. Thus, the serializer could not be rotated:
```ruby
old_verifier = ActiveSupport::MessageVerifier.new("secret", serializer: Marshal)
new_verifier = ActiveSupport::MessageVerifier.new("secret", serializer: JSON)
new_verifier.rotate(serializer: Marshal)
message = old_verifier.generate("message")
new_verifier.verify(message)
# BEFORE:
# => raises JSON::ParserError
# AFTER:
# => "message"
```
* Allow rotation of serializers for `MessageEncryptor` when using a
non-standard initial serializer. Similar to `MessageVerifier`, the
serializer could not be rotated when the initial serializer raised an
error other than `TypeError` or `JSON::ParserError`, such as
`Psych::SyntaxError` or a custom error.
* Raise `MessageEncryptor::InvalidMessage` from `decrypt_and_verify`
regardless of cipher. Previously, when a `MessageEncryptor` was using
a non-AEAD cipher such as AES-256-CBC, a corrupt or tampered message
would raise `MessageVerifier::InvalidSignature` due to reliance on
`MessageVerifier` for verification. Now, the verification mechanism
is transparent to the user:
```ruby
encryptor = ActiveSupport::MessageEncryptor.new("x" * 32, cipher: "aes-256-gcm")
message = encryptor.encrypt_and_sign("message")
encryptor.decrypt_and_verify(message.next)
# => raises ActiveSupport::MessageEncryptor::InvalidMessage
encryptor = ActiveSupport::MessageEncryptor.new("x" * 32, cipher: "aes-256-cbc")
message = encryptor.encrypt_and_sign("message")
encryptor.decrypt_and_verify(message.next)
# BEFORE:
# => raises ActiveSupport::MessageVerifier::InvalidSignature
# AFTER:
# => raises ActiveSupport::MessageEncryptor::InvalidMessage
```
* Support `nil` original value when using `MessageVerifier#verify`.
Previously, `MessageVerifier#verify` did not work with `nil` original
values, though both `MessageVerifier#verified` and
`MessageEncryptor#decrypt_and_verify` do:
```ruby
encryptor = ActiveSupport::MessageEncryptor.new("x" * 32)
message = encryptor.encrypt_and_sign(nil)
encryptor.decrypt_and_verify(message)
# => nil
verifier = ActiveSupport::MessageVerifier.new("secret")
message = verifier.generate(nil)
verifier.verified(message)
# => nil
verifier.verify(message)
# BEFORE:
# => raises ActiveSupport::MessageVerifier::InvalidSignature
# AFTER:
# => nil
```
* Improve performance of verifying a message when it has expired and one
or more rotations have been configured:
```ruby
# frozen_string_literal: true
require "benchmark/ips"
require "active_support/all"
verifier = ActiveSupport::MessageVerifier.new("new secret")
verifier.rotate("old secret")
message = verifier.generate({ "data" => "x" * 100 }, expires_at: 1.day.ago)
Benchmark.ips do |x|
x.report("expired message") do
verifier.verified(message)
end
end
```
__Before__
```
Warming up --------------------------------------
expired message 1.442k i/100ms
Calculating -------------------------------------
expired message 14.403k (± 1.7%) i/s - 72.100k in 5.007382s
```
__After__
```
Warming up --------------------------------------
expired message 1.995k i/100ms
Calculating -------------------------------------
expired message 19.992k (± 2.0%) i/s - 101.745k in 5.091421s
```
Fixes #47185.
|
||
|
Jonathan Hefner
|
91bb5da5fc |
Avoid double serialization of message data
Prior to this commit, messages with metadata were always serialized in
the following way:
```ruby
Base64.strict_encode64(
ActiveSupport::JSON.encode({
"_rails" => {
"message" => Base64.strict_encode64(
serializer.dump(data)
),
"pur" => "the purpose",
"exp" => "the expiration"
},
})
)
```
in which the message data is serialized and URL-encoded twice.
This commit changes message serialization such that, when possible, the
data is serialized and URL-encoded only once:
```ruby
Base64.strict_encode64(
serializer.dump({
"_rails" => {
"data" => data,
"pur" => "the purpose",
"exp" => "the expiration"
},
})
)
```
This improves performance in proportion to the size of the data:
**Benchmark**
```ruby
# frozen_string_literal: true
require "benchmark/ips"
require "active_support/all"
verifier = ActiveSupport::MessageVerifier.new("secret", serializer: JSON)
payloads = [
{ "content" => "x" * 100 },
{ "content" => "x" * 2000 },
{ "content" => "x" * 1_000_000 },
]
if ActiveSupport::Messages::Metadata.respond_to?(:use_message_serializer_for_metadata)
ActiveSupport::Messages::Metadata.use_message_serializer_for_metadata = true
end
Benchmark.ips do |x|
payloads.each do |payload|
x.report("generate ~#{payload["content"].size}B") do
$generated_message = verifier.generate(payload, purpose: "x")
end
x.report("verify ~#{payload["content"].size}B") do
verifier.verify($generated_message, purpose: "x")
end
end
end
puts
puts "Message size:"
payloads.each do |payload|
puts " ~#{payload["content"].size} bytes of data => " \
"#{verifier.generate(payload, purpose: "x").size} byte message"
end
```
**Before**
```
Warming up --------------------------------------
generate ~100B 1.578k i/100ms
verify ~100B 2.506k i/100ms
generate ~2000B 447.000 i/100ms
verify ~2000B 1.409k i/100ms
generate ~1000000B 1.000 i/100ms
verify ~1000000B 6.000 i/100ms
Calculating -------------------------------------
generate ~100B 15.807k (± 1.8%) i/s - 80.478k in 5.093161s
verify ~100B 25.240k (± 2.1%) i/s - 127.806k in 5.066096s
generate ~2000B 4.530k (± 2.4%) i/s - 22.797k in 5.035398s
verify ~2000B 14.136k (± 2.3%) i/s - 71.859k in 5.086267s
generate ~1000000B 11.673 (± 0.0%) i/s - 59.000 in 5.060598s
verify ~1000000B 64.372 (± 6.2%) i/s - 324.000 in 5.053304s
Message size:
~100 bytes of data => 306 byte message
~2000 bytes of data => 3690 byte message
~1000000 bytes of data => 1777906 byte message
```
**After**
```
Warming up --------------------------------------
generate ~100B 4.689k i/100ms
verify ~100B 3.183k i/100ms
generate ~2000B 2.722k i/100ms
verify ~2000B 2.066k i/100ms
generate ~1000000B 12.000 i/100ms
verify ~1000000B 11.000 i/100ms
Calculating -------------------------------------
generate ~100B 46.984k (± 1.2%) i/s - 239.139k in 5.090540s
verify ~100B 32.043k (± 1.2%) i/s - 162.333k in 5.066903s
generate ~2000B 27.163k (± 1.2%) i/s - 136.100k in 5.011254s
verify ~2000B 20.726k (± 1.7%) i/s - 105.366k in 5.085442s
generate ~1000000B 125.600 (± 1.6%) i/s - 636.000 in 5.064607s
verify ~1000000B 122.039 (± 4.1%) i/s - 616.000 in 5.058386s
Message size:
~100 bytes of data => 234 byte message
~2000 bytes of data => 2770 byte message
~1000000 bytes of data => 1333434 byte message
```
This optimization is only applied for recognized serializers that are
capable of serializing a `Hash`.
Additionally, because the optimization changes the message format, a
`config.active_support.use_message_serializer_for_metadata` option has
been added to disable it. The optimization is disabled by default, but
enabled with `config.load_defaults 7.1`.
Regardless of whether the optimization is enabled, messages using either
format can still be read.
In the case of a rolling deploy of a Rails upgrade, wherein servers that
have not yet been upgraded must be able to read messages from upgraded
servers, the optimization can be disabled on first deploy, then safely
enabled on a subsequent deploy.
|
||
|
Jonathan Hefner
|
b95fe24a99 |
Factor rotator tests into dedicated classes
This commit factors tests related to `ActiveSupport::Messages::Rotator` into `MessageEncryptorRotatorTest` and `MessageVerifierRotatorTest` classes, with a shared `MessageRotatorTests` module. In particular, this: * Replaces the `test_rotating_serializer` test in the `MessageEncryptorTest` class because it did not actually rotate the serializer. * Removes the `test_on_rotation_is_called_and_returns_modified_messages` test from the `MessageEncryptorWithJsonSerializerTest` class because its only purpose is to override a test of the same name in the `MessageEncryptorTest` parent class, because the expected value must use string keys instead of symbol keys due to the serializer. * Removes the `test_on_rotation_can_be_passed_at_the_constructor_level` test from the `MessageEncryptorWithJsonSerializerTest` class for the same reason as above. * Removes the `test_on_rotation_option_takes_precedence_over_the_one_given_in_constructor` test from the `MessageEncryptorWithJsonSerializerTest` class for the same reason as above. * Removes the `test_on_rotation_is_called_and_verified_returns_message` test from the `JsonSerializeMarshalFallbackMessageVerifierTest` class because it is rotating the secret and digest, and is not specific to the serializer. * Removes the `test_on_rotation_is_called_and_verified_returns_message` test from the `DefaultMarshalSerializerMessageVerifierTest` class for the same reason as above. * Adds thorough test coverage for the `on_rotation` option for both `MessageVerifier` and `MessageEncryptor` (previous test coverage was mostly just for `MessageEncryptor`), including coverage for when the `on_rotation` proc should _not_ be called. * Adds test coverage for rotating the `url_safe` option, for both `MessageVerifier` and `MessageEncryptor`. |
||
|
Jonathan Hefner
|
c103a4328a |
Refactor message metadata tests
Prior to this commit, there were several places to potentially add a new message metadata test: * `SharedMessageMetadataTests` module * `MessageEncryptorMetadataTest` class (includes `SharedMessageMetadataTests`) * `MessageEncryptorMetadataMarshalTest` class (subclasses `MessageEncryptorMetadataTest`) * `MessageEncryptorMetadataJSONTest` class (subclasses `MessageEncryptorMetadataTest`) * `MessageEncryptorMetadataJsonWithMarshalFallbackTest` class (subclasses `MessageEncryptorMetadataTest`) * `MessageVerifierMetadataTest` class (includes `SharedMessageMetadataTests`) * `MessageVerifierMetadataMarshalTest` class (subclasses `MessageVerifierMetadataTest`) * `MessageVerifierMetadataJsonWithMarshalFallbackTest` class (subclasses `MessageVerifierMetadataTest`) * `MessageVerifierMetadataJsonTest` class (subclasses `MessageVerifierMetadataTest`) * `MessageVerifierMetadataCustomJSONTest` class (subclasses `MessageVerifierMetadataTest`) * `MessageEncryptorMetadataNullSerializerTest` class (subclasses `MessageVerifierMetadataTest`) This commit refactors the message metadata tests, reducing the list to: * `MessageMetadataTests` module * `MessageEncryptorMetadataTest` class (includes `MessageMetadataTests`) * `MessageVerifierMetadataTest` class (includes `MessageMetadataTests`) This makes it easier to add new tests, as well as new testing scenarios (e.g. new encryptor / verifier configurations). Additionally, this commit fixes two tests that were ineffective: * The `test_passing_expires_in_less_than_a_second_is_not_expired` test (which is now simply a part of the "message expires with :expires_in" test) did not use the `with_usec: true` option. Therefore, the time resulting from `travel 0.5.seconds` was truncated, effectively traveling 0 seconds. * The `test_verify_with_use_standard_json_time_format_as_false` test (which is now named "expiration works with ActiveSupport.use_standard_json_time_format = false") did not specify an expiration time. Therefore, the conditional branch that it was supposed to test was never exercised. |
||
Michael Grosser
|
203998c916
|
allow running each test with pure ruby path/to/test.rb
also: - makes test dependencies obvious - makes tests runnable from within subfolders |
||
Kasper Timm Hansen
|
f9a6c00dcc
|
Fix RotationConfiguration test and remove nil-kind rotates. | ||
Michael Coyne
|
39f8ca64ce |
Add key rotation message Encryptor and Verifier
Both classes now have a rotate method where new instances are added for each call. When decryption or verification fails the next rotation instance is tried. |