This allows calls of `ActiveStorage::Blob#url` to have more
interaction with the S3 Presigner, enabling, amongst other options,
custom S3 domain URL Generation.
Closes#42488
It's very hard to understand what happens with the following exception
URI::InvalidURIError:
bad URI(is not URI?): nil
that is raised when trying to generate a URL for Disk service without
setting the ActiveStorage::Current.host first.
This can happen when the ActiveStorage::SetCurrent is not included
in a controller, or when testing URL generation outside of the
controllers layer (eg. testing URL generation in a model).
Co-authored-by: elia <elia@schito.me>
Co-authored-by: filippo <dev@mailvore.com>
Extracted from https://github.com/rails/rails/pull/41543. In that PR, I introduce docs on how to test Active Storage in a variety of scenarios. One specific case is when you're doing parallel tests. To avoid the files uploaded / deleted in your tests from clashing with each other, you should have each test run with a separate storage directory.
Currently `ActiveStorage::Service::DiskService#root` is readonly, which means that to do that you need to change it using `instance_variable_set`. This isn't very elegant. Given parallel tests and Active Storage are both turned on by default in Rails I think it makes sense to have a nice API for this.
This PR just changes the `attr_reader` to an `attr_accessor`. With this change, the [code sample here](https://github.com/rails/rails/pull/41543/files#diff-e61fcdded4b6112b041ff388609c7032d09b8212ce209d67a7debc32f230d77aR1059-R1061) works.
`azure-storage-blob` 2.0.0 has been released.
https://rubygems.org/gems/azure-storage-blob/versions/2.0.0
According to this commit and changelog, `azure-storage-blob` 1.1.0 did not support Ruby 2.6 and higher.
As of right now, Rails master branch should support Ruby 2.5.0, 2.6.0 and 2.7.0
then it should be appropriate to unlock `azure-storage-blob` version.
252e3f06a5
This commit addresses this following error:
```ruby
% cd activestorage
% bundle exec rake test
Traceback (most recent call last):
14: from /Users/yahonda/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:5:in `<main>'
13: from /Users/yahonda/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:5:in `select'
12: from /Users/yahonda/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:17:in `block in <main>'
11: from /Users/yahonda/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/zeitwerk-2.3.0/lib/zeitwerk/kernel.rb:23:in `require'
10: from /Users/yahonda/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/zeitwerk-2.3.0/lib/zeitwerk/kernel.rb:23:in `require'
9: from /Users/yahonda/src/github.com/rails/rails/activestorage/test/service/azure_storage_service_test.rb:7:in `<top (required)>'
8: from /Users/yahonda/src/github.com/rails/rails/activestorage/test/service/azure_storage_service_test.rb:8:in `<class:AzureStorageServiceTest>'
7: from /Users/yahonda/src/github.com/rails/rails/activestorage/lib/active_storage/service.rb:51:in `configure'
6: from /Users/yahonda/src/github.com/rails/rails/activestorage/lib/active_storage/service/configurator.rb:8:in `build'
5: from /Users/yahonda/src/github.com/rails/rails/activestorage/lib/active_storage/service/configurator.rb:17:in `build'
4: from /Users/yahonda/src/github.com/rails/rails/activestorage/lib/active_storage/service/configurator.rb:30:in `resolve'
3: from /Users/yahonda/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/zeitwerk-2.3.0/lib/zeitwerk/kernel.rb:23:in `require'
2: from /Users/yahonda/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/zeitwerk-2.3.0/lib/zeitwerk/kernel.rb:23:in `require'
1: from /Users/yahonda/src/github.com/rails/rails/activestorage/lib/active_storage/service/azure_storage_service.rb:3:in `<top (required)>'
/Users/yahonda/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/rubygems_integration.rb:346:in `block (2 levels) in replace_gem': can't activate azure-storage-blob (~> 1.1), already activated azure-storage-blob-2.0.0. Make sure all dependencies are added to Gemfile. (Gem::LoadError)
11: from /Users/yahonda/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:5:in `<main>'
10: from /Users/yahonda/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:5:in `select'
9: from /Users/yahonda/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/rake_test_loader.rb:17:in `block in <main>'
8: from /Users/yahonda/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/zeitwerk-2.3.0/lib/zeitwerk/kernel.rb:23:in `require'
7: from /Users/yahonda/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/zeitwerk-2.3.0/lib/zeitwerk/kernel.rb:23:in `require'
6: from /Users/yahonda/src/github.com/rails/rails/activestorage/test/service/azure_storage_service_test.rb:7:in `<top (required)>'
5: from /Users/yahonda/src/github.com/rails/rails/activestorage/test/service/azure_storage_service_test.rb:8:in `<class:AzureStorageServiceTest>'
4: from /Users/yahonda/src/github.com/rails/rails/activestorage/lib/active_storage/service.rb:51:in `configure'
3: from /Users/yahonda/src/github.com/rails/rails/activestorage/lib/active_storage/service/configurator.rb:8:in `build'
2: from /Users/yahonda/src/github.com/rails/rails/activestorage/lib/active_storage/service/configurator.rb:17:in `build'
1: from /Users/yahonda/src/github.com/rails/rails/activestorage/lib/active_storage/service/configurator.rb:29:in `resolve'
/Users/yahonda/src/github.com/rails/rails/activestorage/lib/active_storage/service/configurator.rb:33:in `rescue in resolve': Missing service adapter for "AzureStorage" (RuntimeError)
rake aborted!
```
* Update Gemfile.lock to bump azure-storage-blob version
```
% bundle update --conservative faraday azure-storage-blob google-cloud-storage`
```
- How to find gem name to update by setting `gem "azure-storage-blob", "= 2.0.0"` temporarily
```ruby
% git diff
diff --git a/Gemfile b/Gemfile
index 5fdaceab2f..6be7dccf4b 100644
--- a/Gemfile
+++ b/Gemfile
@@ -84,7 +84,7 @@ end
group :storage do
gem "aws-sdk-s3", require: false
gem "google-cloud-storage", "~> 1.11", require: false
- gem "azure-storage-blob", require: false
+ gem "azure-storage-blob", "= 2.0.0", require: false # Use 2.0.0 temporarily to find which gems need bump
gem "image_processing", "~> 1.2"
end
% bundle install
Fetching gem metadata from https://rubygems.org/.........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies.....
Bundler could not find compatible versions for gem "faraday":
In snapshot (Gemfile.lock):
faraday (= 0.17.1)
In Gemfile:
azure-storage-blob (= 2.0.0) was resolved to 2.0.0, which depends on
azure-storage-common (~> 2.0) was resolved to 2.0.1, which depends on
faraday (~> 1.0)
google-cloud-storage (~> 1.11) was resolved to 1.25.0, which depends on
googleauth (~> 0.9) was resolved to 0.10.0, which depends on
faraday (~> 0.12)
Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
% git checkout -f Gemfile
% bundle update --conservative faraday azure-storage-blob google-cloud-storage
```
Services can be configured in `config/storage.yml` with a new key
`public: true | false` to indicate whether a service holds public
blobs or private blobs. Public services will always return a
permanent URL.
Deprecates `Blob#service_url` in favor of `Blob#url`.
* Force content-type to binary on service urls for relevant content types
We have a list of content types that must be forcibly served as binary,
but in practice this only means to serve them as attachment always. We
should also set the Content-Type to the configured binary type.
As a bonus: add text/cache-manifest to the list of content types to be
served as binary by default.
* Store content-disposition and content-type in GCS
Forcing these in the service_url when serving the file works fine for S3
and Azure, since these services include params in the signature.
However, GCS specifically excludes response-content-disposition and
response-content-type from the signature, which means an attacker can
modify these and have files that should be served as text/plain attachments
served as inline HTML for example. This makes our attempt to force
specific files to be served as binary and as attachment can be easily
bypassed.
The only way this can be forced in GCS is by storing
content-disposition and content-type in the object metadata.
* Update GCS object metadata after identifying blob
In some cases we create the blob and upload the data before identifying
the content-type, which means we can't store that in GCS right when
uploading. In these, after creating the attachment, we enqueue a job to
identify the blob, and set the content-type.
In other cases, files are uploaded to the storage service via direct
upload link. We create the blob before the direct upload, which happens
independently from the blob creation itself. We then mark the blob as
identified, but we have already the content-type we need without having
put it in the service.
In these two cases, then, we need to update the metadata in the GCS
service.
* Include content-type and disposition in the verified key for disk service
This prevents an attacker from modifying these params in the service
signed URL, which is particularly important when we want to force them
to have specific values for security reasons.
* Allow only a list of specific content types to be served inline
This is different from the content types that must be served as binary
in the sense that any content type not in this list will be always
served as attachment but with its original content type. Only types in
this list are allowed to be served either inline or as attachment.
Apart from forcing this in the service URL, for GCS we need to store the
disposition in the metadata.
Fix CVE-2018-16477.
The Azure gem uses `Azure::Core::Http::HTTPError` for everything:
checksum mismatch, missing object, network unavailable, and many more.
(https://www.rubydoc.info/github/yaxia/azure-storage-ruby/Azure/Core/Http/HTTPError).
Rescuing that class obscures all sorts of configuration errors. We
should check the type of error in those rescue blocks, and reraise when
needed.
The Azure gem uses `Azure::Core::Http::HTTPError` for everything:
checksum mismatch, missing object, network unavailable, and many more.
(https://www.rubydoc.info/github/yaxia/azure-storage-ruby/Azure/Core/Http/HTTPError).
Rescuing that class obscures all sorts of configuration errors. We
should check the type of error in those rescue blocks, and reraise when
needed.
`ActiveStorage::Blob#download` and `ActiveStorage::Blob#open` raise
`ActiveStorage::FileNotFoundError` when the corresponding file is missing
from the storage service. Services translate service-specific missing
object exceptions (e.g. `Google::Cloud::NotFoundError` for the GCS service
and `Errno::ENOENT` for the disk service) into
`ActiveStorage::FileNotFoundError`.
