Fix: #36065
The IamgeAnalyzer passes a image to ImageMagick without checking if the
image is supported by ImageMagick. This patch checks that image is
supported and if not logs an error and returns an empty hash instead of
raising an error. This is the same error handling we do when we
encounter a LoadError when mini_magick is not installed.
The minimum character length is now 24 characters since it's inherited
from ActiveRecord::Base, but the default behaviour is still using 28
characters.
* Allow plugins to access the global service and alternative service configs before ActiveStorage::Blob loads.
* Make ActiveStorage.service_configurations the default second argument to ActiveStorage::Service.configure. Plugins that just want to use an alternative service defined in config/storage.yml needn't pass in the config themselves.
Sample example ->
Before:
prathamesh@Prathameshs-MacBook-Pro-2 blog *$ rails server thin
DEPRECATION WARNING: Passing the Rack server name as a regular argument is deprecated
and will be removed in the next Rails version. Please, use the -u
option instead.
After:
prathamesh@Prathameshs-MacBook-Pro-2 squish_app *$ rails server thin
DEPRECATION WARNING: Passing the Rack server name as a regular argument is deprecated and will be removed in the next Rails version. Please, use the -u option instead.
Consider a model with `One` and `Many` attachments configured:
class User < ActiveRecord::Base
has_one_attached :avatar
has_many_attached :highlights
end
=== One Attachment
After attaching `One` attachment (`:avatar`), we can see that the associated
`_blob` record (`:avatar_blob`) still returns as `nil`.
user.avatar.attach(blob)
user.avatar_attachment.present? => true
user.avatar_blob.present? => false # Incorrect!
This is a false negative. It happens because after the attachment and blob
are built:
1. The record already has its `_blob` association loaded, as `nil`
2. the `::Attachment` is associated with the record but the `::Blob` only gets
associated with the `::Attachment`, not the record itself
In reality, the blob does in fact exist. We can verify this as follows:
user.avatar.attach(blob)
user.avatar_attachment.blob.present? => true # Blob does exist!
The fix in this change is to simply assign the `::Blob` when assigning
the `::Attachment`. After this fix is applied, we correctly observe:
user.avatar.attach(blob)
user.avatar_attachment.present? => true
user.avatar_blob.present? => true # Woohoo!
=== Many Attachments
We don't see this issue with `Many` attachments because the `_blob` association
is already loaded as part of attaching more/newer blobs.
user.highlights.attach(blob)
user.highlights_attachments.any? => true
user.highlights_blobs.any? => true
Since b21f50d8ae36d9b50b673579e17bccbe55363b34, requiring active_storage
on its own has failed with the following error:
activestorage/lib/active_storage.rb:55:in `<module:ActiveStorage>': undefined method `minutes' for 5:Integer (NoMethodError)
We need this in order to be able to add this migration for users that
use ActiveStorage during update their apps from Rails 5.2 to Rails 6.0.
Related to #33405
`rake app:update` should update active_storage
`rake app:update` should execute `rake active_storage:update`
if it is used in the app that is being updated.
It will add new active_storage's migrations to users' apps during update Rails.
Context https://github.com/rails/rails/pull/33405#discussion_r204239399
Also, see a related discussion in the Campfire:
https://3.basecamp.com/3076981/buckets/24956/chats/12416418@1236713081
We had a discussion on the Core team and we don't want to expose this information
as a JSON endpoint and not by default.
It doesn't make sense to expose this JSON locally and this controller is only
accessible in dev, so the proposed access from a production app seems off.
This reverts commit 8eaffe7e89719ac62ff29c2e4208cfbeb1cd1c38, reversing
changes made to b6e4305c3bca4c673996d0af9db0f4cfbf50215e.
If there is not a `csrf-token` meta tag in the document, the blob record
XHR was including an `X-CSRF-Token` header set to the string
"undefined." Instead of setting it to undefined, it should not be
included in the absence of a meta tag.
Generally followed the pattern for https://github.com/rails/rails/pull/32034
* Removes needless CI configs for 2.4
* Targets 2.5 in rubocop
* Updates existing CHANGELOG entries for fewer merge conflicts
* Removes Hash#slice extension as that's inlined on Ruby 2.5.
* Removes the need for send on define_method in MethodCallAssertions.
Since 06ab7b27ea1c1ab357085439abacdb464f6742bf,
`GCSServiceTest#test_signed_URL_response_headers` is broken.
https://travis-ci.org/rails/rails/jobs/460454477#L7084-L7087
This seems to be due to lack of `content_type` at upload.
This is solved by specifying `conten_type`.
However, since the same content is also tested with `test_upload_with_content_type`,
it will be duplicated content, so I think that can remove `test_signed_URL_response_headers`.
* Force content-type to binary on service urls for relevant content types
We have a list of content types that must be forcibly served as binary,
but in practice this only means to serve them as attachment always. We
should also set the Content-Type to the configured binary type.
As a bonus: add text/cache-manifest to the list of content types to be
served as binary by default.
* Store content-disposition and content-type in GCS
Forcing these in the service_url when serving the file works fine for S3
and Azure, since these services include params in the signature.
However, GCS specifically excludes response-content-disposition and
response-content-type from the signature, which means an attacker can
modify these and have files that should be served as text/plain attachments
served as inline HTML for example. This makes our attempt to force
specific files to be served as binary and as attachment can be easily
bypassed.
The only way this can be forced in GCS is by storing
content-disposition and content-type in the object metadata.
* Update GCS object metadata after identifying blob
In some cases we create the blob and upload the data before identifying
the content-type, which means we can't store that in GCS right when
uploading. In these, after creating the attachment, we enqueue a job to
identify the blob, and set the content-type.
In other cases, files are uploaded to the storage service via direct
upload link. We create the blob before the direct upload, which happens
independently from the blob creation itself. We then mark the blob as
identified, but we have already the content-type we need without having
put it in the service.
In these two cases, then, we need to update the metadata in the GCS
service.
* Include content-type and disposition in the verified key for disk service
This prevents an attacker from modifying these params in the service
signed URL, which is particularly important when we want to force them
to have specific values for security reasons.
* Allow only a list of specific content types to be served inline
This is different from the content types that must be served as binary
in the sense that any content type not in this list will be always
served as attachment but with its original content type. Only types in
this list are allowed to be served either inline or as attachment.
Apart from forcing this in the service URL, for GCS we need to store the
disposition in the metadata.
Fix CVE-2018-16477.
fragment caching was refactored in (I think 5.2) and by default doesn't log cache info
this is confusing in development where rails dev:cache now turns on caching, but doesn't show any different logging output
better to enable debugging by default for dev - and let people turn it off if preferred
Fix an issue in ActiveStorage where a direct upload to disk storage
would fail due to a content type mismatch if the file was uploaded using
a mime-type synonym.
* Use Webpacker by default on new apps
* Stop including coffee-rails by default
* Drop using a js_compressor by default
* Drop extra test for coffeescript inclusion by default
* Stick with skip_javascript to signify skipping webpack
* Don't install a JS runtime by default any more
* app/javascript will be the new default directory for JS
* Make it clear that this is just for configuring the default Webpack framework setup now
* Start using the Webpack tag in the default layout
* Irrelevant test
* jQuery is long gone
* Stop having asset pipeline compile default application.js
* Add rails-ujs by default to the Webpack setup
* Add Active Storage JavaScript to application.js pack by default
* Consistent quoting
* Add Turbolinks to default pack
* Add Action Cable to default pack
Need some work on how to set the global consumer that channels will
work with. @javan?
* Require all channels by default and use a separate consumer stub
* Channel generator now targets Webpack style
* Update task docs to match new generator style
* Use uniform import style
* Drop the JS assets generator
It was barely helpful as it was. It’s no longer helpful in a Webpacked
world. Sayonara!
* Add app/javascript to the stats directories
* Simpler import style
Which match the other imports.
* Address test failures from dropping JS compilation (and compression)
* webpacker-default: Modify `AssetsGeneratorTest`
Before:
```
$ bin/test test/generators/assets_generator_test.rb
Run options: --seed 46201
F
Failure:
AssetsGeneratorTest#test_assets [/Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/assets_generator_test.rb:12]:
Expected file "app/assets/javascripts/posts.js" to exist, but does not
bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/assets_generator_test.rb:10
.
Finished in 0.031343s, 63.8101 runs/s, 95.7152 assertions/s.
2 runs, 3 assertions, 1 failures, 0 errors, 0 skips
```
After:
```
$ bin/test test/generators/assets_generator_test.rb
Run options: --seed 43571
..
Finished in 0.030370s, 65.8545 runs/s, 65.8545 assertions/s.
2 runs, 2 assertions, 0 failures, 0 errors, 0 skips
```
* webpacker-default: Modify `ChannelGeneratorTest`
Before:
```
$ bin/test test/generators/channel_generator_test.rb
Run options: --seed 8986
.F
Failure:
ChannelGeneratorTest#test_channel_with_multiple_actions_is_created [/Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:43]:
Expected file "app/assets/javascripts/channels/chat.js" to exist, but does not
bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:34
.F
Failure:
ChannelGeneratorTest#test_channel_is_created [/Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:29]:
Expected file "app/assets/javascripts/channels/chat.js" to exist, but does not
bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:22
E
Error:
ChannelGeneratorTest#test_cable_js_is_created_if_not_present_already:
Errno::ENOENT: No such file or directory @ apply2files - /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/fixtures/tmp/app/assets/javascripts/cable.js
bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:60
F
Failure:
ChannelGeneratorTest#test_channel_suffix_is_not_duplicated [/Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:87]:
Expected file "app/assets/javascripts/channels/chat.js" to exist, but does not
bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:80
F
Failure:
ChannelGeneratorTest#test_channel_on_revoke [/Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:77]:
Expected file "app/assets/javascripts/cable.js" to exist, but does not
bin/test /Users/ttanimichi/ghq/github.com/ttanimichi/rails/railties/test/generators/channel_generator_test.rb:68
Finished in 0.064384s, 108.7227 runs/s, 481.4861 assertions/s.
7 runs, 31 assertions, 4 failures, 1 errors, 0 skips
```
After:
```
$ bin/test test/generators/channel_generator_test.rb
Run options: --seed 44857
.......
Finished in 0.060243s, 116.1961 runs/s, 697.1764 assertions/s.
7 runs, 42 assertions, 0 failures, 0 errors, 0 skips
```
* Fix shared generator tests.
* webpacker-default: Modify `ControllerGeneratorTest`
The JS assets generator was dropped. ref. 46215b1794
* Revert "Simpler import style". It's currently failing with an error of "TypeError: undefined is not an object (evaluating '__WEBPACK_IMPORTED_MODULE_2_activestorage___default.a.start')". Waiting for @javan to have a look.
This reverts commit 5d3ebb71059f635d3756cbda4ab9752027e09256.
* require webpacker in test app
* Add webpacker without making the build hang/timeout. (#33640)
* use yarn workspaces to allow for installing unreleased packages and only generate js/bootsnap when required
* no longer need to have webpacker in env templates as webpacker moved this config to yml file
* Fix rubocop violation
* Got the test passing for the running scaffold
* update expected lines of code
* update middleware tests to account for webpacker
* disable js in plugins be default to get the tests passing (#34009)
* clear codeclimate report issues
* Anything newer than currently released is good
* Use Webpacker development version during development of Rails
* Edge should get development webpacker as well
* Add changelog entry for Webpacker change
The issue #32584 was fixed in #33405 by adding foreign key constraint
to the `active_storage_attachments` table for blobs.
This commit implements fix on app-level in order to ensure that users
can't delete a blob with attachments even if they don't have the foreign key constraint.
See a related discussion in the Campfire:
https://3.basecamp.com/3076981/buckets/24956/chats/12416418@1236718899
Note that, we should backport it to `5-2-stable` too.
Related to #33405
Applications can configure the route prefix prepended to the Active
Storage routes. By default this maintains the previous prefix
`/rails/active_storage` but supports custom prefixes.
Before this change the route for serving blobs is fixed to
`/rails/active_storage/blobs/:signed_id/*filename`. After this change
it's possible to configure the route to something like
`/files/blobs/:signed_id/*filename`.
The name of the minitest library is spelled that way: regular font, and
lowercase. Lowercase is used even at the beginning of sentences, see
http://docs.seattlerb.org/minitest/
I double-checked this with @zenspider too (thanks!).
The Azure gem uses `Azure::Core::Http::HTTPError` for everything:
checksum mismatch, missing object, network unavailable, and many more.
(https://www.rubydoc.info/github/yaxia/azure-storage-ruby/Azure/Core/Http/HTTPError).
Rescuing that class obscures all sorts of configuration errors. We
should check the type of error in those rescue blocks, and reraise when
needed.
The Azure gem uses `Azure::Core::Http::HTTPError` for everything:
checksum mismatch, missing object, network unavailable, and many more.
(https://www.rubydoc.info/github/yaxia/azure-storage-ruby/Azure/Core/Http/HTTPError).
Rescuing that class obscures all sorts of configuration errors. We
should check the type of error in those rescue blocks, and reraise when
needed.
`ActiveStorage::DiskController#show` generates a 404 Not Found response when
the requested file is missing from the disk service. It previously raised
`Errno::ENOENT`.
`ActiveStorage::Blob#download` and `ActiveStorage::Blob#open` raise
`ActiveStorage::FileNotFoundError` when the corresponding file is missing
from the storage service. Services translate service-specific missing
object exceptions (e.g. `Google::Cloud::NotFoundError` for the GCS service
and `Errno::ENOENT` for the disk service) into
`ActiveStorage::FileNotFoundError`.
This test no longer covers the behavior of ActiveStorage::PurgeJob. Attached blobs are ignored by ActiveStorage::Blob#purge as of 934fccd, which includes an equivalent model test.
Consider the following model definitions:
class User < ApplicationRecord
has_one_attached :avatar
end
class Group < ApplicationRecord
has_one_attached :avatar
end
If you attempt to reflect on the User model's avatar attachment via User.reflect_on_attachment, you could receive a reflection for the Group model's avatar attachment. Fix this by ensuring that each model class uses its own Hash object to track attachment reflections.
As discussed in #33203 rails command already looks for, and runs,
bin/rails if it is present.
We were mixing recommendations within guides and USAGE guidelines,
in some files we recommended using rails, in others bin/rails and
in some cases we even had both options mixed together.
The file `filename.rb` as mentioned in `require "active_storage/filename"`
belongs to the `app` folder while GCSService belongs to the lib folder.
Looking at the git blame, it was added in commit ccac681122 (diff-bda6a610ef1575b2c8458c96b7f12578)
where ActiveStorage::Filename was actually used. But it is no longer
required on master and therefore can be removed.
This allows anyone to use GCSService directly without enabling
ActiveStorage engine.
Don't include `ActiveJob::TestHelper` since there is no test that uses it.
Ensure removing of overridden User's methods.
Related to https://github.com/rails/rails/pull/33085#issuecomment-395548563
Module#remove_method is private in Ruby 2.4.
Related to fd0bd1bf682622f064ac437ceee4e1b2a6b6d3b9
In response to https://github.com/rails/rails/issues/32917
In the current implementation, ActiveStorage passes all options to the underlying processor,
including when a key has a value of false.
For example, passing:
```
avatar.variant(resize: "100x100", monochrome: false, flip: "-90")
```
will return a monochrome image (or an error, pending on ImageMagick configuration) because
it passes `-monochrome false` to the command (but the command line does not allow disabling
flags this way, as usually a user would omit the flag entirely to disable that feature).
This fix only passes those keys forward to the underlying processor if the value responds to
`present?`. In practice, this means that `false` or `nil` will be filtered out before going
to the processor.
One possible use case would be for a user to be able to apply different filters to an avatar.
The code might look something like:
```
variant_options = {
monochrome: params[:monochrome],
resize: params[:resize]
}
avatar.variant(*variant_options)
```
Obviously some sanitization may be beneficial in a real-world scenario, but this type of
configuration object could be used in many other places as well.
- Add removing falsy values from varaints to changelog
- The entirety of #image_processing_transformation inject block was wrapped in `list.tap`
to guard against the default `nil` being returned if no conditional was called.
- add test for explicitly true variant options
Generated attachment getter and setter methods are created within
the model's `GeneratedAssociationMethods` module to allow overriding
and composition using `super`.
Includes tests for new functionality.
Co-authored-by: Josh Susser <josh@hasmanythrough.com>
Co-authored-by: Jamon Douglas <terrildouglas@gmail.com>
### Steps to reproduce
Using Rails 5.2.0
When following this example:
http://api.rubyonrails.org/classes/ActiveStorage/Variant.html
`avatar.variant(resize: "100x100", monochrome: true, flip: "-90")`
### Expected behavior
Image should be rendered as flipped.
### Actual behavior
I get an error:
> failed with error: gm mogrify: Unrecognized option (-90).
### Fix:
According to: https://github.com/minimagick/minimagick the option should be called rotate:
`avatar.variant(resize: "100x100", monochrome: true, rotate: "-90")`
So **flip** changed to **rotate**.
### System configuration
**Rails version**: 5.2.0
**Ruby version**: ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-darwin17]
This adds a boolean argument called identify to ActiveStorage::Blob
methods #create_after_upload, #build_after_upload and #upload. It
allows a user to bypass the automatic content_type inference from
the io.
ImageAnalyzerTest and VideoAnalyzerTest are defining the same helper,
since both use `#create_file_blob` that is defined in TestHelper, it
makes sense to move `#extract_metadata_from` to that side.
Aws::S3::Object#get returns a response with object content wrapped in an
in-memory StringIO object. StringIO#read will return a copy of the
content, which is not necessary because we can return the content
directly using StringIO#string. This halves memory allocation of
S3Service#download, because we remove unnecessary content duplication.
ImageProcessing gem is a wrapper around MiniMagick and ruby-vips, and
implements an interface for common image resizing and processing. This
is the canonical image processing gem recommended in [Shrine], and
that's where it developed from. The initial implementation was extracted
from Refile, which also implements on-the-fly transformations.
Some features that ImageProcessing gem adds on top of MiniMagick:
* resizing macros
- #resize_to_limit
- #resize_to_fit
- #resize_to_fill
- #resize_and_pad
* automatic orientation
* automatic thumbnail sharpening
* avoids the complex and inefficient MiniMagick::Image class
* will use "magick" instead of "convert" on ImageMagick 7
However, the biggest feature of the ImageProcessing gem is that it has
an alternative implementation that uses libvips. Libvips is an
alternative to ImageMagick that can process images very rapidly (we've
seen up 10x faster than ImageMagick).
What's great is that the ImageProcessing gem provides the same interface
for both implementations. The macros are named the same, and the libvips
implementation does auto orientation and thumbnail sharpening as well;
only the operations/options specific to ImageMagick/libvips differ. The
integration provided by this PR should work for both implementations.
The plan is to introduce the ImageProcessing backend in Rails 6.0 as the
default backend and deprecate the MiniMagick backend, then in Rails 6.1
remove the MiniMagick backend.
VariantsController has been merged to RepresentationsController, this
PR fixes outdated references to VariantsController in ActiveStorage documentation.
Trying to pass the current request down to the service so that it can
create full urls instead of paths makes the API messy so use a model
based on ActiveSupport::CurrentAttributes to provide the current host
to services that need it (primarily the disk service).
Addresses rails/rails#32247
Add test that checks identify and analyze work in correct order
Break out direct upload test helper
Review changes for direct-upload test helper
The Active Storage service for Azure Storage has an option called `path`
that is ambiguous in meaning. It needs to be set to the primary blob
storage endpoint but that can be determined from the blobs client anyway.
To simplify the configuration this commit removes the `path` option and
gets the endpoint from the blobs client instead.
Closes#32225.
mutool is licensed under the Affero GPL, which has strict distribution requirements.
Poppler is licensed under the more liberal GPL, making it a good alternative for those who can't use mutool.
Since we started clearing the client-side blob's type in e0867b3, we no longer need to set a blank Content-Type header before issuing the direct upload request. Fixes that Safari 9 would combine the blank Content-Type header with the blank blob type to produce a Content-Type header containing a single comma, invalidating the request.
Since ActiveStorage::Blob::Representable unifies the idea of previews and
variants under one roof as representation, we may as well have the
controllers follow suit.
Thus ActiveStorage::RepresenationsController enters the fray. I've copied
the old tests for both previews and variants and unified those as well.
Prevent older versions of Chrome from appending a Content-Type header containing the Blob type, rendering the request invalid if we intend not to provide a Content-Type. This behavior was observed in Chrome 58.
Remove railties' changelog added by 7340596de45dc4c0f62a287b6acc4e71d8ee6c60
since it was backported to `5-2-stable` via ac99916fcf7bf27bb1519d4f7387c6b4c5f0463d
Remove activesupport's changelog added by 1077ae96b34b5a1dfbf10ee0c40b1ceb1eb6b30b
since it was backported to `5-2-stable` via a2b97e4ffef971607a1be8fc7909f099b6840f36
Remove activesupport's changelog added by 0d41a76d0c693000005d79456dee7f9299f5e8d4
since it was backported to `5-2-stable` via cdce6a709e1cbc98fff009effc3b1b3ce4c7e8db
Remove activestorage's changelog added by d57c52a385eb57c6ce8c6d124ab5e186f931d142
since it was backported to `5-2-stable` via 5292cdf59a2052c453d6016c69b90b790cbf2547
Follow up c113bdc9d0c2cffd535ca97aff85c4bdc46b11f6
`to_prepare` callbacks are run during initialization; using one here
meant that `ActiveStorage::Blob` would be loaded when the app boots,
which would in turn load `ActiveRecord::Base`.
By using a lazy load hook to configure `ActiveStorage::Blob` instead,
we can avoid loading `ActiveRecord::Base` unnecessarily.
* Global ignores at toplevel .gitignore
* Component-specific ignores in each toplevel directory
* Remove `actionview/test/tmp/.keep` for JRuby
```
rm actionview/test/tmp/ -fr
cd actionview/
bundle exec jruby -Itest test/template/digestor_test.rb
```
Related to #11743, #30392.
Closes#29978.
Active Storage is an engine which means its models, jobs and controllers
are autoloaded by Rails rather than Ruby. Unfortunately this means it's
subject to the same gotchas as applications, including this one:
http://guides.rubyonrails.org/v5.1.4/autoloading_and_reloading_constants.html#when-constants-aren-t-missed-qualified-references
On Ruby < 2.5, constants nested under classes can't be autoloaded by
Rails if a top level constant already exists with the same name.
To avoid clashing with constants defined in users' applications or gems,
we can use `require_dependency` to ensure that the nested constants are
loaded before they're used.
This allows activestorage users to ship smaller javascript bundles to
visitors using modern browsers, as demonstrated in this repository:
https://github.com/rmacklin/activestorage-es2015-build-example
In that example, the bundle shrinks by 5K (24%).
In addition to allowing smaller bundles for those who ship untranspiled
code to modern browsers, including the source code in the published
package can be useful in other ways:
1. Users can import individual modules rather than the whole library
2. As a result of (1), users can also monkey patch parts of
activestorage by importing the relevant module, modifying the
exported object, and then importing the rest of activestorage (which
would then use the patched object).
Note:
In order to allow the source code to be depended on rather than the
compiled code, we have to declare the external dependency on spark-md5
as a regular dependency, not a development dependency.
This means that even users who depend on the compiled code will have to
download this package. However, spark-md5 is a small package, so this
tradeoff seems worth it.
