[dreamwidth/dw-free] 3fe67c: Add S3 support to BlobStore (#2220)

[github]

Branch: refs/heads/develop
Home: https://github.com/dreamwidth/dw-free
Commit: 3fe67cd546444c064a9bdf1efa526c68eb5331b9
https://github.com/dreamwidth/dw-free/commit/3fe67cd546444c064a9bdf1efa526c68eb5331b9
Author: Mark Smith <mark@qq.is>
Date: 2017-10-25 (Wed, 25 Oct 2017)

Changed paths:
M bin/checkconfig.pl
A bin/upgrading/migrate-mogilefs.pl
M bin/upgrading/migrate-userpics.pl
M cgi-bin/DW/BlobStore.pm
M cgi-bin/DW/BlobStore/LocalDisk.pm
M cgi-bin/DW/BlobStore/MogileFS.pm
M cgi-bin/DW/BlobStore/S3.pm
M etc/config-private.pl

Log Message:
-----------
Add S3 support to BlobStore (#2220)

* Add S3 support to BlobStore

This implements S3 support for BlobStore using the Paws SDK, which seems
to work fairly well. This also updates the BlobStore initialization code
to not be magical about what fallbacks are valid, it's now just an array
that can be configured directly by the admin.

AWS credentials are by default handled using IAM roles/instance
profiles, which is the secure way to do it if you are running a server
in EC2. This is highly recommended, but fallback support is provided for
specifing the keys directly (for now?).

* Update migrate-userpics.pl

* Add migration script

* Add stats to BlobStore

* Add cascading

This allows the case where store #1 fails to retrieve a file but store #2
succeeds, we then upload the file to store #1. This is a shim for
allowing easier migration between blobstores by letting it happen
organically for traffic.

You still must run a full migration of all old data, of course, but this
just helps the edge case without having to turn the site off and on.

* Update migrator