How to interact with your project’s cloud media storage¶
Your cloud project’s media file storage is held on an S3 service - typically Amazon Web Services’s S3 service, or another S3 provider. Currently, most projects use Amazon’s own S3 service, or Exoscale for projects in our Swiss region.
Locally, your projects store their media in the /data/media
directory, and you can interact
with those directly. Then, you can use the Divio tools to push and pull media to the Cloud if required.
Occasionally you may need direct access to the S3 storage bucket for your project. You can manage this using a client of your choice that supports S3 and the particular storage provider.
Interact with your project’s cloud S3 storage¶
Warning
Note that S3 file operations tend to be destructive and do not necessarily have the same behaviours you may be used to from other models, such as FTP. It’s important that you know what you are doing and understand the consequences of any actions or commands.
Obtain your storage access details¶
Use the Divio CLI to obtain the DEFAULT_STORAGE_DSN
for the environment, for example:
divio project env-vars -s test --all --get DEFAULT_STORAGE_DSN
or:
divio project env-vars -s test --all --get DEFAULT_STORAGE_DSN --remote-id <site id>
if you are in a different directory.
See how to read environment variables.
This value contains the details you will need to use with a file transfer client for access to the
storage bucket. The two examples below show which sections of the DSN correspond to the different
parameters, for the hosts s3.amazonaws.com
and sos.exo.io
:
s3://AKAIIE7LUT6ODIJA:TZJYGCfUZheXG%2BwabbotgBs6d2lxZW06OIbD@example-test-68564d3f78d04c5f-8f20b19.aldryn-media.io.s3-eu-central-1.amazonaws.com/?domain=example-test-68564d3f78d04c5f-8f20b19.aldryn-media.io
key secret bucket name region endpoint
s3://EXO52e55b187195d:iITF12F1tim9zBxITexrvL_bAghgK_z4w1hEuu@example-test-765482644ac540dbb23367cf3837580b-f0596a8.sos-ch-dk-2.exo.io/?auth=s3
The key identifies you as a user.
The secret may contain some symbols encoded as hexadecimal values, and you will need to change them back before using them:
%2B
must be changed to+
%2F
must be changed to/
For any other values beginning with %
use a conversion table.
The bucket name identifies the resource you wish to work with.
The region is contained in the endpoint, the S3 host name. Sometimes it may be implicit, as
in the case of Amazon’s default us-east-1
:
Provider |
Endpoint |
Region |
Location |
---|---|---|---|
Amazon |
|
|
US East (N. Virginia) |
|
|
EU (Frankfurt) |
|
|
|
EU (London) |
|
Exoscale |
|
|
Switzerland |
See Amazon’s S3 regions table for more information about regions and their names.
The endpoint is the address that the client will need to connect to.
Note down the parameters¶
Copy and paste each of these parameters into a text file, so you have them ready for use. Now that you have obtained the connection parameters, you can use them to connect with the client of your choice.
Choose a client¶
How-to guides are provided below for connecting to our storage using:
Connect using AWS CLI¶
AWS CLI documentation is Amazon’s official S3 client. It’s a free, Python-based application.
Install and configure AWS CLI¶
Run:
pip install awscli
aws configure
You will be prompted for some of the storage access parameters values, extracted from the DSN, that you copied earlier.
AWS Access Key ID - key
AWS Secret Access Key - secret key
Default region name - storage region
Default output format - leave blank
Interact with your storage¶
Run aws s3
followed by options, commands and parameters. For example, to list the contents of a
bucket:
➜ aws s3 ls example-test-68564d3f78d0935f-8f20b19.aldryn-media.io
PRE filer_public/
PRE filer_public_thumbnails/
Or, to copy (cp
) a file from your own computer to S3:
➜ aws s3 cp example.png s3://example-test-68564d3f78d04c5f-8f20b19.aldryn-media.io/example.png
upload: ./example.png to s3://example-test-68564d3f78d04c5f-8f20b19.aldryn-media.io/example.png
Using AWS CLI with other providers
For non-AWS providers, such as Exoscale, you will need to add the --url-endpoint
option to
the command, as the AWS CLI assumes an endpoint on .amazonaws.com/
. For the Exoscale
example above, you would use:
aws s3 --endpoint-url=https://sos-ch-dk-2.exo.io [...]
Note that the scheme (typically https://
) must be included.
Additional usage information¶
Run aws s3 help
for more information on commands, or refer to the AWS CLI Command Reference. The AWS CLI can maintain
multiple profiles and offers other features but it’s beyond the scope of this documentation to
explain that here.
The aws configure
command stores the configuration in ~/.aws
.
Connect using s3cmd¶
S3cmd is a free Python-based command line tool and client for uploading, retrieving and managing data in Amazon S3 and other cloud storage service providers that use the S3 protocol.
Install and configure s3cmd¶
Run:
pip install s3cmd
s3cmd --configure
You will be prompted for some of the storage access parameters values, extracted from the DSN, that you copied earlier:
Access Key - enter the key from the DSN
Secret Key - enter the secret key from the DSN
Default Region - enter the storage region
S3 Endoint - enter the endpoint from the DSN
All other settings can be left untouched.
When you have entered the values, s3cmd will offer to test a connection with them (note that when using AWS, this will fail - ignore this).
Interact with your storage¶
Run s3cmd
followed by options, commands and parameters. For example, to list the contents of a
bucket:
s3cmd ls s3://example-test-68564d3f78d04c5f-8f20b19.aldryn-media.io
Note that the scheme (s3://
) is required in front of the bucket name.
Additional usage information¶
Run s3cmd
for more information on commands, or refer to Usage.
Using s3cmd
you can take advantage of --recursive
properties for iterating over the entire
bucket contents; however it’s beyond the scope of this documentation to explain this or other
features here.
s3cmd --configure
creates a configuration file at ~/.s3cfg
.
Connect using Transmit¶
Install the Transmit file transfer application for Macintosh.
Create a new connection. You will need to enter some of the storage access parameters values, extracted from the DSN, that you copied earlier:
Setting |
Value |
---|---|
Protocol |
Amazon S3 |
Address |
endpoint |
Access Key ID |
key |
Password |
secret key |
Remote Path |
bucket name |
Cyberduck¶
Install Cyberduck.
Create a new bookmark (note that you cannot simply use the Open Connection dialog, because this will not allow you to provide the required bucket name in order to proceed). You will be prompted for some of the storage access parameters values, extracted from the DSN, that you copied earlier:
Setting |
Value |
---|---|
Protocol |
Amazon S3 |
Server |
endpoint |
Access Key ID |
key |
Path (in More Options) |
bucket name |
On attempting to connect, you will be prompted for the Secret Access Key; use the secret key.
For Exoscale (Divio Swiss region) deployments, you can also download and install the Exoscale profile for Cyberduck, which includes some prepared configuration.
Use the Divio CLI for local access to Cloud storage¶
The project’s media files can be found in the /data/media
directory, and
can be managed and manipulated in the normal way on your own computer.
Be aware that if you edit project files locally, your operating system may save some hidden files. When you push your media to the cloud, these hidden files will be pushed too. This will however not usually present a problem.
Pushing and pulling media files¶
The Divio CLI includes pull
and push
commands that target the test or live server as
required.
Warning
Note that all push and pull operations completely replace all files at
the destination, and do not perform any merges of assets. Locally, the
/data/media
directory will be deleted and replaced; on the cloud, the
entire bucket will be replaced.
Limitations¶
You may encounter some file transfer size limitations when pushing and pulling media using the Divio CLI. Interacting directly with the S3 storage bucket is a way around this.
It can also be much faster, and allows selective changes to files in the system.
Configuring S3 buckets¶
Storage ACLs (Access Control Lists)¶
When uploading files to your storage, you may need to specify the ACLs explicitly - in effect, the file permissions - on the files. If you don’t set the correct ACLs, you may find that attempts to retrieve them (for example in a web browser) give an “access denied” error.
On AWS S3, the public-read ACL needs to be set (by default it’s private). This is the ACL required for general use.
For example, you can use --acl public-read
as a flag for operations such as cp
, or aws
s3api put-object-acl
and aws s3api get-object-acl
to set set and get ACLs on existing objects.
Enable CORS¶
CORS (cross-origin resource sharing) is a mechanism that allows resources on one domain to be served when requested by a page on another domain.
These requests are blocked by default by S3 media storage; when a request is blocked, you’ll see an error reported in the browser console:
Access to XMLHttpRequest at 'https://example.divio-media.com/images/image.jpg' from origin
'https://example.us.aldryn.io' has been blocked by CORS policy: No
'Access-Control-Allow-Origin' header is present on the requested resource.
In order to resolve this, the storage bucket needs to be configured to allow requests from a different origin.
This can be done using the AWS CLI’s S3 API tool.
As described above, obtain the DEFAULT_STORAGE_DSN for the environment (and if required, run aws configure).
Now you can check for any existing CORS configuration:
aws s3api get-bucket-cors --bucket <bucket-name>
You will receive a The CORS configuration does not exist
error if one is not yet present.
A CORS configuration is specified in JSON. It’s beyond the scope of this documentation to outline how your bucket should be configured for CORS; see AWS’s own Configuring and using cross-origin resource sharing documentation for more.
However an example that allows GET
and HEAD
requests from any origin would be:
{
"CORSRules": [
{
"AllowedHeaders": ["*"],
"AllowedMethods": ["GET", "HEAD"],
"AllowedOrigins": ["*"],
"MaxAgeSeconds": 3000
}
]
}
Save your configuration as a file (cors.json
) and use the API to upload it to the bucket:
aws s3api put-bucket-cors --bucket <bucket-name> --cors-configuration file://cors.json
See the AWS S3 CLI API documentation for further information about available operations.
Note
You may receive a GetBucketCors operation: Access Denied
error when attempting to use the
S3 API with some older buckets. If this occurs, but operations such as aws s3 ls
work as
expected, then your bucket will need to be updated. Please contact Divio support so that we can
do this for you.