Skip to main content

Working with your application's media storage in web applications

Introduction​

File storage on Divio applications is handled by dedicated storage systems entirely separate from the application. The available storage depends on the Divio cloudspace your application uses. Most applications use Amazon Web Services's S3 service, or another S3 provider. Others use Microsoft Azure blob storage.

In our architecture, the same application may be running in multiple parallel container, each with its own local file storage independent of each of the others. Moreover, this storage is not persistent, and exists only for as long as the lifetime of the container.

This means an application should not expect to save files to its local storage, and then expect to find them later.

Good implementations of cloud storage backends or plugins, for both S3 and Azure blog storage, exist for most mature web frameworks and applications languages.

For most Django applications, this won't require any additional work. Django is able to use multiple storage backends, all addressed through a common API. This is the safe and correct way to handle files in Django, so that applications can abstract from details of the storage implementation, and simply not need to know about it.

As long as an application uses Django's storage API, rather than attempting to manipulate Python File objects directly, it doesn't need to do anything differently.

An example configuration for AWS using boto3 would looks something like this:

AWS_STORAGE_BUCKET_NAME = os.environ.get('DEFAULT_STORAGE_BUCKET', '')
AWS_ACCESS_KEY_ID = os.environ.get('DEFAULT_STORAGE_ACCESS_KEY_ID', '')
AWS_SECRET_ACCESS_KEY = os.environ.get('DEFAULT_STORAGE_SECRET_ACCESS_KEY', '')
AWS_S3_CUSTOM_DOMAIN = os.environ.get('DEFAULT_STORAGE_CUSTOM_DOMAIN', '')
AWS_S3_REGION_NAME = os.environ.get('DEFAULT_STORAGE_REGION', '')
AWS_S3_OBJECT_PARAMETERS = {
    'ACL': 'public-read',
    'CacheControl': 'max-age=86400',
}
AWS_S3_FILE_OVERWRITE = False

# Default storage settings, with the staticfiles storage updated.
# See https://docs.djangoproject.com/en/5.1/ref/settings/#std-setting-STORAGES
STORAGE_BACKEND = "django.core.files.storage.FileSystemStorage"

if AWS_SECRET_ACCESS_KEY:
  STORAGE_BACKEND = "storages.backends.s3boto3.S3Boto3Storage"

STORAGES = {
    "default": {
        "BACKEND": STORAGE_BACKEND,
    },
    # ManifestStaticFilesStorage is recommended in production, to prevent
    # outdated JavaScript / CSS assets being served from cache
    # See https://docs.djangoproject.com/en/5.1/ref/contrib/staticfiles/#manifeststaticfilesstorage
    "staticfiles": {
        "BACKEND": "django.contrib.staticfiles.storage.ManifestStaticFilesStorage",
    },
}

Make sure to install the boto3 package into your application through pip install boto3.

Private file storage​

Our storage backend does not support private file storage (i.e. requiring authentication) on S3 objects.

If you need private storage, you can define an additional Django storage backend in your application, which sets S3 objects to be private as required.

Whenever you need to manage private files, you will need to invoke this custom backend.

The backend can use the buckets we provide to do this, but please be aware that if you restore a backup, or use our tools to push files, all the files will become public.

Alternatively, you can use a bucket of your own with this backend.

Loading media files into your applications' pages​

Sometimes an application in your application will need to load media files using JavaScript.

Since your media files are held on a server under a different domain from the application, browsers may refuse to allow this cross-domain loading for security reasons.

There are two solutions to this.

Load media from static or public​

One is to make sure that all files you need to load are in your site's static or public files, rather than media, depending on the technology you are using. (The static/public files are served from the same domain as the application itself, so browsers will be able to load files using JavaScript without complaint).

This has the advantage of not running into the possibility of using JavaScript to load user-submitted material (which could include material uploaded by untrusted users).

Enable CORS headers​

The other solution is to enable CORS ("cross-origin resource sharing") headers on the media bucket, allowing the bucket to serve its resources when requested by a page on a different domain.

Storage speed and performance​

Note that if you need to make many read/write operations to file storage, or are working with very large objects, that the speed you experience on the cloud can be considerably less than what you experience in the local development environment.

The local development environment has the advantage of locally-attached storage, and should not necessarily be taken as a guide to performance on the cloud.

In most cases, this won't actually matter. However, if your code works very intensively with storage, it can be more efficient and faster to do all that work on the application instance's own local filesystem, in a temporary directory, and then send the finished work to the remote storage.