Skip to main content

Working with your application's media storage in web applications

Introduction​

File storage on Divio applications is handled by dedicated storage systems entirely separate from the application. The available storage depends on the Divio cloudspace your application uses. Most applications use Amazon Web Services's S3 service, or another S3 provider. Others use Microsoft Azure blob storage.

In our architecture, the same application may be running in multiple parallel container, each with its own local file storage independent of each of the others. Moreover, this storage is not persistent, and exists only for as long as the lifetime of the container.

This means an application should not expect to save files to its local storage, and then expect to find them later.

Good implementations of cloud storage backends or plugins, for both S3 and Azure blog storage, exist for most mature web frameworks and applications languages.

Working with our storage backends in Django​

For most Django applications, this won't require any additional work. Django is able to use multiple storage backends, all addressed through a common API. This is the safe and correct way to handle files in Django, so that applications can abstract from details of the storage implementation, and simply not need to know about it.

As long as an application uses Django's storage API, rather than attempting to manipulate Python File objects directly, it doesn't need to do anything differently.

Similarly, an application should not rely on knowing or manipulating a File object's file path.

Use Django's defined DEFAULT_FILE_STORAGE, not FileSystemStorage​

Your code may use Django's FileSystemStorage. This provides basic file storage, on a filesystem local to the code. For the reasons described in the Introduction it is therefore not suitable for use on Divio.

Instead, you must use the storage as defined by Django's DEFAULT_FILE_STORAGE - which you can do simply by not explicitly specifying a storage system, and using django.core.files.storage.default_storage.

See also Django's discussion of the subject.

File storage in third-party applications​

Ideally, third-party applications in your application should respect this for their own file handling.

This is not always the case however. In some cases the application may need to be configured explicitly.

For example, you may need to set THUMBNAIL_DEFAULT_STORAGE for Django's Easy Thumbnails library.

More problematically, some applications may have hard-coded expectations for the file storage system, and these will need to be rewritten.

Private file storage​

Our storage backend does not support private file storage (i.e. requiring authentication) on S3 objects.

If you need private storage, you can define an additional Django storage backend in your application, which sets S3 objects to be private as required.

Whenever you need to manage private files, you will need to invoke this custom backend.

The backend can use the buckets we provide to do this, but please be aware that if you restore a backup, or use our tools to push files, all the files will become public.

Alternatively, you can use a bucket of your own with this backend.

Loading media files into your applications' pages​

Sometimes an application in your application will need to load media files using JavaScript.

Since your media files are held on a server under a different domain from the application, browsers may refuse to allow this cross-domain loading for security reasons.

There are two solutions to this.

Load media from static​

One is to make sure that all files you need to load are in your site's static files, rather than media. (The static files are served from the same domain as the application itself, so browsers will be able to load files using JavaScript without complaint).

This has the advantage of not running into the possibility of using JavaScript to load user-submitted material (which could include material uploaded by untrusted users).

Enable CORS headers​

The other solution is to enable CORS ("cross-origin resource sharing") headers on the media bucket, allowing the bucket to serve its resources when requested by a page on a different domain.

Storage speed and performance​

Note that if you need to make many read/write operations to file storage, or are working with very large objects, that the speed you experience on the cloud can be considerably less than what you experience in the local development environment.

The local development environment has the advantage of locally-attached storage, and should not necessarily be taken as a guide to performance on the cloud.

In most cases, this won't actually matter. However, if your code works very intensively with storage, it can be more efficient and faster to do all that work on the application instance's own local filesystem, in a temporary directory, and then send the finished work to the remote storage.