Skip to main content

How to install Python dependencies in an application

See also

It's beyond the scope of this documentation to discuss all the ways in which Python dependencies can be installed in Divio applications. However, the options described here are sufficient to cover most needs.

If you are using Aldryn Django, refer to the section Python package installation in Aldryn Django applications below.

pip install and requirements.txt​

The simplest option is to list Python requirements in a requirements.txt file, and include the command:

RUN pip install -r requirements.txt

in the Dockerfile. See How to configure an existing Django application for deployment on Divio for an example.

However this is only adequate as a quick expedient in the early stages of development and is not recommended beyond that, as it does not allow for complete pinning of all dependencies.

Pin all dependencies​

warning

Unpinned dependencies are the number one cause of deployment failures. Nothing in the codebase may have changed, but a fresh build can unexpectedly pick up a newly-released version of a package.

All Python dependencies, including implicit sub-dependencies, should be pinned to particular versions.

If any dependency is unpinned (that is, a particular version is not specified in the application's requirements) pip will install the latest version it finds, even if a different version was previously installed. This can cause your application to fail with an deployment error or worse, a runtime error, the next time it is built - even if you didn't change anything in it yourself.

To pin all dependencies, your application's requirements should be compiled to a complete list of explicitly specified package versions. This list should then be committed in the application repository, and not be changed until you need to update dependencies.

With pip​

Once you are able to build and run your application successfully, you know have a working set of Python dependencies installed. Use pip freeze to write them in a new file:

docker compose run web pip freeze > compiled_requirements.txt

And then ensure that the pip command in the Dockerfile uses that list:

RUN pip install -r compiled_requirements.txt

Other tools​

There are multiple Python tools such as pip-tools and Poetry that are more sophisticated than pip, that can also generate a complete list of pinned dependencies.

You can use the tool of your choice. In each case, the tool itself needs to be available in the Docker build environment. You can expect to find pip to be installed by default, but other tools will generally need to be installed manually in the Dockerfile.

An example using pip-tools:

RUN pip install pip-tools==5.5.0
RUN pip-compile requirements.in
RUN pip-sync requirements.txt

This installs pip-tools, compiles requirements.in to requirements.txt, then installs the components listed in requirements.txt.

Once you have a working set of dependencies, remove the pip-compile instruction so that the dependencies are pinned and frozen in requirements.txt.

Python package installation in Aldryn Django applications​

By default, applications using an Aldryn Django Dockerfile use our own pip-reqs tool to compile a list wheel URLs from our wheels proxy server, and installs all packages as wheels.

To install Python dependencies an Aldryn application, list them in the requirements.in file. They need to be outside the:

# <INSTALLED_ADDONS>
...
# </INSTALLED_ADDONS>

tags, since that part of the file is maintained automatically and is overwritten automatically with the requirements from the Addons system.

This list is processed by the pip commands in the Dockerfile when the image is built.

Pinning dependencies in an Aldryn application​

Compile requirements.txt​

First, you need to have a working local set-up. Then run:

docker compose run --rm web pip-reqs compile

This will create a requirements.txt file in the application, containing a list of all the packages in the environment, along with their versions.

When your application is built using the new requirements.txt instead of requirements.in, you'll have a guarantee that no unexpected changes will be permitted to find their way in to the application.

Amend the Dockerfile​

In order to have your application built using requirements.txt instead of requirements.in, you need to remove the pip-reqs compile instruction from your application's Dockerfile.

First, remove the Divio-specific comment tags from the Dockerfile:

# <PYTHON>
...
# </PYTHON>

otherwise the Control Panel will simply overwrite your changes.

Then remove the pip-reqs compile instruction, so that requirements.txt will not be amended at the next build.

The next time you need to create a fresh requirements.txt, run:

docker compose run web pip-reqs compile

Specifying packages via a URL​

Please use a commit hash when specifying packages via a URL of a tarballed or zipped archive.

For example:

https://github.com/account/repository/archive/2d8197e2ec4d01d714dc68810997aeef65e81bc1.zip#egg=package-name==1.0
important

Branch names or tags are not supported as part of the archive name and will break. Please use the commit hash as described above.

Recent versions of pip-tools require the use of URLS that provide both the egg fragment and the version fragment (for example, egg=package-name==1.0), and will raise a Bad Request for url error if they encounter URLs lacking it. Older versions would allow you to omit the fragment.

See also Bad request for URL (from from pip-reqs resolve).

pip-tools does note support VCS protocols - for example, you cannot use URLs starting with git+ or hg+, such as git+git@github.com:divio/django-cms.git.