Location>code7788 >text

New version of Django Docker deployment solution with multi-stage builds, automated front-end dependency handling

Popularity：131 ℃/2024-08-13 22:58:51

preamble

In the other day's post, we've got the project using pdm down with dockerThen the next step is to deploy the full DjangoStarter v3 version with docker.

Now it's not as simple as before, because the project uses npm, gulp and other tools to manage the front-end dependencies, and use pdm to manage python dependencies, so this wave I used a multi-stage build (multi-stage build)

And this time also replaced uwsgi. But not to mention uwsgi old to old, performance is still good, but now is already asgi era, wsgi limitations are still a bit much, plus I deployed the project used channels, so it is logical to use it recommended daphne server, feel okay, more usage is still exploring, the follow-up to get out to write articles to record.

After stepping over a lot of potholes, I finally got this set done.

PS: Tossing this stuff is so exhausting... it feels like I'm a relentless internet mover, searching for information based on search engines and official documents (and now GPT), constantly arranging and combining them, and ultimately forming a solution that works 😂

This article documents the tossing and turning while the new docker deployment solution was quickly merged into the master branch of the DjangoStarter project.

Some concepts

A deep understanding of how docker works can help avoid a lot of problems

Every time after encountering a lot of pit burnt out, I feel deeply that I'm still too much of a novice

multistage construction

In a Dockerfile, use multipleFROM command and passes it through theAS Keywords for eachFROM The ability to specify a name is called"multi-stage build" (MSB)。

Multi-stage builds allow you to use multiple base images in a single Dockerfile and optionally copy build results from one stage to another during the build process. The advantage of this is that you can use a larger image for build work (such as compiling code) in the earlier stages, and then keep only the necessary files in the final stage, putting them into a smaller base image to reduce the size of the final image.

Multi-stage builds greatly simplify the process of building complex images, while also helping to keep the size of the final image small.

There are a few things to keep in mind:

  • You don't need to specify the dependencies between each stage, the build will be done at the same time as the build, and you will only need to specify the dependencies if you encounter theCOPY --from=<stage name> maybeCOPY --from=<stage index> This statement is what waits for the dependent stages to finish building.
  • everyoneFROM commands all start a new build phase. These phases are independent; environment variables, filesystem state, etc. from one phase are not automatically passed on to the next, and each phase may choose to copy build results from any previous phase.
  • In a multi-stage build, commands in the Dockerfile are executed sequentially, and earlier stages do not automatically pass files or state to later stages unless explicitly using theCOPY --from=<stage> Instructions.

Docker Volumes Mechanism

Docker volumes are Docker-managed data volumes that are used as a mechanism for persistently saving and sharing data between containers and between containers and the host.

  • Even if the container is deleted, the data stored in volumes still exists
  • You can mount the same volume on multiple containers to share data.
  • volume data is stored in a specific area of the Docker host and is only accessed by the container through a mount point
  • Prioritization (very important)The volume mounted while the container is running will overwrite the contents of the corresponding path in the container. (I've been having trouble updating files in static-dist because of this overwriting problem)

So in a static file sharing scenario, it is important to maintain data consistency.

dockerfile

Let's take a look at the dockerfile I ended up with.

ARG PYTHON_BASE=3.11
ARG NODE_BASE=18

# python construct (sth abstract)
FROM python:$PYTHON_BASE AS python_builder

# set up python environment variable
ENV PYTHONUNBUFFERED=1
# Disable update checking
ENV PDM_CHECK_UPDATE=false

# set up国内源
RUN pip config set -url /pypi/simple/ && \
    # mounting pdm
    pip install -U pdm && \
    # Configuration Mirroring
    pdm config "/pypi/simple/"

# Reproduction of documents
COPY /project/

# mounting依赖项和项目到本地包目录
WORKDIR /project
RUN pdm install --check --prod --no-editable

# node construct (sth abstract)
FROM node:$NODE_BASE as node_builder

# Configuration Mirroring && mounting pnpm
RUN npm config set registry && \
    npm install -g pnpm

# Copying dependency files
COPY /project/

# mounting依赖
WORKDIR /project
RUN pnpm i


# gulp construct (sth abstract)
FROM node:$NODE_BASE as gulp_builder

# Configuration Mirroring && mounting pnpm
RUN npm --registry install -g gulp-cli

# Copying dependency files
COPY /project/

# 从construct (sth abstract)阶段获取包
COPY --from=node_builder /project/node_modules/ /project/node_modules

# Copying dependency files
WORKDIR /project
RUN gulp move


# django construct (sth abstract)
FROM python:$PYTHON_BASE as django_builder

COPY . /project/

# 从construct (sth abstract)阶段获取包
COPY --from=python_builder /project/.venv/ /project/.venv
COPY --from=gulp_builder /project/static/ /project/static

WORKDIR /project
ENV PATH="/project/.venv/bin:$PATH"
# Handling static resource resources
RUN python ./src/ collectstatic


# operational phase
FROM python:$PYTHON_BASE-slim-bookworm as final

# 从construct (sth abstract)阶段获取包
COPY --from=django_builder /project/.venv/ /project/.venv
COPY --from=django_builder /project/static-dist/ /project/static-dist
ENV PATH="/project/.venv/bin:$PATH"
ENV DJANGO_SETTINGS_MODULE=
ENV PYTHONPATH=/project/src
ENV PYTHONUNBUFFERED=1
COPY src /project/src
WORKDIR /project

multi-stage build

The dockerfile has these build phases, which you can see from the names

  • python_builder: install pdm package manager and python dependencies
  • node_builder: installing front-end dependencies
  • gulp_builder: Consolidating front-end resources with gulp tool handling
  • django_builder: takes the python dependencies and front-end resources from the builds of the previous containers and performs work like collectstatic (this is the only one at the moment, but others may be added in the future)
  • final: used to run and generate images after finalization

pivot

There are a few key points to consider when debugging this dockerfile

  • Don't use slim images during the build phase, as you might run into some weird problems with a too-simple environment!
  • Starting with the django_builder phase, which involves running python, the python path in the virtual environment must be added to the environment variable
  • Since DjangoStarter v3 starts with a new project structure where the source code is placed in the src directory in the root directory, you need to add this directory to the PYTHONPATH environment variable during the final phase or you will run into strange package import issues (I've run into this with uwsgi)
  • or the final stage, I also set theDJANGO_SETTINGS_MODULE, PYTHONUNBUFFERED environment variables, regardless of whether they are useful or not, it is important to keep them consistent with the development environment to avoid encountering problems.

docker-compose

I used some environment variables in the compose configuration

Avoid having to change the project name for each project.

Configuration first, more on that later.

services:
  redis:
    image: redis
    restart: unless-stopped
    container_name: $APP_NAME-redis
    expose:
      - 6379
    networks:
      - default
  nginx:
    image: nginx:stable-alpine
    container_name: $APP_NAME-nginx
    restart: unless-stopped
    volumes:
      - ./:/etc/nginx//
      - ./media:/www/media:ro
      - static_volume:/www/static-dist:ro
    depends_on:
      - redis
      - app
    networks:
      - default
      - swag
  app:
    image: ${APP_IMAGE_NAME}:${APP_IMAGE_TAG}
    container_name: $APP_NAME-app
    build: .
    restart: always
    environment:
      - ENVIRONMENT=docker
      - URL_PREFIX=
      - DEBUG=true
    #    command: python src/ runserver 0.0.0.0:8000
    command: >
      sh -c "
      echo 'Starting the application...' &&
      cp -r /project/static-dist/* /project/static-volume/ &&
      exec daphne -b 0.0.0.0 -p 8000 -v 3 --proxy-headers :application
      "
    volumes:
      - ./media:/project/media
      - ./src:/project/src
      - ./db.sqlite3:/project/db.sqlite3
      - static_volume:/project/static-volume
    depends_on:
      - redis
    networks:
      - default

volumes:
  static_volume:

networks:
  default:
    name: $APP_NAME
  swag:
    external: true

A few pointers

nginx

Here I added the nginx container, used to provide web services, because before I have been using uwsgi, but to uwsgi's socket mode is no static file function, to let uwsgi provide static file service, unless the use of HTTP mode, but then again lost the advantage of uwsgi. So I've been using nginx to provide static file access.

However, now not directly on the server to install nginx but changed to a swag container a shuttle, so must be used in compose add a web server, in this case, it is better to continue nginx bar, with more familiar. On this issue, the previous article (Project Completion Summary - Django-React-Docker-Swag Deployment Configuration) has also been discussed.

image name

The image from this build finally has a name...

Paving the way for the next push to the docker hub

data sharing

Since I've always executed collectstatic locally and then uploaded it to the server, I don't have a problem with static files, but it's not elegant at all, and it's very unfriendly to CICDs

DjangoStarter v3 also solves this pain point by integrating front-end dependencies and resource management into the build phase of docker, so you need to use the docker volume to share these static resources for the app and nginx containers.

As I said at the beginning, volume has a higher priority, so even if you modify some static resources later, you will restart with the old version that is already mounted in the volume after build, so here I mount the app container's /project/static-volume to the shared static_volume, and copy the files in /project/static-dist to it instead of directly to /project/static-dist. project/static-dist instead of directly mounting it to /project/static-dist, which would cause the old data in static_volume to overwrite the files from collectstatic.

There are other ways to do this, such as cleaning out the files in static_volume before each boot and then mounting them, but it was a bit of a toss-up when I tried it, so I gave up.

About Application Server Selection

Django (or Python-based back-end frameworks) do not have built-in servers like kestrel, tomcat, etc. like .netcore, springboot, etc., so deployment to production environments can only be done with the help of application servers.

Common Django application servers include uWSGI, Gunicorn, Daphne, Hypercorn, Uvicorn, and others.

Have been using uWSGI , which is a powerful and highly configurable WSGI server , supports multi-threaded , multi-process , asynchronous mode of operation , and has a wealth of plug-in support. Its high performance and flexibility make it the first choice for many large projects. However, uWSGI is relatively complex to configure and may not be friendly to novices. uWSGI's high complexity may lead to misconfiguration or difficult debugging in some scenarios.

Then in the example of this article, the use of uwsgi deployment has been a problem, because a previous project used channels, so this time the use of Daphne, which is the core part of the Django Channels project, is a support for HTTP and WebSocket ASGI server. And also supports HTTP2 and other new features, actually quite good.

Daphne is ideal if you need to handle WebSocket connections or use Django Channels.Daphne can be used in conjunction with reverse proxy servers such as Nginx to handle static files and SSL terminals.

Next I'm going to try Gunicorn and ASGI-based Uvicorn, which seems to be used more in the FastAPI project, and Django has been supporting asynchronous functionality (aka ASGI) since 3.0, so it's actually possible to ditch the traditional WSGI servers?

A few others I'll copy some of the introductions

Gunicorn (Green Unicorn) is a lightweight WSGI server designed for simplicity and ease-of-use.Gunicorn is easy to configure and provides good performance by default, so it is widely used in both development and production environments. Gunicorn has a lower learning curve than uWSGI and is suitable for most Django projects.

Hypercorn is a modern ASGI server that supports multiple protocols such as HTTP/2, WebSocket and HTTP/1.1. It can run in multiple concurrency modes such as asyncio and trio. It can run in multiple concurrency modes such as asyncio and trio. Hypercorn is suitable for projects that need to use Django Channels, WebSocket, or want to support HTTP/2 in the future.

Uvicorn is a lightweight, high-performance ASGI server based on asyncio, designed for speed and simplicity. It supports HTTP/1.1 and WebSocket, and was an early supporter of HTTP/2. Uvicorn is often used with FastAPI, but it is also suitable for Django, especially if you use Django's asynchronous features. Uvicorn is relatively simple to configure and very fast to start, making it suitable for development and production environments that require asynchronous processing.

ASGI is already the future, so it's better to give up on WSGI next...

wrap-up

I've spent a lot of time on this before, but in reality it's not much to summarize, it's just a few key points. But because of the docker, WSGI, ASGI, Python runtime mechanism is not deep enough to understand, so it led to step on a lot of pitfalls, and finally rely on permutations and combinations to complete this set of docker program ......😂

That's it, I'll continue to improve DjangoStarter v3, and there are some other recent experiences with Django that I can document.

bibliography

  • How to Create Efficient Python Docker Images -/
<Previous