From Chaos to Order: An Analysis of Python's Dependency Management Tools

Python The dependency management tools have not been standardized for a number of reasons:

1. Randomness of historical development：PythonInsufficient emphasis on dependency management in the early stages of development and lack of awareness of harmonized planning and design from the outset
2. Decentralized nature of communities：PythonThe community is large and decentralized, with many developers and teams working on their own, developing tools according to their own needs and preferences, and lacking a unified coordination and integration mechanism
3. Diverse usage scenarios：PythonThere are a wide range of application scenarios, from web development to data science, machine learning, and system administration scripts. Different scenarios have different requirements for dependency management
4. The challenge of backward compatibility：Python The language itself has a strong focus on backward compatibility, which somewhat limits the possibility of making fundamental changes to dependency management tools
5. Lack of harmonized governance: with some programming languages (e.g.Java (with Oracle leading the specification development) is different.Python Absence of a strong single entity to lead the standardization of dependency management tools
6. Rapid changes in ecosystems：Python The ecosystem is evolving rapidly, with new libraries and frameworks emerging all the time, which makes dependencies increasingly complex

1. What is dependency management?

Dependency management toolsCommonly used to handle dependencies in software projects.

During software development, a project often depends on many other software libraries, frameworks or components.

Dependency management toolsIt can help developers precisely specify the versions of these dependencies, automatically download and install them, and can ensure the consistency of dependencies across different environments (e.g., development, test, production).

This effectively avoids software failures, compatibility issues, etc. caused by confusing dependency versions.

For example, standards for other programming languagesDependency management toolsThere is:NodeJS(used form a nominal expression)npm，Rust(used form a nominal expression)cargo，Java(used form a nominal expression)MavenAnd so on.

The most critical role of a dependency management tool isrepeatability, meaning that we can follow a series of steps and end up with exactly the same software project.

Especially nowadays, projects (whether open source or internal) basically require multiple people to collaborate, and it is crucial to ensure that everyone's code compiles to run the software with the same results.

Good dependency management can have an impact onexploit (a resource)、construct (sth abstract)respond in singingdeploymentsAll dependencies of the phase are explicitly declared and tracked with the code in version control.

In a nutshell:Application = code + all dependencies。

This can be summarized in the following steps:

1. Creating a definition file: description of the project, declaration of required dependencies, minimum version constraints, etc.
2. Generating Locked Files: fixing the relationship between the version of a dependency and the dependencies
3. Synchronization environment: Usually, we synchronize with each other using a versioning tool such as git.
4. Tracking definition files and locking files: Synchronize definition files and lock files with each other in a timely manner when they change

2. Comparison of dependency management tools

Python Although there is no unified dependency management tool, there are many to choose from, the following one by one to analyze the advantages and disadvantages of each tool.

2.1. pip

pipis the default package manager that comes with it, and is the most used tool, which is characterized as only being used to install thePythonPackage.

dominance：

1. sincePython 3.4Included in thePythonNo additional installation is required.
2. Introduced in 2013wheelsDistributed in a format that is much faster to install
3. Dependency resolution algorithms have been added since 2020 to better maintain the consistency of the environment

shortcoming：

1. dependenciesPythonThis means that using thepipMust be installed firstPython
2. Cannot install non-Python packages
3. No locking of files

2.2. venv

Built-in tool for creating virtual environments in which you can use the pip installer to isolate the environment by setting environment variables.

dominance:: FromPython 3.3Included in thePythonCenter.

shortcoming：

1. bePython Tools, dependenciesPython mounting
2. All environments must use the samePython interpreter
3. Unable to install nonPython contract (to or for)

2.3. virtualenv

existvenv constitutePython Before the built-in tools, we usually use thevirtualenvTo create a virtual environment, you can specify differentPython The interpreter creates the virtual environment, which needs to be passed through thepip Installation.

dominance: Can specify differentPython The interpreter creates the virtual environment.

shortcomingtogether withvenv It's the same.

2.4. pip-tools

Lightweight tool that introduces a lock file mechanism.

Needs to be prepared firstas a definition file, and then use thepip-compilegeneratinglock file, the synchronization environment is using thepip-sync。

dominance：Lightweight and simple, with basic<font style="color:rgba(0, 0, 0, 0.85);">pip/venv</font> The tools work together.

shortcoming：

1. bePython Tools, need to be installed in the project environment, there may be compatibility issues
2. Can only handlepip installable package
3. definition fileRequires manual maintenance

2.5. Pipenv

integratedpip、virtualenv cap (a poem)pip-tools The function of thePipfilecap (a poem)Manage dependencies and virtual environments, automatically update files.

dominance: lightweight and simple, packed with basicpip/venv Tools.

shortcoming：

1. bePython artifact
2. Has its own definition and lock file format
3. Can only handlepip installable package
4. Can only distinguish between development and non-development dependencies, environment definition is not flexible enough

2.6. Poetry

Aims to coverPython The entire development process of a project, including project bootstrapping, virtual environments, dependency management, building and releasing packages.

pass (a bill or inspection etc)Manage dependencies and automate maintenance, supports dependency grouping.

dominance：

1. All-in-one tool covering the entire project development lifecycle
2. Convenient command line interface available
3. Support for dependency grouping

shortcoming：

1. bePython Tools, heavier weight, many dependencies, installation may be problematic
2. Poor interoperability with other tools, no support for other build backends
3. Maintaining a mutually exclusive environment is not supported
4. Has its own dependency definition and lock file format
5. Can only handlepip installable package

2.7. PDM

alikePoetryBut followPEP Standard, can be useduvPerforms dependency resolution and installation, and its build backend can be used independently.

uvis another dependency management tool that will be introduced later.

dominance：

1. comply withPEP (an official) standard
2. availableuvDependency management

shortcoming:: In conjunction with thePoetry Similarly, it isPython tool with more dependencies and associated drawbacks.

2.8. pyenv

For installing and managing different versions ofPythonThe ability to activate a specific version at the global or project level is a simpleshell Utilities, no dependenciesPython Installation.

dominance：

1. unmixedshell Script, noPython dependencies
2. comply withUnix Philosophy, focus on managementPython releases

shortcoming：

1. new installationPython version requires downloading and compiling the source code, which takes time.
2. First time setup can be tricky, multiple build dependencies need to be installed
3. unsupportedWindows

2.9. pipx

commander-in-chief (military)pip packages are installed in separate virtual environments at the user level, avoiding dependency conflicts through thesymlink(soft link) link the entry point to thePATH, easy to call.

dominance: more than directly at the user levelpip Installation tools are better, isolate dependencies, and can use differentPython Interpreter.

shortcoming：

1. bePython artifact
2. Multiple versions of the same tool cannot be installed, all projects need to share the tool version

2.10. uv

expense or outlayRust An all-in-one tool written to replace multiplePython Administration tools that handle the entire development process, including installing packages, managing virtual environments, building and releasing.

comply withPython standard, dependencies are defined in theThe lock file isThe system supports any grouping of dependencies and can be installedpip packages as executables that can be managedPython version that maintains the global package cache.

dominance：

1. expense or outlayRust Written, extremely fast, single binary file, no external dependencies
2. Multi-Platform Support
3. All-in-one tool with comprehensive functionality
4. comply withPython (an official) standard
5. Support for any build backend of your choice
6. Support for dependency grouping.

shortcoming：

1. Maintaining multiple mutually exclusive environments is not supported
2. Can only handlepip installable package

2.11. Conda

leave it (to sb)Anaconda The company develops package managers for different ecosystems, mainly for installingOn the bag.

The ability to create virtual environments withpip Ecosystems are different and have a different impact on [hold or embraceThe definition of ] is broader and includes shared libraries, headers, executables, and so on.

dominance：

1. Multi-platform support
2. There is a global package cache
3. Packages are distributed as compiled binaries
4. Dependency resolution algorithms are robust
5. available atConda Use in the environmentpip
6. Support for global and shared environments

shortcoming：

1. slow
2. Packages are downloaded serially
3. The installation process is somewhat invasive and will modifyshellconfigure
4. with "trump card (in card games)” Python Limited ecosystem interoperability
5. Unlocked files
6. Build and distributeConda be more painful

2.12. Mamba

beCondaAn improved version of theCondapain points, such as slow dependency resolution and parallel download problems, with theC++ implementation, using a different algorithm, the recommended installation has changed.

speed ratioConda Much faster, and otherwise as good asCondaSimilar.

2.13. Pixi

alikeuvBut forConda Ecosystems withRust Written to support multiple platforms.

pass (a bill or inspection etc)maybeconfiguration, a convenient command line interface, support for managing multiple virtual environments and definition files, a lock file mechanism, and support for similarMakefile of project automation tasks that specify system dependencies but do not help build packages.

dominance：

1. Written in Rust, fast, single binary, no external dependencies
2. Multi-Platform Support
3. Convenient command line interface
4. Global Package Cache
5. downloadablePython binary file andupper limitPython hold or embrace
6. usablerespond in singingconfigure
7. Choose any build backend

shortcoming: Limited compatibility with other tools and does not follow theConda of the global environmental concept.

3. Recommendations for tool selection

With so many dependency management tools available, how should we choose?

If our project only has dependencies on Python packages, then the recommendeduvrespond in singingPixi；

If you need to maintain multiple mutually exclusive environments, then the recommendedpip + venv + pip-tools + pyenv；

If processing is required that cannot be done throughpip installed dependencies, then it is recommended to use thePixi。

4. Summary

PythonThere are a lot of dependency management tools out there, but most of them are actually pretty much the same, with a few minor improvements over each other.

When we choose, we try to prioritize the newly released tools in addition to considering the legacy items.

In addition to improving on the shortcomings of the original tool, the new tool will also borrow good dependency management tools from other languages.

Currently, my personal words, useuvto manage the project more.