I want to become a node_modules master! (I): Package manager selection, dependency analysis

Some problems that happened

1. Dependency Hell

• Nested dependency structure: earlier versions of npm use nestednode_modulesStructure, extremely deep dependency levels, can easily lead to excessive path problems (especially on Windows), and even trigger file system restrictions.

• Version conflict: The dependent version management is not strict enough, and it is prone to coexistence of "multiple versions of the same package", which leads to the project volume swelling or difficulty in debugging.

2. Performance issues

• Slow installation speed: npm's installation algorithm (especially before v3) is inefficient and has a long dependency on parsing and downloading time.

• Global lock problem: npm lock file () Design has been criticized for being incompatible with other tools (such as Yarn), and early versions have problems with lock file conflicts.

3. Security historical issues

• Dependency chain risk: npm allows dependency packages to automatically install any sub-dependencies, which has caused multiple security events (for exampleevent-streamMalicious package injection event).

• Permission issue: In the past, npm's package release mechanism was easily abused, and there were "package name squatting" or low-quality packages flooded.

4. Controversy in design philosophy

• Centralized registry: The official registry of npm is a single point of failure, and global developers will be affected once they fail (such as service outages in 2020).

• Abuse of semantic versions (SemVer): Many packages are overly dependent^or~Version scope, resulting in inconsistent dependent versions installed in different environments, which may cause unexpected problems.

5. Comparison of competitors

• Yarn’s impact: After Yarn was launched in 2016, it directly exposed npm’s shortcomings with its offline cache, parallel installation, and more stable lock files.

• Improvements to pnpm: pnpm optimizes storage space and installation speed through hard links and symbolic links, further highlighting the redundancy problem of npm.

despite this

For most normal projects, npm is stable enough, especially the new version (v7+) absorbs the advantages of Yarn and pnpm.

2. Package management tool

3. Specific dependency example analysis

There are now two projects,

Project 1, Dependency requirements: a,b,c a depends on b,c,c,c without dependencies

Project 2. Dependence requirements: a,b,c,d a depends on b,c,c,c depends on d,d depends on b

These are two typical projects,
The first one represents direct dependency
The second one represents nested dependencies

Now I use npm,yarn,pnpm,bun,

We analyze the node_modules folder structure, as well as the package file, and the lock file respectively

3.1. The first project is very simple

Comparison of installation results

3.2. We focus on the second project

See how each tool handles nested dependencies

Let’s see the example picture:

(1)NPM

node_modules

node_modules/
├── a/               # a@1.0.0│ └── # Dependencies: b, c
 ├── b/               # b@1.0.0(Relied by a and d)
 ├── c/               # c@1.0.0│ └── # Dependencies: d
 ├── d/               # d@1.0.0│ └── # Dependencies: b
 └── .bin/# Executable (if any)

Lock the file

{
  "name": "project2",
  "version": "1.0.0",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "node_modules/a": {
      "version": "1.0.0",
      "dependencies": { "b": "^1.0.0", "c": "^1.0.0" }
    },
    "node_modules/b": { "version": "1.0.0" },
    "node_modules/c": {
      "version": "1.0.0",
      "dependencies": { "d": "^1.0.0" }
    },
    "node_modules/d": {
      "version": "1.0.0",
      "dependencies": { "b": "^1.0.0" }
    }
  }
}

(2)Yarn

node_modules

node_modules/
├── a/               # a@1.0.0│ └── # Dependencies: b, c
 ├── b/               # b@1.0.0(Elevated to the top level)
 ├── c/               # c@1.0.0│ └── # Dependencies: d
 ├── d/               # d@1.0.0│ └── # Dependencies: b
 └── .bin/

Lock the file

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
a@1.0.0:
  version "1.0.0"
  dependencies:
    b "^1.0.0"
    c "^1.0.0"

b@1.0.0:
  version "1.0.0"

c@1.0.0:
  version "1.0.0"
  dependencies:
    d "^1.0.0"

d@1.0.0:
  version "1.0.0"
  dependencies:
    b "^1.0.0"

(3)pnpm

node_modules

node_modules/
├── a -> .pnpm/a@1.0.0/node_modules/a # symbolic link
 ├── b-> .pnpm/b@1.0.0/node_modules/b
├── c -> .pnpm/c@1.0.0/node_modules/c
├── d -> .pnpm/d@1.0.0/node_modules/d
└── .pnpm/
    ├── a@1.0.0/
    │   └── node_modules/│ ├── a # a real file
     │ ├── b-> ../../b@1.0.0/node_modules/b # hard link
     │ └── c-> ../../c@1.0.0/node_modules/c
    ├── c@1.0.0/
    │   └── node_modules/│ ├── c # c's real file
     │ └── d-> ../../d@1.0.0/node_modules/d
    ├── d@1.0.0/
    │   └── node_modules/│ ├── d # d's real file
     │ └── b-> ../../b@1.0.0/node_modules/b # hard link
     └── b@1.0.0/
        └── node_modules/└── b # b's real file

Lock the file

lockfileVersion: 5.4
dependencies:
  a:
    specifier: 1.0.0
    version: 1.0.0
    dependencies:
      b: 1.0.0
      c: 1.0.0
  b:
    specifier: 1.0.0
    version: 1.0.0
  c:
    specifier: 1.0.0
    version: 1.0.0
    dependencies:
      d: 1.0.0
  d:
    specifier: 1.0.0
    version: 1.0.0
    dependencies:
      b: 1.0.0

(4)bun

node_modules/
├── a/               # a@1.0.0(Hard link to global storage)
 ├── b/               # b@1.0.0(Hard link)
 ├── c/               # c@1.0.0
├── d/               # d@1.0.0
└── .bin/

Lock the file

For binary

Summarize,

4. A question

Propose a new situation for project 2

Assume that the b package that the project itself depends on is 1.0.0
The version of the b package that the d package depends on is: 2.0.0

What happens to node_modules and lock files?

node_modules/
├── a/               # a@1.0.0│ └── # Dependency: b@1.0.0, c@1.0.0
├── b/               # b@1.0.0(Elevated to the top level)
 ├── c/               # c@1.0.0│ └── # Dependencies: d@1.0.0
├── d/               # d@1.0.0
│   ├── node_modules/
│   │   └── b/       # b@2.0.0(Nested)
 │ └── # Dependency: b@2.0.0
└── .bin/

Due to dependency improvement, the b package version 1.0.0 (meet first) is therefore promoted to the top level

npm will try to improve dependencies to the top level, butOnly one version of the same package can be upgraded, and the rest will be nested。