When we install a package like this – pip install Flask – pip needs to work out Flask’s dependencies, the dependencies of those dependencies, and so on. This recursive process builds a dependency tree or graph where multiple nodes (packages) can point to the same dependency, but different versions. Flask may depend on packages A and B, both of which may depend on C==2.1.0 and C>=2.0.0 respectively.

The exact version of C to be downloaded and installed has to be figured out by pip intelligently. It does not install multiple versions of dependencies (like you might be used to from the Node.js/npm world).

We will try to learn a few things in this article:

• A bit about dependency resolution in pip. For a detailed understanding of dependency resolution and backtracking, you should read the documentation. Although if you don’t have much time, then the description below should be exhaustive for most folks.
• Practical examples of when a dependency (version) conflict occurs.
• Differences in how conflicts are dealt with in the new and legacy dependency resolvers (older and newer versions of pip).
• How to check conflicts between dependencies of packages that are already installed in the current environment.
• Solutions to resolve the dependency version conflicts.

Dependency Resolution

Usually, you will have a lot of packages in your requirements.txt. So you can imagine pip will have to resolve the entire dependency tree and eventually make sure it installs the right version of a dependency that works for all the dependants (like the packages listed). This process is called dependency resolution.

To see all the installed packages – the ones you listed in your requirements.txt, passed directly to pip install and all their dependencies as well – you can do pip list. But if you wanted to see the dependency trees of all the installed packages, you can use the pipdeptree command line utility (available from PyPI).

$ pip install pipdeptree
$ pipdeptree
beanie==1.11.7
  - click [required: >=7, installed: 8.1.3]
  - motor [required: >=2.5,<4.0, installed: 3.0.0]
    - pymongo [required: >=4.1,<5, installed: 4.2.0]
  - pydantic [required: >=1.9.0, installed: 1.9.1]
    - typing-extensions [required: >=3.7.4.3, installed: 4.3.0]
  ...
Flask==2.2.1
  - click [required: >=8.0, installed: 8.1.3]
  - Jinja2 [required: >=3.0, installed: 3.1.2]
    - MarkupSafe [required: >=2.0, installed: 2.1.1]
  - Werkzeug [required: >=2.2.0, installed: 2.2.1]
    - MarkupSafe [required: >=2.1.1, installed: 2.1.1]
  ...
pipdeptree==2.2.1
  - pip [required: >=6.0.0, installed: 22.2.2]
setuptools==49.2.1
wheel==0.37.1

With the output from pipdeptree, you will realise that many packages share the same dependencies, but with different version comparison operators. For instance, in the output above, you can see click is shared by beanie and flask with different version comparison operators (>=7 and >=8.0).

For the sake of this article, we don’t need to worry about what click, beanie and flask do in terms of their functionalities. For us, they’re just random packages. Moving on, the tree or list of dependants can be much more in a large project. For instance, this is an output from one of my “real-world” projects and it shows how many times click is resolved as a dependency of multiple dependants:

$ pipdeptree | grep click
    - click [required: >=4.0, installed: 8.0.3]
    - clickclick [required: >=1.2,<21, installed: 20.10.2]
      - click [required: >=4.0, installed: 8.0.3]
      - click [required: >=8.0, installed: 8.0.3]
    - click [required: >=8.0, installed: 8.0.3]
    - click [required: >=6.7,<9, installed: 8.0.3]
      - click [required: >=8.0, installed: 8.0.3]
        - click [required: >=8.0, installed: 8.0.3]
      - click [required: >=8.0, installed: 8.0.3]
  - click [required: >=7.1.2,<8.0.0, installed: 8.0.3]
      - click [required: Any, installed: 8.0.3]
    - click [required: >=8.0, installed: 8.0.3]
    - click [required: Any, installed: 8.0.3]
  - click [required: >=5.0.0, installed: 8.0.3]

All the entries/lines above, belong to various packages, i.e., each line of click is a dependency for a different package that got resolved by pip’s dependency resolver while building the entire dependency tree.

The most important thing to realise here is that eventually, only one version of click will get installed, which in the output above is 8.0.3. This makes sure when you do import click in your Python script, it always loads the same version.

In Python, you can always import click whether you:

• Installed it as a top-level package (requirements.txt or pip install)
• Or, as a dependency of another package (pip install flask that installs click in turn).

As a side note, this behaviour is different from Node.js where npm can install and maintain multiple versions of the same dependency in multiple node_modules sub-directories.

How does pip decide which version to install from the dependency tree?

From the output above, how did pip decide to install click==8.0.3 though?

If we specify a package with its version specifiers directly in our requirements.txt (or via pip install), then that takes precedence over everything else. So basically in my case, I had the following in my requirements.txt:

...
click==8.0.3
...

This is why you see 8.0.3 as “installed” in the pipdeptree output above. But what if the package is not directly installed? In that case, pip will go through all the version specifiers that you see in the list above (for click) and figure out which exact version of the package would suffice all the constraints at the same time. That exact version will be installed.

As an example, here’s a dependency tree from a different project of mine where click is not installed as a top-level package, i.e., not a part of requirements.txt nor installed via pip install click.

$ pipdeptree | grep click
  - click [required: >=7.1.2,<8.0.0, installed: 7.1.2]
      - click [required: Any, installed: 7.1.2]
  - click [required: >=7.1.2, installed: 7.1.2]
        - click [required: >=7.1.1,<7.2.0, installed: 7.1.2]
      - click [required: >=7.1.1,<7.2.0, installed: 7.1.2]
    - click [required: Any, installed: 7.1.2]

After evaluating all the version specifiers – >=7.1.2,<8.0.0, Any, >=7.1.2, >=7.1.1,<7.2.0, >=7.1.1,<7.2.0 and Any – pip smartly evaluated that click==7.1.2 would suffice all of them and hence decided to install that.

So to summarise, now we know that the version of a package installed (top-level or dependency or dependency of dependency and so on), is determined by:

• The version specifier at the top-level, if present.
• If the previous point is absent, then a version that would conform to all the constraints in the entire dependency tree.

Conflicts

You might have already wondered, but if not, what happens when the versions of two or more dependencies at any level of the dependency tree are in conflict with each other or with the top-level package?

Let’s first see a simple conflicting example. Here’s a sample requirements.txt:

click==8.0.3
beanie==1.7.2 # Depends on click>=7.1.2,<8.0.0 - CONFLICT!

Let’s see what happens when we try to install it:

$ pip install -r requirements.txt
...
INFO: pip is looking at multiple versions of <Python from Requires-Python> to
determine which version is compatible with other requirements. This could
take a while.
INFO: pip is looking at multiple versions of click to determine which version
is compatible with other requirements. This could take a while.
ERROR: Cannot install -r requirements.txt (line 2) and click==8.0.3 because
these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested click==8.0.3
    beanie 1.7.2 depends on click<8.0.0 and >=7.1.2

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency
conflict

ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>

We get an error, a fairly exhaustive description of conflict, some links on how to resolve it and most importantly, none of the packages will be installed, i.e., the entire process comes to a halt.

For a slight variation, here’s another example where click is not a top-level package, but a dependency of multiple top-level packages:

beanie==1.7.2 # Depends on click>=7.1.2,<8.0.0
flask==2.2.1 # Depends on click>=8.0 - CONFLICT!

pip will again cause a ResolutionImpossible error (same as before) and fail the installation:

$ pip install -r requirements.txt
...

The conflict is caused by:
    beanie 1.7.2 depends on click<8.0.0 and >=7.1.2
    flask 2.2.1 depends on click>=8.0

...

ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>

So if your dependency tree has conflicts, then you must fix them. We will look at various solutions to fix them in a bit (below).

Legacy Resolver

I used pip==22.2.2 in the examples above. The behaviour of throwing an error and halting the batch installation process from requirements.txt or pip install pkg1 pkg2 ... pkgN was not the case before pip==20.3.

Even though the older (also now known as legacy) dependency resolver was able to resolve the conflicts and show them as error messages, pip would not halt the installation process. It’d successfully continue the process by installing the first matching dependency in the list of conflicts. If you want to read more about the behaviour of the older dependency resolution algorithm and the challenges around it, head over here.

For instance, installing beanie==1.7.2 and flask==2.2.1 (incompatible due to click) would lead to this:

$ pip --version
pip 20.2.3
$ pip install -r beanie==1.7.2 flask==2.2.1
...
Installing collected packages: pymongo, motor, multidict, idna, yarl,
typing-extensions, pydantic, click, toml, beanie, itsdangerous, zipp,
importlib-metadata, MarkupSafe, Werkzeug, Jinja2, flask

ERROR: pip's dependency resolver does not consider dependency conflicts when
selecting packages. This behaviour is the source of the following dependency
conflicts.
flask 2.2.1 requires click>=8.0, but you'll have click 7.1.2 which is
incompatible.

Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.1 Werkzeug-2.2.1
beanie-1.7.2 click-7.1.2 flask-2.2.1 idna-3.3 importlib-metadata-4.12.0
itsdangerous-2.1.2 motor-2.5.1 multidict-6.0.2 pydantic-1.9.1
pymongo-3.12.3 toml-0.10.2 typing-extensions-4.3.0 yarl-1.8.1 zipp-3.8.1
...

You’ll notice that it shows an error, but continues to install click==7.1.2 which comes from beanie (listed before flask in requirements.txt). If I reverse the order, then it installs click>=8.0 which comes from flask.

Note: If you’re on the older versions of pip, then this re-ordering of packages in your requirements.txt file is actually one of the solutions to resolve dependency version conflicts.

So to summarise, with the older versions of pip (<20.3), you will see an error message regarding the conflict. But the installation of the first version of the conflicting dependency package that was encountered while generating the dependency tree will be downloaded and installed.

Fallback to Older Behaviour

In some circumstances, you may need the older (or legacy) behaviour that we just discussed, in the newer versions of pip. There are two simple ways to do this.

The first is to use the legacy-resolver flag:

$ pip install -r requirements.txt --user-deprecated=legacy-resolver

# or use environment variable

$ PIP_USE_DEPRECATED=legacy-resolver
$ pip install -r requirements.txt

The other way is to sort of trick pip. When you install a list of packages together, then pip does resolve conflicts and halt the installation process in the newer versions as we already know. What you need to know is that this process can become really expensive depending upon how big the dependency tree is/will be. This is why pip will not take into account all the packages that are already installed in your environment/system. This idea can be used to trick pip.

So you cannot do this:

$ pip install "beanie==1.7.2" "flask==2.2.1"

But you can do this:

$ pip install "beanie==1.7.2" # Installs click==7.1.2
$ pip install "flask==2.2.1" # Uninstalls click==7.1.2 and installs click==8.1.3
...
Installing collected packages: zipp, MarkupSafe, itsdangerous, click, Werkzeug, Jinja2, importlib-metadata, flask
  Attempting uninstall: click
    Found existing installation: click 7.1.2
    Uninstalling click-7.1.2:
      Successfully uninstalled click-7.1.2

ERROR: pip's dependency resolver does not currently take into account all the
packages that are installed. This behaviour is the source of the following
dependency conflicts.
beanie 1.7.2 requires click<8.0.0,>=7.1.2, but you have click 8.1.3 which is
incompatible.

Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.1 Werkzeug-2.2.1
click-8.1.3 flask-2.2.1 importlib-metadata-4.12.0 itsdangerous-2.1.2
zipp-3.8.1

When pip is only installing flask, it has only one version of click in its dependency tree. So it installs that, overriding any other versions that were already installed on the system. Then it quickly checks the dependencies of all the existing packages and compares their versions to see if there are any conflicts. If a conflict is found, it is reported in the error message (as you can see above).

Do notice that in this case, the last dependency version is the one that will eventually be installed and available, unlike the legacy resolver where the first dependency version is installed. This is obviously due to differences in the processes.

Checking Compatibility

One small trick to quickly check all kinds of dependency conflicts in your current environment is to run the following command:

$ pip check
beanie 1.7.2 has requirement click<8.0.0,>=7.1.2, but you have click 8.1.3.

Or you can also run pipdeptree to see all the conflicts as warnings at the beginning of the output:

$ pipdeptree
Warning!!! Possibly conflicting dependencies found:
* beanie==1.7.2
 - click [required: >=7.1.2,<8.0.0, installed: 8.1.3]
------------------------------------------------------------------------
beanie==1.7.2
  - click [required: >=7.1.2,<8.0.0, installed: 8.1.3]
  ...
Flask==2.2.1
  - click [required: >=8.0, installed: 8.1.3]
  ...
...

Solutions to Conflicts

Finally, I think once you’ve understood the dependency resolution, which versions of the dependencies get installed and when conflicts arise, it is also important to look at what are our options to resolve the conflicts. You will most likely encounter conflicts when your dependency graph is big, i.e., you are dealing with a lot of dependencies in your requirements.txt and hence in the overall graph or tree.

The solution will of course depend on the situation.

Upgrade pip

To start with, you should make sure to always use the latest version of pip so that the new dependency resolver is in action (make sure the legacy resolver is not being used).

Review Top-level Packages

Once the latest pip version is in place, you should review the top-level packages or requirements in requirements.txt. If top-level package A and B depend on conflicting versions of C, then check if you can upgrade (or downgrade, but generally no) either A or B to resolve the conflict.

For instance, the following fails (as we saw earlier):

$ pip install "beanie==1.7.2" "flask==2.2.1"

But the following works just fine:

pip install "beanie==1.11.7" "flask==2.2.1" # Upgraded beanie from 1.7.2 to 1.11.7

Upgrading doesn’t always mean using a different version with ==. You could also look at making your requirements more flexible, like this:

$ pip install "beanie>=1.7.2" "flask>=2.2.1" # Works just fine!

While reviewing you might also find a bunch of unused packages, make sure to remove them.

In some scenarios, if you’re stuck with using an older version of pip or the legacy-resolver flag, then it is expected that the dependency version to be installed is picked from the first matching (or resolved) top-level package. Sometimes this behaviour may lead to unsuitable dependency version installation.

Instead, you may want the dependency version that the next matching/resolving top-level package points to (in its sub-tree). To achieve change the order of the packages in your requirements.txt. If this is all still confusing then look at this SO post, it has an example. Another option is to make the specific dependency version a top-level package itself.

package1
package2
common_dependency===...

As we learnt earlier, top-level package versions will override all other versions from the dependency tree.

Upgrade Dependencies

Sometimes you might encounter the worst scenario where all the versions of beanie depend on click version constraints/specifiers that are all incompatible with that of what all the versions of flask depends on. Basically, imagine if all versions of package A depended on C<2.0 whereas all versions of package B depended on C>2.0.

Well in such cases, you’ll either have to request the package maintainer to upgrade or downgrade or “loosen” (instead of dep==2.0.0 something like dep>=2.0.0 or dep^=2.0.0) the dependency versions. The other option is to fork the package yourself, do the version changes and then use it.

This will work if A can work with C>=2.0 but it’s just that the package maintainer has not made that change. Or if B can work with C<=2.0 but again the maintainer didn’t want to pin it below 2.0 at all.

But if the pinnings are genuine in the sense that the dependants (A and B) would actually break if any changes were made to C versions, then you may have to opt for the most frustrating option of using a different package with compatible dependencies. Also in that case, it might be a good time to casually read up on Dependency Hell :).