Understanding Python Packages pip Dependency Resolver and Version Conflicts (with Solutions)
When we install a package like this – pip install Flask
– pip
needs to work out Flask’s dependencies, the dependencies of those dependencies, and so on. This recursive process builds a dependency tree or graph where multiple nodes (packages) can point to the same dependency, but different versions. Flask
may depend on packages A
and B
, both of which may depend on C==2.1.0
and C>=2.0.0
respectively.
The exact version of C
to be downloaded and installed has to be figured out by pip
intelligently. It does not install multiple versions of dependencies (like you might be used to from the Node.js/npm world).
We will try to learn a few things in this article:
- A bit about dependency resolution in
pip
. For a detailed understanding of dependency resolution and backtracking, you should read the documentation. Although if you don’t have much time, then the description below should be exhaustive for most folks. - Practical examples of when a dependency (version) conflict occurs.
- Differences in how conflicts are dealt with in the new and legacy dependency resolvers (older and newer versions of
pip
). - How to check conflicts between dependencies of packages that are already installed in the current environment.
- Solutions to resolve the dependency version conflicts.
Dependency Resolution
Usually, you will have a lot of packages in your requirements.txt
. So you can imagine pip
will have to resolve the entire dependency tree and eventually make sure it installs the right version of a dependency that works for all the dependants (like the packages listed). This process is called dependency resolution.
To see all the installed packages – the ones you listed in your requirements.txt
, passed directly to pip install
and all their dependencies as well – you can do pip list
. But if you wanted to see the dependency trees of all the installed packages, you can use the pipdeptree
command line utility (available from PyPI).
$ pip install pipdeptree
$ pipdeptree
beanie==1.11.7
- click [required: >=7, installed: 8.1.3]
- motor [required: >=2.5,<4.0, installed: 3.0.0]
- pymongo [required: >=4.1,<5, installed: 4.2.0]
- pydantic [required: >=1.9.0, installed: 1.9.1]
- typing-extensions [required: >=3.7.4.3, installed: 4.3.0]
...
Flask==2.2.1
- click [required: >=8.0, installed: 8.1.3]
- Jinja2 [required: >=3.0, installed: 3.1.2]
- MarkupSafe [required: >=2.0, installed: 2.1.1]
- Werkzeug [required: >=2.2.0, installed: 2.2.1]
- MarkupSafe [required: >=2.1.1, installed: 2.1.1]
...
pipdeptree==2.2.1
- pip [required: >=6.0.0, installed: 22.2.2]
setuptools==49.2.1
wheel==0.37.1
With the output from pipdeptree
, you will realise that many packages share the same dependencies, but with different version comparison operators. For instance, in the output above, you can see click
is shared by beanie
and flask
with different version comparison operators (>=7
and >=8.0
).
For the sake of this article, we don’t need to worry about what click
, beanie
and flask
do in terms of their functionalities. For us, they’re just random packages. Moving on, the tree or list of dependants can be much more in a large project. For instance, this is an output from one of my “real-world” projects and it shows how many times click
is resolved as a dependency of multiple dependants:
$ pipdeptree | grep click
- click [required: >=4.0, installed: 8.0.3]
- clickclick [required: >=1.2,<21, installed: 20.10.2]
- click [required: >=4.0, installed: 8.0.3]
- click [required: >=8.0, installed: 8.0.3]
- click [required: >=8.0, installed: 8.0.3]
- click [required: >=6.7,<9, installed: 8.0.3]
- click [required: >=8.0, installed: 8.0.3]
- click [required: >=8.0, installed: 8.0.3]
- click [required: >=8.0, installed: 8.0.3]
- click [required: >=7.1.2,<8.0.0, installed: 8.0.3]
- click [required: Any, installed: 8.0.3]
- click [required: >=8.0, installed: 8.0.3]
- click [required: Any, installed: 8.0.3]
- click [required: >=5.0.0, installed: 8.0.3]
All the entries/lines above, belong to various packages, i.e., each line of click is a dependency for a different package that got resolved by pip’s dependency resolver while building the entire dependency tree.
The most important thing to realise here is that eventually, only one version of click
will get installed, which in the output above is 8.0.3
. This makes sure when you do import click
in your Python script, it always loads the same version.
In Python, you can always import click
whether you:
- Installed it as a top-level package (
requirements.txt
orpip install
) - Or, as a dependency of another package (
pip install flask
that installsclick
in turn).
As a side note, this behaviour is different from Node.js where npm
can install and maintain multiple versions of the same dependency in multiple node_modules
sub-directories.
How does pip decide which version to install from the dependency tree?
From the output above, how did pip
decide to install click==8.0.3
though?
If we specify a package with its version specifiers directly in our requirements.txt
(or via pip install
), then that takes precedence over everything else. So basically in my case, I had the following in my requirements.txt
:
...
click==8.0.3
...
This is why you see 8.0.3
as “installed” in the pipdeptree
output above. But what if the package is not directly installed? In that case, pip
will go through all the version specifiers that you see in the list above (for click
) and figure out which exact version of the package would suffice all the constraints at the same time. That exact version will be installed.
As an example, here’s a dependency tree from a different project of mine where click
is not installed as a top-level package, i.e., not a part of requirements.txt
nor installed via pip install click
.
$ pipdeptree | grep click
- click [required: >=7.1.2,<8.0.0, installed: 7.1.2]
- click [required: Any, installed: 7.1.2]
- click [required: >=7.1.2, installed: 7.1.2]
- click [required: >=7.1.1,<7.2.0, installed: 7.1.2]
- click [required: >=7.1.1,<7.2.0, installed: 7.1.2]
- click [required: Any, installed: 7.1.2]
After evaluating all the version specifiers – >=7.1.2,<8.0.0
, Any
, >=7.1.2
, >=7.1.1,<7.2.0
, >=7.1.1,<7.2.0
and Any
– pip
smartly evaluated that click==7.1.2
would suffice all of them and hence decided to install that.
So to summarise, now we know that the version of a package installed (top-level or dependency or dependency of dependency and so on), is determined by:
- The version specifier at the top-level, if present.
- If the previous point is absent, then a version that would conform to all the constraints in the entire dependency tree.
Conflicts
You might have already wondered, but if not, what happens when the versions of two or more dependencies at any level of the dependency tree are in conflict with each other or with the top-level package?
Let’s first see a simple conflicting example. Here’s a sample requirements.txt
:
click==8.0.3
beanie==1.7.2 # Depends on click>=7.1.2,<8.0.0 - CONFLICT!
Let’s see what happens when we try to install it:
$ pip install -r requirements.txt
...
INFO: pip is looking at multiple versions of <Python from Requires-Python> to
determine which version is compatible with other requirements. This could
take a while.
INFO: pip is looking at multiple versions of click to determine which version
is compatible with other requirements. This could take a while.
ERROR: Cannot install -r requirements.txt (line 2) and click==8.0.3 because
these package versions have conflicting dependencies.
The conflict is caused by:
The user requested click==8.0.3
beanie 1.7.2 depends on click<8.0.0 and >=7.1.2
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency
conflict
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>
We get an error, a fairly exhaustive description of conflict, some links on how to resolve it and most importantly, none of the packages will be installed, i.e., the entire process comes to a halt.
For a slight variation, here’s another example where click
is not a top-level package, but a dependency of multiple top-level packages:
beanie==1.7.2 # Depends on click>=7.1.2,<8.0.0
flask==2.2.1 # Depends on click>=8.0 - CONFLICT!
pip
will again cause a ResolutionImpossible
error (same as before) and fail the installation:
$ pip install -r requirements.txt
...
The conflict is caused by:
beanie 1.7.2 depends on click<8.0.0 and >=7.1.2
flask 2.2.1 depends on click>=8.0
...
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>
So if your dependency tree has conflicts, then you must fix them. We will look at various solutions to fix them in a bit (below).
Legacy Resolver
I used pip==22.2.2
in the examples above. The behaviour of throwing an error and halting the batch installation process from requirements.txt
or pip install pkg1 pkg2 ... pkgN
was not the case before pip==20.3
.
Even though the older (also now known as legacy) dependency resolver was able to resolve the conflicts and show them as error messages, pip
would not halt the installation process. It’d successfully continue the process by installing the first matching dependency in the list of conflicts. If you want to read more about the behaviour of the older dependency resolution algorithm and the challenges around it, head over here.
For instance, installing beanie==1.7.2
and flask==2.2.1
(incompatible due to click
) would lead to this:
$ pip --version
pip 20.2.3
$ pip install -r beanie==1.7.2 flask==2.2.1
...
Installing collected packages: pymongo, motor, multidict, idna, yarl,
typing-extensions, pydantic, click, toml, beanie, itsdangerous, zipp,
importlib-metadata, MarkupSafe, Werkzeug, Jinja2, flask
ERROR: pip's dependency resolver does not consider dependency conflicts when
selecting packages. This behaviour is the source of the following dependency
conflicts.
flask 2.2.1 requires click>=8.0, but you'll have click 7.1.2 which is
incompatible.
Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.1 Werkzeug-2.2.1
beanie-1.7.2 click-7.1.2 flask-2.2.1 idna-3.3 importlib-metadata-4.12.0
itsdangerous-2.1.2 motor-2.5.1 multidict-6.0.2 pydantic-1.9.1
pymongo-3.12.3 toml-0.10.2 typing-extensions-4.3.0 yarl-1.8.1 zipp-3.8.1
...
You’ll notice that it shows an error, but continues to install click==7.1.2
which comes from beanie
(listed before flask
in requirements.txt
). If I reverse the order, then it installs click>=8.0
which comes from flask
.
Note: If you’re on the older versions of pip
, then this re-ordering of packages in your requirements.txt
file is actually one of the solutions to resolve dependency version conflicts.
So to summarise, with the older versions of pip (<20.3
), you will see an error message regarding the conflict. But the installation of the first version of the conflicting dependency package that was encountered while generating the dependency tree will be downloaded and installed.
Fallback to Older Behaviour
In some circumstances, you may need the older (or legacy) behaviour that we just discussed, in the newer versions of pip
. There are two simple ways to do this.
The first is to use the legacy-resolver
flag:
$ pip install -r requirements.txt --user-deprecated=legacy-resolver
# or use environment variable
$ PIP_USE_DEPRECATED=legacy-resolver
$ pip install -r requirements.txt
The other way is to sort of trick pip
. When you install a list of packages together, then pip does resolve conflicts and halt the installation process in the newer versions as we already know. What you need to know is that this process can become really expensive depending upon how big the dependency tree is/will be. This is why pip
will not take into account all the packages that are already installed in your environment/system. This idea can be used to trick pip
.
So you cannot do this:
$ pip install "beanie==1.7.2" "flask==2.2.1"
But you can do this:
$ pip install "beanie==1.7.2" # Installs click==7.1.2
$ pip install "flask==2.2.1" # Uninstalls click==7.1.2 and installs click==8.1.3
...
Installing collected packages: zipp, MarkupSafe, itsdangerous, click, Werkzeug, Jinja2, importlib-metadata, flask
Attempting uninstall: click
Found existing installation: click 7.1.2
Uninstalling click-7.1.2:
Successfully uninstalled click-7.1.2
ERROR: pip's dependency resolver does not currently take into account all the
packages that are installed. This behaviour is the source of the following
dependency conflicts.
beanie 1.7.2 requires click<8.0.0,>=7.1.2, but you have click 8.1.3 which is
incompatible.
Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.1 Werkzeug-2.2.1
click-8.1.3 flask-2.2.1 importlib-metadata-4.12.0 itsdangerous-2.1.2
zipp-3.8.1
When pip
is only installing flask
, it has only one version of click
in its dependency tree. So it installs that, overriding any other versions that were already installed on the system. Then it quickly checks the dependencies of all the existing packages and compares their versions to see if there are any conflicts. If a conflict is found, it is reported in the error message (as you can see above).
Do notice that in this case, the last dependency version is the one that will eventually be installed and available, unlike the legacy resolver where the first dependency version is installed. This is obviously due to differences in the processes.
Checking Compatibility
One small trick to quickly check all kinds of dependency conflicts in your current environment is to run the following command:
$ pip check
beanie 1.7.2 has requirement click<8.0.0,>=7.1.2, but you have click 8.1.3.
Or you can also run pipdeptree
to see all the conflicts as warnings at the beginning of the output:
$ pipdeptree
Warning!!! Possibly conflicting dependencies found:
* beanie==1.7.2
- click [required: >=7.1.2,<8.0.0, installed: 8.1.3]
------------------------------------------------------------------------
beanie==1.7.2
- click [required: >=7.1.2,<8.0.0, installed: 8.1.3]
...
Flask==2.2.1
- click [required: >=8.0, installed: 8.1.3]
...
...
Solutions to Conflicts
Finally, I think once you’ve understood the dependency resolution, which versions of the dependencies get installed and when conflicts arise, it is also important to look at what are our options to resolve the conflicts. You will most likely encounter conflicts when your dependency graph is big, i.e., you are dealing with a lot of dependencies in your requirements.txt
and hence in the overall graph or tree.
The solution will of course depend on the situation.
Upgrade pip
To start with, you should make sure to always use the latest version of pip
so that the new dependency resolver is in action (make sure the legacy resolver is not being used).
Review Top-level Packages
Once the latest pip
version is in place, you should review the top-level packages or requirements in requirements.txt
. If top-level package A
and B
depend on conflicting versions of C
, then check if you can upgrade (or downgrade, but generally no) either A
or B
to resolve the conflict.
For instance, the following fails (as we saw earlier):
$ pip install "beanie==1.7.2" "flask==2.2.1"
But the following works just fine:
pip install "beanie==1.11.7" "flask==2.2.1" # Upgraded beanie from 1.7.2 to 1.11.7
Upgrading doesn’t always mean using a different version with ==
. You could also look at making your requirements more flexible, like this:
$ pip install "beanie>=1.7.2" "flask>=2.2.1" # Works just fine!
While reviewing you might also find a bunch of unused packages, make sure to remove them.
In some scenarios, if you’re stuck with using an older version of pip
or the legacy-resolver
flag, then it is expected that the dependency version to be installed is picked from the first matching (or resolved) top-level package. Sometimes this behaviour may lead to unsuitable dependency version installation.
Instead, you may want the dependency version that the next matching/resolving top-level package points to (in its sub-tree). To achieve change the order of the packages in your requirements.txt
. If this is all still confusing then look at this SO post, it has an example. Another option is to make the specific dependency version a top-level package itself.
package1
package2
common_dependency===...
As we learnt earlier, top-level package versions will override all other versions from the dependency tree.
Upgrade Dependencies
Sometimes you might encounter the worst scenario where all the versions of beanie
depend on click
version constraints/specifiers that are all incompatible with that of what all the versions of flask
depends on. Basically, imagine if all versions of package A depended on C<2.0 whereas all versions of package B
depended on C>2.0
.
Well in such cases, you’ll either have to request the package maintainer to upgrade or downgrade or “loosen” (instead of dep==2.0.0
something like dep>=2.0.0
or dep^=2.0.0
) the dependency versions. The other option is to fork the package yourself, do the version changes and then use it.
This will work if A
can work with C>=2.0
but it’s just that the package maintainer has not made that change. Or if B can work with C<=2.0 but again the maintainer didn’t want to pin it below 2.0
at all.
But if the pinnings are genuine in the sense that the dependants (A
and B
) would actually break if any changes were made to C
versions, then you may have to opt for the most frustrating option of using a different package with compatible dependencies. Also in that case, it might be a good time to casually read up on Dependency Hell :).