Understanding Python Packages pip Dependency Resolver and Version Conflicts (with Solutions)
When we install a package like this –
pip install Flask –
pip needs to work out Flask’s dependencies, the dependencies of those dependencies, and so on. This recursive process builds a dependency tree or graph where multiple nodes (packages) can point to the same dependency, but different versions.
Flask may depend on packages
B, both of which may depend on
The exact version of
C to be downloaded and installed has to be figured out by
pip intelligently. It does not install multiple versions of dependencies (like you might be used to from the Node.js/npm world).
We will try to learn a few things in this article:
- A bit about dependency resolution in
pip. For a detailed understanding of dependency resolution and backtracking, you should read the documentation. Although if you don’t have much time, then the description below should be exhaustive for most folks.
- Practical examples of when a dependency (version) conflict occurs.
- Differences in how conflicts are dealt with in the new and legacy dependency resolvers (older and newer versions of
- How to check conflicts between dependencies of packages that are already installed in the current environment.
- Solutions to resolve the dependency version conflicts.
Usually, you will have a lot of packages in your
requirements.txt. So you can imagine
pip will have to resolve the entire dependency tree and eventually make sure it installs the right version of a dependency that works for all the dependants (like the packages listed). This process is called dependency resolution.
To see all the installed packages – the ones you listed in your
requirements.txt, passed directly to
pip install and all their dependencies as well – you can do
pip list. But if you wanted to see the dependency trees of all the installed packages, you can use the
pipdeptree command line utility (available from PyPI).
$ pip install pipdeptree $ pipdeptree beanie==1.11.7 - click [required: >=7, installed: 8.1.3] - motor [required: >=2.5,<4.0, installed: 3.0.0] - pymongo [required: >=4.1,<5, installed: 4.2.0] - pydantic [required: >=1.9.0, installed: 1.9.1] - typing-extensions [required: >=184.108.40.206, installed: 4.3.0] ... Flask==2.2.1 - click [required: >=8.0, installed: 8.1.3] - Jinja2 [required: >=3.0, installed: 3.1.2] - MarkupSafe [required: >=2.0, installed: 2.1.1] - Werkzeug [required: >=2.2.0, installed: 2.2.1] - MarkupSafe [required: >=2.1.1, installed: 2.1.1] ... pipdeptree==2.2.1 - pip [required: >=6.0.0, installed: 22.2.2] setuptools==49.2.1 wheel==0.37.1
With the output from
pipdeptree, you will realise that many packages share the same dependencies, but with different version comparison operators. For instance, in the output above, you can see
click is shared by
flask with different version comparison operators (
For the sake of this article, we don’t need to worry about what
flask do in terms of their functionalities. For us, they’re just random packages. Moving on, the tree or list of dependants can be much more in a large project. For instance, this is an output from one of my “real-world” projects and it shows how many times
click is resolved as a dependency of multiple dependants:
$ pipdeptree | grep click - click [required: >=4.0, installed: 8.0.3] - clickclick [required: >=1.2,<21, installed: 20.10.2] - click [required: >=4.0, installed: 8.0.3] - click [required: >=8.0, installed: 8.0.3] - click [required: >=8.0, installed: 8.0.3] - click [required: >=6.7,<9, installed: 8.0.3] - click [required: >=8.0, installed: 8.0.3] - click [required: >=8.0, installed: 8.0.3] - click [required: >=8.0, installed: 8.0.3] - click [required: >=7.1.2,<8.0.0, installed: 8.0.3] - click [required: Any, installed: 8.0.3] - click [required: >=8.0, installed: 8.0.3] - click [required: Any, installed: 8.0.3] - click [required: >=5.0.0, installed: 8.0.3]
All the entries/lines above, belong to various packages, i.e., each line of click is a dependency for a different package that got resolved by pip’s dependency resolver while building the entire dependency tree.
The most important thing to realise here is that eventually, only one version of
click will get installed, which in the output above is
8.0.3. This makes sure when you do
import click in your Python script, it always loads the same version.
In Python, you can always
import click whether you:
- Installed it as a top-level package (
- Or, as a dependency of another package (
pip install flaskthat installs
As a side note, this behaviour is different from Node.js where
npm can install and maintain multiple versions of the same dependency in multiple
How does pip decide which version to install from the dependency tree?
From the output above, how did
pip decide to install
If we specify a package with its version specifiers directly in our
requirements.txt (or via
pip install), then that takes precedence over everything else. So basically in my case, I had the following in my
... click==8.0.3 ...
This is why you see
8.0.3 as “installed” in the
pipdeptree output above. But what if the package is not directly installed? In that case,
pip will go through all the version specifiers that you see in the list above (for
click) and figure out which exact version of the package would suffice all the constraints at the same time. That exact version will be installed.
As an example, here’s a dependency tree from a different project of mine where
click is not installed as a top-level package, i.e., not a part of
requirements.txt nor installed via
pip install click.
$ pipdeptree | grep click - click [required: >=7.1.2,<8.0.0, installed: 7.1.2] - click [required: Any, installed: 7.1.2] - click [required: >=7.1.2, installed: 7.1.2] - click [required: >=7.1.1,<7.2.0, installed: 7.1.2] - click [required: >=7.1.1,<7.2.0, installed: 7.1.2] - click [required: Any, installed: 7.1.2]
After evaluating all the version specifiers –
pip smartly evaluated that
click==7.1.2 would suffice all of them and hence decided to install that.
So to summarise, now we know that the version of a package installed (top-level or dependency or dependency of dependency and so on), is determined by:
- The version specifier at the top-level, if present.
- If the previous point is absent, then a version that would conform to all the constraints in the entire dependency tree.
You might have already wondered, but if not, what happens when the versions of two or more dependencies at any level of the dependency tree are in conflict with each other or with the top-level package?
Let’s first see a simple conflicting example. Here’s a sample
click==8.0.3 beanie==1.7.2 # Depends on click>=7.1.2,<8.0.0 - CONFLICT!
Let’s see what happens when we try to install it:
$ pip install -r requirements.txt ... INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of click to determine which version is compatible with other requirements. This could take a while. ERROR: Cannot install -r requirements.txt (line 2) and click==8.0.3 because these package versions have conflicting dependencies. The conflict is caused by: The user requested click==8.0.3 beanie 1.7.2 depends on click<8.0.0 and >=7.1.2 To fix this you could try to: 1. loosen the range of package versions you've specified 2. remove package versions to allow pip attempt to solve the dependency conflict ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>
We get an error, a fairly exhaustive description of conflict, some links on how to resolve it and most importantly, none of the packages will be installed, i.e., the entire process comes to a halt.
For a slight variation, here’s another example where
click is not a top-level package, but a dependency of multiple top-level packages:
beanie==1.7.2 # Depends on click>=7.1.2,<8.0.0 flask==2.2.1 # Depends on click>=8.0 - CONFLICT!
pip will again cause a
ResolutionImpossible error (same as before) and fail the installation:
$ pip install -r requirements.txt ... The conflict is caused by: beanie 1.7.2 depends on click<8.0.0 and >=7.1.2 flask 2.2.1 depends on click>=8.0 ... ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>
So if your dependency tree has conflicts, then you must fix them. We will look at various solutions to fix them in a bit (below).
pip==22.2.2 in the examples above. The behaviour of throwing an error and halting the batch installation process from
pip install pkg1 pkg2 ... pkgN was not the case before
Even though the older (also now known as legacy) dependency resolver was able to resolve the conflicts and show them as error messages,
pip would not halt the installation process. It’d successfully continue the process by installing the first matching dependency in the list of conflicts. If you want to read more about the behaviour of the older dependency resolution algorithm and the challenges around it, head over here.
For instance, installing
flask==2.2.1 (incompatible due to
click) would lead to this:
$ pip --version pip 20.2.3 $ pip install -r beanie==1.7.2 flask==2.2.1 ... Installing collected packages: pymongo, motor, multidict, idna, yarl, typing-extensions, pydantic, click, toml, beanie, itsdangerous, zipp, importlib-metadata, MarkupSafe, Werkzeug, Jinja2, flask ERROR: pip's dependency resolver does not consider dependency conflicts when selecting packages. This behaviour is the source of the following dependency conflicts. flask 2.2.1 requires click>=8.0, but you'll have click 7.1.2 which is incompatible. Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.1 Werkzeug-2.2.1 beanie-1.7.2 click-7.1.2 flask-2.2.1 idna-3.3 importlib-metadata-4.12.0 itsdangerous-2.1.2 motor-2.5.1 multidict-6.0.2 pydantic-1.9.1 pymongo-3.12.3 toml-0.10.2 typing-extensions-4.3.0 yarl-1.8.1 zipp-3.8.1 ...
You’ll notice that it shows an error, but continues to install
click==7.1.2 which comes from
beanie (listed before
requirements.txt). If I reverse the order, then it installs
click>=8.0 which comes from
Note: If you’re on the older versions of
pip, then this re-ordering of packages in your
requirements.txt file is actually one of the solutions to resolve dependency version conflicts.
So to summarise, with the older versions of pip (
<20.3), you will see an error message regarding the conflict. But the installation of the first version of the conflicting dependency package that was encountered while generating the dependency tree will be downloaded and installed.
Fallback to Older Behaviour
In some circumstances, you may need the older (or legacy) behaviour that we just discussed, in the newer versions of
pip. There are two simple ways to do this.
The first is to use the
$ pip install -r requirements.txt --user-deprecated=legacy-resolver # or use environment variable $ PIP_USE_DEPRECATED=legacy-resolver $ pip install -r requirements.txt
The other way is to sort of trick
pip. When you install a list of packages together, then pip does resolve conflicts and halt the installation process in the newer versions as we already know. What you need to know is that this process can become really expensive depending upon how big the dependency tree is/will be. This is why
pip will not take into account all the packages that are already installed in your environment/system. This idea can be used to trick
So you cannot do this:
$ pip install "beanie==1.7.2" "flask==2.2.1"
But you can do this:
$ pip install "beanie==1.7.2" # Installs click==7.1.2 $ pip install "flask==2.2.1" # Uninstalls click==7.1.2 and installs click==8.1.3 ... Installing collected packages: zipp, MarkupSafe, itsdangerous, click, Werkzeug, Jinja2, importlib-metadata, flask Attempting uninstall: click Found existing installation: click 7.1.2 Uninstalling click-7.1.2: Successfully uninstalled click-7.1.2 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. beanie 1.7.2 requires click<8.0.0,>=7.1.2, but you have click 8.1.3 which is incompatible. Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.1 Werkzeug-2.2.1 click-8.1.3 flask-2.2.1 importlib-metadata-4.12.0 itsdangerous-2.1.2 zipp-3.8.1
pip is only installing
flask, it has only one version of
click in its dependency tree. So it installs that, overriding any other versions that were already installed on the system. Then it quickly checks the dependencies of all the existing packages and compares their versions to see if there are any conflicts. If a conflict is found, it is reported in the error message (as you can see above).
Do notice that in this case, the last dependency version is the one that will eventually be installed and available, unlike the legacy resolver where the first dependency version is installed. This is obviously due to differences in the processes.
One small trick to quickly check all kinds of dependency conflicts in your current environment is to run the following command:
$ pip check beanie 1.7.2 has requirement click<8.0.0,>=7.1.2, but you have click 8.1.3.
Or you can also run
pipdeptree to see all the conflicts as warnings at the beginning of the output:
$ pipdeptree Warning!!! Possibly conflicting dependencies found: * beanie==1.7.2 - click [required: >=7.1.2,<8.0.0, installed: 8.1.3] ------------------------------------------------------------------------ beanie==1.7.2 - click [required: >=7.1.2,<8.0.0, installed: 8.1.3] ... Flask==2.2.1 - click [required: >=8.0, installed: 8.1.3] ... ...
Solutions to Conflicts
Finally, I think once you’ve understood the dependency resolution, which versions of the dependencies get installed and when conflicts arise, it is also important to look at what are our options to resolve the conflicts. You will most likely encounter conflicts when your dependency graph is big, i.e., you are dealing with a lot of dependencies in your
requirements.txt and hence in the overall graph or tree.
The solution will of course depend on the situation.
To start with, you should make sure to always use the latest version of
pip so that the new dependency resolver is in action (make sure the legacy resolver is not being used).
Review Top-level Packages
Once the latest
pip version is in place, you should review the top-level packages or requirements in
requirements.txt. If top-level package
B depend on conflicting versions of
C, then check if you can upgrade (or downgrade, but generally no) either
B to resolve the conflict.
For instance, the following fails (as we saw earlier):
$ pip install "beanie==1.7.2" "flask==2.2.1"
But the following works just fine:
pip install "beanie==1.11.7" "flask==2.2.1" # Upgraded beanie from 1.7.2 to 1.11.7
Upgrading doesn’t always mean using a different version with
==. You could also look at making your requirements more flexible, like this:
$ pip install "beanie>=1.7.2" "flask>=2.2.1" # Works just fine!
While reviewing you might also find a bunch of unused packages, make sure to remove them.
In some scenarios, if you’re stuck with using an older version of
pip or the
legacy-resolver flag, then it is expected that the dependency version to be installed is picked from the first matching (or resolved) top-level package. Sometimes this behaviour may lead to unsuitable dependency version installation.
Instead, you may want the dependency version that the next matching/resolving top-level package points to (in its sub-tree). To achieve change the order of the packages in your
requirements.txt. If this is all still confusing then look at this SO post, it has an example. Another option is to make the specific dependency version a top-level package itself.
package1 package2 common_dependency===...
As we learnt earlier, top-level package versions will override all other versions from the dependency tree.
Sometimes you might encounter the worst scenario where all the versions of
beanie depend on
click version constraints/specifiers that are all incompatible with that of what all the versions of
flask depends on. Basically, imagine if all versions of package A depended on C<2.0 whereas all versions of package
B depended on
Well in such cases, you’ll either have to request the package maintainer to upgrade or downgrade or “loosen” (instead of
dep==2.0.0 something like
dep^=2.0.0) the dependency versions. The other option is to fork the package yourself, do the version changes and then use it.
This will work if
A can work with
C>=2.0 but it’s just that the package maintainer has not made that change. Or if B can work with C<=2.0 but again the maintainer didn’t want to pin it below
2.0 at all.
But if the pinnings are genuine in the sense that the dependants (
B) would actually break if any changes were made to
C versions, then you may have to opt for the most frustrating option of using a different package with compatible dependencies. Also in that case, it might be a good time to casually read up on Dependency Hell :).