When you develop with Python, you normally use PyPI as the source for your package dependencies whether they're third party or your own. Poetry is a great way to handle this administrative burden - at least it's better than using the regular pip
and requirements.txt
method.
But what if you don't want everything to come from PyPI? What if you want to implement your own private package index on your own network? What if you want to mix and match the sources on a package-by-package basis, so you can run development packages against your system?
Don't worry, it's pretty straightforward. Let's dive in...
Why Would I Set Up My Own Version Of PyPI?
If you want to test new features or updates that you've made to a package, you can use feature flags or some other method, and deploy to PyPI. This makes your updates available to everyone to use in their production systems, thereby increasing your credibility with numerous LinkedIn influencers. Many people don't specify versions for packages, so your new one automatically gets dragged down instantly. You did test it FULLY, didn't you? Users get really angry if you release bugs! Even if you're giving your software away! Anyway, cross your fingers and run that release command! Do it!
Alternatively, you might not want to live on that particular knife edge. You can avoid this by using your own package index, and building up the automated tests while you're developing your modules. You won't win any virtue points from the "coding experts" and "leaders" on social media for doing this, but it keeps customers happy and that's what usually matters.
I Don't Want To Upset The Users! How Do I Set Up My Server?
Rather than writing your own PyPI server code, it's probably easier to use one of the servers that are freely available. The Python Packaging User Guide has a list of them - just follow the instructions for the one you choose. Your package index can be on a local or networked file system, cloud storage, or anywhere else depending upon the server you use.
I Don't Suppose I Can Just Use The test.PyPI Index...
Not normally, no.
Why Not? You Used It In Your Last Article (Which Was Excellent By The Way!)
As I mentioned in How To Use Poetry to Publish a Python Package on PyPI, the test.PyPI index is a great thing, which allows you to test builds and uploads. The problem is that when your module has dependencies, they're not always available in the index, and if they are, you don't know if they're really what you think.
What Do You Mean?
The test.PyPI index is a separate system from the regular PyPI index. There is nothing stopping anyone from adding any package to the test index if it doesn't already exist. So, you could end up with a package whose name matches one in PyPI, but which does something totally different. It could even be malicious. This is called a dependency confusion attack.
Alright, I've Created My Own Package Index! How Do I Use It?
That's the easy part, well, it is to start with. First, you need to set up your Package Index as a repository in Poetry. If you want to call the new repository myprivaterepo
, then from the command line type:
poetry config repositories.myprivaterepo <Private Repo Upload URL>
poetry config PyPI-token.myprivaterepo <API Token>
To make sure that the configuration is correct, use the dry run feature again to test it:
poetry publish -r myprivaterepo --build --dry-run
The output of this command should describe the process of building and then publishing your package to your private repository.
Once you've got this working, you can delete the --dry-run option and run the command again to make sure everything gets built and uploaded. The command you need is:
poetry publish -r myprivaterepo --build
Go to the project that will be using the test package. Open a terminal window in the directory that contains the pyproject.toml
file and run the following:
poetry source add myprivaterepo <Private Repo Upload URL>
This will add the following section to the pyproject.toml
file:
[[tool.poetry.source]]
name = "myprivaterepo"
url = <Private Repo Upload URL>
priority = "primary"
This means that Poetry will try to download the dependencies from the myprivaterepo
package index first. If it can't find them, it will revert to using the standard PyPI index.
Can I Specify The Specific Packages To Download From Each Index?
Yes! The method above tries to download all packages from the primary source before PyPI. This is great, until you use a package that isn't in your private repository.
You could change the priority of the source to "supplemental". This means Poetry will only download packages from that source if it can't find them in PyPI.
You'd add this using:
poetry source add --priority=supplemental myprivaterepo <Private Repo Upload URL>
which would add the following to the pyproject.toml
file:
[[tool.poetry.source]]
name = "myprivaterepo"
url = <Private Repo Upload URL>
priority = "supplemental"
As with the primary priority, this applies to ALL dependencies. If you want to be able to specify the package index to be used on a package-by-package basis, use the explicit priority:
poetry source add --priority=explicit myprivaterepo <Private Repo Upload URL>
This gives a pyproject.toml
entry of:
[[tool.poetry.source]]
name = "myprivaterepo"
url = <Private Repo Upload URL>
priority = "explicit"
This will let you specify dependency versions, and where to get them, on the command line:
poetry add --source myprivaterepo primegenerators==0.0.8
which gets added to pyproject.toml
as:
[tool.poetry.dependencies]
...
primegenerators = {version = "0.0.8", source = "myprivaterepo"}
Of course, if you want to make things even more complex, there's no reason you can't add one or more of your own package indexes, and drag each package from a different location. I don't know why you'd do it, but it's possible.
So What Does This All Mean?
All this means that you can set up your development and test environments to use modules from any combination of PyPI and your own private package indexes. If you want to test one or more locally-developed modules, put them into your own repository, refer to that in your poetry configuration, and use PyPI for the rest. If you do this, you know any bugs in your own code won't affect external modules, and you can take your time updating, debugging and testing them. Whatever you do, your changes can't upset the users of your modules. Your users won't be complaining on social media about the software you supply them with for free, and should keep using it happily.