Handling private php repositories with Composer & Satis

Getting started

As an organisation producing a handful of quality handmade sites per year, we quickly felt the need to capitalize on some core functionalities. Even though some of them were public or open sourced, such as our wonderwp framework for example or the pew.js library, some others required to be kept private.

Composer is at the heart of our build processes, we use it to easily manage our dependencies so we wanted our private plugins to be managed with it too. The first thing we therefore did at the time was to look at the official composer documentation regarding private repositories. Here’s a summary of what I found relevant in our case:

• By default, when you require a dependency, composer will look for it in its own packagist index. If your library or repository is not in this index, which will be the case for private repositories, then you have to tell composer where to find it thanks to the repositories section in your composer.json file.
• You have the choice between different types of repositories, but more precisely in our case we’re looking at vcs or composer. In other words, either you add a vcs entry per private dependency in your composer.json file so composer will know where to find your private repos, or you create your own private composer.
• For our projects, we typically have more or less 6 to 10 private dependencies to add. So by choosing the vcs way of doing, we’d need to add as much vcs entries by hand as we have private dependencies. You can see the example here. Those vcs entries need to have a full GitHub path, and the machine making the composer install call need to be authenticated with GitHub via its ssh key to be authorized to pull the packages. As you can see that’s quite some manual work to do, and this work is proportional to the (growing) list of our private packages. Surely there must be a better way.
• This better way seems to be the other type called composer. I don’t know if you tried to read the paragraph in the official documentation but then I did I felt a bit confused. It did feel like what we were looking for though so I decided to try it out.

Trying out to make a private composer index

Like I said, when I first read the documentation, I was left a bit confused. So I thought I’d figure it out by doing. Okay so first we need to create a packages.json file. We’ll put in this file all the private repositories information, then we’ll reference this file in our composer.json file to make composer aware of our private repos.

The thing that then caught my eye was what they called the minimal package definition, and that looks like this.

{
    “name”: “smarty/smarty”,
    “version”: “3.1.7”,
    “dist”: {
        “url”: “http://www.smarty.net/files/Smarty-3.1.7.zip”,
        “type”: “zip”
    }
}

What I understand from this is something like : “Composer, if we’re trying to require smarty/smarty at version 3.1.7, you can find the zip here : http://www.smarty.net/files/Smarty-3.1.7.zip”. The dist part can also be an hosted git repository url, and that’s probably more the case with living proprietary private packages.

So to build our packages.json file, we should pile up a list of package definitions. For each private library we want to use, we should add one package definition per version number (and there could be quite a lot of them)

Some notable difference with vcs repos versus zips, is that you also need to provide in the dist part, a reference number, which is actually the commit id. And that’s very important because keeping that reference accurate and up to date plus the fact that we need to have one package definition per library version make up the fact we’ll need a tool such as private packagist or satis to automate this packages.json build, and therefore make this composer option truly viable.

Without automated solution, you’d have to go back to edit your packages.json file every time you add a new version to one of your repositories, and even worse, every time you add a new set of commits to any version to update this version’s reference number. That’s truly not acceptable to do all this by hand.

Automating the build

I looked for a solution to automate the packages.json build, and by looking at different resources around the internet, there are quickly two players that often come back: private packagist and satis. At the time of my investigations private packagist was just released, and was not free for an organisation like ours. I therefore decided to give satis a try because I also must say that I quite like to tackle those kind of industrilasation challenges, made of investigations and trials and errors.

Private packagist was founded by Jordi Boggiano and Nils Adermann, the creators of Composer. They put in some regular efforts to propose a product that helps packages maker to host their private repos just like you would do on the open source packagist.org, which is exactly the point of this article. Therefore you should pay them a visit to see if the quality of their services would actually be a more interesting alternative for you than building it on your own.

If you’re adventurous like me then read on :)!

Installing Satis

Like explained in the satis documentation, to install satis you need to run : composer create-project composer/satis:dev-master --keep-vcs, and follow up the command prompt.

The satis command

Satis can be invoked like so: php bin/satis build <configuration-file> <output-dir>

As you can see you have three parts in this command. Firstly you need to provide the path to your satis executable, then you need to specify which configuration file you’d like to use (typically the satis.json file we’ll talk about in the next paragraph), and finally a folder output where it will generate all the necessary files for your private composer to run, the packages.json file being one of them. But it will also generate a webpage presenting all of your available packages, their available version number and so on.

The output directory should then be made accessible online so we can use its url inside our project composer.json file, within the repositories section. In order to avoid https warnings, it’s better to setup https on the url you’re going to use, for example : https://yourprivatecomposer.com

The satis.json file

This file is there to help you configure satis, mainly to tell him where to find your private packages so it can work with them. The documentation explains the available options. In mine, I’ve listed all of my private packages GitHub urls, and I’ve specified that I want to build every branch satis will find in them through the option named “require-all”: true. You can find my satis.json file attached to this article.

We are now at a point where we have a satis command ruled by a proper statis.json file, that generates our private composer index. But we still have a manual step that is to launch the satis command every time we want to update our index. The ideal would be to automate the launch of the command every time we commit to any of our repositories.

To achieve this, we need to prepare a script that would then be called via GitHub webhooks (or equivalent), every time we push new commits or new version tags, and the job of this script would be to execute the satis build command to regenerate our private composer index.

The update script

I’ve attached an update script inside the zip file that accompanies this article.

Let’s see how you can configure it. Line 9 you have a $config array where you need to specify where your satis executable is located (that’s the very first part of the statis build command), where your satis.json file is (that’s the second part), and where you want the packagist index to be build, (that’s the third part).

That’s the only thing you should need to configure. Those arguments will be used to invoke the satis build command that the script will then execute for you.

Executing the update can be quite long, and this fact gives us two conclusions.

• Firstly, there’s no point rebuilding the entire index with all of your private packages if you only sent a push to one of them. We should consider setting up a partial rebuild.
• Secondly, as you don’t really know how long it will take, it’s nice to setup some sort of notification in place so you know when the script ends. That’s important because this end is the signal that you can now do a composer update from within your project to get the latest changes from your private packages.

Webhooks

Put the update script online where it can be accessed via a url, for example https://yourprivatecomposer.com/satis_webhook.php. We’ll use this url as a target for our webhooks.

In GitHub, to setup a webhook, you need to go to one of your repositories, then in settings / webhooks, add a new webhook, then specify that on the push event you’d like to make a request to your remote update script url at https://yourprivatecomposer.com/satis_webhook.php.

Considering partial rebuilds, depending on how the webhook is setup, we can therefore know in the update script which repository is concerned, and only rebuild the index parts that are relevant. When pushing commits to a repository, this could mean we added some commits to an existing version number, or that we’ve created a new version number.

Satis has an option named –repository-url which can be used to tell it to only concentrate on scanning this one and only repository. We’ve paired our webhooks with this option so that all the webhooks carry the repository name in a special parameter, for example : https://yourprivatecomposer.com/satis_webhook.php?pkg=yourcompagny/yourpackagename. In case of GitHub webhooks, you might not need to pass parameters in the url because it posts you a json payload with all this information as well. The aim is that when the update script receives the webhook it should know which repository url it should update. This improvement made our script 20 times quicker to execute (because we’ve had around 20 private repositories at the time) so as you can see it’s a huge performance improvement.

Notification

When the update script first ran, I was sure it was triggering properly thanks to webhooks, but when I had no idea if it had finished, or if it was still running. So I tried my luck by running composer update in my project and one times out of two, it either served me the ‘nothing to update or install’ sentence, or it started updating or installing the updated dependencies.

To avoid this waiting game, I setup a slack notification in a dedicated channel that warned everyone following this channel that one package index had been updated and was available to everyone. That was much much better, I definitely encourage you to do something similar within your own communicate medium.

Referencing your private packagist inside your projects

Now that you have a private composer index accessible online somewhere, open up your project composer.json file, and locate the repositories section. Then add up an entry of type composer with your index url like so:

“repositories”: [{
    “type”: “composer”,
    “url”: “https://wpackagist.org”
},
{
    “type”: “composer”,
    “url”: “https://yourprivatecomposer.com”
}]

Conclusion

Hosting your own private packagist can be quite challenging if you’ve never done it before. I hope this little guide would help make the process easier. For example, as I wasn’t a composer specialist, it took me a few bunch of days of investigation, trial and errors before getting to a setup that was production ready. That can be seen as quite an investment. I do hope that you’ll go quicker than me thanks to this article though 🙂 .

And once you’ve got the whole thing running, for your developers it will be so much easier to use the private repositories. Apart from the additional line to put inside the repositories section of your project, requiring private dependencies would be nearly as straightforward as public ones. And that’s the whole point of all this really.