Using Composer and Satis to handle private php repositories

Using Composer and Satis to handle private php repositories

Composer is an awesome and powerful tool for dependency management in PHP. It allows you to declare the libraries your project depends on and it will manage (install/update) them, and their respective dependencies for you. To make your own libraries or packages usable by composer users, you need to declare them towards composer. When the package is public, less steps are required. You could host your repository online on GitHub for example, then declare it towards the packagist.org index for example. When the repository is private, it's a bit trickier, and I spent some time figuring it out, so here's a little feedback on my attempts.

Thank You for Downloading

I'm glad that you found my work useful. If you did, it might help some other people too. So why not helping me spread the word with a tweet? You could also buy me a coffee to thank me for my code and explanations if you prefer ;).
I hope you'll enjoy your download. Regards. Jeremy.

Getting started

As an organisation producing a handful of quality handmade sites per year, we quickly felt the need to capitalize on some core functionalities. Even though some of them were public or open sourced, such as our wonderwp framework for example or the pew.js library, some others required to be kept private.

Composer is at the heart of our build processes, we use it to easily manage our dependencies so we wanted our private plugins to be managed with it too. The first thing we therefore did at the time was to look at the official composer documentation regarding private repositories. Here's a summary of what I found relevant in our case:

  • By default, when you require a dependency, composer will look for it in its own packagist index. If your library or repository is not in this index, which will be the case for private repositories, then you have to tell composer where to find it thanks to the repositories section in your composer.json file.
  • You have the choice between different types of repositories, but more precisely in our case we're looking at vcs or composer. In other words, either you add a vcs entry per private dependency in your composer.json file so composer will know where to find your private repos, or you create your own private composer.
  • For our projects, we typically have more or less 6 to 10 private dependencies to add. So by choosing the vcs way of doing, we'd need to add as much vcs entries by hand as we have private dependencies. You can see the example here. Those vcs entries need to have a full GitHub path, and the machine making the composer install call need to be authenticated with GitHub via its ssh key to be authorized to pull the packages. As you can see that's quite some manual work to do, and this work is proportional to the (growing) list of our private packages. Surely there must be a better way.
  • This better way seems to be the other type called composer. I don't know if you tried to read the paragraph in the official documentation but then I did I felt a bit confused. It did feel like what we were looking for though so I decided to try it out.

Trying out to make a private composer index

Like I said, when I first read the documentation, I was left a bit confused. So I thought I'd figure it out by doing. Okay so first we need to create a packages.json file. We'll put in this file all the private repositories information, then we'll reference this file in our composer.json file to make composer aware of our private repos.

The thing that then caught my eye was what they called the minimal package definition, and that looks like this.

  1. {
  2. "name": "smarty/smarty",
  3. "version": "3.1.7",
  4. "dist": {
  5. "url": "http://www.smarty.net/files/Smarty-3.1.7.zip",
  6. "type": "zip"
  7. }
  8. }

What I understand from this is something like : "Composer, if we're trying to require smarty/smarty at version 3.1.7, you can find the zip here : http://www.smarty.net/files/Smarty-3.1.7.zip". The dist part can also be an hosted git repository url, and that's probably more the case with living proprietary private packages.

So to build our packages.json file, we should pile up a list of package definitions. For each private library we want to use, we should add one package definition per version number (and there could be quite a lot of them)

Some notable difference with vcs repos versus zips, is that you also need to provide in the dist part, a reference number, which is actually the commit id. And that's very important because keeping that reference accurate and up to date plus the fact that we need to have one package definition per library version make up the fact we'll need a tool such as private packagist or satis to automate this packages.json build, and therefore make this composer option truly viable.

Without automated solution, you'd have to go back to edit your packages.json file every time you add a new version to one of your repositories, and even worse, every time you add a new set of commits to any version to update this version's reference number. That's truly not acceptable to do all this by hand.

Automating the build

I looked for a solution to automate the packages.json build, and by looking at different resources around the internet, there are quickly two players that often come back: private packagist and satis. At the time of my investigations private packagist was just released, and was not free for an organisation like ours. I therefore decided to give satis a try because I also must say that I quite like to tackle those kind of industrilasation challenges, made of investigations and trials and errors.

Private packagist was founded by Jordi Boggiano and Nils Adermann, the creators of Composer. They put in some regular efforts to propose a product that helps packages maker to host their private repos just like you would do on the open source packagist.org, which is exactly the point of this article. Therefore you should pay them a visit to see if the quality of their services would actually be a more interesting alternative for you than building it on your own.

If you're adventurous like me then read on :)!

Installing Satis

Like explained in the satis documentation, to install satis you need to run : composer create-project composer/satis:dev-master --keep-vcs, and follow up the command prompt.

The satis command

Satis can be invoked like so: php bin/satis build <configuration-file> <output-dir>

As you can see you have three parts in this command. Firstly you need to provide the path to your satis executable, then you need to specify which configuration file you'd like to use (typically the satis.json file we'll talk about in the next paragraph), and finally a folder output where it will generate all the necessary files for your private composer to run, the packages.json file being one of them. But it will also generate a webpage presenting all of your available packages, their available version number and so on.

The output directory should then be made accessible online so we can use its url inside our project composer.json file, within the repositories section. In order to avoid https warnings, it's better to setup https on the url you're going to use, for example : https://yourprivatecomposer.com

The satis.json file

This file is there to help you configure satis, mainly to tell him where to find your private packages so it can work with them. The documentation explains the available options. In mine, I've listed all of my private packages GitHub urls, and I've specified that I want to build every branch satis will find in them through the option named "require-all": true. You can find my satis.json file attached to this article.

We are now at a point where we have a satis command ruled by a proper statis.json file, that generates our private composer index. But we still have a manual step that is to launch the satis command every time we want to update our index. The ideal would be to automate the launch of the command every time we commit to any of our repositories.

To achieve this, we need to prepare a script that would then be called via GitHub webhooks (or equivalent), every time we push new commits or new version tags, and the job of this script would be to execute the satis build command to regenerate our private composer index.

The update script

I've attached an update script inside the zip file that accompanies this article.

Let's see how you can configure it. Line 9 you have a $config array where you need to specify where your satis executable is located (that's the very first part of the statis build command), where your satis.json file is (that's the second part), and where you want the packagist index to be build, (that's the third part).

That's the only thing you should need to configure. Those arguments will be used to invoke the satis build command that the script will then execute for you.

Executing the update can be quite long, and this fact gives us two conclusions.

  • Firstly, there's no point rebuilding the entire index with all of your private packages if you only sent a push to one of them. We should consider setting up a partial rebuild.
  • Secondly, as you don't really know how long it will take, it's nice to setup some sort of notification in place so you know when the script ends. That's important because this end is the signal that you can now do a composer update from within your project to get the latest changes from your private packages.

Webhooks

Put the update script online where it can be accessed via a url, for example https://yourprivatecomposer.com/satis_webhook.php. We'll use this url as a target for our webhooks.

In GitHub, to setup a webhook, you need to go to one of your repositories, then in settings / webhooks, add a new webhook, then specify that on the push event you'd like to make a request to your remote update script url at https://yourprivatecomposer.com/satis_webhook.php.

Considering partial rebuilds, depending on how the webhook is setup, we can therefore know in the update script which repository is concerned, and only rebuild the index parts that are relevant. When pushing commits to a repository, this could mean we added some commits to an existing version number, or that we've created a new version number.

Satis has an option named --repository-url which can be used to tell it to only concentrate on scanning this one and only repository. We've paired our webhooks with this option so that all the webhooks carry the repository name in a special parameter, for example : https://yourprivatecomposer.com/satis_webhook.php?pkg=yourcompagny/yourpackagename. In case of GitHub webhooks, you might not need to pass parameters in the url because it posts you a json payload with all this information as well. The aim is that when the update script receives the webhook it should know which repository url it should update. This improvement made our script 20 times quicker to execute (because we've had around 20 private repositories at the time) so as you can see it's a huge performance improvement.

Notification

When the update script first ran, I was sure it was triggering properly thanks to webhooks, but when I had no idea if it had finished, or if it was still running. So I tried my luck by running composer update in my project and one times out of two, it either served me the 'nothing to update or install' sentence, or it started updating or installing the updated dependencies.

To avoid this waiting game, I setup a slack notification in a dedicated channel that warned everyone following this channel that one package index had been updated and was available to everyone. That was much much better, I definitely encourage you to do something similar within your own communicate medium.

Referencing your private packagist inside your projects

Now that you have a private composer index accessible online somewhere, open up your project composer.json file, and locate the repositories section. Then add up an entry of type composer with your index url like so:

  1. "repositories": [
  2. {
  3. "type": "composer",
  4. "url": "https://wpackagist.org"
  5. },
  6. {
  7. "type": "composer",
  8. "url": "https://yourprivatecomposer.com"
  9. }
  10. ]

Conclusion

Hosting your own private packagist can be quite challenging if you've never done it before. I hope this little guide would help make the process easier. For example, as I wasn't a composer specialist, it took me a few bunch of days of investigation, trial and errors before getting to a setup that was production ready. That can be seen as quite an investment. I do hope that you'll go quicker than me thanks to this article though :) .

And once you've got the whole thing running, for your developers it will be so much easier to use the private repositories. Apart from the additional line to put inside the repositories section of your project, requiring private dependencies would be nearly as straightforward as public ones. And that's the whole point of all this really.

One Response to Using Composer and Satis to handle private php repositories

  1. I know this site gives quality depending posts and additional data, is there any other site
    which gives these kinds of stuff in quality?

Leave a Reply

Your email address will not be published. Required fields are marked *

Other Useful Resources

Creating a simple image gallery with PHP & jQuery illustration

An image gallery is a very common feature for a website to have. It is an efficient way to nicely present a set of pictures. With the increasing popularity of JavaScript plugin, image galleries are nowadays implemented with rich features, effects, and transitions, in order to emphasize the given content.

When you wish to create a picture gallery, there are several steps involved in the process. In this tutorial, we will see how you can set everything up, and then add, remove, or edit pictures in your gallery, in a simple and dynamic manner. Finally, we will explain how we can setup the jQuery lightbox plugin in order to work with our pictures.

What are the benefits of this solution? This methods is sooo easy to setup, but still achieves quite a lot. Little efforts, great rewards. You only have to put your images in a folder, and give that folder path to the class. You also tell the class how big you want your pictures to be, and it handles all the hard work for you. Just by calling a method, you can see your gallery on the page. So if you are looking for a fast and convenient way to put an image gallery on a site, read on.

Read more
How to create an Ajax, validated form illustration

Even on some very famous websites and blogs based on the WordPress platform, such as Smashing Magazine or Webdesigner Depot for example, the comment forms are not checked prior to submission. Which means that if the user hits the submit button without actually filling any information, he is redirected to the famous error: "Error: please fill the required fields (name, email).".

Thanks to jQuery and its plugins, it is possible to change the way those forms are handled, in order to obtain slick effects, and an overall better user experience.

In this tutorial, we'll associate the validation plugin, and the form plugin, to get a contact form that is validated before submission, and submitted via ajax. Why don't you give it a try?

Read more
Standardizing development environments with Vagrant illustration

This is the first article of a series in which we'll explain our development process industrialisation. We'll describe how we went from a team of individuals doing things together but artisanally, to a more industrial and qualitative approach. This first topic talks about how we standardized our development environments with Vagrant.

Read more
Just a reminder that this method allows a php developer that has no JS knowledge to create php component classes, a… https://t.co/7A9s8EgzXn10 Nov / 23:53