Getting started with Jenkins Blue Ocean Pipelines

The first thing that struck me when dipping a toe into Continuous Integration is the barrier of entry that I feel is quite high. I’ve had some hard time finding information about how to to properly get on with the subject, get a grasp of the tools and the code needed. I can say it properly kicked me out of my comfort zone, especially as I was not surrounded by people who had done it before from whom I could have learned more easily. Special thanks to James Dumay for his time and help. I’ve also found this google group full of helpful people.

Strategy

First lesson that I learned is that it’s better to establish a strategy. Jenkins is a powerful tool that can be used to do many different things. So what do you want it to do? I did not know at the time and that diserved me, but I ended up finding one : I consider Jenkins as a team mate, whom I can ask to put our code into production (or staging) when we push code on specific branches (master and develop).

If that were you that would be asked to push code into production manually, which steps would be required?

  • Pull the latest changes from the develop branch
  • Install composer vendors
  • Install node vendors
  • Compile assets (Sass to CSS for example), combine, minify, timestamp
  • Test that everything is all right all around
  • If yes send the code to staging
  • Migrate db changes to staging
  • Test that everything is all right all around
  • Notify project managers that the changes are online

That’s what I’m going to ask Jenkins to do for me. That would automate all the process to push code into staging here, and apparently that is called a pipeline.

The obvious benefit of automating this is to save the time it takes a developer to do this manually every time, but there are also some other advantages.

Security for example, as all the data required to run these scripts would be hidden from the developers, kept secretly by Jenkins. This is especially useful if you work with remote developers for example (like freelancers that are not part of your permanent team), and you’re not at ease with the idea of giving them an ssh access to your server.

This approach also removes the pain that could be associated by doing all those steps manually. As a result, you might get an easier on boarding from your team, and more frequent pushes to the staging environment, because in the end the developer’s job ends when he has merged his code on the develop branch.

Delegating the knowledge to the machine is also a good knowledge capitalisation in the long run. Maybe there’s a deployment expert in your team, what will you do if this person leaves the company? Or maybe you know all the steps to deploy a project right now, but if you append to work on something else that works differently for 6 months, then go back to this project for a fix, will you still remember everything that needs to be done? What about if you delegate this fix to a new teammate? Jenkins will always remember and do all the necessary tasks in the right order. One less burden on your shoulders.

And you? What’s your CI strategy?

Source Control Management (SCM)

Continous integration puts SCM at the heart of the process. It’s by doing certain actions on the SCM that you’ll trigger CI builds.

Here again you’ve got several strategies at your disposal, we’ve chosen to use GitFlow. That’s a bit beyond the scope of this article, I’m not going to expand too much on GitFlow here, but there are nonetheless some aspects to mention.

For us, in GitFlow, the master branch illustrates the state of what is currently on the production server, and the develop branch illustrates the state of what is currently on the staging server. When we develop a new feature, we open a feature branch that is forked from develop. We work on this branch until we are ready to merge it back to develop, by opening a pull request for instance.

We’ve configured our GitHub repositories to send a web hook to Jenkins when a change happens to them, this way Jenkins gets notified of the change and launches a new CI build. You could also ask Jenkins to periodically poll your SCM, but that’s less efficient.

To add this kind of webhook, go to your github repository page, then Settings / Webhooks:

As ou can see, SCM is really at the heart of all this as CI relies on branch state and branch actions to know what to build.

Installing blue ocean

I’ll consider that you already have a running Jenkins instance to concentrate on the addition of blue ocean.

Blue ocean is actually a Jenkins plugin, so to install it you can go to > Manage Jenkins > Manage Plugins > Available, then type Blue ocean inside the filter input search.

Once you’ve located the blue ocean plugin, install it. And that is all really because Jenkins will take care of downloading all the required dependencies for you.

Once the plugin installed and the Jenkins restarted, you should have an ‘open blue ocean’ link in the left hand side sidebar.

The Jenkinsfile

You didn’t need this kind of file before to make CI pipelines with Jenkins, this is new. This file allows you to declare a pipeline directly in your project instead of declaring it by configuring it inside jenkins.

By default, Jenkins looks for a Jenkinsfile directly at the root of your project files, but you can tell him to look elsewhere.

Depending on the kind of job you’ll declare later on in Jenkins, this file could also serve as a trigger to create jobs automatically, from within GitHub organizations for example.

The pipeline

Like we’ve said earlier, our aim is to automate different tasks that we repeatedly do when pushing code online.

Here’s the essence the pipeline script that I usually use:

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        script{
        	defineVariables();

            echo "Starting Build #${env.BUILD_ID}, triggered by $BRANCH_NAME";

            if(env.runComposer=='true'){
                try {
	                sh 'composer install --no-dev --prefer-dist';
                } catch(exc){
                    handleException('Composer install failed', exc);
                }
	        } else {
	        	echo 'skipped composer install';
	        }

	        if(env.runNpm=='true'){
	            try {
	        	    sh 'npm install';
                } catch(exc){
                    handleException('npm install failed',exc);
                }
	        } else {
	        	echo 'skipped npm install';
	        }

            if(env.runBuild=='true'){
	            try {
                    sh 'npm run sprites';
                    if(BRANCH_NAME=='master'){
                        sh 'npm run build:prod';
                    } else {
                        sh 'npm run build';
                    }
                } catch(exc){
                    handleException('Building the front failed',exc);
                }
            } else {
            	echo 'skipped npm sprites & build';
            }
        }
      }
    }
    stage('Deploy') {
        steps {
            script {
            	try{
	                echo "Deploying $BRANCH_NAME branch"
	                def creds = loadCreds("livraison_occitanie_${BRANCH_NAME}_credentials");
	                deployCode(creds);
	                finalizeDistantMigration(creds);
	            } catch(exc){
	            	  handleException("The $BRANCH_NAME branch deployment failed",exc);
              }
            }
        }
    }
    stage('Integration tests') {
        steps {
            script {
                try {
                    if(env.runCypress=='true'){
                        def host = '';
                        if(env.siteUrl){
                            host = env.siteUrl;
                            echo "Starting integration tests on $host"
                            sh "cypress run --env host=$host"
                        } else {
                            echo 'No host defined to run cypress against';
                        }
                    } else {
                        echo 'Skipped integration tests'
                    }
	            } catch(exc){
	            	handleException("Cypress tests failed, which means you have a problem on your $BRANCH_NAME live environment",exc);
                }
            }
        }
    }
    stage('notify'){
        steps {
            script {
    			    notify(env.slackMsg,env.slackColor);
            }
        }
    }
  }
}

I’ve omitted function declarations for the sake of clarity, but you can download the full script here. Let’s explain what it does in a bit more details, but remember, I’m in now way an expert, this is truly experimental. That works for my needs though, so let’s see if it can be useful for you as well.

Local build

The first big step is to have Jenkins create a local build for him. In a GitHub multi branch pipeline, to do so it first pulls the latest changes from your source control. I realized that’s an important thing to know, because it implies that Jenkins needs to have enough space to do so. And the more branches you build, the more space you need. Some other CI tools do not work like this by pulling the code first, they’re just scripting tools. But in a GitHub multi branch pipeline, Jenkins pulls the code first. That’s probably a reason why there are so many cloud based CI services online actually.

Once the code has been pulled, I ask him to log a few messages with information about the build. You can find the list of accessible variables from inside a job there: Pipeline syntax > Global variable reference.

Then I run a composer install and a npm install. I’ve had troubles with these instructions when I started, because I realized that it depends on the capacities of the agent in which you choose to run your pipeline. So it worked on my machine, but not online because my online Jenkins didn’t have natively access to those tools. This is because the agent I have chosen is ‘any’. You could choose a docker agent instead, if so make sure this docker agent has the right capabilities you then use in your pipeline script.

Tests

The thing I realised with tests is that the build that Jenkins creates while running a pipeline should be considered as autonomous and disposable. In other words you should be able to recreate the entire environment with code and database, run your tests against it, and if it works carry on with the pipeline, to eventually destroy the entire local build at the end, hence the autonomous and disposable. If your unit tests require a complete database bootstrapping for example, you should make sure that your pipeline is able to recreate the complete environment for your tests to run correctly, and I find that this is not an easy task.

Once your environment is ready to run your tests, you can run many different types of tests against it, in sequence or in parallel. For example, you could run unit tests first, then a set of parallel browser tests, and if all goes well, let the pipeline carry on.

I’ve tried a unit test setup with phpunit, and Jenkins is able to understand your test report directly out of the box and abort the build if tests do not pass. If you want to produce some code coverage stats, it won’t work unless you have on your Jenkins machine a tool that can do, such as Xdebug for example.

Delivery

In my strategy, I imagined to deploy the code the the remote environment by using rsync. I know how to use this command, that wasn’t really the issue here, it was more how to handle credentials safely. I didn’t feel like writing the complete rsync command along with the user authentication in it directly in a script that is versioned inside the project (remember the security considerations we evoked at the beginning). Furthermore, the user used by the command changes depending if I need to deploy the code to staging or production environment.

That’s when I learned about credentials. With credentials you can store sensitive information in different format against an id you can use to retrieve that information from within the pipeline. What I like to do is to create a JSON file where I put all the information I need for the build, then store this file as a secret file credential. Inside the pipeline I load up this file and have access to all the secret login, passwords, db names, path…

Also bear in mind that in this kind of setup it’s the Jenkins machine that works for you and will execute the commands you want in a non-interactive mode. That means you won’t have the opportunity to enter parameters by hand along the way. So you need to parameterize your commands, and make sure no password prompt will show up during a command. This is usually avoided by authorizing the Jenkins machine on the remote host via an ssh key.

Notification

If you don’t want to open up your Jenkins every time you push code on a branch wondering if the build is running or not, it’s nice to have a notification mechanism in place. You could send notifications when a build starts, fails, succeeds, and this way everyone that is interested in the project could follow what’s going on.

Enabling a slack notification once every build is finished has had a very positive impact on my dev teams, but also beyond that, on project managers and POs for example. They found it quite useful to follow what was pushed on the staging environment along the way, it was a good informal communication from the developer to them that let them know the work is progressing. (Side note, this requires the slack notification plugin).

Closing thoughs on the pipeline script

I know this script is far from perfect, there are a number of things that bother me, for example there’s a bit of duplicate code, and I’d like the script to send a slack notification when the build fails. I’d also like to launch certain actions only if a particular file changed, like composer install only if composer.lock changed for example, I’d like to have near 0 downtime deploys with current folders symlinked and so on.

But I’m also pleased that it does perform the tasks I wanted to automate. It’s able to create its own build on its own, it’s able to deploy the code remotely by looking up credentials securely, it’s able to perform some db changes, and it notifies in slack when a build’s online, and so far that’s been a great achievement for me knowing the fact that I’m not a sysadmin and that I’m not a CI expert at all. I figured that with CI, it’s better to start small and iterate than to imagine the ultimate pipeline right away. I wasn’t really confident but I tried a few things out, figured out pitfalls, made some baby steps, and eventually got something useful working.

I’d love if a proficient jenkins pipeline user would give me some advice on the points to improve. In the meantime you can download the entire script here.

If you’re looking at a good resource to learn the pipeline syntax, I recommend this one : Declarative Pipeline With Jenkins

Declaring a new multibranch pipeline in jenkins

Having a new Jenkinsfile ready to be read by Jenkins in your project is good, but not enough, you also need to create a new Jenkins project to associate your dev project.

To do so, go to new Item, and set the project type to multi branch pipeline.

Remember that SCM is at the core of this multi branch pipeline concept, so that’s not a surprise to see an SCM source settings here next. In the branch source section, connect your SCM source. I tend to use GitHub so that’s what I choose in the dropdown. This has for effect to add a GitHub settings panel to fill up.

One of the thing it asks you is to attach a credential here, I’ve therefore created a credential with my GitHub account information so Jenkins can use that to connect to my account on my behalf to retrieve the list of all the repositories I have access to, be it public or private. The owner input is the GitHub owner, yourself or your organisation most probably.

Once your credential and owner filled, you should have access to all of your repositories in the Repository dropdown, so pick your project with the Jenkinsfile inside.

By default, Jenkins will clone every branch you push to your remote repository to its local machine, and run the Jenkinsfile on it. This could require a very good machine if you have many branches to build, and / or many projects managed by Jenkins. I wasn’t really aware of that before I started CI. Cloning locally many different projects could use up a huge amount of disk space. So I’ve instructed Jenkins to only build certain branch patterns automatically (The filter by name (with regular expression) settings) , and to only keep a limited number of builds per project, and within a limited period of time (Days to keep old items and Max # of old items to keep).

So as you can see, even with a limited number of settings, you can get your new job running quite rapidly.

Now that you have your Jenkinsfile in your project, and a Jenkins job configured properly, (and the github webhook we talked about earlier setup properly), once you start pushing code to your remote repository, Jenkins should be listening for the push event to grab your code, and run your pipeline script against it.

Conclusion

I find the new blue ocean UI very very nice compared to the ageing previous UI. It’s fresh and bright, feels flat and modern, but it’s not just a pretty face. The new UX is nice as well.

For example, you can browse the list of all your pipelines, put the ones you care the most about at the moment in favourites which is nice because it puts them at the top of your pipeline list.

When viewing the job detail page, the new UI looks so much better than the legacy view! You can see all the pipeline steps, and if anything goes wrong the view will point you to the issue right away, saving you from browsing 3 kilometres long logs.

The new pipeline syntax is more declarative, and therefore might require more work from the developer to master its syntax, and much time from him to implement what he wants. But once you have a solid pipeline running, the gain of time and energy in the long run is definitely worth it.

I’d always be thankful to my loyal Jenkins for helping me pushing my team’s code online every time we send a few commits, and I’m delighted to use this stable and nice looking tool daily.

I hope this feedback on my personal experience will be useful to any fellow developer out there. Good luck setting up your own pipelines! Let me know what you’ve built in the comments.

Handling private php repositories with Composer & Satis

Getting started

As an organisation producing a handful of quality handmade sites per year, we quickly felt the need to capitalize on some core functionalities. Even though some of them were public or open sourced, such as our wonderwp framework for example or the pew.js library, some others required to be kept private.

Composer is at the heart of our build processes, we use it to easily manage our dependencies so we wanted our private plugins to be managed with it too. The first thing we therefore did at the time was to look at the official composer documentation regarding private repositories. Here’s a summary of what I found relevant in our case:

  • By default, when you require a dependency, composer will look for it in its own packagist index. If your library or repository is not in this index, which will be the case for private repositories, then you have to tell composer where to find it thanks to the repositories section in your composer.json file.
  • You have the choice between different types of repositories, but more precisely in our case we’re looking at vcs or composer. In other words, either you add a vcs entry per private dependency in your composer.json file so composer will know where to find your private repos, or you create your own private composer.
  • For our projects, we typically have more or less 6 to 10 private dependencies to add. So by choosing the vcs way of doing, we’d need to add as much vcs entries by hand as we have private dependencies. You can see the example here. Those vcs entries need to have a full GitHub path, and the machine making the composer install call need to be authenticated with GitHub via its ssh key to be authorized to pull the packages. As you can see that’s quite some manual work to do, and this work is proportional to the (growing) list of our private packages. Surely there must be a better way.
  • This better way seems to be the other type called composer. I don’t know if you tried to read the paragraph in the official documentation but then I did I felt a bit confused. It did feel like what we were looking for though so I decided to try it out.

Trying out to make a private composer index

Like I said, when I first read the documentation, I was left a bit confused. So I thought I’d figure it out by doing. Okay so first we need to create a packages.json file. We’ll put in this file all the private repositories information, then we’ll reference this file in our composer.json file to make composer aware of our private repos.

The thing that then caught my eye was what they called the minimal package definition, and that looks like this.

{
    “name”: “smarty/smarty”,
    “version”: “3.1.7”,
    “dist”: {
        “url”: “http://www.smarty.net/files/Smarty-3.1.7.zip”,
        “type”: “zip”
    }
}

What I understand from this is something like : “Composer, if we’re trying to require smarty/smarty at version 3.1.7, you can find the zip here : http://www.smarty.net/files/Smarty-3.1.7.zip”. The dist part can also be an hosted git repository url, and that’s probably more the case with living proprietary private packages.

So to build our packages.json file, we should pile up a list of package definitions. For each private library we want to use, we should add one package definition per version number (and there could be quite a lot of them)

Some notable difference with vcs repos versus zips, is that you also need to provide in the dist part, a reference number, which is actually the commit id. And that’s very important because keeping that reference accurate and up to date plus the fact that we need to have one package definition per library version make up the fact we’ll need a tool such as private packagist or satis to automate this packages.json build, and therefore make this composer option truly viable.

Without automated solution, you’d have to go back to edit your packages.json file every time you add a new version to one of your repositories, and even worse, every time you add a new set of commits to any version to update this version’s reference number. That’s truly not acceptable to do all this by hand.

Automating the build

I looked for a solution to automate the packages.json build, and by looking at different resources around the internet, there are quickly two players that often come back: private packagist and satis. At the time of my investigations private packagist was just released, and was not free for an organisation like ours. I therefore decided to give satis a try because I also must say that I quite like to tackle those kind of industrilasation challenges, made of investigations and trials and errors.

Private packagist was founded by Jordi Boggiano and Nils Adermann, the creators of Composer. They put in some regular efforts to propose a product that helps packages maker to host their private repos just like you would do on the open source packagist.org, which is exactly the point of this article. Therefore you should pay them a visit to see if the quality of their services would actually be a more interesting alternative for you than building it on your own.

If you’re adventurous like me then read on :)!

Installing Satis

Like explained in the satis documentation, to install satis you need to run : composer create-project composer/satis:dev-master --keep-vcs, and follow up the command prompt.

The satis command

Satis can be invoked like so: php bin/satis build <configuration-file> <output-dir>

As you can see you have three parts in this command. Firstly you need to provide the path to your satis executable, then you need to specify which configuration file you’d like to use (typically the satis.json file we’ll talk about in the next paragraph), and finally a folder output where it will generate all the necessary files for your private composer to run, the packages.json file being one of them. But it will also generate a webpage presenting all of your available packages, their available version number and so on.

The output directory should then be made accessible online so we can use its url inside our project composer.json file, within the repositories section. In order to avoid https warnings, it’s better to setup https on the url you’re going to use, for example : https://yourprivatecomposer.com

The satis.json file

This file is there to help you configure satis, mainly to tell him where to find your private packages so it can work with them. The documentation explains the available options. In mine, I’ve listed all of my private packages GitHub urls, and I’ve specified that I want to build every branch satis will find in them through the option named “require-all”: true. You can find my satis.json file attached to this article.

We are now at a point where we have a satis command ruled by a proper statis.json file, that generates our private composer index. But we still have a manual step that is to launch the satis command every time we want to update our index. The ideal would be to automate the launch of the command every time we commit to any of our repositories.

To achieve this, we need to prepare a script that would then be called via GitHub webhooks (or equivalent), every time we push new commits or new version tags, and the job of this script would be to execute the satis build command to regenerate our private composer index.

The update script

I’ve attached an update script inside the zip file that accompanies this article.

Let’s see how you can configure it. Line 9 you have a $config array where you need to specify where your satis executable is located (that’s the very first part of the statis build command), where your satis.json file is (that’s the second part), and where you want the packagist index to be build, (that’s the third part).

That’s the only thing you should need to configure. Those arguments will be used to invoke the satis build command that the script will then execute for you.

Executing the update can be quite long, and this fact gives us two conclusions.

  • Firstly, there’s no point rebuilding the entire index with all of your private packages if you only sent a push to one of them. We should consider setting up a partial rebuild.
  • Secondly, as you don’t really know how long it will take, it’s nice to setup some sort of notification in place so you know when the script ends. That’s important because this end is the signal that you can now do a composer update from within your project to get the latest changes from your private packages.

Webhooks

Put the update script online where it can be accessed via a url, for example https://yourprivatecomposer.com/satis_webhook.php. We’ll use this url as a target for our webhooks.

In GitHub, to setup a webhook, you need to go to one of your repositories, then in settings / webhooks, add a new webhook, then specify that on the push event you’d like to make a request to your remote update script url at https://yourprivatecomposer.com/satis_webhook.php.

Considering partial rebuilds, depending on how the webhook is setup, we can therefore know in the update script which repository is concerned, and only rebuild the index parts that are relevant. When pushing commits to a repository, this could mean we added some commits to an existing version number, or that we’ve created a new version number.

Satis has an option named –repository-url which can be used to tell it to only concentrate on scanning this one and only repository. We’ve paired our webhooks with this option so that all the webhooks carry the repository name in a special parameter, for example : https://yourprivatecomposer.com/satis_webhook.php?pkg=yourcompagny/yourpackagename. In case of GitHub webhooks, you might not need to pass parameters in the url because it posts you a json payload with all this information as well. The aim is that when the update script receives the webhook it should know which repository url it should update. This improvement made our script 20 times quicker to execute (because we’ve had around 20 private repositories at the time) so as you can see it’s a huge performance improvement.

Notification

When the update script first ran, I was sure it was triggering properly thanks to webhooks, but when I had no idea if it had finished, or if it was still running. So I tried my luck by running composer update in my project and one times out of two, it either served me the ‘nothing to update or install’ sentence, or it started updating or installing the updated dependencies.

To avoid this waiting game, I setup a slack notification in a dedicated channel that warned everyone following this channel that one package index had been updated and was available to everyone. That was much much better, I definitely encourage you to do something similar within your own communicate medium.

Referencing your private packagist inside your projects

Now that you have a private composer index accessible online somewhere, open up your project composer.json file, and locate the repositories section. Then add up an entry of type composer with your index url like so:

“repositories”: [{
    “type”: “composer”,
    “url”: “https://wpackagist.org”
},
{
    “type”: “composer”,
    “url”: “https://yourprivatecomposer.com”
}]

Conclusion

Hosting your own private packagist can be quite challenging if you’ve never done it before. I hope this little guide would help make the process easier. For example, as I wasn’t a composer specialist, it took me a few bunch of days of investigation, trial and errors before getting to a setup that was production ready. That can be seen as quite an investment. I do hope that you’ll go quicker than me thanks to this article though 🙂 .

And once you’ve got the whole thing running, for your developers it will be so much easier to use the private repositories. Apart from the additional line to put inside the repositories section of your project, requiring private dependencies would be nearly as straightforward as public ones. And that’s the whole point of all this really.

ipsum Praesent vel, at risus. lectus velit, Donec justo elementum nec in