Continuous Integration Servers: Should you self-host or rely on SaaS?

Picture of Simon Mathieu
Simon Mathieu, Friday August 8, 2014

At Rainforest, we've spent a lot of time helping our amazing customers build the perfect deployment workflow. An important component in deploying applications effectively is a Continuous Integration (CI) server.

Traditionally, the CI server was a box in your office the you would manage yourself. Nowadays, it's pretty frequent for that software to be hosted on a VPS. But there's also a newer and fast growing way of doing CI. That way is pay a company to host and manage your CI server.

Lets compare the two approaches and try to figure out which one is best for you.

Who are the players?

Lets start with some name dropping.

In the DIY category, the leader is without a doubt Jenkins. Other popular tools are Bamboo[^1] and Teamcity.

In the SaaS category, the major players are CircleCI, Solano Labs, Codeship and Travis CI.

Self Hosted CI

Self hosted CI servers are the old guard, yet they still have many upsides that you can't really outsource.

One of their benefits is that they give you absolute control of your build process. Most of the popular CI servers have a large (intimidating?) list of possible plugins and extensions. You can even write some yourself if it does not behave the way you want. This can however become a very time consuming endeavor.

Another important benefit is that if you're using a configuration management tool such as Chef, Puppet or Ansible, you can use that tool to provision your CI server itself. One major advantage of doing this is that this ensures that your CI server is provisioned the same way as your production machines are. This means that the packages installed on your CI server will match the versions of packages on your production machines. This can catch sneaky bugs that only appear under some very specific configurations.

The biggest downside to doing it yourself is that it's hard. If your organization does not have the operational resources to dedicate to configuring a CI server correctly and keeping it running 24/7, you'll have a bad time. Remember that the CI server is closely tied to your development workflow; if it's not working correctly, then it's affecting the productivity of your engineering organization.

Hosted CI

Hosted CI is easy. The companies building these tools have spent a whole lot of effort improving the on-boarding flow of their products. If your product is built on a popular framework, probably all you need to do to get started is to give them access to your source code repository. This means you get a working CI server in minutes, not days.

Hosted CI is also fast... for the right price. Many of the hosted solutions can partition your tests and split their execution between several servers. This makes your build process much faster, although you do have to pay for the privileges. [^2]

Which brings us to our next point. Hosted CI can be very expensive. It usually becomes more expensive the more often your build runs or the more parallel servers you use. It should however be noted that many hosted services have free tiers or are free for open source projects.

Finally, hosted CI servers need access to your code. They will also require access to your servers if you want them to deploy your code for you (and you really want this). This has some pretty obvious security implications. You must be able to trust these providers with an extremely large part of your business.

Which should I choose?

My recommendation is that if you can, use a hosted CI provider. They cannot do everything yet, but they are so easy to setup and use that it seems silly to spend the resources to do that yourself.

They might however not be a good fit for your organization if you have stringent security requirements or an atypical testing setup.

Who's your money on?

Is our money where our mouth is? You betcha. We use CircleCI to run the tests and deploy all of our code at Rainforest. We're nearing 15,000 builds across all our projects. However, most of our open source projects use Travis CI as it's the de-facto open source CI server.

*Update: We did a webinar with Kevin Bell of CircleCI and discussed how we use CircleCI in a bit more depth. Check it out here.

The future

I think that hosted CI servers are going to be the norm very soon. There have been a few trends over the last few years that make me think so.

First, a lot of companies are not running their own infrastructure anymore. If you're entrusting all of your code to Heroku, then why not trust a CI provider too?

Then, even if you're running your own code, container technology such as Docker is becoming more popular at an incredible rate. If you can run your app in a container, you can re-use the same container to run your tests. This makes the problem of having a CI server that differs form your production environment almost completely disappear.

Finally, it seems clear that our industry has accepted that using services is better. You're not going to do a better job running a CI server than a company that does this as their core business.

No matter what solution you go with, Rainforest integrates with Continuous Integration servers like CircleCI and Jenkins. Find out more about using Rainforest with your favorite CI server here.

[^1]: Bamboo offers both hosted and self-hosted options. [^2]: You could achieve the same things with a self hosted solution, but not as easily.