Optimal Test Environment Setup

Contents

In this post we’re going to look at optimal environments for web apps.

Most web applications are run in more than one place – usually at least two. The first one is the actual server that serves the application to the users. The second is the developer’s machine. It’s extremely important for developers to be able to test locally before deploying new code to all users.

In this post, we’ll examine the importance of having multiple environments and how they can help you isolate your application and prevent accidentally damaging your production systems.

Contents

What are environments?

An environment is a collection of all the configurations needed to run a particular instance of the application. This includes things like database credentials, all the API keys for the services you use, various internal settings of your application and more.

The environment is, in short, all the things you would need to change if you wanted to run a completely different and isolated version of the same application.

The exact content of the environment will vary with your application and technological stack, but the idea is applicable to all frameworks and web applications. I’ll shortly demonstrate how to create an environment from scratch in Ruby.

Why should I use environments?

The main reason to use different configurations per environment is isolation.

Lets consider an application where there’s no environment whatsoever. The configuration on the developer’s machine is the same as the one on the server (which is pretty rare). Because the configuration is the same, it means that the code running on the developer’s machine is connecting to the same database and same services as the production site. By definition, the role of the developer is to change the code of the application. What happens when the developer creates a bug locally and destroy records in the database? While he or she tests his or her code, that same code might be destroying important customer’s information, send emails to users or charging credit cards wrongfully.

As previously mentioned, it’s pretty rare for developers to connect straight to the production database. It’s however not that rare to connect to the same services (external APIs, mail servers).

The services are often not as critical as a database, but can still cause serious headaches if misused. An obvious example is that you probably don’t want to charge a real credit card from a development environment.

Now lets run over an optimal version of the most basic environments that everyone should have.

Production

Production is where your website runs and serves customers. It’s sometimes also referred to as ‘live’. An optimal Production environment should have the following qualities:

Stable – doesn’t break repeatedly
Performant – fast enough not to annoy your users
Backed-up regularly – you can recover in-case of a disaster / hack
Easy to manage – simple, doesn’t require constant maintenance
Repeatable – as in, repeatably creatable (avoid hand-made unicorns)

To achieve these, you’ll need to pick everything from your hosting provider to software stack with care – much like baking a cake – to end up with a great result.

Backed-up regularly is the key. Getting this wrong (and being unlucky) can be lethal for your company.

Whatever your production environment ends up being (software- and hardware-wise), it will become a defacto template for later environments. Therefore making Production choices that fit within your other constraints (e.g. are your developers using Mac / Windows machines?) will make adopting the optimal set of non-production environments easier.

Let’s look at those non-prod environments.

Development & Test

Assuming you’re no longer editing code in Production, the next two most common and essential environments are test and development:

Test – for running tests against
Development – for a working local version of your project

For best results, you should run the same software versions as in Production.

How far should you go?

It’s easy to get carried away and create environments for every possible use-case. However, setting them up takes time away from developing your actual product and so should be balanced against any potential upside. Since this is highly context-dependent, I’m going to outline the optimal set of environments for individuals, small teams and more advanced or complex projects separately.

A practical example

Here’s an extracted and annotated configuration from one of our apps that loads settings for the current environment. The application itself is written in Ruby and uses Bundler and Rack. Even though this example is in Ruby, the same can easily be achieved with any programming language and framework.

Get the name of the current environment from an environment variable.

ENV[‘RACK_ENV’] ||= ‘development’

Sets the root path of our application.

APPROOT = File.expandpath File.join(dir, “..”)

Set it to a constant for convenience

RACKENV = ENV[‘RACKENV’]

Require the environment specific dependencies

Bundler.require(:default, RACK_ENV)

Load a YAML file containing ALL of our API keys and environment-specific settings.

module MyApp def self.config @config ||= parseconfig(readconfig_hash) end

private def self.readconfighash YAML.load(File.read(“#{APPROOT}/config/#{RACKENV}.yml”)) end end

Use the loaded configuration file to configure our exception manager with the correct api key and to tag all exceptions with this environment.

Raven.configure do |config| config.dsn = MyApp.config.raven.dsn config.tags = { environment: RACK_ENV } config.logger = logger end “`

Best practices for QA environments

Every web app should have 3 environments at the very least. One for development, one for production and one for testing. I would also recommend a fourth one for a staging server.

Once you have these environments setup, you need to make sure that every API key in your environment is configured on a per-environment basis. For the test environment, it might be fine to not even specify a key for some services. You probably don’t want your unit tests to send actual emails or charge a credit card.

Finally, if you setup a staging environment, you should make it as close to production as possible. A staging environment is meant to test new features before deploying them to production. You want to avoid any discrepancies between the two.

Dealing with services

One question we often get at Rainforest is how to deal with services. Some services, such as Stripe, provide a sandbox mode. This means that you can use the same account with a different key, but none of the requests done in Sandbox mode will have any effect on your real customers. You should check whether or not the API you’re using supports such a mode.

If it doesn’t, then the next solution is to create a new account for each environment. This can be tedious and expensive, but it beats breaking important things in production.

What should I do to get started?

Take the following steps to start using environments effectively:

Identify all of your external dependencies (email servers, APIs, etc)
Extract the configuration of all these services to a configuration file
Create a copy of that file for your new environment with a different key for each service
Load the correct configuration file based on the current environment

There are more steps that are application and language specific, but they should become obvious one you have a working environment setup.

Finally, you should should setup a different environment for your QA tests and use Rainforest for easy QA.

Optimal environment setup for Individuals

Assuming you’ve got the basics of test, development and production, what should you do for the simplest of projects? For smaller projects, generally one with a single developer, it’s all too easy to go overboard with environments. I suggest you keep it simple, and just add a Staging environment to the mix.

Staging

Staging is simple: it should be as close a copy to Production as you can afford. Staging’s aim is to allow you to test larger and / or more dangerous changes in as similar a setting to Production as possible.

An optimal Staging setup:

Is running the code which you are planning to release
Has a recent (optionally sanitized1), version of Production data
Includes the same services and versions as Production (except when testing upgrades of those services i.e. does Apache 2.2 -> 2.4 really work?)
Has the same OS as Production
Is running on the same hardware as Production

If you can’t do all of the above, due to time or money, start from the top and work your way down.

The second item, having a recent copy of production data is surprisingly effective at catching missed errors fast. Without this, it’s common to find migrations which work locally but fail in Production, causing all kinds of issues. Having a database that’s very similar to production (a few days – a week old) makes it more likely that migrations will work as they have to run over much of the same data.

Note: this can become impractical at larger scale; moving the data, let alone sanitation can take too long to be practical. In this case, taking a subset of your production data can be a good halfway house.

For Small Teams

For smaller teams, adding an extra environment for QA is essential. It’s different to staging in that it should have a known and consistent state; having this allows members of your team to manually (or hopefully for you, automatically via Rainforest) test your software in a rigorous way.

A great QA environment setup should have the following properties in order of importance:

Be running the code which you are planning to release
Have a known consistent state that is set or reset during the deploy process
Run the same services and versions as production (unless you’re testing a service upgrade)
Have the same OS and hardware as Production

Number 2 is something that catches people out. It’s essential for any software that has state to avoid confusion and missed bugs. The easiest way to achieve this is to reset your database and seed it with known data after each deploy.

Getting this setup can be as easy as using basic tools like rake db:seed, or if you have a lot of data to seed – e.g. Rainforest has > 1000 accounts with data for QA testing, which takes ~25 minutes to regenerate – then a database backup restore is faster, though less flexible.

An example

We use the awesome CircleCI to run our tests and manage our deployments. This is a cut-down version of what we run live. It does the following:

Runs our tests (including Jasmine)
Development branch is deployed to two Heroku apps; herokuapp-stg and herokuapp-qa
Staging has it’s DB migrated, and is then restarted to make sure all the changes are noticed
For QA we additionally un-gzip a seeded dumb file, which provides ~1000 consistent accounts for our testing.
Finally, Rainforest is triggered (we test Rainforest’s QA env with Rainforest. Meta++)

This is our circle.yaml file:

test:
 post: - RAILS_ENV=test bundle exec rake spec:javascript

deployment:
 staging:
   branch: develop
   commands:
     - git fetch --unshallow
     - git push -f git@heroku.com:herokuapp-stg.git $CIRCLE<em>SHA1:master
     - heroku run rake db:migrate --app herokuapp-stg: timeout: 1800
     - heroku restart --app herokuapp-stg
     - git push -f git@heroku.com:herokuapp-qa.git $CIRCLE</em>SHA1:master
     - heroku pg:reset DATABASE --app herokuapp-qa --confirm herokuapp-qa
     - gzip -dc db/seeded.dump.gz | heroku pg:psql --app herokuapp-qa
     - heroku run rake db:migrate --app herokuapp-qa: timeout: 1800
     - heroku restart --app herokuapp-qa
     - "curl https://herokuapp-qa.herokuapp.com/ &gt; /dev/null"
     - bundle exec rainforest run --token $RF_TOKEN --tag run-me --conflict abort --fail-fast -fg

 production:
   branch: master
   commands:
       - git fetch --unshallow
       - git push -f git@heroku.com:herokuapp-prd.git $CIRCLE_SHA1:master
       - heroku run rake db:migrate --app herokuapp-prd: timeout: 1800
       - heroku restart --app herokuapp-prd

In addition to automating deployment and resetting databases, an oft-missed essential is documentation. Having a super-fancy deployment process is great, but it’s a hindrance if (when?) it fails when you’re on vacation and your whole team is blocked figuring things out. Document it.

For Complex Projects

More complicated projects, or teams with more developers have some interesting additional requirements to make them optimal.

Some important questions to be considered when designing an environment setup for complex projects and / or larger teams:

Who really needs prod access?
Key storage: where are keys stored? Does every developer need access?
Have you automated your backups? Do you test them regularly? Is this automated?

Optimal things, in order of importance:

Test each Pull Request like you would with Staging. With a larger team this provides an extra layer on top of code review and is faster3. This results in less things being missed before getting to Staging / QA, which makes everyone else more efficient.
Make separate accounts (not just keys) for each environment. Specifically for services such as AWS, Stripe, Mixpanel; things that matter, that affect your business if they go wrong. Splitting accounts is partly for isolation, mainly for security – this mitigates many types of accidents as well as malicious changes to Production environments.
Automate your backups AND automate the testing of them. The most basic way to test this is to restore it to staging and have a culture of using staging in your team.
Document all the things. The bare minimum is:

How do new team members get started? Cover:

Requirements
Installation
What’s expected of them to contribute (e.g. tested code -> pull request tagging @rainforestapp/dev)

On-call procedures:

Who’s on call
SEV levels and what they mean and are4
How to deal with indicents. Who to call? What SEV things are. Playbooks. If you’re interested in this topic more, watch this talk by Blake Gentry: Every Minute Counts: Coordinating Heroku’s Incident Response.

Regarding #1: If you’re lucky enough to use Heroku, Fourchette is a life changer; once you try it, you won’t go back. We’ve got an upcoming post on this subject, so subscribe at the bottom to get emailed when it comes out.