How We Use Webhooks to Reset Database States for QA

Picture of Yu Usami
Yu Usami, Thursday July 14, 2016

As you may have noticed in our recent post on how we ship code at Rainforest, we use the Rainforest platform extensively for QAing our product. The key to successful, effective QA testing is preparing seed data in the database for each test.

The problem is that some tests will cause changes in the database, thus we have to refresh database state to run QA tests again - otherwise the test result would be inconsistent. That means we must prepare a specific data or record (such as a special user) for testing a certain situation or feature in advance. This makes it easier to find bugs, and might also shorten the time needed for QA testing. In other words, for reliable QA, it is necessary to reset database and reproduce the same state whenever we run tests. So how do we do database resets for QA tests in Rainforest? We use webhooks.

What are Webhooks?

“Webhook? What is that?” you might ask. TL;DR: Webhooks are the mechanism that allows you to integrate housekeeping tasks like database reset before or after runs. Rainforest can notify the registered URL that your QA test run is about to start or has completed, and you can use these notifications to trigger your own processes before or after your QA runs.

Webhook in Practice

To use Rainforest webhooks, you'll first need to prepare a webhook API to receive calls from Rainforest. We run an app server exclusively for handling these webhook calls, as we believe in (at least) some benefits of micro-service architecture. For the sake of convenience, I'll call this app a "reset service" in this post. The app just has one API endpoint to receive all calls from Rainforest. To enable webhooks, register your webhook API endpoint in Environment settings (go to Menu -> Sites). Do not forget to check the "Webhook Enabled?" checkbox! Once you specify the environment you've set up, you'll get POST requests to the registered endpoint before a run starts or after a run has completed. When our reset service receives the POST requests, it will trigger the following tasks:

  1. Database reset
  2. QA environment scale-up (before run)
  3. QA environment scale-down (after run)

Let's take a close look at each task one by one.

1. Database Reset

A POST request sent from Rainforest to your webhook API contains a body like this:

{ "callbacktype": "beforerun", "options": { "run_id": "12345" }, "digest": "ADIGESTSTRING" }

After the endpoint of our reset service receives the POST request, it'll first verify the signature in it. (You should also check Rainforest Auth to see how to verify signatures.) And see the callback_type field if it's a valid request from Rainforest. Currently, Rainforest sends requests with before_run or after_run in the callback_type field, just before starting a run or just after completed a run accordingly. For before_run, the reset service triggers the database reset script on QA server. We store the gzipped dump of seed data for QA in our code base - we use PostgreSQL, so the following command is used to generate seed database dump:

pgdump --no-acl --no-owner -U postgres $databaseNAME | gzip -9 > seeded.dump.gz

The database reset script is pretty simple one; it'll drop all the record on QA server, then restore the seed dump. Here are the core commands in the database reset script:

psql $DATABASEURL -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;' > /dev/null 2>&1 zcat seeded.dump.gz | psql $DATABASEURL > /dev/null

2. QA environment scale-up (before run)

The reset service also does QA environment scale-up for before_run requests. The reason to scale-up before run is because the QA environment will get a lot of accesses by testers when runs make progress. Preparing for these incoming accesses beforehand enables us to avoid unexpected errors due to high load. It will save us some costs on servers as well. The reset service scales up to the saved good server formation via Heroku API after the database reset.

3. QA environment scale-down (after run)

When a run has completed, the reset service gets a after_run POST request. Then it'll scale-down the QA environment to the minimum formation of servers via Heroku API.

Want to enable webhook for your runs? Let us know!

As I've described in this post, we use webhooks to make our Rainforest QA runs more flexible and efficient. Of cause, there are a ton of other possible ways to utilize webhooks. Please refer to our support documents for more details about webhooks. And if you have any questions or problems on utilizing webhooks for your Rainforest workflow, let us know so that our customer success representatives or engineers can help you.