A sustainable Rails upgrade workflow that slowly but surely brought us to the point when all our apps run on latest Rails.

Rainforest main application is a Rails monolith with 160k lines of code and tests. Bumping a major version of rails in a project this big has overwhelming results, in fact no rails command would even start when we first tried to just bundle update from 4 to 5. Here’s what we did it to ease the upgrade pain.

Rails at Rainforest QA

Rainforest started in 2013 as a Rails 3 application. We’ve been following patches and upgraded to Rails 4 in 2014, but things started going downhill when Rails 5 was released in 2016 while Rainforest remained behind. The scope of changes required to upgrade kept growing and we didn’t make any attempts until almost 3 years later. Our Rails 5 upgrade went live in July 2019 and was followed by consecutive migrations to Rails 5.1, 5.2 and then 6.0 in July 2020.

A rough timeline of Rainforest Rails versions vs. Rails official releases
A rough timeline of Rainforest Rails versions vs. Rails official releases

Even though it was a conscious decision to focus on other projects in 2017-2019, allowing the app to stay so far behind accrued a lot of technical debt. In fact, the priorities haven’t changed since but we found a way to upgrade that doesn’t interrupt product work and helps us detect any breaking changes early. The fact that we test Rainforest extensively with both RSpec and a suite of Rainforest Tests played a key role too.

The Upgrade Workflow

Step 1: Avoid a long-lived upgrade branch with dual-boot

One of our first learnings was that a long-lived upgrade feature branch is not an option, being too time consuming and risky. Instead, we came up with a slower but non-disruptive upgrade process using the BUNDLE_GEMFILE option and dual-gemfile approach.

Here’s the setup:

# main Gemfile
gem "rails", "~> 5.2"


# Gemfile_NEXT
gem "rails", "~> 6.0"

With all other gems copied to Gemfile_NEXT, Bundler could install all NEXT dependencies and save them in a new lockfile without affecting any of the original production bundle:

BUNDLE_GEMFILE=Gemfile_NEXT bundle

Great, now we can run specs against Rails 6 with:

BUNDLE_GEMFILE=Gemfile_NEXT bundle exec rspec

That’s the idea. In practice, our application wouldn’t even complete bundle install, because of so many gem version incompatibilities. It was the time to comment out the offending gems from Gemfile_NEXT. Yes, the app wasn’t functional at this stage and many (actually, thousands of) specs were failing, but it can be dealt with later. We completed the first milestone: that any developer could install NEXT dependencies and start a functional rails console to test things out.

This setup was soon merged without affecting any production code and lead to the benefit of no long-lived upgrade branch becoming a nightmare to merge. Then, we could work on small incremental PRs to address issues one-by-one and prepare Gemfile_NEXT for release.

Step 2: Wire up CI/CD

Our dual-boot means we can test each new feature branch against both the production Gemfile and Gemfile_NEXT, to make sure we’re getting closer to our upgrade goal as the team keeps working on new features day to day. Fortunately, it was easy to duplicate our existing test CircleCI config and just add a BUNDLE_GEMFILE environment variable to run tests against the new version.

The optional test_rails_next build check verifies that future code changes are compatible with both Gemfile and Gemfile_NEXT:

build check

The caveat here is that we started off far from a green test_rails_next build with all the failures and Rails deprecations still not fixed.

Step 3: Manage and exclude Gemfile_NEXT failures

Once we had a failing test suite we wanted to get it passing as soon possible so that we could spot new bugs introduced by our ongoing work on the product, and skip breaking changes and deprecation errors until there was time to address each one. To do this we added a Rainforest.rails_next? method:

# config/application.rb
module Rainforest
  def self.rails_next?
    Gem::Version.new(Rails.version) > Gem::Version.new('5.0.2.1')
  end
end

‍This method was used by a fixme_rails_next RSpec tag that we added to specs failing on Gemfile_NEXT. We could see how many specs there are left to address by running rspec –dry-run –format=documentation –tag fixme_rails_next and exclude the failing ones from default spec runs with:

# spec/spec_helper.rb
RSpec.configure do |config|
  config.filter_run_excluding(fixme_rails_next: true) if Rainforest.rails_next?
end

‍On top of that, one more build step was added to our CI that ensures any gem updates we do in Gemfile are reflected in Gemfile_NEXT. The CI script you can see below will fail with a message like ERROR: GEMFILE_NEXT needs to update grape from 1.1.0 to at least 1.3.0 in case someone forgot to reflect an update to Gemfile in Gemfile_NEXT.

This method turned out useful as we had to update other dependencies while the Rails upgrade process was in flight and here’s what that script looks like:

# frozen_string_literal: true
# rubocop:disable Lint/NoENV

# This script can take two Gemfile.locks and compare them.
# The command will print a diff between added/removed gems
# and exit 1 if any of the GEMFILE_NEW gems is stale w.r.t. to GEMFILE_OLD.
# Use EXCLUDED_GEMS (comma separated list) to ignore certain gems from validation.

require 'bundler'

def fetch_gems_and_versions(lockfile_name)
  lockfile = Bundler::LockfileParser.new(Bundler.read_file(lockfile_name))
  lockfile.specs.map do |spec|
    [spec.name, spec.version]
  end.to_h
end

def usage
  puts "Validates differences between two Gemfile locks\n" \
       'Usage: ruby script/validate_gemfile_next.rb GEMFILE_OLD.lock GEMFILE_NEW.lock'
end

unless ARGV.length == 2
  usage
  exit 1
end

old_bundle = fetch_gems_and_versions(ARGV[0])
new_bundle = fetch_gems_and_versions(ARGV[1])

removed_gems = old_bundle.keys - new_bundle.keys
puts "WARN: Removed in GEMFILE_NEW: #{removed_gems.join(', ')}" if removed_gems.any?
added_gems = new_bundle.keys - old_bundle.keys
puts "WARN: Added in GEMFILE_NEW: #{added_gems.join(', ')}" if added_gems.any?

EXCLUDED_GEMS = ENV['EXCLUDED_GEMS'].to_s.downcase.split(',')

status = 0
old_bundle.each do |gem_name, old_ver|
  new_ver = new_bundle[gem_name]
  if new_ver && old_ver > new_ver
    if EXCLUDED_GEMS.include?(gem_name)
      puts "WARN: excluding #{gem_name}: #{old_ver} vs #{new_ver}"
      next
    end

    puts "ERROR: GEMFILE_NEW needs to update #{gem_name} from #{new_ver} " \
         "to at least #{old_ver}"
    status = 1
  end
end

if status == 1
  puts "\nUpdate the Gemfile_NEXT.lock file by running `BUNDLE_GEMFILE=Gemfile_NEXT bundle ...`"
end

exit status

Step 4: Config, dependencies, and deprecations

The above three steps have established a process to work on multiple short-lived changes to tick all the remaining items off an upgrade workflow checklist.

Talking about the 5 to 6 migration specifically, it started by working through all config changes generated by rake app:update. Those included the migration from sprockets to zeitwerk for constant autoloading which only required defining a few custom constant inflectors and fixing a few namespace locations that the new autoloader found confusing.

Next came dependency updates. We updated our in-house gems like queue_classic to support Rails 6 and bumped other gem dependencies (state_machines and grape were probably the most risky and time consuming). We also decided to drop a dependency on schema_validations which isn’t compatible with Rails 6 and made little sense to keep. This gem dynamically infers ActiveModel validations from the database schema constraints, so in order to remove it safely we wrote a code generator that stored all of these implicit validators as declarations within each class in the app/models/ directory.

The dependency updates let us remove a big portion of rails_next? exclusions both in the codebase and specs, but those final ones required working through failures and addressing API changes and deprecations one-by-one to get to a green Gemfile_NEXT build with zero fixme_rails_next exclusions.

Step 5: Run regression tests with RainforestQA

Once the build was green, we did one more thing before Gemfile_NEXT finally replaced the “old” Gemfile: run the entire Rainforest test suite, which is comprised of 250+ Rainforest test scenarios ran both by crowdsourced human testers as well as our visual automation technology (learn more on our website). In case of Rails upgrades, we ran the entire suite instead of a subset of about 100 tests normally included in every backend release, which ensured highest confidence of no functional regressions being present in the final upgrade PR.

Final Thoughts

Stale dependencies are technical debt

Small changes with limited scope are easier to ship and have smaller potential blast radius if things go wrong. Once the scope becomes too big, the upgrade project suddenly turns into a nightmare: a project impossible to deliver without jeopardising other product initiatives. We’ve been there: this is what blocked Rainforest from upgrading from Rails 4 to 5 for a long time. Fortunately, the upgrade process you’ve just read about helped us pay back the technical debt and future-proof our codebase.

Rainforest is ready for future upgrades

Even though we’re currently on the latest version of Rails, test_rails_next has still a place in our build pipeline and points to the 6.1-stable branch. This way we can test every change against the cutting edge, get notified about any breaking changes quickly, and also ensure the upgrade process we described here keeps working as expected.

To recap, these are the main concepts to help you ensure sustainable and stress-free upgrades:

  1. Dual-boot with BUNDLE_GEMFILE=Gemfile_NEXT
  2. Keep a ToDo list of breaking changes, deprecations and fixme_rails_next exclusions that can be addressed in small chunks
  3. Update Gemfile dependencies often, e.g. with automated help from Dependabot
  4. Cover your application with an adequate number of unit tests. We use rspec and undercover (try it on GitHub) for ensuring test coverage for all new changes
  5. Finally, use Rainforest for functional regression testing on every release

If you’re looking for more good content about Rails upgrades, check out these links:

Thanks for reading, I hope this article helped with your Rails upgrade journey!

‍