The Complete Guide: Rails, Sidekiq and Puma on Heroku/Linux Server

There are a ton of resources out there that talks about setting up Rails, Sidekiq and Puma on Heroku/Linux, but somehow during my own personal endeavour into the process, I felt that there were always little bits of information that was left out. So I decided to write my own version and put together all the little things which I've found answers to.


Introduction

We will first briefly talk about the technologies used here. After that, we will talk about setting up Redis, configuring Sidekiq and linking them it all with Puma. Finally, we will talk about getting it all to work on Heroku and Linux.


What?

Sidekiq
Sidekiq is a neat gem that allows us to easily implement efficient background processing for Ruby applications. It works by spawning multiple worker threads from a single master worker process.

In Rails, its mostly used as a means to offload long running tasks to the background. This can include heavy tasks such as uploading photos to S3, or doing additional post-processing to PDF files. It can also be scheduled to perform tasks at an arbitrary time in the future. Using Sidekiq in these manner allows for your Rails application to appear more responsive because the user's request will complete faster (while the heavy lifting is done in the background and users doesn't need to wait for it to finish).

On a high level, Sidekiq consists of 2 "parts" that interface with a Redis instance (Redis is a lightweight, persistent, key-value pair based database). The 2 parts are Sidekiq Client and Sidekiq Server. A Sidekiq client is anything that "pushes" jobs to a queue stored in the Redis instance. A Sidekiq server on the other hand is the process that "pulls" jobs from the queue and executes it. Sometimes, a Sidekiq server can also behave as a Sidekiq client. This happens when the Sidekiq server itself also pushes jobs into the queue (eg: a worker that queues another worker.).

Puma
Puma is a concurrent web server for Ruby.

Puma increases the responsiveness of our Rails application by either spawning multiple threads to handle requests, or forking multiple process instances of our app (cluster mode) or by using a combination of both.

Puma replaced Unicorn as Heroku's recommended web server for Rails because of its ability to gracefully handle slow requests better. You can read more about it here.


Let's Start!

Setting up Redis
We begin by first making sure that Redis is installed. This step is easily missed, when I was setting up Sidekiq for the first time, I was completely caught off-guard about needing to install redis.

Localhost - Mac
Install Redis by typing brew install redis into your console.
Tada! Walla! Redis is now installed! That was easy wasn't it?

Localhost/Remote Server - Linux
Installing redis on Linux can be a little bit lengthy (if you choose to do additional configurations and customisations).

There are two ways to install Redis on Linux:
Method 1: Redis official documentation discourages the use of native Linux package managers because it may be outdated. Instead, you can use the following commands taken from the official redis site.

wget http://download.redis.io/redis-stable.tar.gz  
tar xvzf redis-stable.tar.gz  
cd redis-stable  
sudo make install  

Method 2 (My recommendation): Alternatively, to circumvent the problem of outdated redis packages on the native Linux package managers, we can just manually point the package manager to reference an updated PPA. After that, it's installation is a breeze.

$ sudo add-apt-repository ppa:chris-lea/redis-server
$ sudo apt-get update
$ sudo apt-get install redis-server

Once you've installed redis, if you are using a Linux server for production, you will want to make sure that redis server starts automatically when the system boots up. To do this, you will need to make sure a proper init script is exists in the /etc/init.d path (If I am not mistaken, Method 2 does this automatically for you. I have not tried Method 1, so I am not too sure).
For an awesome comprehensive explanation on how to do that, you can visit here. I encourage you to test out starting, accessing and saving redis once the setup process is complete. I've personally ran into folder permission issues the first time I tried setting up redis on EC2.

Localhost - Windows
I'm sorry my Windows friends, but apparently Redis doesn't play nice with Windows right out of the box. See here for potential solutions.

Heroku
Setting up Redis on Heroku is extremely easy. After you created your app, you just need to provision a Redis database add-on. There are several providers to choose from. RedisToGo and RedisCloud are the popular ones with free tiers.

After provisioning a Redis add-on, the redis instance will be exposed to your Heroku application via the an environment variable. Usually it will default to ENV['REDIS_URL'], but sometimes it can be some other variable like ENV['REDISCLOUD_URL'].

Setting up Sidekiq
Now that redis is installed on our system, let's get Sidekiq into our Rails application. Add the following line into your Gemfile and then run bundle install.

    gem 'sidekiq'

Once that's done, Sidekiq is ready to go. However, most of the time, it is advisable to include a config file for Sidekiq. The config file allows us to configure things such as the number of workers to spawn, the types of queues to run and the location to store pid and log files. For more information about config files and queues, you can checkout the Sidekiq wiki.

In our case, we will be using the configuration below, defined in the file /config/sidekiq.yml (create one if it doesn't exists):

development:  
  :concurrency: 5
production:  
  :concurrency: 20
:queues:
  - default

A concurrency of 20 means that Sidekiq will spawn up to 20 workers to process jobs in the queue. These workers are only spawned when there are jobs to run.

Setting up Puma
Ok, time to install the big cat! First off, add Puma to your Gemfile and run bundle install.

gem 'puma'  

Puma integrates well with Rails. Just by running the rails s command now, a Puma server instead of webrick will start. For development, this behaviour is fine, but in production, every rails app and server resources are different, so it is advisable to create a config file for Puma as well.

Create a file /config/puma.rb and fill in the following:

workers Integer(ENV['WEB_CONCURRENCY'] || 2)  
threads_count = Integer(ENV['MAX_THREADS'] || 1)  
threads threads_count, threads_count

preload_app!

rackup      DefaultRackup  
port        ENV['PORT']     || 3000  
environment ENV['RACK_ENV'] || 'development'

# Because we are using preload_app, an instance of our app is created by master process (calling our initializers) and then memory space
# is forked. So we should close DB connection in the master process to avoid connection leaks.
# https://github.com/puma/puma/issues/303
# http://stackoverflow.com/questions/17903689/puma-cluster-configuration-on-heroku
# http://www.rubydoc.info/gems/puma/2.14.0/Puma%2FDSL%3Abefore_fork
# Dont have to worry about Sidekiq's connection to Redis because connections are only created when needed. As long as we are not
# queuing workers when rails is booting, there will be no redis connections to disconnect, so it should be fine.
before_fork do  
  puts "Puma master process about to fork. Closing existing Active record connections."
  ActiveRecord::Base.connection.disconnect!
end

on_worker_boot do  
  # Worker specific setup for Rails 4.1+
  # See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
  ActiveRecord::Base.establish_connection
end  

Let's take a closer look at what we've filled in:

  • workers basically specifies the number of worker process Puma will fork out from the master process. Each worker process runs a copy of your Rails app. Setting this number more than 1 will enable cluster mode.
  • threads basically specifies the number of threads each worker process can have. A threads 0,16 just means that minimum there can be 0 threads and maximum there can be 16 threads running. Threads are only spawned when your app needs them (So you can save server resource when there are no requests). In this code above, I set the minimum and maximum number of threads to be the same, as defined by the variable thread_count (This config is optimized for Heroku as charges are based on number and size of dyno instead of how much resources is used.).
  • Each Puma thread can handle 1 request at a time. Therefore, 16 threads can handle 16 request at a time. If you have 1 worker with up to 16 threads, that means you can only handle 16 requests at a time. If you have 2 workers, that means you have a total of 32 threads, thus your rails app can handle 32 requests at a time.
  • So why does Puma have workers and threads? Why not just have 1 worker and 32 threads? To answer this, there are several points to understand.
    • workers, in general, are more memory dependant. Each Puma worker is basically an entirely new OS-level process with its own memory address space containing a copy of your Rails app. 2 workers means 2 copies of your Rails application code in memory.
    • threads, in general, are more CPU reliant. Multiple threads run in a single worker process. The processing of the threads gets interleaved at the OS and CPU level. Additionally, all threads share the same memory address space.
    • Using threads require your application to be thread-safe. This is because threads run in the same process and share memory address. Simple things like manipulation of class variables by one thread can easily mess up the results in another thread.
    • Using workers on the other hand does not require your application to be thread-safe (if you only have 1 thread per worker.). Memory address is isolated between processes, so variable changes in one worker will not affect the other. However, using workers can quickly bloat up your memory footprint. If your rails app usually takes 100mb of memory, then running 2 workers will require 200mb of memory. However, most rails applications are small enough to not have to worry about this.
    • As a result, one must find the right balance between the number of workers and threads. If your code is not thread-safe, then using multiple workers with a 1 thread each is the way to go. However, if you can ensure the thread-safety of your application, then using threads alongside workers will allow you to maximise both the CPU and RAM available to you.
  • preload_app! basically specifies that your Rails app should be loaded by Puma master process first, and then when workers are spawned, they will just fork the memory address for your rails app from the master process. The purpose of this is to speed up the worker creation process. Without preload_app!, each worker will have to reload your app again.
  • The before_fork and on_worker_boot blocks are only required if you are using preload_app!. When the master process loads your Rails application, all the initialization will be done. This includes connection to the DB and also Redis. The code in before_fork basically closes any active connections to the DB to make sure no connections are leaked. After that, the workers will fork and boot up. That is when on_worker_boot is called and we re-establish ActiveRecord connection to the DB for each worker.

Configuring Sidekiq and Connecting to Redis
Under default circumstances, for localhost development and on Heroku, Sidekiq will be able to connect to redis without any configurations.

However, in the event that is not the case for you, such as because you changed the redis port, or ENV['REDIS_URL'] was not populated correctly, then you will have to configure Sidekiq to connect to redis in an initializer file.

First, create a file /config/initializers/sidekiq.rb (actually, you can name it anything you want, or even put it inside another file in the initializers folder.), then fill in the following:

if Rails.env.production?

  Sidekiq.configure_client do |config|
    config.redis = { url: ENV['REDIS_URL'], size: 2 }
  end

  Sidekiq.configure_server do |config|
    config.redis = { url: ENV['REDIS_URL'], size: 20 }

    Rails.application.config.after_initialize do
      Rails.logger.info("DB Connection Pool size for Sidekiq Server before disconnect is: #{ActiveRecord::Base.connection.pool.instance_variable_get([email protected]')}")
      ActiveRecord::Base.connection_pool.disconnect!

      ActiveSupport.on_load(:active_record) do
        config = Rails.application.config.database_configuration[Rails.env]
        config['reaping_frequency'] = ENV['DATABASE_REAP_FREQ'] || 10 # seconds
        # config['pool'] = ENV['WORKER_DB_POOL_SIZE'] || Sidekiq.options[:concurrency]
        config['pool'] = 16
        ActiveRecord::Base.establish_connection(config)

        Rails.logger.info("DB Connection Pool size for Sidekiq Server is now: #{ActiveRecord::Base.connection.pool.instance_variable_get([email protected]')}")
      end
    end
  end

end  

That looks confusing, but it's not. Let me explain.

The Sidekiq.configure_client block basically tells all Sidekiq Clients which redis instance to push their jobs to.

The Sidekiq.configure_server block on the other hand, yes, you guessed it, tells all Sidekiq Server which redis instance to pull their jobs from.

The redis instance location is specified using the url key.

Note: A Redis instance url typically has the following format (< and > not included):
redis://<user>:<secret>@<url path>:<port>/<db number>.

In your Linux production server, you can set the ENV['REDIS_URL'] to contain the url of your redis provider. If you are running redis on the same server with default configurations (locally, with respect to the Rails app), then you can use the default url redis://localhost:6379. On Heroku, if ENV['REDIS_URL'] is already set, then everything will work fine. If it is not set, then you will have to set it manually after getting the url of your redis instance from your redis provider. For example, if you are using REDISCLOUD, you can easily set ENV['REDIS_URL'] by typing in the console:

heroku config:set REDIS_URL = REDISCLOUD_URL  

Note: You might have to revise url: ENV['REDIS_URL'] to url: ENV[ENV['REDIS_URL']] for the REDISCLOUD case.

The size key on the other hand merely specifies the number of connection per Sidekiq client/server has to redis. In this case, for the Sidekiq client, every web dyno on Heroku, or every Puma worker running your app will have up to 2 connections to redis. On the other hand, for the Sidekiq server, each worker dyno can have up to 20 connections to redis. If you are deploying to Heroku, especially if you're using the Free tiers, you will want to take note of the values you set here.

How do I know what size value to use? You count. The first thing you have to understand is that it is a connection pool and each process has its own connection pool. A single webrick Rails app is 1 process. A rails app launched with Puma will have a connection pool for each Puma worker.
As a rule of thumb, the Sidekiq client, which is usually a rails app will only need 1 connection to redis. I used 2 in the config above to handle rare cases where one of the connection hangs and at least the client has a backup to use.
For Sidekiq server, setting a size value equivalent or less-but-close-to your Sidekiq concurrency will suffice.

Note: The general principle of counting and allocating Redis connections is also applicable to regular ActiveRecord DB connections. Read on and you will see the similarities.

Alright, so far that was easy, but what is all that extra code doing in the Sidekiq.configure_server block? What is with all the ActiveRecord..disconnect! and ActiveRecord..establish_connection? Well, to explain this section of code, I will first base my explanation on deployment to Heroku.

In Heroku, each dyno is supposed to be running only a single process (in theory, but you can actually work around it ;p ). Your rails app will be running on the web dyno while Sidekiq will be running in the worker dyno. The bunch of disconnect! and establish_connection code in the Sidekiq.configure_server block is to allow us to customise the number of ActiveRecord DB connections the worker dyno will have.

Note: The Sidekiq.configure_server block is only called by the dyno/instance/process that executes bundle exec sidekiq.

For example, let's assume that in config/database.yml, we declare pool: 5. Without the additional code in Sidekiq.configure_server block, both the web dyno and worker dyno will each have only up to 5 ActiveRecord connections to the DB. This is OK for the web dyno, but definitely not OK for the worker dyno. Why? Because the workerdyno runs your Sidekiq workers and if each Sidekiq worker uses ActiveRecord and accesses the DB, then each Sidekiq worker will need 1 ActiveRecord DB connection. If you set your concurrency for Sidekiq to be a large number, in our case, 20, then 5 connections shared among 20 workers is definitely not enough and you will start seeing errors such as the one below in your logs.

ActiveRecord::ConnectionTimeoutError: could not obtain a database connection within 5.000 seconds (waited 5.001 seconds)  

So, in order to avoid this problem, the code in the Sidekiq.configure_server block essentially disconnects ActiveRecord's connection to the DB for the worker dyno. Then it re-establishes a new connection with a more suitable connection pool, in this case creating a pool equal to the number of possible Sidekiq workers - ENV['WORKER_DB_POOL_SIZE'] || Sidekiq.options[:concurrency].

The same principle applies to deployment to Linux servers such as EC2.

Note: The code in Sidekiq.configure_server matters more for Heroku because of the limited number of DB connections provided in the free database tiers (Postgres Free Tier: 30 connections only). If database connection limit is not an issue, then all that code can be thrown away and we can simply set pool: <Some large number equivalent to Sidekiq concurrency> in /config/database.yml.

Starting Redis
Ok, with all the installation and configuration done, its time to start the "engine"! Brooooom Brooooom! Redis is not part of Rails, therefore we will need to manually start a redis server instance so that our rails app and Sidekiq can connect to it.

Localhost - Mac/Linux/Windows
You can start redis by typing redis-server into your console. To switch off, just do a ctrl-c.
This will "use up" the console, so you'll need to launch another console for any other console related work.

Linux Notes: If you've gone through the process of getting redis server to start automatically, then you can skip this step. Just type ps aux | grep redis to check if the redis instance is already running or not.

PS: After exiting from redis-server, a file called dump.rdb (which contains all your data stored in redis) will be created. Best not to run redis-server command while in your project folder to avoid random files being added to git.

Heroku
Yay! Nothing to do here. The redis instance is managed by the add-on provider you chosed, so it is always on and always ready for you to access.


Finally! Launch Your App!

With Redis started and your configurations all set, you can now launch your rails app!

Localhost - Mac/Linux/Windows
First start Sidekiq in one console:

bundle exec sidekiq  

And then in another console, start the rails server:

bundle exec puma -C config/puma.rb  

Note: Just append -e production to both commands to run in production mode.

Heroku
Enter the following into your Procfile and push to Heroku.

web: bundle exec puma -C config/puma.rb  
worker: bundle exec sidekiq -e production -C config/sidekiq.yml  

After that, provision a worker dyno by typing into the console:

heroku ps:scale worker=1  

That's it! Sidekiq should be all set up in manner that plays nice with Puma on Heroku or your Linux server! :D

I hope this guide was able to help in your setup of Rails, Sidekiq and Puma on Heroku/Linux. If you spot any misinterpretations or room for improvements then let me know!

Thanks! ;D

Cheers,
Julian Ee.


Extra: Database and Redis Connections

I just felt like expanding more on this topic, so lets keep it simple.

  • You need a connection to access your Database/Redis instance to run queries and whatnot (herein DB).
  • Most of the time, connections are provided to you in a connection pool which consists of several connections.
  • Connection pools are shared within a process. Different process will have their own connection pool.
  • A standard Rails app (webrick) will typically only need 1 connection to the DB.
  • A Puma worker will have its own connection pool (defined in /config/database.yml). It is not shared with other Puma workers. However, the connection pool in a Puma worker is shared among the Puma threads in it.
  • Each Puma thread will need a connection to the DB. A Puma worker with 1 thread will need 1 connection to the DB.
  • If you have more Puma threads than connections in the connection pool, it is fine if the difference is not too big. A connection pool of 2-3 is usually sufficient for 5 Puma threads (They share. When one uses finish, the connection is returned to the pool for another thread to use. In the end, it depends on how DB intensive your Rails app is.).
  • If you run Puma with 5 workers and 6 threads each. Then a pool: 6 should be good enough.
  • To calculate the max number of connections your application uses, (5 worker * 6 threads each) = 30 total threads. Therefore 30 total DB connection required for your app.
  • If you run Sidekiq with a concurrency of 20. Then the number of DB connections required by Sidekiq is (number of worker dyno * concurrency) = (1 * 20) = 20 DB connections.
  • Combining the 2, you will need at least 50 DB connections for your Rails app to run smoothly. Please take note of this, especially if you are using Heroku Postgresql free tier as it has only 30 DB connections available.
  • When possible, try to allocate 1-2 extra connections in the pool so that in the event the connection encounters an error, there are backup connections for your app to use.
  • It is also a good idea not to allocate all the available DB connections. You should keep 1 or 2 in reserve so that you still can connect to it for debugging or error fixing such as from the console or some PG/MySQL Admin interface.