Article updated on 1/31/2009 for Rails 2.2.2 compatibility.
So you've had some success using sendmail to send one-off emails like order receipts, password reset confirmations and welcome letters. Now, you need to be able to send 5,000 newsletter emails to your entire contact database.
This tutorial assumes that you are already sending email in Rails. This is a tutorial on implementing queued mailing with ar_mailer.
A great way to handle a massive amount of emails that you need to send is simply by queueing the emails and sending them in smaller bursts. I like to use the plugin "ar_mailer" to get this working. How it works is simple: Whenever you attempt to send an email by calling a method like "deliver_welcome_letter," instead of using sendmail to send the email at that moment, ar_mailer will instead store the outgoing email into a database table. Then, in the background, the ar_sendmail executable will periodically query that database table and attempt to deliver the messages.
When you use a shared Ruby on Rails hosting plan, you will typically only be allowed to send out a certain number of emails per hour. The host that we use here at Ameravant actually limits us to 250 outgoing emails per hour (for each of our accounts). If a client of ours wants to send out a newsletter to 1,000 contacts, only the first 250 would be delivered if we used sendmail directly with no queue in place. Therefore, we need to limit the number of emails we send per hour so all 1,000 contacts will eventually receive their emails.
Let's get started
First off, you need to have the ar_mailer plugin installed for your Rails application.ar_mailer is available as a gem, but it is not yet compatible for use with Rails 2.2.2. So, for now you should be using the plugin or you will run into problems. The ar_mailer plugin is maintained on GitHub using git for version control. You can install it with:
$ script/plugin install git://github.com/adzap/ar_mailer.git
We need to setup a model in our application that will be used to store the outgoing emails. There are two commands that ar_sendmail (note that while the plugin is called ar_mailer, the actual executable is ar_sendmail) provides to give us the model and migration. They arear_sendmail --create-migration and ar_sendmail --create-model. Running these commands will print out the migration and model code for you to use. However, you can simply make the model yourself and use the code I give you here, to save some time.
The model and database migration
$ script/generate model email
class CreateEmails < ActiveRecord::Migration def self.up create_table :emails do |t| t.string :from, :to t.integer :last_send_attempt, :default => 0 t.text :mail t.datetime :created_on # ar_mailer still uses deprecated created_on field end end def self.down drop_table :emails end end
You will notice that ar_mailer still uses the old "created_on" field name as opposed to the newer "created_at" convention. This is the table that will hold all of the outgoing email information. It will store the to and from addresses, the entire MIME email body, and some meta information like when the email was added to the queue and the last timestamp of when the last delivery was attempted. When the ar_sendmail process runs, it uses the "created_on" field to determine if an email is expired. If an email is unable to be delivered for a long period of time, it will simply be removed from the queue (the default expiration time is one week).
We only have some simple validations here to ensure that no emails are added to the database that are missing information.
class Email < ActiveRecord::Base validates_presence_of :from, :to, :mail end
We also need change our application's ActionMailer delivery method from sendmail to activerecord. In your production.rb and development.rb environment files, enter the following:
ActionMailer::Base.delivery_method = :activerecord
You should already have a mailer class created for your application. If not, create one like so:
script/generate mailer post_office
Slight tangent — If you are going to have a lot of automated emails sent from your application, I suggest making multiple mailer classes, such as user_mailer.rb, event_mailer.rb, etc. It is also worth looking into Rails observers to trigger your email delivery.
Moving on... the mailer class needs to inherit from ActionMailer's now-available ARMailer subclass instead of Base.
class PostOffice < ActionMailer::ARMailer ... end
Let's test sending some messages. Use your application to trigger the email delivery, then go into your console and look in the Emails table for your queued emails.
$ script/console Loading development environment (Rails 2.1.0) >> Email.all
If you see your email records, it worked properly!
The magic in the background
We've got our emails queued and ready to be sent, but they will not actually be delivered until we start using the ar_mailer executable ar_sendmail. We have two options for making this work: We can use the process in daemon mode so it runs constantly in the background, or we can call it on a recurring schedule by setting up a cron job.
The problem I have with running the process as a daemon is that it can unexpectedly and randomly halt. This is not such a problem locally, as we'll likely be developing and testing our application and will know the process has stopped. We can then simply restart it. However, on a production server, we will likely not be monitoring the processes and therefore will usually only know it has stopped running when a client calls us, frustrated that their newsletter is not being sent.
I suggest running it as a daemon locally, but using a cron job on your production server.
Running ar_sendmail in daemon mode
This command is pretty simple. All we do is run the ar_sendmail command from our terminal. To get a list of all the arguments you can pass to it, simply run ar_sendmail -h. For development purposes right now, I'm going to just have ar_sendmail one a single time, so we can test that everything is configured properly.
First, we can check out our email queue with this:
We will be shown a list of emails in our database that are ready to be sent. If you see emails in there, great! That means that ar_sendmail is properly querying our database. Now let's run it:
The arguments instruct it to run one time only (-o), and to be verbose about what it is doing (-v). If all goes well, you should see some emails appearing in your inbox! If you want it to run on a recurring basis on your development machine, set it to run in daemon mode and give it parameters to tell it how many to send in each burst, and how often. This, for example, will send 20 emails every 5 minutes:
ar_sendmail -d --batch-size 20 --delay 300
Running as a cron job on your production server
When we run ar_sendmail on our production server, we need to be a bit more specific in our command, to ensure that the proper application path and environment is used. As explained before, I prefer to run a cron job that commands ar_sendmail to run a single time, on a recurring basis. For our host, we are limited to 250 emails an hour. So, I will setup the cron job to run at 5 minute intervals. This means we can send 20 emails every 5 minutes, giving us the safe number of sending 240 per hour.
Here's my entry in my production server's crontab:
*/5 * * * * /usr/local/bin/ruby /usr/local/bin/ar_sendmail -o --batch-size 20 --chdir /home/myusername/myapplication --environment production
If you've read my cron job post, you'll understand this command pretty easily.
That's it, we're up and rolling with queued emails. But for some extra credit, a few neat things that I like to do is provide a section for the website administrator to manage their email queue. I let them see:
- A list of all emails that are in the queue
- A time estimate of how long it will take the remaining emails to be sent (we know how many we send an hour, so this is cake!)
- The ability to delete a single email from the queue
- The ability to clear the entire queue in case they mistakenly sent something and then realized there was a huge typo in the subject line
This kind of flexibility is easy to add and your customers will appreciate it!