Say what???: Category SysAdmin

Getting a grip on archiving mail threads

Kelly McCauley — Fri, 21 Dec 2007 13:32:00 -0600

I’m subscribed to both the Ruby Talk and Ruby on Rails mailing lists. Both are high volume. I typically don’t have enough time to read all that is going on, but I do like to have the emails around so I can search for a specific topic.

I like to keep my high volume mailing lists’ threads archived by month. This means that the topic thread head’s Date header determines where the entire thread is archived, even if the thread children’s Date header is a different month. For a low volume lists, this can be done by hand using any mail client. For high volume lists, doing it by hand is tedious and prone to mistakes. Computers are for this type of task. It is time to work hard at being lazy…

Here’s what I did to tackle this problem. My time was limited, I only had a couple of hours to create something to do the above for my two high volume lists. I had two Maildirs containing the Ruby Talk (~/.maildir/.ruby.talk) and Ruby on Rails (~/.maildir/.ruby.rails) mailing lists. Each contained more than 50,000 emails stored in individual files in the lists’ /cur directory.

So my ~/.maildir is organized like the following:

.ruby.rails/
.ruby.rails.200603/
.ruby.rails.200604/
.ruby.rails.200605/
.ruby.rails.200606/
.ruby.rails.200607/
.ruby.rails.200608/
.ruby.rails.200609/
.ruby.rails.200610/
.ruby.rails.200611/
.ruby.talk/
.ruby.talk.200603/
.ruby.talk.200604/
.ruby.talk.200605/
.ruby.talk.200606/
.ruby.talk.200607/
.ruby.talk.200608/
.ruby.talk.200609/
.ruby.talk.200610/
.ruby.talk.200611/

The requirements were:

Archive threads into the appropriate archive directories (should correctly archive 99.9% of the time).
Keep track of thread heads and their associated archive location so subsequent runs catch thread children dated after the previous run.
Shouldn’t consume excessive amounts of memory.

Since I intended to be the sole user of this program and the scope of functionality was so narrow, I decided to write a self contained script to flesh out the logic and behavior. This meant that testing by hand was OK for me (if this was developed for someone else, I would not choose this path). Future development iterations, I will break out the functionality into classes and modules along with real test specs.

The next decision I had was to decided how to process email headers. Since TMail is being maintained again, I decided to use it instead of parsing the email headers my self.

The following is the heavily commented script that I created. The most current source can be found at http://svn.drotner.org/repos/unix_tools/trunk/bin/mail_sort.rb

#!/usr/bin/env ruby
# Author: Kelly McCauley
# Copyright 2007 Kelly McCauley
# Source: http://svn.drotner.org/repos/unix_tools/trunk/bin/mail_sort.rb
# License: version 0.1 is Public Domain

require 'rubygems'
require 'optparse'        # Parses commandline options
require 'tmail'           # Handles the email parsing
require 'date'            # Ruby's date library
require 'fileutils'       # File and directory manipulation libarary

$VERBOSE = true

@version = '0.1'
@debug = 0
@quiet = false

@days_ago = 30              # Default "Sort and archive mail up to @days_ago".
@src_mail_dir = nil         # Maildir to sort and archive.
@thread_heads = {}          # Maps a thread head's Message-ID to
                            # its associated archive directory.
@thread_head_index = nil    # Location of a saved version of @thread_heads
                            # from a previous run.
@total_orphans = 0          # Count of thread children that have no parents.
@total_emails = 0           # Total emails read.
@total_emails_archived = 0  # Total emails that were moved to an archive
                            # directory.


#
# Methods
#

# Prints out the given msgs and opts to STDERR and then exits
def error_exit(opts, *msgs)
  msgs.each {|m| $stderr.print m}
  $stderr.puts opts
  exit(1)
end


# Loads a saved @thread_heads from a previous run into memory.
def load_thread_head_index(index_file)
  if File.file?(index_file)
    File.open(index_file) do |file|
      file.each_line do |line|
        key, year, sum, mon = line.chomp.split(/\t/)
        @thread_heads[key.to_sym] = [year.to_sym, sum.to_i, mon.to_sym]
      end
    end
  end
end

# Dumps @thread_heads that are less than 365 days ago to a file.
#
# I didn't serialize it to YAML because I didn't want the extra processing
# overhead or memory consumption.  I didn't Marshal it since I wanted the
# saved file to be tied to the particular version of Marshal.
def dump_thread_head_index(index_file)
  File.open(index_file, 'w') do |file|
    @thread_heads.each do |key,value|
      next if value[1] < @th_index_cutoff_sum
      file << "#{key.to_s}\t#{value.map{|x| x.to_s}.join("\t")}\n"
      @th_index_dump_count += 1
    end
  end
end


# Adds the given email to the @thread_heads lookup table.
def add_thread_head(email)
  unless @thread_heads.key?(email['message-id'].id.to_sym)
    $stderr.puts "th subject: '#{email['subject'].to_s}'" if @debug > 2
    @thread_heads[email['message-id'].id.to_sym] = [
      email['date'].date.year.to_s.to_sym,
      (
        email['date'].date.year.to_s +
        sprintf('%02d', email['date'].date.mon) +
        sprintf('%02d', email['date'].date.day)
      ).to_i,
      sprintf('%02d', email['date'].date.mon).to_sym,
    ]
  end
end

# Creates the archive maildir
def create_archive_maildir(root_archive_dir)
  sub_dirs = []
  sub_dirs << File.join(root_archive_dir, 'cur')
  sub_dirs << File.join(root_archive_dir, 'new')
  sub_dirs << File.join(root_archive_dir, 'tmp')
  options = {}
  options[:noop] = true if @debug > 2
  options[:verbose] = true if @debug > 1

  sub_dirs.each do |dir|
    unless File.directory?(dir)
      FileUtils.mkdir_p(dir, options)
    end
  end
  return sub_dirs
end


# Archives the given file to the give archive directory
def archive_email(root_archive_dir, filename)
  archive_dir = create_archive_maildir(root_archive_dir).shift
  options = {}
  options[:noop] = true if @debug > 2
  options[:verbose] = true if @debug > 1

  if @debug > 0
    FileUtils.cp(filename, archive_dir, options)
  else
    FileUtils.mv(filename, archive_dir)
  end

  @total_emails_archived += 1

end


# Archives the thread child email into the appropriate maildir
def archive_thread_child(thread_head, src_mail_dir, filename)
  $stderr.puts "tc #{filename}:  #{@thread_heads[thread_head][1]} <= #{@cutoff_sum}" if @debug > 2
  if (@thread_heads[thread_head][1] <= @cutoff_sum)
    $stderr.puts "tc filename: #{filename}" if @debug > 2
    root_archive_dir = "#{File.expand_path(src_mail_dir)}.#{@thread_heads[thread_head].first.to_s}#{@thread_heads[thread_head].last.to_s}"
    archive_email(root_archive_dir, filename)
  end
end


# Archives the thread head email into the appropriate maildir
def archive_thread_head(email, src_mail_dir, filename)
  # Determine this email's date sum.
  email_sum = (
    email['date'].date.year.to_s +
    sprintf('%02d', email['date'].date.mon) +
    sprintf('%02d', email['date'].date.day)
  ).to_i
  $stderr.puts "th #{filename}:  #{email_sum} <= #{@cutoff_sum}" if @debug > 2

  # Is the email before the cutoff date?
  if email_sum <= @cutoff_sum
    # Yes.
    $stderr.puts "th filename: #{filename}" if @debug > 2
    root_archive_dir = "#{File.expand_path(src_mail_dir)}.#{email['date'].date.year}#{sprintf('%02d', email['date'].date.mon)}"

    # Archive it.
    archive_email(root_archive_dir, filename)
  end
end


#
# Handle the commandline arguments
#

opts = OptionParser.new do |opts|
  opts.banner = "Usage: #{$0} [OPTIONS] MAILDIR"
  opts.separator("")
  opts.separator("OPTIONS")

  opts.on(
    '-D','--days-ago NUMBER',
    'Sort and archive mail up to --days-ago'
  ) do |days|
    @days_ago = days
  end

  opts.on(
    '-i','--thread-head-index FILE',
    'Specify the thread head index file'
  ) do |file|
    @thread_head_idx = file
  end

  opts.on_tail(
    '-q','--quiet',
    'Turns off all output including error output'
  ) do |q|
    @quiet = true
  end

  opts.on_tail(
    '-d','--debug',
    'Turns on debugging output'
  ) do |debug|
    @debug += 1
  end

  # help
  opts.on_tail(
    '-h', '--help', 'Shows this message'
  ) do ||
    error_exit(opts)
  end

  # version
  opts.on_tail(
    '-V', '--version',
    'Shows the version and copyright of db_diff'
  ) do ||
    error_exit(opts, "#{$0} version #{@version}\n")
  end

end

opts.parse!(ARGV)

# Make sure that the source Maildir is given and that the directory exists.
@src_mail_dir = ARGV.shift
error_exit(
  opts,
  "ERROR: failed to specify a MAILDIR\n"
) unless @src_mail_dir

error_exit(
  opts,
  "ERROR: MAILDIR does not exist: #{@src_mail_dir}\n"
) unless File.directory?(@src_mail_dir)


#
# Determine the cut-off dates.  Used in simple numerical comparison of dates.
#

# The cut-off date for determining if thread heads are targeted for archival.
@cutoff = Date.today - @days_ago
@cutoff_sum = (
  @cutoff.year.to_s +
  sprintf('%02d', @cutoff.mon) +
  sprintf('%02d', @cutoff.day)
).to_i

# The cut-off date for storing thread heads in @thread_heads.
thi = Date.today - 365
@th_index_cutoff_sum = (
  thi.year.to_s +
  sprintf('%02d', thi.mon) +
  sprintf('%02d', thi.day)
).to_i
@th_index_dump_count = 0

# Compose the location of the thread head index file
if @thread_head_index.nil?
  @thread_head_index = "#{File.expand_path(@src_mail_dir)}.mail_sort.idx"
end


# Pre-run debugging
if @debug > 0
  $stderr.puts "@debug: '#{@debug}'"
  $stderr.puts "@src_mail_dir: '#{@src_mail_dir}'"
  $stderr.puts "@thread_head_index: '#{@thread_head_index}'"
  $stderr.puts "@days_ago: '#{@days_ago}'"
  $stderr.puts "@cutoff: '#{@cutoff.to_s}'"
  $stderr.puts "@cutoff_sum: '#{@cutoff_sum.to_s}'"
  $stderr.puts "@th_index_cutoff_sum: '#{@th_index_cutoff_sum}'"
end


#
# Do the run.
#

# Load the thread head index if it exists.
load_thread_head_index(@thread_head_index)

# The location of the Maildir's cur directory.
src_mail_dir_cur = File.join(File.expand_path(@src_mail_dir),'cur')

# Iterate through each file in the Maildir's cur directory.
Dir.foreach(src_mail_dir_cur) do |filename|
  # Skip . and ..
  next if filename == '.'
  next if filename == '..'

  filename = File.join(src_mail_dir_cur, filename)

  # Skip any directories.
  next unless File.file?(filename)

  $stderr.puts "filename: #{filename}" if @debug > 2

  # Parse the file into an email.
  email = TMail::Mail.parse(IO.read(filename))

  if email['references'].nil? && email['in-reply-to'].nil?
    # This email is a thread head

    if email['message-id'].id.nil?
      # This email is a malformed email.
      $stderr.puts "No message-id for #{filename}" unless @quiet

    else
      # Add this email as a thread head.
      add_thread_head(email)

      # Archive this email.
      archive_thread_head(email, @src_mail_dir, filename)

    end

  else
    # This email is a thread child
    thread_head = nil

    # Determine the thread's head (Simple case first since it is the most
    # common)
    if !email['references'].nil? && !email['references'].ids.empty?
      # This email as a References header and it is not empty
      thread_head = email['references'].ids.first.to_sym

    elsif !email['in-reply-to'].nil? && !email['in-reply-to'].empty?
      # This email only has a In-Reply-To header which is not empty
      thread_head = email['in-reply-to'].to_s.to_sym

    end

    # Lookup the thread head in @thread_heads.
    if @thread_heads.key?(thread_head)
      # Found it, so archive this email in the thread head's archive directory.
      archive_thread_child(thread_head, @src_mail_dir, filename)

    else
      # Possibly an orphaned child.  See if any of its other references are
      # known thread heads.
      thread_head = nil

      if email['references'].nil? && !email['in-reply-to'].empty?
        # No References header so use the In-Reply-To header.
        ref = email['in-reply-to'].to_s.to_sym
        thread_head = ref if @thread_heads.key?(ref)

      elsif !email['references'].nil? && !email['references'].empty?
        # Use References header.  Iterate through each of the references and
        # use the first that matches as the thread's head.
        email['references'].ids.each do |ref|
          ref = ref.to_s.to_sym
          if @thread_heads.key?(ref)
            thread_head = ref
            break
          end
        end

      end

      # Do we now have the thread's head?
      if thread_head
        # Yes, so archive this email in the thread head's archive directory.
        archive_thread_child(thread_head, @src_mail_dir, filename)

      else
        # No.  We have an orphan.
        $stderr.puts "th orphan" if @debug > 2
        @total_orphans += 1

        # Archive it as a thread head.
        add_thread_head(email)
        archive_thread_head(email, @src_mail_dir, filename)

      end

    end

  end

  @total_emails += 1

end

# The run is done, so save @thread_heads.
dump_thread_head_index(@thread_head_index)

# Post-run debugging.
if @debug > 0
  $stderr.puts "@thread_heads.length: #{@thread_heads.length}"
  $stderr.puts "@total_orphans: #{@total_orphans}"
  $stderr.puts "@total_emails: #{@total_emails}"
  $stderr.puts "@total_emails_archived: #{@total_emails_archived}"
  $stderr.puts "@th_index_dump_count: #{@th_index_dump_count}"
end

Invoking it is as simple as ./mail_sort.rb -h.

Apache Setup for a Non-Root Rails App

Kelly McCauley — Mon, 17 Sep 2007 16:19:00 -0500

Introduction

Over the weekend, I wanted to put a small Ruby on Rails application that I had written for myself into production. This small article documents the setup and configuration that I used to get my application running with Apache and a Mongrel cluster back end. The article assumes that the reader knows the basics of Apache administration, Mongrel configuration, and deployment of Ruby on Rails applications.

There are plenty of documentation references for installing a rails application as the root web page. The Agile Web Development with Rails book, in the Setting Up a Deployment Enviroment section, gives a fairly detailed description of the steps needed to deploy an application into production. The Mongrel: Apache documentation is another excellent reference. But I didn’t want to install my rails application at the root of my URL….

My rails application is just a small URL manager called Blink and I wanted it to be served from the private section of my web pages:

https://drotner.org:8443/blink/

I am using Apache as the front end to Blink because I’m using mod_svn to host my Subversion repositories and didn’t want to have yet another front end web server to administer. Blink itself will be running on a cluster of mongrel servers (initially, only one mongrel server).

Setup and Configuration

I first checked out a copy of Blink’s source code, configured Blink, and created the database. The next step was to configure the mongrel cluster. My mongre_cluster init.d script looks for server configurations in /etc/mongrel, so I created /etc/mongrel/blink.drotner.org.8443.mongrel_cluster.yml

--- 
prefix: /blink
cwd: /var/www/ssl_drotner_org/apps/blink
log_file: log/mongrel.log
port: "8200"
environment: production
address: 127.0.0.1
servers: 1
user: mongrel
group: mongrel

I set mongrel’s prefix option (line 2) since I want Blink to run under the /blink/ URL path. I started up the mongrel cluster and then used curl to verify that Blink’s mongrel server was running.

root@drotner:/etc/mongrel # curl http://127.0.0.1:8200/blink/
<html><body>You are being <a href="http://127.0.0.1:8200/blink/authn_sessions/new">redirected</a>.</body></html>

curl outputted exactly what I was expecting. The next step was configuring Apache to proxy requests for /blink/ to the back end mongrel cluster.

<VirtualHost 66.241.137.71:8443>
  DocumentRoot "/var/www/ssl_drotner_org/htdocs-secure"
  ServerName drotner.org:8443
  ErrorLog /var/log/apache2/ssl_drotner_org_error_log
  CustomLog /var/log/apache2/ssl_drotner_org_access_log combinedio

  # ... SNIP ...

  ProxyRequests Off
  ProxyVia Full
  RequestHeader set X_FORWARDED_PROTO 'https'

  <Proxy balancer://mongrel_cluster>
    BalancerMember http://127.0.0.1:8200
    #BalancerMember http://127.0.0.1:8201
  </Proxy>

  <Proxy *>
    Order Deny,Allow
    Deny from all
    Allow from all
  </Proxy>

  RewriteEngine On
  # Check for  maintenance file and redirect all requests
  RewriteCond  %{DOCUMENT_ROOT}/system/maintenance.html -f
  RewriteCond  %{SCRIPT_FILENAME} !maintenance.html
  RewriteRule  ^/blink/.*$ /system/maintenance.html [L]
  # Rewrite /blink/ to check for static index page
  # RewriteRule ^/blink/$ /index.html [QSA]
  # Rewrite to check for Rails cached page
  RewriteRule ^/blink/([^.]+)$ /blink/$1.html [QSA]
  # Redirect all non-static requests to the mongrel cluster
  RewriteCond %{DOCUMENT_ROOT}/blink/%{REQUEST_FILENAME} !-f
  RewriteRule ^/blink/(.*)$ balancer://mongrel_cluster%{REQUEST_URI} [P,QSA,L]

</VirtualHost>

ProxyRequests Off (line 9) is critical and keeps the Apache server from being an open proxy server.

Note that Blink is being served over TLS/SSL. RequestHeader set X_FORWARDED_PROTO 'https' (line 11), in a nutshell, tells rails to write URLs with https:// for location redirects instead of the default http://.

<Proxy balancer://mongrel_cluster>... (lines 13 – 15) tells Apache where to find Blink’s mongrel servers. Each BalanceMember must map to a mongrel server running on the given port.

The rewrite rules, lines 24 – 35, do the grunt work of deciding whether the requests go to the Blink application or elsewhere.

Finally, I restarted my Apache server, and pointed my browser to https://drotner.org:8443/blink/ and got the appropriate index page. Now, back to what I enjoy doing … programming.