Say what???: Tag ruby

Getting a grip on archiving mail threads

Kelly McCauley — Fri, 21 Dec 2007 13:32:00 -0600

I’m subscribed to both the Ruby Talk and Ruby on Rails mailing lists. Both are high volume. I typically don’t have enough time to read all that is going on, but I do like to have the emails around so I can search for a specific topic.

I like to keep my high volume mailing lists’ threads archived by month. This means that the topic thread head’s Date header determines where the entire thread is archived, even if the thread children’s Date header is a different month. For a low volume lists, this can be done by hand using any mail client. For high volume lists, doing it by hand is tedious and prone to mistakes. Computers are for this type of task. It is time to work hard at being lazy…

Here’s what I did to tackle this problem. My time was limited, I only had a couple of hours to create something to do the above for my two high volume lists. I had two Maildirs containing the Ruby Talk (~/.maildir/.ruby.talk) and Ruby on Rails (~/.maildir/.ruby.rails) mailing lists. Each contained more than 50,000 emails stored in individual files in the lists’ /cur directory.

So my ~/.maildir is organized like the following:

.ruby.rails/
.ruby.rails.200603/
.ruby.rails.200604/
.ruby.rails.200605/
.ruby.rails.200606/
.ruby.rails.200607/
.ruby.rails.200608/
.ruby.rails.200609/
.ruby.rails.200610/
.ruby.rails.200611/
.ruby.talk/
.ruby.talk.200603/
.ruby.talk.200604/
.ruby.talk.200605/
.ruby.talk.200606/
.ruby.talk.200607/
.ruby.talk.200608/
.ruby.talk.200609/
.ruby.talk.200610/
.ruby.talk.200611/

The requirements were:

Archive threads into the appropriate archive directories (should correctly archive 99.9% of the time).
Keep track of thread heads and their associated archive location so subsequent runs catch thread children dated after the previous run.
Shouldn’t consume excessive amounts of memory.

Since I intended to be the sole user of this program and the scope of functionality was so narrow, I decided to write a self contained script to flesh out the logic and behavior. This meant that testing by hand was OK for me (if this was developed for someone else, I would not choose this path). Future development iterations, I will break out the functionality into classes and modules along with real test specs.

The next decision I had was to decided how to process email headers. Since TMail is being maintained again, I decided to use it instead of parsing the email headers my self.

The following is the heavily commented script that I created. The most current source can be found at http://svn.drotner.org/repos/unix_tools/trunk/bin/mail_sort.rb

#!/usr/bin/env ruby
# Author: Kelly McCauley
# Copyright 2007 Kelly McCauley
# Source: http://svn.drotner.org/repos/unix_tools/trunk/bin/mail_sort.rb
# License: version 0.1 is Public Domain

require 'rubygems'
require 'optparse'        # Parses commandline options
require 'tmail'           # Handles the email parsing
require 'date'            # Ruby's date library
require 'fileutils'       # File and directory manipulation libarary

$VERBOSE = true

@version = '0.1'
@debug = 0
@quiet = false

@days_ago = 30              # Default "Sort and archive mail up to @days_ago".
@src_mail_dir = nil         # Maildir to sort and archive.
@thread_heads = {}          # Maps a thread head's Message-ID to
                            # its associated archive directory.
@thread_head_index = nil    # Location of a saved version of @thread_heads
                            # from a previous run.
@total_orphans = 0          # Count of thread children that have no parents.
@total_emails = 0           # Total emails read.
@total_emails_archived = 0  # Total emails that were moved to an archive
                            # directory.


#
# Methods
#

# Prints out the given msgs and opts to STDERR and then exits
def error_exit(opts, *msgs)
  msgs.each {|m| $stderr.print m}
  $stderr.puts opts
  exit(1)
end


# Loads a saved @thread_heads from a previous run into memory.
def load_thread_head_index(index_file)
  if File.file?(index_file)
    File.open(index_file) do |file|
      file.each_line do |line|
        key, year, sum, mon = line.chomp.split(/\t/)
        @thread_heads[key.to_sym] = [year.to_sym, sum.to_i, mon.to_sym]
      end
    end
  end
end

# Dumps @thread_heads that are less than 365 days ago to a file.
#
# I didn't serialize it to YAML because I didn't want the extra processing
# overhead or memory consumption.  I didn't Marshal it since I wanted the
# saved file to be tied to the particular version of Marshal.
def dump_thread_head_index(index_file)
  File.open(index_file, 'w') do |file|
    @thread_heads.each do |key,value|
      next if value[1] < @th_index_cutoff_sum
      file << "#{key.to_s}\t#{value.map{|x| x.to_s}.join("\t")}\n"
      @th_index_dump_count += 1
    end
  end
end


# Adds the given email to the @thread_heads lookup table.
def add_thread_head(email)
  unless @thread_heads.key?(email['message-id'].id.to_sym)
    $stderr.puts "th subject: '#{email['subject'].to_s}'" if @debug > 2
    @thread_heads[email['message-id'].id.to_sym] = [
      email['date'].date.year.to_s.to_sym,
      (
        email['date'].date.year.to_s +
        sprintf('%02d', email['date'].date.mon) +
        sprintf('%02d', email['date'].date.day)
      ).to_i,
      sprintf('%02d', email['date'].date.mon).to_sym,
    ]
  end
end

# Creates the archive maildir
def create_archive_maildir(root_archive_dir)
  sub_dirs = []
  sub_dirs << File.join(root_archive_dir, 'cur')
  sub_dirs << File.join(root_archive_dir, 'new')
  sub_dirs << File.join(root_archive_dir, 'tmp')
  options = {}
  options[:noop] = true if @debug > 2
  options[:verbose] = true if @debug > 1

  sub_dirs.each do |dir|
    unless File.directory?(dir)
      FileUtils.mkdir_p(dir, options)
    end
  end
  return sub_dirs
end


# Archives the given file to the give archive directory
def archive_email(root_archive_dir, filename)
  archive_dir = create_archive_maildir(root_archive_dir).shift
  options = {}
  options[:noop] = true if @debug > 2
  options[:verbose] = true if @debug > 1

  if @debug > 0
    FileUtils.cp(filename, archive_dir, options)
  else
    FileUtils.mv(filename, archive_dir)
  end

  @total_emails_archived += 1

end


# Archives the thread child email into the appropriate maildir
def archive_thread_child(thread_head, src_mail_dir, filename)
  $stderr.puts "tc #{filename}:  #{@thread_heads[thread_head][1]} <= #{@cutoff_sum}" if @debug > 2
  if (@thread_heads[thread_head][1] <= @cutoff_sum)
    $stderr.puts "tc filename: #{filename}" if @debug > 2
    root_archive_dir = "#{File.expand_path(src_mail_dir)}.#{@thread_heads[thread_head].first.to_s}#{@thread_heads[thread_head].last.to_s}"
    archive_email(root_archive_dir, filename)
  end
end


# Archives the thread head email into the appropriate maildir
def archive_thread_head(email, src_mail_dir, filename)
  # Determine this email's date sum.
  email_sum = (
    email['date'].date.year.to_s +
    sprintf('%02d', email['date'].date.mon) +
    sprintf('%02d', email['date'].date.day)
  ).to_i
  $stderr.puts "th #{filename}:  #{email_sum} <= #{@cutoff_sum}" if @debug > 2

  # Is the email before the cutoff date?
  if email_sum <= @cutoff_sum
    # Yes.
    $stderr.puts "th filename: #{filename}" if @debug > 2
    root_archive_dir = "#{File.expand_path(src_mail_dir)}.#{email['date'].date.year}#{sprintf('%02d', email['date'].date.mon)}"

    # Archive it.
    archive_email(root_archive_dir, filename)
  end
end


#
# Handle the commandline arguments
#

opts = OptionParser.new do |opts|
  opts.banner = "Usage: #{$0} [OPTIONS] MAILDIR"
  opts.separator("")
  opts.separator("OPTIONS")

  opts.on(
    '-D','--days-ago NUMBER',
    'Sort and archive mail up to --days-ago'
  ) do |days|
    @days_ago = days
  end

  opts.on(
    '-i','--thread-head-index FILE',
    'Specify the thread head index file'
  ) do |file|
    @thread_head_idx = file
  end

  opts.on_tail(
    '-q','--quiet',
    'Turns off all output including error output'
  ) do |q|
    @quiet = true
  end

  opts.on_tail(
    '-d','--debug',
    'Turns on debugging output'
  ) do |debug|
    @debug += 1
  end

  # help
  opts.on_tail(
    '-h', '--help', 'Shows this message'
  ) do ||
    error_exit(opts)
  end

  # version
  opts.on_tail(
    '-V', '--version',
    'Shows the version and copyright of db_diff'
  ) do ||
    error_exit(opts, "#{$0} version #{@version}\n")
  end

end

opts.parse!(ARGV)

# Make sure that the source Maildir is given and that the directory exists.
@src_mail_dir = ARGV.shift
error_exit(
  opts,
  "ERROR: failed to specify a MAILDIR\n"
) unless @src_mail_dir

error_exit(
  opts,
  "ERROR: MAILDIR does not exist: #{@src_mail_dir}\n"
) unless File.directory?(@src_mail_dir)


#
# Determine the cut-off dates.  Used in simple numerical comparison of dates.
#

# The cut-off date for determining if thread heads are targeted for archival.
@cutoff = Date.today - @days_ago
@cutoff_sum = (
  @cutoff.year.to_s +
  sprintf('%02d', @cutoff.mon) +
  sprintf('%02d', @cutoff.day)
).to_i

# The cut-off date for storing thread heads in @thread_heads.
thi = Date.today - 365
@th_index_cutoff_sum = (
  thi.year.to_s +
  sprintf('%02d', thi.mon) +
  sprintf('%02d', thi.day)
).to_i
@th_index_dump_count = 0

# Compose the location of the thread head index file
if @thread_head_index.nil?
  @thread_head_index = "#{File.expand_path(@src_mail_dir)}.mail_sort.idx"
end


# Pre-run debugging
if @debug > 0
  $stderr.puts "@debug: '#{@debug}'"
  $stderr.puts "@src_mail_dir: '#{@src_mail_dir}'"
  $stderr.puts "@thread_head_index: '#{@thread_head_index}'"
  $stderr.puts "@days_ago: '#{@days_ago}'"
  $stderr.puts "@cutoff: '#{@cutoff.to_s}'"
  $stderr.puts "@cutoff_sum: '#{@cutoff_sum.to_s}'"
  $stderr.puts "@th_index_cutoff_sum: '#{@th_index_cutoff_sum}'"
end


#
# Do the run.
#

# Load the thread head index if it exists.
load_thread_head_index(@thread_head_index)

# The location of the Maildir's cur directory.
src_mail_dir_cur = File.join(File.expand_path(@src_mail_dir),'cur')

# Iterate through each file in the Maildir's cur directory.
Dir.foreach(src_mail_dir_cur) do |filename|
  # Skip . and ..
  next if filename == '.'
  next if filename == '..'

  filename = File.join(src_mail_dir_cur, filename)

  # Skip any directories.
  next unless File.file?(filename)

  $stderr.puts "filename: #{filename}" if @debug > 2

  # Parse the file into an email.
  email = TMail::Mail.parse(IO.read(filename))

  if email['references'].nil? && email['in-reply-to'].nil?
    # This email is a thread head

    if email['message-id'].id.nil?
      # This email is a malformed email.
      $stderr.puts "No message-id for #{filename}" unless @quiet

    else
      # Add this email as a thread head.
      add_thread_head(email)

      # Archive this email.
      archive_thread_head(email, @src_mail_dir, filename)

    end

  else
    # This email is a thread child
    thread_head = nil

    # Determine the thread's head (Simple case first since it is the most
    # common)
    if !email['references'].nil? && !email['references'].ids.empty?
      # This email as a References header and it is not empty
      thread_head = email['references'].ids.first.to_sym

    elsif !email['in-reply-to'].nil? && !email['in-reply-to'].empty?
      # This email only has a In-Reply-To header which is not empty
      thread_head = email['in-reply-to'].to_s.to_sym

    end

    # Lookup the thread head in @thread_heads.
    if @thread_heads.key?(thread_head)
      # Found it, so archive this email in the thread head's archive directory.
      archive_thread_child(thread_head, @src_mail_dir, filename)

    else
      # Possibly an orphaned child.  See if any of its other references are
      # known thread heads.
      thread_head = nil

      if email['references'].nil? && !email['in-reply-to'].empty?
        # No References header so use the In-Reply-To header.
        ref = email['in-reply-to'].to_s.to_sym
        thread_head = ref if @thread_heads.key?(ref)

      elsif !email['references'].nil? && !email['references'].empty?
        # Use References header.  Iterate through each of the references and
        # use the first that matches as the thread's head.
        email['references'].ids.each do |ref|
          ref = ref.to_s.to_sym
          if @thread_heads.key?(ref)
            thread_head = ref
            break
          end
        end

      end

      # Do we now have the thread's head?
      if thread_head
        # Yes, so archive this email in the thread head's archive directory.
        archive_thread_child(thread_head, @src_mail_dir, filename)

      else
        # No.  We have an orphan.
        $stderr.puts "th orphan" if @debug > 2
        @total_orphans += 1

        # Archive it as a thread head.
        add_thread_head(email)
        archive_thread_head(email, @src_mail_dir, filename)

      end

    end

  end

  @total_emails += 1

end

# The run is done, so save @thread_heads.
dump_thread_head_index(@thread_head_index)

# Post-run debugging.
if @debug > 0
  $stderr.puts "@thread_heads.length: #{@thread_heads.length}"
  $stderr.puts "@total_orphans: #{@total_orphans}"
  $stderr.puts "@total_emails: #{@total_emails}"
  $stderr.puts "@total_emails_archived: #{@total_emails_archived}"
  $stderr.puts "@th_index_dump_count: #{@th_index_dump_count}"
end

Invoking it is as simple as ./mail_sort.rb -h.

Ruby Method of the Day - Array.reject!

Kelly McCauley — Wed, 21 Nov 2007 00:01:00 -0600

Signature

array.reject! {|element| block}  #=> array or nil

array.reject {|element| block} does the exact same thing as Array.delete_if except that it returns nil if no changes were made to array.

Examples

a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

b = a.clone                 #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
b.reject! {true}            #=> []
b                           #=> []

b = a.clone                 #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
b.reject! {false}           #=> nil
b                           #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

b = a.clone                 #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
b.reject! {|n| n == 3}      #=> [1, 2, 4, 5, 6, 7, 8, 9, 10]
b                           #=> [1, 2, 4, 5, 6, 7, 8, 9, 10]

b = a.clone                 #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
b.reject! {|n| n % 2 == 0}  #=> [1, 3, 5, 7, 9]
b                           #=> [1, 3, 5, 7, 9]

Documentation Reference

Ruby version 1.8.6

www.ruby-doc.org : Array.reject!

Ruby Method of the Day - Array.reject

Kelly McCauley — Tue, 20 Nov 2007 00:01:00 -0600

Signature

array.reject {|element| block}    #=> new_array

array.reject {|element| block} iterates over array’s elements and returns new_array that contains any element in array where the block returns either nil or false.

Examples

a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

a.reject {|n| nil}                        #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
a.reject {|n| false}                      #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
a.reject {|n| true}                       #=> []
a.reject {|n| ''}                         #=> []
a.reject {|n| 0}                          #=> []

a.reject {|n| n == 3}                     #=> [1, 2, 4, 5, 6, 7, 8, 9, 10]
a.reject {|n| n % 2 == 0 }                            #=> [1, 3, 5, 7, 9]
a.reject {|n| true if (n % 3 == 0) || (n % 5 == 0) }  #=> [1, 2, 4, 7, 8]

Documentation Reference

Ruby version 1.8.6

www.ruby-doc.org : Array.reject

Ruby Method of the Day - Array.last

Kelly McCauley — Mon, 19 Nov 2007 00:01:00 -0600

Signature

array.last            #=> object or nil
array.last(number)    #=> new_array

array.last returns the last element of array or it returns nil if array is empty. array.last(number) returns the last number elements of array or it returns an empty array if array is empty.

Examples

a = ["a", "b", "c", "d", "e", "f"]

a.last        #=> "f"
[].last       #=> nil

a.last(0)     #=> []
a.last(1)     #=> ["f"]
a.last(4)     #=> ["c", "d", "e", "f"]
a.last(99)    #=> ["a", "b", "c", "d", "e", "f"]
[].last(10)   #=> []

Documentation Reference

Ruby version 1.8.6

www.ruby-doc.org : Array.last

Ruby Method of the Day - Array.first

Kelly McCauley — Fri, 16 Nov 2007 00:01:00 -0600

Signature

array.first           #=> object or nil
array.first(number)   #=> new_array

array.first returns the first element of array or it returns nil if array is empty. array.first(number) returns the first number elements of array or it returns an empty array if array is empty.

Examples

a = ["a", "b", "c", "d", "e", "f"]

a.first           #=> "a"
[].first          #=> nil

a.first(0)        #=> []
a.first(1)        #=> ["a"]
a.first(99)       #=> ["a", "b", "c", "d", "e", "f"]
[].first(10)      #=> []

Documentation Reference

Ruby version 1.8.6

www.ruby-doc.org : Array.first

Ruby Method of the Day - Array.fetch

Kelly McCauley — Thu, 15 Nov 2007 00:01:00 -0600

Signature

array.fetch(position)             #=> object

array.fetch(position) returns the object located at position. If position is outside array an IndexError is thrown. Note that if position is negative and outside of array then the index number reported in the exception will be reported as position - array.length. I’ve submitted a patch to fix it. If position is 0 or positive then start counting from the beginning of the array. If position is negative then start counting from the end of the array.

array.fetch(position, default)    #=> object

array.fetch(position, default) returns the object located at position. If position is outside of array then default is returned. If position is 0 or positive then start counting from the beginning of the array. If position is negative then start counting from the end of the array.

array.fetch(position) {|position| block}  #=> object

array.fetch(position) {|position| block} returns the object located at position. If position is outside of array then block’s returned results is returned. If position is 0 or positive then start counting from the beginning of the array. If position is negative then start counting from the end of the array.

Examples

a = ['a','b','c','d','e','f']   #=> ["a", "b", "c", "d", "e", "f"]
a.fetch(2)                      #=> "c"
a.fetch(-2)                     #=> "e"

begin
  a.fetch(99)
rescue Exception => e
  e.inspect                     #=> "#<IndexError: index 99 out of array>"
end

begin
  a.fetch(-99)
rescue Exception => e
  # Note that it reports an index of -93 (i.e. -99 - a.length).
  e.inspect                     #=> "#<IndexError: index -93 out of array>"
end


begin
  a.fetch(-7)
rescue Exception => e
  # Note that it reports an index of -1 (i.e. -7 - a.length).
  e.inspect                     #=> "#<IndexError: index -1 out of array>"
end


a.fetch(4, 'z')                 #=> "e"
a.fetch(99, 'z')                #=> "z"

a.fetch(4){false}               #=> "e"
a.fetch(99){false}              #=> false

def default_fetch(ary, position)
  element = nil
  if position > ary.length
    element = ary.last
  else
    element = ary.first
  end
end

a.fetch(99){|position| default_fetch(a, position)}    #=> "f"
a.fetch(4){|position| default_fetch(a, position)}     #=> "e"
a.fetch(-99){|position| default_fetch(a, position)}   #=> "a"

Documentation Reference

Ruby version 1.8.6

www.ruby-doc.org : Array.fetch

Ruby Method of the Day - Array.delete_if

Kelly McCauley — Wed, 14 Nov 2007 00:01:00 -0600

Signature

array.delete_if {|element| block}   #=> array

array.delete_if {|element| block} iterates over all elements of array and deletes an element from array if block returns true for that element.

Examples

123	a = [1,2,3,4,5,6,7,8,9,10] #=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]a.delete_if {\|n\| n % 2 == 0} #=> [1, 3, 5, 7, 9]a #=> [1, 3, 5, 7, 9]

The following is another usage example.

#!/usr/bin/env ruby
#
# A contrived example of using Array.delete_if
#

$VERBOSE = true

require 'pp'

# Member of an organization.
class Member
  attr_accessor :name
  attr_accessor :current
  def current?
    @current
  end
end


# Raw member data
members_data = [
  ['Bob', true],
  ['Fred', false],
  ['Bill', false],
  ['Alice', true]
]

# Create the members based on the raw member data
members = members_data.map do |entry|
  member = Member.new
  member.name = entry[0]
  member.current = entry[1]
  member
end

puts 'Before purging of non-current members...'
pp members
puts "\n"

# Purge non-current members.
members.delete_if {|member| !member.current?}

puts 'After purging of non-current members...'
pp members

Produces…

Before purging of non-current members...
[#<Member:0x87780 @current=true, @name="Bob">,
 #<Member:0x877a8 @current=false, @name="Fred">,
 #<Member:0x87794 @current=false, @name="Bill">,
 #<Member:0x8776c @current=true, @name="Alice">]

After purging of non-current members...
[#<Member:0x87780 @current=true, @name="Bob">,
 #<Member:0x8776c @current=true, @name="Alice">]

Documentation Reference

Ruby version 1.8.6

www.ruby-doc.org : Array.delete_if

Ruby Method of the Day - Array.delete

Kelly McCauley — Tue, 13 Nov 2007 00:01:00 -0600

Signature

array.delete(object)                #=> object or nil
array.delete(object) {|o| block }   #=> object or returned_block_result

array.delete(object) removes all occurances of object from array and returns object if object is found in array or it returns nil if object is not found in array. If block is given and object is not found in array then the the block’s returned results is returned by array.delete.

Examples

a = [1,2,3,4,5,6,7]       #=> [1, 2, 3, 4, 5, 6, 7]
a.delete(7)               #=> 7
a                         #=> [1, 2, 3, 4, 5, 6]
a.delete('foo')           #=> nil

a = [7,1,2,3,7,4,5,6,7]   #=> [7, 1, 2, 3, 7, 4, 5, 6, 7]
a.delete(7)               #=> 7
a                         #=> [1, 2, 3, 4, 5, 6]

a.delete('foo') {false}   #=> false

a.delete('foo') do |item|
  "'#{item}' not found"
end                       #=> "'foo' not found"

Documentation Reference

Ruby version 1.8.6

www.ruby-doc.org : Array.delete

Ruby Method of the Day - Array.compact and Array.compact!

Kelly McCauley — Mon, 12 Nov 2007 00:01:00 -0600

Signature

array.compact   #=> new_array
array.compact!  #=> array or nil

array.compact returns new_array that contains all non-nil elements in array (nil elements removed). array.compact! either returns array with all nil elements removed or returns nil if no nil elements were removed.

Examples

a = [1, nil, nil, 4, nil]       #=> [1, nil, nil, 4, nil]
a.compact                       #=> [1, 4]
a                               #=> [1, nil, nil, 4, nil]
a.compact!                      #=> [1, 4]
a                               #=> [1, 4]
a.compact!                      #=> nil

Documentation Reference

Ruby version 1.8.6

www.ruby-doc.org : Array.compact, Array.compact!

Ruby Method of the Day - Array.pop

Kelly McCauley — Fri, 09 Nov 2007 00:01:00 -0600

Signature

array.pop   #=> object or nil

array.pop removes and returns the last object from array. nil is returned if array is empty (which is not a good test for an empty array since the last object of the array could itself be nil).

Examples

stack = []
stack.pop           #=> nil
stack.push(1,2,3)   #=> [1, 2, 3]
stack.pop           #=> 3
stack               #=> [1, 2]
stack.push(nil)     #=> [1, 2, nil]
stack.pop           #=> nil

Documentation Reference

Ruby version 1.8.6