Sisimai: Mail Analyzing Interface

How To Parse Using Sisimai

Basic Usage

How to parse bounce mails, to get structured data, and to make parsed data as JSON. This is the most simple and easy way to parse using Sisimai.

Read

From STDIN

Sample code in this description shows a way to read bounce mail data from Standard-In. It is useful for reading bounce mails from pipeline on one-liner.

Read

From Variable

Sample code in this description shows a way to read bounce mail data from variable. It is useful for parsing data stored in a database or job queue system.

Read

Include Delivered

The default behaviour of Sisimai is parsing a bounce message only, but Sisimai can also include a delivery success (delivery notification) in the parsed results.

Read

Except Mails

This feature has been removed at Sisimai 4.25.5.
Sisimai can also parse a bounce object (JSON) got from Cloud Email Services: As of present, Amazon SES and SendGrid are supported.

Read

Callback

Callback feature allows you to read each bounce message, to pick a specific header field, to set custom values into the parsed results before being parsed.

Read

Read From STDIN

make() and dump() methods of Sisimai accept STDIN as the first argument for reading bounce mail data from Standard-In. This feature had been implemented at earlier version of Sisimai, However, sometimes, it did not work properly caused by some bugs. The bugs have been fixed in the Perl version and the Ruby version at Sisimai 4.18.1.

Perl

STDIN File Handle

dump() and make() methods of Sisimai class read bounce mail data from Standard-In when a file handle STDIN is specified as the first argument. 

% cat ./path/to/email | perl -MSisimai -lE 'print Sisimai->dump(STDIN)' | jq
[
  {
    "listid": "",
    "smtpagent": "Sendmail",
    ...
                
% cat ./path/to/email | perl -MSisimai -lE 'print $_->recipient->address for @{ Sisimai->make(STDIN) }'
kijitora@example.jp
sironeko@example.org
Ruby

STDIN Object

Similarly, In Ruby version of Sisimai, dump() and make() methods of Sisimai class read bounce mail data from Standard-In when a STDIN object is specified as the first argument. 

% cat ./path/to/email | ruby -Ilib -rsisimai -e 'puts Sisimai.dump(STDIN)' | jq
[
  {
    "catch": "",
    "token": "d295b517a3ad6f8748031e1068e07ebd089c0277",
    ...
                
% cat ./path/to/email | ruby -rsisimai -e 'p Sisimai.make(STDIN)'
[#<Sisimai::Data:0x007fa1b0993050 @addresser=#<Sisimai::Address:0x007fa1b0990e40 @user="kijitora", ...

Read From Variable

Beginning with Sisimai 4.23.0, make() and dump() methods accept a variable (keeping entire bounce message) as the first argument. The variable as the first argument should be a scalar reference in Perl, a String object in Ruby.

This feature is useful to read bounce messages stored in a database or job queue system.

Perl

Scalar Reference

When a scalar reference which is holding entire bounce message is specified as the 1st argument of make() and dump() methods, Sisimai calls Sisimai::Mail::Memory class and return the parsed results. 

#! /usr/bin/env perl
use Sisimai;
my $f = 'From MAILER-DAEMON Thu Dec 22 17:54:04 2015...';   # Entire bounce message
my $v = Sisimai->make(\$f);
for my $e ( @$v ) {
    print $e->action;           # delayed
    print $e->deliverystatus;   # 4.7.0
    print $e->reason;           # expired
    print $e->diagnosticcode;   # Envelope expired
    print $e->softbounce;       # 1
}
Ruby

String object

Similarly, In Ruby version of Sisimai, When a String object which is holding entire bounce message is specified as the 1st argument of make() and dump() methods, Sisimai calls Sisimai::Mail::Memory class and return the parsed results. 

#! /usr/bin/env ruby
require 'sisimai'
f = 'From MAILER-DAEMON Thu Dec 22 17:54:04 2015...'    # Entire bounce message
v = Sisimai.make(f)
v.each do |e|
  puts e.action         # delayed
  puts e.deliverystatus # 4.7.0
  puts e.reason         # expired
  puts e.diagnosticcode # Envelope expired
  puts e.softbounce     # 1
end

Include Delivery Success

make() and dump() methods return the parsed results including a delivery success (delivery notification) when delivered option is specified as the 2nd argument.

email

Succeeded Delivery as a Sample

An email as a sample that has delivered successfully is available as rfc3464-28.eml at github.com/sisimai/set-of-emails repository. The following code shows a sample for parsing the email.

Perl

delivered => 1

When delivered => 1 is specified as the 2nd argument of make() and dump() methods, an email that has delivered successfully is included in the parsed results. If the value of delivered is 0, "" or undef, Sisimai does not generate the parsed results from the succeeded email. 

#! /usr/bin/env perl
use Sisimai;
my $f = './set-of-emails/maildir/bsd/rfc3464-28.eml';
my $v = Sisimai->make($f, 'delivered' => 1);
for my $e ( @$v ) {
    print $e->action;           # deliverable
    print $e->deliverystatus;   # 2.1.5
    print $e->reason;           # delivered
    print $e->diagnosticcode;   # 250 2.1.5 Ok
    print $e->softbounce;       # -1
}
Ruby

delivered: true

Similarly, In Ruby version of Sisimai, when delivered: true is specified as the 2nd argument of make() and dump() methods, an email that has delivered successfully is included in the parsed results. If the value of delivered is nil or false, Sisimai does not generate the parsed results from the succeeded email. 

#! /usr/bin/env ruby
require 'sisimai'
f = './set-of-emails/maildir/bsd/rfc3464-28.eml'
v = Sisimai.make(f, delivered: true)
v.each do |e|
  puts e.action         # deliverable
  puts e.deliverystatus # 2.1.5
  puts e.reason         # delivered
  puts e.diagnosticcode # 250 2.1.5 Ok
  puts e.softbounce     # -1
end

Parse Except Mails

This feature has been removed at Sisimai 4.25.5.
Sisimai can parse bounce data except emails using input as the 2nd argument of make() and dump() methods. This feature has implemented at Sisimai v.20.0. As of present, only bounce objects retrieved from Amazon SES (Bounce object thet can be got from SNS) and SendGrid (Bounce object that can be got with Web API) could be parsed

JSON

Bounce Object(JSON)

Some JSON files including bounce object retrieved from Cloud Email Delivery Services (Amazon SES and SendGrid) are available in jsonobj/ directory of github.com/sisimai/set-of-emails repository. The following code shows a sample for parsing json-sendgrid-03.json (Bounce object from SendGrid).

Perl

input => 'json'

When you set a decoded JSON (bounce object) to the 1st argument of, set input => 'json' to the 2nd argument of make() and dump() method of Sisimai, you can get the parsed results. 

#! /usr/bin/env perl
use Sisimai;
use JSON;
my $q = '[{"status": "5.1.1", "created": "2011-10-08 14:05:44", "reason": "550...';
my $j = JSON->new;
my $v = Sisimai->make($j->decode($q), 'input' => 'json');
for my $e ( @$v ) {
    print $e->smtpagent;        # JSON::SendGrid
    print $e->deliverystatus;   # 5.1.1
    print $e->reason;           # userunknown
    print $e->diagnosticcode;   # 550 5.1.1 The email account that you tried to...
    print $e->softbounce;       # 0
}
Ruby

input: 'json'

Like the perl version, when you set a decoded JSON (bounce object) to the 1st argument of, set input: 'json' to the 2nd argument of make() and dump() method of Sisimai, you can get the parsed results. 

#! /usr/bin/env ruby
require 'sisimai'
require 'json'
q = '[{"status": "5.1.1", "created": "2011-10-08 14:05:44", "reason": "550...'
v = Sisimai.make(JSON.load(q), input: 'json')
v.each do |e|
  puts e.smtpagent      # JSON::SendGrid
  puts e.deliverystatus # 5.1.1
  puts e.reason         # userunknown
  puts e.diagnosticcode # 550 5.1.1 The email account that you tried to...
  puts e.softbounce     # 0
end

Callback Feature

Beginning with Sisimai 4.19.0, Callback feature has implemented. This feature allows you to read the header part and the body part of each bounce mail before being parsed by Sisimai, and receive the results returned from a hook method you specified via catch() method of Sisimai::Data or Sisimai::Message object.

It is useful to get entire text of the original message, or the value of a certain header of the original message such as a delivery ID, X-Mailer, and so on.

argv

Argument of Hook Method

A hook method in Perl (sub routine) receives a Hash reference, a hook method in Ruby receives a Hash object. Hash reference and object have the following structure.

Key Name Data Type Description Email JSON
datasrc String Kind of data source to be parsed Available (email) Available (json)
headers Hash Header part of a bounce mail (as a Hash Data) Available undef, nil
message String Body part of a bounce mail including the original message Available undef, nil
bounces Hash or Array Bounce object (Decoded JSON) undef, nil Available
email

email-postfix-16.eml

The following code shows an example for getting the following values from email-postfix-16.eml using a hook method given as the 2nd argument of make() method of Sisimai.

  • The value of Received-SPF in the header part
  • The value of X-Postfix-Queue-ID in the message body
Return-Path: <>
Received: by 192.0.2.22 with SMTP id fffffff000;
    Thu, 20 Jan 2011 23:34:45 +0900 (JST)
Received: from mx0.example.jp (mx0.example.jp [192.0.2.9])
    by mx.example.org with ESMTP id 0000000000000000000000;
    Thu, 20 Jan 2011 23:34:45 +0900 (JST)
Received-SPF: pass (example.org: best guess record for domain of mx0.example.jp designates 192.0.2.9 as permitted sender) client-ip=192.0.2.9;
Received: by mx0.example.jp (Postfix)
    id FFFFFFFFFFFF; Thu, 20 Jan 2011 23:34:45 +0900 (JST)
Date: Thu, 20 Jan 2011 23:34:45 +0900 (JST)
From: MAILER-DAEMON@example.jp (Mail Delivery System)
Subject: Undelivered Mail Returned to Sender
To: shironeko@cat.example.com
Auto-Submitted: auto-replied
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
    boundary="FFFFFFFFFF.00000000/mx0.example.jp"
Content-Transfer-Encoding: 8bit
Message-Id: <000000000000000.FFFFFFFFFFFF@mx0.example.jp>

This is a MIME-encapsulated message.

--FFFFFFFFFF.00000000/mx0.example.jp
Content-Description: Notification
Content-Type: text/plain; charset=us-ascii

This is the mail system at host mx0.example.jp.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to <postmaster>

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

                   The mail system

<kijitora@example.go.jp>: host mx1.example.go.jp[192.0.2.127] said: 550 5.1.6 recipient
    no longer on server: kijitora@example.go.jp (in reply to RCPT TO command)

--FFFFFFFFFF.00000000/mx0.example.jp
Content-Description: Delivery report
Content-Type: message/delivery-status

Reporting-MTA: dns; mx0.example.jp
X-Postfix-Queue-ID: FFFFFF000000
X-Postfix-Sender: rfc822; shironeko@cat.example.com
...

The following is an example of an argument that the hook method you defined receives. The argument have the header part (Hash) and the body part (String) generated from email-postfix-16.eml . 

{
  'bounces' => undef,
  'datasrc' => 'email',
  'headers' => {
     'x-mailer' => undef,
     'return-path' => '<>',
     'content-type' => 'multipart/report; report-type=delivery-status; boundary="FFFFFFFFFF.00000000/mx0.example.jp"',
     'subject' => 'Undelivered Mail Returned to Sender',
     'date' => 'Thu, 20 Jan 2011 23:34:45 +0900 (JST)',
     'to' => 'shironeko@cat.example.com',
     'received-spf' => 'pass (example.org: best guess record for domain of mx0.example.jp designates 192.0.2.9 as permitted sender) client-ip=192.0.2.9;',
     'reply-to' => undef,
     'message-id' => '<000000000000000.FFFFFFFFFFFF@mx0.example.jp>',
     'from' => 'MAILER-DAEMON@example.jp (Mail Delivery System)',
     'content-transfer-encoding' => '8bit',
     'received' => [
         'by 192.0.2.22 with SMTP id fffffff000; Thu, 20 Jan 2011 23:34:45 +0900 (JST)',
         'from mx0.example.jp (mx0.example.jp [192.0.2.9]) by mx.example.org with ESMTP id 0000000000000000000000; Thu, 20 Jan 2011 23:34:45 +0900 (JST)',
         'by mx0.example.jp (Postfix) id FFFFFFFFFFFF; Thu, 20 Jan 2011 23:34:45 +0900 (JST)'
       ],
     'auto-submitted' => 'auto-replied'
     },
  'message' => 'This is a MIME-encapsulated message.
...
Reporting-MTA: dns; mx0.example.jp
X-Postfix-Queue-ID: FFFFFF000000
X-Postfix-Sender: rfc822; shironeko@cat.example.com
Arrival-Date: Thu, 20 Jan 2011 23:34:45 +0900 (JST)

Final-Recipient: rfc822; kijitora@example.go.jp
Action: failed
Status: 5.1.6
...
Subject: Nyaaaaan
To: Kijitora 
Message-Id: <0000000000000.FFFFFF000000@mx0.example.jp>

Nyaaaan
'
  }
Perl

Sisimai->make($f, 'hook' => $x, 'field' => [])

make() and dump() methods of Sisimai class receive a code reference for being executed as the 2nd argument hook like the following code:

Argument field receives an email header list as an array reference which you want to capture in the hook method. This argument is implemented at Sisimai 4.20.2. 

#! /usr/bin/env perl
use Sisimai;
my $f = './set-of-emails/maildir/bsd/email-sendmail-24.eml';
my $x = sub {
    my $argv = shift;
    my $data = { 'queue-id' => '', 'received-spf' => '' };

    $data->{'received-spf'} = $argv->{'headers'}->{'received-spf'} || '';
    if( $argv->{'message'} =~ m/^X-Postfix-Queue-ID:\s*(.+)$/m ) {
        $data->{'queue-id'} = $1;
    }
    return $data;
};
my $h = ['Received-SPF'];
my $j = Sisimai->dump($f, 'hook' => $x, 'field' => $h); # JSON文字列を得る
my $v = Sisimai->make($f, 'hook' => $x, 'field' => $h);
print $v->[0]->{'catch'}->{'received-spf'};  # pass (example.org:...
print $v->[0]->{'catch'}->{'queue-id'};      # FFFFFF000000
Ruby

Sisimai.make(f, hook: x, field: h)

make() and dump() methods of Sisimai class receive a Proc object for being executed as the 2nd argument hook: like the following code:

Argument field: receives an email header list as an array reference which you want to capture in the hook method. This argument is implemented at Sisimai 4.20.2. 

#! /usr/bin/env ruby
require 'sisimai'
f = './set-of-emails/maildir/bsd/email-postfix-16.eml'
x = lambda do |argv|
  data = { 'queue-id' => '', 'received-spf' => '' }

  data['received-spf'] = argv['headers']['received-spf'] || ''
  if cv = argv['message'].match(/^X-Postfix-Queue-ID:\s*(.+)$/)
    data['queue-id'] = cv[1]
  end
  return data
end

h = ['Received-SPF']
j = Sisimai.dump(f, hook: x, field: h)    # JSON文字列を得る
v = Sisimai.make(f, hook: x, field: h)
puts v[0]['catch']['received-spf'] # pass (example.org: ...
puts v[0]['catch']['queue-id']     # FFFFFF000000
JSON

JSON string by dump()

JSON string generated by dump() method of Sisimai with a hook method is like the following: 

[
  {
    "timestamp": 1270043983,
    "smtpcommand": "RCPT",
    "subject": "Nyaaan",
    "recipient": "kijitora@example.jp",
    "destination": "example.jp",
    "senderdomain": "example.co.jp",
    "token": "bb2a5dfb6161f396300b24acd9bf637c1c45a58a",
    "smtpagent": "qmail",
    "replycode": "450",
    "softbounce": 1,
    "catch": {
      "received-spf": "pass (example.org: best guess record for domain of mx0.example.jp designates 192.0.2.9 as permitted sender) client-ip=192.0.2.9;",
      "queue-id": "FFFFFF000000",
    },
    "rhost": "192.0.2.135",
    "timezoneoffset": "+0900",
    "addresser": "shironeko@example.co.jp",
    "diagnostictype": "SMTP",
    "action": "failed",
    "listid": "",
    "feedbacktype": "",
    "diagnosticcode": "192.0.2.135 does not like recipient. Remote host said: 450 4.2.2 ... Mailbox Full Giving up on 192.0.2.135. I'm not going to try again; this message has been in the queue too long.",
    "reason": "mailboxfull",
    "deliverystatus": "4.2.2",
    "alias": "",
    "lhost": "",
    "messageid": "57D03121-12F7-4801-B30F-9E44B15E56DC@example.co.jp"
  }
]
Ojbect

Bounce Object: json-amazonses-01.eml

The following code shows an example for getting value of mail.sendingAccountId from a bounce object (retrieved or notified from AWS SNS) json-amazonses-01.json using a hook method given as the 2nd argument of make() method of Sisimai. 

{
  "notificationType": "Bounce",
  "bounce": {
    "bounceType": "Permanent",
    "bounceSubType": "General",
    "bouncedRecipients": [
      {
        "emailAddress": "bounce@simulator.amazonses.com",
        "action": "failed",
        "status": "5.1.1",
        "diagnosticCode": "smtp; 550 5.1.1 user unknown"
      }
    ],
    "timestamp": "2016-10-21T00:06:40.502Z",
    "feedbackId": "01010157e48fa03f-c7e948fe-3c34-403e-b681-02a497797067-000000",
    "reportingMTA": "dsn; a27-23.smtp-out.us-west-2.amazonses.com"
  },
  "mail": {
    "timestamp": "2016-10-21T00:06:39.000Z",
    "source": "kijitora@neko.example.org",
    "sourceArn": "arn:aws:ses:us-west-2:123456789012:identity/kijitora@neko.example.org",
    "sendingAccountId": "123456789012",
    ...

The following is an example of an argument that the hook method you defined receives. The argument have a bounce object (decoded JSON) in "bounces" key decoded from json-amazonses-01.json . 

{
    'datasrc' => 'json',
    'headers' => undef,
    'message' => undef,
    'bounces' => {
        'bounce' => {
            'feedbackId' => '01010157e48fa03f-c7e948fe-3c34-403e-b681-02a497797067-000000',
            'timestamp' => '2016-10-21T00:06:40.502Z',
            'reportingMTA' => 'dsn; a27-23.smtp-out.us-west-2.amazonses.com',
            'bounceType' => 'Permanent',
        },
        'mail' => {
            'source' => 'kijitora@neko.example.org',
            'sendingAccountId' => '123456789012',
            'destination' => [
                'bounce@simulator.amazonses.com'
            ],
            'timestamp' => '2016-10-21T00:06:39.000Z',
            'sourceArn' => 'arn:aws:ses:us-west-2:123456789012:identity/kijitora@neko.example.org',
            'messageId' => '01010157e48f9b9b-891e9a0e-9c9d-4773-9bfe-608f2ef4756d-000000'
        },
        'notificationType' => 'Bounce'
    }
}
Perl

Sisimai->make($f, 'hook' => $x, 'input' => 'json')

make() and dump() methods of Sisimai class receive a code reference for being executed as the 2nd argument hook like the following code: 

#! /usr/bin/env perl
use Sisimai;
use JSON;
my $q = '{"notificationType": "Bounce", "bounce": {...';
my $o = JSON->new;
my $x = sub {
    my $argv = shift;
    my $data = { 'account-id' => '' };

    $data->{'account-id'} = $argv->{'bounces'}->{'mail'}->{'sendingAccountId'};
    return $data;
};
my $j = Sisimai->dump($o->decode($q), 'hook' => $x, 'input' => 'json'); # get JSON string
my $v = Sisimai->make($o->decode($q), 'hook' => $x, 'input' => 'json');
print $v->[0]->{'catch'}->{'account-id'};  # 123456789012
Ruby

Sisimai.make(f, hook: x, input: 'json')

make() and dump() methods of Sisimai class receive a Proc object for being executed as the 2nd argument hook: like the following code: 

#! /usr/bin/env ruby
require 'sisimai'
require 'json'
q = '{"notificationType": "Bounce", "bounce": {...'
o = JSON.load(q)
x = lambda do |argv|
  data = { 'acount-id' => nil }

  data['account-id'] = argv['bounces']['mail']['sendingAccountId']
  return data
end

j = Sisimai.dump(o, hook: x, input: 'json')  # get JSON string
v = Sisimai.make(o, hook: x, input: 'json')
puts v[0]['catch']['account-id']  # 123456789012
JSON

JSON string by dump()

JSON string generated by dump() method of Sisimai with a hook method is like the following: 

[
  {
    "timezoneoffset": "+0900",
    "reason": "userunknown",
    "replycode": "550",
    "messageid": "01010157e48f9b9b-891e9a0e-9c9d-4773-9bfe-608f2ef4756d-000000",
    "lhost": "a27-23.smtp-out.us-west-2.amazonses.com",
    "destination": "simulator.amazonses.com",
    "senderdomain": "neko.example.org",
    "diagnosticcode": "550 5.1.1 user unknown",
    "addresser": "kijitora@neko.example.org",
    "catch": {
      "account-id": 123456789012
    },
    "subject": "nyaan",
    "deliverystatus": "5.1.1",
    "token": "79f14902d1e4b8f9c40727609d7a752a1700564e",
    "feedbacktype": "",
    "softbounce": 0,
    "rhost": "",
    "smtpcommand": "",
    "smtpagent": "JSON::AmazonSES",
    "recipient": "bounce@simulator.amazonses.com",
    "alias": "",
    "diagnostictype": "SMTP",
    "listid": "",
    "action": "failed",
    "timestamp": 1476976000
  }
]