How To Parse Using Sisimai

Basic Usage

How to parse bounce mails, to get structured data, and to make parsed data as JSON.

Read

From STDIN

Sample codes in Perl and Ruby for reading bounce mail data from Standard-In.

Read

Delivered

How to include a delivery success (delivery notification) in the parsed results.

Read

Except Mails

Sisimai can also parse a bounce object (JSON) got from Cloud Email Services.

Read

Callback

Callback feature allows you to read each bounce message before being parsed.

Read

Read From STDIN


make() and dump() methods of Sisimai accept STDIN as the first argument for reading bounce mail data from Standard-In. This feature had been implemented at earlier version of Sisimai, However, sometimes, it did not work properly caused by some bugs. The bugs have been fixed in the Perl version and the Ruby version at Sisimai 4.18.1.

Perl

STDIN File Handle

dump() and make() methods of Sisimai class read bounce mail data from Standard-In when a file handle STDIN is specified as the first argument.

% cat ./path/to/email | perl -MSisimai -lE 'print Sisimai->dump(STDIN)' | jq
[
  {
    "listid": "",
    "smtpagent": "Sendmail",
    ...
                
% cat ./path/to/email | perl -MSisimai -lE 'print $_->recipient->address for @{ Sisimai->make(STDIN) }'
kijitora@example.jp
sironeko@example.org
                
Ruby

STDIN Object

Similarly, In Ruby version of Sisimai, dump() and make() methods of Sisimai class read bounce mail data from Standard-In when a STDIN object is specified as the first argument.

% cat ./path/to/email | ruby -Ilib -rsisimai -e 'puts Sisimai.dump(STDIN)' | jq
[
  {
    "catch": "",
    "token": "d295b517a3ad6f8748031e1068e07ebd089c0277",
    ...
                
% cat ./path/to/email | ruby -rsisimai -e 'p Sisimai.make(STDIN)'
[#<Sisimai::Data:0x007fa1b0993050 @addresser=#<Sisimai::Address:0x007fa1b0990e40 @user="kijitora", ...
                

Include Delivery Success

make() and dump() methods return the parsed results including a delivery success (delivery notification) when delivered option is specified as the 2nd argument.

email

Succeeded Delivery as a Sample

An email as a sample that has delivered successfully is available as rfc3464-28.eml at github.com/sisimai/set-of-emails repository. The following codes show a sample for parsing the email.

Perl

delivered => 1

When delivered => 1 is specified as the 2nd argument of make() and dump() methods, an email that has delivered successfully is included in the parsed results. If the value of delivered is 0, "" or undef, Sisimai does not generate the parsed results from the succeeded email.

#! /usr/bin/env perl
use Sisimai;
my $f = './set-of-emails/maildir/bsd/rfc3464-28.eml';
my $v = Sisimai->make($f, 'delivered' => 1);
for my $e ( @$v ) {
    print $e->action;           # deliverable
    print $e->deliverystatus;   # 2.1.5
    print $e->reason;           # delivered
    print $e->diagnosticcode;   # 250 2.1.5 Ok
    print $e->softbounce;       # -1
}
                
Ruby

delivered: true

Similarly, In Ruby version of Sisimai, when delivered: true is specified as the 2nd argument of make() and dump() methods, an email that has delivered successfully is included in the parsed results. If the value of delivered is nil or false, Sisimai does not generate the parsed results from the succeeded email.

#! /usr/bin/env ruby
require 'sisimai'
f = './set-of-emails/maildir/bsd/rfc3464-28.eml'
v = Sisimai.make(f, delivered: true)
v.each do |e|
  puts e.action         # deliverable
  puts e.deliverystatus # 2.1.5
  puts e.reason         # delivered
  puts e.diagnosticcode # 250 2.1.5 Ok
  puts e.softbounce     # -1
end
                

Parse Except Mails

Sisimai can parse bounce data except emails using input as the 2nd argument of make() and dump() methods. This feature has implemented at Sisimai v.20.0. As of present, only bounce objects retrieved from Amazon SES (Bounce object thet can be got from SNS) and SendGrid (Bounce object that can be got with Web API) could be parsed

JSON

Bounce Object(JSON)

Some JSON files including bounce object retrieved from Cloud Email Delivery Services (Amazon SES and SendGrid) are available in jsonapi/ directory of github.com/sisimai/set-of-emails repository. The following codes show a sample for parsing ced-us-sendgrid-03.json (Bounce object from SendGrid).

Perl

input => 'json'

When you set a decoded JSON (bounce object) to the 1st argument of, set input => 'json' to the 2nd argument of make() and dump() method of Sisimai, you can get the parsed results.

#! /usr/bin/env perl
use Sisimai;
use JSON;
my $q = '[{"status": "5.1.1", "created": "2011-10-08 14:05:44", "reason": "550...';
my $j = JSON->new;
my $v = Sisimai->make($j->decode($q), 'input' => 'json');
for my $e ( @$v ) {
    print $e->smtpagent;        # CED::US::SendGrid
    print $e->deliverystatus;   # 5.1.1
    print $e->reason;           # userunknown
    print $e->diagnosticcode;   # 550 5.1.1 The email account that you tried to...
    print $e->softbounce;       # 0
}
                
Ruby

input: 'json'

Like the perl version, when you set a decoded JSON (bounce object) to the 1st argument of, set input: 'json' to the 2nd argument of make() and dump() method of Sisimai, you can get the parsed results.

#! /usr/bin/env ruby
require 'sisimai'
require 'json'
q = '[{"status": "5.1.1", "created": "2011-10-08 14:05:44", "reason": "550...'
v = Sisimai.make(JSON.load(q), input: 'json')
v.each do |e|
  puts e.smtpagent      # CED::US::SendGrid
  puts e.deliverystatus # 5.1.1
  puts e.reason         # userunknown
  puts e.diagnosticcode # 550 5.1.1 The email account that you tried to...
  puts e.softbounce     # 0
end
                

Callback Feature

Beginning with Sisimai 4.19.0, Callback feature has implemented. This feature allows you to read the header part and the body part of each bounce mail before being parsed by Sisimai, and receive the results returned from a hook method you specified via catch() method of Sisimai::Data or Sisimai::Message object.

It is useful to get entire text of the original message, or the value of a certain header of the original message such as a delivery ID, X-Mailer, and so on.

argv

Argument of Hook Method

A hook method in Perl (sub routine) receives a Hash reference, a hook method in Ruby receives a Hash object. Hash reference and object have the following structure.

Key Name Data Type Description Email JSON
datasrc String Kind of data source to be parsed Available (email) Available (json)
headers Hash Header part of a bounce mail (as a Hash Data) Available undef, nil
message String Body part of a bounce mail including the original message Available undef, nil
bounces Hash or Array Bounce object (Decoded JSON) undef, nil Available
email

mta-postfix-16.eml

The following codes show an example for getting the following values from mta-postfix-16.eml using a hook method given as the 2nd argument of make() method of Sisimai.

  • The value of Received-SPF in the header part
  • The value of X-Postfix-Queue-ID in the message body
Return-Path: <>
Received: by 192.0.2.22 with SMTP id fffffff000;
    Thu, 20 Jan 2011 23:34:45 +0900 (JST)
Received: from mx0.example.jp (mx0.example.jp [192.0.2.9])
    by mx.example.org with ESMTP id 0000000000000000000000;
    Thu, 20 Jan 2011 23:34:45 +0900 (JST)
Received-SPF: pass (example.org: best guess record for domain of mx0.example.jp designates 192.0.2.9 as permitted sender) client-ip=192.0.2.9;
Received: by mx0.example.jp (Postfix)
    id FFFFFFFFFFFF; Thu, 20 Jan 2011 23:34:45 +0900 (JST)
Date: Thu, 20 Jan 2011 23:34:45 +0900 (JST)
From: MAILER-DAEMON@example.jp (Mail Delivery System)
Subject: Undelivered Mail Returned to Sender
To: shironeko@cat.example.com
Auto-Submitted: auto-replied
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
    boundary="FFFFFFFFFF.00000000/mx0.example.jp"
Content-Transfer-Encoding: 8bit
Message-Id: <000000000000000.FFFFFFFFFFFF@mx0.example.jp>

This is a MIME-encapsulated message.

--FFFFFFFFFF.00000000/mx0.example.jp
Content-Description: Notification
Content-Type: text/plain; charset=us-ascii

This is the mail system at host mx0.example.jp.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to <postmaster>

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

                   The mail system

<kijitora@example.go.jp>: host mx1.example.go.jp[192.0.2.127] said: 550 5.1.6 recipient
    no longer on server: kijitora@example.go.jp (in reply to RCPT TO command)

--FFFFFFFFFF.00000000/mx0.example.jp
Content-Description: Delivery report
Content-Type: message/delivery-status

Reporting-MTA: dns; mx0.example.jp
X-Postfix-Queue-ID: FFFFFF000000
X-Postfix-Sender: rfc822; shironeko@cat.example.com
...
                

The following is an example of an argument that the hook method you defined receives. The argument have the header part (Hash) and the body part (String) generated from mta-postfix-16.eml .

{
  'bounces' => undef,
  'datasrc' => 'email',
  'headers' => {
     'x-mailer' => undef,
     'return-path' => '<>',
     'content-type' => 'multipart/report; report-type=delivery-status; boundary="FFFFFFFFFF.00000000/mx0.example.jp"',
     'subject' => 'Undelivered Mail Returned to Sender',
     'date' => 'Thu, 20 Jan 2011 23:34:45 +0900 (JST)',
     'to' => 'shironeko@cat.example.com',
     'received-spf' => 'pass (example.org: best guess record for domain of mx0.example.jp designates 192.0.2.9 as permitted sender) client-ip=192.0.2.9;',
     'reply-to' => undef,
     'message-id' => '<000000000000000.FFFFFFFFFFFF@mx0.example.jp>',
     'from' => 'MAILER-DAEMON@example.jp (Mail Delivery System)',
     'content-transfer-encoding' => '8bit',
     'received' => [
         'by 192.0.2.22 with SMTP id fffffff000; Thu, 20 Jan 2011 23:34:45 +0900 (JST)',
         'from mx0.example.jp (mx0.example.jp [192.0.2.9]) by mx.example.org with ESMTP id 0000000000000000000000; Thu, 20 Jan 2011 23:34:45 +0900 (JST)',
         'by mx0.example.jp (Postfix) id FFFFFFFFFFFF; Thu, 20 Jan 2011 23:34:45 +0900 (JST)'
       ],
     'auto-submitted' => 'auto-replied'
     },
  'message' => 'This is a MIME-encapsulated message.
...
Reporting-MTA: dns; mx0.example.jp
X-Postfix-Queue-ID: FFFFFF000000
X-Postfix-Sender: rfc822; shironeko@cat.example.com
Arrival-Date: Thu, 20 Jan 2011 23:34:45 +0900 (JST)

Final-Recipient: rfc822; kijitora@example.go.jp
Action: failed
Status: 5.1.6
...
Subject: Nyaaaaan
To: Kijitora 
Message-Id: <0000000000000.FFFFFF000000@mx0.example.jp>

Nyaaaan
'
  }
                
Perl

Sisimai->make($f, 'hook' => $x, 'field' => [])

make() and dump() methods of Sisimai class receive a code reference for being executed as the 2nd argument hook like the following code:

Argument field receives an email header list as an array reference which you want to capture in the hook method. This argument is implemented at Sisimai 4.20.2.

#! /usr/bin/env perl
use Sisimai;
my $f = './set-of-emails/maildir/bsd/mta-sendmail-24.eml';
my $x = sub {
    my $argv = shift;
    my $data = { 'queue-id' => '', 'received-spf' => '' };

    $data->{'received-spf'} = $argv->{'headers'}->{'received-spf'} || '';
    if( $argv->{'message'} =~ m/^X-Postfix-Queue-ID:\s*(.+)$/m ) {
        $data->{'queue-id'} = $1;
    }
    return $data;
};
my $h = ['Received-SPF'];
my $j = Sisimai->dump($f, 'hook' => $x, 'field' => $h); # JSON文字列を得る
my $v = Sisimai->make($f, 'hook' => $x, 'field' => $h);
print $v->[0]->{'catch'}->{'received-spf'};  # pass (example.org:...
print $v->[0]->{'catch'}->{'queue-id'};      # FFFFFF000000
                
Ruby

Sisimai.make(f, hook: x, field: h)

make() and dump() methods of Sisimai class receive a Proc object for being executed as the 2nd argument hook: like the following code:

Argument field: receives an email header list as an array reference which you want to capture in the hook method. This argument is implemented at Sisimai 4.20.2.

#! /usr/bin/env ruby
require 'sisimai'
f = './set-of-emails/maildir/bsd/mta-postfix-16.eml'
x = lambda do |argv|
  data = { 'queue-id' => '', 'received-spf' => '' }

  data['received-spf'] = argv['headers']['received-spf'] || ''
  if cv = argv['message'].match(/^X-Postfix-Queue-ID:\s*(.+)$/)
    data['queue-id'] = cv[1]
  end
  return data
end

h = ['Received-SPF']
j = Sisimai.dump(f, hook: x, field: h)    # JSON文字列を得る
v = Sisimai.make(f, hook: x, field: h)
puts v[0]['catch']['received-spf'] # pass (example.org: ...
puts v[0]['catch']['queue-id']     # FFFFFF000000
                
JSON

JSON string by dump()

JSON string generated by dump() method of Sisimai with a hook method is like the following:

[
  {
    "timestamp": 1270043983,
    "smtpcommand": "RCPT",
    "subject": "Nyaaan",
    "recipient": "kijitora@example.jp",
    "destination": "example.jp",
    "senderdomain": "example.co.jp",
    "token": "bb2a5dfb6161f396300b24acd9bf637c1c45a58a",
    "smtpagent": "qmail",
    "replycode": "450",
    "softbounce": 1,
    "catch": {
      "received-spf": "pass (example.org: best guess record for domain of mx0.example.jp designates 192.0.2.9 as permitted sender) client-ip=192.0.2.9;",
      "queue-id": "FFFFFF000000",
    },
    "rhost": "192.0.2.135",
    "timezoneoffset": "+0900",
    "addresser": "shironeko@example.co.jp",
    "diagnostictype": "SMTP",
    "action": "failed",
    "listid": "",
    "feedbacktype": "",
    "diagnosticcode": "192.0.2.135 does not like recipient. Remote host said: 450 4.2.2 ... Mailbox Full Giving up on 192.0.2.135. I'm not going to try again; this message has been in the queue too long.",
    "reason": "mailboxfull",
    "deliverystatus": "4.2.2",
    "alias": "",
    "lhost": "",
    "messageid": "57D03121-12F7-4801-B30F-9E44B15E56DC@example.co.jp"
  }
]
                
Ojbect

Bounce Object: ced-us-amazonses-01.eml

The following codes show an example for getting value of mail.sendingAccountId from a bounce object (retrieved or notified from AWS SNS) ced-us-amazonses-01.json using a hook method given as the 2nd argument of make() method of Sisimai.

{
  "notificationType": "Bounce",
  "bounce": {
    "bounceType": "Permanent",
    "bounceSubType": "General",
    "bouncedRecipients": [
      {
        "emailAddress": "bounce@simulator.amazonses.com",
        "action": "failed",
        "status": "5.1.1",
        "diagnosticCode": "smtp; 550 5.1.1 user unknown"
      }
    ],
    "timestamp": "2016-10-21T00:06:40.502Z",
    "feedbackId": "01010157e48fa03f-c7e948fe-3c34-403e-b681-02a497797067-000000",
    "reportingMTA": "dsn; a27-23.smtp-out.us-west-2.amazonses.com"
  },
  "mail": {
    "timestamp": "2016-10-21T00:06:39.000Z",
    "source": "kijitora@neko.example.org",
    "sourceArn": "arn:aws:ses:us-west-2:123456789012:identity/kijitora@neko.example.org",
    "sendingAccountId": "123456789012",
    ...
                

The following is an example of an argument that the hook method you defined receives. The argument have a bounce object (decoded JSON) in "bounces" key decoded from ced-us-amazonses-01.json .

{
    'datasrc' => 'json',
    'headers' => undef,
    'message' => undef,
    'bounces' => {
        'bounce' => {
            'feedbackId' => '01010157e48fa03f-c7e948fe-3c34-403e-b681-02a497797067-000000',
            'timestamp' => '2016-10-21T00:06:40.502Z',
            'reportingMTA' => 'dsn; a27-23.smtp-out.us-west-2.amazonses.com',
            'bounceType' => 'Permanent',
        },
        'mail' => {
            'source' => 'kijitora@neko.example.org',
            'sendingAccountId' => '123456789012',
            'destination' => [
                'bounce@simulator.amazonses.com'
            ],
            'timestamp' => '2016-10-21T00:06:39.000Z',
            'sourceArn' => 'arn:aws:ses:us-west-2:123456789012:identity/kijitora@neko.example.org',
            'messageId' => '01010157e48f9b9b-891e9a0e-9c9d-4773-9bfe-608f2ef4756d-000000'
        },
        'notificationType' => 'Bounce'
    }
}
                
Perl

Sisimai->make($f, 'hook' => $x, 'input' => 'json')

make() and dump() methods of Sisimai class receive a code reference for being executed as the 2nd argument hook like the following code:

#! /usr/bin/env perl
use Sisimai;
use JSON;
my $q = '{"notificationType": "Bounce", "bounce": {...';
my $o = JSON->new;
my $x = sub {
    my $argv = shift;
    my $data = { 'account-id' => '' };

    $data->{'account-id'} = $argv->{'bounces'}->{'mail'}->{'sendingAccountId'};
    return $data;
};
my $j = Sisimai->dump($o->decode($q), 'hook' => $x, 'input' => 'json'); # get JSON string
my $v = Sisimai->make($o->decode($q), 'hook' => $x, 'input' => 'json');
print $v->[0]->{'catch'}->{'account-id'};  # 123456789012
                
Ruby

Sisimai.make(f, hook: x, input: 'json')

make() and dump() methods of Sisimai class receive a Proc object for being executed as the 2nd argument hook: like the following code:

#! /usr/bin/env ruby
require 'sisimai'
require 'json'
q = '{"notificationType": "Bounce", "bounce": {...'
o = JSON.load(q)
x = lambda do |argv|
  data = { 'acount-id' => nil }

  data['account-id'] = argv['bounces']['mail']['sendingAccountId']
  return data
end

j = Sisimai.dump(o, hook: x, input: 'json')  # get JSON string
v = Sisimai.make(o, hook: x, input: 'json')
puts v[0]['catch']['account-id']  # 123456789012
                
JSON

JSON string by dump()

JSON string generated by dump() method of Sisimai with a hook method is like the following:

[
  {
    "timezoneoffset": "+0900",
    "reason": "userunknown",
    "replycode": "550",
    "messageid": "01010157e48f9b9b-891e9a0e-9c9d-4773-9bfe-608f2ef4756d-000000",
    "lhost": "a27-23.smtp-out.us-west-2.amazonses.com",
    "destination": "simulator.amazonses.com",
    "senderdomain": "neko.example.org",
    "diagnosticcode": "550 5.1.1 user unknown",
    "addresser": "kijitora@neko.example.org",
    "catch": {
      "account-id": 123456789012
    },
    "subject": "nyaan",
    "deliverystatus": "5.1.1",
    "token": "79f14902d1e4b8f9c40727609d7a752a1700564e",
    "feedbacktype": "",
    "softbounce": 0,
    "rhost": "",
    "smtpcommand": "",
    "smtpagent": "CED::US::AmazonSES",
    "recipient": "bounce@simulator.amazonses.com",
    "alias": "",
    "diagnostictype": "SMTP",
    "listid": "",
    "action": "failed",
    "timestamp": 1476976000
  }
]