How to parse bounce mails, to get structured data, and to make parsed data as JSON. This is the most simple and easy way to parse using Sisimai.
Sample code in this description shows a way to read bounce mail data from Standard-In. It is useful for reading bounce mails from pipeline on one-liner.
Sample code in this description shows a way to read bounce mail data from variable. It is useful for parsing data stored in a database or job queue system.
The default behaviour of Sisimai is parsing a bounce message only, but Sisimai can also include a delivery success (delivery notification) in the parsed results.
Callback feature allows you to read each bounce message, to pick a specific header field, to set custom values into the parsed results before being parsed.
make() and dump() methods of Sisimai accept STDIN as the first argument for reading bounce mail data from Standard-In. This feature had been implemented at earlier version of Sisimai, However, sometimes, it did not work properly caused by some bugs. The bugs have been fixed in the Perl version and the Ruby version at Sisimai 4.18.1.
dump() and make() methods of Sisimai class read bounce mail data from Standard-In when a file handle STDIN is specified as the first argument.
% cat ./path/to/email | perl -MSisimai -lE 'print Sisimai->dump(STDIN)' | jq [ { "listid": "", "smtpagent": "Sendmail", ...
% cat ./path/to/email | perl -MSisimai -lE 'print $_->recipient->address for @{ Sisimai->make(STDIN) }' kijitora@example.jp sironeko@example.org
Similarly, In Ruby version of Sisimai, dump() and make() methods of Sisimai class read bounce mail data from Standard-In when a STDIN object is specified as the first argument.
% cat ./path/to/email | ruby -Ilib -rsisimai -e 'puts Sisimai.dump(STDIN)' | jq [ { "catch": "", "token": "d295b517a3ad6f8748031e1068e07ebd089c0277", ...
% cat ./path/to/email | ruby -rsisimai -e 'p Sisimai.make(STDIN)' [#<Sisimai::Data:0x007fa1b0993050 @addresser=#<Sisimai::Address:0x007fa1b0990e40 @user="kijitora", ...
Beginning with Sisimai 4.23.0, make() and dump() methods accept a variable (keeping entire bounce message) as the first argument. The variable as the first argument should be a scalar reference in Perl, a String object in Ruby.
This feature is useful to read bounce messages stored in a database or job queue system.
When a scalar reference which is holding entire bounce message is specified as the 1st argument of make() and dump() methods, Sisimai calls Sisimai::Mail::Memory class and return the parsed results.
#! /usr/bin/env perl
use Sisimai;
my $f = 'From MAILER-DAEMON Thu Dec 22 17:54:04 2015...'; # Entire bounce message
my $v = Sisimai->make(\$f);
for my $e ( @$v ) {
print $e->action; # delayed
print $e->deliverystatus; # 4.7.0
print $e->reason; # expired
print $e->diagnosticcode; # Envelope expired
print $e->softbounce; # 1
}
Similarly, In Ruby version of Sisimai, When a String object which is holding entire bounce message is specified as the 1st argument of make() and dump() methods, Sisimai calls Sisimai::Mail::Memory class and return the parsed results.
#! /usr/bin/env ruby
require 'sisimai'
f = 'From MAILER-DAEMON Thu Dec 22 17:54:04 2015...' # Entire bounce message
v = Sisimai.make(f)
v.each do |e|
puts e.action # delayed
puts e.deliverystatus # 4.7.0
puts e.reason # expired
puts e.diagnosticcode # Envelope expired
puts e.softbounce # 1
end
make() and dump() methods return the parsed results including a delivery success (delivery notification) when delivered option is specified as the 2nd argument.
An email as a sample that has delivered successfully is available as
rfc3464-28.eml
at
github.com/sisimai/set-of-emails
repository.
The following code shows a sample for parsing the email.
When delivered => 1 is specified as the 2nd argument of make() and dump() methods, an email that has delivered successfully is included in the parsed results. If the value of delivered is 0, "" or undef, Sisimai does not generate the parsed results from the succeeded email.
#! /usr/bin/env perl
use Sisimai;
my $f = './set-of-emails/maildir/bsd/rfc3464-28.eml';
my $v = Sisimai->make($f, 'delivered' => 1);
for my $e ( @$v ) {
print $e->action; # deliverable
print $e->deliverystatus; # 2.1.5
print $e->reason; # delivered
print $e->diagnosticcode; # 250 2.1.5 Ok
print $e->softbounce; # -1
}
Similarly, In Ruby version of Sisimai, when delivered: true is specified as the 2nd argument of make() and dump() methods, an email that has delivered successfully is included in the parsed results. If the value of delivered is nil or false, Sisimai does not generate the parsed results from the succeeded email.
#! /usr/bin/env ruby
require 'sisimai'
f = './set-of-emails/maildir/bsd/rfc3464-28.eml'
v = Sisimai.make(f, delivered: true)
v.each do |e|
puts e.action # deliverable
puts e.deliverystatus # 2.1.5
puts e.reason # delivered
puts e.diagnosticcode # 250 2.1.5 Ok
puts e.softbounce # -1
end
Beginning with Sisimai 4.19.0, Callback feature has implemented. This feature allows you to read the header part and the body part of each bounce mail before being parsed by Sisimai, and receive the results returned from a hook method you specified via catch() method of Sisimai::Data or Sisimai::Message object.
It is useful to get entire text of the original message, or the value of a certain header of the original message such as a delivery ID, X-Mailer, and so on.
A hook method in Perl (sub routine) receives a Hash reference, a hook method in Ruby receives a Hash object. Hash reference and object have the following structure.
Key Name | Data Type | Description | JSON | |
---|---|---|---|---|
headers | Hash | Header part of a bounce mail (as a Hash Data) | Available | undef, nil |
message | String | Body part of a bounce mail including the original message | Available | undef, nil |
The following code shows an example for getting the following values from lhost-postfix-16.eml using a hook method given as the 2nd argument of make() method of Sisimai.
Return-Path: <> Received: by 192.0.2.22 with SMTP id fffffff000; Thu, 20 Jan 2011 23:34:45 +0900 (JST) Received: from mx0.example.jp (mx0.example.jp [192.0.2.9]) by mx.example.org with ESMTP id 0000000000000000000000; Thu, 20 Jan 2011 23:34:45 +0900 (JST) Received-SPF: pass (example.org: best guess record for domain of mx0.example.jp designates 192.0.2.9 as permitted sender) client-ip=192.0.2.9; Received: by mx0.example.jp (Postfix) id FFFFFFFFFFFF; Thu, 20 Jan 2011 23:34:45 +0900 (JST) Date: Thu, 20 Jan 2011 23:34:45 +0900 (JST) From: MAILER-DAEMON@example.jp (Mail Delivery System) Subject: Undelivered Mail Returned to Sender To: shironeko@cat.example.com Auto-Submitted: auto-replied MIME-Version: 1.0 Content-Type: multipart/report; report-type=delivery-status; boundary="FFFFFFFFFF.00000000/mx0.example.jp" Content-Transfer-Encoding: 8bit Message-Id: <000000000000000.FFFFFFFFFFFF@mx0.example.jp> This is a MIME-encapsulated message. --FFFFFFFFFF.00000000/mx0.example.jp Content-Description: Notification Content-Type: text/plain; charset=us-ascii This is the mail system at host mx0.example.jp. I'm sorry to have to inform you that your message could not be delivered to one or more recipients. It's attached below. For further assistance, please send mail to <postmaster> If you do so, please include this problem report. You can delete your own text from the attached returned message. The mail system <kijitora@example.go.jp>: host mx1.example.go.jp[192.0.2.127] said: 550 5.1.6 recipient no longer on server: kijitora@example.go.jp (in reply to RCPT TO command) --FFFFFFFFFF.00000000/mx0.example.jp Content-Description: Delivery report Content-Type: message/delivery-status Reporting-MTA: dns; mx0.example.jp X-Postfix-Queue-ID: FFFFFF000000 X-Postfix-Sender: rfc822; shironeko@cat.example.com ...
The following is an example of an argument that the hook method you defined receives. The argument have the header part (Hash) and the body part (String) generated from lhost-postfix-16.eml .
{ 'headers' => { 'x-mailer' => undef, 'return-path' => '<>', 'content-type' => 'multipart/report; report-type=delivery-status; boundary="FFFFFFFFFF.00000000/mx0.example.jp"', 'subject' => 'Undelivered Mail Returned to Sender', 'date' => 'Thu, 20 Jan 2011 23:34:45 +0900 (JST)', 'to' => 'shironeko@cat.example.com', 'received-spf' => 'pass (example.org: best guess record for domain of mx0.example.jp designates 192.0.2.9 as permitted sender) client-ip=192.0.2.9;', 'reply-to' => undef, 'message-id' => '<000000000000000.FFFFFFFFFFFF@mx0.example.jp>', 'from' => 'MAILER-DAEMON@example.jp (Mail Delivery System)', 'content-transfer-encoding' => '8bit', 'received' => [ 'by 192.0.2.22 with SMTP id fffffff000; Thu, 20 Jan 2011 23:34:45 +0900 (JST)', 'from mx0.example.jp (mx0.example.jp [192.0.2.9]) by mx.example.org with ESMTP id 0000000000000000000000; Thu, 20 Jan 2011 23:34:45 +0900 (JST)', 'by mx0.example.jp (Postfix) id FFFFFFFFFFFF; Thu, 20 Jan 2011 23:34:45 +0900 (JST)' ], 'auto-submitted' => 'auto-replied' }, 'message' => 'This is a MIME-encapsulated message. ... Reporting-MTA: dns; mx0.example.jp X-Postfix-Queue-ID: FFFFFF000000 X-Postfix-Sender: rfc822; shironeko@cat.example.com Arrival-Date: Thu, 20 Jan 2011 23:34:45 +0900 (JST) Final-Recipient: rfc822; kijitora@example.go.jp Action: failed Status: 5.1.6 ... Subject: Nyaaaaan To: KijitoraMessage-Id: <0000000000000.FFFFFF000000@mx0.example.jp> Nyaaaan ' }
make() and dump() methods of Sisimai class receive a code reference for being executed as the 2nd argument hook like the following code:
#! /usr/bin/env perl use Sisimai; my $f = './set-of-emails/maildir/bsd/lhost-sendmail-24.eml'; my $x = sub { my $argv = shift; my $data = { 'queue-id' => '', 'received-spf' => '' }; $data->{'received-spf'} = $argv->{'headers'}->{'received-spf'} || ''; if( $argv->{'message'} =~ m/^X-Postfix-Queue-ID:\s*(.+)$/m ) { $data->{'queue-id'} = $1; } return $data; }; my $j = Sisimai->dump($f, 'hook' => $x); # JSON文字列を得る my $v = Sisimai->make($f, 'hook' => $x); print $v->[0]->{'catch'}->{'received-spf'}; # pass (example.org:... print $v->[0]->{'catch'}->{'queue-id'}; # FFFFFF000000
make() and dump() methods of Sisimai class receive a Proc object for being executed as the 2nd argument hook: like the following code:
#! /usr/bin/env ruby
require 'sisimai'
f = './set-of-emails/maildir/bsd/lhost-postfix-16.eml'
x = lambda do |argv|
data = { 'queue-id' => '', 'received-spf' => '' }
data['received-spf'] = argv['headers']['received-spf'] || ''
if cv = argv['message'].match(/^X-Postfix-Queue-ID:\s*(.+)$/)
data['queue-id'] = cv[1]
end
return data
end
j = Sisimai.dump(f, hook: x) # JSON文字列を得る
v = Sisimai.make(f, hook: x)
puts v[0]['catch']['received-spf'] # pass (example.org: ...
puts v[0]['catch']['queue-id'] # FFFFFF000000
JSON string generated by dump() method of Sisimai with a hook method is like the following:
[ { "timestamp": 1270043983, "smtpcommand": "RCPT", "subject": "Nyaaan", "recipient": "kijitora@example.jp", "destination": "example.jp", "senderdomain": "example.co.jp", "token": "bb2a5dfb6161f396300b24acd9bf637c1c45a58a", "smtpagent": "qmail", "replycode": "450", "softbounce": 1, "catch": { "received-spf": "pass (example.org: best guess record for domain of mx0.example.jp designates 192.0.2.9 as permitted sender) client-ip=192.0.2.9;", "queue-id": "FFFFFF000000", }, "rhost": "192.0.2.135", "timezoneoffset": "+0900", "addresser": "shironeko@example.co.jp", "diagnostictype": "SMTP", "action": "failed", "listid": "", "feedbacktype": "", "diagnosticcode": "192.0.2.135 does not like recipient. Remote host said: 450 4.2.2... Mailbox Full Giving up on 192.0.2.135. I'm not going to try again; this message has been in the queue too long.", "reason": "mailboxfull", "deliverystatus": "4.2.2", "alias": "", "lhost": "", "messageid": "57D03121-12F7-4801-B30F-9E44B15E56DC@example.co.jp" "origin": "/var/spool/bounce/1.eml" } ]