This section introduces the simplest and most basic way to use Sisimai. Simply prepare a bounce email in an environment where Sisimai can be used, and you can immediately obtain the decoded results, such as email decoding, structured data, and JSON string acquisition.
This is an example of how to read bounce email data from standard input (STDIN) and decode it with Sisimai. This is a convenient input method when reading data using pipelines in one-liners of Perl or Ruby.
This is an example of how to read bounce email data from a variable and it with Sisimai. This is a convenient method when reading and decoding bounce emails from databases or job queue systems.
By default, Sisimai only outputs the decoded results of emails that could not be delivered. However, this article introduces how to use the method of the Sisimai class to include emails that were delivered successfully (delivery notifications) in the decoded results.
By default, Sisimai only outputs the decode results of emails that could not be delivered. However, this article introduces how to use the method of the Sisimai class to include emails that are returned with an out-of-office (vacation) response in the decoded results.
The callback feature allows you to perform custom processing on the data before it is decoded by Sisimai. This includes retrieving specific headers, partially rewriting the data, and including the results in the decoded data.
The make() method of the Sisimai class has been deprecated in Sisimai 5.0.0. Please use the newly implemented rise() method instead.
The rise() and dump() methods of Sisimai class can read bounce email data from standard input (STDIN) in addition to specifying a filename as an argument. This feature was implemented in early versions of Sisimai, but there was a bug that sometimes prevented it from working properly. This bug has been fixed in Sisimai 4.18.1 Perl version and the Ruby version.
The dump() and rise() methods of the
Sisimai class read bounce email data from standard input and return
the decoded results when the first argument is STDIN as the file handle.
% cat ./path/to/email | perl -MSisimai -lE 'print Sisimai->dump(STDIN)' | jq [ { "listid": "", "smtpagent": "Sendmail", ...
% cat ./path/to/email | perl -MSisimai -lE 'print $_->recipient->address for @{ Sisimai->rise(STDIN) }' kijitora@example.jp sironeko@example.org
The Ruby version of Sisimai also works similarly. When the first
argument of the dump() and rise() methods of the
Sisimai class is the STDIN object, it reads bounce email data from
standard input and returns the decoded results.
% cat ./path/to/email | ruby -Ilib -rsisimai -e 'puts Sisimai.dump(STDIN)' | jq [ { "catch": "", "token": "d295b517a3ad6f8748031e1068e07ebd089c0277", ...
% cat ./path/to/email | ruby -rsisimai -e 'p Sisimai.rise(STDIN)' [#<Sisimai::Fact:0x007fa1b0993050 @addresser=#<Sisimai::Address:0x007fa1b0990e40 @user="kijitora", ...
The Dump() and Rise() functions of the
libsisimai.org/sisimai package read bounce email data from standard
input and return the decoded results when the first argument is STDIN as a string.
package main
import "fmt"
import "libsisimai.org/sisimai"
func main() {
args := sisimai.Args()
json, _ := sisimai.Dump("STDIN", args)
if json != nil && *json != "" { fmt.Printf("%s\n", *json) }
}
% cat ./path/to/email | go run ./sisid.go | jq [ { "catch": "", "token": "d295b517a3ad6f8748031e1068e07ebd089c0277", ...
The make() method of the Sisimai class has been deprecated in Sisimai 5.0.0. Please use the newly implemented rise() method instead.
The rise() and dump() methods of the Sisimai class allow you to read bounce email data from sources other than files and standard input by specifying a variable (a scalar reference in Perl and a String object in Ruby) as the first argument. This feature (Sisimai::Mail::Memory), which is useful for reading bounce emails from databases and job queue systems, was implemented in Sisimai 4.23.0.
When a scalar reference is specified as the first argument to rise()
method and dump() method, Sisimai::Mail::Memory
is called internally, and the decoded results are obtained in the same way as when reading
from a mail file or standard input.
#! /usr/bin/env perl
use Sisimai;
my $f = 'From MAILER-DAEMON Thu Dec 22 17:54:04 2015...'; # Entire bounce message
my $v = Sisimai->rise(\$f);
for my $e ( @$v ) {
print $e->action; # "delayed"
print $e->deliverystatus; # "4.7.0"
print $e->reason; # "expired"
print $e->diagnosticcode; # "Envelope expired"
print $e->hardbounce; # 0
}
In the Ruby version of Sisimai, when a String
object is specified as the first argument to rise()
method and dump() method, Sisimai::Mail::Memory
is called internally, and the decoded results are obtained in the same way as when reading from a mail file or standard
input.
#! /usr/bin/env ruby
require 'sisimai'
f = 'From MAILER-DAEMON Thu Dec 22 17:54:04 2015...' # Entire bounce message
v = Sisimai.rise(f)
v.each do |e|
puts e.action # "delayed"
puts e.deliverystatus # "4.7.0"
puts e.reason # "expired"
puts e.diagnosticcode # "Envelope expired"
puts e.hardbounce # false
end
If you specify a string containing the entire bounce email as the first argument to the
Rise() and Dump() functions,
Sisimai internally calls libisismai.org/sisimai/mail/memory,
and the decoded results are obtained in the same way as when reading from a mail file or standard input.
package main
import "fmt"
import "libsisimai.org/sisimai"
func main() {
mail := 'From MAILER-DAEMON Fri Feb 2 18:30:22 2018...' // Entire bounce message
args := sisimai.Args()
data, _ := sisimai.Rise(mail, args)
if len(*data) > 0 { fmt.Printf("%s\n", data[0].Recipient.Address) }
}
The make() method of the Sisimai class has been deprecated in Sisimai 5.0.0. Please use the newly implemented rise() method instead.
The rise() and dump() methods of the Sisimai class can include emails that notify you of successful delivery by specifying delivered as the second argument.
Emails that were successfully delivered are available as samples in the
github.com/sisimai/set-of-emails
repository. Specifically, the
rfc3464-28.eml
can be used as an example.
This section provides sample code for decoding this email.
The second argument of rise() and dump()
takes a delivered key-value pair. When set to 1, the decoded results will include emails
that were successfully delivered (delivery notifications). If delivered
is set to a false value (0, undef, or an empty string),
the decoded results will not be generated.
#! /usr/bin/env perl
use Sisimai;
my $f = './set-of-emails/maildir/bsd/rfc3464-28.eml';
my $v = Sisimai->rise($f, 'delivered' => 1);
for my $e ( @$v ) {
print $e->action; # "delivered"
print $e->deliverystatus; # "2.1.5"
print $e->reason; # "delivered"
print $e->diagnosticcode; # "250 2.1.5 Ok"
print $e->hardbounce; # 0
}
Similarly to the Perl version, in the Ruby version of Sisimai,
specifying delivered: true as the second argument of rise()
and dump() will include emails that were successfully delivered
(delivery notifications) in the decoded results. If a false value (nil
or false) is specified for delivered, the decoded results will not
be generated.
#! /usr/bin/env ruby
require 'sisimai'
f = './set-of-emails/maildir/bsd/rfc3464-28.eml'
v = Sisimai.rise(f, delivered: true)
v.each do |e|
puts e.action # "delivered"
puts e.deliverystatus # "2.1.5"
puts e.reason # "delivered"
puts e.diagnosticcode # "250 2.1.5 Ok"
puts e.hardbounce # false
end
In the Go version of Sisimai, as with the Perl and Ruby versions,
if you specify true for Delivered in the
sis.DecodingArgs
struct, which is specified as the
second argument to Rise() and Dump(),
the decoded results will include successfully delivered emails (delivery notifications).
The initial value of Delivered is false.
package main
import "fmt"
import "libsisimai.org/sisimai"
func main() {
path := "./set-of-emails/maildir/bsd/rfc3464-28.eml"
args := sisimai.Args()
args.Delivered = true
sisi, _ := sisimai.Rise(path, args)
if len(*sisi) > 0 { fmt.Printf("Recipient = %s\n", *sisi[0].Recipient.Address }
}
The make() method of the Sisimai class has been deprecated in Sisimai 5.0.0. Please use the newly implemented rise() method instead.
Starting with Sisimai 5.0.0, bounce emails with a bounce reason of "vacation" will no longer be decoded by default. If you need this functionality, you must specify the vacation parameter to the rise() method.
The rise() and dump() methods of the Sisimai class can include vacation response emails in the decoded results by specifying vacation as the second argument.
A sample of a vacation response email is available in the
rfc3834-01.eml
in the
github.com/sisimai/set-of-emails
repository. Here is a sample code that decodes this email.
Specifying vacation => 1 as the second argument of rise()
and dump() will include vacation response emails in the decoded results.
If a value that evaluates to false for vacation is specified
(0, undef, ''), no decoded results will be generated.
#! /usr/bin/env perl
use Sisimai;
my $f = './set-of-emails/maildir/bsd/rfc3834-01.eml';
my $v = Sisimai->rise($f, 'vacation' => 1);
for my $e ( @$v ) {
print $e->action; # ""
print $e->deliverystatus; # ""
print $e->reason; # "vacation"
print $e->diagnosticcode; # "I am currently away returning to the office on May 5th."
print $e->hardbounce; # 0
}
Similar to the Perl version, in the Ruby version of Sisimai,
specifying vacation: true as the second argument of rise()
and dump() will include vacation notification emails in the decoded
results. If a value that evaluates to false for vacation is specified
(nil, false), no decoded results will be generated.
#! /usr/bin/env ruby
require 'sisimai'
f = './set-of-emails/maildir/bsd/rfc3834-01.eml'
v = Sisimai.rise(f, vacation: true)
v.each do |e|
puts e.action # ""
puts e.deliverystatus # ""
puts e.reason # "vacation"
puts e.diagnosticcode # "I am currently away returning to the office on May 5th."
puts e.hardbounce # false
end
In the Go version of Sisimai, as with the Perl and Ruby versions,
if you specify true for Vacation in the
sis.DecodingArgs
struct, which is specified as the
second argument to the Rise() and Dump()
functions, the decoded results will include vacation reply emails.
The initial value of Vacation is false.
package main
import "fmt"
import "libsisimai.org/sisimai"
func main() {
path := "./set-of-emails/maildir/bsd/rfc3834-01.eml"
args := sisimai.Args()
args.Vacation = true
sisi, _ := sisimai.Rise(path, args)
if len(*sisi) > 0 { fmt.Printf("Recipient = %s\n", *sisi[0].Recipient.Address }
}
The make() method of the Sisimai class has been deprecated in Sisimai 5.0.0. Please use the newly implemented rise() method instead.
The callback feature specification (number and type of arguments) has changed in Sisimai 5.0.0. Due to the incompatible change, please check the GitHub/Callback Feature of the 4-stable branch for the callback feature in Version 4.
The callback feature implemented in Sisimai 4.19.0 allows you to get the results of your own bounce email processing through the catch() method of Sisimai::Fact class or Sisimai::Message class. This feautre is useful for extracting the entire original message that Sisimai does not include in the decoded results, or specific header values of the original message (such as the delivery ID).
The arguments that can be passed to the c___ (c
and three underscores, looks like a fishing hook) in the rise() method of the
Sisimai class are one array reference for Perl hook methods (subroutines)
and one Array object for Ruby hook methods. The structure of the
arguments is as follows:
The indices of the array passed to c___ | Target of processing |
---|---|
[0] ($code0in the following code example) | Email headers headers and message text message |
[1] ($code1in the following code example) | The original email file |
my $code0 = sub { my $args = shift; my $head = $args->{'headers'}; # Email headers my $body = $args->{'message'}; # Message body }; my $code1 = sub { my $args = shift; my $kind = $args->{'kind'}; # (String) Sisimai::Mail->kind my $mail = $args->{'mail'}; # (*String) Entire email message my $path = $args->{'path'}; # (String) Sisimai::Mail->path my $fact = $args->{'fact'}; # (*Array) List of Sisimai::Fact }; my $list = Sisimai->rise('/path/to/mbox', 'c___' => [$code0, $code1]);
The string obtained from the message is the body
of the bounce email. It does not include the headers of the bounce email itself,
so please refer to the hashed headers.
Key Name | Data Type | Description |
---|---|---|
headers | Hash | Header part of a bounce mail (as a Hash Data) |
message | String | Body part of a bounce mail(including no headers of the bounce mail) |
The string obtained from the mail is the entire
bounce email, including the headers of the bounce email itself.
Key Name | Type | Description |
---|---|---|
kind | String | The kind of the original email file |
*String | Entire message of the bounce mail including all headers | |
path | String | The path to the original email file |
fact | Array | The List of Sisimai::Factobjects |
The arguments and return values of Callback0 and
Callback1 in the
sis.DecodingArgs
struct in the libsisimai.org/sisimai package's
Rise() function have the following structure:
sis.DecodingArgs | Argument | Return values |
---|---|---|
Callback0 |
*sis.CallbackArg0
|
|
Callback1 |
*sis.CallbackArg1
|
|
args := sisimai.Args() args.Callback0 = func(arg *sisimai.CallbackArg0) (map[string]interface{}, error) { fmt.Printf("Headers = %v\n", (*arg).Headers) // All the headers fmt.Printf("- From: = %s\n", (*arg).Headers["from"][0]) fmt.Printf("- Subject: = %s\n", (*arg).Headers["subject"][0]) fmt.Printf("Body = %s\n", (*arg).Payload) // Message body data := make(map[string]interface{}) data["from"] = (*arg).Headers["from"][0] return data, nil } args.Callback1 = func(arg *sisimai.CallbackArg1) (bool, error) { fmt.Printf("- Path = %s\n", arg.Path) // Path to the bounce mail file fmt.Printf("- Kind = %s\n", arg.Kind) // Kind of the email fmt.Printf("Body = %s\n", (*arg.Mail)) // Message body fmt.Printf("Body = %s\n", (*arg.Fact)) // Decoded results return true, nil } sisi, nyaan := sisimai.Rise(path, args)
The following code snippet demonstrates how to operate the original email by obtaining the contents listed below from rhost-google-03.eml via a callback.
Return-Path: <> Received: from mail4.example.com (mx4.example.com [192.0.2.24]) by secure.example.jp (Postfix) with ESMTPS id 4SnNbh0r3mz24sXs for; Thu, 9 Apr 2022 23:34:45 +0900 (JST) Received: by mail4.example.com (Postfix) id t8WLKlmm99zqL71C; Thu, 9 Apr 2022 23:34:45 +0900 (JST) Authentication-Results: secure.example.jp; spf=fail smtp.helo=mail4.example.com Date: Thu, 9 Apr 2022 23:34:45 +0900 (JST) From: MAILER-DAEMON@mail4.example.com (Mail Delivery System) Subject: Undelivered Mail Returned to Sender To: nginx@mail4.example.com ... (expanded from ): host gmail-smtp-in.l.google.com[64.233.188.26] said: 550-5.7.26 The MAIL FROM domain [mail4.example.com] has an SPF record with a 550-5.7.26 hard fail policy (-all) but it fails to pass SPF checks with the ip: 550-5.7.26 [192.0.2.24]. To best protect our users from spam and phishing, 550-5.7.26 the message has been blocked. For instructions on setting up 550-5.7.26 authentication, go to 550 5.7.26 https://support.google.com/mail/answer/81126#authentication 2-222222630d46222222b002b2d2neko2cat2222222cat.22 - gsmtp (in reply to end of DATA command) --Lms5yfCw0Vz5CRp3.1647426971/mail4.example.com Content-Description: Delivery report Content-Type: message/delivery-status Reporting-MTA: dns; mail4.example.com X-Postfix-Queue-ID: Lms5yfCw0Vz5CRp3 X-Postfix-Sender: rfc822; nginx@mail4.example.com Arrival-Date: Thu, 9 Apr 2022 23:34:45 +0900 (JST) Final-Recipient: rfc822; kijitora@google.example.com Original-Recipient: rfc822;neko@example.co.jp Action: failed Status: 5.7.26 Remote-MTA: dns; gmail-smtp-in.l.google.com Diagnostic-Code: smtp; 550-5.7.26 The MAIL FROM domain [mail4.example.com] has ...
You can pass two code references as the second argument c___
to the rise() and dump() methods
of the Sisimai class.
These code references should contain the code for the subroutine that you want
to execute.
% cat set-of-emails/maildir/bsd/rhost-google-03.eml | perl ./sisid.pl
#! /usr/bin/env perl
# sisid.pl
use Sisimai;
my $c0 = sub {
my $args = shift;
my $data = { 'queue-id' => '', 'from' => '', 'return-path' => '' };
$data->{'return-path'} = $args->{'headers'}->{'return-path'} || '';
$data->{'from'} = $args->{'headers'}->{'from'} || '';
if( $args->{'message'} =~ m/^X-Postfix-Queue-ID:\s*(.+)$/m ) {
$data->{'queue-id'} = $1;
}
return $data;
};
my $c1 = sub {
my $args = shift;
my $kind = $args->{'kind'}; # (String) Sisimai::Mail->kind
my $mail = $args->{'mail'}; # (*String) Entire email message
my $path = $args->{'path'}; # (String) Sisimai::Mail->path
if $kind eq 'stdin' {
my $f = IO::File->new("/tmp/sisimai-decoded-1.eml", 'w');
print $f $$mail; $f->close;
}
};
my $rv = Sisimai->rise(STDIN, 'c___' => [$c0, $c1]);
printf("From: %s\n", $rv->[0]->{'catch'}->{'from'});
printf("Queue-ID: %s\n", $rv->[0]->{'catch'}->{'queue-id'});
printf("Return-Path: %s\n", $rv->[0]->{'catch'}->{'return-path'});
You can pass a Proc object containing the code for
the function that you want to execute to the rise()
and dump() methods of the Sisimai
class as the second argument c___.
% cat set-of-emails/maildir/bsd/rhost-google-03.eml | ruby ./sisid.rb
#! /usr/bin/env ruby
# sisid.rb
require 'sisimai'
c0 = lambda do |args|
data = { 'queue-id' => '', 'from' => '', 'return-path' => '' }
data['return-path'] = args['headers']['return-path'] || ''
data['from'] = args['headers']['from'] || ''
if cv = args['message'].match(/^X-Postfix-Queue-ID:\s*(.+)$/)
data['queue-id'] = cv[1]
end
return data
end
c1 = lambda do |args|
kind = args['kind'] # (String) Sisimai::Mail.kind
mail = args['mail'] # (*String) Entire email message
path = args['path'] # (String) Sisimai::Mail.path
if kind == 'stdin'
File.open("/tmp/sisimai-decoded-1.eml", "w") do |f|
f.puts(mail)
end
end
end
rv = Sisimai.rise(STDIN, c___: [c0, c1])
printf("From: %s\n", rv[0].catch['from'])
printf("X-Postfix-Queue-ID: %s\n", rv[0].catch['queue-id'])
printf("Return-Path: %s\n", rv[0].catch['return-path'])
You can assign and pass functions containing the processing you want to execute to
Callback0 and Callback1
of the
sis.DecodingArgs
struct, which is passed to the
Rise() and Dump() functions
of the libsisimai.org/sisimai package.
% cat set-of-emails/maildir/bsd/rhost-google-03.eml | go run ./sisid.go
package main
// sisid.go
import "fmt"
import "strings"
import "libsisimai.org/sisimai"
func main() {
args := sisimai.Args() // sis.DecodingArgs{}
list := []string{"Return-Path", "From"}
args.Callback0 = func(arg *sisimai.CallbackArg0) (map[string]interface{}, error) {
data := make(map[string]interface{})
data["queue-id"] = ""
for _, e := range list {
ff := strings.ToLower(e)
if len((*arg).Headers[ff]) == 0 || (*arg).Headers[ff][0] == "" { continue }
data[ff] = (*arg).Headers[ff][0]
}
if arg.Payload != nil && len(*arg.Payload) > 0 {
name := "X-Postfix-Queue-ID"
mesg := *arg.Payload
p0 := strings.Index(mesg, "\n" + name + ": "); if p0 < 0 { return data, nil }
p1 := p0 + len(name) + 3
p2 := strings.Index(mesg[p1:], "\n"); if p2 < 0 { return data, nil }
data["queue-id"] = mesg[p1:p1 + p2]
}
return data, nil
}
args.Callback1 = func(arg *sisimai.CallbackArg1) (bool, error) {
if nyaan := ioutil.WriteFile("/tmp/sisimai-decoded-1.eml", []byte(*arg.Mail), 0600); nyaan != nil {
return false, nyaan
}
return true, nil
}
// sisi is a pointer to []sis.Facti
sisi, nyaan := sisimai.Rise("STDIN", args)
if len(*sisi) > 0 {
list = append(list, "Queue-ID")
for _, e := range *sisi {
// e is a sis.Fact struct
fmt.Printf("Recipient = %s\n", e.Recipient.Address)
re, as := e.Catch.(map[string]interface{})
if as == false { continue }
for _, f := range list {
ff := strings.ToLower(f)
if ca, ok := re[ff].(string); ok {
fmt.Printf("- Catch[%s] = %s\n", f, ca)
}
}
}
}
// nyaan is a pointer to []sis.NotDecoded
if len(*nyaan) > 0 { fmt.Fprintf(os.Stderr, "%v\n", *nyaan) }
}
The JSON string obtained by passing a hook method to the dump() method of the Sisimai class will have the following contents:
[ { "addresser": "kijitora@google.example.com", "messageid": "vDcsF1hr4zzywp69@mail4.example.com", "recipient": "kijitora@google.example.com", "diagnosticcode": "host gmail-smtp-in.l.google.com[64.233.188.26] said: The MAIL FROM domain [mail4.example.com] has an SPF record with a hard fail policy (-all) but it fails to pass SPF checks with the ip: [192.0.2.24]. To best protect our users from spam and phishing, the message has been blocked. For instructions on setting up authentication, go to https://support.google.com/mail/answer/81126#authentication 2-222222630d46222222b002b2d2neko2cat2222222cat.22 - gsmtp (in reply to end of DATA command)", "smtpagent": "Postfix", "destination": "google.example.com", "alias": "neko@example.co.jp", "reason": "authfailure", "lhost": "gmail-smtp-in.l.google.com", "senderdomain": "google.example.com", "token": "fe75b94dd0489b106fa041c460064dd6afa8196b", "subject": "Nyaan", "deliverystatus": "5.7.26", "timezoneoffset": "+0900", "listid": "", "origin": "set-of-emails/maildir/bsd/rhost-google-03.eml", "action": "failed", "timestamp": 1649514885, "hardbounce": 0, "rhost": "gmail-smtp-in.l.google.com", "catch": { "size": 4436, "kind": "Mailbox", "authentication-results": "secure.example.jp; spf=fail smtp.helo=mail4.example.com", "return-path": "<>", "queue-id": "Lms5yfCw0Vz5CRp3" }, "diagnostictype": "SMTP", "replycode": 550, "feedbacktype": "", "smtpcommand": "DATA" } ]