How To Decode Using Sisimai

Basic Usage

This section introduces the simplest and most basic way to use Sisimai. Simply prepare a bounce email in an environment where Sisimai can be used, and you can immediately obtain the decoded results, such as email decoding, structured data, and JSON string acquisition.

Read

From STDIN

This is an example of how to read bounce email data from standard input (STDIN) and decode it with Sisimai. This is a convenient input method when reading data using pipelines in one-liners of Perl or Ruby.

Read

From Variable

This is an example of how to read bounce email data from a variable and it with Sisimai. This is a convenient method when reading and decoding bounce emails from databases or job queue systems.

Read

Include "Delivered"

By default, Sisimai only outputs the decoded results of emails that could not be delivered. However, this article introduces how to use the method of the Sisimai class to include emails that were delivered successfully (delivery notifications) in the decoded results.

Read

Include "Vacation"

By default, Sisimai only outputs the decode results of emails that could not be delivered. However, this article introduces how to use the method of the Sisimai class to include emails that are returned with an out-of-office (vacation) response in the decoded results.

Read

Callback

The callback feature allows you to perform custom processing on the data before it is decoded by Sisimai. This includes retrieving specific headers, partially rewriting the data, and including the results in the decoded data.

Read

Read From STDIN


The make() method of the Sisimai class has been deprecated in Sisimai 5.0.0. Please use the newly implemented rise() method instead.

The rise() and dump() methods of Sisimai class can read bounce email data from standard input (STDIN) in addition to specifying a filename as an argument. This feature was implemented in early versions of Sisimai, but there was a bug that sometimes prevented it from working properly. This bug has been fixed in Sisimai 4.18.1 Perl version and the Ruby version.

Perl

STDIN File Handle

The dump() and rise() methods of the Sisimai class read bounce email data from standard input and return the decoded results when the first argument is STDIN as the file handle.

% cat ./path/to/email | perl -MSisimai -lE 'print Sisimai->dump(STDIN)' | jq
[
  {
    "listid": "",
    "smtpagent": "Sendmail",
    ...
                
% cat ./path/to/email | perl -MSisimai -lE 'print $_->recipient->address for @{ Sisimai->rise(STDIN) }'
kijitora@example.jp
sironeko@example.org
                
Ruby

STDIN Object

The Ruby version of Sisimai also works similarly. When the first argument of the dump() and rise() methods of the Sisimai class is the STDIN object, it reads bounce email data from standard input and returns the decoded results.

% cat ./path/to/email | ruby -Ilib -rsisimai -e 'puts Sisimai.dump(STDIN)' | jq
[
  {
    "catch": "",
    "token": "d295b517a3ad6f8748031e1068e07ebd089c0277",
    ...
                
% cat ./path/to/email | ruby -rsisimai -e 'p Sisimai.rise(STDIN)'
[#<Sisimai::Fact:0x007fa1b0993050 @addresser=#<Sisimai::Address:0x007fa1b0990e40 @user="kijitora", ...
                

Read From Variable

The make() method of the Sisimai class has been deprecated in Sisimai 5.0.0. Please use the newly implemented rise() method instead.

The rise() and dump() methods of the Sisimai class allow you to read bounce email data from sources other than files and standard input by specifying a variable (a scalar reference in Perl and a String object in Ruby) as the first argument. This feature (Sisimai::Mail::Memory), which is useful for reading bounce emails from databases and job queue systems, was implemented in Sisimai 4.23.0.

Perl

Scalar Reference

When a scalar reference is specified as the first argument to rise() method and dump() method, Sisimai::Mail::Memory is called internally, and the decoded results are obtained in the same way as when reading from a mail file or standard input.

#! /usr/bin/env perl
use Sisimai;
my $f = 'From MAILER-DAEMON Thu Dec 22 17:54:04 2015...';   # Entire bounce message
my $v = Sisimai->rise(\$f);
for my $e ( @$v ) {
    print $e->action;           # "delayed"
    print $e->deliverystatus;   # "4.7.0"
    print $e->reason;           # "expired"
    print $e->diagnosticcode;   # "Envelope expired"
    print $e->hardbounce;       # 0
}
                
Ruby

String object

In the Ruby version of Sisimai, when a String object is specified as the first argument to rise() method and dump() method, Sisimai::Mail::Memory is called internally, and the decoded results are obtained in the same way as when reading from a mail file or standard input.

#! /usr/bin/env ruby
require 'sisimai'
f = 'From MAILER-DAEMON Thu Dec 22 17:54:04 2015...'    # Entire bounce message
v = Sisimai.rise(f)
v.each do |e|
  puts e.action         # "delayed"
  puts e.deliverystatus # "4.7.0"
  puts e.reason         # "expired"
  puts e.diagnosticcode # "Envelope expired"
  puts e.hardbounce     # false
end
                

Include Delivery Success

The make() method of the Sisimai class has been deprecated in Sisimai 5.0.0. Please use the newly implemented rise() method instead.

The rise() and dump() methods of the Sisimai class can include emails that notify you of successful delivery by specifying delivered as the second argument.

email

Succeeded Delivery as a Sample

Emails that were successfully delivered are available as samples in the github.com/sisimai/set-of-emails repository. Specifically, the rfc3464-28.eml can be used as an example. This section provides sample code for decoding this email.

Perl

delivered => 1

The second argument of rise() and dump() takes a delivered key-value pair. When set to 1, the decoded results will include emails that were successfully delivered (delivery notifications). If delivered is set to a false value (0, undef, or an empty string), the decoded results will not be generated.

#! /usr/bin/env perl
use Sisimai;
my $f = './set-of-emails/maildir/bsd/rfc3464-28.eml';
my $v = Sisimai->rise($f, 'delivered' => 1);
for my $e ( @$v ) {
    print $e->action;           # "delivered"
    print $e->deliverystatus;   # "2.1.5"
    print $e->reason;           # "delivered"
    print $e->diagnosticcode;   # "250 2.1.5 Ok"
    print $e->hardbounce;       # 0
}
                
Ruby

delivered: true

Similarly to the Perl version, in the Ruby version of Sisimai, specifying delivered: true as the second argument of rise() and dump() will include emails that were successfully delivered (delivery notifications) in the decoded results. If a false value (nil or false) is specified for delivered, the decoded results will not be generated.

#! /usr/bin/env ruby
require 'sisimai'
f = './set-of-emails/maildir/bsd/rfc3464-28.eml'
v = Sisimai.rise(f, delivered: true)
v.each do |e|
  puts e.action         # "delivered"
  puts e.deliverystatus # "2.1.5"
  puts e.reason         # "delivered"
  puts e.diagnosticcode # "250 2.1.5 Ok"
  puts e.hardbounce     # false
end
                

Include Vacation

The make() method of the Sisimai class has been deprecated in Sisimai 5.0.0. Please use the newly implemented rise() method instead.

Starting with Sisimai 5.0.0, bounce emails with a bounce reason of "vacation" will no longer be decoded by default. If you need this functionality, you must specify the vacation parameter to the rise() method.

The rise() and dump() methods of the Sisimai class can include vacation response emails in the decoded results by specifying vacation as the second argument.

email

Vacation email as a sample

A sample of a vacation response email is available in the rfc3834-01.eml in the github.com/sisimai/set-of-emails repository. Here is a sample code that decodes this email.

Perl

vacation => 1

Specifying vacation => 1 as the second argument of rise() and dump() will include vacation response emails in the decoded results. If a value that evaluates to false for vacation is specified (0, undef, ''), no decoded results will be generated.

#! /usr/bin/env perl
use Sisimai;
my $f = './set-of-emails/maildir/bsd/rfc3834-01.eml';
my $v = Sisimai->rise($f, 'vacation' => 1);
for my $e ( @$v ) {
    print $e->action;           # ""
    print $e->deliverystatus;   # ""
    print $e->reason;           # "vacation"
    print $e->diagnosticcode;   # "I am currently away returning to the office on May 5th."
    print $e->hardbounce;       # 0
}
                
Ruby

vacation: true

Similar to the Perl version, in the Ruby version of Sisimai, specifying vacation: true as the second argument of rise() and dump() will include vacation notification emails in the decoded results. If a value that evaluates to false for vacation is specified (nil, false), no decoded results will be generated.

#! /usr/bin/env ruby
require 'sisimai'
f = './set-of-emails/maildir/bsd/rfc3834-01.eml'
v = Sisimai.rise(f, vacation: true)
v.each do |e|
  puts e.action         # ""
  puts e.deliverystatus # ""
  puts e.reason         # "vacation"
  puts e.diagnosticcode # "I am currently away returning to the office on May 5th."
  puts e.hardbounce     # false
end
                

Callback Feature

The make() method of the Sisimai class has been deprecated in Sisimai 5.0.0. Please use the newly implemented rise() method instead.

The callback feature specification (number and type of arguments) has changed in Sisimai 5.0.0. Due to the incompatible change, please check the GitHub/Callback Feature of the 4-stable branch for the callback feature in Version 4.

The callback feature implemented in Sisimai 4.19.0 allows you to get the results of your own bounce email processing through the catch() method of Sisimai::Fact class or Sisimai::Message class. This feautre is useful for extracting the entire original message that Sisimai does not include in the decoded results, or specific header values of the original message (such as the delivery ID).

c___

Arguments passed to c___

The arguments that can be passed to the c___ (c and three underscores, looks like a fishing hook) in the rise() method of the Sisimai class are one array reference for Perl hook methods (subroutines) and one Array object for Ruby hook methods. The structure of the arguments is as follows:

The indices of the array passed to c___ Target of processing
[0] ($code0in the following code example) Email headers headers and message text message
[1] ($code1in the following code example) The original email file
my $code0 = sub {
    my $args = shift;
    my $head = $args->{'headers'}; # Email headers
    my $body = $args->{'message'}; # Message body
};
my $code1 = sub {
    my $args = shift;
    my $kind = $args->{'kind'}; # (String)  Sisimai::Mail->kind
    my $mail = $args->{'mail'}; # (*String) Entire email message
    my $path = $args->{'path'}; # (String)  Sisimai::Mail->path
    my $fact = $args->{'fact'}; # (*Array)  List of Sisimai::Fact
};
my $list = Sisimai->rise('/path/to/mbox', 'c___' => [$code0, $code1]);
                
c___[0]

Arguments inside of c___[0]

The string obtained from the message is the body of the bounce email. It does not include the headers of the bounce email itself, so please refer to the hashed headers.

Key Name Data Type Description
headers Hash Header part of a bounce mail (as a Hash Data)
message String Body part of a bounce mail(including no headers of the bounce mail)
c___[1]

Arguments inside of c___[1]

The string obtained from the mail is the entire bounce email, including the headers of the bounce email itself.

Key Name Type Description
kind String The kind of the original email file
mail *String Entire message of the bounce mail including all headers
path String The path to the original email file
fact Array The List of Sisimai::Factobjects
email

Email: rhost-google-03.eml

The following code snippet demonstrates how to operate the original email by obtaining the contents listed below from rhost-google-03.eml via a callback.

  • Retrieve the value of Return-Path in the bounce email header.
  • Retrieve the value of Authentication-Results in the bounce email header.
  • Retrieve the value of X-Postfix-Queue-ID in the body.
  • Add a custom header to the original email and save it to a different path.
Return-Path: <>
Received: from mail4.example.com (mx4.example.com [192.0.2.24])
	by secure.example.jp (Postfix) with ESMTPS id 4SnNbh0r3mz24sXs
	for ; Thu,  9 Apr 2022 23:34:45 +0900 (JST)
Received: by mail4.example.com (Postfix)
	id t8WLKlmm99zqL71C; Thu,  9 Apr 2022 23:34:45 +0900 (JST)
Authentication-Results: secure.example.jp; spf=fail smtp.helo=mail4.example.com
Date: Thu,  9 Apr 2022 23:34:45 +0900 (JST)
From: MAILER-DAEMON@mail4.example.com (Mail Delivery System)
Subject: Undelivered Mail Returned to Sender
To: nginx@mail4.example.com
...
 (expanded from ): host
    gmail-smtp-in.l.google.com[64.233.188.26] said: 550-5.7.26 The MAIL FROM
    domain [mail4.example.com] has an SPF record with a 550-5.7.26 hard fail
    policy (-all) but it fails to pass SPF checks with the ip: 550-5.7.26
    [192.0.2.24]. To best protect our users from spam and phishing,
    550-5.7.26 the message has been blocked. For instructions on setting up
    550-5.7.26 authentication, go to 550 5.7.26
    https://support.google.com/mail/answer/81126#authentication
    2-222222630d46222222b002b2d2neko2cat2222222cat.22 - gsmtp (in reply to end
    of DATA command)

--Lms5yfCw0Vz5CRp3.1647426971/mail4.example.com
Content-Description: Delivery report
Content-Type: message/delivery-status

Reporting-MTA: dns; mail4.example.com
X-Postfix-Queue-ID: Lms5yfCw0Vz5CRp3
X-Postfix-Sender: rfc822; nginx@mail4.example.com
Arrival-Date: Thu,  9 Apr 2022 23:34:45 +0900 (JST)

Final-Recipient: rfc822; kijitora@google.example.com
Original-Recipient: rfc822;neko@example.co.jp
Action: failed
Status: 5.7.26
Remote-MTA: dns; gmail-smtp-in.l.google.com
Diagnostic-Code: smtp; 550-5.7.26 The MAIL FROM domain [mail4.example.com] has
...
                
Perl

Sisimai->rise($f, 'c___' => [$c0, $c1])

You can pass two code references as the second argument c___ to the rise() and dump() methods of the Sisimai class. These code references should contain the code for the subroutine that you want to execute.

#! /usr/bin/env perl
use Sisimai;
my $fn = 'set-of-emails/maildir/bsd/rhost-google-03.eml';
my $c0 = sub {
    my $args = shift;
    my $data = { 'queue-id' => '', 'authentication-results' => '', 'return-path' => '' };

    $data->{'return-path'} = $args->{'headers'}->{'return-path'} || '';
    $data->{'authentication-results'} = $args->{'headers'}->{'authentication-results'} || '';
    if( $args->{'message'} =~ m/^X-Postfix-Queue-ID:\s*(.+)$/m ) {
        $data->{'queue-id'} = $1;
    }
    return $data;
};
my $c1 = sub {
    my $args = shift;
    my $kind = $args->{'kind'}; # (String)  Sisimai::Mail->kind
    my $mail = $args->{'mail'}; # (*String) Entire email message
    my $path = $args->{'path'}; # (String)  Sisimai::Mail->path
    my $fact = $args->{'fact'}; # (*Array)  List of Sisimai::Fact

    for my $e ( @$fact ) {
        # Store custom information in the "catch" accessor.
        $e->{'catch'} ||= {};
        $e->{'catch'}->{'size'} = length $$mail;
        $e->{'catch'}->{'kind'} = ucfirst $kind;

        # Save the original email with an additional "X-Sisimai-Parsed:" header to a different PATH.
        my $a = sprintf("X-Sisimai-Parsed: %d\n", scalar @$fact);
        my $f = IO::File->new(sprintf("/tmp/sisimai-%s.eml", $e->token), 'w');
        my $v = $$mail; $v =~ s/^(From:.+)$/$a$1/m;
        print $f $v; $f->close;
    }

    # Remove the email file in Maildir/ after decoding
    unlink $path if $kind eq 'maildir';
};

my $js = Sisimai->dump($fn, 'c___' => [$c0, $c1]);
my $rv = Sisimai->rise($fn, 'c___' => [$c0, $c1]);
printf("Authentication-Results: %s\n", $rv->[0]->{'catch'}->{'authentication-results'});
printf("X-Postfix-Queue-ID: %s\n", $rv->[0]->{'catch'}->{'queue-id'});
printf("Return-Path: %s\n", $rv->[0]->{'catch'}->{'return-path'});
                
Ruby

Sisimai.rise(f, c___: [c0, c1])

You can pass a Proc object containing the code for the function that you want to execute to the rise() and dump() methods of the Sisimai class as the second argument c___.

#! /usr/bin/env ruby
require 'sisimai'
fn = 'set-of-emails/maildir/bsd/rhost-google-03.eml'
c0 = lambda do |args|
  data = { 'queue-id' => '', 'authentication-results' => '', 'return-path' => '' }
  data['return-path'] = args['headers']['return-path'] || ''
  data['authentication-results'] = args['headers']['authentication-results'] || ''
  if cv = args['message'].match(/^X-Postfix-Queue-ID:\s*(.+)$/)
    data['queue-id'] = cv[1]
  end
  return data
end

c1 = lambda do |args|
  kind = args['kind'] # (String)  Sisimai::Mail.kind
  mail = args['mail'] # (*String) Entire email message
  path = args['path'] # (String)  Sisimai::Mail.path
  fact = args['fact'] # (*Array)  List of Sisimai::Fact

  fact.each do |e|
    # Store custom information in the "catch" accessor.
    e.catch ||= {}
    e.catch['size'] = mail.size
    e.catch['kind'] = kind

    # Save the original email with an additional "X-Sisimai-Parsed:" header to a different PATH.
    a = sprintf("X-Sisimai-Parsed: %d", fact.size)
    b = mail.sub(/^(To:.+$)/,'\1'+ "\n" + a)
    File.open(sprintf("/tmp/sisimai-%s.eml", e.token), "w") do |f|
      f.puts(b)
    end

    # Remove the email file in Maildir/ after decoding
    File.delete(path) if kind == 'maildir'
  end
end

js = Sisimai.dump(fn, c___: [c0, c1]) # JSON文字列を得る
rv = Sisimai.rise(fn, c___: [c0, c1])
printf("Authentication-Results: %s\n", rv[0].catch['authentication-results'])
printf("X-Postfix-Queue-ID: %s\n", rv[0].catch['queue-id'])
printf("Return-Path: %s\n", rv[0].catch['return-path'])
                
JSON

JSON string by dump()

The JSON string obtained by passing a hook method to the dump() method of the Sisimai class will have the following contents:

[
  {
    "addresser": "kijitora@google.example.com",
    "messageid": "vDcsF1hr4zzywp69@mail4.example.com",
    "recipient": "kijitora@google.example.com",
    "diagnosticcode": "host gmail-smtp-in.l.google.com[64.233.188.26] said: The MAIL FROM domain [mail4.example.com] has an SPF record with a hard fail policy (-all) but it fails to pass SPF checks with the ip: [192.0.2.24]. To best protect our users from spam and phishing, the message has been blocked. For instructions on setting up authentication, go to https://support.google.com/mail/answer/81126#authentication 2-222222630d46222222b002b2d2neko2cat2222222cat.22 - gsmtp (in reply to end of DATA command)",
    "smtpagent": "Postfix",
    "destination": "google.example.com",
    "alias": "neko@example.co.jp",
    "reason": "authfailure",
    "lhost": "gmail-smtp-in.l.google.com",
    "senderdomain": "google.example.com",
    "token": "fe75b94dd0489b106fa041c460064dd6afa8196b",
    "subject": "Nyaan",
    "deliverystatus": "5.7.26",
    "timezoneoffset": "+0900",
    "listid": "",
    "origin": "set-of-emails/maildir/bsd/rhost-google-03.eml",
    "action": "failed",
    "timestamp": 1649514885,
    "hardbounce": 0,
    "rhost": "gmail-smtp-in.l.google.com",
    "catch": {
      "size": 4436,
      "kind": "Mailbox",
      "authentication-results": "secure.example.jp; spf=fail smtp.helo=mail4.example.com",
      "return-path": "<>",
      "queue-id": "Lms5yfCw0Vz5CRp3"
    },
    "diagnostictype": "SMTP",
    "replycode": 550,
    "feedbacktype": "",
    "smtpcommand": "DATA"
  }
]