May 222011
 

In The last article “Postfix Randomizing Outgoing IP Using TCP_TABLE And Perl“, i was writing about my experiment randomizing outbound ip. someone asked if the script can be mimicked as in the iptables statistic module. indeed, when using the previous script, the results obtained will really random. although that is what the script was created.

I found this wiki explaining about Weighted Round-Robin Scheduling.

Supposing that there is a server set S = {S0, S1, …, Sn-1};
W(Si) indicates the weight of Si;
i indicates the server selected last time, and i is initialized with -1;
cw is the current weight in scheduling, and cw is initialized with zero;
max(S) is the maximum weight of all the servers in S;
gcd(S) is the greatest common divisor of all server weights in S;

while (true) {
    i = (i + 1) mod n;
    if (i == 0) {
        cw = cw - gcd(S);
        if (cw <= 0) {
            cw = max(S);
            if (cw == 0)
            return NULL;
        }
    }
    if (W(Si) >= cw)
        return Si;
}

anyway, I do not have much time to implement that algorithm into the script, so I use an existing perl module List:: util:: WeightedRoundRobin

I’ve modified the script a bit. This script is not exactly imitate iptables stastitic module, but if we set the right weight value, we’re going to get interesting patterns and results.

#!/usr/bin/perl -w
# author: Hari Hendaryanto <hari.h -at- csmcom.com>
use strict;
use warnings;
use Sys::Syslog qw(:DEFAULT setlogsock);
use List::Util::WeightedRoundRobin;
use Storable;

# $count variable latest value stored in file.hash for next script execution
# reference http://www.perlmonks.org/?node_id=510202
my $hashfile="/tmp/file.hash";
store {}, $hashfile unless -r $hashfile;

#
# our transports lists, we will define this in master.cf as transport services
# Queued using Weighted Round-Robin Scheduling
#
my $list = [
    {
        name    => 'smtp1:',
        weight  => 4,
    },
    {
        name    => 'smtp2:',
        weight  => 2,
    },
    {
        name    => 'smtp3:',
        weight  => 2,
    },
    {
        name    => 'smtp4:',
        weight  => 2,
    },
    {
        name    => 'smtp5:',
        weight  => 3,
    },
    {
        name    => 'smtp6:',
        weight  => 2,
    },
    {
        name    => 'smtp7:',
        weight  => 1,
    },
    {
        name    => 'smtp8:',
        weight  => 2,
    },
    {
        name    => 'smtp9:',
        weight  => 2,
    },
    {
        name    => 'smtp10:',
        weight  => 2,
    },

  ];

my $WeightedList = List::Util::WeightedRoundRobin->new();
my $weighted_list = $WeightedList->create_weighted_list( $list );

# $maxinqueue max number of queue in smtp list
my $maxinqueue = scalar(@{$weighted_list});

#
# Initalize and open syslog.
#
openlog('postfix/randomizer','pid','mail');
#
# Autoflush standard output.
#
select STDOUT; $|++;

while (<>) {
	chomp;
	my $count;
	my $hash=retrieve($hashfile);

	if (!defined $hash->{"index"})
	{
        	$count = 0;
	} else {
        	$count = $hash->{"index"};
	}

	if ($count >= $maxinqueue)
	{
        	$hash->{"index"} = 0;
        	$count = 0;
	}

	$hash->{"index"}++;
	store $hash, $hashfile;
	my $random_smtp = ${$weighted_list}[$count];
	if (/^get\s(.+)$/i) {
		print "200 $random_smtp\n";
		syslog("info","Using: %s Transport Service", $random_smtp);
		next;
	}
	print "200 smtp:\n";
}

we can set the script to rotate / queueing output in this manner.

smtp1: smtp2: smtp3: smtp4: smtp5: smtp6: smtp7: smtp8: smtp9: smtp10:

Simply by setting all weight with the same value. we can also set different weight to get a different pattern.

    {
        name    => 'smtp1:',
        weight  => 4,
    },
.....
.....
.....

This simple script bellow also performing weighted random choice. This script require List::Util::WeightedChoice module. the result will be random, but one element can be selected more often by setting weight value greater than others.

#!/usr/bin/perl -w
# author: Hari Hendaryanto <hari.h -at- csmcom.com>
use strict;
use warnings;
use Sys::Syslog qw(:DEFAULT setlogsock);
use List::Util::WeightedChoice qw( choose_weighted );

my $smtp_lists =
[
	'smtp1:',
	'smtp2:',
	'smtp3:',
	'smtp4:',
	'smtp5:',
	'smtp6:',
	'smtp7:',
	'smtp8:',
	'smtp9:',
	'smtp10:'
];
my $weights =
[
	50,
	25,
	1,
	30,
	10,
	5,
	25,
	35,
	45,
	15
];

#
# Initalize and open syslog.
#
openlog('postfix/randomizer','pid','mail');
#
# Autoflush standard output.
#
select STDOUT; $|++;

while (<>) {
	chomp;
	my $random_smtp = choose_weighted( $smtp_lists, $weights );
	if (/^get\s(.+)$/i) {
		print "200 $random_smtp\n";
		syslog("info","Using: %s Transport Service", $random_smtp);
		next;
	}
	print "200 smtp:\n";
}

good luck.

Share

  5 Responses to “Postfix Rotating Outgoing IP Using TCP_TABLE And Perl”

Comments (5)
  1. Incoming mail rejecting…. Log is appended here

    Dec 24 07:30:51 host postfix/smtpd[6219]: 989408406FB: client=mail-ve0-f174.google.com[209.85.128.174]
    Dec 24 07:30:51 host postfix/randomizer[6225]: Using: smtp16: Transport Service
    Dec 24 07:30:51 host postfix/randomizer[6225]: Using: smtp17: Transport Service
    Dec 24 07:30:51 host postfix/smtpd[6219]: 989408406FB: reject: RCPT from mail-ve0-f174.google.com[209.85.128.174]: 550 5.1.1 : Recipient address rejected: User unknown in local recipient table; from= to= proto=ESMTP helo=
    Dec 24 07:30:51 host postfix/cleanup[6226]: 989408406FB: message-id=
    Dec 24 07:30:51 host opendkim[2281]: 989408406FB: mail-ve0-f174.google.com [209.85.128.174] not internal
    Dec 24 07:30:51 host opendkim[2281]: 989408406FB: not authenticated
    Dec 24 07:30:51 host opendkim[2281]: 989408406FB: DKIM verification successful
    Dec 24 07:30:51 host opendkim[2281]: 989408406FB: s=20120113 d=gmail.com SSL
    Dec 24 07:30:51 host postfix/qmgr[6216]: 989408406FB: from=, size=2010, nrcpt=2 (queue active)
    Dec 24 07:30:51 host postfix/randomizer[6225]: Using: smtp18: Transport Service
    Dec 24 07:30:51 host postfix/randomizer[6225]: Using: smtp1: Transport Service
    Dec 24 07:30:51 host postfix/smtpd[6219]: disconnect from mail-ve0-f174.google.com[209.85.128.174]
    Dec 24 07:30:51 host postfix-rotate1/smtp[6227]: 989408406FB: to=, relay=none, delay=0.34, delays=0.18/0.01/0.14/0, dsn=5.4.6, status=bounced (mail for amritanews.com loops back to myself)
    Dec 24 07:30:51 host postfix-rotate1/smtp[6228]: 989408406FB: to=, relay=none, delay=0.34, delays=0.18/0.01/0.14/0, dsn=5.4.6, status=bounced (mail for amritanews.com loops back to myself)
    Dec 24 07:30:51 host postfix/cleanup[6226]: E8CF38406FD: message-id=
    Dec 24 07:30:51 host postfix/qmgr[6216]: E8CF38406FD: from=, size=4273, nrcpt=1 (queue active)
    Dec 24 07:30:51 host postfix/randomizer[6225]: Using: smtp2: Transport Service
    Dec 24 07:30:51 host postfix/bounce[6230]: 989408406FB: sender non-delivery notification: E8CF38406FD
    Dec 24 07:30:51 host postfix/qmgr[6216]: 989408406FB: removed
    Dec 24 07:30:52 host postfix-rotate1/smtp[6231]: connect to gmail-smtp-in.l.google.com[2607:f8b0:400d:c02::1a]:25: Network is unreachable
    Dec 24 07:30:52 host postfix-rotate1/smtp[6231]: E8CF38406FD: to=, relay=gmail-smtp-in.l.google.com[173.194.68.26]:25, delay=0.99, delays=0.01/0.01/0.59/0.38, dsn=2.0.0, status=sent (250 2.0.0 OK 1387888253 j7si17727126qab.167 – gsmtp)
    Dec 24 07:30:52 host postfix/qmgr[6216]: E8CF38406FD: removed

  2. First of all congratulation for you work and thank you for sharing.
    I build a test server with few public IPs, used you script and everything works great up to the point where I’m trying to randomize slow transport maps.
    Tried this suggestion http://www.kutukupret.com/2010/12/06/postfix-randomizing-outgoing-ip-using-tcp_table-and-perl/#comment-1026
    but the message matching the regex goes through that specific transport and then to the default IP bypassing the randomizer.

    $hash->{“index”}++;
    store $hash, $hashfile;
    my $random_smtp = ${$weighted_list}[$count];

    if (/^get\s+(.+@+exampledomain+\..{2,6})/) {
    print “200 slowest:\n”;
    next;
    }

    if (/^get\s(.+)$/i) {
    my $rcpt_domain = $1;
    print “200 $random_smtp\n”;
    syslog(“info”,”using transport: service for: “, $random_smtp, $rcpt_domain);
    next;
    }

    Unfortunately your script was my first interaction with perl and it’s difficult for me to make substantial modifications to it.

    Any suggestions?

    Thanks
    Mike

  3. my original pearl script was not involving any weighted feature , if you follow that, that should be works fine.

  4. Hi, I modified your original script, kept the custom transport maps settings in master.cf and settings in main.cf.
    When sending an email I get the same behavior as with weighted script, it’s passed through that transport and then relayed through default IP.
    Should I replicate the random smtp client services in master.cf and add the settings specific to that transport?

    Mike

  5. Do you think it would be possible to fall-back from IPv6 to IPv4 address when receiving smtp server doesn’t support it?

    David

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

(required)

(required)


*