How to upgrade a mailserver with short downtime and no mail loss?

Hi all,

I am looking to understand how a mail server (Postfix and IMAP Server)
which is installed in a VM (KVM) can be upgraded without having a long
downtime or mail loss.

At the moment I see 2 strategies:

1. long downtime

Put VM offline

Cone VM into VM2 for backup only.

Upgrade VM

Test VM

Put VM online again

Result: We have no lost e-mails, but the downtime can be long, if there
are many mails or if there will be some issues to solve during upgrade.

2. short downtimes but mail loss in the IMAP folders

Put VM offline.

Clone VM into VM2.

Put VM online.

Upgrade VM2.

Test VM2.

Put VM offline.

Put VM2 online.

Result: We have only 2 very short offline moments, but the mails handled
by VM during the upgrade phase will get lost in the IMAP folder, because
they do not get written to the IMAP folders of VM2.

Does someone on the list have any suggestions on hoe to solve this
problem? I am surely missing sonthing very easy, because I am sure there
is an easy way to upgrade Postfix and IMAP without long downtimes and
without loosing e-mails in the folders of the upgraded machine.

Suggestions welcome!

Patrick

Hi Patrick,

First: if you have a good setup you probably have a backup MX configured that will hold your mail during downtime. I think a downtime of roughly 24 hours would yield to no loss of messages.

Second: you can snapshot your VM while it's running if your guest is using lvm-hosted disks. You should be able to limit the downtime to a few seconds/minutes IMO.

Third: after you practiced you are welcome to migrate our mail server *biggrin*

Ciao,
Daniele

Does someone on the list have any suggestions on hoe to solve this
problem? I am surely missing sonthing very easy, because I am sure
there
is an easy way to upgrade Postfix and IMAP without long downtimes and
without loosing e-mails in the folders of the upgraded machine.

Hi,

define "long"!

If "long" is an hour or two, just take your MX offline. Good peers will
retry delivering the mail for days... You will not lose (proper) mails
only because your offline a bit.

If "long" is more than four hours (typical), senders will get "Delayed
Mail (still being retried)" messages, though. I guess you would want to
avoid this. In that case see Daniele:

First: if you have a good setup you probably have a backup MX configured that will hold your mail during downtime.

If your setup is not too complex (i.e. just a domain or so, not an
entire hosting service and or complex antispam/virus/signing things),
you might even want to setup a backup MX for the purpose of the
maintenance, it shouldn't be too complex.

Or just get rid of everything and outsource email handling :wink:

Bye,
Chris.

I almost found a volunteer to migrate our mail server, don't do this! :wink:

Daniele

First: if you have a good setup you probably have a backup MX
configured that will hold your mail during downtime. I think a
downtime of roughly 24 hours would yield to no loss of messages.

Also without a backup MX, any decent smtp server should retry for some
time (24-48h or more; there's no guarantee, but there's no guarantee at
all in smtp :slight_smile:

## diff to your proposed 2. solution

stop VM smtpd (eg postfix stop or service postfix stop etc)

check outgoing queue (or you will have duplicate outgoing email
when you fire up the vm clone)

if mailq is empty then
  put vm offline
  clone vm into vm2
  etc
else
  disable smtp automatic startup
  put vm offline
  clone vm into vm2
  restart vm
  restart vm mta/smtp
  start vm2
  clear (no delivery) mailq
  start vm2 mta/smtp
  enable automatic startup for vm2 mta/smtp
  etc
end

>Put VM offline.

>Clone VM into VM2.
>
>Put VM online.
>
>Upgrade VM2.
>
>Test VM2.
>

Stop VM smtpd (eg postfix stop or service postfix stop etc)
Stop VM imapd (eg /etc/init.d/dovesomething stop)
Stop VM2 smtpd (eg postfix stop or service postfix stop etc)
Stop VM2 imapd (eg /etc/init.d/dovesomething stop)

(r)sync IMAP Folders VM->VM2
  -> this is of course easier if you use "virtual mailboxes" (eg.
     a single directory for all (virtual) users maildir, eg.
     vm2(a)root:/home/mailboxes# rsync --delete -avz vm1:/home/mailboxes/ .
  -> handle rsync with care :slight_smile:

(check mailq again, see below)

Halt VM
Change VM2 IP a/o reboot a/o restart smtp/imapd, etc.

ciao,
I.

> First: if you have a good setup you probably have a backup MX
> configured that will hold your mail during downtime. I think a
> downtime of roughly 24 hours would yield to no loss of messages.

Also without a backup MX, any decent smtp server should retry for some
time (24-48h or more; there's no guarantee, but there's no guarantee at
all in smtp :slight_smile:

## diff to your proposed 2. solution

stop VM smtpd (eg postfix stop or service postfix stop etc)

check outgoing queue (or you will have duplicate outgoing email
when you fire up the vm clone)

if mailq is empty then
  put vm offline
  clone vm into vm2
  etc
else
  disable smtp automatic startup
  put vm offline
  clone vm into vm2
  restart vm
  restart vm mta/smtp
  start vm2
  clear (no delivery) mailq
  start vm2 mta/smtp
  enable automatic startup for vm2 mta/smtp
  etc
end

> >Put VM offline.

> >Clone VM into VM2.
> >
> >Put VM online.
> >
> >Upgrade VM2.
> >
> >Test VM2.
> >

Stop VM smtpd (eg postfix stop or service postfix stop etc)
Stop VM imapd (eg /etc/init.d/dovesomething stop)
Stop VM2 smtpd (eg postfix stop or service postfix stop etc)
Stop VM2 imapd (eg /etc/init.d/dovesomething stop)

(r)sync IMAP Folders VM->VM2
  -> this is of course easier if you use "virtual mailboxes" (eg.
     a single directory for all (virtual) users maildir, eg.
     vm2(a)root:/home/mailboxes# rsync --delete -avz vm1:/home/mailboxes/ .
  -> handle rsync with care :slight_smile:

(check mailq again, see below)

Ops, not so easy..

if VM mailq is empty then
  piece of cake
else
  configure VM2 as smart relay for VM
  enable relay to 3rd party domains for VM in VM2 smtpd
  while mailq != empty
    flush mail queue (eg mail -q)
  end
end

Halt VM
Change VM2 IP a/o reboot a/o restart smtp/imapd, etc.

ciao,
I.

SCNR: I love Igor's pseudocode :slight_smile:

attachment.htm (3.33 KB)

what-if
  we will talk like that in daily life?

diego

Hi all,

thank you all for the valuable inputs!

The problem is not only the downtime for external mails coming in -
which shouldn't be a problem with an Backup-MX, but also the problem of
active users in need to send e-mails all the time. In larger
organizations it becomes difficult to coordinate with all users, because
some departments are more and more relying on working e-mails to match
deadlines etc. To schedule a downtime becomes annoying and difficult.
Therefore the idea of being able to upgrade the mail server without
internal users even noticing is a goal.

We followed Igors suggestions and it work very good.

Thank you all and have a nice weekend!

Patrick

Large organizations and a single SMTP outgoing server? Uhm... You can do better than that :wink:

Ciao,
Daniele

What would you suggest?

BTW. There are 2 SMTP servers, but the users are distributed between the 2. That means the user doesn't switch automatically from one SMTP to the other.

Patrick

attachment.htm (1.81 KB)

I'm not suggesting anything, just saying that you can do better :slight_smile:

SMTP outgoing usually just needs to authenticate your users against some directory, you can have lots of VMs doing this in parallel...

Ciao,
Daniele