Getting away with data – Migrating to Kolab Now – Mail Part #2 (migration proper)

By   2015-09-25

Short recap: in the previous installment I described some tips and best practices on preparing your mailboxes for migration.


Using imapsync

Once the source mailbox is freed from all the layers of dust we can begin the migration proper.

I used imapsync ver. 1.607 on openSUSE, Windows users will have some differences and describing them is out of scope of this article.

Please refer to the imapsync FAQ and also check the options for a full set of instructions on how the command line switches work.

The most important parts are:

  • syncinternaldates – this is probably the single most important switch: it makes sure that all time-stamps are preserved when copying the mail. Thanks to this, you can have mail older than the mailbox that hosts it and all dates will still be coherent;
  • ssl1 – don’t transfer your mail over the network without encryption, make sure SSL is on;
  • exclude – you may want to ignore certain folders, like SPAM;
  • skipcrossduplicates – makes sure you won’t have duplicate messages on the destination mailbox. A must have when going from Gmail to Kolab;
  • justfolders – it’s sometimes needed to just create the folders and then copy the mail, see the imapsync mailing list for details;
  • dry – triggers simulation or “dry” mode, no changes will be made for real;

This is one of my scripts:

Command line used:
 /usr/bin/imapsync --host1 --user1 {redacted} --passfile1 /home/{redacted} --host2 --user2 {redacted}@{redacted}.pl --passfile2 /home/{redacted} --syncinternaldates --ssl1 --ssl2 --noauthmd5 --split1 100 --split2 100 --exclude Wszystkie Spam --checkmessageexists --dry --justfolders

One of the best things in imapsync is the logging facility which is on by default. Here’s a piece of one of the logs:

++++ Calculating sizes on Host1
 Host1 folder [INBOX] Size: 2639 Messages: 1 Biggest: 2639
 Host1 folder [Templates] Size: 0 Messages: 0 Biggest: 0
 Host1 folder [[Gmail]] Host1 Folder [Gmail]: Could not select: 66 NO [NONEXISTENT] Unknown Mailbox: [Gmail] (now in authenticated state) (Failure)
 Host1 folder [[Gmail]/Kosz] Size: 1640615 Messages: 30 Biggest: 190601
 Host1 folder [[Gmail]/Oznaczone gwiazdk&AQU-] = [[Gmail]/Oznaczone gwiazdką] Size: 0 Messages: 0 Biggest: 0
 Host1 folder [[Gmail]/Spam] Size: 2618 Messages: 1 Biggest: 2618
 Host1 folder [[Gmail]/Wa&AXw-ne] = [[Gmail]/Ważne] Size: 2639 Messages: 1 Biggest: 2639
 Host1 folder [[Gmail]/Wersje robocze] Size: 0 Messages: 0 Biggest: 0
 Host1 folder [[Gmail]/Wys&AUI-ane] = [[Gmail]/Wysłane] Size: 663 Messages: 1 Biggest: 663
 Host1 folder [migrationSent] Size: 663 Messages: 1 Biggest: 663
 Host1 folder [migrationTest] Size: 0 Messages: 0 Biggest: 0
 Host1 folder [migrationTestEmpty] Size: 0 Messages: 0 Biggest: 0
 Host1 Nb messages: 35 messages
 Host1 Total size: 1649837 bytes (1.573 MiB)
 Host1 Biggest message: 190601 bytes (186.134 KiB)
 Host1 Time spent: 2.9 seconds

Gmail and its gotchas

There are some things worth knowing when migrating from Gmail (if you migrate from one Gmail instance to another, most of them don’t matter).

The most mind boggling thing is Gmails implementation of the IMAP folder store. You see, Gmail doesen’t use “folders”, it has “tags”. In more traditional mail storage systems (like Kolab or Exchange even) folders are “boxes” you put the mail (imagine them as envelopes) into those boxes. With Gmail it’s the other way around: you have a stack of messages and you apply tags like you were working on a scrapbook. Every message exists in Gmail in only one copy and moving it between “folders” or removing it from “folders” is actually manipulating the tags.

Here you can find an explanation that covers more ground.

Migrating to Gmail is therefore rather straightforward: duplicates will most likely get purged and the sole survivors will get their tags (don’t take my word, I never tested this).

On the other hand migrating from Gmail requires you to decide what to do with duplicates. If a message has two tags: cooking and vacation one of the following will happen:

  • The message will get copied during sync and you’ll have two identical messages, one in folder cooking and one in vacation, or;
  • Only one will survive (when you use –skipcrossduplicates);
  • Third option: during the cleanup phase you can chose where to put the message. I deleted so many trash, the message sorting wasn’t that hard. If your post-cleaning mailbox isn’t huge, give it a shot.

Does Gmail have any kind of folders? Surprisingly yes! The spam folder and “all”. The inbox is also a tag, yes, Google went to the extreme with this scheme and maybe that’s why it works. A message has to be in the “ALL” folder with a tag “inbox” or “trash”. If it has no tags, it’s considered “archived”. The only way to kick it out of this realm is to move it to spam (or to trash and then empty the trashcan). This will really matter since things you “archived” in Gmail won’t go to Kolab’s archive, but to a separate folder “all”. At first this sounds counter intuitive, but after a while makes sense: Gmail has its own way of imagining reality and when you copy mail to Kolab you actually translate this state of mind into something else. Much like translating literature. Did you know in Polish we have double negatives that don’t cancel each other out and they’re totally legit?

“Nie trzeba nam żadnej edukacji” means “We don’t need no education”.


Other things to consider:

  • Different language versions have different folder names (like “Kosz” or “Trash”);
  • Watch out for special characters like ‘@’ or diacritic characters.

Pro-tip: before migrating add “migration” to the tags in Gmail. That’s to avoid any collisions with previously created folders. The word “collision” is inaccurate: if imapsync sees an existing folder of the same name it just copies inside. This may be something you want or not.

Word of caution: make sure none of your mail folders in Gmail has the same name like ANY folder in Kolab (this applies also to calendars, address books etc.). imapsync may by accident throw mail into such folders… that’s where the “migration” bit may help. See bug 5021 for more info.

Ready? You wrote the script and tested it? Have back-up? Remove the –dry switch and enjoy your noodles.


One Comment on “Getting away with data – Migrating to Kolab Now – Mail Part #2 (migration proper)

  1. Pingback: Getting away with data – Migrating to Kolab Now – Mail Part #3 (post post-migration) – Fractured Lens

Comments are closed.