Hello Thunderbird - Goodby Outlook

Summary
(Ad Lagendijk @ Scientist Surbibal Blog) This post is the ultimate, mobile-site-friendly, migration guide from Outlook to Thunderbird. If you want the short version, go immediately to the section To wrap it all up. The author describes in detail the design flaws in Microsoft’s Outlook and how these shortcomings are avoided in Thunderbird. If you need a Windows program to handle a large number of email messages coming from various accounts the open source and free Thunderbird should be your choice. With this guide Outlook will be history.

Requirements

For scientists communicating and connecting to the outside world through email is the most important communication channel. My email messages are also my to-do list. Dealing with my email messages has been a struggle for the last 10 years. I receive typically 30 to 40 messages a day. These numbers are quite common for scientists, I guess. Here are my requirements for a digital solution to my problem:

I want to be able to store my messages in a hierarchical, collapsible tree. My set of messages contains in addition to recent messages, all access details to Internet sites where I have an account, all my software licenses, and all other email messages that contain information that I have to or want to keep. I am quite bad in throwing emails away and still keep 75% of all messages I receive, going back more than 10 years. On a number of occasions – including in a legal question – this storage mania of mine came in quite handy. I want my whole email archive to be arranged and stored in a systematic way, like we do in a filing cabinet, but then with folders in folders in folders etc.. I want to have all of the stored items available and visible within three or four mouse clicks. As these mails contain for instance a number of passwords I do not want to them to be retrievable when I check my email from a public computer, such as present at scientific conferences. Key logging software is too easily installed there.
I want to be able to search all of those messages with a convenient search engine. This engine does not necessarily need to be part of the email client. As a matter of fact Google Desktop did a wonderful job. So good that Google decided to abandon it. To develop a general program searching through email messages is difficult as the various email clients like Outlook, do not store their messages in an ASCII formatless text file. I wrote an article why this is really bad practice.
I want to be able to retrieve any piece of information from my email store when I do not have a LAN or Internet connection.
I want the collection of emails to be up to date and synchronized from wherever in the world I am checking and handling my mail.
I prefer not to go through a browser interface unless I am not behind one of my own computers, as could the case when I am at a conference. Developers of web interfaces are crippled by naff development languages as client-side JavaScript and server-side script languages as PHP. It is a mainly through the use of AJAX technology that smart developers can still make smooth interfaces as Google Calendar. Making smooth web interfaces for email clients is much more difficult than for a web calendar as larger parts of the web-mail pages have to be changed as result of a user click. The web interface of Google’s free web mail client Gmail is poor and substandard for Google. I must admit that Microsoft Outlook‘s web interface, however, is indeed of superb quality. Part of this success of the Redmond giant is due to the use of the hierarchical navigation bar in their web-mail interface.
I would like a solution where I can use a mobile phone app to peek at (not necessarily download) the most recent email messages and send an occasional message or reply

Up to a few months ago I used Outlook, partly dictated by the fact that my various employers used a Microsoft Exchange Server for their mail server, and up to a few years ago there were no acceptable web interfaces for this server.

Why not using Gmail?

Many of my colleagues manage all their email traffic from Gmail. Gmail certainly satisfies my criterion 4 (world-wide synchronization) and criterion 3 (searchable) and obviously satisfies criterion 6 (mobile-phone app) as could be expected from the driving force behind the Android operating system. But it fails dramatically on my most important criterion 1: a collapsible navigation tree. Google will never ever introduce such a navigation three because it violates Google’s philosopy and raison d’être. The search giant has become big with this philosopy: “Dump anything digital everything you got in a structureless container, and our search machine will find it. Google cannot afford a navigation tree.

To bring some structure in the mess Gmail creates, Google implements labels and sub-labels. But these labels are horrible to use, partly due to their flat structure and partly due to the terribly invasive use of JavaScript on the Gmail site. The company Google seems to be addicted to JavaScript and its overuse on every Google site causes chagrin for the users. A case is the unavoidable irritating autocomplete suggestions populating your screen when you are busy typing in a search query in Google’s search engine.

Gmail fails also on requirement 3 (no need of Internet connection). One could make a local copy of emails stored at the server, but as this is most conveniently done with an email client on the local computer this solution begs the question.

Using a local client like Outlook or Thunderbird

For the last ten years I used Outlook as local email client and saved my email messages, as pst files, on a fast USB drive, a drive that I take with me wherever I go (In my case a Western Digital 500 GB 100$ drive). Everyday I have to synchronize my files at most four times: (i) in the morning at work, (ii) at work just before I leave for home, (iii) early in the evening at home and (iv) late at home or early in the morning before I go to work. To be a success this synchronization process requires discipline, but with a simple utility as Beyond Compare 2 (its newer version Beyond Compare 3 is not nearly as good) or Microsoft’s free command-line utility Robocopy (download here , it contains a good help command) this task becomes light. Maintaining this discipline is not difficult for me as I have to synchronize a lot of other stuff as well, like edited scientific papers, grant proposals and presentations.

On all my computers I have a made a data-only partition (drive d:) that is also fully mapped on my USB disk. The synchronization procedure could be totally automated, but I like to see what files are going to be refreshed before I overwrite them.

Whatever email program one uses in this approach one has to carry all synchronized data on a disk wherever one goes. When going abroad a more convenient solution, without bringing the USB disk will be described later here. Also further on in this article I will indicate that using the cloud, like Dropbox, is not a good solution for parking your email archive.

Problems with Outlook as email client

Outlook suffers from the Microsoft philosophy and as a result is built to optimize incompatibility with non-Microsoft products and is built to maximize the hurdle for non-Microsoft developers to produce useful add-ons to Outlook. Writing add-ons for any Microsoft program requires the complexity of Microsoft-only

COM objects. Another way is to write snail’s pace macros with the hopelessly unstructured, outdated Visual Basic language. Both type of solutions make Outlook slower.In the old days an email client as the free Eudora saved its emails in a large number of flat text files. Such a design choice is a pleasure from the point of view of synchronization. A few new emails will only lead to a few new or changed files to be synchronized.

impatient Goodbye Outlook, enter Thunderbird

Surprise, surprise Microsoft has chosen for a different design. Its design choice is a logical consequence of its closed-shop philosophy, but is steadily getting a major inconvenience in the era of the cloud computing. Outlook saves all its emails in large database files, so-called pst files, files of a very complicated structure. Such a file can easily become of gigabytes size. To keep the pst files manageable the user can create his own pst files, with default name “Personal Folder” showing itself in the navigation tree of Outlook and where one can drag mail folders and mail messages. Here comes the weird Microsoft design: if you open Outlook and not even downloads or send any email message all the pst files of that Outlook profile are changed and get a new file modification time. This is a synchronization nightmare: everyday you have to synchronize a few gigabytes. In my case my social life started to suffer from it. I had to synchronize for minutes and minutes per synchronization cycle, while colleagues waiting in my office to take me out for dinner or while my my wife was waiting in the car wondering what was keeping me. Outlook was keeping me.

The pst files of Outlook are so complex and so fragile that they get easily corrupted. Microsoft realizes this shortcoming and supplies a repair utility for pst files scanpst.exe, shipping standard with the Microsoft Office suite. I had to use this repair facility at least once per month as Outlook warned me regularly that a certain pst file was corrupt and could not be used any longer.

There are quite a number of other less important inconveniences with Outlook. It is quite complex to change the view of newly created folders (Change Current View) . It is quite difficult to change defaults, like showing number of items in a folder rather than showing the number of unread items. I wrote a Visual Basic macro to accomplish this. What normal user masters Visual Basic? Another shortcoming is that there is no connection to the favorite folder list and the navigation tree (there is in Thunderbird). Outlook has the annoying habit of not storing the email addresses of people it knows the email address of. It just stores the name of that person. This gives problems when you forward a mail under the condition that the address book is not available on the computer you work or hen you want to unsubscribe and you cannot find out to which of your accounts the mail was sent..

Start-up times of Outlook were very long, getting longer with each newer version and getting longer due to the growth of my pst files.

Problems with Microsoft Exchange Server

Amusing in the battle with the pst files is that Microsoft itself acknowledges that the structure of its pst files makes synchronization a disaster. Even if you and your employer or other provider are Microsoft-abiding customers using on the server side Microsoft Exchange Server and on the client side Outlook. If you really let the Microsoft server interact directly with your computer your whole computer’s bandwidth will be taken by an unstoppable synchronization process: the server is continuously trying to send these gigabyte over your Internet connection. This is not very pleasant when you are in a hotel with your laptop fitted with a slow connection and trying to find on the Internet the address of a garage to repair your car.

Microsoft has found the solution: cache mode. In this mode Outlook tells the server basically to shut up. Outlook stores all the data on your local computer, on a file of the type .ost which of course can also become huge. You better be sure that before you shut down your computer and you want to continue to work somewhere else you switch off cache mode. Otherwise you are in big trouble. One solution is not get it out of cache mode before you shut down but synchronizes the ost file as well. But even so you can have big problems. As Microsoft is quite good in inventing new file structures that easily get corrupted the company also supplies a repair tool for the ost files: scanost.exe. If you are able to master all this you can apply for a vacancy in an IT department.

For the above reasons in this section I always configure Exchange Servers that I have to use in a downgrade mode: that of a the good old POP server (it means just adding your account as a POP server and not as an Exchange Server in the setting up of the account in Outlook. Then and all my mails end up in a pst file. In such a solution there are no ost files. The profile of Outlook is configured to store its pst data files on the USB disk directly. In case I forgot to synchronize there is no harm done as the USB disk always has the newest data. Great, the IT supporting people continuously were warning me against using the Exchange Server in POP mode, but I knew what I did and it always worked perfectly. My point here is that the necessity of using a server in cache mode demonstrates the failure of the pst file model of Microsoft.

Why not everything on server of employer?

A convenient solution could be to keep all your emails on the Exchange Server of your provider. This is a cloud solution avant la lettre. The first problem that arises in such an approach is what do you do with your email messages of other accounts. If you also let them sitting with their server you will end up with the chaos of having to check different mail servers manually. You will have to choose one “master” account and redirect all your mails from the other accounts to the master account. The redirection will work fine as long as you never log in tot he non-master servers to prevent ending up with massive numbers of duplicate mails or the disaster of lost mails.

A serious complication is that “master” mail server providers always limit the storage capacity you are allowed to keep. This constraint might go away in the future. The decisive factor in my case not using this solution is that web interfaces do not allow to creation of personal folders. This means that you end up with one unmanageable fat navigation branch, rather than a rich tree with many roots. And in case of lost emails I must get into contact with the IT department of my employer. Be my guest.

Why not Outlook in the Cloud?

An ideal seems to have one’s email in the cloud. Forced by the competition Microsoft is now offering all its office applications in the cloud, thereby robbing quite a number of its certified partners from its clients. I studied the solution in quite detail. First of all it is not very expensive. On the order of 10$/month for the lite version. Email works through the Outlook web interface, so that is fine. One can create all kinds of personal folders, so that is also fine. The big problem occurs when you decide you want to have some local copy a mail messages. Then the cloud server of Microsoft sends the whole pst file again. At that time is best you ask your colleagues for a coffee break.

Enter Thunderbird and goodbye Outlooks

After all this grieve with Outlook I decided to try Thunderbird. I downloaded and installed. The normal procedure of account details. My first positive experience: Thunderbird allows to store email in a navigation tree. Fantastic.

Thunderbird allows you tell it where to store its emails. This location is called to profile location. As soon as possible you must set this location to a suitable empty directory on your USB disk. Close Thunderbird and find the Thunderbird file profiles.ini. Use search to find it. It is somewhere in c:\Users\your_username\AppData \Roaming\Thunderbird . The content looks like like:

The Path variable tells Thunderird where during start up it can find the email tree. The drive “g:” denotes my USB disk. In your case you must put your location of where you want to store your emails on your USB disk. Because you always want to synchronize the email tree to your desktop, necessary in case of a mishap, you also must make a similar directory on your hard disk. The more the two directies have comparable names the easier the synchronization. In my case their names are identical except for the name of the drive.

What to do with all my email history in the form Outlook pst files that I want to import in Thunderbird. Next task is to import all these pst files from the Outlook profile.

Importing pst files into Thunderbird

Thunderbird has an Outlook import facility. If you have more profiles for Outlook Thunderbird will only import the default profile. The following steps will import all your pst files flawlessly.

Configure Outlook to start up automatically in the default profiles.
Configure Outlook not to send or receive any email. You might even disconnect the Internet.
Run the scanpst utility on all the pst files. You must do this.
Open Outlook and move all files out of the inbox account file into your pst files. As a result all your default pst file (that is the first root in the navigation bar) is empty
Close all pst files except the default account . Open one pst file and close Outlook. The procedure of importing them one by one is safe and in case of error you know exactly what pst is not (fully) imported.
Use Thunderbird to import Outlook pst files. It will import two files, the default file (containing the Inbox) and the other pst files you had opened. You will get an error of your pst file has a very deep tree (7 levels deep or so). If you get this error you have to go back to Outlook and move some items to make the tree less deep.
Thunderbird will now show the two imported files. Rename them or put them in a local folder you like.
Repeat step 5 to 7 for all pst files you want to import in Thunderbird. That is to say you open Outlook, close the pst file you just imported and open the other pst file you need

You could import more pst files simultaneously but it gives easily rise to import errors (you are importing gigabyte files)

Add-ons make Thunderbird the best email client

Thunderbird is open source and has a css-oriented approach to its looks. That is to say on start-up Thunderbird loads its css files. Developers can easily make add-ons and tailor how they look. I found the following add-ons very useful:

Silvermel: this add-on is a skin for Thunderbird and changes the whole look and feel. Superb skin.
Remove Duplicate Messages (Alternate) delete replicates. When your employer uses IMAP servers or your pop servers are are leaving messages on the server you end up with duplicate messages. With this add-on one click removes them from a folder
Extra Folder Columns I always want to know how many items a folder contains. With this add-on this is possible
QuickFolders: this add-on really great. It enables a toolbar with favorite folders, with drag and drag and drop and sub-folder display. Absolute superior to Outlook’s favorites list.

At first I found the looks of Thunderbird a little old-fashioned, but the Silvermel skin changed Thunderbird in a beautifully looking application. Together with all the add-ons described above I come to the conclusion that Thunderbird looks as nice as Outlook, has a much better User Interface. How about the synchronization speed?

Synchronization speed

Thunderbird saves its emails in a large number of files – not a file for each message because that would slow down the program too much, but the fragmentation is good enough to speed up synchronization tremendously. Synchronization of all my emails, again gigabytes, now takes 10 seconds. Ten to twenty times faster than Outlook.

Thunderbird tree of Ad Lagendijk Goodbye Outlook, enter Thunderbird

Checking recent email without your PC.

When you go on a trip you might find it inconvenient to use your USB disk when your are sitting with your laptop somewhere in the back of a conference room. There is an easy solution to this.

Right before you leave home synchronize the USB disk with your laptop.
Edit the profiles.ini file and change the location to the location where you keep the copy of your emails on your hard disk. Thunderbird will now start without using the USB disk
When you return home, the first thing you do is to synchronize by updating the USB disk and change back in the profiles.ini file the location of your email archive to the location on your USB disk.

To simplify the editing of profiles.ini you could put a shortcut on the desktop pointing to the ini file. Even better is to make two batch files (.bat) that do changing of the path variable automatically and then one mouse click is sufficient.

Using your smart phone

For this we need a Gmail account that we we will only use for occasional reading the most recent messages and for sending an occasional emergency reply. As we want to minimize traffic to our Gmail account, you have to configure the “reply-to” address of my Gmail to one of my other accounts. People that reply to your Gmail message will not send that reply to your Gmail account.

We configure all our mail accounts through the web-mail interface of our providers, except our Gmail account, to redirect a copy of all incoming messages on the server to our Gmail account. In case of a Microsoft Exchange Server you better set this redirection filter while running Outlook (so do not uninstall Outlook).

For the real work we use Thunderbird as email client. We supply Thunderbird with all our account data, except my Gmail account. All incoming fetched mails are filtered – using Thunderbird filters – to automatically arrive in one Inbox. We also tell Thunderbird to send a copy of each sent email to our GMail account.

Some emails are necessarily directly sent to a Google email account, for instance Google Calendar remainder emails. On the web interface of Gmail I filter these message to send a copy (redirection is unfortunately not possible) to one of my email accounts. Redirection and forwarding can give rise to infinite loops but Gmail and Thunderbird recognizes these and stop any loop.

Now we end up with a situation that all emails, received or sent, are registered by Thunderbird as well as Gmail.

I activate my Gmail account on my Android phone. Now when I do not have my laptop available, but want to check my recent messages I check my Gmail app. Regularly I have to clean (meaning delete) my Google inbox. I only send emergency emails from there and then always with sending in addition a copy to one of my other accounts (Google does not let you automate this). In this way Thunderbird will always tell me what I did with my phone. If you know you will need some important email messages while you are at a conference, you keep them on your Gmail account.

To wrap it all up

Here I summarize the steps to take to migrate form Outlok to Thunderbird.

If you do not have a Gmail account get one, or get an equivalent one.
Use Thunderbird as email client with a number of useful add-ons and the Chameleon skin. All your mail accounts, except your Gmail account, are included in the accounts managed by Thunderbird. All incoming mails are filtered within Thunderbird to automatically arrive in one Inbox. Of all sent email messages a copy is sent to your Gmail account.
At the web interface of your employer or professional relation you should set filters to redirect any incoming message to your Gmail account.
Some emails are necessarily sent to your Google email account, for instance Google Calendar’s reminder emails. On the web interface of Gmail I filter these message to send a copy (redirection is unfortunately not possible) to one of your other accounts managed by Thunderbird.
You make two batch files of which you set shortcuts on your desktop to switch from a Thunderbird profile on your USB disk (standard situation) to a profile residing locally on the computer. This trick is useful at conferences, but beware when you return home and do not forget to synchronize from the laptop to USB disk and do not forget to switch the profile back to the USB disk.
Activate your Gmail account on your Android phone or iPhone.

If you have both your computer and your USB disk connected, you have everything to manage all of your email business. If there is no Internet connection Thunderbird will wait fetching and sending messages until the connection becomes available. Not having synchronized your emails for a view days is not a problem as everything is run from the disk. But do not wait too long to get them back into sync.

You can check all your recent emails on your mobile phone and even send mails (do not forget to cc. yourself to a non-Gmail account as otherwise Thunderbird will never know you sent that mail. Clean your Gmail Inbox regularly

Log in to post comments
4440 reads