KDEPIM sprint #2 (Understanding and diagnosing Akonadi)

In this second part I’m going to explain a few techniques to debug Akonadi we use in order to understand why my KMail installation was working bad.

My configuration
I’m a heavy email user, every day  I receive and read hundreds of them, and I love to have my email synchronized across devices (laptop, workstation, phone, tablet) and particulary in my laptop and workstation I want to have all my email offline so I can make use of it with or without internet, specially useful on my laptop while traveling. Because of all those reasons I have to use something called Disconnect IMAP.

My EMail account
The account has 5.6GB of data and according to GMail it is split in 700K emails (more or less) 200K of those emails are in a folder (label in GMail) called kde-commits with a size of 1.8GB, then it has 5 folders with more than 25K emails and around 200MB of data each, and finally the rest are just small folders with 500-5000 emails. In total there are 70 folders.

First import experience
The first problem I have with KMail2 is the experience out of the first email import. Since my account is so huge it takes a while (around 15 hours) to make the full import and I can’t use the application while doing it because either the data won’t be available or all my CPU power will be used in the task.

To fix this we designed something called “Spanish sync” which basically consist on splitting the task of “Downloading all email while indexing it” into the following steps:

  1. Download all the envelopes (subject,from, to, and some other data) starting from the most recent. Once this is done KMail will be fully functional (full email will be downloaded when user tries to read it).
  2. Download in a controlled way all the email bodies (where the content of the email is). Right now this goes as fast as your internet line allows which can be a problem for people having a fast internet connection (100Mb in my case) and slow machine (my laptop is 6yrs old). Once this finishes the email content will be available offline.
  3. Finally we will index all emails. If the user tries to search before this is done we will show an progress indicator.

The full synchronization of my email is still going to take 15 hours but I will be able to use my application almost since moment one (since it will import most recent emails first).

KMail not showing email body
Even though I set my account as “Disconnected IMAP”, more or less 50% of my emails were not being downloaded so I kept getting this “loading email” message from KMail. This is specially annoying when I’m on my laptop since the internet availability while traveling is not reliable. We have not been able to identify the reason why 1 out 2 emails is not being downloaded but at least I learned how to diagnose this.

Execute the debugging application “akonadiconsole” and go to the “DB console” tab and execute the following SQL query:

SELECT count(*) FROM parttable WHERE name = “PLD:RFC822″ AND data is null;

The result should be a small amount of your total email (in my workstation it is 4K while in my laptop it is 300K). If you set your account as “Disconnected IMAP” and you are getting a high number you may be suffering the same bug.

Unfortunately there is no way to force Akonadi to download the missing body for all those emails… so you will have to either live with it or re-import your account.

KMail not updating email status, not loading anything, not working !
If all your email is offline and you do not suffer from the bug I mentioned before, you won’t even notice this… but if you are using “Normal IMAP” or you suffer the bug this is the most annoying behavior.

Basically all the actions you perform are put in a queue to be processed, for example if you read 100 emails while offline, those “100 read actions” will be stored locally and pushed to the server once online. Where is the problem? the problem is that right now “pushing” this information blocks any other operation to be sure that we are using fresh data so if for instance email XXX is requested akonadi won’t provide it until the the “pushing” finishes.

In most IMAP servers pushing this data is FAST, but for some reason in GMail each modification takes around 1/2 seconds… so if I set as read 100 email or I delete 2000 make the math :p

The solution for this is conceptually quite easy, we should only block operations that affect email XXX instead of blocking everything. The technical implementation is a little more difficult than that :p

You can use akonadiconsole to check this “todo list”, open it go to “Agents” tab, and in there open the context menu on the agent you want to check and click on “show change-notification log”.

Akonadi and Kontact2 are an awesome piece of software, I have no doubts that once the issues I mentioned here are fixed we will offer a better user experience than the one you get either in osx or Thunderbird.

As a note, Thunderbird or iMail take also 15 hour to fetch all my email, with the difference that I can use my email right the way, we are not that far.

Finally, a screenshot of how I’d like KMail to look like by default… and I’m not the only one asking this :p

Edit: to have this config just do:

Settings–>Appearance–>Layout–>Show the message preview pane next to the message list
Settings->Appearance–>Message List–>Default Thene: Fancy with Clickable Status

  • Bugsbane

    All of this looks like great progress forward, but even as a KDE lover, I won’t be using Kmail until there’s some kind of Gmail style “conversations” view (see Thunderbird’s “Conversations” plugin for exact details of what and how).

    I love though that KDE PIM is looking at handling large, Gmail size accounts and the potential of Akonadi to really integrate email and other PIM data deeply into the OS. All the effort is appreciated, and I remain hopeful that I’ll be able to ditch Thunderbird (as Mozilla seems to be) soon.

    • http://www.afiestas.org afiestas

      Hehe I understand, in my case it is exactly the contrary, I’m so used to KMail2 thread interface that I can’t use anything else, specially those conversation vews :/

  • block

    I’m using KMail (for > 2 years), I have the same use case and noticed the same issues.
    Also for your screenshot, it looks like my current configuration ;)

  • walo

    Yo lo que hecho de menos es que detecte eutomaticamente las caracteristicas del servisor (como mozilla isp database) y la opcion de exportar cuentas para poder importarlas de nuevo después de un formateo….

    • http://www.afiestas.org afiestas

      Hay cosas… pero estoy de acuerdo que no es perfecto :/

  • http://blog.bauermann.eng.br/ Thiago Jung Bauermann

    If you use a dovecot imap server locally and have kmail2 connect to that (using regular IMAP instead of Disconnected IMAP (there shouldn’t be separate kinds of accounts for that btw, just a checkbox in settings)), then the pushes should be fast.

    And you would have an offlineimap instance which would sync GMail and dovecot every now and then.

    That’s the setup I use (with Evolution, not KMail2) and it makes a huge difference.

    • Will Stephenson

      That would be…weird. Since Akonadi is basically a custom-developed IMAP server with some auxiliary agents itself. I’d prefer we fix the protocol deadlocking properly.

      • http://blog.bauermann.eng.br/ Thiago Jung Bauermann

        Having a local dovecot server running and connecting to it via localhost is actually fairly common among developers, actually.

        I agree it’s weird and I don’t like this solution. But it’s the only bearable way to use Evolution with IMAP, and at least for now it looks like it’s the same with Kmail2. So it’s just a workaround while Akonadi isn’t fixed.

  • http://agateau.com Aurélien Gâteau

    So you want my face in the top-right corner by default? I am honored, sir :)

    • http://www.afiestas.org afiestas

      Mueheheh :p

  • Will Stephenson

    Where is Spanish Sync to be found? Sounds interesting. How long does the initial envelope fetch take for your large account? I feel that initial setup time matters, in case the user gets tired of waiting for the progress bar to complete and breaks the network connection. What Thunderbird does is only fetch the folder list, then fetch envelopes when entering a folder for the first time, then start fetching bodies. A hybrid of Spanish Sync (for inbox) and Mexican Sync ;) for other folders. A way to efficiently select a complete sync for other ‘important’ folders would be the icing on the cake.

    • http://www.afiestas.org afiestas

      It has not been developed yet, some pieces are missing (being able to tell akonadi somehow to fetch bodies from emails that were fetched as “envelope” only).

  • Kishore

    Just a paper cut… Notice that the message list view in the middle is slightly misaligned (wrong margins?) at the bottom when compared with the folder list on the left and the preview pane to the right.

  • Alex Elsayed

    I ran into the ‘empty messages’ thing as well, and found a (somewhat icky) method of using akonadiconsole to force a resync of a specific collection.

    WARNING: This is hackish and ugly and may cause breakage I am unaware of.

    If you go into the akonadiconsole browser tab, you can right click on a collection and open up the folder properties dialog. In that window, go to the ‘Attributes’ tab and set both uidnext and timestamp to ‘0’ (the number) and then restart the akonadi server. This makes Akonadi think that it a.) has no messages, and b.) last synced as of the epoch. That makes it resync the collection.

    I used this to resync the inbox of my gmail account, without having to recreate the whole thing (or resync all directories)

  • Pingback: My personal KDEPIM upgrade (again): laptopBoredome | Boredome()

  • Pingback: My personal KDEPIM upgrade (again): laptop - Bartle Doo()

  • http://testdomain.com test

    I hope more blogs like this will come into existence.

  • http://www.youtube.com/watch?v=sUyjoU5Ua4k Kina

    Hey, I think your blog might be having browser compatibility issues.
    When I look at your blog in Safari, it looks fine but when opening in Internet Explorer,
    it has some overlapping. I just wanted to give you a quick heads up!
    Other then that, fantastic blog!