KDEPIM sprint #2 (Understanding and diagnosing Akonadi)

In this second part I’m going to explain a few techniques to debug Akonadi we use in order to understand why my KMail installation was working bad.

My configuration
I’m a heavy email user, every day  I receive and read hundreds of them, and I love to have my email synchronized across devices (laptop, workstation, phone, tablet) and particulary in my laptop and workstation I want to have all my email offline so I can make use of it with or without internet, specially useful on my laptop while traveling. Because of all those reasons I have to use something called Disconnect IMAP.

My EMail account
The account has 5.6GB of data and according to GMail it is split in 700K emails (more or less) 200K of those emails are in a folder (label in GMail) called kde-commits with a size of 1.8GB, then it has 5 folders with more than 25K emails and around 200MB of data each, and finally the rest are just small folders with 500-5000 emails. In total there are 70 folders.

First import experience
The first problem I have with KMail2 is the experience out of the first email import. Since my account is so huge it takes a while (around 15 hours) to make the full import and I can’t use the application while doing it because either the data won’t be available or all my CPU power will be used in the task.

To fix this we designed something called “Spanish sync” which basically consist on splitting the task of “Downloading all email while indexing it” into the following steps:

  1. Download all the envelopes (subject,from, to, and some other data) starting from the most recent. Once this is done KMail will be fully functional (full email will be downloaded when user tries to read it).
  2. Download in a controlled way all the email bodies (where the content of the email is). Right now this goes as fast as your internet line allows which can be a problem for people having a fast internet connection (100Mb in my case) and slow machine (my laptop is 6yrs old). Once this finishes the email content will be available offline.
  3. Finally we will index all emails. If the user tries to search before this is done we will show an progress indicator.

The full synchronization of my email is still going to take 15 hours but I will be able to use my application almost since moment one (since it will import most recent emails first).

KMail not showing email body
Even though I set my account as “Disconnected IMAP”, more or less 50% of my emails were not being downloaded so I kept getting this “loading email” message from KMail. This is specially annoying when I’m on my laptop since the internet availability while traveling is not reliable. We have not been able to identify the reason why 1 out 2 emails is not being downloaded but at least I learned how to diagnose this.

Execute the debugging application “akonadiconsole” and go to the “DB console” tab and execute the following SQL query:

SELECT count(*) FROM parttable WHERE name = “PLD:RFC822″ AND data is null;

The result should be a small amount of your total email (in my workstation it is 4K while in my laptop it is 300K). If you set your account as “Disconnected IMAP” and you are getting a high number you may be suffering the same bug.

Unfortunately there is no way to force Akonadi to download the missing body for all those emails… so you will have to either live with it or re-import your account.

KMail not updating email status, not loading anything, not working !
If all your email is offline and you do not suffer from the bug I mentioned before, you won’t even notice this… but if you are using “Normal IMAP” or you suffer the bug this is the most annoying behavior.

Basically all the actions you perform are put in a queue to be processed, for example if you read 100 emails while offline, those “100 read actions” will be stored locally and pushed to the server once online. Where is the problem? the problem is that right now “pushing” this information blocks any other operation to be sure that we are using fresh data so if for instance email XXX is requested akonadi won’t provide it until the the “pushing” finishes.

In most IMAP servers pushing this data is FAST, but for some reason in GMail each modification takes around 1/2 seconds… so if I set as read 100 email or I delete 2000 make the math :p

The solution for this is conceptually quite easy, we should only block operations that affect email XXX instead of blocking everything. The technical implementation is a little more difficult than that :p

You can use akonadiconsole to check this “todo list”, open it go to “Agents” tab, and in there open the context menu on the agent you want to check and click on “show change-notification log”.

Summary:
Akonadi and Kontact2 are an awesome piece of software, I have no doubts that once the issues I mentioned here are fixed we will offer a better user experience than the one you get either in osx or Thunderbird.

As a note, Thunderbird or iMail take also 15 hour to fetch all my email, with the difference that I can use my email right the way, we are not that far.

Finally, a screenshot of how I’d like KMail to look like by default… and I’m not the only one asking this :p

Edit: to have this config just do:

Settings–>Appearance–>Layout–>Show the message preview pane next to the message list
Settings->Appearance–>Message List–>Default Thene: Fancy with Clickable Status

Results from my first KDEPIM sprint #1

A few weeks ago I went to Berlin, this time to attend the October KDEPIM sprint.

Though I have met some PIM hackers through the years I have been around KDE, I never had the opportunity to actually work with them, and try to figure things out, and as it is usual in our community they haven’t disappointed me.

I had 2 objectives for this sprint, first get a few bugs that I discover while working on WebAccounts fixed, and secondly try to track down all the KMail issues I have when I run it in my laptop. In this blog post I will only cover the first block.

Collections that can’t be de/activated

This is a really stupid bug, but in my humble opinion these are the kind of things that makes the user feel that they are using inferior quality software.

If you take a look at the following pic, you will see how there is a checkbox left of my email address that apparently will activate or deactivate all the contact groups provided by my Google account, well it doesn’t do anything so clicking on it results on basically nothing.

Resources bug

I tried to fix it and successfully tracked the bug down to a piece of code, but I needed help from the PIM hackers to actually fix it, and of course they didn’t disappointed, in a matter of hours it was fixed :)

KMail and the “Broken” imap resource

Just after a new IMAP resource was added (doesn’t matter if from KMail or Kontact or any other method) a notification will appear saying that the IMAP resource is broken. Wait what? is it broken? if I have just added it!

Well, not this is fixed since KMail now knows how to differentiate between a broken IMAP resource and one that has not been yet configured.

Activate resources when added

Another small bug or behavior that really bothers me is that when a new resource is added (for example a Google account is configured), the new added Calendars or Agendas won’t be “activated/checked” by default. I think that if you just added something is because you want to use it, don’t you think?

Well, this problem may seem simple but it isn’t, so after taking a look at it with the PIM hackers we thought a good way to fix it, though we didn’t had time to fix it.

http://lists.kde.org/?l=kde-pim&m=135040360317368&w=2

If nobody does, I will try to fix it when possible.

See you in part 2!

Meet the KDE TeaTime !

KDE TeaTime LogoFor a few weeks now 3 other KDE hackers and I have been doing a videocast/podcast/show/thing about KDE. It is a completely unprofessional unprepared amateur show where we basically talk about a different topic every week without having any script, we say things as we think them. it Doesn’t matter if they are politically correct or not.

My main motivation for having this show is to open a door to the insides of KDE. This door is open both ways so the communication should and must be from Developers to Users and the other way around, so please participate !

Some urls of interest:

And of course #kdeteatime in both identi.ca and twitter.
We are looking for all kind of participation, just to mention a few from the top of my head:
  • Leave feedback so we can improve
  • Ask questions of any kind!
  • Propose topics for future teatimes
  • Propose people to interview (and questions for these people)
  • Whatever you can think of :p
We usually do the shows every Tuesday at 16:00 CET but in anycase check the latest news and dates in our social sites.
So, join us and have a good time!