Introduction

As a company we receive messages from multiple sources (emails, feedback, contact requests,….).

Actualy we use Microsoft Outlook to manage and montior emails. And we developed an in-house application that collects and displays the other messages that are saved in multiple databases.

Actual Solution

For the moment we have only one person resoponsible for monitoring and handling all the messages. If he can't handle a specific message, than he will just forwarded to someone who could.

Not all messages require actions, some of them are just spams.

We use two tools to monitor incoming messages, Microsoft Outlook and a customized dashboard :

Microsoft Oulook

Outlook is used to monitor all incoming emails in all mail boxes. It is configured with all our accounts.

Customized dashboard

We have a custom web application that collect data from our databases and summarized it in from of a dashboard. There are a varieties of data including total number of site visits, total number of downloads, latest messages…etc.

The dashboard is checked daily for incoming messages.

Problems with the current solution

I noticed with time the following issues with the current implementation:

• It takes time to configure Outlook with all the accounts

People don't work always from the same machine. And they perform clean system install from time to time. It happens that they procrastinate or they don't have the time to configure all the accounts in outlook.

• Some messages that requires action slip from time to time

With manual handling, some messages are not handled or followed up properly.

• We need to put more people handling messages

Having multiple people handling the same messages without an appropriate tracking system, will create a lot of conflicts and let more messages to slip.

People have a tendency to answer easy messages, and let the rest of the team handle the hard ones. We can also have people handling the same message twice.

Putting more people make it tricky to keep track of what was handled.

• Supervise what's going is not obvious

Putting a supervisor in CC will not inform him about unhandled messages. Supervisors need to be notified if a message is not handled in time.

The solution

We need a tool to monitor all incoming feedbacks and messages and track their handling.

What this tool is supposed to do?

This tool needs to monitor all email boxes as well as messages stored in different databases. When a new message arrives, the team is informed that a message need to be handled. After a period of time, if the message was not handled, a supervisor is informed about it. The supervisor will kept notified until the message is handled.

The interaction between this tool and the team is done via a task list. This BPML diagram shows the process described above:

Figure 1 : BPML diagram that describe the handling and monitoring process

How this process will be implemented?

The process will be implemented as windows workflow application and executes as a Windows Azure Worker Role.

The worker role will loop and check for incoming messages. When a new message is found, a workflow instance is created to notify the team and monitor the handling.

To notify the team, the application (WF) creates a task in the team's task list. Team members will pick up tasks, do the work, and mark the task as completed. If someone pick up a message that he can't handle, he will just assign the task to someone who could. In the meantime, the workflow keeps monitoring the status of the task. If it still uncompleted for 2 days, a supervisor is notified by email. The supervisor will keep be notified each 2 days, until the task is done.

First release

In this first release we will monitor only emails. I write it in the last weekend as a prototype to try the solution.

Implementation

The rest of this article will describe how I implemented the application.

Source Code

You can download the source code from my GitHub page.

Building blocks

In this section I will describe how each elementary task is implemented. An elementary task is a small unit of work that is represented by a task in my BPMN diagram. In my case those units of work are:

• Connecting to IMAP server and getting new message
• Create a task in our Smarter Mail Server and Check their status
• Send Notification Emails

Checking for new emails

In order to monitor emails, I used the library MailSystem.Net. The library allows to query IMAP servers for emails.

When I add a new email account to the application to monitor, I process all unread messages. After that I will check only for new message whether they are read or not.

How to query IMAP for new messages?

In IMAP, as described by RFC 3501 each message have a unique id represented by UID. A UID is a 32 bit integer where for each two messages A and B, if A was received after B then the UID of A is great than the UID of B. IMAP allows us to search for messages by UID range. Check RFC 3501 page 48.

Each time we get a message, the application saves the last (the bigger) UID that will be used to search for new messaged in the next iteration.

// Fetching new messages
// For newly configured accounts, we fetch only unread messages for the first time. After that
// we look for new messaged whether or not they are read.
var searchQuery = account.LastUid <= 0
     ? "UNSEEN"
     : string.Format("UID {0}:*", account.LastUid + 1);

return inbox.Search(searchQuery)…

Creating tasks

Currently our email servers uses the latest version Smarter Mail Enterprise. We already have a shared task list that we use to track some of our tasks. We will use it to track the handling of each message. As I said earlier, for each newly arrived message, a task is created in this shared list.

Smarter Mail have many web services that we can use to perform different tasks on the server. The service at https://serveraddress/Services/svcOutlookAddin.asmx exposes a series of methods used to manipulate tasks. In our scenario I used the following two methods:

AddTask    to add new task
GetTask    to check if the task is completed or not

Tasks are compatible with Outlook tasks and have the same format. They are represented by "ExTaskItem" object. Basically, each object maintain two separate string arrays, the first array contain a list of all properties names such as "Subject", "Description", "Status",…etc. While the 2^nd array contains the values of these properties encoded in Base 64. The following code shows how we create a simple task:

var task = new ExTaskItem { key = new ArrayOfString(), val = new ArrayOfString() };
// The following two lines set the task subject to "This is my task"
task.key.Add("Subject");
task.val.Add(Convert.ToBase64String(Encoding.UTF8.GetBytes("This is my task"))); // values are Base 64 encoded

To make it easy to manipulate "ExTaskItem" object, I create a class called "OutlookTask" which wrap ExTaskItem. The class have all the properties of a task used in the application. The class constructor automatically initializes default values for us. For the status of the task, the web server don't allow the "Not started" status. I used "In Progress" instead.

All operations needed by the application are wrapped in the TasksManager class. The class have the following two methods:

public static string CreateTask(string subject, string description, DateTime dueDate) // Create a new task and return its id. The id will be used later to check for the task completion
public static bool IsTaskCompleted(string id)    // Checks if the task is completed or not

Sending emails notification

Us I described earlier, I want the application to send notifications to a supervisor if a message didn't get handled in time.

Notifications are sent by email. I also used the library MailSystem.Net to send emails. The syntax is very easy, you just create a SmtpMessage, initialize its properties and call send message.

The send operation is encapsulated in the EmailsManager class.

Warning: Don't monitor the supervisor email address. During the first test I put the supervisor email address in the list of the accounts to monitor. In the morning I've found more than 7000 emails, and more than 7000 workflow instances. The system went in a loop!

Glue things together

After playing with the different building blocks, it's time to integrate everything together in a working system. I have divided the system on two processes as the following:

1. A windows Azure worker role: its job is to loop for incoming messages.
2. A WF workflow to handle messages: for each new message, a workflow instance is created to handle the message according to the process described above.

The windows Azure Role

The azure role is just an infinite loop that checks for new messages. When a new message is received, the role creates a workflow instance to handle it.

while (true)
{

      var msgs = EmailsManager.GetAllNewMessages(); // Get All new messages since the last check       

      msgs.ForEach(EmailHandlerService.HandleEmail);    // EmailHanderService.HandleEmail creates a workflow instance to handle the email.    

      Thread.Sleep(TimeSpan.FromMinutes(15));    // Sleep for 15 minutes before the next check

}

The EmailsManager.GetAllNewMessages() returns the list of all new messages. The worker than loops through them and creates a workflow instances by calling EmailHandlerServices.HandleEmail.

When I receive many new messages, I noticed during tests that the process hangs up while creating the new workflow instances. To avoid this problem, I just sleep a little to give a little time to the previous workflow instance to start.

The Workflow

The workflow is implemented using Windows Workflow foundation. Here are the steps of the current version:

Figure 2: Email Handling Workflow

1. Save the IMAP UID in the database. As I explained in "Checking for new emails", we search for new messages using the last received message UID. I intentionally save the uid in the workflow rather than the worker. By this I'm sure that the workflow is started. If I save in the worker or in the GetAllNewMessages and the process get stopped and the workflow instance failed to start, I will lose the message. I also persisted the workflow state before saving the UID for that same reason.
2. Create a task in our smarter mail server.
3. Sleep for two days since the message real received time
4. Check if the task was completed
5. If the task was completed, then the workflow will end
6. If the task was not completed, the supervisor is informed via email that the task still undone.
7. Sleep for two days
8. Repeat from step 4

Hosting requirements

If you want to host this application to monitor your emails you need the following:

1. A SQL Server database to store IMAP accounts
2. A SQL Server database to persist workflow states
3. A Windows Azure subscription to host the worker. For this worker an A0 (shared core, 768 MB memory) instance was enough for me.

Conclusion

In this post I explained how I created a system that will help me automate email handling (and other messages in the future). The idea is that the system creates a task for every incoming message and monitors its handling.

By integrating messages handling in our tasks list. I will save the team the time that was necessary for checking emails. And I will save my time tracking answers by just looking at tasks status.