Archive for the ‘Programming’ Category

SSL vs TLS vs STARTTLS

Wednesday, October 9th, 2013

There’s often quite a confusion about the different terms: SSL, TLS, STARTTLS and STLS.

SSL and TLS

SSL and TLS are cryptographic protocols, both provide a way to encrypt communication channel between two machines over the Internet (e.g. client computer and a server). SSL stands for Secure Sockets Layer and current version is 3.0. TLS stands for Transport Layer Security and the current version is 1.2. TLS is the successor to SSL. The terms SSL and TLS can be used interchangeably, unless you’re referring to a specific protocol version.

Version numbering is inconsistent between SSL and TLSs. When TLS took over SSL as the preferred protocol name, it began with a new version number. The ordering of protocols in terms of oldest to newest is: SSLv2, SSLv3, TLSv1.0, TLSv1.1, TLSv1.2.

STARTTLS and STLS

STARTTLS is a protocol command, that is issued by an email client. It indicates, that the client wants to upgrade existing, insecure connection to a secure connection using SSL/TLS cryptographic protocol. STARTTLS command name is used by SMTP and IMAP protocols, whereas POP3 protocol uses STLS as the command name.

Despite having TLS in the name, STARTTLS doesn’t mean TLS will be used. Both SSL and TLS are acceptable protocols for securing the communication.

Clear text/Plain text

No security protocol is used at all. All commands, responses and data are transferred in plain text.

client.Connect("mail.example.com");

Implict SSL mode

Implict SSL mode means, that you connect to SSL/TLS encrypted port.

client.ConnectSSL("mail.example.com");

Explicit SSL mode

Explicit SSL mode means, that you connect to plaint text port and secure the connection by issuing STARTTLS (or STLS) command afterwards (you explicitly secure the connection).

client.Connect("mail.example.com");
client.StartTLS();

Securing the connection

Regardless of whether you use implict (connecting to an SSL/TLS encrypted port) or explicit (using STARTTLS to upgrade an existing connection) mode, both sides will negotiate which protocol and which version to use. This negotiation is based on how client and server have been configured and what each side supports.

SSL/TLS support

Support for SSL/TLS is virtually universal, however which versions are supported is variable. Pretty much everything supports SSLv3. Most machines support TLSv1.0.

TLS vs STARTTLS naming problem

One significant complicating factor is that some email software incorrectly uses the term TLS when they should have used “STARTTLS” or “explicit SSL/TLS”. Older versions of Thunderbird used “TLS” to mean “enforce use of STARTTLS to upgrade the connection, and fail if STARTTLS is not supported” and “TLS, if available” to mean “use STARTTLS to upgrade the connection, if the server advertises support for it, otherwise just use an insecure connection” (very problematic, as we’ll see below).

Port numbers

To add security to some existing protocols (IMAP, POP3, SMTP), it was decided to just add SSL/TLS encryption as a layer underneath the existing protocol. However to distinguish that software should talk the SSL/TLS encrypted version of the protocol rather than the plaintext one, a different port number was used for each protocol:

Protocol Plain text SSL
IMAP 143 993
POP3 110 995
SMTP 587 or 25 465

Too many ports? Solution: Plain text + STARTTLS

At some point, it was decided that having 2 ports for every protocol was wasteful, and instead it’s better to have 1 port, that starts off as plain text, but clients can upgrade the connection to an SSL/TLS encrypted one using STARTTLS (or STLS for POP3 protocol) command.

STARTTLS problems

There were a few problems with this. There exists lots of software, that used the alternate port numbers with pure SSL/TLS connections. Client software can be very long lived, so you can’t just disable the encrypted ports until all software has been upgraded.

Each protocol received mechanisms to tell clients that the server supported upgrading to SSL/TLS (e.g. STARTTLS in IMAP’s CAPABILITY response), and that they should not attempt to login without doing the STARTTLS upgrade (LOGINDISABLED in IMAP’s CAPABILITY response). This created two unfortunate situations:

  • Some software just ignored the “login disabled until upgraded” announcement (LOGINDISABLED, STARTTLS) and just tried to log in anyway, sending the user login name and password over clear text channel. The server rejected the login and password, but the details had already been sent over the Internet in plain text.
  • Other software saw the “login disabled until upgraded” announcement, but then wouldn’t upgrade the connection automatically, and thus reported login errors back to the user, which caused confusion about what was wrong.
    • Both of these problems resulted in significant compatibility issues with existing clients, and so most system administrators continued to just use plain text connections on one port, and encrypted connections on a separate port number.

      Disable plain text for IMAP and POP3

      Many companies (e.g. Gmail, Outlook.com) disabled plain IMAP (port 143) and plain POP3 (port 110), so people must use a SSL/TLS encrypted connection – this removes the need for having STARTTLS command completely.

      SMTP STARTTLS stays

      The one real exception to the above is SMTP. Most email software used SMTP on port 25 to submit messages to the email server for onward transmission to the destination. However SMTP was originally designed for transfer, not submission. So yet another port (587) was defined for message submission.

      Port 587 doesn’t mandate requiring STARTTLS, however the use of port 587 became popular around the same time as the realization that SSL/TLS encryption of communications between clients and servers was an important issue. The result is that most systems, that offer message submission over port 587 require clients to use STARTLS to upgrade the connection. Login and password to authenticate is also required.

      There has been an additional benefit to this approach as well. By moving users away from using port 25 for email submission, ISPs can block outgoing port 25 connections from users’ computers, which were a significant source of spam, due to user computers infected with spam sending viruses.

      Further readings

      Using SSL/TLS with IMAP
      Using SSL/TLS with SMTP
      Using SSL/TLS with POP3

Copy to public folder: “There is no replica for that mailbox on this server.”

Sunday, September 1st, 2013

When copying an mail from the personal folder to a Public folder, you may receive “There is no replica for that mailbox on this server.” error.

This error is generated by Exchange server and unfortunately this is the Exchange limitation. Here’s Microsoft’s response to this problem:

Copying from user mailboxes to public folders does not work with Exchange IMAP4.
Your users need to copy the message to a personal folder and then back up to the public folder (append) or forward it to an address that gets archived to a public folder.

Workaround

It seems the only way to workaround this is by downloading a message and uploading it to public folder. Important thing is that you don’t need to parse email message at all – you just download and upload raw eml data. Please also note that public folder may be in fact stored on a another IMAP server instance (different server address) – this is usually indicated during logon with REFERRAL error.

// C#

using(Imap imap = new Imap())
{
    imap.Connect("imap.example.com");   // or ConnectSSL for SSL
    imap.UseBestLogin("user", "password");

    imap.SelectInbox();
    List<long> uids = imap.Search(Flag.All);

    foreach (long uid in uids)
    {
        var eml = imap.GetMessageByUID(uid);
        IMail email = new MailBuilder().CreateFromEml(eml);

        if (email.Subject.Contains("[REF"))
            UploadToPublic(email);
    }
    imap.Close();
}
' VB.NET

Using imap As New Imap()
	imap.Connect("imap.example.com")	' or ConnectSSL for SSL
	imap.UseBestLogin("user", "password")

	imap.SelectInbox()
	Dim uids As List(Of Long) = imap.Search(Flag.All)

	For Each uid As Long In uids
		Dim eml = imap.GetMessageByUID(uid)
		Dim email As IMail = New MailBuilder().CreateFromEml(eml)

		If email.Subject.Contains("[REF") Then
			UploadToPublic(email)
		End If
	Next
	imap.Close()
End Using

Here is the body of UploadToPublic method:

// C# code

private void UploadToPublic(IMail email)
{
    using (Imap imap = new Imap())
    {
        imap.Connect("server");  // or ConnectSSL for SSL
        imap.Login("user", "password");
    
        imap.UploadMessage("#Public/Cases", email);
    
        imap.Close();
    }
}
' VB.NET code

Private Sub UploadToPublic(email As IMail)
	Using imap As New Imap()
		imap.Connect("server")
		' or ConnectSSL for SSL
		imap.Login("user", "password")

		imap.UploadMessage("#Public/Cases", email)

		imap.Close()
	End Using
End Sub

Get IIS pickup directory location

Thursday, August 22nd, 2013

If you plan to use local IIS SMTP service to send emails you created using Mail.dll, you’ll need to save those emails to IIS pickup folder.

Default folder location is “c:\Inetpub\mailroot\Pickup”

There is a way to get IIS pickup folder location directly from IIS metabase. To get this path programmatically we’ll use IisPickupDirectory class. Unfortunatelly this class is not public, we’ll use its name to get its type and Activator class to invoke private static method GetPickupDirectory.

To get type reference, we need to specify fully qualified type name “System.Net.Mail.IisPickupDirectory System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089”. We can also search CurrentDomain’s assemblies to find System assembly and get its full name.

Assembly system = AppDomain.CurrentDomain.GetAssemblies()
    .First(x => x.GetName().Name == "System");

Type iisPickupType = Type.GetType(
    "System.Net.Mail.IisPickupDirectory, "
    + system.GetName().FullName,
    true);
// -or- use fully qualified assembly name directly:
//Type iisPickupType = Type.GetType(
//    "System.Net.Mail.IisPickupDirectory, "
//    + "System, Version=4.0.0.0, Culture=neutral, " 
//    + "PublicKeyToken=b77a5c561934e089",
//    true);

string pickupFolder = (string)iisPickupType.InvokeMember(
    "GetPickupDirectory",
    BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.NonPublic,
    null, null, null);

“Cannot get IIS pickup directory.” error

Unfortunately, this exception is raised when any kind of problem occurs, while trying to determine the location of IIS/SMTP pickup directory.

A common cause is simply missing IIS SMTP service.

The pickup directory is stored in the IIS Metabase, so if the account that your web-app runs as does not have access to the required nodes, this error can be thrown. Metabase permissions are separate from file permissions, so you explore it with Metabase explorer (part of the IIS resource kit).

These nodes need to have read permission given to your web-app user: \LM, \LM\Smtpsrv\ and \LM\Smtpsrv\1

Read system.net/mailSettings/smtp settings from web.config

Sunday, August 18th, 2013

There is a standard way of specifying SMTP settings in .NET applications. .NET uses config files (app.config or web.config in case of ASP.NET) and element to specify the appropriate SMTP parameters to send e-mail.

Sample configuration (in this case Gmail SMTP settings) looks as follows:

<configuration>

<system.net>
  <mailSettings>
    <smtp deliveryMethod="network" from="pam@gmail.com">
      <network
        host="smtp.gmail.com"
        port="465"
        enableSsl="true"
        userName="pam@gmail.com"
        password="password"
    />
    </smtp>
  </mailSettings>
</system.net>

</configuration>

If port attribute is omitted default value (25) is used. SMTP protocol typically uses ports 587 and 25 for non SSL connections, and port 465 for SSL ones.

Although Mail.dll SMTP component does not support reading from web.config directly, it is quite easy to read those settings programmatically and use them with Mail.dll classes.

Here’s the simple sample that reads from mailSettings section:

SmtpSection section = (SmtpSection)ConfigurationManager.GetSection("system.net/mailSettings/smtp");

string from = section.From;
string host = section.Network.Host;
int port = section.Network.Port;
bool enableSsl = section.Network.EnableSsl;
string user = section.Network.UserName;
string password = section.Network.Password;

Use web.config’s mailSettings with Mail.dll

In most cases you want to send email via SMTP server (DeliveryMethod set to SmtpDeliveryMethod.Network). Here’s the sample that uses web.config settings and Mail.dll’s STMP component to send an email message:

SmtpSection section = (SmtpSection)ConfigurationManager.GetSection("system.net/mailSettings/smtp");

IMail email = Fluent.Mail
            .Text("Hi, how are you?")
            .Subject("Hello")
            .To("to@example.com")
            .From(section.From)
            .Create();

using (Smtp client = new Smtp())
{
    client.Connect(section.Network.Host, section.Network.Port, 
        section.Network.EnableSsl);
    client.UseBestLogin(section.Network.UserName, section.Network.Password);
    client.SendMessage(email);
    client.Close();
}

IIS pickup folder

If you plan to use local IIS SMTP service to send emails you created using Mail.dll, you’ll need to save those emails to IIS pickup folder.

You can specify folder location explicitly using SpecifiedPickupDirectory.PickupDirectoryLocation (default location is “c:\Inetpub\mailroot\Pickup”). You can also use SmtpDeliveryMethod.PickupDirectoryFromIis constant – in this case we’ll get pickup folder location directly from IIS metabase.

Following is the code that recognizes different DeliveryMethods and acts accordingly:

SmtpSection section = (SmtpSection)ConfigurationManager.GetSection("system.net/mailSettings/smtp");
IMail email = Fluent.Mail
            .Text("Hi, how are you?")
            .Subject("Hello")
            .To("lesnikowski@limilabs.com")
            .From(section.From)
            .Create();

if (section.DeliveryMethod == SmtpDeliveryMethod.Network)
{
    using (Smtp client = new Smtp())
    {
        client.Connect(section.Network.Host, section.Network.Port, 
            section.Network.EnableSsl);
        client.UseBestLogin(section.Network.UserName, section.Network.Password);
        client.SendMessage(email);
        client.Close();
    }
}
else if (section.DeliveryMethod == SmtpDeliveryMethod.SpecifiedPickupDirectory)
{
    string pickupFolder = section.SpecifiedPickupDirectory.PickupDirectoryLocation;
    email.Save(Path.Combine(pickupFolder, "email.eml"));
}
else if (section.DeliveryMethod == SmtpDeliveryMethod.PickupDirectoryFromIis)
{
    Assembly system = AppDomain.CurrentDomain.GetAssemblies()
        .First(x => x.GetName().Name == "System");
    
    Type iisPickupType = Type.GetType(
        "System.Net.Mail.IisPickupDirectory, " 
        + system.GetName().FullName, 
        true);
    // -or- use fully qualified system assembly name directly:
    //Type iisPickupType = Type.GetType(
    //    "System.Net.Mail.IisPickupDirectory, " 
    //    + "System, Version=4.0.0.0, Culture=neutral, "
    //    + "PublicKeyToken=b77a5c561934e089",
    //    true);

    string pickupFolder = (string)iisPickupType.InvokeMember(
        "GetPickupDirectory", 
        BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.NonPublic, 
        null, null, null);

    email.Save(Path.Combine(pickupFolder, "email.eml"));
}

How to shorten Connect timeout

Thursday, March 21st, 2013

Connect and BeginConnect methods take quite a long time to timeout, when you use incorrect server address. It usually takes almost 20 seconds for those methods to decide that connection is impossible. Setting SendTimeout/ReceiveTimeout doesn’t influence this in any way.

Not only Imap, Pop3, and Smtp classes suffer from this problem, it’s the same for regular Socket class.

There is a solution however, using BeginConnect and simply waiting for specified amount of time (AsyncWaitHandle.WaitOne) without calling EndConnect (which blocks):

// C#

using(Imap imap =new Imap())
{
    IAsyncResult result = imap.BeginConnectSSL("imap.example.com");

    // 5 seconds timeout
    bool success = result.AsyncWaitHandle.WaitOne(5000, true);

    if (success == false)
    {
        throw new Exception("Failed to connect server.");
    }
    imap.EndConnect(result);

    //...

    imap.Close();
}
' VB.NET

Using imap As New Imap()
	Dim result As IAsyncResult = imap.BeginConnectSSL("imap.example.com")

	' 5 seconds timeout
	Dim success As Boolean = result.AsyncWaitHandle.WaitOne(5000, True)

	If success = False Then
		Throw New Exception("Failed to connect server.")
	End If
	imap.EndConnect(result)

	'...

	imap.Close()
End Using