Using RSA encryption

Question

I have another question about S/MIME that I cannot quite see in the documentation.

One of the counter-parties we are working with has implemented HSM in the FIPS140-2 mode (pretty sure you have heard of the US government’s cryptographic bureaucracy…) and now they are failing to decrypt our messages because their device does not support the RSAES-OAEP algorithm (OID is 1.2.840.113549.1.1.7).

Others apparently use RSA (1.2.840.113549.1.1.1).

I do not quite see how we can affect the algorithm choice - in the signature configuration we can only specify hashing algorithm OID and there are no CommonOIDs members for any RSA algorithms, so I am assuming the RSAES-OAEP is picked by default.
Thank you,

Answer 1

First, RSAES-OAEP is newer than RSAES-PKCS1-v1_5 (RSA).

Second, some background:

You use asymmetric system to exchange key and symmetric system to encrypt/decrypt data using this key.

This means the message is encrypted using 3DES or AES256 (content encryption), key - later used by 3DES or AES - is exchanged using some asymmetric algorithm (key exchange).

Key exchange is selected by EnvelopedCms - internal .NET class.

You can check RecipientInfo.KeyEncryptionAlgorithm property (through EnvelopedCms.RecipientInfos) to get key exchange algorithm used:

IMail email = new MailBuilder().CreateFromEml(eml);

EnvelopedCms envelope = email.GetEnvelopedCms()

// Key exchange algorithm used for particular recipient:
Assert.AreEqual(
    "1.2.840.113549.1.1.7",
    envelope.RecipientInfos[0].KeyEncryptionAlgorithm.Oid.Value);

Assert.AreEqual(
    "RSAES_OAEP",
    envelope.RecipientInfos[0].KeyEncryptionAlgorithm.Oid.FriendlyName);

// Encryption algorithm used:
Assert.AreEqual(
    "aes256",
    encrypted.GetEnvelopedCms().ContentEncryptionAlgorithm.Oid.FriendlyName);

When encrypting SMIME message, 3DES is used by default for content encryption. You can specify different encryption algorithm to use (e.g. AES 256):

EncryptionConfiguration config = new EncryptionConfiguration();
config.Algorithm = new Oid(CommonOids.Aes256); // use AES 256

config.Certificates.Add(new X509Certificate2(...));
config.Certificates.Add(new X509Certificate2(...));

MailBuilder builder = new MailBuilder();
builder.Text = "Encrypted";
builder.EncryptWith(config);

IMail encrypted = builder.Create();

Reference:
https://www.limilabs.com/blog/send-encrypted-email-smime-aes-256

It turns out, that when using 3DES (default), RSA (1.2.840.113549.1.1.1) is used as key exchange algorithm.

For AES 256, RSAES-OAEP (1.2.840.113549.1.1.7) is used. To reiterate: that happens only when you explicitly use AES 256:

configuration.Algorithm = new Oid(CommonOids.Aes256);

As a side note: 3DES is FIPS 140-2 compliant.

There is a way to force .NET to use RSA key exchange and AES:

1.
Add an UnprotectedAttribute:

envelopeCms.UnprotectedAttributes.Add(new Pkcs9DocumentName("a"));

2.
Use SubjectKeyIdentifier identification:

CmsRecipient recipient = new CmsRecipient(
    SubjectIdentifierType.SubjectKeyIdentifier, 
    certificate);

Option 1. is not available in Mail.dll, you can go with option 2. however:

EncryptionConfiguration config = new EncryptionConfiguration();
config.Algorithm = new Oid(CommonOids.Aes256); // use AES 256

config.RecipientIdentifierType = SubjectIdentifierType.SubjectKeyIdentifier;

configuration.Certificates.Add(...);
configuration.Certificates.Add(...);

MailBuilder builder = new MailBuilder();
    builder.Text = "Encrypted";
    builder.EncryptWith(config);

IMail encrypted = builder.Create();

There is a small disclaimer though:

"make sure that the certificate has an explicit SubjectKeyIdentifier extension before using this identification form. .NET/Windows will make up an implicit value if there isn't an explicit one, and not all CMS implementations will match the recipient certificate in that case (e.g. OpenSSL)."

Reference:
...and internal .NET code (encryptParam.useCms variable):
https://referencesource.microsoft.com/#System.Security/system/security/cryptography/pkcs/envelopedpkcs7.cs,377