This is not a download error.
In your code you must be casting object of type MailAddress to MailBox explicitly.
MailAddress can be MailBox -or- MailGroup (both inherit from MailAddress).
Please use GetMailboxes method:
var mailboxes = email.To.SelectMany(address => address.GetMailboxes())
handle mailboxes and groups differently.
You can find more on this here: