The Mystery of the Disappearing Bccs

I’m supposed to be an email expert, so you can imagine that I was pretty frustrated not being able to understand why Bcc recipients were disappearing from my sent emails.

Let’s set the scene: I am nursing my old Windows Vista desktop through the end of its life, and setting up shop on a new laptop. I use Office 2007 as my ‘productivity’ tools, and this includes Outlook: I connect to my external email provider using IMAP, and manage all my emails that way. So I installed Office on my laptop, and configured Outlook to connect to my email account.

The problem: however, the emails in my Sent Items on the laptop don’t quite look like the same emails in my Sent Items on the desktop. More specifically, the desktop emails, when opened (not using the preview pane, which never shows Bcc recipients), properly reflect any Bcc recipients I had set, whereas the laptop emails don’t. Interestingly, the laptop emails contain populated Internet headers (and I can see the Bcc recipients listed in there), while the desktop emails don’t – this will become important later.

Quite a few of the emails I send contain Bcc recipients; for instance my Neighbourhood Watch messages do, and I often use the “Resend This Message” on these to avoid having to re-type a fairly long list of Bccs every time.

So it was important that I figured out what was wrong. Was it a different configuration of Outlook? Nope. Was is a different configuration of the email account? Not that I could see. Was it a different set of Add-Ins, with one of them (missing on the laptop) somehow properly populating the Bcc field from the Internet headers from the emails on the server? No, but these internet headers were nagging me.

One simple test provided an important clue: sending a Bcc mail from my laptop. Of course, it also appeared in the Sent Items in Outlook on my desktop, but there, it was lacking the Bcc recipient (and had Internet headers), whereas on the laptop I could see the email with its Bcc recipient intact (and no Internet headers) – the situation was reversed between the laptop and the desktop.

Ah, now I understood! The mystery was pretty simple after all. You see, Outlook stores emails etc. locally in a PST file (Personal Folders File). When you set up an email account in Outlook, you’ll have a PST file (or a PSO file for Exchange emails) on your hard disk mirroring the content of your account as it is on the email server (with Exchange accounts, you can turn that off, but that’s a different matter). This way, you can work offline, and not need to be connected to access your mailbox. Of course, initially the PST file is empty, and right after you set up a new email account in Outlook, it will spend a little time downloading the emails (at least all the ‘headers’ – the actual contents get downloaded on ‘first-read’) from the email server and copy them into the local PST file…

Also, when you compose and send an email, it’s usual to save a copy in a “Sent” folder (which may go by slightly different names). In my case, I had selected to have all sent emails copied to the Sent Items folder in my email account – the default is to save a copy in the Sent Items folder in the local-only “Outlook Personal Folders” (also a PST file, but a different one). This way, my sent emails were always saved in the email account, and accessible from anywhere. At the same time, a copy of these emails was obviously kept in my local PST file.

But that’s where the difference, and the lack of Internet headers, explained what was happening:

When I send an email from my desktop, Outlook at some point proceeds with the sending itself (talking SMTP to the email server), and then copies the email to the Sent Items folder in the PST file (it’s more a move from the Outbox folder, but that’s a technicality). That copy of the email is also sent to the email server (either through synchronisation of the PST file’s content via IMAP, or actively by Outlook sending the individual email via IMAP to the email server, or through an accessory command during the SMTP process – it doesn’t matter here). The important bit is that what’s in the PST file is a copy of the email in Outlook’s internal format, an original MAPI message – there are no Internet headers (the email never passed through the Internet). What gets stored on the email server however is an Internet format email – with Internet headers.

In turn, on the laptop, Outlook spots that a new email has appeared in the Sent Items folder on the server, and it pulls it in to the local PST file for the email account. While doing this and converting it to a MAPI message, it does not convert faithfully and fully. The Internet headers are preserved, but in particular the Bcc recipients listed in there are not transferred to the list of recipients in the Outlook format email (MAPI) in the PST file.

So there you have it, in short:

Send → copy in Sent Items in PST file (original MAPI, with Bcc recipients)
→ copy sent to Sent Items in email account (Internet format)

Separate synchronisation, email in Sent Items in email account
→ pull copy to Sent Items in PST file (Internet format→MAPI, no Bcc recipients)

In other words, only a copy of the sent email to the local PST file for the mailbox by the sending Outlook will have Bcc recipients preserved. If Outlook didn’t send the email, or not in this ‘lifetime’ (PST file from when the account was set up), it won’t have Bcc recipients.

Additional info:

Why doesn’t the sending Outlook later pull in the sent email from the email server, during a re-sync for instance? Well, the email is already there in the PST file (each email has a unique ID), so there is no point in pulling it from the email server to replace what’s already in the PST file. Zap your PST file though, and Outlook will pull everything back in, and convert to MAPI format, and you’ll magically now have Internet headers and no Bcc recipients on all these emails.

Is there other data that doesn’t make it from the email server? Probably, but I didn’t look into it.

Does this happen with all versions of Outlook? I don’t know. It definitively works that way with Outlook 2007, but Microsoft may have changed that in later versions.

Did I find out exactly how the PST file gets synchronised with the email server? No. It wasn’t necessary to find out.

So I’ve got this bunch of emails with Bccs I want to preserve to use with Outlook. What do I do? Come back later, I might have a solution. I’m going to try to ‘transplant’ the PST file from my desktop to the laptop… That didn’t work; don’t try it.

About Philippe Lasnier

A French- and British-educated professional software engineer with over 33 years of broad experience and a solid history of developing high-quality software for large and small companies across a range of applications and different industries; particularly experienced in C, C++; the development of middleware, internal components, core code, and APIs; solo, in a team, or as lead of small teams. Recognised for high-quality development, great attention to detail, keen analytic and investigative skills. Bilingual, English and French, just in case you were wondering. Philippe currently works for Spirent Communications, but in any case, the opinions here are his own.
This entry was posted in Email musing. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s