Tag Archives: exchange 2003

Folder item limits for various versions of Exchange

UPDATE, Nov 2015: I’ve added the item counts for Exchange 2013 and 2016, and the limits for Outlook in cached mode.

High item count may not be the root of all evil, but it’s certainly up there. What constitutes high item count varies according to your environment, but it’s pretty low in exchange 2003. It’s possible to mistake a high item count issue for poor hardware performance, and in any scenario where you suspect your hardware, it’s worth looking for high item count folders and trying to reduce them to the recommended maxima, or possibly below. The recommended maxima are for “good” hardware environments and optimal configurations, not your mess. You should be familiar with the literature, and think of item count whenever presented with a performance issue.

Folder item count has a huge effect on Exchange performance; more so than mailbox size. Much of the time, rather than looking at a folder, a user may be looking at a server generated view of a folder, at which point item count becomes crucial to how long it takes the server to put together the view, as each item is evaluated.
Most of our customers are familiar with the Exchange 2007 article “Understanding the Performance Impact of High Item Counts and Restricted Views”, and many have noticed the Exchange 2003 limits given in there (If you haven’t read this article, then you really, really should). What many miss, however, are the caveats Microsoft put in there. Abiding by these limits you may expect to achieve “acceptable” performance.

“With properly architected hardware”, (and exchange 2007), “an acceptable user experience can still be maintained with item counts as high as 20,000 items.”

The definition of “acceptable” may not be what you expect, however. From the same article:

“This recommended maximum also depends on the performance capability of your Exchange environment. Your specific hardware choices may result in lower maximum numbers. Ideally, it is best to keep the Inbox and Sent Items folders less than 20,000 items, and the Contacts and Calendar item counts less than 5,000. Even when maintaining item counts that are at or under the recommended maximum values, there are some operations which may still take noticeable time (usually this is approximately one minute).”

So, with properly specified and correctly performing hardware, and with an item count below the maximum recommended limit, there are some operations that will have some of your users gnashing their teeth with frustration. I know this, because people log calls with us about delays of a minute. They generally don’t call it a minute, however; usually, it’s “literally hours”.

It’s also worth noting that for commonly used folders, the “critical” ones, the limits are very much lower. The real bad boy is the calendar folder, as that is the one most likely to be accessed by other users, and requires a search to filter out private items. 20,000 calendar items will bring a server to its knees, I’ve found.

Also, the Microsoft published maxima are not necessarily the same as those the engineers believe to be reasonable. For instance, Nicole Allen, an Outlook engineer writing in the official exchange technet blog, recommends that critical folders on Exchange 2003 have no more than 1000 items in them.

So, bearing that in mind, what does good look like?

Version General Folders Critical Folders (inbox, calendar, contacts, sent items)
Exchange 2003 5,000 1,000
Exchange 2007 20,000 5,000
Exchange 2010 100,000 10,000
Exchange 2013 1,000,000 1,000,000
Exchange 2016 1,000,000 1,000,000

Critical folders vary depending on how your users interact with each other – most of us only allow one or two people to our inbox, so for 2007 and 2010 Microsoft don’t refer to it as critical; Calendar and Contacts most definitely are, though.

To summarise sources, and extract a sentence or two from each that I think you should particularly watch for:

Exchange 2003 (http://blogs.technet.com/b/exchange/archive/2005/03/14/395229.aspx):

5000, Nicole allen says 1000 for inbox and calendar, here.

I usually recommend no more than about 2500 – 5000 messages in any of the critical path folders.  The critical path folders are the Calendar, Contacts, Inbox, and Sent Item folder. Ideally, keep the Inbox, Contacts and Calendar to 1000 or less.  Other folders, particularly custom folders created by the user, can handle having larger numbers of items without having a broad impact on the user experience (20,000 items in my “Cookie Recipes” folder?  No problem – except when I need to find that recipe from last Christmas!).

exchange 2007 (http://blogs.technet.com/b/exchange/archive/2005/03/14/395229.aspx):

20000, but keep critical path folders such as inbox and calendar below 5000:

With Exchange Server 2003, the recommended maximum item count per folder was 5,000 items. In Exchange 2007, improvements in I/O, larger page size, and increased cache can help enable an increase in the recommended maximum item count. With properly architected hardware, an acceptable user experience can still be maintained with item counts as high as 20,000 items.

exchange 2010 (http://technet.microsoft.com/en-us/library/ee832791.aspx ):

100,000, but no more than 10,000 in calendar and contacts.

“A challenging scenario occurs when a user has exceeded the number of indexes that Exchange will store. This is 11 indexes in Exchange 2010. When the user chooses to sort a new way, and thereby creates a twelfth index, this causes additional disk I/O activity. Because the index isn’t stored, this additional disk activity cost occurs every time that this sort is performed. Because of the high I/O activity that can be generated in this scenario, we strongly recommend that you store no more than 100,000 items in core folders, such as the Inbox and Sent Items folders, and no more than 10,000 items in the Calendar and Contacts folders. The creation of more top-level folders, or of subfolders beneath the Inbox and Sent Items folders, greatly reduces the costs that are associated with this index creation. This is true as long as the number of items in any folder doesn’t exceed 100,000.”

Exchange 2013 and 2016:

The recommended limit for 2013 is given in a footnote to the table “Mailbox folder limits across standalone plans” in the Exchange Online Limits article. It’s the same as the hard limit in Exchange Online; 1 million items. In his presentation on the Exchange 2016 Preferred Architecture given at Ignite in May 2015, Ross Smith IV states that the recommended upper limit for 2016 is also 1 million items. He states a bunch of other interesting stuff also. go and watch it.

Outlook Cached Mode (https://support.microsoft.com/en-us/kb/2768656):

Yeah, all this nonsense applies to Exchange… so what about Outlook? These figures are for cached mode.

Version Total Folders Items per Folder
Outlook 2007 500 50,000
Outlook 2010 500 100,000
Outlook 2013 500 100,000
Outlook 2016 ? ?

There are also interactions between exchange and different versions of outlook. Outlook 2003 in particular needs to be on SP1, as there is a bug in the RTM version. Of course, you’re all on SP3, by now, and looking to upgrade before it goes completely out of support in April 2014. Outlook 2007 and Outlook 2010 have differing caching behaviours; outlook 2010 will cache other users’ MAIL folders by default, which can lead to some long delays.

So what are the indicators of a possible high item count issue? From a user perspective, “oh my god it’s so sloooooow…” is a common one. But then, it always is. Take a perfmon, and have a look – there may be obvious things on there, like disk access or high cpu, which initially will look like a hardware bottleneck. It would be easy, at this point, to say “hardware’s rubbish” and go down the canteen, rather than start running powershell commands to find high item count folders or using pfdavadmin. Sometimes, however, there’s no obvious hardware bottleneck – what then? Have a look at the “client related search” counters under msExchangeIS mailbox – the two biggies are slow findrow rate, which is explained by Mike Lagase here, and slow qp threads.

Everyone should read the following two articles on this, the first from an outlook perspective, and the second from an exchange one.
Outlook users experience poor performance when they work with a folder that contains many items on a server that is running Exchange Server
Understanding the Performance Impact of High Item Counts and Restricted Views

Preventing Automatic Forwarding by Client Side Rules

A customer has had a query regarding how to prevent automatic forwarding and automatic replies in Exchange.

in Exchange 2003 this is controlled by the properties of Internet Message Formats:

Exchange Organization|Global Settings|Internet Message Formats.

chances are there is only one format, called “default”. right click properties and away you go.

Things are slightly different in exchange 2007 and 2010:

  1. Open Exchange Management Console
  2. Expand Organization Configuration-> Hub Transport
  3. In the right pane select the Remote Domains tab
  4. Right click Default and choose Properties
  5. On the General tab you can set which type of Out of Office Messages you will allow to be sent out. By default only external OOF messages are allowed. You can change the option to also allow OOF messages created by Outlook 2003 and previous.
    On the tab named “Format of original message sent as attachment to journal report:” (Exchange 2007) or “Message Format” (Exchange 2010) you can enable or disable the automatic replying/forwarding.

The call became interesting, however, when the customer wanted to know if this would affect rules set by the Out of Office Assistant. I couldn’t find a definitive answer, so I set about playing.

I set a test account to have out of office on, auto forward to my external email account (nickxxxxx). In IMF advanced settings on the server I had the following:

I sent a test message via telnet to the test account, and in message tracking saw the following:

The auto forwarded message got stuck at the categorizer stage, which is expected behaviour.

I then enabled automatic forwarding, bounced the store, sent another test message from an external account and got the following:

As you can see in this instance, the message is queued for remote delivery. If my test system were connected to the internet, the message would be delivered.

want to stop it by default, but allow it for some users? This article is your friend:

http://support.microsoft.com/kb/317652

Auditing Extreme Public Folder Deletion

A quick and easy call this morning. We have a customer who has had around 80GB of public folders deleted mysteriously. The change has been replicated round, but they have restored the data and would like to know how it happened, and if there is any way of auditing events that happened in the past. The short answer is no.
Public folders exist within the exchange database, not active directory, so there is no way of tracking it via AD tools. A quick troll through the MS partner forums confirms that there is no other way t ofind this information other than following the procedures below.
It is possible to turn auditing on for exchange 2003 sp2 and later, by adjusting the diagnostic logging for the msexchangeis/public folder/general object to medium. this will produce a 9682 information event in the application event log that looks like this:
9682 info event
In Exchange 2007 sp1 you need to use the shell, and the following command:

set-eventloglevel “msexchangeis/9001 public/general” -level medium.

in sp2 it is possible to use “set diagnostic logging” in the action pane if you select the server object. This also works for Exchange 2010.

diagnostic logging option ex2k10

Once you have the logging enabled you can trawl the event logs using a script from the blog post here:

http://gsexdev.blogspot.com/2005/11/displaying-deleted-public-folder.html
So what caused it? don’t know. my money would be on a user, but it might also be a policy, although the customer says not, or a third party tool that’s been mis-set.

Checking the permissions on the folders would be a good place to start – anyone with owner permission could delete the folder.

Exchange 2003: One foot in the Grave.

Exchange 2003 is getting on, there’s no doubt about it, but there’s still three years to go before it moves into PAL support, so there’s life in the old dog yet.

2003 is my favourite Exchange product. It does what it does well and without making a big song and dance about it. There are no judderingly mad features (like the M: drive!) and it’s relatively easy to administer once you’ve got the hang of the permissions model.

It works. All the features run properly. It doesn’t suffer from any major unfixables. Exchange 2000 had memory fragmentation and instant messaging, Exchange 5.5 had the glass-jawed SMTP service. Pretty much everything in Exchange 2003 is functional. There’s the odd exception – the RUS is pretty pathetic, but most installations aren’t really affected.

It’s mature. There are plenty of hotfixes out there, there’s a huge fund of knowledge on the internet describing how to get it to do stuff. There are plenty of people who know how to administer it, and users (by and large) know what they can expect from it.

It’s robust. OK, this may be a relative thing, but i see about half the calls with Exchange 2003 as i did with Exchange 5.5. things like recovery storage groups, a well understood backup and restore system, outsourced directory all add up to a product that doesn’t cause much pain. Well, anything like as much pain as its predecessors, anyway.

However, to keep it running into the next decade, there are some things we all need to be mindful of. Messages are getting bigger and more frequent. There’s more data out there in general. Constant pressure has resulted in mailbox limits being increased, if they are present at all. There’s a LOT of third party stuff out there that wants a piece of your database. You might well have succumbed to running anti-virus on your Exchange server, rather than separately. There are also some quite specific things that can bite the unwary in the backside.

The biggie is pool memory depletion[i]. Exchange 2003 is a 32 bit system, in an increasingly 64 bit world. If you use the /3GB switch (you DO use the /3GB switch, don’t you?) you are reducing your paged pool to 256MB and non-paged pool to 128MB (on windows 2003, anyway – it’s a different story on 32 bit windows 2008, but, seeing as how you can’t install Exchange 2003 on windows 2008, a different and irrelevant story). “Oh, well, that’s how big it’s always been, we’ve coped for years, we’ll be fine” I hear you say. Well, that’s what they all say, and they’re often wrong. I’ve got two words. Smart. Phone. Oh, and two more. Outlook. 2007. Strictly, 2007 isn’t a word, but hey hoe.

If you’re using Exchange 2003 with Outlook 2003, exclusively, pool depletion will pass you by, most likely, as these two products were designed to work together and be scalable. You may have a problem if you have users in scores or hundreds of security groups, but otherwise don’t worry. You’ll be fine.

Smart phones, Blackberries and later versions of Outlook however are very, very selfish things. A small number of users won’t be a problem, but as they become more and more frequent things will become unpleasant. You’ll start getting users moaning about being disconnected. In the event logs you’ll start to see 9646 errors. You’ll implement the fixes here:

http://support.Microsoft.com/kb/842022

and increase the number of mapi sessions allowed per user[ii]. And then the fun will begin. Outlook 2003 will typically open between five and ten mapi sessions. Blackberries will open significantly more than this. Some smartphones more again. Add in desktop search, active sync etc., and users can be trying to open a hundred or more sessions.

Why is this a problem? Every session requires an access token. Access tokens consume paged pool memory[iii]. Say you have a reasonably typical user, with a token size of about 2KB. They use Outlook 2003 fairly heavily, so are using say, 20KB of paged pool. Generally speaking, we only see problems when token consumption hits about 130MB of paged pool memory. That means you could have 6500 relatively heavy users concurrently connected before you hit pool depletion problems. You’ll run out of CPU, storage or hair before this is an issue, I reckon. Start replacing those users with smart-phone/Blackberry/desktop searchers, with a hundred connections at once.  650 concurrent users. Now start, say, an ad migration, and increase the numbers of security groups they belong to. Token size increases proportionally to 4KB. 300 concurrent users. Once you hit 80 security groups token size doubles to 8kb (One of our customers had multiple users in over 200 security groups – it happens). 150 concurrent users. Worried yet? There’s more on what to do at Mike Lagase’s blog, here: http://blogs.technet.com/b/mikelag/archive/2009/09/15/how-to-monitor-and-troubleshoot-the-use-of-nonpaged-pool-memory-in-Exchange-server-2003-or-in-Exchange-2000-server.aspx

it’s not just paged pool, either – that neat-o VSS backup solution you’ve installed? Gobbling up your non-paged pool.[iv]

Another major problem that’s only recently come to light is transaction log filename depletion. That’s right; we’re running out of things to call our transaction logs.

http://support.Microsoft.com/?kbid=830408

So long as you’re prepared for it, it’s not a problem. Unfortunately, we’ve had a number of customers who were shocked to discover that a finite resource eventually runs out.

So, my top tips for keeping your Exchange server frisky, with a glossy coat and cold nose for the next three years:

Keep an eye on it. If you run the BPA[v] monthly, and are baselining server performance then you’ll have a good idea what’s coming, and when things are going to start falling over. If you have addressed all the issues raised by the BPA, got rid of all the warnings and errors from your event logs and can keep all your perf counters in the green, then yours is the earth and all that’s in it…

Shoot iphone users on sight. If you can’t do this, use the Exchange user monitor[vi] (exmon) to quantify the impact that new versions of Outlook and the like are having on your system. In conjunction with perfmon you can produce some pretty persuasive data. You can also spot the “power” users who are ruining it for everybody else.

Don’t put more than 4GB of physical RAM in your Exchange server. It can’t use it, and it WILL make things worse.

Stop playing call of nature on your Exchange server. Put the base Microsoft video driver on[vii]. You’ll have to make do with Castle Wolfenstein.