Tag Archives: exchange 2007

exchange stuff may 2016

Hi all – it is traditional to start this with “it’s been a while”…

Contents:

Rollups and cumulative updates

Useful and interesting links

Blog articles

Interesting kb articles

Hot news…

FREE exchange 2016 online training course provided by Microsoft on edX, complete with exams and hands on labs (but no videos- it’s all reading, by the look). There are three pretty cheap ones there as well as the initial infrastructure course. It looks like about 20 hours of free stuff (they reckon…). Given the lack of 2016 material on MVA at the moment (all MVA exchange stuff), it’ll have to do, I guess. Give it a go. It’s free.


Rollups

First up – We appear to be falling behind a bit with our rollup rollouts, so it is likely there will be a new Exchange Stabilisation project shortly. For the avoidance of doubt, I’ll restate the Microsoft support policies:

Exchange 2013/2016 – you are supported by Microsoft on the latest and next latest CU only.

Exchange 2010 – if you are on service pack 3 you are in extended support until November 2020

Exchange 2007 – if you are on service pack 3 you are in extended support until November 2017

Exchange 2003 – you are unsupported. No, really.

With 2007/2010, while you may be supported on sp3 rollup1, it is my experience that unless the problem is a simple configuration issue you will be asked to update to the latest rollup as part of the troubleshooting process. They hate analysing old code.

the latest rollups were released March 15th, and are here:


Useful and interesting links

As part of an investigation last week, I came across a highly useful article that references loadbalancer settings for exchange 2013 – they are the same as for 2010, but it’s nice to have that stated explicitly – it also suggests a great list of other useful and exciting things to do to stave off ever having to log a support call. I know I sent it out earlier, but it bears repeating.

Our colleague Mark Bodley has hunted out a number of extremely useful links:

Database corruption and dirty shutdown decigeons* tree on Exchange server pro. There’s a picture. It’s great. print it out and stick it over your desk.

He emailed a reminder of how to upgrade the CU on Exchange 2013 and 2016, along with some extremely useful links to scripts written by our friend and colleague, Michael Van Hybrid (yes, he’s on the GAL!):

  Not sure if you are aware that the recommend steps for installing CU updates on E2013/2016 are a bit different to what we may have gotten used to with Exchange 2010 – assuming that your local hostile SDM ever agreed to an update!!

There’s an article on the steps required for Exchange 2013 SP2 2016 at http://exchangeserverpro.com/installing-cumulative-updates-on-exchange-server-2016/ with 2013 being quite similar. Confusingly though Exchange continues to ship with the StartDAGServerMaintenance.ps1 scripts, these appear really to be for 2010 and should not be used on 2013 or later.

Michael van Horenbeeck has written 2 scripts for starting and stopping maintenance mode on Exchange 2013 and as they support a multi-role deployment they should work for 2016. Certainly looking at the above article, the steps for 2016 are the same as for an E2013 Multi-role server.

The scripts are at :-

· Stop-ExchanegServerMaintenanceMode.ps1 : https://gallery.technet.microsoft.com/scriptcenter/Exchange-Server-2013-77a71eb2

· Start-ExchanegServerMaintenanceMode.ps1 : https://gallery.technet.microsoft.com/scriptcenter/Exchange-Server-2013-ff6c942f

The exchange 2010 search troubleshooting guide – invaluable. Note this is for exchange search, not instant search – outlook uses exchange search in online mode, and instant search in cached mode – instant search is based on windows search, and searches the .ost file, not the server. Shall I bore you with exchange store search…? No? ok. Be aware that exchange 2013 has a different search engine again – Microsoft search foundation. try this article.

Understanding the Outlook Connection Status window. Recently (slightly) updated.

How to troubleshoot free/busy issues in a hybrid deployment of on-premises Exchange Server and Exchange Online in Office 365

The latest guided walkthroughs for Office servers – lync, exchange online, office 365 etc – are here. Some of them are brand shiny and new, others are a little long in the tooth. PF repl for 2003? Wow.


Blog articles

Rhoderick Milne explains why Network Location Profiles are giving you neckache – I’ve come across this myself, it stops Exchange working properly, and is tricky to spot.

The PFE Exchange 2013 tips and tweaks post (also Mr Milne). The 2010 article was a standard. You need to read this if you look after 2013, or are about to. CSAs should read it also…

Anyone looking at moving from 2007 to 2013 should review the links from this 2013 upgrade workshop.

My friend Ingo has an update to his activesync user script here: Get-ActiveExchangeUsers 2.0

Released: March 2016 Quarterly Exchange Updates on the official “you had me at EHLO…” blog.

Deferred Lagged Copy playdown in Exchange 2016

Messing around with how powershell proxying works causes headaches. Read this to make sure you are aware of how cu11/cu12 will affect you.

Exmon is finally available for Exchange 2013 and 2016


Interesting KB articles

Outlook

May 3, 2016, update for Outlook 2016 (KB3115101)

Office 2016 Applications crash or cannot start

Lync 2013 (Skype for Business) or Outlook 2013 Crash after installing the april 2016 upates

Performance problems when you try to access folders in a secondary mailbox in Outlook –I know at least one account is struggling with this.

Exchange

“The remote server returned an Error 404” or "HTTP request has exceeded the allotted timeout" error when you move a mailbox from on-premises Exchange Server to Exchange Online

Intermittent "500" error occurs for EWS requests in an Exchange Server 2013/2007 coexistence scenario

"Cannot display the folder properties" or "could not be updated" error when Exchange hybrid deployment users open a room calendar in Outlook

Can’t open a shared folder in Outlook on the web in Exchange Server

Users in your Exchange 2013-based hybrid deployment experience mail issues after April 15, 2016

Incorrect output when you run the Get-CASMailbox cmdlet to view the HasActiveSyncDevicePartnership attribute

"Cannot display the folder properties" or "could not be updated" error when Exchange hybrid deployment users open a room calendar in Outlook

"Nullable object must have a value" error when you run the Hybrid Configuration wizard

"Execution of the Get-WebServicesVirtualDirectory cmdlet has thrown an exception" error when you run the Hybrid Configuration wizard

Exchange Online users cannot access free/busy information of users in a non-Internet-facing Active Directory site

"The user isn’t assigned to any management roles" error when you run the Hybrid Configuration wizard

"Secure Mail Certificate on server is not bound to the SMTP Service" error when you run the Hybrid Configuration wizard

"The SMTP address template is invalid" error when you run the Hybrid Configuration wizard

"The length of the property is too long. The maximum length is 64" error message when you run the Hybrid Configuration wizard

"RequiredTls flag should be set to true if TlsCertificateName is specified" error when you run the Hybrid Configuration wizard

"The remote server returned an error: (403) Forbidden" error when you try to move mailboxes from on-premises Exchange Server to Exchange Online

"The term ‘Get-HybridMailflowDatacenterIPs’ is not recognized" error when you run the Hybrid Configuration wizard

On-premises users in an Exchange hybrid deployment can see availability but not capacity or description information of a resource when they schedule a meeting

Can’t reserve a resource for a meeting after the resource mailbox is migrated to Exchange Online

"An error occurred while working on your domain" when you try to verify your domain in Office 365 in an Exchange hybrid deployment

Users in a hybrid deployment can’t access a shared mailbox that was created in Exchange Online

Out-of-office replies and voting options in email messages between on-premises users and Exchange Online users do not appear correctly in a hybrid deployment

Slow mail delivery in an Exchange environment that has transport rules configured – I’m particularly keen that you understand this article – I’ve been asked a few times lately about transport rules, so anyone thinking of implementing them needs to understand that they can have a performance impact.

Outlook Anywhere users prompted for credentials when they try to connect to Exchange Server 2013 or Exchange Server 2016

Information about the Calendar Checking Tool for Outlook (CalCheck)

Feedback is of course welcome.

 


Roderick, by John Sladek. Probably my favouritest book about robots ever.

clip_image001

Advertisements

Some useful exchange monitoring tools

Quite often I get asked about ways to find out basic stuff about exchange. Basic stuff that, frankly, is often quite difficult to find out unless you are already a powershell wizard. Now, we should all be working toward becoming powershell wizards, but if you’re like me, you may not be there quite yet…

 

So here’s a collection of tools that may be of use. Some of these are a little old – 18 months or so, but I was prompted to dig them out after seeing the new “exchange reports” utility on codeplex (http://exchangereports.codeplex.com/downloads/get/635540#). It’s very pretty but is unfortunately an .exe, which may make it tricky for some people.

exchangereports1exchangereports2

 

 

 

 

 

 

 

 

 

I’d suggest you download it and play with it in your labs – I certainly will.

 

If executables are bit out of your price range, then here’s a few powershell scripts that are worth investigating. The first one is from Steve Goodman – it’s a little long in the tooth, but has just been updated and had some bugs fixed.

 

http://www.stevieg.org/2011/06/exchange-environment-report/

 

this is great – total number and average size of mailboxes, DB size and the amount of white space, last full backup and lots more besides. Of course, being a powershell script, the software is already installed on your server – you could just type it all out longhand and run it.

 

Finally here’s a couple from Paul Cunningham at ExchangeServerPro. First a DAG health checker (http://exchangeserverpro.com/get-daghealth-ps1-database-availability-group-health-check-script) which will check the replication state of copies and content indexes and then a more comprehensive checker (http://exchangeserverpro.com/powershell-script-health-check-report-exchange-2010). This second one takes a bit of configuring, but runs a series of automated checks – hopefully you’re running  a chunk of this stuff already with SCOM. You can run it to test a particular server:

.\test-exchangeserverhealth.ps1 –server <servername>

exchangeserverhealth

 

 

Dns – is the name resolvable?

Services are running?

Total queue is the sum of items in all the queues on that server.

Mailbox test is on a server at random.

 

Run the without parameters to go through every server in the organization. This may take an hour or more for very large organisations (ie lots of servers, rather than lots of mbxs). If you add the following parameters:

.\test-exchangeserverhealth.ps1 –reportmode $true  -sendemail $true

The script will send a html report to the email address in the Email Settings section.

 

The great thing about powershell scripts is they are very easy to reverse engineer and repurpose if necessary. In the last article there’s also a section on how to schedule the script, which is easily applicable to the other scripts if you prefer them, or want to run them as well.

 

 

Enhanced by Zemanta

Folder item limits for various versions of Exchange

UPDATE, Nov 2015: I’ve added the item counts for Exchange 2013 and 2016, and the limits for Outlook in cached mode.

High item count may not be the root of all evil, but it’s certainly up there. What constitutes high item count varies according to your environment, but it’s pretty low in exchange 2003. It’s possible to mistake a high item count issue for poor hardware performance, and in any scenario where you suspect your hardware, it’s worth looking for high item count folders and trying to reduce them to the recommended maxima, or possibly below. The recommended maxima are for “good” hardware environments and optimal configurations, not your mess. You should be familiar with the literature, and think of item count whenever presented with a performance issue.

Folder item count has a huge effect on Exchange performance; more so than mailbox size. Much of the time, rather than looking at a folder, a user may be looking at a server generated view of a folder, at which point item count becomes crucial to how long it takes the server to put together the view, as each item is evaluated.
Most of our customers are familiar with the Exchange 2007 article “Understanding the Performance Impact of High Item Counts and Restricted Views”, and many have noticed the Exchange 2003 limits given in there (If you haven’t read this article, then you really, really should). What many miss, however, are the caveats Microsoft put in there. Abiding by these limits you may expect to achieve “acceptable” performance.

“With properly architected hardware”, (and exchange 2007), “an acceptable user experience can still be maintained with item counts as high as 20,000 items.”

The definition of “acceptable” may not be what you expect, however. From the same article:

“This recommended maximum also depends on the performance capability of your Exchange environment. Your specific hardware choices may result in lower maximum numbers. Ideally, it is best to keep the Inbox and Sent Items folders less than 20,000 items, and the Contacts and Calendar item counts less than 5,000. Even when maintaining item counts that are at or under the recommended maximum values, there are some operations which may still take noticeable time (usually this is approximately one minute).”

So, with properly specified and correctly performing hardware, and with an item count below the maximum recommended limit, there are some operations that will have some of your users gnashing their teeth with frustration. I know this, because people log calls with us about delays of a minute. They generally don’t call it a minute, however; usually, it’s “literally hours”.

It’s also worth noting that for commonly used folders, the “critical” ones, the limits are very much lower. The real bad boy is the calendar folder, as that is the one most likely to be accessed by other users, and requires a search to filter out private items. 20,000 calendar items will bring a server to its knees, I’ve found.

Also, the Microsoft published maxima are not necessarily the same as those the engineers believe to be reasonable. For instance, Nicole Allen, an Outlook engineer writing in the official exchange technet blog, recommends that critical folders on Exchange 2003 have no more than 1000 items in them.

So, bearing that in mind, what does good look like?

Version General Folders Critical Folders (inbox, calendar, contacts, sent items)
Exchange 2003 5,000 1,000
Exchange 2007 20,000 5,000
Exchange 2010 100,000 10,000
Exchange 2013 1,000,000 1,000,000
Exchange 2016 1,000,000 1,000,000

Critical folders vary depending on how your users interact with each other – most of us only allow one or two people to our inbox, so for 2007 and 2010 Microsoft don’t refer to it as critical; Calendar and Contacts most definitely are, though.

To summarise sources, and extract a sentence or two from each that I think you should particularly watch for:

Exchange 2003 (http://blogs.technet.com/b/exchange/archive/2005/03/14/395229.aspx):

5000, Nicole allen says 1000 for inbox and calendar, here.

I usually recommend no more than about 2500 – 5000 messages in any of the critical path folders.  The critical path folders are the Calendar, Contacts, Inbox, and Sent Item folder. Ideally, keep the Inbox, Contacts and Calendar to 1000 or less.  Other folders, particularly custom folders created by the user, can handle having larger numbers of items without having a broad impact on the user experience (20,000 items in my “Cookie Recipes” folder?  No problem – except when I need to find that recipe from last Christmas!).

exchange 2007 (http://blogs.technet.com/b/exchange/archive/2005/03/14/395229.aspx):

20000, but keep critical path folders such as inbox and calendar below 5000:

With Exchange Server 2003, the recommended maximum item count per folder was 5,000 items. In Exchange 2007, improvements in I/O, larger page size, and increased cache can help enable an increase in the recommended maximum item count. With properly architected hardware, an acceptable user experience can still be maintained with item counts as high as 20,000 items.

exchange 2010 (http://technet.microsoft.com/en-us/library/ee832791.aspx ):

100,000, but no more than 10,000 in calendar and contacts.

“A challenging scenario occurs when a user has exceeded the number of indexes that Exchange will store. This is 11 indexes in Exchange 2010. When the user chooses to sort a new way, and thereby creates a twelfth index, this causes additional disk I/O activity. Because the index isn’t stored, this additional disk activity cost occurs every time that this sort is performed. Because of the high I/O activity that can be generated in this scenario, we strongly recommend that you store no more than 100,000 items in core folders, such as the Inbox and Sent Items folders, and no more than 10,000 items in the Calendar and Contacts folders. The creation of more top-level folders, or of subfolders beneath the Inbox and Sent Items folders, greatly reduces the costs that are associated with this index creation. This is true as long as the number of items in any folder doesn’t exceed 100,000.”

Exchange 2013 and 2016:

The recommended limit for 2013 is given in a footnote to the table “Mailbox folder limits across standalone plans” in the Exchange Online Limits article. It’s the same as the hard limit in Exchange Online; 1 million items. In his presentation on the Exchange 2016 Preferred Architecture given at Ignite in May 2015, Ross Smith IV states that the recommended upper limit for 2016 is also 1 million items. He states a bunch of other interesting stuff also. go and watch it.

Outlook Cached Mode (https://support.microsoft.com/en-us/kb/2768656):

Yeah, all this nonsense applies to Exchange… so what about Outlook? These figures are for cached mode.

Version Total Folders Items per Folder
Outlook 2007 500 50,000
Outlook 2010 500 100,000
Outlook 2013 500 100,000
Outlook 2016 ? ?

There are also interactions between exchange and different versions of outlook. Outlook 2003 in particular needs to be on SP1, as there is a bug in the RTM version. Of course, you’re all on SP3, by now, and looking to upgrade before it goes completely out of support in April 2014. Outlook 2007 and Outlook 2010 have differing caching behaviours; outlook 2010 will cache other users’ MAIL folders by default, which can lead to some long delays.

So what are the indicators of a possible high item count issue? From a user perspective, “oh my god it’s so sloooooow…” is a common one. But then, it always is. Take a perfmon, and have a look – there may be obvious things on there, like disk access or high cpu, which initially will look like a hardware bottleneck. It would be easy, at this point, to say “hardware’s rubbish” and go down the canteen, rather than start running powershell commands to find high item count folders or using pfdavadmin. Sometimes, however, there’s no obvious hardware bottleneck – what then? Have a look at the “client related search” counters under msExchangeIS mailbox – the two biggies are slow findrow rate, which is explained by Mike Lagase here, and slow qp threads.

Everyone should read the following two articles on this, the first from an outlook perspective, and the second from an exchange one.
Outlook users experience poor performance when they work with a folder that contains many items on a server that is running Exchange Server
Understanding the Performance Impact of High Item Counts and Restricted Views

Automating collection of performance monitor counters for Exchange, on windows 2008.

Summary

It is frequently necessary to collect performance counters to troubleshoot problems on servers. Unfortunately, quite often the condition we are trying to troubleshoot is transient, and by the time an administrator is alerted to the problem it has already passed. This document describes a method to set up automated performance monitoring that can be triggered by a particular performance condition being met.

It involves setting up a performance alert to write an event to the application log, and then using the latest version of task scheduler to start a batch job based upon the appearance of that event.

This article applies to Windows 2008 and 2008 R2, and Exchange 2007 and 2010.

Method

Part 1. Setting an alert condition

First, set up alert condition by following http://technet.microsoft.com/en-us/library/cc722414.aspx

To create a Data Collector Set to monitor Performance counters

  1. To open Windows Performance Monitor, click Start, click in the Start Search box, type perfmon, and then press ENTER.
  2. In the Windows Performance Monitor navigation pane, expand Data Collector Sets, right-click User Defined, point to New, and click Data Collector Set. The Create new Data Collector Set Wizard starts.
  3. Enter a name for your Data Collector Set.
  4. Select the Create manually option and click Next.
  5. Select the Performance Counter Alert option and click Next.
  6. Click Add to open the Add Counters dialog box. When you are finished adding counters, click OK to return to the wizard.
  7. Define alerts based on the values of performance counters you have selected.
    1. From the list of Performance counters, select the counter to monitor and trigger an alert.
    2. From the Alert when drop-down, choose whether to alert when the performance counter value is above or below the limit.
    3. In the Limit box, enter the threshold value. Eg, if you want to gather performance data when cpu utilisation is greater than 95%, enter “95”
  8. When you are finished defining alerts, click Next to continue configuration.
  9. After clicking Next, you can configure the Data Collector Set to run as a particular user. Click the Change button to enter the user name and password for a different user than the default listed.
Note
  • If you are a member of the Performance Log Users group, you must configure Data Collector Sets you create to run under your own credentials. Data Collector Sets run as the System user by default. As a security best practice, you should accept this default value unless you have a compelling reason to change it.
  1. Click Finish to return to Windows Performance Monitor.
  • To view the properties of the Data Collector Set or make additional changes, select Open properties for this data collector set. For more information about the properties of the Data Collector Set, see Data Collector Set Properties.
  • To start the Data Collector Set immediately (and begin saving data to the location specified in Step 8), select Start this data collector set now.
  • To save the Data Collector Set without starting collection, select Save and close.

To configure alert actions

  1. Expand Reliability and Performance in the navigation pane.
  2. Expand Data Collector Sets, expand User Defined, and click the name of the Data Collector Set with performance counter alerts.
  3. In the console pane, right-click the name of a Data Collector whose type is Alert and click Properties.
  4. On the Data Collector Properties page, click the Alerts tab. The data collectors and alerts already configured should appear.
  5. Click the Alert Action tab to choose to write an entry to the event log Applications and Services Logs/Microsoft/Windows/Diagnosis-PLA/Operational when the alert criteria are met. You can also start a Data Collector Set when the alert criteria are met; Don’t do this.
  6. Click the Alert Task tab to choose a Windows Management Interface (WMI) task and arguments to run when the alert criteria are met. Don’t do this.

This will give you  an alert that writes a 2031 event to the log named in step 15, above.

Part 2. Creating the batch file

First, set the PowerShell environment. Open the Exchange Shell and run the following:

Set-ExecutionPolicy RemoteSigned

Then download and extract the ExPerfWiz powershell script from the Microsoft website:

http://archive.msdn.microsoft.com/ExPerfwiz

Make sure to download the latest version – 1.3.7 or higher.

Then Create a batch file to run the experfwiz script:

  1. Open notepad
  2. Copy the text below into the open document, and save it as C:\perfwiz\experfwiz.bat, or something equally usable. Try and avoid spaces or non standard characters in your path.

PowerShell.exe -command “. ‘c:\Program Files\Microsoft\Exchange Server\V14\bin\RemoteExchange.ps1’; Connect-ExchangeServer -auto; c:\experfwiz\experfwiz.ps1 -duration 00:15:00 -interval 1 -quiet”

  1. You will need to check:

Location of Exchange binaries for the RemoteExchange.ps1 script.

Location of ExPerfWiz script

Version of ExPerfWiz script – only 1.3.7 and later support the –quiet switch.

Change duration and interval to something suitable. In the example above the duration is 15 minutes, and the interval is two seconds.

Part 3. Scheduling the batch file

The batch file will be run by setting up a scheduled task that will run every time an event id is logged. The Event ID that you will use to trigger the Log:

Log: Microsoft-windows-diagnosis-PLA/operational

Source: Diagnosis-PLA

Event id:  2031

Now to create a Scheduled Task.

  1. If Task Scheduler is not open, start Task Scheduler. For more information, see Start Task Scheduler.
  2. Find and click the task folder in the console tree that you want to create the task in. If you want to create the task in a new task folder, see Create a New Task Folder to create the folder.
  3. In the Actions Pane, click Create Task.
  4. On the General tab of the Create Task dialog box, enter a name for the task. Make sure it is running under an admin account, and that it is set to run only when the user is logged on. You will need to remain logged on to the server for the duration.
  5. On the Triggers tab of the Create Task dialog box, click the New… button to create a trigger for the task, and supply information about the trigger in the New Trigger dialog box. Select On an Event to start the task.Fill in the details for the event as shown in the picture below.
  6. On the Actions tab of the Create Task dialog box, click the New… button to create an action for the task, and supply information about the action in the New Action dialog box. Select Start a Program, and browse to the batch file you created in part 2, above. Once this action is saved you can, if you choose, set it to also send you an email when the condition occurs.
  7. Click the OK button on the Create Task dialog box.

Part 4. Final checks, and what to do next.

  • Make sure that the data collector set is running.
  • Make sure that the batch file will work by starting it manually. Check the c:\perflogs folder for the generated log file.
  • Make sure that the process works – do this by setting the data collector set to alert on a commonly met condition, such as cpu utilization = 20%. Check that:
    • The event is logged in the Applications and Services Logs/Microsoft/Windows/Diagnosis-PLA/Operational event log.
    • The scheduled task is triggered by selecting the task, and checking the “history” tab in the lower window of the scheduler console.
    • The perfmon log is generated and saved to the C:\perflogs directory.

When the condition is met, the task triggered, the batch file ran and the log generated, why not use PAL to analyze it. I would.

The batch file doesn’t have to point to ExPerfWiz, of course. It could quite easily point to a batch file that triggers logman, or you could avoid using a batch file altogether and set perfmon up to trigger a data collector set instead of an event. For more details on using logman with a batchfile see our nutshell “Scripting Perfmon for Win XP through to Win 2008”

Thanks to Mike Lagase for the excellent ExPerfWiz script, and Amit Tank at exchangeshare.wordpress.com for his article on scheduling Exchange tasks. And obviously thanks to technet, where I’ve ripped great chunks of this from.

References:

UPDATE:

if you want the script to mail you a little email to say it’s done, or similar, then you’ll want to read Morgen Simonsen’s blog, here:

http://morgansimonsen.wordpress.com/2009/12/15/exploring-task-scheduler/

 

you may also find it useful to have a mail method that avoids CDOSYS altogether:

http://stackoverflow.com/questions/11868120/task-scheduler-to-send-email-through-batch

Recovery Storage Groups are Making Your Life Hell

An interesting call this week. High severity issue with a CCR cluster with geographically separated nodes. The customer was following http://technet.microsoft.com/en-us/library/bb676320%28EXCHG.80%29.aspx, which is the TechNet article on how to patch a CCR to sp1 or 2 – it’s valid for sp3 as well, but MS haven’t updated the article to reflect this. They customer had got to step 9, but things were then going wrong trying to move the cluster from the active node to the passive node (top tip: “active” and “passive” refer to the state of the nodes; don’t use those words to name your nodes, or we will fall out). The move would fail, and then fail to move back to the original node as well, leaving the cluster in a down state. The quick fix to restore service was to shut both nodes off, then power up the sp2 machine. Once the sp2 machine was running, the sp3 machine was turned on. At this point we were called.

First things first was to get a worst case action plan sorted. Sp3 cannot be uninstalled (http://technet.microsoft.com/en-us/library/ff607233%28EXCHG.80%29.aspx) so basically, they would need t ouninstall exchange, evict the node from the cluster, reinstall exchange and recluster. Henrik Walther has documented this process perfectly in his blog: http://www.msexchange.org/articles-tutorials/exchange-server-2007/high-availability-recovery/re-installing-cluster-nodes-exchange-2007-ccr-based-mailbox-server-setup-part1.html. the second part is linked from that page. You can do this with 100% availability, pretty much (except for the failover). Reseeding the db can be done online.

 

Once the customer was happy that we had a backout plan we got some basic troubleshooting evidence collected; a bpa in healthcheck mode, mps reports from both nodes, with cluster and exchange options ticked.

With the collection under way, we started to look at the state of the cluster.

The “clustered mailbox server” tab in the properties of each node showed everything ok – both nodes were listed on each machine, the correct node was listed as operational.

get-storagegroupcopystatus showed all storage groups as healthy, with copy and replay queue lengths of 0, and a timely last inspection timestamp. All storage groups except the recovery storage group, that is. Not supported.

Get-ClusteredMailboxServerStatus (http://technet.microsoft.com/en-us/library/aa998632%28EXCHG.80%29.aspx ) and
Test-ReplicationHealth (http://technet.microsoft.com/en-us/library/bb691314%28EXCHG.80%29.aspx ) likewise showed everything cool.

in system event logs i could see event id 1069, source: clussvc
Cluster resource ‘rsg1db1/sg4db1 (<servername>)’ in Resource Group ‘<servername>’ failed.
more details here:
http://www.microsoft.com/technet/support/ee/transform.aspx?ProdName=Windows%20Operating%20System&ProdVer=5.2&EvtID=1069&EvtSrc=ClusSvc&LCID=1033
this implicates the recovery storage group that they shouldn’t be running in the problem. the recovery storage group that can’t take part in a clustered environment, but is a resource in the cluster. hmmm.

A little digging got me to this:

Databases in an RSG cannot be set to mount automatically when the Exchange Information Store service is started. You must always start the databases manually. If mounted at the time of a cluster failover, databases will not mount automatically after failover is completed.
From:
http://technet.microsoft.com/en-us/library/bb124039%28EXCHG.80%29.aspx
the implication being if the rsg being online is a dependency, then failover will not complete successfully in either direction.

 

Now, the literature all states that while a RSG cannot mount, it doesn’t say that it will prevent failover. However as it was set as a cluster resource (as shown in the 1069 error, above) in this case it will cause failover to crash out when it doesn’t come online.

 

The agreed plan was that the customer would remove the rsg, as per the best practice article here: http://technet.microsoft.com/en-gb/library/aa995895%28EXCHG.80%29.aspx

Once this was complete, they would rerun the prereq tests, repatch the sp3 server to ensure that there was no issue there, and failover the cluster as per step 9 of the upgrade document. With no RSG to bugger things up, this went great. They successfully patched their now passive sp2 node the following day, and away they go…

 

So to summarise, recovery storage groups are making your life hell. If you’re not using them, get rid of them. Don’t have them hanging about on your box. Especially not if it’s a cluster.

Auditing Extreme Public Folder Deletion

A quick and easy call this morning. We have a customer who has had around 80GB of public folders deleted mysteriously. The change has been replicated round, but they have restored the data and would like to know how it happened, and if there is any way of auditing events that happened in the past. The short answer is no.
Public folders exist within the exchange database, not active directory, so there is no way of tracking it via AD tools. A quick troll through the MS partner forums confirms that there is no other way t ofind this information other than following the procedures below.
It is possible to turn auditing on for exchange 2003 sp2 and later, by adjusting the diagnostic logging for the msexchangeis/public folder/general object to medium. this will produce a 9682 information event in the application event log that looks like this:
9682 info event
In Exchange 2007 sp1 you need to use the shell, and the following command:

set-eventloglevel “msexchangeis/9001 public/general” -level medium.

in sp2 it is possible to use “set diagnostic logging” in the action pane if you select the server object. This also works for Exchange 2010.

diagnostic logging option ex2k10

Once you have the logging enabled you can trawl the event logs using a script from the blog post here:

http://gsexdev.blogspot.com/2005/11/displaying-deleted-public-folder.html
So what caused it? don’t know. my money would be on a user, but it might also be a policy, although the customer says not, or a third party tool that’s been mis-set.

Checking the permissions on the folders would be a good place to start – anyone with owner permission could delete the folder.