Background
Recently I installed and configured a SharePoint 2010 farm at a client site. The environments that were installed and configured are Development, Staging, and Production environments. The development environment is a single Application Server with a separate Database Server. The Staging and Production environments have an Application Server, 2 Web Front End Servers, and a Database Server.
While in operation the end users began complaining about some peculiar behavior. In addition we also noticed some strange errors in the event logs. During my initial investigation of the errors I began to notice a similar theme in my research. The posts I was reading were addressing issues with the RTM release of SharePoint 2010. Therefore I decided to verify the SharePoint release I had installed.
The build number for the RTM release of SharePoint 2010 is 14.0.4762.1000. To validate the build number we installed, I went to the Central Administration site and opened the Manage Servers in Farm page. As suspected the build number I had installed was 14.0.4762.1000.
Upgrade to latest Cumulative Update
Technet provides a good article for upgrading your SharePoint environment to the latest cumulative updates. The article can be found at http://technet.microsoft.com/en-us/sharepoint/ff800847.aspx. After reviewing the article the determined approach was to apply the updates to the Development environment first, Staging environment second, and Production environment third. The approach used was the following.
Backups
· Create a support ticket within the client support systems
· Validate the latest VM Snapshot of the servers
· Create a farm level backup
· Create site collection backups of every site collection
· Create database backups of all of the content databases
Apply Updates
· Apply the latest SharePoint Foundation updates to the Application Server
· Apply the latest SharePoint Foundation updates to the remaining Web Front End Servers
· Apply the latest SharePoint Server updates to the Application Server
· Apply the latest SharePoint Server updates to the remaining Web Front End Servers
· Run the SharePoint Products Configuration Wizard on the Application Server
· Run the SharePoint Products Configuration Wizard on the remaining Web Front End Servers
Testing
· Validate index crawler works
· Validate Content Type Hub works
· Validate Profile Synch works
· Run automated performance web tests created in Visual Studio Ultimate
The Issue
The upgrade was successful in the Development environment. However, as I went to run the SharePoint Products Configuration Wizard on the Application Server on the Staging environment I saw an error that indicated the updates were not installed on the Web Front End servers in the Staging environment.
As the error message showed a disabled Next button I was not able to continue on with the SharePoint Products Configuration Wizard so I had to validate the installation of the updates on the Web Front End Servers. I decided to review the installed updates page under the Programs and Features page of both Web Front End Servers.
After reviewing the Installed Updates page I knew that the updates had been applied to the Web Front End Servers. So I decided to rerun the SharePoint Products Configuration Wizard again only to receive the same message from earlier.
Given I was in a maintenance window and that I needed to get the servers backup within the window I started to search for posts of others encountering a similar situation. I could not find anyone having encountered this scenario so I was left to ponder what should be done to fix it.
The options I considered were the following.
· Create a support ticket with Microsoft and have them work with me to fix it
· Work with the network team to restore the latest snapshot of the server
· Fix it myself
I decided to try and fix this issue myself. My approach was to uninstall SharePoint on the Web Front End Servers. I knew that this would uninstall the updates on the servers. Then reinstall SharePoint back onto the Web Front End Servers and reinstall the updates again. I also determined that if I made the environment worse that I was going to work with the System Administrators to restore the latest snapshot onto the server and that I would restore the backups that I had taken at the beginning of the process.
Root Cause
Before uninstalling SharePoint from the Web Front End Servers a colleague noticed that the SharePoint Timer Service had been stopped on the servers. The reason it had stopped was the password had expired on the farm admin account. Our system administrators require us to go through a 3 month review to gain an exception to the non expiring password policy on domain accounts. Therefore the admin account did not have non expiring passwords.
Resolution
To fix it I went to the Services page of the Web Front End Servers and started the SharePoint Timer Service. Then I went back to the Application Server and reran the SharePoint Products Configuration Wizard. The problem was fixed and I was able to complete the steps on the wizard.
Conclusion
I intentionally encountered the same issue while upgrading the Production environment. Starting the Timer Services on the Web Front End Servers fixed the issue. After the completed upgrade, all of our peculiar errors and bugs have been fixed. The end users are happy again. Cheers.
