SharePoint 2010 Upgrade Standard License To Enterprise License Failure

Background

The client I am working on initially purchased a Standard SharePoint license.  After using the product for about 6 months my client decided to upgrade to the Enterprise license.  The approach to upgrade to the Enterprise edition was the following.

1.       Perform upgrade on the Development environment.  Perform smoke screen tests of the existing content and functionality.  Perform testing of some of the new enterprise functionality.

2.       Perform upgrade on the Staging environment.  Perform smoke screen tests of the existing content and functionality.  Perform testing of some of the new enterprise functionality.

3.       Perform upgrade on the Production environment.  Perform smoke screen tests of the existing content and functionality.  Perform testing of some of the new enterprise functionality.

The upgrade went smoothly in the Development environment but the upgrade did not work in the Staging environment.  The difference between the environments is the Development environment is a single server connecting to a Database server.  The Staging environment consists of 2 Web Front End servers, an Application server, and a Database server.  Below is a description of the error that I was seeing in the log files and the event logs.

Failed to register SharePoint services.
An exception of type Microsoft.SharePoint.Administration.SPUpdatedConcurrencyException was thrown.  Additional exception information: An update conflict has occurred, and you must re-try this action. The object SearchAdminWebService was updated by ***********, in the PSCONFIG (4788) process, on machine *******************.  View the tracing log for more information about the conflict.
Microsoft.SharePoint.Administration.SPUpdatedConcurrencyException: An update conflict has occurred, and you must re-try this action. The object SearchAdminWebService was updated by **********, in the PSCONFIG (4788) process, on machine *******************.  View the tracing log for more information about the conflict.
   at Microsoft.SharePoint.Administration.SPConfigurationDatabase.StoreObject(SPPersistedObject obj, Boolean storeClassIfNecessary, Boolean ensure)
   at Microsoft.SharePoint.Administration.SPConfigurationDatabase.Microsoft.SharePoint.Administration.ISPPersistedStoreProvider.PutObject(SPPersistedObject persistedObject, Boolean ensure)
   at Microsoft.SharePoint.Administration.SPPersistedObject.BaseUpdate()
   at Microsoft.SharePoint.Administration.SPIisWebService.Update()
   at Microsoft.SharePoint.PostSetupConfiguration.ServicesTask.InstallServiceInConfigDB(Boolean provisionTheServiceToo, String serviceRegistryKeyName)
   at Microsoft.SharePoint.PostSetupConfiguration.ServicesTask.InstallServices(Boolean provisionTheServicesToo)
   at Microsoft.SharePoint.PostSetupConfiguration.ServicesTask.Run()
   at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

I posted a thread on the technet forums at http://social.technet.microsoft.com/Forums/en-US/sharepoint2010setup/thread/08756dcf-7735-426a-81df-42ae0202c509.  The response I received was to clear the config cache as mentioned in http://support.microsoft.com/kb/944267.  Although I tried this I still was not able to get the upgrade to work and the same error was being given.

Solution Approach

Given the Development environment was a single server connecting to a Database server.  I decided that it may be worth making the Staging environment similar to that of the Development environment.  My approach was to try the following.

1.       Disconnect the 2 Web Front End servers from the Staging Farm leaving Application server as the only server on the Staging farm

2.       Clear the configuration cache on the Application server

3.       Perform the Enterprise upgrade on the farm by using the Central Administration site

4.       If the upgrade is successful then uninstall SharePoint from the 2 Web Front End servers

5.       Reboot the 2 Web Front End servers

6.       Install SharePoint on the 2 Web Front End servers using the Enterprise license key

7.       Install all of the cumulative updates on the 2 Web Front End servers

8.       Connect both Web Front End servers to the farm

9.       Perform smoke screen tests

10.   Test existing content

11.   Test existing functionality

12.   Test enterprise functionality

The upgrade was successful.  The only downside to this approach is the outage that was taken to disconnect the Web Front End servers from the farm.  The time it took to perform steps 1-8 was about 2 hours.

SharePoint 2010 Cumulative Updates Missing On Web Front End Servers

Background

Recently I installed and configured a SharePoint 2010 farm at a client site.  The environments that were installed and configured are Development, Staging, and Production environments.  The development environment is a single Application Server with a separate Database Server.  The Staging and Production environments have an Application Server, 2 Web Front End Servers, and a Database Server. 

While in operation the end users began complaining about some peculiar behavior.  In addition we also noticed some strange errors in the event logs.  During my initial investigation of the errors I began to notice a similar theme in my research.  The posts I was reading were addressing issues with the RTM release of SharePoint 2010.  Therefore I decided to verify the SharePoint release I had installed. 

The build number for the RTM release of SharePoint 2010 is 14.0.4762.1000.  To validate the build number we installed, I went to the Central Administration site and opened the Manage Servers in Farm page.  As suspected the build number I had installed was 14.0.4762.1000.

Upgrade to latest Cumulative Update

Technet provides a good article for upgrading your SharePoint environment to the latest cumulative updates. The article can be found at http://technet.microsoft.com/en-us/sharepoint/ff800847.aspx. After reviewing the article the determined approach was to apply the updates to the Development environment first, Staging environment second, and Production environment third.  The approach used was the following.

Backups

·         Create a support ticket within the client support systems

·         Validate the latest VM Snapshot of the servers

·         Create a farm level backup

·         Create site collection backups of every site collection

·         Create database backups of all of the content databases

Apply Updates

·         Apply the latest SharePoint Foundation updates to the Application Server

·         Apply the latest SharePoint Foundation updates to the remaining Web Front End Servers

·         Apply the latest SharePoint Server updates to the Application Server

·         Apply the latest SharePoint Server updates to the remaining Web Front End Servers

·         Run the SharePoint Products Configuration Wizard on the Application Server

·         Run the SharePoint Products Configuration Wizard on the remaining Web Front End Servers

Testing

·         Validate index crawler works

·         Validate Content Type Hub works

·         Validate Profile Synch works

·         Run automated performance web tests created in Visual Studio Ultimate

The Issue

The upgrade was successful in the Development environment.  However, as I went to run the SharePoint Products Configuration Wizard on the Application Server on the Staging environment I saw an error that indicated the updates were not installed on the Web Front End servers in the Staging environment.

As the error message showed a disabled Next button I was not able to continue on with the SharePoint Products Configuration Wizard so I had to validate the installation of the updates on the Web Front End Servers.  I decided to review the installed updates page under the Programs and Features page of both Web Front End Servers.

After reviewing the Installed Updates page I knew that the updates had been applied to the Web Front End Servers.  So I decided to rerun the SharePoint Products Configuration Wizard again only to receive the same message from earlier.

Given I was in a maintenance window and that I needed to get the servers backup within the window I started to search for posts of others encountering a similar situation.  I could not find anyone having encountered this scenario so I was left to ponder what should be done to fix it.

The options I considered were the following.

·         Create a support ticket with Microsoft and have them work with me to fix it

·         Work with the network team to restore the latest snapshot of the server

·         Fix it myself

I decided to try and fix this issue myself.  My approach was to uninstall SharePoint on the Web Front End Servers.  I knew that this would uninstall the updates on the servers.  Then reinstall SharePoint back onto the Web Front End Servers and reinstall the updates again.  I also determined that if I made the environment worse that I was going to work with the System Administrators to restore the latest snapshot onto the server and that I would restore the backups that I had taken at the beginning of the process.

Root Cause

Before uninstalling SharePoint from the Web Front End Servers a colleague noticed that the SharePoint Timer Service had been stopped on the servers.  The reason it had stopped was the password had expired on the farm admin account.  Our system administrators require us to go through a 3 month review to gain an exception to the non expiring password policy on domain accounts.  Therefore the admin account did not have non expiring passwords.

Resolution

To fix it I went to the Services page of the Web Front End Servers and started the SharePoint Timer Service.  Then I went back to the Application Server and reran the SharePoint Products Configuration Wizard.  The problem was fixed and I was able to complete the steps on the wizard.

Conclusion

I intentionally encountered the same issue while upgrading the Production environment.  Starting the Timer Services on the Web Front End Servers fixed the issue.  After the completed upgrade, all of our peculiar errors and bugs have been fixed.  The end users are happy again.  Cheers.