EventLog Error Fix

Overview: After building my farms I trawl through the ULS and event logs to look for logs messages to identify any issues.  This post contains errors from my event logs that hopefully will help me in future.

Problem: My event log shows a Windows/IIS error whereby the IIS sites application pool uses a service account that does not have a user profile on the machine.  The error message reads "Event Id: 1511 Windows cannot find the local profile and is logging you on with a temporary profile. Changes you make to this profile will be lost when you log off."

Verify the issue:

Resolution: (IEDaddy's post gave me the resolution)
1.> Stop the processes that use the account (I stopped the web sites that used the application pool account "demo\OD_Srv")
2.> cmd prompt> net localgroup administrators demo\OD_Srv /add
3.> cmd prompt> runas /u:demo\OD_Srv /profile cmd
4.> in the new cmd prompt run > echo %userprofile%
5.> Check the user profiles and verify the profile store for the account (demo\OD_Srv) has a status of "Local"
6.> Remove the account from the local administrators group ie cmd> net localgroup administrators demo\OD_Srv /delete

More Info:
http://www.brainlitter.com/2010/06/08/how-to-resolve-event-id-1511windows-cannot-find-the-local-profile-on-windows-server-2008/
http://todd-carter.com/post/2010/05/03/give-your-application-pool-accounts-a-profile/

*************************

Problem: EVENT ID: 8321 - Task Category: Topology
A certificate validation operation took 120053.1569 milliseconds and has exceeded the execution time threshold. 

Resolution:  I performed various steps:
1.> Host entry add the host entry:
127.0.0.1  crl.microsoft.com
2.> Trust the SP root cert
http://support.microsoft.com/kb/2625048

Import the Trusted certificate

3.>  Reduce the time when the crl check is done (not a fix but it will fail quicker and carry on)

This post may also help - but it wasn't my issue: http://stevesps.blogspot.co.uk/2013/01/sharepoint-foundation-event-id-8321.html

 *************************

Problem: Event Id: 8313 - Task Category: Topology
A failure was reported when trying to invoke a service application: EndpointFailure
Process Name: w3wp
Process ID: 5640
AppDomain Name: /LM/W3SVC/1647355528/ROOT-1-13036555135555957
AppDomain ID: 2
Service Application Uri: urn:schemas-microsoft-com:sharepoint:service:649a3e7c090555059555c7a101555576#authority=urn:uuid:55b29cf855594c76555658fca66dac65&authority=https://sv-sp-web1:32844/Topology/topology.svc
Active Endpoints: 2
Failed Endpoints:1
Affected Endpoint: http://sv-sp-app2:32843/649a3e7c0904495552e4c7a555d64555/MetadataWebService.svc

Initial Hypothesis: It looks like the Web front ends cannot coomunicate with the MetadataWebService.svc, run mmc > file > add/remove snapin > snap-in "certificates" > Add > Computer Account > Local Computer > OK.
Expand "Certificates" > SharePoint > Certificates.  Open the certs and check if they are verified.  In my case my wfe's are good but my app servers do not have a valid certificate as shown below.

Resolution:
PS> $rootCert = (Get-SPCertificateAuthority).RootCertificate
PS> $rootCert.Export(“Cer”) | Set-Content C:\SharEPointRootAutority.cer –Encoding Byte

http://khalidstech.blogspot.co.uk/2012/11/certificate-validation-errors-in.html

Automation to add the SharePoint Root Certificate is done very nicely in this post: http://lennytech.wordpress.com/2013/06/18/powershell-install-sp-root-cert-to-trusted-root/

 **************

Disable CRL check (I believe this is from AutoSPInstaller)
Set-ItemProperty -path "HKCU:\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing" -name State -value 146944
set-ItemProperty -path "REGISTRY::\HKEY_USERS\.Default\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing" -name State -value 146944
get-ChildItem REGISTRY::HKEY_USERS | foreach-object {set-ItemProperty -ErrorAction silentlycontinue -path ($_.Name + "\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing") -name State -value 146944 

************* 

Problem: Event Log is capturing EventId: 2159 Source: SharePoint Foundation Error message refers to Event 8306 within the ULS logs.

Resolution: Edit the web.config allowing the ULS to capture additional information relating to the error.  The resulting error show the common SharePoint COM class factory error.  In this scenario changing the "SecurityTokenService" app pool "Load User Profile" property to true correct the underlysing issue.

***************
 

Digital Signatures and Install Software gotcha

Problem: In automating SQL Server and SharePoint images, the actual installation is taking a long time on my managed environment whereas my developer laptop is fast.  All installations are done without Internet access (offline).

I have a dev environment build on my laptop that runs SSD and I run 3 VM using VMware workstation 9 (all use Windows 2008 R2 SP1).  I create an 1) AD with 1GB or RAM and 1 CPU 2) SQL 20012 with 10GB RAM and 4 CPU's 3) SP2010 CU Aug 2012 10GB 4 CPU's.  All the installation is automated using slip streamed images.

So for simplicity on the CI environment I will explain a simplified comparable setup. I have 3 machines with the same roles however the SQL 2012 and SP2010 install take considerably longer.  The CI environment is on ESX (Cisco blades & chassis, and Violin (SSD) storage.  The CPU/compute is connected to the storage via SAS/Fibre channel made no difference either).  I have summarised the results below:

                                                                 SQL2012 (duration)      SP2010 (duration)
Laptop(VMworkstation Workstation)           15 min                              16 min
CI (ESX)                                                        22 min                              92 min 

Finding: My hardcore/good ESX infrastructure is taking 9 minutes longer to install SQL Server 2012 on beter hardware and an amazing 70 minutes longer to install SP2010.

Update 21 Feb 2013: Don't use PowerShell 3 with AutoSPInstaller (including using the version switch i.e. -version 2), it doesn't work and even changing AutoSPInstallers internal web call fail.  It can be made to work with the version 2 switch but it isn't worth the effort.

Initial Hypothesis:
After many many hours between service providers managing the infrastructure, it was not hardware or ESX configuration/setup.  However if the network card on the VM is disabled, the performance change improves to:

                                                          SQL2012 (duration)            SP2010 (duration)
CI (ESX)                                                     13 min                       5 min and 5 seconds

Pretty hefty improvement.  Using netstat is looks like there are requests to the Internet.  After adding Wireshark to monitor all traffic.  I can see requests being sent to crl.mirosoft.com (certificate revocation lists) and ctldl.windowsupdate.com

Issue shown in Wireshark

Issue Shown in Fiddler

This is the 1st time I have seen this issue in a clients production environment.  If the WFEs/SP servers have internet access (less preferable) or the servers don't have access the install work in a timely fashion.  The symtoms of the issue are when the WFE's/SP Servers don't have internet access but think they do.  All the binaries are digitally signed and the install will try validate the signatures despite this being an offline install.

I confirmed the problem being how the networking is setup.  My issue shows up on the VM NIC adapter, Originally the IPv4 Connectivity has a status of "No Internet Access", once I ping google I get a reply and the status changes to "Internet".  I can ping google but not browse to it.

Resolution:  The problem is that executable code is digitally signed.  This is good, all code should be digitally signed so it can be authenticated.  However in this situation a lot of requests are being sent out from the VM as the install tries to verify all the SharePoint complied code.  The install on the local VM acts as if there is an Internet connection (which there is not).

It takes unique networking to get into this issue and SP/any digitally signed code will check the digital certs.

There are a few fixes such as:
1.> Allowing the servers to get out to the Internet, so open the firewall or set a proxy on the local VM.
2.> Add host entries to the cert fails immediately but will continue installing (This is not working for me).
3.> Make the following registry change:

set-ItemProperty -path "HKCU:\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing" -name State -value 146944

set-ItemProperty -path "REGISTRY::\HKEY_USERS\.Default\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing" -name State -value 146944
get-ChildItem REGISTRY::HKEY_USERS | foreach-object {set-ItemProperty -ErrorAction silentlycontinue -path ($_.Name + "\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing") -name State -value 146944}


More Information:

Certificates for installing sofware is cause slow install:
http://joelblogs.co.uk/2011/09/20/certificate-revocation-list-check-and-sharepoint-2010-without-an-internet-connection/

http://ddkonline.blogspot.co.uk/2010/05/fix-sharepoint-very-slow-to-start-after.html

If you want to verify if a machine is having problems with a poarticlar process Process Explorer (Usefule if a machine has high memory, CPU or IO issues)