Applying updates to SfB Servers (Oct 2015 CU - 6.0.9319.88)

Have not done too many updates on the new Skype for Business pools, so made some notes based of my experiences with the Lync 2013 servers. There are new PS commands in SfB that really make it so much easier to go thru the whole process more efficiently and are more reliable (especially in a multi-member pool). No more guessing if the Fabric is "really" ready to take down the next member.

I upgraded a couple of SfB pools from RTM to version 6.0.9319.88 recently, and it was mostly painless on 2-member as well as 3-member pools. I say mostly, as my servers have "Windows Firewall" disabled by GPO so the pool would not start up because of that, giving me yet another grey hair :-). 

Step 1: Download and verify the update package. Right-click on the executable file and look at the properties. Ensure that the version information in the Details tab, matches up with the one you intend to apply to your servers.

Also look at the documentation to see if there are any prerequisites for the update. 

LS Storage Service Events 32023, 32021 and 32019 are continuously logged on FEs

Have noticed these on Lync 2013 as well as SfB server recently. Usually after some kind of loss of network connectivity.

Per MS Support, this is a know bug and has been addressed in Skype for Business Server 2015 CU1 (https://support.microsoft.com/en-us/kb/3061064). Does not seem to be fixed in the Lync 2013 code stream though.

Workaround is to restart the OS on the impacted Lync server. That clears out the issue. 


Log entries look like these:

Log Name:      Lync Server
Source:        LS Storage Service
Event ID:      32019
Task Category: (4006)
Level:         Warning
Keywords:      Classic
User:          N/A

Description: Asynchronous Task timed out for a Storage Adaptor.

Storage Adaptor=36aa818f-00bb-43bc-88e7-6840eca732c6

Cause: Either the Asynchronous Task is hung and/or the Storage Adaptor is not reporting progress.
Resolution: No action needed. The Storage Service will auto heal the Asynchronous Task by attempting a graceful cancellation of the task and restarting it.

Log Name:      Lync Server
Source:        LS Storage Service
Event ID:      32021
Task Category: (4006)
Level:         Warning
Keywords:      Classic
User:          N/A

Description: Request to cancel a Storage Adaptor Asynchronous Task timed out.

Storage Adaptor=36aa818f-00bb-43bc-88e7-6840eca732c6

Cause: The cancel request would have been generated by the system as a result of the task having timed out earlier or the service is shutting down and draining. The Storage Adaptor has failed to cancel the task. Either the Asynchronous Task is hung and/or the Storage Adaptor has not implemented support for cancelling Asynchronous Task.
Resolution: No action needed yet. The Storage Service will check the Asynchronous Task again and log an error event if the condition persists.


Log Name:      Lync Server
Source:        LS Storage Service
Event ID:      32023
Task Category: (4006)
Level:         Error
Keywords:      Classic
User:          N/A

Description: Storage Service failed to stop the Asynchronous Task for a Storage Adaptor.

Storage Adaptor=36aa818f-00bb-43bc-88e7-6840eca732c6

Cause: The stop request would have been generated by the system as a result of the task having timed out earlier or the service is shutting down and draining. The Storage Adaptor has failed to stop the task. Either the Asynchronous Task is hung and/or the Storage Adaptor has not implemented support for cancelling Asynchronous Task.

Resolution: This condition should not be left unattended as data loss can occur. If the problem persists, notify your organization's support team with the event detail.

September 2015 Security update for Lync server causes multiple issues (KB3080353)

Applying the Sep 2015 security patch can cause multiple issues, including:

1: Dialin.domain.com sign-in ability is broken
2: Mobility stops working
3: External Clients can’t sign in
4: Address book web queries fails
5: Credential Prompts after Signing in to Lync

The above issues can be avoided by ensuring your Lync servers are at the latest update level BEFORE applying KB3080353. Per MS support, the minimum requirement is July 2015 Lync Server 2013 CU 5.0.8308.920.

Short-term fix is to uninstall this security update and restart your Lync server(s). This will give you some breathing room to test/evaluate/plan the deployment of the July 2015 Lync server update. Since KB3080353 is a security update, it should be applied back on the server asap.


September 2015 Security update info - https://support.microsoft.com/en-us/kb/3080353

May 2015 - New or updated KB articles for Lync and SfB

• The LastActive attribute provides inaccurate information about a user’s status in Lync Online
• Issues when the Lync Server File Store share is in an unsupported configuration
• Menu bar is not displayed in Lync 2013 and Skype for Business after you set the AlwaysShowMenu registry entry
• Attached IM is lost in Lync 2013 when you send a file through an Office application
• Lync or Skype for Business client does not start when you click a meeting link if the default browser is Google Chrome
• Cannot join meetings in Lync 2010 after you install OneDrive for Business
• CPU usage increases largely when you have multiple animated emoticons in conversations in Skype for Business
• Memory leak occurs when you transfer a file that is larger than 5 megabytes (MB) in a conversation in Skype for Business
• Cannot open links without the "http://" prefix and the links that are to a OneNote page in Skype for Business
• Receiver cannot open or save transferred files in a chat room in Skype for Business
• Copy and "Select All" menu items are disabled for the first instant message in a conversation in Skype for Business
• Help isn't working error occurs when you open the "Skype for Business Help" window in Skype for Business
• The setting for the "team call" feature is deleted after you sign in to Lync 2013
• Lync 2013 crashes when you add a video to an existing audio call that is to Polycom RMX 2000 (v8.4)

Starting and shutting down Lync services using Powershell

The "Disable/Enable all Lync services" ones are especially helpful when the OS will need to be restarted a couple of times, and you don't want to deal with the hassle of waiting for the Pool/Windows Fabric to settle down first.

Get status - Get-Service | ? {$_.displayname –like “*Lync Server*”} | select status,displayname

Start all Lync services - Start-CSWindowsService

Stop all Lync services - Stop-CsWindowsService -Graceful

Disable all Lync services - Get-Service | ? {$_.displayname –like “*Lync Server*”} | Set-Service -StartupType disabled

Enable all Lync services - Get-Service | ? {$_.displayname –like “*Lync Server*”} | Set-Service -StartupType Automatic

SfB uses SQL express 2014

Interesting find: Skype for Business installs SQL Express 2014 on the FE servers.

Hmmm...wonder if that was to support WinFab v3 ...

Lync for Mac 2011 - No Delegate support

Symptom:  Lync for Mac 2011 | Delegates not able to see delegators or manage Lync meetings in Lync for Mac 2011.

Cause:  Delegate functionality is not currently present and supported in Lync for Mac 2011.  Delegator (add delegate) functionality is only delegate/delegator functionality currently supported.

Storage Service had an EWS Autodiscovery failure - ErrorIncorrectExchangeServerVersion

These events can be ignored if you have Exchange 2010.

Log Name:      Lync Server

Source:        LS Storage Service

Event ID:      32054

Task Category: (4006)

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      FE03.domain.com

Description:

Storage Service had an EWS Autodiscovery failure.

UnsupportedStoreException: code=ErrorIncorrectExchangeServerVersion, reason=GetUserSettings failed, smtpAddress=User2@domain.com, Autodiscover Uri=https://autodiscover.domain.com/autodiscover/autodiscover.svc, Autodiscover WebProxy=<NULL> ---> Microsoft.Exchange.WebServices.Data.ServiceRequestException: The request failed. The remote server returned an error: (401) Unauthorized. ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized.

   at System.Net.HttpWebRequest.GetResponse()

   at Microsoft.Exchange.WebServices.Data.EwsHttpWebRequest.Microsoft.Exchange.WebServices.Data.IEwsHttpWebRequest.GetResponse()

   at Microsoft.Exchange.WebServices.Autodiscover.AutodiscoverRequest.InternalExecute()

   --- End of inner exception stack trace ---

   at Microsoft.Exchange.WebServices.Autodiscover.AutodiscoverRequest.InternalExecute()

   at Microsoft.Exchange.WebServices.Autodiscover.AutodiscoverService.InternalGetUserSettings(List`1 smtpAddresses, List`1 settings, Nullable`1 requestedVersion, Uri& autodiscoverUrl)

   at Microsoft.Exchange.WebServices.Autodiscover.AutodiscoverService.GetSettings[TGetSettingsResponseCollection,TSettingName](List`1 identities, List`1 settings, Nullable`1 requestedVersion, GetSettingsMethod`2 getSettingsMethod, Func`1 getDomainMethod)

   at Microsoft.Exchange.WebServices.Autodiscover.AutodiscoverService.GetUserSettings(List`1 smtpAddresses, List`1 settings)

   at Microsoft.Exchange.WebServices.Autodiscover.AutodiscoverService.InternalGetSoapUserSettings(String smtpAddress, List`1 requestedSettings)

   at Microsoft.Exchange.WebServices.Autodiscover.AutodiscoverService.GetUserSettings(String userSmtpAddress, UserSettingName[] userSettingNames)

   at Microsoft.Rtc.Internal.Storage.Exchange.ExchangeContext.SendGetUserSettingsRequest(StoreContext ctx, String smtpAddress)

   --- End of inner exception stack trace ---

   at Microsoft.Rtc.Internal.Storage.Exchange.ExchangeContext.SendGetUserSettingsRequest(StoreContext ctx, String smtpAddress)

   at Microsoft.Rtc.Internal.Storage.Exchange.ExchangeContext.GetUserEwsSettings(StoreContext ctx, String smtpAddress, CacheMode cacheMode)

Cause: Autodiscovery Uri was not correctly configured or unreachable, that there is a problem with the Proxy, or other errors.

Resolution:

Check event details.  Check autodiscovery Uri is properly configured and reachable. Check that proxy setting is properly configured and reachable.  Validate Skype for Business to Exchange Autodiscovery configuration by following the trouble shooting guide. If problem persists, notify your organization's support team with the event details.

Wish there was a way: to sort the open conversation tabs

Tabbed conversations are great, especially for folks that have more than a handful of chats going on at the same time, however I wish there was a way to sort them alphabetically, so it's easy to find your contacts.

Even the popup window when you hover on the Lync/SfB program on your task bar seems to order the list by placing the most recent new IM session at the top, but as you see in the  screenshot below, the tow lists don't exactly match up either.

Skype for Business client - Tabbed conversations.
Conversation window in the background, and the taskbar list when you hover on the client, in the foreground.

If anybody has found a way to sort either of these lists, please let me know :-)

Gotchas with SfB upgrades

There is a ton of good information and lengthy guides on the actual in-place upgrade process for Skype for Business, so no point regurgitating it here. Instead, I've provided a couple of gotcha and tips to help make your upgrade go a little smoother.

1.	Make sure you have at least 32GB of free diskspace before running the in-place upgrade. The installer will verify that as one of the pre-reqs too.
2.	Shutdown and restart the Powershell window. If you had the Lync Management Shell console open while your server was getting upgraded to SfB, make sure you shut down that window, and start a new SfB Management Shell window. The fancy new start-cspool and other new SfB command will not run in the old version of that shell (which has also been overwritten by new bits).
3.	This one will hopefully be fixed in an upcoming release, but you MUST have the Windows Firewall Service running the first time you run start-cspool command after the upgrade is complete. The Windows Fabric seems to do a bunch of configuration updates at first launch, and if the Wnidows Firewall is not running, it will fail to start. On the console windows you will see the start-cspool command fail with something similar to this error. Start-CsPool : An error occurred while making the HTTP request to https://****fe06.domain.com/LiveServer/UserPinManagement/FabricManagement/. This could be due to the fact that the server certificate is not configured properly with HTTP.SYS in the HTTPS case. This could also be caused by a mismatch of the security binding between the client and the server. ... ... Start-CsPool : Please make sure at least 2 machine(s) are in running or starting state to achieve fabric network ring quorum from the following machine list. Also makes sure Windows Fabric Host Service is running on those machines. ****FE06.DOMAIN.COM ****FE07.DOMAIN.COM ****FE08.DOMAIN.COM ... ... Not enough seed nodes are running. ,Microsoft.Rtc.Management.Hadr.StartOcsPoolCmdlet If you also look at the events under Microsoft-Windows-WindowsFabric, you will see it referring to FabricDeployer.FirewallManager.EnableFirewallSettings. Just start the Windows Firewall service, and let the Fabric do it's thing. After the pool is up, you can stop the service if you want to.
4.	Check your back-end database versions to ensure they are at the proper versions. Run Test-CsDatabase -ConfiguredDatabases -SqlServerFqdn pool.domain.com | select databasename,installedversion,expectedversion to verify, and if need be update the DBs use Install-CsDatabase -ConfiguredDatabases -SqlServerFqdn SQLServerFQDN  -UseDefaultSqlPaths  -verbose or using the Topology builder.
5.	Run the SfB Setup Wizard at the end on each FE to enable/disable all the new changes in the topology.
6.	Run the Get-csServerVersion command and Congratulate yourself for a job well done :-)

Issue: The content file store is full or unavailable - event ID 32050 & 32026 on FEs

Recently we noticed a rash of Event ID 32050 & 32026 on the FE servers. Started all of a sudded in the middle of the work day (so not related to reboots or patches). The MS engineer who I worked with was aware of the issue, and said its a know bug, that should get resolved in the next server update.

Will notice a lot of these events on all except 1 of your FE servers

32026 - Conference rollover failed. 

Conference: 'sip:user@domain.com;gruu;opaque=app:conf:focus:id:xxxxxx' MCU Type: 'data-conf' Existing Mcu: 'https://fe03.domain.com:444/liveserver/datamcu/'

Cause: Conference rollover occurs when an MCU becomes unavailable or is restarted. A rollover failure indicates that no MCU is available for the requested MCU type.
Resolution: This operation will be automatically retried. This error can be ignored if MCU outage is planned. Otherwise ensure that at least one MCU for the specified type is available and that there are no connectivity issues between this server and the MCU.

32050 - A Create Conference request sent to an Mcu was rejected. It will be retried but if this error continues to occur conferencing functionality will be affected.

Mcu: https://fe05.domain.com:444/liveserver/datamcu/ Conference: sip:user2@domain.com;gruu;opaque=app:conf:focus:id:xxxxxx Error: otherFailure
Cause: Overloaded or incorrectly functioning MCU.
Resolution: Ensure that the Mcu is functioning correctly.

These seem to be related to another event ID 41027

41027 - The content file store is full or unavailable, and no more data can be written to it.

Content file store location: \\FS01.domain.com\ConfLyncShare\1-WebServices-7\CollabJournal Reason: The specified network name is no longer available. Cause: The file store either has run out of space or is unavailable. Resolution: Verify write access to the file store or make more space available on it. Note that this event is logged only once. When the problem is fixed a success event ID=41028 will be logged.

Cause : The front ends are attempting to access a file lock under the existing handle and not creating a new file handle to access the share.

Resolution: 

Run "Get-CsWindowsService" on all FEs

Check which FEs have 0 active conferences on the RTCDATAMCU service

Those MCUs are not responding, so restart the service on those FEs with

     stop-csWindowsService RTCDATAMCU

     start-csWindowsService RTCDATAMCU

  

Note: Sometimes the Backup Service can also lock a file in the file-share in a similar fashion, and restarting that service on the Primary or Backup pool can also resolve the lock issue. In those cases, you will notice a "stale" file in the CollabJournal folder.

MS is aware of this issue and the next release (CU5 HF9) should address it. (Update: Seems to be address in May'15 Lync server patch - https://support.microsoft.com/en-us/kb/3057445)

Outlook 2010 breaks with the latest round of Windows updates

KB2825678 seems to be causing issues with Outlook 2010. After this update is applied, Outlook 2010 crashes at launch time.

Related to a Facebook add-in for Outlook called Microsoft Outlook Social Connector. Uninstalling this update from the PC, fixes the issue.

MS technotes on the issue:

                https://support.microsoft.com/en-us/kb/2825678

                https://support.microsoft.com/en-us/kb/3058207