"Have you tried turning it off and back on again?" for MS Teams

One of the most powerful tools in a support professional's arsenal is the tried-tested-and-true method of restarting a computer to resolve strange issues. I can't tell you how many times this fixed the oddest issues over the years. Why does it work? You might as well ask where socks go once placed in the clothes dryer.

I'm excited/sad to report that the same method works in Microsoft Teams to resolve strange issues that defy explanation. I'm not talking about client-side issues. I'm talking about issues at the Office 365 end of things. How does that work when there are no administrator-accessible servers to restart? You simply toggle the relevant setting off and back on again.

We use Direct Routing for our PSTN services, and for the most part it works great. In the past few weeks, I've had some of our users experience strange issues with telephony in Teams.

1. A few users were unable to receive any PSTN calls. Troubleshooting at the SBC would show 487 Request Terminated - Phone number not found.
2. Another user's outbound PSTN calls were showing as Anonymous/Private. There weren't any caller ID policies being applied that would account for the behaviour.

For each of those issues, I verified that things were indeed setup properly in Teams via PowerShell. Get-CsOnlineUser showed that their LineURI and OnpremLineURI settings were correct, and Enterprise Voice was enabled. Get-CsOnlineVoiceUser showed they were all licensed, PSTNConnectivity was OnPremises and their Number was correct.

In short, there was nothing on the Teams side of things that indicated an issue. I decided to try the "Have you turned it off and back on again?" trick, which consisted of:

Set-CsUser -Identity UserID@contoso.com -EnterpriseVoiceEnabled:$FALSE
Set-CsUser -Identity UserID@contoso.com -EnterpriseVoiceEnabled:$TRUE

I waited patiently all of 10 seconds between commands. I then instructed the users to log off and back onto Teams. Lo and behold, in each case, the problem went away.

Yeah, I know. I don't get it either, but whatever...it works.

via GIPHY

Response Groups Stop Responding

Our company (Event Zero - makers of the best Skype for Business analytics software out there BTW) relies on Skype for Business response groups for our sales and support queues.  Last week, I noticed that every single one of them on two separate pools were no longer accepting calls.  It wasn't a SIP trunk or mediation issue, because I couldn't get to them by directly entering in their SIP address in the Skype for Business client.  They appeared available (green presence) but would not accept calls. Snooper logs showed they were throwing 480 Temporarily Unavailable errors.

It was especially odd that it happened on two separate S4B pools at roughly the same time.  I tried numerous things, from restarting the RGS service on the affected servers to restarting the servers.  So, yeah, not a lot of tools in my RGS troubleshooting arsenal apparently.

What I found DID work, was to change the Tel URI of one of the workflows to a slightly different number, then changing it back.  Within a few minutes, that particular workflow started working again. 

Rather than doing the same thing to all 20-odd response groups (which would take a LOOONG time because the RGS Workflow web page is so slow), I created a Powershell script to do the same thing.

WARNING: Use script at your own risk. It worked fine for me, but hey I'm not a programmer. Also, this script will use the Description field of the workflow to store the original Tel URI of the response group.  Whatever is there now will get wiped out. You can do this script in other ways, but I was strapped for time and we weren't using the Description field for anything.

$Workflows = Get-CsRgsWorkflow 

Foreach ($WF in $Workflows)
{
 Write-Host "Adding dummy extension to " $WF.Name
 $WF.Description = $WF.LineURI
 $WF.LineURI = $WF.LineURI + ";ext=0"
 Set-CSRgsWorkflow -Instance $WF
}
Foreach ($WF in $Workflows)
{
 Write-Host "Reverting back to original number for " $WF.Name
 $WF.LineURI = $WF.Description
 Set-CSRgsWorkflow -Instance $WF
}

I don't know what caused the problem to start with, but at least this fixed it. Presumably, something happened to "break" the connection between RGS and the associated contact objects, and resetting the Tel URI re-linked them.

Hopefully, this might help others who come across the same thing. If anybody has any insight into why it broke, and why this fix worked, please enlighten me!

Lync 2013 Edge server bug when adding new server pools

I had this happen to me a few weeks ago, and a co-worker had it happen to him today, so I figure it should be mentioned since nobody on the Internet seems to have a proper fix for it.

I installed a new Lync 2013 Standard Edition server in an existing Lync 2013 environment with an established edge server pool.  I created a test account on the new pool and began running through the usual testing scenarios.

All seemed well and fine until we tested federated communications.  An external party could establish an IM or A/V session with the internal user homed on the new pool and see their presence, but the internal user couldn't see presence nor establish a new IM or A/V session with anybody outside the company.

Puzzling to be sure, since everybody else in the company was OK.  I first checked to see if I could telnet to the typical ports required for edge functionality, and all was good. I then logged onto the edge to see if there were any relevant events.

Lo and behold, there was this interesting little warning:

LS Protocol Stack Event 14402
Multiple incoming connections on internal edge from non-internal servers.
In the past 289 minutes the server received 4 incoming connections on internal edge from non-internal servers. The last one was from host newlyncfe.contoso.com.
Cause: This can happen if an internal server is not present in the list of internal servers on the Access Edge Server.
Resolution:
If the server is a valid one, you need to add it to the list of internal servers on the Access Edge Server. If the server is invalid, you may be under an attack from that server.

This left me scratching my head.  How do you add a Lync 2013 server to the "list of internal servers"???  I recalled back to the olden times of OCS, where you had to manually add every valid server name to the edge, but you don't do that anymore, thanks to the wonder of the Topology Builder.

Internet searches brought up several cobweb-covered webpages from the late 2000's, all of them relating to OCS.  No good.  I re-ran Setup on the edge server, hoping that would trigger something and make things all good, but nope.

Finally, I decided to restart the Lync Server Access Edge service (during a period of low activity of course).  Once it restarted, all my troubles went away.  The user on the new front-end was able to initiate sessions to external users, and the 14402 errors on the edge went away.

My next step was going to be blindly restarting other services followed by a last-ditch server reboot. Since that turned out to be unnecessary, if this saves someone from doing a server restart (which would fix the problem too), that's awesome.

So, while Lync normally does an excellent job of detecting new servers and functioning fine without service restarts or server reboots, this is one situation where its not doing that.  It seems the Access Edge service caches the list of valid internal servers at service startup, and doesn't refresh that cache until the next service restart.

Because this seems to be one of the few instances where a service restart is required to maintain connectivity with a new downstream server, I would classify this behaviour as a...

Lync Address Book Weirdness

So, I'm at a fairly large client that's moved from Lync 2010 to 2013.  The server-side migration completed a while ago, and the client migration is in progress.  Enterprise Voice is enabled for a select group of people.

Some people have noticed that when they search for a specific Lync user, the only object that got returned was an object for their Active Directory administrative account that had only a phone number defined (not a Lync-enabled account).  The Lync-enabled "normal" Lync entry didn't show up in the search, even though it did appear in the GALContacts.db file (the Lync locally cached address book file).

The really weird thing was that the object showed up like a phone contact object, but one that had presence working.  The affected user could toggle their presence, and the contact object would dutifully change itself.  However, since it wasn't a Lync object, you couldn't start an IM (but you could call the associated number).

This user's first name isn't actually "Admin" in case you were wondering.

Another weird thing was that the very first time you pulled up that person's name from a search, it would briefly show the proper Lync-enabled account, but switch to the phone contact object view after a half-second or so.

Yet ANOTHER weird thing, was that after viewing the admin account object in Lync a few times, usually after viewing the contact card, the next time I did a search for the same name, only the phone number appeared.  However, I could still look at the contact object for the phone number and it would show the user's email address.

As mentioned previously, the address book had the correct contact information (verified by opening the GALContacts.db file in Notepad....the results ain't pretty, but it works), as well as the information from the AD admin account, but Lync was choosing to show the non-Lync enabled admin account in searches.

When I compared the affected admin accounts with others, I noted that some users showed up properly, but others didn't.  More comparisons showed that for the incorrect information to show up, the following had to be true:

1. Both the Lync-enabled account and the AD admin account had a phone number defined
2. Both the Lync-enabled account and the AD admin account had the same email address defined.

Removing the email address from the admin account, re-generating the Lync address book via Update-CSAddressBook, and re-downloading the updated address book file fixed the issue. 

So, what happened?  When you do a search, the Lync client will look at the address book, your own Outlook contacts, and any social connectors you may have associated with Outlook (like Facebook or LinkedIn).  If the same name shows up in multiple places, Lync will try to consolidate all the information and show only one contact object.

Sometimes, the process goes awry. In this case, it appears that Lync saw the same email address for both the Lync-enabled AD account, and the non-Lync enabled AD admin account, and incorrectly presented the non-Lync enabled account as the single object.

I'm guessing this is a bug, because Lync really should place priority on a Lync-enabled object over a non-Lync one for presentation.  

Meet URL Gives 404 error

I was recently at a company that did a big switchover from Lync 2010 to 2013.  The new environment consisted of 3 Enterprise Edition front-end servers with an F5 load balancer taking care of web services load balancing.

On the first business day of full Lync 2013 operations, some people were complaining they could not join meetings.  When some users clicked on the Join Lync Meeting link in the email, they were greeted with a 404 - File or directory not found error in IE.  If they tried several times, eventually they got in.  

Further analysis showed that one specific front-end server was serving up the 404 errors, while the others were working fine.  Thanks to the F5, we were able to easily remove that server from the load-balanced pool while we troubleshooted (troubleshot?) the problem. 

The first thing to note about troubleshooting issues with the meet URL on an Enterprise Edition pool is that you can't connect directly to a specific server in a pool and expect to get the proper meeting join experience.  For example, if your meet URL is meet.contoso.com and your Lync pool members are FE01.contoso.com, FE02.contoso.com etc, you would normally connect to a meeting via something like https://meet.contoso.com/user.name/FY3DFSE4.  You can't troubleshoot issues with a specific server by connecting to https://FE01.contoso.com/user.name/FY3DFSE4.  You'll get a 404 error, thanks to the way the URL Rewrite module processes URLs. 

To get around this, you need to add a temporary HOSTS entry to your testing workstation for meet.contoso.com pointing to the server having the issue. This will bypass the load balancer and allow you to connect directly to the server you want to test.

Back to the problem....

After setting my HOSTS file to point directly to the "bad" server, I found I could browse to https://meet.contoso.com successfully, but not to a specific meeting like https://meet.contoso.com/user.name/FY3DFSE4, which threw a 404.  Event logs didn't show anything wrong. 

I'll spare you the hours of dead ends I tried and just give you the solution (because that's why you're here, right?)

I went to Control Panel and removed the IIS URL Rewrite Module 2.  Then I re-ran Lync setup via the Lync 2013 Deployment Wizard, which re-added the URL Rewrite Module and reset the default URL rewrite rules Lync put in place.  All this was done without reboots or service interruption.  As soon as Lync setup completed, meeting joins happened without error. 

So, it appears that something went wrong with one or more of the URL rewrite rules, which wasn't cleaned up by simply re-running Lync setup, or Enable-CSComputer, which was things other people suggested in various places I looked. 

I hope this helps others who are having this issue.

SChannel Errors on Lync Server Preventing Client Logon

I was at a client setting up a brand-spanking new Lync 2013 deployment on Windows 2012.  I was setting up two pools in two datacenters. The server deployment went without a hitch and we got everything up and running in no time flat. However, we could not sign on with a Lync 2013 client to either pool.  The client just complained it couldn't log on. 

Looking at the server event logs, we saw numerous SChannel errors as below:

Event ID: 36874 - TLS 1.2 connection request was received from a remote client application, but none of the cipher suites supported by the client application are supported by the server. The SSL connection request has failed.

Event ID: 36888 - A fatal alert was generated and sent to the remote endpoint. This may result in termination of the connection. The TLS protocol defined fatal error code is 40. The Windows SChannel error state is 1205.

Looking around for solutions on the web, I came across these two apparent gems:
http://social.technet.microsoft.com/Forums/lync/en-US/41718327-203f-445f-8657-87b0a8545ead/lync-2013-client-signin-issue-with-lync-2013-server?forum=lyncprofile (Look towards the bottom for the answer)
and
http://www.logicspot.net/index.php?id=50

If you don't feel like reading the aforementioned links, the answer was to use Regedit to disable TLS 1.2 on the Lync front-ends. This was the solution provided by MS Support. Sure enough, doing that fixed the problem, but as noted in the links above, this broke Windows Update.  To get Windows Update to work, you would have to remove the registry entry, restart the server, run Windows Update, re-add the registry entry and reboot the server once more.

Since this was a brand-new Lync deployment on brand new Windows 2012 servers, I had a hard time believing this was the only fix for the problem. Since the problem was affecting two independent pools, I figured there must be some common feature shared between them causing the issue. After much flailing about, I turned my attention to the recently installed Windows Certificate Authority installation. Another consultant had installed a CA for the company in preparation for Lync.

Comparing against known good installations, we noticed the signature hash algorithm used for the root certificate was SHA512, but other working deployments used SHA256 or lower. We reissued the root certificate using SHA256, and installed new certificates on the Lync front-ends using this hash algorithm. After a server restart, clients were able to log on successfully, and the SChannel errors went away.

I'm not a cryptography expert, so I'm not exactly sure why SHA512 caused issues with TLS 1.2. Poking around the Internet gave me the impression that SHA512 and TLS 1.2 just don't work together (but damned if I can find where I saw that again).

Regardless, this just goes to show that even if a workaround provided by Microsoft themselves might solve an issue, it doesn't necessarily mean its the right way to do it.

July 2013 Update for Lync 2013 Icon Weirdness

So, along with probably a bazillion other people, I installed the July 2013 client update for Lync 2013 (CU2) to see all the new awesomeness around Q & A, picture cut'n'paste and other assorted goodies as shown on several other blogs like Richard Brynteson from MasteringLync.com.

While the new features worked great, I couldn't help but notice that the new icons were either completely missing or showed incorrectly, as in the below screenshots:

Note the missing icon for "Q & A"

Note the weird icon on the right where the meeting icon should be

After removing and reinstalling both the Lync 2013 client update and even Lync 2013 itself, the situation never changed. After posting a question in the Lync MVP mailing list, Dave Howe from MS came back asking about the version numbers of MSO.DLL and MSORES.DLL.  Apparently, those DLLs contain the Lync 2013 icons, so if they're the incorrect version, then you'll see issues.  In my Office 2013 x32 installations, those DLLs were found in C:\Program Files (x86)\Common Files\microsoft shared\OFFICE15.

Keeping an eye on those files, I reinstalled the Lync 2013 CU2 update once again.  However, they never changed from their default value.  Going on what I heard from others, I went to Windows Update to install the latest updates for Office 2013.  Several were found on my first pass.  I noticed that MSORES.DLL seemed to be updated by KB2817489 to 15.0.4517.1003 , but MSO.DLL was untouched.  With that update, the wonky icon showed up correctly, but the icons for "Q & A" and "No Meeting IM" were still missing.

Re-running Windows Update again installed a few more updates, notably KB2817491. This updated MSO.DLL to 15.0.4517.1005, at which point all the icons showed up normally.

If you read the description for KB2817489, it says the update should fix the Q & A and ME icons (whatever that is). The description page for KB2817491 is currently not available, so I can't tell what it does. The patch is apparently for Office 2013 x64, but I was running x32, and it updated the x32 version of MSO.DLL.  There may be some documentation fixing to do.

So, if you install Lync 2013 CU2, make sure you run Windows Update not just once, but TWICE to ensure you get all the updates necessary for Lync 2013 to work properly.

What did I learn from all this? MSORES.DLL stores the icons for the main Lync page, while MSO.DLL stores other icons.  Also, not all Lync file updates come in one nice package, as you would hope.

On a final note, I have to say that I am not a fan of this process.  I understand that Lync 2013 is now part of Office 2013, but is it too much to ask that if an update is put out for Lync 2013, that update should include ALL the required DLL updates?  I shouldn't have to install multiple seemingly unrelated updates just to get a fully functional Lync 2013 client.  This took up a good half a day of troubleshooting that would have been better spent doing other things.

By the way, I want to give special thanks to Dave Howe from Microsoft. If it weren't for him asking about MSO/MSORES, it would have taken a lot longer to figure out the solution.

Errors After Installing Office Web App Server on Non-Standard Drive

I haven't come across anything noting this, but beware of installing Office Web App Server 2013 on anything other than the standard C:\Program Files\Microsoft Office Web Apps on Windows Server 2012.  I installed OWA on the D: drive to keep with a corporate policy for all installed applications. The installation itself went without a hitch, and I was even able to create a new Office Web App farm.  However, when I tried to run any other OWA Powershell command (like Get-OfficeWebAppFarm), I would get the following error:

Get-OfficeWebAppsFarm : The content type text/html; charset=utf-8 of the response message does not match the content type of the binding (application/soap+msbin1). If using a custom encoder, be sure that the IsContentTypeSupported method is implemented properly. The first 1024 bytes of the response were: '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>IIS 8.0 Detailed Error - 500.19 - Internal Server Error</title>
<style type="text/css">
<!--
body{margin:0;font-size:.7em;font-family:Verdana,Arial,Helvetica,sans-serif;}
code{margin:0;color:#006600;font-size:1.1em;font-weight:bold;}
.config_source code{font-size:.8em;color:#000000;}
pre{margin:0;font-size:1.4em;word-wrap:break-word;}
ul,ol{margin:10px 0 10px 5px;}
ul.first,ol.first{margin-top:5px;}
fieldset{padding:0 15px 10px 15px;word-break:break-all;}
.summary-container fieldset{padding-bottom:5px;margin-top:4px;}
legend.no-expand-all{padding:2px 15px 4px 10px;margin:0 0 0 -12px;}
legend{color:#333333;;margin:4px 0 8px -12px;_margin-top:0px;
font-weight:bold;font-size:1em;}
a:link,a:visited{color:#007EFF;font-weight:bold;}
a:hover{text-decoration:none;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0'.
At line:1 char:1
+ Get-OfficeWebAppsFarm
+ ~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-OfficeWebAppsFarm], ProtocolException
    + FullyQualifiedErrorId : System.ServiceModel.ProtocolException,Microsoft.Office.Web.Apps.Administration.GetFarmCommand

Uninstalling Office Web App Server from the D: drive and reinstalling on the default path of C:\Program Files\Microsoft Office Web App fixed the issue.  Again, this was on Windows Server 2012.  No idea if the same thing happens on Windows Server 2008 R2.

Just an FYI for anyone looking for answers for this specific issue.

Fixing Call Issues in a Mitel 3300 - Lync 2013 Deployment

We recently deployed Lync Server 2013 Enterprise Voice for a customer who at the same time replaced their old Mitel PBX with a newer Mitel 3300. The customer wasn't ready to make a full commitment to Lync for voice traffic, but probably 80% of the customer's users are using Lync exclusively for voice.

The Lync infrastructure was connected to the PSTN via the Mitel 3300 PBX using direct SIP.  While things initially seemed to be fine, we were getting numerous reports of sporadic failed calls throughout each day.

Digging through the monitoring server reports (deploying monitoring server reports should be considered ESSENTIAL for any Lync Enterprise Voice deployment), we could see a clear trend for the call failures: Looking at the Failure Distribution Report, we could see the top failed call reason was Gateway side Media negotiation failed.

Clicking on the sessions number hyperlink brought us to the list of calls that failed due to that cause.  Selecting one shows the session detail report for that call:

At the bottom is the Diagnostic Report.  Clicking the Detail link on the server side finally shows the exact reason for the call failure:

10010; source="LyncFE.contoso.com"; reason="Gateway side Media negotiation failed"; component="MediationServer"; ValidationError="SDP Exchange already failed before SetAnswer: Session c line = 0.0.0.0 or ::, call hold not allowed for initial INVITE offer"; GatewayFqdn="10.1.1.2; trunk=NA-OH-Dayton-Mitel"

I did a search through my normal channels and didn't come up with much.  I saw a few references to clients who set the Lync client audio port range to 3, rather than the recommended 20-40 which caused their particular issue, but we were already using 40 as a client port range.

It seemed as though the issue was coming from the Mitel side.  I tried a few things like setting the trunk's RTCPActiveCalls and RTCPCallsonHold values to FALSE on the Lync side, but that had no effect.

We tried to contact the Mitel partner who put in the Mitel system, but they were of very little help.  I asked if I could poke around the Mitel web console to look for a potential fix. Surprisingly, the client said "Sure".  Having never seen a Mitel before, I thought that things were about to go very wrong, or at least we wouldn't get any closer to a solution.

Since the error was related to SDP, we focussed on the SDP Options tab under SIP Peer Profile on the Mitel.  The most promising option was Avoid Signalling Hold to the Peer.  It was promising because the error message from the Lync monitoring reports mentioned both SDP and hold, so.......we set it to Yes, and at the same time we set Enable Mitel Proprietary SDP to No.  Everything else we left alone.

Almost immediately, our problems came to an end. Calls to numbers that would fail every 2nd or 3rd attempt suddenly worked without error every single time. Another side benefit was that we were now getting music on hold when transferring to Mitel call queues, where we weren't before.

I was amazed that we had brought up the issues around both music on hold and the failed calls on numerous occasions, and the Mitel partner could not find a solution, while someone with zero experience with Mitel was able to find a solution in under 10 minutes of poking around.

I was feeling pretty good about fixing that issue, so I tried to solve the other remaining issue where outbound calls would only show the main office number rather than the individual Lync user's DID. However, my beginners luck ran out and I couldn't find a solution.

So, if you're having similar issues with a Mitel - Lync deployment, hopefully this will steer you in the right direction.

Lync Loses Connection Every 8min 28sec

If you enjoy CSI-like detective stories (and by CSI, I mean cheesy quips as he puts on his sunglasses, NOT this) centered around isoteric computer issues involving Lync, then you'll love this post.  For the other 6.99999 billion people in the world, move along....nothing to see here.

Late last week, my Lync client on my main work desktop started acting oddly.  Every so often, it would spontaneously lose its connection, log out, and re-login again - all within about 5-10 seconds.  At first, I suspected a network problem.  Running a steady ping against the Lync edge server showed a constant connection.  I looked at the Lync servers for any potential issue, thinking it was something that could be affecting everybody, but all seemed well.  It was late in the day on Friday, so I dealt with it like I deal with all my problems.....ignore it and hope it just goes away.

Monday morning - same thing.  It was starting to get annoying.  I'd be in the middle of an IM conversation, and BAM, log out and back in again.  Audio calls were unaffected, except for the message about limited connectivity popping up.  Weirdly enough, when I logged on via my laptop, the issue never came up, which lead me to believe it was a problem with my desktop computer.  Even more weird was that if I logged onto another Lync account on the same pool via my desktop, the problem never came up.

The first thing I tried was to restart my computer.  Didn't fix it.  Then I exited Lync and deleted all my settings in my %userprofile%\AppData\Local\Microsoft\Communicator folder.  No change in behaviour.

There were no relevant messages in the Event Log, even though Event Log logging was turned on.  I decided to turn on detailed logging via Options - General - Turn on logging in Lync.  You should only turn on this option when troubleshooting an issue, because it can take up a lot of disk space.  Logs are stored in the %userprofile%\Tracing folder.  The Communicator-uccapi-0.uccapilog is the log file to look at.  This file will grow to about 50 MB before it rolls over into Communicator-uccapi-1.uccapilog and so on.  It's Notepad-friendly. 

Digging through the log, I focussed on the time where the issue arose.  I started to notice a pattern.  My client would lose its connection EXACTLY every 8 minutes 28 seconds.  I racked my mind to think of what would be happening every 8:28, but nothing came to mind.  I ripped out all the log data surrounding that time to try to find a pattern.  This is the sequence that repeated every 8:28:

10/18/2011|12:23:30.119 1674:2350 INFO  :: UCCP:ClientAllowedAuthProts0x10004
10/18/2011|12:23:30.120 1674:2350 INFO  :: SIP_REGISTER::RefreshRegistration force(1),refreshSA(1),refreshRoute(0),state(2)
10/18/2011|12:23:30.120 1674:2350 INFO  :: SIP_REGISTER:State (2) => (4)
10/18/2011|12:23:30.120 1674:2350 INFO  :: SA(b0b8c38) Dropped
10/18/2011|12:23:30.120 1674:2350 INFO  :: Out trxn corr-id (0CE823A8)
10/18/2011|12:23:30.127 1674:2350 INFO  :: SignMsg:NoSA for request(ce823a8)
10/18/2011|12:23:30.127 1674:2350 INFO  :: SignMsg: send request without signature for trans(ce823a8)
10/18/2011|12:23:30.127 1674:2350 INFO  :: Trxn corr-id (0CE823A8), SIP msg corr-id (7aa8ef7f)
10/18/2011|12:23:30.127 1674:2350 INFO  :: Sending Packet - 209.xx.xxx.xxx:443 (From Local Address: 10.0.0.2:55072) 793 bytes:
10/18/2011|12:23:30.127 1674:2350 INFO  :: REGISTER sip:contoso.com SIP/2.0
Via: SIP/2.0/TLS 10.0.0.2:55072

The client would then try re-registering itself multiple times (about 10) for about 3-5 seconds, which would generate a SIP/2.0 401 Unauthorized from the server.  Then, it would just log back in as if nothing ever happened. 

It looked to me like the line that had SIP_REGISTER::RefreshRegistration force(1),refreshSA(1),refreshRoute(0),state(2) was the likely culprit, but I still had no idea why it was happening.  I ran log captures on working machines and never saw that line appear anywhere else.  Internet searches came up blank.  I even ran traces from the front-end Lync server, which only showed me the multiple 401 Unauthorized messages it was returning to the client.  No indication why it was not allowing me in.

I tried uninstalling and reinstalling Lync.  Nope.  I tried uninstalling and searching and deleting any trace of Lync in folders and the registry.  I deleted any cached credentials in Windows via Credential Manager.  Nope.  In frustration, I even deleted my Lync account and re-created it.  Nope.....Chuck Testa.

Then I tried creating a new local account on my desktop and logged into Lync.  Great success!!!  The problem was gone.  However, I wasn't all that thrilled because I didn't want to go through the pain of moving all my profile data to a new account.  Plus, it felt like giving up....kind of like when you re-install the OS to deal with an issue.

So, I reasoned that there must be something about my profile that was causing the issue.  So, I logged back into my main account, deleted all the temp files in %userprofile%\TEMP, cleaned out my browser history and cache, and restarted.  Amazingly, the problem was gone.  It's been 3 days and my client has been working normally.

I'm not sure if it was cleaning out the TEMP folder or the IE cache that fixed the problem.  Ideally, I should have tried one or the other first, but I was getting tired of troubleshooting by this point.  If anybody has any insight as to what was really going on, please drop me a line.