Tuesday, March 4, 2014

Meet URL Gives 404 error

I was recently at a company that did a big switchover from Lync 2010 to 2013.  The new environment consisted of 3 Enterprise Edition front-end servers with an F5 load balancer taking care of web services load balancing.

On the first business day of full Lync 2013 operations, some people were complaining they could not join meetings.  When some users clicked on the Join Lync Meeting link in the email, they were greeted with a 404 - File or directory not found error in IE.  If they tried several times, eventually they got in.  

Further analysis showed that one specific front-end server was serving up the 404 errors, while the others were working fine.  Thanks to the F5, we were able to easily remove that server from the load-balanced pool while we troubleshooted (troubleshot?) the problem. 

The first thing to note about troubleshooting issues with the meet URL on an Enterprise Edition pool is that you can't connect directly to a specific server in a pool and expect to get the proper meeting join experience.  For example, if your meet URL is meet.contoso.com and your Lync pool members are FE01.contoso.com, FE02.contoso.com etc, you would normally connect to a meeting via something like https://meet.contoso.com/user.name/FY3DFSE4.  You can't troubleshoot issues with a specific server by connecting to https://FE01.contoso.com/user.name/FY3DFSE4.  You'll get a 404 error, thanks to the way the URL Rewrite module processes URLs. 

To get around this, you need to add a temporary HOSTS entry to your testing workstation for meet.contoso.com pointing to the server having the issue. This will bypass the load balancer and allow you to connect directly to the server you want to test.

Back to the problem....

After setting my HOSTS file to point directly to the "bad" server, I found I could browse to https://meet.contoso.com successfully, but not to a specific meeting like https://meet.contoso.com/user.name/FY3DFSE4, which threw a 404.  Event logs didn't show anything wrong. 

I'll spare you the hours of dead ends I tried and just give you the solution (because that's why you're here, right?)

I went to Control Panel and removed the IIS URL Rewrite Module 2.  Then I re-ran Lync setup via the Lync 2013 Deployment Wizard, which re-added the URL Rewrite Module and reset the default URL rewrite rules Lync put in place.  All this was done without reboots or service interruption.  As soon as Lync setup completed, meeting joins happened without error. 

So, it appears that something went wrong with one or more of the URL rewrite rules, which wasn't cleaned up by simply re-running Lync setup, or Enable-CSComputer, which was things other people suggested in various places I looked. 

I hope this helps others who are having this issue.

Friday, February 21, 2014

Lync Conference 2014 Recap

Just got back from another amazing Lync Conference, this time at Aria in Las Vegas. It was great to see all my Lync buddies from around the world and to have the opportunity to participate in some very informative sessions given by Microsoft employees and many of my fellow Lync MVP friends.

There were several announcements, most of which I'm sure everyone has already heard about.
  • The next version of Lync is currently known as Lync vNext. Not sure if this is a codeword, or the final name
  • LyncvNext will include a new server role which will allow other video-conferencing systems (like Tandberg/Cisco) to join Lync-hosted video conferences. This server role can be either co-located on front-end or separate. 
  • Feature-parity on all mobile platforms, including Android tablets, which have not seen a Lync release as of yet.
  • Video calling between Lync and Skype. We all knew it was coming, but nice to see it finally show up. I think they're targetting go-live in June. 
  • A set of Javascript libraries called jLync, which will allow for all kinds of web development possibilities
  • The introduction of hosted-PSTN connectivity on Office365. No details on where it will be offered, but the US is a good bet.
Myself, I hosted two very popular sessions on Lync 2013 Enterprise Voice Best Practices to packed rooms. I had a lot of fun doing it, and look forward to doing more. The famous Jamie Stark even mentioned it on several occasions:

Feedback was very positive, including one fellow who threw a pair of underwear at me at the end of my second session as a joke.
I had a great time at this year's Lync conference. The venue was beautiful, the sessions were informative, and the after-hours parties were fun. I'm already looking forward to LyncConf15, hosted in Hawaii (I hope!).

Sunday, February 2, 2014

February 2014 Lync Optimizer Updates

Since moving the Optimizer back-end to a SQL database and enabling Microsoft authentication, its allowed me to explore adding new features not previously possible.

Ruleset History

The first feature I'm releasing is a ruleset history option. Every ruleset run by users is stored in SQL, so it was relatively easy to enable a history feature, so users can call up past rulesets (it was harder to make it secure). This can come in especially handy when working with extensions, as it is currently rather time consuming to enter extension details.

When you log in now, you'll see a View History button beside the Input heading.

Clicking it will bring up your entire ruleset history since the introduction of authentication and the SQL backend (late October 2013).  Clicking on any row will load that particular ruleset into the Optimizer.

Selective Caller-ID Block

Sometimes, users want the ability to block their outgoing caller ID, but not all the time.  Lync has a feature where you can do this at the voice route level, but it takes a bit of work to make this work "on-demand".  The Optimizer can now accomplish this for you, simply by checking the Allow Call ID Block checkbox and entering the desired caller ID block code (ie *67 in the US/Canada), and the replacement caller ID to use.

The Optimizer will change the default normalization rules to allow the entry of the caller ID block code.  Normalized numbers using this code will look something like: *67+12123334444.  The Optimizer then creates a route for that pattern that selects the Suppress caller ID option.  A trunk translation rule strips the block code before sending to the next hop.

Other Things

I've also included numerous tweaks to improve the overall experience and to ensure consistency. I've also nearly completed the data move from XML to SQL which gives me more options for the future.

I've been working behind the scenes trying to make these work as seamlessly and easy as possible. If you find the Optimizer helpful and a timesaver, think of the Hoff and send him a donation (this guy here, not the real Hoff....he doesn't need any more money).

Any new feature requests, please drop me a line.

Thursday, January 23, 2014

Presenting at LyncConf 2014!

I'm excited to say that I'll be presenting two sessions on Lync Enterprise Voice Best Practices at LyncConf14 in Las Vegas.

These sessions will focus on the WHYs behind the HOWs of Enterprise Voice in Lync 2013.  I'll talk about the best ways to manage your dialplans, voice policies, routes and trunk translation rules to provide a consistent, functional and manageable Lync EV deployment.

So, if you're at LyncConf, come check out my session. Otherwise, the only people who will be there will be my fellow Lync "friends" who just want to see me make a fool of myself, and are threatening to bring various fruits and vegetables to fling at me.

Session Details:

Setting up Enterprise Voice is a big project even for seasoned Lync experts. The interplay between dial plans, voice policies, routes, PSTN usages and trunk translation rules can make it complicated to figure out how to start. Come and join Lync MVP Ken Lasko, the creator of the Lync Dialing Rule Optimizer, and learn the WHYs behind the HOWs of configuring Lync Enterprise Voice – including E.164 numbering, extension dialing and least-cost routing - to provide the most flexibility and easiest migration path from legacy PBXs. And of course, this session wouldn’t be complete without a demonstration of the Lync Dialing Rule Optimizer and how it puts all these best practices into play.

Session Date/Time:

Tuesday, February 18 2014 2:00 PM - 3:15 PM
Room: Copperleaf 1
Thursday, February 20 2014 9:00 AM - 10:15 AM
Room: Copperleaf 1

If there's something you'd like to see me talk about in the session, leave a comment below.

Presentation Issues After Moving Lync 2013 Fileshare

Recently, we moved the Lync 2013 file share for an enterprise pool to a new location that was more resilient than the original. Everything seemed to go well, except that file sharing from internal to external users stopped working. Also, Lync 2010 users were suddenly unable to share Powerpoint presentations.  Lync 2013 users seemed fine.

When attempting to view or share a Powerpoint presentation on Lync 2010, the users got the following message:
This slide couldn’t be downloaded. Please contact your support team. Error reason: File not found.
The fact that 2013 clients were unaffected by the Powerpoint presentation is not unusual.  Lync 2013 offloads Powerpoint content rendering to the Office Web App server.  Lync 2010 clients are unable to utilize this feature, and fall back to the original method, which is managed and rendered from within the Lync server itself.

Of course, my first step in troubleshooting is to plug the error message into my favourite search engine. It came up with several different options, mostly dealing with issues with Office Web Apps, which we weren't experiencing. One blog post in particular caught my attention, because it talked about the same issue happening after a file share move:  http://paulbrown.us/blog/2011/11/02/how-to-change-lync-server-file-store-location/.  Incidentally, it appears that the writer (Paul Brown) enjoys sitting on active train tracks in his spare time, which may account for his lack of activity since May 2013.

So, Paul Brown (RIP) managed to solve the issue for Lync 2010, which definitely helped me solve it for 2013.  The key difference between 2010 and 2013 in this case is that the MeetingContent and MeetingFiles virtual directories in Lync 2010 don't exist in 2013.  All that seems to have been rolled into the CollabContent virtual directory in Lync 2013.

When I looked at the advanced settings for the CollabContent virtual directory, the Physical Path still pointed to the old file share location.  I updated the location using the new path in both the Lync Server External Web Site and Lync Server Internal Web Site and repeated this on each server in the affected front-end pool, followed by an IISReset.  This was enough for both file sharing and presentation sharing to function properly for Lync 2010 and 2013 clients.

This is obviously a bug within the Topology Builder/Lync Deployment Wizard, because this should have been changed automatically by running the Deployment Wizard after the topology change to move the file share.

As a final note, if you're planning on sitting on train tracks, make sure to keep an eye out for any oncoming trains, because, well you know, YOU'RE SITTING ON TRAIN TRACKS!

Thursday, January 16, 2014

High Processor Utilization on Lync 2013 Front-End Servers

We have a customer who is about to migrate from Lync 2010 to Lync 2013.  They've got a few lightly loaded Lync 2013 Enterprise Edition pools with 3 servers each.  All are running Windows 2008 R2 Standard Edition on VMWare.  All patches are up-to-date.

For inexplicable reasons, some of the servers will suddenly see their processor utilization spike to near 100% for extended periods of time, when their typical utilization is less than 5%. A look at Task Manager shows two instances of the W3WP.exe service (IIS web service) that are consuming large amounts of processor resources.  There are no events in the Event Logs to indicate an issue.

Performing an IISReset on the affected node makes the processor go back to normal, but this is obviously not a real solution.  We opened a ticket with Microsoft PSS, and they confirmed there are others seeing the same thing.  It seems the source of the problem is the "garbage collection" process in the LyncIntFeature and LyncExtFeature application pools in IIS.  Recycling those pools makes processor utilization return to normal (for a while at least).

Microsoft is actively working to resolve the issue, and I will post a permanent solution for all to see as soon as one becomes available.

UPDATE:  Thanks to @dannydpa  on Twitter, it appears the trigger may be Lync topology publishing. I confirmed this by updating the topology and publishing it.  Less than 10 minutes later, all the servers processor utilization spiked.  Recycling the aforementioned apppools resolved the issue.

To help others with this issue, I've created a little Powershell script that will recycle the LyncIntFeature and LyncExtFeature app pools for all servers in the pool hosting the Central Management Store.

$CMPool = (Get-CSService -CentralManagement | Where-Object {$_.Active}).PoolFQDN
$CMMembers = (Get-CSPool $CMPool).Computers
Foreach ($Computer in $CMMembers)
$Session = New-PSSession -ComputerName $Computer
Invoke-Command -session $Session -ScriptBlock {Restart-WebAppPool LyncExtFeature}
Invoke-Command -session $Session -ScriptBlock {Restart-WebAppPool LyncIntFeature}
Remove-PSSession $Session

Friday, December 6, 2013

SChannel Errors on Lync Server Preventing Client Logon

I was at a client setting up a brand-spanking new Lync 2013 deployment on Windows 2012.  I was setting up two pools in two datacenters. The server deployment went without a hitch and we got everything up and running in no time flat. However, we could not sign on with a Lync 2013 client to either pool.  The client just complained it couldn't log on. 

Looking at the server event logs, we saw numerous SChannel errors as below:
Event ID: 36874 - TLS 1.2 connection request was received from a remote client application, but none of the cipher suites supported by the client application are supported by the server. The SSL connection request has failed.
Event ID: 36888 - A fatal alert was generated and sent to the remote endpoint. This may result in termination of the connection. The TLS protocol defined fatal error code is 40. The Windows SChannel error state is 1205.
Looking around for solutions on the web, I came across these two apparent gems:
http://social.technet.microsoft.com/Forums/lync/en-US/41718327-203f-445f-8657-87b0a8545ead/lync-2013-client-signin-issue-with-lync-2013-server?forum=lyncprofile (Look towards the bottom for the answer)

If you don't feel like reading the aforementioned links, the answer was to use Regedit to disable TLS 1.2 on the Lync front-ends. This was the solution provided by MS Support. Sure enough, doing that fixed the problem, but as noted in the links above, this broke Windows Update.  To get Windows Update to work, you would have to remove the registry entry, restart the server, run Windows Update, re-add the registry entry and reboot the server once more.

Since this was a brand-new Lync deployment on brand new Windows 2012 servers, I had a hard time believing this was the only fix for the problem. Since the problem was affecting two independent pools, I figured there must be some common feature shared between them causing the issue. After much flailing about, I turned my attention to the recently installed Windows Certificate Authority installation. Another consultant had installed a CA for the company in preparation for Lync.

Comparing against known good installations, we noticed the signature hash algorithm used for the root certificate was SHA512, but other working deployments used SHA256 or lower. We reissued the root certificate using SHA256, and installed new certificates on the Lync front-ends using this hash algorithm. After a server restart, clients were able to log on successfully, and the SChannel errors went away.

I'm not a cryptography expert, so I'm not exactly sure why SHA512 caused issues with TLS 1.2. Poking around the Internet gave me the impression that SHA512 and TLS 1.2 just don't work together (but damned if I can find where I saw that again).

Regardless, this just goes to show that even if a workaround provided by Microsoft themselves might solve an issue, it doesn't necessarily mean its the right way to do it.

Tuesday, November 12, 2013

November 2013 Lync Dialing Rule Optimizer Updates

Since the move to use authentication in the Lync Dialing Rule Optimizer, I've been busy working behind the scenes to prepare the back-end for some cool new updates.

Back-End Changes

Firstly, I've been steadily moving away from XML for my data sources to a full-fledged SQL back-end. XML was great for the first while, but its been getting difficult to manage.  SQL offers much more robust querying, searching and sorting than XML, and opens up all kinds of possibilities for future features.  Now, changes and updates only have to be done in the database, and I don't have to touch the web pages.

Area Code Improvements

With the change to SQL for back-end databases, I've been able to drastically increase the number of area codes stored for countries like Germany.  Germany has thousands of area codes, which would overwhelm the drop-down style of listing area codes I've always done.  So for countries like Germany, you now enter the area code, and the Optimizer will show you the available cities from that area code.

Extension Extensions

You may notice additional options for extension entry than before.  Firstly, I've upped the extension limit from 10 to 20.  Secondly, I've added options to create your own rule suffixes for extension ranges.  So, if you're creating an extension range for your London, UK head office, you can assign a suffix like "HeadOffice", which will make the resulting normalization/routing rules use UK-London-20-HeadOffice, instead of the default UK-London-20-Internal-1.

You may also notice an additional checkbox column for "Single".  Sometimes, you may have users with their own DID, but maps to an internal extension that doesn't hold any relation to the DID.

For example, the company president may have a DID of +14165551234, and an internal extension of x200.  Your vice president may have a DID of +14165559876 and an extension of x201. Since there is no relationship between the DID and extension, you can't create a blanket normalization rule that will work with both of these.

With the new iteration of the Optimizer, you can easily tell the Optimizer to create individual normalization rule for each of these, simply by entering their DID and extension, and checking the box for Single.

Future updates may include a more Excel-like interface for extension entry that would allow cutting-and-pasting from Excel spreadsheets.  If you have the information already in a spreadsheet, it will make data entry MUCH simpler. 

Until then, enjoy, and if you have questions or problems, let me know.

Thursday, November 7, 2013

Location Based Routing Bug with External Users

Location-based routing is a relatively new addition to Lync 2013.  It wasn't part of the initial release, but the first cumulative update added this much asked for feature.

In a nutshell, location-based routing routes calls based on the network subnet the user is calling from, rather than their defined home server pool. So, if a user is in the Rome office today, all their calls can route via the Rome PSTN gateway.  If that same user goes to the London office tomorrow, their calls will route out the London PSTN gateway.

Since then, I've done a number of deployments where location-based routing was used extensively.  During one remote deployment at a large company, I noticed that my calls were not routing via my assigned voice policy.  They were routing via a PSTN gateway that was not defined in the policy I should have been using. In fact, my calls were routing out from South America (I'm in Canada)!

After much troubleshooting, I realized that Lync was routing my calls based on the subnet of my home network.  Turns out that my home network subnet of 192.168.2.x matched up with a corporate subnet assigned to that South American location. Location-based routing was configured to route calls for that subnet out through the South American PSTN gateway.  To verify this, I changed my home subnet to another one used by location-based routing. Lo and behold, my calls started routing out that PSTN gateway.  Finally, I changed my subnet to one not defined for location-based routing, and my calls began routing as per my assigned voice policy.

I wasn't using a VPN, and as such, I was connecting through the Lync edge server. Lync was incorrectly using my home's private subnet for call routing decisions. Since administrators have no control over the subnets used by external users, this could obviously lead to many issues, not to mention increased telephony charges for calls routing out through the wrong location.

I filed a bug report with Microsoft, who confirmed this bug and promised a fix in a soon-to-be-released cumulative update.  I don't know the details of the fix, but I imagine it will be one of two things:

  1. Lync's behaviour will change to ignore private subnet information for external connections for the purposes of location-based routing. This is probably an easy fix, and my money's on this one. 
  2. Lync's behaviour will change to use the detected public IP address assigned by the ISP for routing decisions.  I like this one, because it gives administrators the option to include public networks for location-based call routing decisions. Its unlikely that many administrators would go through the trouble to do this, but it would be nice to have the option.  
I doubt this bug will affect may deployments, but its always good to be aware.

UPDATE (08-Jan-2014): The issue has been fixed in the January 2014 Cumulative Update.  Read the KB article here

Wednesday, October 23, 2013

LiveID Authentication Coming to the Lync Dialing Rule Optimizer

Ever since its inception, the Lync Dialing Rule Optimizer has been totally free for use by anyone. I've always just assumed that people will use the tool for good instead of evil.  But the Internet is the Internet, and lately, I've been noticing a rather large uptick in fraudulent entries being done by parties unknown.

While this hasn't had any apparent impact on usability so far, I'm trying to get in front of it by figuring out ways to stem the bleeding before the patient goes terminal. The most obvious way is to introduce authentication into the Lync Dialing Rule Optimizer.

Thanks to the fantastic assistance from Richard Brynteson at Avtex, I've been able to get Microsoft Live ID authentication working in the Optimizer.  I've designed it to be as unobtrusive as possible.  All you have to do is click the Sign in button on the top-right corner, and a popup will direct you to the Live ID sign-in page. If its your first time, you'll be asked to confirm the permissions being requested by the application.  Once logged in, you can continue as normal.  If you try to generate a ruleset without logging in, you will be blocked.

I will be capturing basic information, including first/last name, Live User ID and email address. At this time, I have no plans on what to do with this information, but anything I do with it will be strictly limited to within the realm of the Lync Dialing Rule Optimizer.

This new feature has already given me all sorts of ideas for future improvements to the Optimizer.  Things like keeping a history of the scripts run and the ability to come back to make changes to extension lists.

I've put the question regarding authentication out to the Twitter community and I got back an even split of "Yeah, go for it" and "No, don't like it".  What are your feelings on the topic?  Let me know in the comments.