Bug in UCDialplans North American Rulesets

I was troubleshooting an issue where people couldn't dial US/CA toll-free numbers in Teams the other day, when using Direct Routing. Initially, the root issue wasn't that easy to solve, due to the lack of a Voice Routing Test Case tester like we have in Skype for Business. Normally, one of the first things I do when troubleshooting issues like this, is to make sure the user is actually ALLOWED to make such phone calls, based on their assigned voice policy. Teams doesn't have an equivalent to the SfB voice route tester. If it did, I would have found it much sooner.

The first thing that I did was look at the user's call history in the Teams Admin Centre. Sure enough, every toll-free call made by the user failed with the below error logged in Teams:

"The destination failed because it doesn't exist"

No information about this error came up in a search. Reviewing logs from the SBC showed the call never got to the SBC. Client logs didn't provide any usable information. Finally, I looked deeper into the applied dialplan. At this point, I saw the problem, and it was me.

Of course, I had applied a ruleset generated by UCDialplans.com. Everybody was automatically assigned the global voice policy, which was set to allow dialing only within the US. This utilizes a feature in UCDialplans.com where you can decide exactly what constitutes a "National" call.

If you're not familiar with the intricacies of the North American dialplan (NANPA), just know that the US, Canada and 26 Caribbean countries share the country code +1.  Each country has its own set of one or more area codes. There is no way to distinguish whether a number is in the US, Canada or a Caribbean country just by looking at it.

UCDialplans.com can configure your National voice policy so that a "National" call can include any number that starts with +1, limit to US/CA or limit it to US OR CA. When creating the ruleset, UCDialplans.com has a drop-down option where you decide what you want to "Treat as National".

If you select "In-Country Only" (which is the default option), UCDialplans.com will create a national voice route that only includes the area codes for the selected country. The current US example is below, which explicitly details every area code in the US, and is derived programmatically from the current list of area codes:

^\+1(?!(900|976))(20[^04]|21[^1]|22[^1267]|23[149]|24[08]|25[12346]|26[0279]|27[0269]|30[^06]|31[^1]|32[0135]|33[^358]|34[0167]|35[12]|36[014]|38[056]|40[^03]|41[^168]|42[345]|43[0245]|44[0235]|46[39]|47[0589]|48[04]|50[^06]|51[^149]|53[0149]|54[01]|55[19]|56[12347]|57[01345]|58[056]|60[^04]|61[^13]|62[03689]|63[016]|64[016]|65[017]|66[01279]|67[018]|68[0124]|70[^059]|71[^01]|72[04567]|73[1247]|74[037]|75[47]|76[02359]|77[^1678]|78[1567]|80[^079]|81[^19]|83[0128]|84[3578]|85[^1235]|86[02345]|87[028]|90[^025]|91[^1]|92[0589]|93[^235]|94[0179]|95[12469]|97[^4567]|98[0459]|281|458|520|828)[2-9]\d{6}

For the US/Canada option, rather than explicitly including every area code, it EXCLUDES Caribbean area codes, which is a much smaller list. Similarly, for US/Canada/Caribbean there is no need to exclude any area code (except premium), so the list is even smaller.

In an effort to reduce route selection time and processing overhead, the National voice policy doesn't explicitly include PSTN usages for local dialing, UNLESS least-cost routing is being used. For the US/Canada and US/Canada/Caribbean route options, toll-free numbers are captured by the route pattern because they're NOT in the exclusion list of Caribbean area codes. Unfortunately, I didn't account for toll-free numbers in the US or Canada Only route options, which means that toll-free numbers can't be dialed if you use the default options.

You could fix this by adding the Local PSTN usage to your voice policies, but that adds unnecessary overhead. A better option is to explicitly include toll-free numbers in the computed route.

^\+1(?!(900|976))(20[^04]|21[^1]|22[^1267]|23[149]|24[08]|25[12346]|26[0279]|27[0269]|30[^06]|31[^1]|32[0135]|33[^358]|34[0167]|35[12]|36[014]|38[056]|40[^03]|41[^168]|42[345]|43[0245]|44[0235]|46[39]|47[0589]|48[04]|50[^06]|51[^149]|53[0149]|54[01]|55[19]|56[12347]|57[01345]|58[056]|60[^04]|61[^13]|62[03689]|63[016]|64[016]|65[017]|66[01279]|67[018]|68[0124]|70[^059]|71[^01]|72[04567]|73[1247]|74[037]|75[47]|76[02359]|77[^1678]|78[1567]|80[^079]|81[^19]|83[0128]|84[3578]|85[^1235]|86[02345]|87[028]|90[^025]|91[^1]|92[0589]|93[^235]|94[0179]|95[12469]|97[^4567]|98[0459]|281|458|520|828|8(00|33|44|55|66|77|88))[2-9]\d{6}

The latest version of UCDialplans.com does this now, but if you've previously applied a UCDialplans.com ruleset using the "In-Country Only" option, you can either manually update your National voice route, or you can run one of the below scripts, which will seek out any US/CA-National voice route and apply the fix. If you use any of the other options, you aren't affected.

How can I tell which option was used?

Easily. The voice route for either of the other options are MUCH smaller in size. If your voice routes look like this, you are not affected.

US/Canada: (?!(24[26]|26[48]|284|345|441|473|649|664|721|[67]58|767|784|8[024]9|86[89]|876|900|976))[2-9]\d\d[2-9]\d{6}

US/Canada/Caribbean: (?!(900|976))[2-9]\d\d[2-9]\d{6}

The Fix is In

The below PS commands will append tollfree prefixes to the problematic voice route. Run whichever is appropriate for your environment, and BACKUP before you run this.

Skype for Business Fix

$Routes = Get-CsVoiceRoute | Where {($_.Identity -eq 'US-National' -or $_.Identity -eq 'CA-National') -and $_.NumberPattern -notlike '*[2-9]\d\d[2-9]\d{6}' -and $_.NumberPattern -notlike '*8(00|33|44|55|66|77|88)*'}; ForEach ($Route in $Routes) {Set-CsVoiceRoute -Identity $Route.Identity -NumberPattern $Route.NumberPattern.replace(")[2-9]\d{6}","|8(00|33|44|55|66|77|88))[2-9]\d{6}")}

Microsoft Teams Fix

$Routes = Get-CsOnlineVoiceRoute | Where {($_.Identity -eq 'US-National' -or $_.Identity -eq 'CA-National') -and $_.NumberPattern -notlike '*[2-9]\d\d[2-9]\d{6}' -and $_.NumberPattern -notlike '*8(00|33|44|55|66|77|88)*'}; ForEach ($Route in $Routes) {Set-CsOnlineVoiceRoute -Identity $Route.Identity -NumberPattern $Route.NumberPattern.replace(")[2-9]\d{6}","|8(00|33|44|55|66|77|88))[2-9]\d{6}")}


My apologies to anyone caught up by this bug. If you see me in the streets, feel free to slap me.

Undocumented behaviour with dialplans on MS Teams

Who am I talking to? Eartha Kitt? Sorry, wrong number. I was trying to call my car.

I've been spending a lot of time recently messing around with MS Teams dialplans thanks to the recent release of SBC-specific translation rules. For the most part, the overall structure is the same as with Skype for Business on-prem. Some PowerShell commands are different, but most of what you already know is usable in Teams.

The biggest issue with Teams is that there hasn't been a UI for managing your Teams Enterprise Voice layout. This can be an issue if you are big into Direct Routing.  Thankfully, Microsoft recently made available what is hopefully the first round of UI enhancements to the Teams Admin Center. You can now create and manage dialplans in the UI. Microsoft has dutifully updated their documentation to help people navigate these new features.

However, there are some undocumented features that open up some interesting scenarios. Along with these are some extremely frustrating bugs that you should be aware of.

The Hidden Feature

In Skype for Business on-prem, there is a global dialplan that applies to users when there isn't an appropriate site or user-level dialplan. In the Teams documentation, the tenant Global dialplan serves a similar purpose, but there are additional features outside of what can be done on-prem. 

In Teams, Microsoft has published a series of very simple pre-defined dialplans called service country dialplans that apply to nearly every country in the world. You can see these by running Get-CsDialPlan and Get-CsVoiceNormalizationRule. They do the bare minimum, and don't do any real validation. These dialplans cannot be modified. 

The documentation goes on to say that you can define normalization rules in the tenant global dialplan or any number of tenant user dialplans. If a user tenant dialplan isn't assigned to a user, they will be assigned an effective dialplan of whatever is in the tenant global dialplan and the default service country dialplan for that user. Similarly, if a user is assigned to a tenant user dialplan, their effective dialplan will consist of the contents of the assigned tenant user dialplan along with the user's service country dialplan. The tenant dialplans have higher priority than the service country ones. 

The hidden feature is that if you have defined any normalization rules in the tenant global dialplan, those normalization rules will appear at the top of the list of any new tenant user dialplans created afterwards. This can be very useful in large organizations with many sites, but have a common set of extension dial rules. You simply have to define the rules once in the tenant global dialplan, and they will show up in every new tenant user dialplan created from that point forward. This applies to dialplans created in both PowerShell and the Teams Admin Center.

Unfortunately, there isn't a real link between the tenant global and tenant user dialplans. They are simply copies of the original. If you make a change to the tenant global dialplan, it won't be reflected in all your other dialplans. As with Skype for Business, you're still stuck with using PowerShell to manage bulk changes. Overall, it might make deploying new dialplans just a little bit simpler if you choose to take advantage of this undocumented feature.

The Bugs

Unfortunately, there are also a number of annoying bugs/oversights with the new UI. Some are cosmetic, but some will force you to re-create your dialplan if you accidentally encounter one.

Description Discrepancies

If you've used UCDialplans.com to create your dialplans, you may have noticed that I make extensive use of the Description field to let people know where the rules came along with a copyright notice. I even put in linebreaks to make it readable in the SfB Control Panel UI. These descriptions are rather lengthy and go well beyond 255 characters (sorry, not sorry).

Here are my grievances with the Description field in the MAIN dialplan UI:

1. The Teams admin center dialplan UI doesn't respect the linebreaks and just renders a wall of text.
2. The UI throws an error that you can only have up to 255 characters, when PowerShell allows a much higher number.
3. The UI also throws errors about "invalid characters", such as !@#$%^*()=/\[]{}:;?<>+' Again, PowerShell has no issue with any of these characters.

What makes this even more odd, is that the UI for the individual normalization rules exhibits NONE of these issues. It renders the linebreaks properly, and doesn't complain when the description is longer than 255 characters and includes "special" characters.

The External Dialing Prefix Bug

This is a big bug, and am quite surprised it made it through QA. For a bit of background, you can configure Teams so that users can dial an external access prefix (like 9), before dialing an external number. For users coming from a legacy PBX, this is undoubtedly a boon and doesn't force them to retrain years of muscle memory. It also makes off-hook dialing more reliable. For an extended explanation of this feature, go waaay back to one of my posts from 2011 on the topic. I haven't tested it myself, but the details should still apply to Teams.

Most customers don't bother with this feature, and have been happily creating dialplans in Teams/Skype for Business Online for years via PowerShell without defining an external dialing prefix. However, if you create a new dialplan or edit an existing one in the UI, you will not be able to save your changes unless you set an external dialing prefix.

Also, this seems to have found itself into the PowerShell commands themselves, but not completely. You can create a new dialplan with New-CsTenantDialplan without an external dialing prefix, and you can make changes to an existing dialplan with Set-CsTenantDialplan, but if you try to remove an external dialing prefix via PowerShell by setting it to either $NULL or "", you will be denied.

As long as you don't set a prefix, you should be fine. Don't do what I did and set a prefix in the UI to see what happens, and expect to be able to remove it in PowerShell. This forced me to redo my entire dialplan from scratch. Thankfully, UCDialplans.com took care of that in a minute.

Spelling Mistakes

A minor quibble, but one that will force me to keep a close eye on (boo hoo me). The New/Get/Set-CsOnlinePSTNGateway command has options for setting inbound and outbound translation rules:

• InboundPstnNumberTranslationRules
• InboundTeamsNumberTranslationRules
• OutboundPstnNumberTranslationRules
• OutbundTeamsNumberTranslationRules

Do you see the problem?

Someone's spellcheck didn't work.

Conclusion

I'm trying to get this in front of the right people at Microsoft to get them to fix. Hopefully, this half of my post will have a limited shelf life. Have any of you experienced any of these issues or come across any others? Let me know in the comments.

Lync 2013 Edge server bug when adding new server pools

I had this happen to me a few weeks ago, and a co-worker had it happen to him today, so I figure it should be mentioned since nobody on the Internet seems to have a proper fix for it.

I installed a new Lync 2013 Standard Edition server in an existing Lync 2013 environment with an established edge server pool.  I created a test account on the new pool and began running through the usual testing scenarios.

All seemed well and fine until we tested federated communications.  An external party could establish an IM or A/V session with the internal user homed on the new pool and see their presence, but the internal user couldn't see presence nor establish a new IM or A/V session with anybody outside the company.

Puzzling to be sure, since everybody else in the company was OK.  I first checked to see if I could telnet to the typical ports required for edge functionality, and all was good. I then logged onto the edge to see if there were any relevant events.

Lo and behold, there was this interesting little warning:

LS Protocol Stack Event 14402
Multiple incoming connections on internal edge from non-internal servers.
In the past 289 minutes the server received 4 incoming connections on internal edge from non-internal servers. The last one was from host newlyncfe.contoso.com.
Cause: This can happen if an internal server is not present in the list of internal servers on the Access Edge Server.
Resolution:
If the server is a valid one, you need to add it to the list of internal servers on the Access Edge Server. If the server is invalid, you may be under an attack from that server.

This left me scratching my head.  How do you add a Lync 2013 server to the "list of internal servers"???  I recalled back to the olden times of OCS, where you had to manually add every valid server name to the edge, but you don't do that anymore, thanks to the wonder of the Topology Builder.

Internet searches brought up several cobweb-covered webpages from the late 2000's, all of them relating to OCS.  No good.  I re-ran Setup on the edge server, hoping that would trigger something and make things all good, but nope.

Finally, I decided to restart the Lync Server Access Edge service (during a period of low activity of course).  Once it restarted, all my troubles went away.  The user on the new front-end was able to initiate sessions to external users, and the 14402 errors on the edge went away.

My next step was going to be blindly restarting other services followed by a last-ditch server reboot. Since that turned out to be unnecessary, if this saves someone from doing a server restart (which would fix the problem too), that's awesome.

So, while Lync normally does an excellent job of detecting new servers and functioning fine without service restarts or server reboots, this is one situation where its not doing that.  It seems the Access Edge service caches the list of valid internal servers at service startup, and doesn't refresh that cache until the next service restart.

Because this seems to be one of the few instances where a service restart is required to maintain connectivity with a new downstream server, I would classify this behaviour as a...

Lync Address Book Weirdness

So, I'm at a fairly large client that's moved from Lync 2010 to 2013.  The server-side migration completed a while ago, and the client migration is in progress.  Enterprise Voice is enabled for a select group of people.

Some people have noticed that when they search for a specific Lync user, the only object that got returned was an object for their Active Directory administrative account that had only a phone number defined (not a Lync-enabled account).  The Lync-enabled "normal" Lync entry didn't show up in the search, even though it did appear in the GALContacts.db file (the Lync locally cached address book file).

The really weird thing was that the object showed up like a phone contact object, but one that had presence working.  The affected user could toggle their presence, and the contact object would dutifully change itself.  However, since it wasn't a Lync object, you couldn't start an IM (but you could call the associated number).

This user's first name isn't actually "Admin" in case you were wondering.

Another weird thing was that the very first time you pulled up that person's name from a search, it would briefly show the proper Lync-enabled account, but switch to the phone contact object view after a half-second or so.

Yet ANOTHER weird thing, was that after viewing the admin account object in Lync a few times, usually after viewing the contact card, the next time I did a search for the same name, only the phone number appeared.  However, I could still look at the contact object for the phone number and it would show the user's email address.

As mentioned previously, the address book had the correct contact information (verified by opening the GALContacts.db file in Notepad....the results ain't pretty, but it works), as well as the information from the AD admin account, but Lync was choosing to show the non-Lync enabled admin account in searches.

When I compared the affected admin accounts with others, I noted that some users showed up properly, but others didn't.  More comparisons showed that for the incorrect information to show up, the following had to be true:

1. Both the Lync-enabled account and the AD admin account had a phone number defined
2. Both the Lync-enabled account and the AD admin account had the same email address defined.

Removing the email address from the admin account, re-generating the Lync address book via Update-CSAddressBook, and re-downloading the updated address book file fixed the issue. 

So, what happened?  When you do a search, the Lync client will look at the address book, your own Outlook contacts, and any social connectors you may have associated with Outlook (like Facebook or LinkedIn).  If the same name shows up in multiple places, Lync will try to consolidate all the information and show only one contact object.

Sometimes, the process goes awry. In this case, it appears that Lync saw the same email address for both the Lync-enabled AD account, and the non-Lync enabled AD admin account, and incorrectly presented the non-Lync enabled account as the single object.

I'm guessing this is a bug, because Lync really should place priority on a Lync-enabled object over a non-Lync one for presentation.  

Presentation Issues After Moving Lync 2013 Fileshare

Recently, we moved the Lync 2013 file share for an enterprise pool to a new location that was more resilient than the original. Everything seemed to go well, except that file sharing from internal to external users stopped working. Also, Lync 2010 users were suddenly unable to share Powerpoint presentations.  Lync 2013 users seemed fine.

When attempting to view or share a Powerpoint presentation on Lync 2010, the users got the following message:

This slide couldn’t be downloaded. Please contact your support team. Error reason: File not found.

The fact that 2013 clients were unaffected by the Powerpoint presentation is not unusual.  Lync 2013 offloads Powerpoint content rendering to the Office Web App server.  Lync 2010 clients are unable to utilize this feature, and fall back to the original method, which is managed and rendered from within the Lync server itself.

Of course, my first step in troubleshooting is to plug the error message into my favourite search engine. It came up with several different options, mostly dealing with issues with Office Web Apps, which we weren't experiencing. One blog post in particular caught my attention, because it talked about the same issue happening after a file share move:  http://paulbrown.us/blog/2011/11/02/how-to-change-lync-server-file-store-location/.  Incidentally, it appears that the writer (Paul Brown) enjoys sitting on active train tracks in his spare time, which may account for his lack of activity since May 2013.

So, Paul Brown (RIP) managed to solve the issue for Lync 2010, which definitely helped me solve it for 2013.  The key difference between 2010 and 2013 in this case is that the MeetingContent and MeetingFiles virtual directories in Lync 2010 don't exist in 2013.  All that seems to have been rolled into the CollabContent virtual directory in Lync 2013.

When I looked at the advanced settings for the CollabContent virtual directory, the Physical Path still pointed to the old file share location.  I updated the location using the new path in both the Lync Server External Web Site and Lync Server Internal Web Site and repeated this on each server in the affected front-end pool, followed by an IISReset.  This was enough for both file sharing and presentation sharing to function properly for Lync 2010 and 2013 clients.

This is obviously a bug within the Topology Builder/Lync Deployment Wizard, because this should have been changed automatically by running the Deployment Wizard after the topology change to move the file share.

As a final note, if you're planning on sitting on train tracks, make sure to keep an eye out for any oncoming trains, because, well you know, YOU'RE SITTING ON TRAIN TRACKS!

High Processor Utilization on Lync 2013 Front-End Servers

UPDATE (2015-Oct-02): The problem has finally been fixed!  Took nearly 2 years, but here's a link to the KB article. The fix is in the September 2015 CU for Lync Server 2013.  The cause is "because the topology snapshot was recomputed multiple times by the Response Group Service."  Thanks to @sublimeashish for telling me. 

We have a customer who is about to migrate from Lync 2010 to Lync 2013.  They've got a few lightly loaded Lync 2013 Enterprise Edition pools with 3 servers each.  All are running Windows 2008 R2 Standard Edition on VMWare.  All patches are up-to-date.

For inexplicable reasons, some of the servers will suddenly see their processor utilization spike to near 100% for extended periods of time, when their typical utilization is less than 5%. A look at Task Manager shows two instances of the W3WP.exe service (IIS web service) that are consuming large amounts of processor resources.  There are no events in the Event Logs to indicate an issue.

Performing an IISReset on the affected node makes the processor go back to normal, but this is obviously not a real solution.  We opened a ticket with Microsoft PSS, and they confirmed there are others seeing the same thing.  It seems the source of the problem is the "garbage collection" process in the LyncIntFeature and LyncExtFeature application pools in IIS.  Recycling those pools makes processor utilization return to normal (for a while at least).

Microsoft is actively working to resolve the issue, and I will post a permanent solution for all to see as soon as one becomes available.

UPDATE:  Thanks to @dannydpa  on Twitter, it appears the trigger may be Lync topology publishing. I confirmed this by updating the topology and publishing it.  Less than 10 minutes later, all the servers processor utilization spiked.  Recycling the aforementioned apppools resolved the issue.

To help others with this issue, I've created a little Powershell script that will recycle the LyncIntFeature and LyncExtFeature app pools for all Lync servers.  For the script to work, you need to make sure that remote management is enabled on all Lync servers.  On Windows Server 2012, this is on by default, but in Windows 2008 R2, you need to log on locally and run: Enable-PSRemoting -Force before running the script.

$WebPools = (Get-CSService -WebServer).PoolFQDN

ForEach ($Pool in $WebPools)
{
  $PoolMembers = (Get-CSPool $Pool).Computers
  Foreach ($Computer in $PoolMembers)
  {
    Write-Host "Resetting LyncExtFeature and LyncIntFeature app pools on $Computer"
    $Session = New-PSSession -ComputerName $Computer
    Invoke-Command -session $Session -ScriptBlock {Restart-WebAppPool LyncExtFeature}
    Invoke-Command -session $Session -ScriptBlock {Restart-WebAppPool LyncIntFeature}
    Remove-PSSession $Session
  }
}

Location Based Routing Bug with External Users

Location-based routing is a relatively new addition to Lync 2013.  It wasn't part of the initial release, but the first cumulative update added this much asked for feature.

In a nutshell, location-based routing routes calls based on the network subnet the user is calling from, rather than their defined home server pool. So, if a user is in the Rome office today, all their calls can route via the Rome PSTN gateway.  If that same user goes to the London office tomorrow, their calls will route out the London PSTN gateway.

Since then, I've done a number of deployments where location-based routing was used extensively.  During one remote deployment at a large company, I noticed that my calls were not routing via my assigned voice policy.  They were routing via a PSTN gateway that was not defined in the policy I should have been using. In fact, my calls were routing out from South America (I'm in Canada)!

After much troubleshooting, I realized that Lync was routing my calls based on the subnet of my home network.  Turns out that my home network subnet of 192.168.2.x matched up with a corporate subnet assigned to that South American location. Location-based routing was configured to route calls for that subnet out through the South American PSTN gateway.  To verify this, I changed my home subnet to another one used by location-based routing. Lo and behold, my calls started routing out that PSTN gateway.  Finally, I changed my subnet to one not defined for location-based routing, and my calls began routing as per my assigned voice policy.

I wasn't using a VPN, and as such, I was connecting through the Lync edge server. Lync was incorrectly using my home's private subnet for call routing decisions. Since administrators have no control over the subnets used by external users, this could obviously lead to many issues, not to mention increased telephony charges for calls routing out through the wrong location.

I filed a bug report with Microsoft, who confirmed this bug and promised a fix in a soon-to-be-released cumulative update.  I don't know the details of the fix, but I imagine it will be one of two things:

1. Lync's behaviour will change to ignore private subnet information for external connections for the purposes of location-based routing. This is probably an easy fix, and my money's on this one. 
2. Lync's behaviour will change to use the detected public IP address assigned by the ISP for routing decisions.  I like this one, because it gives administrators the option to include public networks for location-based call routing decisions. Its unlikely that many administrators would go through the trouble to do this, but it would be nice to have the option.  

I doubt this bug will affect may deployments, but its always good to be aware.

UPDATE (08-Jan-2014): The issue has been fixed in the January 2014 Cumulative Update.  Read the KB article here. 

Errors After Installing Office Web App Server on Non-Standard Drive

I haven't come across anything noting this, but beware of installing Office Web App Server 2013 on anything other than the standard C:\Program Files\Microsoft Office Web Apps on Windows Server 2012.  I installed OWA on the D: drive to keep with a corporate policy for all installed applications. The installation itself went without a hitch, and I was even able to create a new Office Web App farm.  However, when I tried to run any other OWA Powershell command (like Get-OfficeWebAppFarm), I would get the following error:

Get-OfficeWebAppsFarm : The content type text/html; charset=utf-8 of the response message does not match the content type of the binding (application/soap+msbin1). If using a custom encoder, be sure that the IsContentTypeSupported method is implemented properly. The first 1024 bytes of the response were: '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>IIS 8.0 Detailed Error - 500.19 - Internal Server Error</title>
<style type="text/css">
<!--
body{margin:0;font-size:.7em;font-family:Verdana,Arial,Helvetica,sans-serif;}
code{margin:0;color:#006600;font-size:1.1em;font-weight:bold;}
.config_source code{font-size:.8em;color:#000000;}
pre{margin:0;font-size:1.4em;word-wrap:break-word;}
ul,ol{margin:10px 0 10px 5px;}
ul.first,ol.first{margin-top:5px;}
fieldset{padding:0 15px 10px 15px;word-break:break-all;}
.summary-container fieldset{padding-bottom:5px;margin-top:4px;}
legend.no-expand-all{padding:2px 15px 4px 10px;margin:0 0 0 -12px;}
legend{color:#333333;;margin:4px 0 8px -12px;_margin-top:0px;
font-weight:bold;font-size:1em;}
a:link,a:visited{color:#007EFF;font-weight:bold;}
a:hover{text-decoration:none;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0'.
At line:1 char:1
+ Get-OfficeWebAppsFarm
+ ~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-OfficeWebAppsFarm], ProtocolException
    + FullyQualifiedErrorId : System.ServiceModel.ProtocolException,Microsoft.Office.Web.Apps.Administration.GetFarmCommand

Uninstalling Office Web App Server from the D: drive and reinstalling on the default path of C:\Program Files\Microsoft Office Web App fixed the issue.  Again, this was on Windows Server 2012.  No idea if the same thing happens on Windows Server 2008 R2.

Just an FYI for anyone looking for answers for this specific issue.

Viewing Extensions in the Lync Client

In companies that use extensions off the main number, I’ve always recommended they format their numbers using the main office number for the “base” and followed by the extension (ie. tel:+14165551111;ext=222 for North America, or tel:+4420555111;ext=222 for elsewhere).  In Active Directory, I’ve recommended they use the same type of format (ie. +1.416.555.1111 X222), and use the Company_Phone_Number_Normalization_Rules.txt to enforce the same formatting.

Background

For those unfamiliar with how Lync displays phone numbers, the Lync client will only display numbers it can parse from Active Directory.  It doesn't show the actual TelURI defined in the Lync Control panel, which is a good thing, because there are often additional settings applied there that administrators don't want to have appear in the address list.  The Company_Phone_Number_Normalization_Rules.txt file is helpful in re-formatting phone numbers in AD to the E.164 format we like to see in Lync.  For a good overview of the Company_Phone_Number_Normalization_Rules.txt, see Jeff Schertz's blog entry on the topic.

The Problem

I've found that when using the Lync 2010 client, North American phone numbers would always display with the typical North American formatting properly along with the extension when dialed, (ie. +1 (416) 555-1111 X222).  However, when applying similar rules to international numbers, the extension would never display.  It would show +4420555111 instead of +4420555111 X222.  The number would be dialed properly, but it would be confusing to the user, because the extension wouldn't appear.

Fast forward to Lync 2013 client, and the situation appeared even worse.  Even North American phone numbers with extensions wouldn't appear properly when dialled.  I assumed this was a client bug that was getting worse with each client revision.

The Solution...Sort of

Fellow Lync MVP Tom Pacyk of ConfusedAmused.com brought to my attention a little known (well, little known by me) feature added with the October 2012 update for the Lync 2010 client.  In short, the client update finally fixed the bug for international extensions, but with a catch....you had to first add a new client policy entry to your Lync client policy.  If only the global client policy is being used, the commands are:

$x = New-CsClientPolicyEntry -Name "ShowExtensionInFormattedDisplayString" -Value "True"
$y = Get-CsClientPolicy -Identity Global
$y.PolicyEntry.Add($x)
Set-CsClientPolicy -Instance $y

Since this policy setting was a part of the Lync 2010 client updates, it would be very easily missed by someone whose focus has changed to Lync 2013.  I can only assume the same settings are required in a pure Lync 2013 environment, although there is very little information on the topic.

If you're running Lync Server 2010, it appears as though you need at least the October 2012 Lync 2010 server updates for this to work.  I tried to apply this client policy entry to a client's deployment whose got Lync 2010 and 2013 in parallel, but the Lync 2010 environment is a bit out of date.  The policy entry showed up properly when I ran the following:

$y = Get-CsClientPolicy -Identity Global
$y.PolicyEntry

...but neither my Lync 2010 or 2013 clients would show extensions properly.  Applying the policy entry to a 2010/2013 server mix where the 2010 environment was up-to-date worked fine...at least for 2010 clients...mostly.

So, make sure your Lync 2010 infrastructure is up-to-date before attempting to apply this setting.

What's Still Not Working

I setup a few test users with extensions off a fictitious Toronto, Canada and London, UK main office number.  The Toronto user's office number in AD was set to +1.416.555.1111 X111, and the London user's number in AD was set to +44.20.444.2222 X999.  Both users were setup in Lync with their TelURI properly defined as tel:+14165551111;ext=111 and tel:+44204442222;ext=999 respectively.

A Company_Phone_Number_Normalization_Rules.txt file was placed in the ABFiles folder on the shared folder for the Lync server pool with the following rule to ensure Lync would be able to parse phone numbers  with extensions in Active Directory properly:

\+(\d{6,14})\D+(\d+)
+$1;ext=$2

When testing with the Lync 2010 client, North American numbers showed up properly in both the list view and the dialing view, as shown below:

However, international numbers did show numbers a bit differently between the views, using the ;ext= from the TelURI instead of the X:

Interestingly enough, the Lync 2013 client doesn't seem to fully honour the extension setting for either North American or international numbers.  The number shows up properly when hovering over the phone icon, but only shows the main number when dialing:

North America

United Kingdom

Needless to say, this can be very confusing to the average user.

This appears to be a bug introduced with the Lync 2013 client.  I've brought this up with the Lync product managers, and we should see a future patch that will address the issue for Lync 2013 clients.

UPDATE (10-July-2013): The July 2013 Update for Lync 2013 addresses the extension presentation for North American numbers, as shown below.

However, non-North American numbers with extensions still don't show up properly.  I've been told this should be in a separate fix coming later.

UPDATE (16-June-2014):  I didn't notice but at some point in the past year, a patch seemed to mostly fix non-North America number presentation during dialing.  The number shows the extension as part of the TelURI, but it really should show the display number. At least its consistent with how the Lync 2010 client formats things.

Lync Dialing Rule Optimizer Gets Optimized

It was recently brought to my attention that some of the normalization rules created by the Lync Dialing Rule Optimizer in certain cases were not being used by the Lync client.  Specifically, the issue only arises if you select the option for the Optimizer to create 7-digit local dialing rules (only available for North America dialing rules).  The 7-digit rules are simply never used.  If you enter a 7-digit number that should be normalized to a 11-digit E.164 North American phone number, it doesn't happen.  Interestingly, if you use the Lync Voice Routing Test Case applet in the Lync Control Panel, you'll see that it normalizes just fine.

I did some testing of my own, and found out that the first part of the 7-digit rules were causing the Lync client to ignore the entire rule.  The first part of each 7-digit normalization rule is this cryptic piece of regex:

(?=^\d{7}$)  

This bit of regex says that whatever number is entered has to be a total of 7 digits long. The rest of the regular expression dictates the allowable first 2 or 3 digits for that particular area code.  At the time, this was the only way I could think of to ensure the total number of digits entered was exactly 7.

I put the question to Microsoft, who acknowledged that the server and the client can use different criteria for evaluating the validity of regular expressions.  It may be fixed in a future patch, but rather than waiting, I went about figuring out how to ensure 7-digit numbers without that bit of cryptic regex at the beginning.

After a good amount of research, work, and testing, I was able to figure out a way to ensure 7 digits in a much simpler way.  At the same time, I got a bit of regex schooling by Dan Berry of Acrodex. He told me I had way too many brackets in my regular expressions, so with his prompting, I was able to reduce the number of brackets by quite a bit.  He also gave me some other ideas for reducing the length and complexity of my regular expressions. 

The end result is a much shorter and more robust set of regular expressions for all the North American local dialing rules.  For example, one ruleset for Toronto, ON used to be 820 characters long.  With the new optimizations, the character count is down to 621.  This reduction can result in fewer rules, especially in larger cities. 

If you've previously used the Optimizer to create your rulesets for 7-digit dialing, I recommend you apply the updated rules.  If you subscribe to the monthly email rule update, then you'll get the updated ruleset starting next month.  If you come across any issues with the new rules, please let me know.

Inbound Number Normalization Bug in Lync (FIXED)

I came across an issue recently where a North American company had deployed Enterprise Voice using the Lync Dialing Rule Optimizer.  Outbound calling would work fine, but inbound calls would fail with a busy signal.  I was testing against a number that was supposed to route to an Exchange auto-attendant.

I ran a trace using the Snooper tool and found a big glaring red error staring at me:

404 - No matching rule has been found in the dial plan for the called number. 

The detailed error looked like this:

Direction: outgoing;source="local"
Peer: lyncpool.contoso.com:58964
Message-Type: response
Start-Line: SIP/2.0 404 No matching rule has been found in the dial plan for the called number.
From: "604xxxxxxx";epid=5A81C7C2F0;tag=b356e0ebc3
To: ;tag=FCA83E847F99452AC4A563DB1552D6C4
CSeq: 2389 INVITE
Call-ID: 9d03fadf-282b-461b-912b-fbefe95a111b
ms-application-via: LYNCMON.contoso.com_LyncMonitoring;ms-server=LYNCFE.contoso.com;ms-pool=lyncfepool.contoso.com;ms-application=51FB453D-5B9F-45df-83B4-ADD1F7E604A8
Via: SIP/2.0/TLS 10.0.5.10:58964;branch=z9hG4bK71da34d1;ms-received-port=58964;ms-received-cid=18FC00
ms-diagnostics: 14010;reason="Unable to find an exact match in the rules set";source="LYNCFE.contoso.com";CalledNumber="4165551111";ProfileName="HeadOffice";appName="TranslationService"
Server: TranslationService/4.0.0.0

The inbound phone number was coming in as 10-digits, and excluded the North American country code 1 (which isn't unusual for a lot of phone providers).  I knew the normalization rules were working properly for outbound calls, but I couldn't figure out why inbound was failing.

I zeroed in on my NA-National rule.  The rule is formatted as follows:

^1?([2-9]\d\d[2-9]\d{6})$  ----NormalizeTo----> +1$1

This rule will accept any 10-digit valid North American formatted telephone number OR any valid 11-digit North American formatted telephone number starting with a 1.  Users in many areas tend to use 10-digits and exclude the leading 1 when dialing phone numbers, or they may use the full 11-digit proper format. The NA-National rule deals with both these cases by starting the rule with 1?.  When a question mark is present in a regular expression, it means that the preceding element is optional.  So, in our case, the NA-National rule will match both 10-digit and 11-digit North American numbers.

Unfortunately, there seems to be a bug in earlier versions of Lync Server 2010 (prior to the March 2012 update from what I can tell) that results in inbound numbers failing to normalize against a rule that includes a question mark.  When I removed the 1? from the rule, inbound calls worked as expected.

Thankfully, it appears that someone at MS has already caught this and fixed it somewhere between the November 2011 Lync Server update and the March 2012 update.  I didn't try to figure out which update fixed it, but I knew it was broken on a server running the November 2011 updates, and was fixed with the March 2012 update.

If you keep up-to-date with your Lync server patches, you won't come across this bug.  So, make sure you have the latest Lync Server updates applied before running the Optimizer for North American deployments.

Finally! Exchange 2010 OWA Redirect Bug Fixed!

Ever since the release of Exchange 2010 SP1, there have been numerous reports of users being prompted that they have to be redirected to another server, instead of the silent redirect that had worked like a charm until SP1.  I blogged about this in September after I experienced the issue for myself.  Since then, it's been one of the most-read pages on my blog.

I've just received word that the issue has finally been fixed in Exchange 2010 SP1 Rollup 2.  This KB article describes the issue and the confirmation that it's been fixed by RU2.

You can get Rollup 2 here.

E2K7 OWA Redirect Bug Introduced with Exchange 2010 SP1

I just deployed Exchange 2010 SP1 to an enterprise customer that has a mix of Exchange 2007 and Exchange 2010 users.  It seems that SP1 has introduced a rather aggravating and obvious bug.  Before I get into that, I'll give some background on how Exchange 2010 coexists with previous versions of Exchange.

When you have a mix of Exchange 2010 and older versions in your environment, you have to do a bit of work to make the two work together for your external users.  In a nutshell, you use Exchange 2010 Client Access Server (CAS) as your primary entry point for all external users. 

Say you use owa.company.com as your externally accessible URL.  If an Exchange 2010 user logs in from the Internet, the Exchange 2010 CAS will do its thing and the user will get a nice Outlook Web App screen.

If an Exchange 2007 user logs in using owa.company.com, the Exchange 2010 CAS will redirect the user to an externally accessible Exchange 2007 CAS using a different URL (like legacy.company.com).  The redirection is silent, but the user may notice their browser changed to legacy.company.com. 

How the redirect is handled is managed by the LegacyRedirectType setting in the Exchange 2010 OWA virtual directory.  In most cases, LegacyRedirectType is set to Silent.  To see what the setting is in your environment, run:

Get-OWAVirtualDirectory -Server <CASservername> | FL Identity, LegacyRedirectType

In SP1, this redirection is no longer silent.  When your Exchange 2007 user logs in via owa.company.com, they are presented with this screen:

The text reads:

A temporary change has occurred that requires you to connect to a different server.  To connect, click the button below.  For security reasons, you'll be asked to enter your user name and password again.

Sure enough, when you click Connect, you are redirected to legacy.company.com, where you have to re-enter your user information.

Thankfully, the same sort of thing doesn't seem to happen with Outlook Anywhere or ActiveSync clients.

I checked the LegacyRedirectType value on my 2010 SP1 CAS boxes and they are all still set to Silent.  I've read the issue occurs because the OWA virtual directory value for LegacyRedirectType is being ignored.  Apparently, this bug was to be addressed in Exchange 2010 SP1 RU1, but that wasn't the case.  Hopefully, Rollup 2 will fix the issue.

This is an extraordinarily unfortunate thing to have been introduced with SP1.   If you have a mixed Exchange 2007/2010 environment, I suggest you wait before deploying SP1.

UPDATE (01-Dec-2010):  Thanks to an anonymous commenter below, there is a workaround for the OWA redirect issue.  Navigate to C:\Program Files\Microsoft\Exchange\v14\ClientAccess\Owa (or whereever you installed Exchange) and edit the casredirect.aspx with Notepad.

Add the following line just under the existing meta-tag that starts with <meta http-equiv...:

<meta http-equiv="refresh" content="0;URL=https://legacy.domain.com/owa">

Replace legacy.domain.com with whatever you are using for your redirect URL.  Save the file and issue an IISRESET from the command line.  When your legacy users logon to OWA, they will still see the redirect page, but users will not have to press the button to continue.  It should automatically switch them to the legacy URL. 

It's not perfect, but at least its something.  Thanks again to the anonomous user who brought this to my attention!

FINAL UPDATE (14-Dec-2010): The redirect issue has finally been fixed in Exchange 2010 SP1 Rollup 2!  Read more about it here.

Warning about OCS R2 RGS and July 2010 Updates

If you've applied the July 2010 cumulative updates to your OCS R2 environment and you use the Response Group Service, you'll want to take note of this update from Microsoft:  http://support.microsoft.com/default.aspx?scid=kb;en-us;2401878&sd=rss&spid=14030

In a nutshell, if you make any changes to a response group or create a new one, calls to the affected response group could fail unless you restart the service.  This issue will be fixed in the September 2010 cumulative update package.

Thanks to RPasztor for bringing this to my attention.