The Importance of Lync Location Policies

Lync Location Policies are used to specify how to deal with emergency calls originating from Lync clients.  Quite a bit has been written about location policies and how they are used in E911 emergency dialing scenarios (for a few examples, see Technet or Mark King's UnplugthePBX blog).

Unfortunately, E911 only exists in the US, so anybody reading the documentation outside the US might conclude that location policies aren't required, and that emergency calls can be routed as other normal calls. While this is technically true, location policies still provide some critically important functions in specific scenarios that may not be apparent until you have to deal with them firsthand.

The Problem

Imagine the following two common scenarios:

1. Your company has a single site spread across several floors of a building and everyone is assigned an extension, instead of a full directly-dialable phone number (DID). Your company has a single main office external phone number which rings to reception.
2. Your company uses a centralized SIP trunk for normal calls, but all remote sites are configured with small PSTN gateways used to provide Lync-based local emergency dialing via a few analog lines.  Each user have their own dedicated directly-dialable phone number.  Users are assigned to a voice policy that ensures that emergency calls will route out the local PSTN gateway. 

For scenario #1, if someone within the office calls the emergency service applicable for their country (lets use 112, a common emergency number in Europe and many parts of the world), the number that will be presented to emergency services will be your main office number. 

For scenario #2, if someone from a remote office calls 112, the phone number that will be presented to emergency services will be that of the analog line assigned to the PSTN gateway. 

For both scenarios, you might be thinking "OK, so emergency calls go out the proper gateway, and emergency services will know where the office is. What's your point?"  You're correct in that emergency services will know the office location and will be able to dispatch personnel as required, but now think about these two factors:

1. The user who called 112 accidentally hangs up because that zombie finally caught up to him.  What happens next (other than you now have two zombies)?  We'll call it the "Hang-up of Death".
2. What happens when someone travels from their usual office to another and has to call emergency services because of....yeah, zombies?  It's the "Travelling Salesman of Doom".

The Hang-Up of Death

For #1, the typical response from emergency services is to re-dial the number that will show up on their console. In the single office/extension scenario, they will reach reception who may have no idea about who called 112, which can waste valuable time, depending on how fast the zombie horde grows.

In the remote site scenario with a dedicated analog phone line, emergency services would redial the analog number, and the call would ring endlessly, since its not likely that anybody considered someone calling inbound to an emergency number.  Or if someone was bright enough to forward incoming calls to reception, they still might not have any idea on who called 112 in the first place, again wasting valuable zombie killing time.

Travelling Salesman of Doom

For the unfortunate travelling user in #2, remember that unless you've implemented location-based routing, the user's calling behaviour won't change based on their current location.  If they call 112, then the call will route out the user's normal PSTN gateway which means emergency services will show up in their normal office, while our poor guy gets chased down by zombies.  If location-based routing has been implemented, then we still have to deal with the fact that reception at the local site still has no idea where the zombie attack is taking place.

The Solution

As you hopefully have discerned, location policies can help with both of these situations very well.

Firstly, location polices can be configured to notify one or more users by IM that someone's called emergency services. This way, when emergency services shows up, or if they have to call back, reception or security or whoever will have a bit more information about the situation than they would otherwise.

Secondly, location policies can be assigned to Lync network sites, which means that emergency calls will go out the proper gateway no matter where the user is typically situated.  This works very similarly to location-based routing, except that it's only for emergency calls.

Takeaways

If you are outside the US, you should still configure location policies for every site that has a PSTN gateway.  Consider doing the following:

• Configure Lync network sites and subnets for everywhere that Lync users could go.
• Configure location policies and assign them to each Lync network site, and set it to use the local PSTN gateway for emergency calls. Check the box marked "Enable Enhanced 9-1-1 (E9-1-1)" even if you don't use E911. This seems to be a prerequisite for emergency dialing via Location Policies to work at all, despite what Technet says.

• Assign the appropriate default location policy to every user, to act as a fallback in case they are outside a Lync-defined subnet. 
• Configure the notification URI field* in the location policy so that the appropriate people are aware whenever the configured emergency number is dialed.
• If using analog gateways for emergency dialing only, make sure that inbound calls route to the appropriate person, should emergency services need to call back (ideally the same person who's been assigned to the notification URI field).
• Consider populating the location information services (LIS) database even if outside the US, because this can help reception figure out where the person who called emergency services is located.

* A special note about the the Conferencing URI fields: You probably noticed these fields, which implies you can conference someone in when they dial the emergency number. Unfortunately, this doesn't work unless you're actually using an E911 provider in the US. If you're outside the US, you can enter values here to your heart's content, but it won't do anything. Boo-urns! Thanks to C. Anthony Caragol (check out his LyncFix blog) for bringing this annoying little tidbit to my attention.

For detailed instructions on how to configure Lync network sites and location policies for non-US locations, I strongly recommend Elan Shudnow's blog post on the subject.

Lync Enterprise Voice Misconceptions #3 - Inter-trunk Routing

During my travels across the Lync Enterprise Voice landscape, I come across numerous Lync deployments that make some fundamental mistakes in how to manage phone numbers, call routing and other details.  I've covered a lot of these details in parts of my OTHER Enterprise Voice Best Practices, but its obviously a lot to take in, and I've seen enough of the same sort of errors in the wild that prompted me to do this series.

Last week's episode talked about how to properly use the external access prefix in Lync. This week, we're talking about...

#3 - Inter-Trunk Routing

If you've ever manually created trunk translation rules for a PSTN gateway in Lync, you've probably noticed the "Associated PSTN Usages" section.

The wording is vague, and even Technet's description of the setting doesn't tell you much.

From what I've seen in multiple deployments, people tend to add the PSTN Usages that were created specifically for the particular trunk, which sort of makes sense if you use Technet's description as your only guide.

Some other places on the Internet take this idea even further, and imply that this section is used to define the PSTN usages that are allowed to use the specified trunk. If you think about it, you already accomplish this by defining the PSTN Usage/route combination without having to do anything else, so that can't be right either.

Now, you might be aware that Microsoft mentions in their Technet planning documentation that Lync can do "inter-trunk routing".  From their own documentation:

This new capability enables Lync Server to provide call control functionalities to downstream telephony systems. Intertrunk routing can interconnect an IP-PBX to a public switched telephone network (PSTN) gateway so that calls from a private branch exchange (PBX) phone can be routed to the PSTN, and incoming PSTN calls can be routed to a PBX phone. Similarly, Lync Server can interconnect two or more IP-PBX systems so that calls can be placed and received between PBX phones from the different IP-PBX systems.

In more general terms, Lync can pass inbound phone calls on to a different destination, which could be another PBX system or even back out to the PSTN.  So, now you're thinking "That's a great feature, but what does this have to do with Associated PSTN Usages?"  Since the title of this article says "Inter-Trunk Routing", it should dawn on you that "Associated PSTN Usages" has something to do with inter-trunk routing.  If it doesn't, then lie down a while and come back after a good nap.

As you hopefully have surmised by now, inter-trunk routing is done using "Associated PSTN Usages" from within a trunk.  The details behind how to do this are helpfully discussed in some of my own blog posts, Technet and also those by fellow Lync MVP Richard Brynteson:
http://ucken.blogspot.com/2013/06/inter-trunk-routing-in-lync-2013.html
http://masteringlync.com/2013/06/07/inter-trunk-routing-deep-dive/
http://technet.microsoft.com/en-us/library/gg425831.aspx

To summarize those blog posts, if an incoming call from a trunk can't be routed to an internal user, it will attempt to route the call to an alternate trunk, if there is a matching PSTN Usage/route combination assigned to the incoming trunk.

When people add the PSTN usages to a trunk that are already assigned to that very same trunk, incoming calls could potentially be sent back out the same way they came in, if there isn't a matching user in Lync. This could result in odd behaviour, including calls bouncing back and forth between a PBX and Lync, in what I call the "Telecom Routing Donut of Doom".  Fortunately, in most cases, adding PSTN usages to the trunk has no effect, since there is almost always a Lync user assigned to the incoming phone numbers.

I've seen numerous deployments where they've added PSTN usages willy-nilly (my Word-of-the-Day calendar entry for today).  If I see those, my next question is "Oh, so you're using inter-trunk routing."  If the reply I get back is some variation of  "What's inter-trunk routing?", then I remove those PSTN usages from the trunk configuration.

There you have it.  Tune in next time (not necessarily next week, since I'm bad at reading calendars) where we talk about something else that I haven't quite figured out yet.

Lync Enterprise Voice Misconceptions #2 - External Access Prefixes

During my travels across the Lync Enterprise Voice landscape, I come across numerous Lync deployments that make some fundamental mistakes in how to manage phone numbers, call routing and other details.  I've covered a lot of these details in parts of my OTHER Enterprise Voice Best Practices, but its obviously a lot to take in, and I've seen enough of the same sort of errors in the wild that prompted me to do this series.

Last week's episode talked about common errors when setting up resiliency/failover routing. This week, we're talking about...

#2 - External Access Prefixes

A lasting legacy of the PBX era is the concept of dialing an external access prefix to make a call to the outside world.  In many deployments, you have to dial a 9 or 8 or something to open a channel to the PSTN.  In some systems, when you press 9 (or whatever the prefix is, I'm going to assume the external access prefix is 9 throughout this article), you can hear the change in dial tone as you actually open a connection to the PSTN.

There are many reasons for using an external access prefix, one of the biggest is to provide a clear demarcation between internal and external calls. When dialing "off-hook" - meaning the user picks up the handset, hears a dialtone then starts dialing the number on the keypad - the PBX needs to know where to send the call based on the digits entered on the keypad.  This might sound obvious, but if you've punched in the digits 2-1-2-3, the PBX needs to know if you want to connect to someone at extension 2123, or if you're just in the process of dialing an external New York number that happens to start with 2123.  If you punched in 2123 and meant to reach extension 2123, then the PBX will connect your call to that extension and everybody's happy. But if you meant to dial New York number 2123334444, then you will be understandably frustrated that you were connected to extension 2123 instead.

The external access prefix deals with this issue in a neat and tidy way. When you dial off-hook, if you enter the external access prefix before dialing your number, then the PBX will not attempt to connect you to an internal extension as you punch in digits.  So, when you dial 92123, the PBX won't attempt to route your call to extension 2123, and will instead wait for the user to enter the remaining digits necessary to complete a call on the PSTN.

In many Lync Enterprise Voice deployments, Lync is connected to the PSTN through an existing PBX which gives Lync both access to the PSTN as well as phones on the PBX. Since Lync's PSTN access is via the PBX, then this means that calls that come from Lync usually needs to have the PBX's external access prefix added in front of PSTN-bound calls.

I've seen many deployments where administrators do one of two things:

1. They normalize dialed numbers in Lync to include the 9
2. They create a normalization rule where users have to dial 9, but they strip it out, leaving a properly formatted E.164 number, using a rule like ^9(\d{10})$ --> +1$1. They then add the 9 using a trunk translation rule before sending to the PBX.

With #1, this breaks Ken's universal E.164 rule, which states: "NORMALIZE EVERY NUMBER TO E.164!!!"  If you don't know what E.164 is, I recommend you check out this post on the topic.  For example, if your normalized number for our earlier example is +912123334444, that number is technically a phone number in Chinchwad, India (which incidentally is now on my list of funny-sounding place names).  91 is the country code for India, and 212 is the area code for Chinchwad.

The ramifications of choosing this path is that you have to adjust everything else in Lync to accommodate what you started with adding the 9 to all normalized numbers.  This means routes, and even phone numbers in Active Directory.  You may argue that this is OK for you, because you have a single site, and you don't have any need to call India. But AD phone numbers can populate users' contact lists on their mobile phones, and they most certainly don't want to be dialing India when they mean to call New York. You could do some additional work to deal with this situation, but it gets messy very fast and doesn't scale beyond a single site.  For example, what if you have a second site that doesn't use 9 for dialing externally?  If you have to route calls from the first site for redundancy, then you'd have to make all sorts of rules in Site B to deal with how you initially set things up in Site A.  As you add sites, this scheme would quickly become unmanageable. 

Case #2 is closer to the correct way to do things. Dialed numbers are ultimately formatted to E.164 standards which is good. Since the PBX ultimately requires numbers are prepended with a 9, we can do this with trunk translation rules. Its trivial to add a 9 to every call that routes out to the PBX. However, you shouldn't have to manually create normalization rules that force users to enter a 9.  If you recall from the beginning of this article, the external access prefix is meant to help people who do off-hook dialing. When dialing from the Lync softclient, there is no off-hook dialing.  You enter the digits, then you press the call button, or ENTER or whatever. Entering a 9 is superfluous in those cases.  However, you do want to ensure a good experience when people do off-hook dialing from a Lync desk phone. How do you support on-hook and off-hook dialing without forcing EVERYONE to enter an external access prefix in Lync?  Use the External Access Prefix option in the Lync dial plan, as shown below.   

This option does several things, firstly it allows people to dial phone numbers in Lync the same way they're accustomed to on the PBX without having to adjust normalization rules.  It also makes Lync work the same way a PBX phone works when dialing off-hook.  The kicker is that you're not forced to enter a 9 when dialing. You either enter a 9 when dialing from the Lync softclient, or you don't. Either way, the call goes through as expected. Same thing for the Lync deskphone. You're not forced to enter a 9, but if you habitually dial off-hook, then you'll want to, or you'll stand a good chance of reaching an internal extension instead of the PSTN number you dialed. 

Rather than rehash what's been said before, have a read of one of my blog posts from waaay back in Lync 2010 days on this topic (don't worry, things haven't changed since then).  It shows how to set your Lync deployment up for extension dialing and external access prefixes the no-muss, no-fuss way. It also shows some examples on how the Lync softclient behaves when using an external access prefix.

You can use this in pure Lync Enterprise Voice deployments, or when integrated with an existing PBX.  In cases where you connect to a PBX, you still have to create a trunk translation rule to add the 9, if necessary. For more information on trunk translation rules, you can read my blog post on the subject, or refer to Technet. 

Tune in next week, where we talk about inter-trunk routing, and how people inadvertently set it up without meaning to.

Lync Enterprise Voice Misconceptions #1 - Failover Routing

During my travels across the Lync Enterprise Voice landscape, I come across numerous Lync deployments that make some fundamental mistakes in how to manage phone numbers, call routing and other details.  They're always fixed by the end of the engagement, but I'd like to share some of these to help keep others from making the same mistakes.  This will be an ongoing series of unknown duration.

#1 - Failover/Resiliency Routing

This is one of the more misunderstood concepts in Lync Enterprise Voice. Its a question that often trips people up on the Lync Voice exam (70-337), and is something that even respected websites like Windows IT Pro gets wrong (despite my attempts to get them to correct it).

The idea behind failover or resiliency routing is that you want to use a particular PSTN gateway for your normal day-to-day call routing, but have a backup PSTN trunk ready in case the primary one is down. This is different from load-balanced routing, where your goal is to evenly spread call traffic between two or more trunks.

For example, say you have a Lync deployment in your Boston office and another one in your Seattle office.  For reasons that should be obvious, you want to ensure that your Boston users always use the Boston PSTN gateway for outbound calls to reduce WAN traffic and call latency. If the Boston trunk is down, calls should failover to the Seattle PSTN trunk.  The same goes for your Seattle users. We could make things more interesting by bringing least-cost routing into play, but we're going to keep it simple here.

A common error people make in an attempt to make things simple and easier to manage is to create a single route for all calls, and place both the Boston and Seattle trunks inside that route.  Since they added the Boston trunk first, it appears first in the list, and they are lulled into thinking that calls will always use the Boston trunk unless its down, at which point the Seattle trunk is used.

In actual fact, calls will load-balance themselves between the Boston and Seattle trunks. Sure, if Boston is down, all calls will route via Seattle, but any well-designed EV deployment should never needlessly send calls across the country on the corporate WAN unless absolutely necessary.  If your other site is in a different country, you could be left with some pretty high phone bills since half of your national calls will be routed out the other country's gateway as an international call.

Voice Policy	PSTN Usage	Route	Trunks
Global	AllCalls	AllCalls	BostonGW SeattleGW

A clue to this behaviour can be found when you look at a route in the Control Panel.  You'll notice that while you can add/remove trunks to a route, there is no way to change the order of the trunks. This should be a clue to the fact that you have no control over which trunk is used within a route.

The proper way to achieve failover routing is via PSTN Usage ordering. Instead of creating a single route with multiple gateways inside it, you create a Boston PSTN Usage with a route to the Boston PSTN gateway, and another Seattle PSTN Usage with a route to the Seattle gateway.  In the voice policy assigned to your Boston users, add both the Boston and Seattle PSTN usages, ensuring the Boston one is above the Seattle one.  For your Seattle voice policy, put Seattle first.

Voice Policy	PSTN Usage	Route	Trunks
Boston	Boston-AllCalls	Boston-AllCalls	BostonGW
	Seattle-AllCalls	Seattle-AllCalls	SeattleGW

Now, you have achieved true failover routing.  For Boston users, all calls will route out the Boston trunk, unless it is down.  The same thing works for your Seattle users.

Incidentally, putting multiple trunks  in a single route does have its uses. If you have multiple PSTN gateways within the same datacentre for redundancy, its OK to do things this way, since you usually don't really care which trunk is used for outbound calls. Calls will be load-balanced between all the trunks, skipping any gateways marked as down. The above picture shows exactly that situation (although its not obvious since most of the trunk name is truncated). However this should never be considered failover routing, simply round-robin or load-balanced routing.

Voice Policy	PSTN Usage	Route	Trunks
Boston	Boston-AllCalls	Boston-AllCalls	BostonGW1 BostonGW2
	Seattle-AllCalls	Seattle-AllCalls	SeattleGW1 SeattleGW2

Essentially, you want to avoid placing trunks from different locations within the same route. This ensures you have control over how your calls are routed, keeping WAN utilization and costs low.

Tune in next week for our next exciting episode: "PBX External Access Prefixes"

Command Line Options in Lync Optimizer Scripts

A lot of what drives development in the Lync Optimizer is my own personal needs. Every once in a while, I come across a deployment that has some requirement that the Optimizer doesn't cater to.  Most often, it's related to a country that isn't in the Optimizer, which is easy enough to remedy in most cases, with a lot of research and testing. Other times, its a quirk of the deployment that forces me to change how the Optimizer script runs.

On my latest project, there are over 100 sites that have to have their own routes/PSTN usages for places around the globe. Normally, I generate all the dial rules, and run the resulting scripts, answering the questions as they come up (Site ID? Least cost routing? Location based routing? Would you like fries with that?).  Until this past week, I was happy to press 1, Yes, No, No, Supersize me etc.  Not anymore.

Many of the finest PowerShell scripts out there (I'm looking at you, Lync Scriptboy) allow you to enter parameters as required for execution.  I figured it was time to add that feature to the Lync Optimizer generated scripts.

So, if you don't want to answer questions as the Lync Optimizer script is running, you can enter answers to those questions as parameters in the command line prior to execution. All parameters are optional. If you don't add the parameter in the command line, the script will ask you during execution as it always has.

An interesting new feature I enabled along with the command line options is the ability to control which sites are included as part of the least-cost/failover routing logic. Previously, if you have lots of sites with Enterprise Voice routes/policies etc. and you choose to enable least-cost/failover routing in the script, your resulting voice policy PSTN usage list would be very long.  This option now allows you to limit which sites to apply least-cost/failover routing to.  Just use the option -LCRSites followed by the ruleset prefix of the sites to apply least-cost failover routing enclosed with quotes.  For example:  Scriptname.ps1 -LeastCostRouting:$TRUE -LCRSites "CA-ON-Toronto, UK-London, SG-Singapore"

If you need to run a bunch of Optimizer-generated scripts, this can be a real timesaver, especially when you have to run the same script multiple times to account for multiple gateways, or applying least-cost routing and that sort of thing.

The Optimizer script also now has the proper syntax for getting help, so you can type HELP and it will spit out all the available options.

The available options are:

SiteID
The numeric identifier associated with the Lync site to apply the script to. If a value is not provided, script will ask during execution. To see a list of sites and the associated site IDs, run Get-CSSite

DialPlanType
Create a site-level or user-level dialplan. Site-level applies to all users at a site, while user-level have to be explicitly applied. Possible values are 'Site' or 'User'. If a value is not provided, script will ask during execution.

LeastCostRouting
Either apply or don't apply least cost routing to the given site. Only applies if multiple Lync Optimizer generated rulesets are detected. If a value is not provided (and multiple rulesets are detected), script will ask during execution.

LCRSites
Apply least-cost/failover routing only to rulesets defined in this list. Separate site names with commas. Input site names exactly the same as the prefix for the desired site (ie CA-ON-Toronto, UK-London etc).

OverwriteSiteVoicePolicy
Overwrite any existing site-level voice policy if values are already there. Only applies if a site-level voice policy exists. If a value is not provided, script will ask during execution, if necessary.

LocationBasedRouting
Apply location-based routing to a selected site. Only applies if Lync network sites have been created. If a value is not provided, script will ask during execution, if it detects Lync network sites.

LBRNetworkSite
The name of the Lync network site to apply location-based routing. Only applies if Lync network sites exist. For a list of Lync network sites, run Get-CSNetworkSite. Works in conjunction with LocationBasedRouting parameter. If a value is not provided, script will ask during execution, if network sites exist, and the user selects location-based routing.

LocalOnly
Creates only local routes and PSTN usages for the selected site. Useful when using a central SIP trunk for all dialing but want to have local-only voice policies for remote sites without their own local PSTN access. Assumes that the users at the remote site are assigned numbers appropriate to their site, and the SIP provider enforces the appropriate local dialing area for those numbers.

MediationPool
The FQDN of a mediation pool to apply the script to. Only applies if multiple mediation pools exist within the selected Lync site. If a value is not provided and multiple mediation pools exist, script will ask during execution.

PSTNGateway
The FQDN of a PSTN gateway/trunk to apply the script to. Only applies if multiple PSTN gateways/trunks are assigned to the selected mediation pool. If a value is not provided and multiple PSTN gateways/trunks exist, script will ask during execution.

ApplicationPool
The FQDN of an application pool to apply the script to. Only applies if multiple application pools exist within the selected Lync site, and call park or premium call blocking is being used. If a value is not provided and multiple application pools exist, and either call park or premium call blocking is being used,  script will ask during execution.

Enter any desired options on the command line using the following syntax:

ScriptName.ps1 -SiteID digit -DialPlanType user/site -LeastCostRouting true/false -LCRSites "sitename1,sitename2,sitename3" -OverwriteSiteVoicePolicy true/false -LocationBasedRouting true/false -LBRNetworkSite NetworkSiteName -MediationPool MedPoolFQDN -PSTNGateway PSTNGatewayFQDN -ApplicationServer AppServerFQDN

Lync 2013 Edge server bug when adding new server pools

I had this happen to me a few weeks ago, and a co-worker had it happen to him today, so I figure it should be mentioned since nobody on the Internet seems to have a proper fix for it.

I installed a new Lync 2013 Standard Edition server in an existing Lync 2013 environment with an established edge server pool.  I created a test account on the new pool and began running through the usual testing scenarios.

All seemed well and fine until we tested federated communications.  An external party could establish an IM or A/V session with the internal user homed on the new pool and see their presence, but the internal user couldn't see presence nor establish a new IM or A/V session with anybody outside the company.

Puzzling to be sure, since everybody else in the company was OK.  I first checked to see if I could telnet to the typical ports required for edge functionality, and all was good. I then logged onto the edge to see if there were any relevant events.

Lo and behold, there was this interesting little warning:

LS Protocol Stack Event 14402
Multiple incoming connections on internal edge from non-internal servers.
In the past 289 minutes the server received 4 incoming connections on internal edge from non-internal servers. The last one was from host newlyncfe.contoso.com.
Cause: This can happen if an internal server is not present in the list of internal servers on the Access Edge Server.
Resolution:
If the server is a valid one, you need to add it to the list of internal servers on the Access Edge Server. If the server is invalid, you may be under an attack from that server.

This left me scratching my head.  How do you add a Lync 2013 server to the "list of internal servers"???  I recalled back to the olden times of OCS, where you had to manually add every valid server name to the edge, but you don't do that anymore, thanks to the wonder of the Topology Builder.

Internet searches brought up several cobweb-covered webpages from the late 2000's, all of them relating to OCS.  No good.  I re-ran Setup on the edge server, hoping that would trigger something and make things all good, but nope.

Finally, I decided to restart the Lync Server Access Edge service (during a period of low activity of course).  Once it restarted, all my troubles went away.  The user on the new front-end was able to initiate sessions to external users, and the 14402 errors on the edge went away.

My next step was going to be blindly restarting other services followed by a last-ditch server reboot. Since that turned out to be unnecessary, if this saves someone from doing a server restart (which would fix the problem too), that's awesome.

So, while Lync normally does an excellent job of detecting new servers and functioning fine without service restarts or server reboots, this is one situation where its not doing that.  It seems the Access Edge service caches the list of valid internal servers at service startup, and doesn't refresh that cache until the next service restart.

Because this seems to be one of the few instances where a service restart is required to maintain connectivity with a new downstream server, I would classify this behaviour as a...

Lync Address Book Weirdness

So, I'm at a fairly large client that's moved from Lync 2010 to 2013.  The server-side migration completed a while ago, and the client migration is in progress.  Enterprise Voice is enabled for a select group of people.

Some people have noticed that when they search for a specific Lync user, the only object that got returned was an object for their Active Directory administrative account that had only a phone number defined (not a Lync-enabled account).  The Lync-enabled "normal" Lync entry didn't show up in the search, even though it did appear in the GALContacts.db file (the Lync locally cached address book file).

The really weird thing was that the object showed up like a phone contact object, but one that had presence working.  The affected user could toggle their presence, and the contact object would dutifully change itself.  However, since it wasn't a Lync object, you couldn't start an IM (but you could call the associated number).

This user's first name isn't actually "Admin" in case you were wondering.

Another weird thing was that the very first time you pulled up that person's name from a search, it would briefly show the proper Lync-enabled account, but switch to the phone contact object view after a half-second or so.

Yet ANOTHER weird thing, was that after viewing the admin account object in Lync a few times, usually after viewing the contact card, the next time I did a search for the same name, only the phone number appeared.  However, I could still look at the contact object for the phone number and it would show the user's email address.

As mentioned previously, the address book had the correct contact information (verified by opening the GALContacts.db file in Notepad....the results ain't pretty, but it works), as well as the information from the AD admin account, but Lync was choosing to show the non-Lync enabled admin account in searches.

When I compared the affected admin accounts with others, I noted that some users showed up properly, but others didn't.  More comparisons showed that for the incorrect information to show up, the following had to be true:

1. Both the Lync-enabled account and the AD admin account had a phone number defined
2. Both the Lync-enabled account and the AD admin account had the same email address defined.

Removing the email address from the admin account, re-generating the Lync address book via Update-CSAddressBook, and re-downloading the updated address book file fixed the issue. 

So, what happened?  When you do a search, the Lync client will look at the address book, your own Outlook contacts, and any social connectors you may have associated with Outlook (like Facebook or LinkedIn).  If the same name shows up in multiple places, Lync will try to consolidate all the information and show only one contact object.

Sometimes, the process goes awry. In this case, it appears that Lync saw the same email address for both the Lync-enabled AD account, and the non-Lync enabled AD admin account, and incorrectly presented the non-Lync enabled account as the single object.

I'm guessing this is a bug, because Lync really should place priority on a Lync-enabled object over a non-Lync one for presentation.  

Lync Conference 2014 Content Online

If you weren't able to get to Las Vegas for Lync Conference 2014, you can now see all the content online at Microsoft's Channel 9 page.

I did two very well attended sessions on Enterprise Voice Best Practices.  You can see both of them online, if you're so inclined.  I recommend checking out the second one on Day 3 at 9am (BEST301-R).  For whatever reason, MS didn't have my most up-to-date slide deck for my first run-through, and there were technical difficulties which meant I almost wasn't able to demonstrate the Lync Optimizer.

And if you're going to be at TechEd 2014 in Houston on May 12-15, come see me perform the same session on Tuesday, May 13 at 10:15 AM (OFC-B339)

March/April 2014 Lync Optimizer Updates

I'm always working to provide new features for the Lync Optimizer.  Some have been rolled out silently to fix an issue.  Others are related to country specific dialing rules.  Here are a few things I've done recently that you may or may not have noticed.

NANPA Dial Rule Updates

Recently, I was looking to implement dialing rules for Namibia, when I was faced with an issue I hadn't considered.  The country code for Namibia is the same code I use for all North American dial rules (NA).

To fix the issue meant I had to do something I've been meaning to do for a very long time: separate out dialing rules for the US, Canada and the other 20-odd Caribbean countries that are part of the North American Numbering Plan Administration dialing area (NANPA).  All the countries in NANPA use +1 as the country code, which makes it difficult to work with programatically.  To switch things so that US dial rules have a US- prefix, Canada dial rules have a CA- prefix and so on, required quite a bit of work. 

The end result is that no matter the country you select, the dial rule prefix will use the actual country name, so instead of NA-ON-Toronto-Local or NA-TX-Dallas-National, you'll get CA-ON-Toronto-Local or US-TX-Dallas-National.  You'll still select North America as the country name.  The Optimizer will be smart enough to know if the area code is in Texas, Ontario or the Dominican Republic (for example).

This change is effective as of May 1, 2014.  Anybody who signed up for rule updates prior to this will be unaffected. Your rule updates will still come to you using the old format.  However, if you've started building dial rules for multiple North American locations prior to May 1st, and are adding more dial rules after May 1st, things may not work as expected if you have a mix of US-, CA-, and NA- prefixes. If this does affect you, please let me know and I'll put up an option to use the "legacy" dial rule format, or give you access to the "old" code.

New North America Dialing Options

North America has an interesting situation regarding international dialing.  There are 26 countries that share the same country code: +1.  In the US and Canada, long distance charges to Caribbean countries that are part of NANPA are often the same price as international calls. Since it is difficult to distinguish a Caribbean country from US/Canada based solely on the area code, it is easy for people to accidentally incur high phone charges when calling these countries.

I've added a new option to prevent this by allowing users to select one of the following options with regards to dialing other countries within NANPA:

1. In-Country Only - Treat all calls to NANPA countries other than your own as international, even though users don't have to dial 011 to reach them. As such, users will have to be a member of a voice policy that allows international dialing.
2. US/Canada - Treat calls to anywhere in US/Canada as national calls, excluding the Caribbean. To dial Caribbean countries, users will have to be a member of a voice policy that allows international dialing. Not available for Caribbean rulesets.
3. US/Canada/Caribbean - Treats calls to anywhere in NANPA (US/Canada/Caribbean) as national calls (along with the potentially higher call costs).

If you are creating a ruleset for a Caribbean country that uses +1 as the country code, you won't be able to select US/Canada, since this would make dialing within that Caribbean country difficult.

Selecting the Simple Ruleset option prevents usage of this feature, and will default to US/Canada/Caribbean. You won't be able to control how people dial other NANPA countries when there is only a single simple routing rule. 

New Extension Options

I had a request to create more flexibility around how extensions are used in the Optimizer. One addition is the ability to control how many digits of the extension are part of the associated DID. For example, a company might have an extension range of 2100-2199 that maps to a DID that matches up with the last 3-digits of the extension, like +12125558100 to +12125558199 (Note the extension starts with a 2, while the DID has an 8 in its place).

The Optimizer now supports this via the addition of a new column called "# of Ext Digits in DID". This will only appear when you select the DID option, and you can use the dropdown to select how many digits of the extension is part of the DID.

New Country Additions

I've added support for more countries, including Singapore and finally, Mexico.  Mexico has a very strange dial plan where you have to dial mobile numbers differently than land lines, but nothing about the number tells you in advance whether or not you should dial those additional digits.  If you use the Optimizer to generate dialing rules for Mexico, you should make sure your mobile numbers are stored in AD with 1 after the country code  (like +521331234567), and not with 044 or 045 as you would dial.  Lync will add the 044 or 045 as appropriate before it leaves Lync.

Conclusion

I hope you like the new additions to the Lync Optimizer.  If you require North American dial rules to follow the old format (NA-TX-Dallas-Local), please drop me a line and I'll help you out. 

Getting Creative with Lync Dial-in Conferencing Phone Numbers

I'm at a client who is moving their PSTN teleconferencing from one of your typical teleconferencing providers to an in-house Lync 2013 deployment.  In the old deployment, they configured an email message that got sent out every time one of the teleconferencing bridges had been booked. This email message contained a long list of both toll-free and locally dialable numbers for many countries around the world.  The client wanted something similar for the automatically generated Lync 2013 New Lync Meeting invitation from Outlook.

When you create a dial plan in the Lync Control Panel, you can set a value for Dial-in conferencing region.  This value is used to group dial-in conferencing phone numbers and present those to users assigned to that particular dial plan.

Dial-in conferencing phone numbers are assigned to the appropriate dial-in conferencing region as shown below.

When a user clicks New Online Meeting in Outlook, the values for dial-in conferencing numbers are automatically populated based on the numbers assigned to the dial-in conferencing region/dial plan associated with the user.  Notice that the region name shows up in brackets after the phone number.

The client wished to show all the available toll-free numbers by default, and order them so that the first one showing was specific to the region associated with the user booking the meeting.  When someone clicked on Find a local number, they would be shown all the local dialing numbers available.

The Lync Server environment consisted of two Enterprise Edition pools in two separate central sites: one in North America and one in South America.  We had configured two site-level dial plans: North America (English) and South America (Spanish).  The dial-in conferencing region names were the same as the central site name (North/South America).

To meet the requirements, we decided to change the dial-in conferencing region name to Toll Free for the North America site-level dial plan, and Llamada Gratuita (Spanish for Toll Free) for the South America site-level dial plan.  We also changed the default Global dial plan's dial-in conferencing region name to International.  You could use user-level dial plans just as effectively, if desired.

Then we created all the dial-in conferencing access numbers.  The Display Number field was set to include both the phone number and the location of that number.  I explicitly did not include the plus sign at the beginning of the display number, as this meant that we could apply normalization rules should someone click the number in the email, instead of the "Join Meeting" link (see my update towards the end of this post for more information).

All the toll-free numbers were assigned to both the Toll Free and Llamada Gratuita dial-in conferencing regions. All the local numbers were added to the International dial-in conferencing region.  To make sure the order was correct, we used the Set-CSDialInConferencingAccessNumber command to set the order appropriately:

Set-CSDialInConferencingAccessNumber -Identity AccessNumSIPAddress -Priority x -ReorderedRegion RegionName

Once complete, users would see only the toll-free numbers (ordered correctly for their region) when booking a meeting.  In the below example, a North American user is creating the meeting, so the North America toll-free number is shown first.  If a South American user created a meeting, the Chile number would appear first. Note that for North American users, the Toll-Free in brackets represents the region name, but users will be none the wiser.  South American users would see (Llamada Gratuita) instead.

Clicking the Find a local number shows all the local numbers available.

The client was thrilled with the results. The only caveat with this method is that the links for clicking the phone numbers in the invite won't work because of the text added.  However, the reasoning is that nobody will ever click those links because they will normally just click Join Lync Meeting.  It would be nice if Lync would use the Line URI in place of the display number for the link, which would avoid the problem entirely.

UPDATE (2015-Oct-07) A BETTER (BUT NOT THE BEST) WAY TO ENSURE "CLICKABILITY":  Greig Sheridan emailed me to see if I ever came up with a solution to this, and after much messing around, I came up with a rather simple solution: remove the plus sign from the Display Number (not the actual Line URI). When the plus sign is in place and you click the link, the plus sign gets copied over along with the actual number and the numeric representation of every letter after the number.

This is because the Lync/Skype for Business client lets you type in something like 1 800 GOT JUNK, and it will helpfully translate it to +1 (800) 468-5865.  Unfortunately, it will also do the same thing for the "- North America" (or whatever text) you added to the end of the display number. When you click the link for +1 800 222 3333 - North America, it gets translated to +18002223333667842637422.  Because the plus sign is already there, Lync won't normalize the number and will try to dial that mess.

When you remove the plus from the display number, normalization rules can be applied. If you've followed my advice in my blog post on overdialing (again thanks to Greig for that idea....yet another Foster's lager inbound), then any extraneous numbers will get stripped and the number will dial happily.  Of course, this relies on every company configuring their normalization rules this way, but its better than nothing.

UPDATE (2015-Oct-08) THE BEST WAY TO ENSURE "CLICKABILITY": Another fantastic option, this time from Ari Protheroe.  Instead of relying on normalization rules that other companies may not have configured "correctly", Ari found this way of getting around the issue.

Start the Display Number with the text you want, followed by a space, then a colon (:), another space, then the number in E.164 format.  Something like this:

The space before and after the colon are critical to this.  This seems to be enough to trick Outlook into sending only the number to Lync/SfB.  I tested this myself and it worked amazingly.  So, ignore all that I put before, and use Ari's method.  Thanks Ari! Now I owe YOU a beer. I'll stick with Foster's for consistency's sake.

In any case, this is definitely a creative way to get around some of the limitations of dial-in conferencing number display.  I hope you find it useful.

Meet URL Gives 404 error

I was recently at a company that did a big switchover from Lync 2010 to 2013.  The new environment consisted of 3 Enterprise Edition front-end servers with an F5 load balancer taking care of web services load balancing.

On the first business day of full Lync 2013 operations, some people were complaining they could not join meetings.  When some users clicked on the Join Lync Meeting link in the email, they were greeted with a 404 - File or directory not found error in IE.  If they tried several times, eventually they got in.  

Further analysis showed that one specific front-end server was serving up the 404 errors, while the others were working fine.  Thanks to the F5, we were able to easily remove that server from the load-balanced pool while we troubleshooted (troubleshot?) the problem. 

The first thing to note about troubleshooting issues with the meet URL on an Enterprise Edition pool is that you can't connect directly to a specific server in a pool and expect to get the proper meeting join experience.  For example, if your meet URL is meet.contoso.com and your Lync pool members are FE01.contoso.com, FE02.contoso.com etc, you would normally connect to a meeting via something like https://meet.contoso.com/user.name/FY3DFSE4.  You can't troubleshoot issues with a specific server by connecting to https://FE01.contoso.com/user.name/FY3DFSE4.  You'll get a 404 error, thanks to the way the URL Rewrite module processes URLs. 

To get around this, you need to add a temporary HOSTS entry to your testing workstation for meet.contoso.com pointing to the server having the issue. This will bypass the load balancer and allow you to connect directly to the server you want to test.

Back to the problem....

After setting my HOSTS file to point directly to the "bad" server, I found I could browse to https://meet.contoso.com successfully, but not to a specific meeting like https://meet.contoso.com/user.name/FY3DFSE4, which threw a 404.  Event logs didn't show anything wrong. 

I'll spare you the hours of dead ends I tried and just give you the solution (because that's why you're here, right?)

I went to Control Panel and removed the IIS URL Rewrite Module 2.  Then I re-ran Lync setup via the Lync 2013 Deployment Wizard, which re-added the URL Rewrite Module and reset the default URL rewrite rules Lync put in place.  All this was done without reboots or service interruption.  As soon as Lync setup completed, meeting joins happened without error. 

So, it appears that something went wrong with one or more of the URL rewrite rules, which wasn't cleaned up by simply re-running Lync setup, or Enable-CSComputer, which was things other people suggested in various places I looked. 

I hope this helps others who are having this issue.

Lync Conference 2014 Recap

Just got back from another amazing Lync Conference, this time at Aria in Las Vegas. It was great to see all my Lync buddies from around the world and to have the opportunity to participate in some very informative sessions given by Microsoft employees and many of my fellow Lync MVP friends.

There were several announcements, most of which I'm sure everyone has already heard about.

• The next version of Lync is currently known as Lync vNext. Not sure if this is a codeword, or the final name
• LyncvNext will include a new server role which will allow other video-conferencing systems (like Tandberg/Cisco) to join Lync-hosted video conferences. This server role can be either co-located on front-end or separate. 
• Feature-parity on all mobile platforms, including Android tablets, which have not seen a Lync release as of yet.
• Video calling between Lync and Skype. We all knew it was coming, but nice to see it finally show up. I think they're targetting go-live in June. 
• A set of Javascript libraries called jLync, which will allow for all kinds of web development possibilities
• The introduction of hosted-PSTN connectivity on Office365. No details on where it will be offered, but the US is a good bet.

Myself, I hosted two very popular sessions on Lync 2013 Enterprise Voice Best Practices to packed rooms. I had a lot of fun doing it, and look forward to doing more. The famous Jamie Stark even mentioned it on several occasions:

@darylhunter @unplugthepbx @kenlasko Ken's #lyncconf14 session will be PACKED - arrive early! Its also being repeated Thurs 9am
— Jamie Stark (@nomorephones) February 18, 2014

I am impressed #lyncconf14 - @kenlasko repeat all full pic.twitter.com/IIUwxAOS38
— Jamie Stark (@nomorephones) February 20, 2014

Feedback was very positive, including one fellow who threw a pair of underwear at me at the end of my second session as a joke.

Someone just launched underwear at @kenlasko ?!!? Hope they were clean #LyncConf14
— Mark King (@unplugthepbx) February 20, 2014

@greiginsydney Man I knew that @kenlasko guy had fanboys but this is ridiculous! @msftLync #Lyncconf14
— John A Cook (@JohnACook) February 20, 2014

@unplugthepbx @kenlasko ROFL - another #lyncconf14 first!
— Jamie Stark (@nomorephones) February 20, 2014

I had a great time at this year's Lync conference. The venue was beautiful, the sessions were informative, and the after-hours parties were fun. I'm already looking forward to LyncConf15, hosted in Hawaii (I hope!).

February 2014 Lync Optimizer Updates

Since moving the Optimizer back-end to a SQL database and enabling Microsoft authentication, its allowed me to explore adding new features not previously possible.

Ruleset History

The first feature I'm releasing is a ruleset history option. Every ruleset run by users is stored in SQL, so it was relatively easy to enable a history feature, so users can call up past rulesets (it was harder to make it secure). This can come in especially handy when working with extensions, as it is currently rather time consuming to enter extension details.

When you log in now, you'll see a View History button beside the Input heading.

Clicking it will bring up your entire ruleset history since the introduction of authentication and the SQL backend (late October 2013).  Clicking on any row will load that particular ruleset into the Optimizer.

Selective Caller-ID Block

Sometimes, users want the ability to block their outgoing caller ID, but not all the time.  Lync has a feature where you can do this at the voice route level, but it takes a bit of work to make this work "on-demand".  The Optimizer can now accomplish this for you, simply by checking the Allow Call ID Block checkbox and entering the desired caller ID block code (ie *67 in the US/Canada), and the replacement caller ID to use.

The Optimizer will change the default normalization rules to allow the entry of the caller ID block code.  Normalized numbers using this code will look something like: *67+12123334444.  The Optimizer then creates a route for that pattern that selects the Suppress caller ID option.  A trunk translation rule strips the block code before sending to the next hop.

Other Things

I've also included numerous tweaks to improve the overall experience and to ensure consistency. I've also nearly completed the data move from XML to SQL which gives me more options for the future.

I've been working behind the scenes trying to make these work as seamlessly and easy as possible. If you find the Optimizer helpful and a timesaver, think of the Hoff and send him a donation (this guy here, not the real Hoff....he doesn't need any more money).

Any new feature requests, please drop me a line.

Presenting at LyncConf 2014!

I'm excited to say that I'll be presenting two sessions on Lync Enterprise Voice Best Practices at LyncConf14 in Las Vegas.

These sessions will focus on the WHYs behind the HOWs of Enterprise Voice in Lync 2013.  I'll talk about the best ways to manage your dialplans, voice policies, routes and trunk translation rules to provide a consistent, functional and manageable Lync EV deployment.

So, if you're at LyncConf, come check out my session. Otherwise, the only people who will be there will be my fellow Lync "friends" who just want to see me make a fool of myself, and are threatening to bring various fruits and vegetables to fling at me.

Session Details:

Setting up Enterprise Voice is a big project even for seasoned Lync experts. The interplay between dial plans, voice policies, routes, PSTN usages and trunk translation rules can make it complicated to figure out how to start. Come and join Lync MVP Ken Lasko, the creator of the Lync Dialing Rule Optimizer, and learn the WHYs behind the HOWs of configuring Lync Enterprise Voice – including E.164 numbering, extension dialing and least-cost routing - to provide the most flexibility and easiest migration path from legacy PBXs. And of course, this session wouldn’t be complete without a demonstration of the Lync Dialing Rule Optimizer and how it puts all these best practices into play.

Session Date/Time:

Tuesday, February 18 2014 2:00 PM - 3:15 PM

Room: Copperleaf 1

Thursday, February 20 2014 9:00 AM - 10:15 AM

Room: Copperleaf 1

If there's something you'd like to see me talk about in the session, leave a comment below.

Presentation Issues After Moving Lync 2013 Fileshare

Recently, we moved the Lync 2013 file share for an enterprise pool to a new location that was more resilient than the original. Everything seemed to go well, except that file sharing from internal to external users stopped working. Also, Lync 2010 users were suddenly unable to share Powerpoint presentations.  Lync 2013 users seemed fine.

When attempting to view or share a Powerpoint presentation on Lync 2010, the users got the following message:

This slide couldn’t be downloaded. Please contact your support team. Error reason: File not found.

The fact that 2013 clients were unaffected by the Powerpoint presentation is not unusual.  Lync 2013 offloads Powerpoint content rendering to the Office Web App server.  Lync 2010 clients are unable to utilize this feature, and fall back to the original method, which is managed and rendered from within the Lync server itself.

Of course, my first step in troubleshooting is to plug the error message into my favourite search engine. It came up with several different options, mostly dealing with issues with Office Web Apps, which we weren't experiencing. One blog post in particular caught my attention, because it talked about the same issue happening after a file share move:  http://paulbrown.us/blog/2011/11/02/how-to-change-lync-server-file-store-location/.  Incidentally, it appears that the writer (Paul Brown) enjoys sitting on active train tracks in his spare time, which may account for his lack of activity since May 2013.

So, Paul Brown (RIP) managed to solve the issue for Lync 2010, which definitely helped me solve it for 2013.  The key difference between 2010 and 2013 in this case is that the MeetingContent and MeetingFiles virtual directories in Lync 2010 don't exist in 2013.  All that seems to have been rolled into the CollabContent virtual directory in Lync 2013.

When I looked at the advanced settings for the CollabContent virtual directory, the Physical Path still pointed to the old file share location.  I updated the location using the new path in both the Lync Server External Web Site and Lync Server Internal Web Site and repeated this on each server in the affected front-end pool, followed by an IISReset.  This was enough for both file sharing and presentation sharing to function properly for Lync 2010 and 2013 clients.

This is obviously a bug within the Topology Builder/Lync Deployment Wizard, because this should have been changed automatically by running the Deployment Wizard after the topology change to move the file share.

As a final note, if you're planning on sitting on train tracks, make sure to keep an eye out for any oncoming trains, because, well you know, YOU'RE SITTING ON TRAIN TRACKS!

High Processor Utilization on Lync 2013 Front-End Servers

UPDATE (2015-Oct-02): The problem has finally been fixed!  Took nearly 2 years, but here's a link to the KB article. The fix is in the September 2015 CU for Lync Server 2013.  The cause is "because the topology snapshot was recomputed multiple times by the Response Group Service."  Thanks to @sublimeashish for telling me. 

We have a customer who is about to migrate from Lync 2010 to Lync 2013.  They've got a few lightly loaded Lync 2013 Enterprise Edition pools with 3 servers each.  All are running Windows 2008 R2 Standard Edition on VMWare.  All patches are up-to-date.

For inexplicable reasons, some of the servers will suddenly see their processor utilization spike to near 100% for extended periods of time, when their typical utilization is less than 5%. A look at Task Manager shows two instances of the W3WP.exe service (IIS web service) that are consuming large amounts of processor resources.  There are no events in the Event Logs to indicate an issue.

Performing an IISReset on the affected node makes the processor go back to normal, but this is obviously not a real solution.  We opened a ticket with Microsoft PSS, and they confirmed there are others seeing the same thing.  It seems the source of the problem is the "garbage collection" process in the LyncIntFeature and LyncExtFeature application pools in IIS.  Recycling those pools makes processor utilization return to normal (for a while at least).

Microsoft is actively working to resolve the issue, and I will post a permanent solution for all to see as soon as one becomes available.

UPDATE:  Thanks to @dannydpa  on Twitter, it appears the trigger may be Lync topology publishing. I confirmed this by updating the topology and publishing it.  Less than 10 minutes later, all the servers processor utilization spiked.  Recycling the aforementioned apppools resolved the issue.

To help others with this issue, I've created a little Powershell script that will recycle the LyncIntFeature and LyncExtFeature app pools for all Lync servers.  For the script to work, you need to make sure that remote management is enabled on all Lync servers.  On Windows Server 2012, this is on by default, but in Windows 2008 R2, you need to log on locally and run: Enable-PSRemoting -Force before running the script.

$WebPools = (Get-CSService -WebServer).PoolFQDN

ForEach ($Pool in $WebPools)
{
  $PoolMembers = (Get-CSPool $Pool).Computers
  Foreach ($Computer in $PoolMembers)
  {
    Write-Host "Resetting LyncExtFeature and LyncIntFeature app pools on $Computer"
    $Session = New-PSSession -ComputerName $Computer
    Invoke-Command -session $Session -ScriptBlock {Restart-WebAppPool LyncExtFeature}
    Invoke-Command -session $Session -ScriptBlock {Restart-WebAppPool LyncIntFeature}
    Remove-PSSession $Session
  }
}