Adventures in Normalization

One of the things that differentiate Lync from its competitors is its reliance on 3rd party peripherals like deskphones. This is often touted as a Good Thing, since you're not locked into a single vendor for phones. Phones from single-vendor solutions are often more expensive than an equivalent Lync phone simply because you can't get them from anywhere else, so competition is low and prices are higher.

Since Lync phones are provided by different vendors, there is naturally a lot of variation in the specific details on how things work. Microsoft has alleviated this somewhat by instituting a Lync-qualified devices program.  This is a fairly thorough program that puts devices through their paces to ensure they function properly within Lync. 

However rigorous the program, little details slip through the cracks. One of the details that I've noticed is how different phones (and even the Lync client) handle regular expressions in different ways. For anyone familiar with the minutiae (thanks Word of the Day Calendar) of Lync Enterprise Voice, regular expressions are used for phone number normalization and routing. 

All Lync-qualified devices handle basic regex in the same manner. For most people, this is not an issue, but sometimes you need to use lesser-known regex patterns to accomplish certain goals. When you try to get fancy with regex, things sometimes break down with disparate results between clients and phones. 

Over the years, I've heard about and personally experienced situations where Lync Optimizer-generated regex doesn't work on specific phones. The affected vendors have eventually fixed these issues, but I figured that I should do some hardcore 60 Minutes style investigation into just how well different devices handle some of the more esoteric regex patterns out there. 

To do this, I used my wide variety of Lync deskphones I have available for testing in my home lab.  (You should hear what it sounds like when someone phones me.  Everything is ringing and flashing, and I usually go "AAAAhhhhh" and run out of the room.)  I used a Polycom VVX 600, a Polycom CX600, a Snom 760, and an old LG-Nortel "Tanjay" series phone.  My AudioCodes 440HD wasn't connecting to our Lync server at the time, so it was left out of testing. Every phone was updated with the latest firmware available at testing time. 

For the first round of the Lync Normalization Olympics, I decided to keep it simple, using some of the rule formats commonly used by the Lync Optimizer.  I input the normalization rules into Lync via the Control Panel, and I used the "Create voice routing test case information" to see how Lync Control Panel would interpret the regex, and included that in the results too. For all the rulesets, I used a unique "identifier" for each rule to allow me to setup a series that I could test all at once.  

Commas

First off, I used a regex rule that included a comma, which I had heard caused normalization issues with one particular vendor. In the below example, I'm looking for a pattern of numbers that starts with 999 followed by 10 or 11 digits. The 999 gets stripped and puts a + in its place.

Pattern: ^999(\d{10,11})$

Translation: +$1
Input number: 99915552223333 then 9995552223333
Expected output: +15552223333 then +5552223333

Control Panel	Lync Client	Tanjay Phone	Polycom CX	Polycom VVX	Snom
+15552223333	+15552223333	+15552223333	+15552223333	+15552223333	+15552223333
+5552223333	+5552223333	+5552223333	+5552223333	+5552223333	+5552223333

Everything passed that one with flying colours, so rumours of a phone type not handling commas seem to no longer apply.

Dealing with Extensions, Part I

Another common rule used in the Optimizer is a rule to accept any dialed number, but strip out any extensions and leave it as an externally routable E.164 number. This rule was meant mostly for click-to-dial from the Lync client, but I did hear about a particular vendor's phone not being able to parse that particular rule. In the next example, I'm looking for a pattern of numbers that starts with 998 followed by 11 digits, followed by some regex that would strip anything else beyond the 11 digits. The 998 gets stripped and puts a + in its place.

Pattern: ^998(\d{11})\d*(\D+\d+)?$

Translation: +$1
Input number: 99815552223333
Expected output: +15552223333

Control Panel	Lync Client	Tanjay Phone	Polycom CX	Polycom VVX	Snom
+15552223333	+15552223333	+15552223333	+15552223333	+15552223333	+15552223333

Everything passed that one with flying colours, so rumours of a phone type not parsing those rules seem to no longer apply.

Dealing with Extensions, Part II

The next rule test is a variation on the first example above, using a slightly different regex pattern to strip any extension. This format was used a long time ago in the Lync Optimizer, but was replaced with the "cleaner" looking version above.  One person told me he used this format to get around an issue with a particular phone vendor's issue parsing the previous example's format.

Pattern: ^997(\d{11})(\s*\S*)*?$

Translation: +$1
Input number: 99715552223333
Expected output: +15552223333

Control Panel	Lync Client	Tanjay Phone	Polycom CX	Polycom VVX	Snom
+15552223333	+15552223333	+15552223333	+15552223333	+15552223333	+15552223333

Again, no issues on any platform.

Extensions with Optional Site Code

In companies with multiple sites that provide its users with extension dialing, its often necessary to use site codes to properly differentiate one site from another. This is especially true if there are overlapping extension ranges originating from legacy PBXs. So, for a company using 4-digit extensions, they may prepend a site code to ensure uniqueness. So, Site #101 could use 1015xxx for their extension range, and Site #102 could use 1025xxx for their extension range.

One particular company wanted to ensure that users within any given site could dial their own users by extension without the prepended site code, but still allow them to enter the site code if they wanted to. I did this in one rule by using a bit of fancy regex. In the below example, a user should be able to enter the extension with or without the 996 "site code".

Pattern: ^(?:996)?(5\d{3})$$

Translation: +1555222$1
Input number: 9965000 then 5000
Expected output: +15552225000

Control Panel	Lync Client	Tanjay Phone	Polycom CX	Polycom VVX	Snom
+15552225000	+15552225000	+15552225000	+15552225000	+15552225000	+15552225000
+15552225000	+15552225000	+15552225000	+15552225000	+15552225000	+15552225000

Sigh, once again, no issues on any platform.

OK, boooooring. All the phones passed the basic regex tests without issue. Not at all "exciting". I use "exciting" in quotes, because how exciting can the topic of normalization be?

Multiple Replacement Strings

Sometimes you need to use multiple replacement strings in a rule (ie $1, $2 etc). Rumour has it some phones have issues with this.

Pattern: ^996(\d{3})000(\d{7})$

Translation: +1$1$2
Input number: 9965550002223333
Expected output: +15552223333

Control Panel	Lync Client	Tanjay Phone	Polycom CX	Polycom VVX	Snom
+15552223333	+15552223333	+15552223333	+15552223333	+15552223333	+15552223333

Running out of ways to say "No issues".

Advanced Normalization Tests 

To really gauge how well a phones' regex engine functions, we have to throw at it some regex that would only be used in very specific circumstances. 

For example, let's say I have a need to translate a set of digits into a pattern where those digits are buried within other numbers.  For example, I want to take 92xx and translate it to +155592xx999.  So, 9201 translates to +15559201999.  If you try to create a pattern like ^(92\d{2})$ and translate it to +1555$1999, there is ambiguity on whether you mean +1555 $1 999 or +1555 $19 99. 

Normally, I would follow the .NET regex reference which has different ways of dealing with this, but none of them work in the Lync client.  I did stumble upon one that did work, even though it isn’t a valid .NET expression.  Not only that, but testing with other phones gave different results most of the time.

I tried various patterns, and plugged in 9100, 9200, 9300 etc to see the results in both Control Panel and Lync softclient and desk phones. The results proved interesting.

Pattern: ^(91\d{2})$

Translation: +1555$1999
Input number: 9100
Expected output: Expect failure, but want to see how various devices handle this situation.

Control Panel	Lync Client	Tanjay Phone	Polycom CX	Polycom VVX	Snom
+1555$1999	+155599	+15559100999	+15559100999	No match	+1555

Seems as though the old Tanjay and CX-series phone work in this situation, but nothing else. No big deal, because I wouldn't expect any consistent behaviour here.

Pattern: ^(92\d{2})$

Translation: +1555${1}999
Input number: 9200
Expected output: +15559200999

Control Panel	Lync Client	Tanjay Phone	Polycom CX	Polycom VVX	Snom
+15559200999	No match	+15559200999	+15559200999	+15559200999	No match

Using ${1} is a valid way to prevent any ambiguity between $1 or $19 in this example. Sadly only the Lync client and the Snom don't respect this valid regex.

Pattern: ^(93\d{2})$

Translation: +1555$_999
Input number: 9300
Expected output: +15559300999

Control Panel	Lync Client	Tanjay Phone	Polycom CX	Polycom VVX	Snom
+15559300999	No match	No match	No match	+15559300999	No match

Using $_ is a way to substitute the entire string. Not ideal code-wise, but wanted to see if it would work.  Only the Control Panel and Polycom VVX phones properly parsed it.

Pattern: ^(?<1>94\d{2})$

Translation: +1555${1}999
Input number: 9400
Expected output: +15559400999

Control Panel	Lync Client	Tanjay Phone	Polycom CX	Polycom VVX	Snom
+15559400999	No match	No match	No match	+15559400999	No match

This is a valid .NET regex method to use named replacement strings instead of numeric. So, I could have easily used ^(?<DigItDude>94\d{2})$ -->  +1555${DigItDude}999.  Again, only the Control Panel and Polycom VVX phones properly parsed it.

Pattern: ^(95\d{2})$

Translation: +1555($1)999
Input number: 9500
Expected output: +15559500999

Control Panel	Lync Client	Tanjay Phone	Polycom CX	Polycom VVX	Snom
+1555(9500)999	+15559500999	+15559500999	+15559500999	+15559500999	+1555(9500)999

This is actually not a valid .NET regex method. Surprisingly, all the devices except for the Snom managed to parse it correctly.

Conclusions

As long as your phone number normalization requirements don't require any digit replacement buried within other digits, you can safely use just about any Lync-qualified device.  However, if you know in advance that you have such a requirement (and its very unlikely that you would), I would avoid Snom phones at this time, simply because they couldn't handle any of the advanced regex I threw at them in my attempts to solve that specific issue.

The way I see it, if I build a regex pattern in Lync Control Panel, I should be able to trust that every single device will parse that regex pattern the same way.  I find it rather annoying that there isn't any rigorous checking to see if vendors adhere to .NET regex standards. On that front, I feel Microsoft needs to be more diligent with their phone qualification process.  Its these little details that can lead to frustration for anybody deploying Lync Enterprise Voice.

If you got to the end of this post and are still not asleep.....congratulations!  This post has been entered in the "Most Boring Tech Post of the Year" for 2015. Yes, I know its early, but I think this is an early front-runner for the win.

The Importance of Lync Location Policies

Lync Location Policies are used to specify how to deal with emergency calls originating from Lync clients.  Quite a bit has been written about location policies and how they are used in E911 emergency dialing scenarios (for a few examples, see Technet or Mark King's UnplugthePBX blog).

Unfortunately, E911 only exists in the US, so anybody reading the documentation outside the US might conclude that location policies aren't required, and that emergency calls can be routed as other normal calls. While this is technically true, location policies still provide some critically important functions in specific scenarios that may not be apparent until you have to deal with them firsthand.

The Problem

Imagine the following two common scenarios:

1. Your company has a single site spread across several floors of a building and everyone is assigned an extension, instead of a full directly-dialable phone number (DID). Your company has a single main office external phone number which rings to reception.
2. Your company uses a centralized SIP trunk for normal calls, but all remote sites are configured with small PSTN gateways used to provide Lync-based local emergency dialing via a few analog lines.  Each user have their own dedicated directly-dialable phone number.  Users are assigned to a voice policy that ensures that emergency calls will route out the local PSTN gateway. 

For scenario #1, if someone within the office calls the emergency service applicable for their country (lets use 112, a common emergency number in Europe and many parts of the world), the number that will be presented to emergency services will be your main office number. 

For scenario #2, if someone from a remote office calls 112, the phone number that will be presented to emergency services will be that of the analog line assigned to the PSTN gateway. 

For both scenarios, you might be thinking "OK, so emergency calls go out the proper gateway, and emergency services will know where the office is. What's your point?"  You're correct in that emergency services will know the office location and will be able to dispatch personnel as required, but now think about these two factors:

1. The user who called 112 accidentally hangs up because that zombie finally caught up to him.  What happens next (other than you now have two zombies)?  We'll call it the "Hang-up of Death".
2. What happens when someone travels from their usual office to another and has to call emergency services because of....yeah, zombies?  It's the "Travelling Salesman of Doom".

The Hang-Up of Death

For #1, the typical response from emergency services is to re-dial the number that will show up on their console. In the single office/extension scenario, they will reach reception who may have no idea about who called 112, which can waste valuable time, depending on how fast the zombie horde grows.

In the remote site scenario with a dedicated analog phone line, emergency services would redial the analog number, and the call would ring endlessly, since its not likely that anybody considered someone calling inbound to an emergency number.  Or if someone was bright enough to forward incoming calls to reception, they still might not have any idea on who called 112 in the first place, again wasting valuable zombie killing time.

Travelling Salesman of Doom

For the unfortunate travelling user in #2, remember that unless you've implemented location-based routing, the user's calling behaviour won't change based on their current location.  If they call 112, then the call will route out the user's normal PSTN gateway which means emergency services will show up in their normal office, while our poor guy gets chased down by zombies.  If location-based routing has been implemented, then we still have to deal with the fact that reception at the local site still has no idea where the zombie attack is taking place.

The Solution

As you hopefully have discerned, location policies can help with both of these situations very well.

Firstly, location polices can be configured to notify one or more users by IM that someone's called emergency services. This way, when emergency services shows up, or if they have to call back, reception or security or whoever will have a bit more information about the situation than they would otherwise.

Secondly, location policies can be assigned to Lync network sites, which means that emergency calls will go out the proper gateway no matter where the user is typically situated.  This works very similarly to location-based routing, except that it's only for emergency calls.

Takeaways

If you are outside the US, you should still configure location policies for every site that has a PSTN gateway.  Consider doing the following:

• Configure Lync network sites and subnets for everywhere that Lync users could go.
• Configure location policies and assign them to each Lync network site, and set it to use the local PSTN gateway for emergency calls. Check the box marked "Enable Enhanced 9-1-1 (E9-1-1)" even if you don't use E911. This seems to be a prerequisite for emergency dialing via Location Policies to work at all, despite what Technet says.

• Assign the appropriate default location policy to every user, to act as a fallback in case they are outside a Lync-defined subnet. 
• Configure the notification URI field* in the location policy so that the appropriate people are aware whenever the configured emergency number is dialed.
• If using analog gateways for emergency dialing only, make sure that inbound calls route to the appropriate person, should emergency services need to call back (ideally the same person who's been assigned to the notification URI field).
• Consider populating the location information services (LIS) database even if outside the US, because this can help reception figure out where the person who called emergency services is located.

* A special note about the the Conferencing URI fields: You probably noticed these fields, which implies you can conference someone in when they dial the emergency number. Unfortunately, this doesn't work unless you're actually using an E911 provider in the US. If you're outside the US, you can enter values here to your heart's content, but it won't do anything. Boo-urns! Thanks to C. Anthony Caragol (check out his LyncFix blog) for bringing this annoying little tidbit to my attention.

For detailed instructions on how to configure Lync network sites and location policies for non-US locations, I strongly recommend Elan Shudnow's blog post on the subject.

Lync Enterprise Voice Misconceptions #3 - Inter-trunk Routing

During my travels across the Lync Enterprise Voice landscape, I come across numerous Lync deployments that make some fundamental mistakes in how to manage phone numbers, call routing and other details.  I've covered a lot of these details in parts of my OTHER Enterprise Voice Best Practices, but its obviously a lot to take in, and I've seen enough of the same sort of errors in the wild that prompted me to do this series.

Last week's episode talked about how to properly use the external access prefix in Lync. This week, we're talking about...

#3 - Inter-Trunk Routing

If you've ever manually created trunk translation rules for a PSTN gateway in Lync, you've probably noticed the "Associated PSTN Usages" section.

The wording is vague, and even Technet's description of the setting doesn't tell you much.

From what I've seen in multiple deployments, people tend to add the PSTN Usages that were created specifically for the particular trunk, which sort of makes sense if you use Technet's description as your only guide.

Some other places on the Internet take this idea even further, and imply that this section is used to define the PSTN usages that are allowed to use the specified trunk. If you think about it, you already accomplish this by defining the PSTN Usage/route combination without having to do anything else, so that can't be right either.

Now, you might be aware that Microsoft mentions in their Technet planning documentation that Lync can do "inter-trunk routing".  From their own documentation:

This new capability enables Lync Server to provide call control functionalities to downstream telephony systems. Intertrunk routing can interconnect an IP-PBX to a public switched telephone network (PSTN) gateway so that calls from a private branch exchange (PBX) phone can be routed to the PSTN, and incoming PSTN calls can be routed to a PBX phone. Similarly, Lync Server can interconnect two or more IP-PBX systems so that calls can be placed and received between PBX phones from the different IP-PBX systems.

In more general terms, Lync can pass inbound phone calls on to a different destination, which could be another PBX system or even back out to the PSTN.  So, now you're thinking "That's a great feature, but what does this have to do with Associated PSTN Usages?"  Since the title of this article says "Inter-Trunk Routing", it should dawn on you that "Associated PSTN Usages" has something to do with inter-trunk routing.  If it doesn't, then lie down a while and come back after a good nap.

As you hopefully have surmised by now, inter-trunk routing is done using "Associated PSTN Usages" from within a trunk.  The details behind how to do this are helpfully discussed in some of my own blog posts, Technet and also those by fellow Lync MVP Richard Brynteson:
http://ucken.blogspot.com/2013/06/inter-trunk-routing-in-lync-2013.html
http://masteringlync.com/2013/06/07/inter-trunk-routing-deep-dive/
http://technet.microsoft.com/en-us/library/gg425831.aspx

To summarize those blog posts, if an incoming call from a trunk can't be routed to an internal user, it will attempt to route the call to an alternate trunk, if there is a matching PSTN Usage/route combination assigned to the incoming trunk.

When people add the PSTN usages to a trunk that are already assigned to that very same trunk, incoming calls could potentially be sent back out the same way they came in, if there isn't a matching user in Lync. This could result in odd behaviour, including calls bouncing back and forth between a PBX and Lync, in what I call the "Telecom Routing Donut of Doom".  Fortunately, in most cases, adding PSTN usages to the trunk has no effect, since there is almost always a Lync user assigned to the incoming phone numbers.

I've seen numerous deployments where they've added PSTN usages willy-nilly (my Word-of-the-Day calendar entry for today).  If I see those, my next question is "Oh, so you're using inter-trunk routing."  If the reply I get back is some variation of  "What's inter-trunk routing?", then I remove those PSTN usages from the trunk configuration.

There you have it.  Tune in next time (not necessarily next week, since I'm bad at reading calendars) where we talk about something else that I haven't quite figured out yet.

Lync Enterprise Voice Misconceptions #2 - External Access Prefixes

During my travels across the Lync Enterprise Voice landscape, I come across numerous Lync deployments that make some fundamental mistakes in how to manage phone numbers, call routing and other details.  I've covered a lot of these details in parts of my OTHER Enterprise Voice Best Practices, but its obviously a lot to take in, and I've seen enough of the same sort of errors in the wild that prompted me to do this series.

Last week's episode talked about common errors when setting up resiliency/failover routing. This week, we're talking about...

#2 - External Access Prefixes

A lasting legacy of the PBX era is the concept of dialing an external access prefix to make a call to the outside world.  In many deployments, you have to dial a 9 or 8 or something to open a channel to the PSTN.  In some systems, when you press 9 (or whatever the prefix is, I'm going to assume the external access prefix is 9 throughout this article), you can hear the change in dial tone as you actually open a connection to the PSTN.

There are many reasons for using an external access prefix, one of the biggest is to provide a clear demarcation between internal and external calls. When dialing "off-hook" - meaning the user picks up the handset, hears a dialtone then starts dialing the number on the keypad - the PBX needs to know where to send the call based on the digits entered on the keypad.  This might sound obvious, but if you've punched in the digits 2-1-2-3, the PBX needs to know if you want to connect to someone at extension 2123, or if you're just in the process of dialing an external New York number that happens to start with 2123.  If you punched in 2123 and meant to reach extension 2123, then the PBX will connect your call to that extension and everybody's happy. But if you meant to dial New York number 2123334444, then you will be understandably frustrated that you were connected to extension 2123 instead.

The external access prefix deals with this issue in a neat and tidy way. When you dial off-hook, if you enter the external access prefix before dialing your number, then the PBX will not attempt to connect you to an internal extension as you punch in digits.  So, when you dial 92123, the PBX won't attempt to route your call to extension 2123, and will instead wait for the user to enter the remaining digits necessary to complete a call on the PSTN.

In many Lync Enterprise Voice deployments, Lync is connected to the PSTN through an existing PBX which gives Lync both access to the PSTN as well as phones on the PBX. Since Lync's PSTN access is via the PBX, then this means that calls that come from Lync usually needs to have the PBX's external access prefix added in front of PSTN-bound calls.

I've seen many deployments where administrators do one of two things:

1. They normalize dialed numbers in Lync to include the 9
2. They create a normalization rule where users have to dial 9, but they strip it out, leaving a properly formatted E.164 number, using a rule like ^9(\d{10})$ --> +1$1. They then add the 9 using a trunk translation rule before sending to the PBX.

With #1, this breaks Ken's universal E.164 rule, which states: "NORMALIZE EVERY NUMBER TO E.164!!!"  If you don't know what E.164 is, I recommend you check out this post on the topic.  For example, if your normalized number for our earlier example is +912123334444, that number is technically a phone number in Chinchwad, India (which incidentally is now on my list of funny-sounding place names).  91 is the country code for India, and 212 is the area code for Chinchwad.

The ramifications of choosing this path is that you have to adjust everything else in Lync to accommodate what you started with adding the 9 to all normalized numbers.  This means routes, and even phone numbers in Active Directory.  You may argue that this is OK for you, because you have a single site, and you don't have any need to call India. But AD phone numbers can populate users' contact lists on their mobile phones, and they most certainly don't want to be dialing India when they mean to call New York. You could do some additional work to deal with this situation, but it gets messy very fast and doesn't scale beyond a single site.  For example, what if you have a second site that doesn't use 9 for dialing externally?  If you have to route calls from the first site for redundancy, then you'd have to make all sorts of rules in Site B to deal with how you initially set things up in Site A.  As you add sites, this scheme would quickly become unmanageable. 

Case #2 is closer to the correct way to do things. Dialed numbers are ultimately formatted to E.164 standards which is good. Since the PBX ultimately requires numbers are prepended with a 9, we can do this with trunk translation rules. Its trivial to add a 9 to every call that routes out to the PBX. However, you shouldn't have to manually create normalization rules that force users to enter a 9.  If you recall from the beginning of this article, the external access prefix is meant to help people who do off-hook dialing. When dialing from the Lync softclient, there is no off-hook dialing.  You enter the digits, then you press the call button, or ENTER or whatever. Entering a 9 is superfluous in those cases.  However, you do want to ensure a good experience when people do off-hook dialing from a Lync desk phone. How do you support on-hook and off-hook dialing without forcing EVERYONE to enter an external access prefix in Lync?  Use the External Access Prefix option in the Lync dial plan, as shown below.   

This option does several things, firstly it allows people to dial phone numbers in Lync the same way they're accustomed to on the PBX without having to adjust normalization rules.  It also makes Lync work the same way a PBX phone works when dialing off-hook.  The kicker is that you're not forced to enter a 9 when dialing. You either enter a 9 when dialing from the Lync softclient, or you don't. Either way, the call goes through as expected. Same thing for the Lync deskphone. You're not forced to enter a 9, but if you habitually dial off-hook, then you'll want to, or you'll stand a good chance of reaching an internal extension instead of the PSTN number you dialed. 

Rather than rehash what's been said before, have a read of one of my blog posts from waaay back in Lync 2010 days on this topic (don't worry, things haven't changed since then).  It shows how to set your Lync deployment up for extension dialing and external access prefixes the no-muss, no-fuss way. It also shows some examples on how the Lync softclient behaves when using an external access prefix.

You can use this in pure Lync Enterprise Voice deployments, or when integrated with an existing PBX.  In cases where you connect to a PBX, you still have to create a trunk translation rule to add the 9, if necessary. For more information on trunk translation rules, you can read my blog post on the subject, or refer to Technet. 

Tune in next week, where we talk about inter-trunk routing, and how people inadvertently set it up without meaning to.

Lync Enterprise Voice Misconceptions #1 - Failover Routing

During my travels across the Lync Enterprise Voice landscape, I come across numerous Lync deployments that make some fundamental mistakes in how to manage phone numbers, call routing and other details.  They're always fixed by the end of the engagement, but I'd like to share some of these to help keep others from making the same mistakes.  This will be an ongoing series of unknown duration.

#1 - Failover/Resiliency Routing

This is one of the more misunderstood concepts in Lync Enterprise Voice. Its a question that often trips people up on the Lync Voice exam (70-337), and is something that even respected websites like Windows IT Pro gets wrong (despite my attempts to get them to correct it).

The idea behind failover or resiliency routing is that you want to use a particular PSTN gateway for your normal day-to-day call routing, but have a backup PSTN trunk ready in case the primary one is down. This is different from load-balanced routing, where your goal is to evenly spread call traffic between two or more trunks.

For example, say you have a Lync deployment in your Boston office and another one in your Seattle office.  For reasons that should be obvious, you want to ensure that your Boston users always use the Boston PSTN gateway for outbound calls to reduce WAN traffic and call latency. If the Boston trunk is down, calls should failover to the Seattle PSTN trunk.  The same goes for your Seattle users. We could make things more interesting by bringing least-cost routing into play, but we're going to keep it simple here.

A common error people make in an attempt to make things simple and easier to manage is to create a single route for all calls, and place both the Boston and Seattle trunks inside that route.  Since they added the Boston trunk first, it appears first in the list, and they are lulled into thinking that calls will always use the Boston trunk unless its down, at which point the Seattle trunk is used.

In actual fact, calls will load-balance themselves between the Boston and Seattle trunks. Sure, if Boston is down, all calls will route via Seattle, but any well-designed EV deployment should never needlessly send calls across the country on the corporate WAN unless absolutely necessary.  If your other site is in a different country, you could be left with some pretty high phone bills since half of your national calls will be routed out the other country's gateway as an international call.

Voice Policy	PSTN Usage	Route	Trunks
Global	AllCalls	AllCalls	BostonGW SeattleGW

A clue to this behaviour can be found when you look at a route in the Control Panel.  You'll notice that while you can add/remove trunks to a route, there is no way to change the order of the trunks. This should be a clue to the fact that you have no control over which trunk is used within a route.

The proper way to achieve failover routing is via PSTN Usage ordering. Instead of creating a single route with multiple gateways inside it, you create a Boston PSTN Usage with a route to the Boston PSTN gateway, and another Seattle PSTN Usage with a route to the Seattle gateway.  In the voice policy assigned to your Boston users, add both the Boston and Seattle PSTN usages, ensuring the Boston one is above the Seattle one.  For your Seattle voice policy, put Seattle first.

Voice Policy	PSTN Usage	Route	Trunks
Boston	Boston-AllCalls	Boston-AllCalls	BostonGW
	Seattle-AllCalls	Seattle-AllCalls	SeattleGW

Now, you have achieved true failover routing.  For Boston users, all calls will route out the Boston trunk, unless it is down.  The same thing works for your Seattle users.

Incidentally, putting multiple trunks  in a single route does have its uses. If you have multiple PSTN gateways within the same datacentre for redundancy, its OK to do things this way, since you usually don't really care which trunk is used for outbound calls. Calls will be load-balanced between all the trunks, skipping any gateways marked as down. The above picture shows exactly that situation (although its not obvious since most of the trunk name is truncated). However this should never be considered failover routing, simply round-robin or load-balanced routing.

Voice Policy	PSTN Usage	Route	Trunks
Boston	Boston-AllCalls	Boston-AllCalls	BostonGW1 BostonGW2
	Seattle-AllCalls	Seattle-AllCalls	SeattleGW1 SeattleGW2

Essentially, you want to avoid placing trunks from different locations within the same route. This ensures you have control over how your calls are routed, keeping WAN utilization and costs low.

Tune in next week for our next exciting episode: "PBX External Access Prefixes"

Command Line Options in Lync Optimizer Scripts

A lot of what drives development in the Lync Optimizer is my own personal needs. Every once in a while, I come across a deployment that has some requirement that the Optimizer doesn't cater to.  Most often, it's related to a country that isn't in the Optimizer, which is easy enough to remedy in most cases, with a lot of research and testing. Other times, its a quirk of the deployment that forces me to change how the Optimizer script runs.

On my latest project, there are over 100 sites that have to have their own routes/PSTN usages for places around the globe. Normally, I generate all the dial rules, and run the resulting scripts, answering the questions as they come up (Site ID? Least cost routing? Location based routing? Would you like fries with that?).  Until this past week, I was happy to press 1, Yes, No, No, Supersize me etc.  Not anymore.

Many of the finest PowerShell scripts out there (I'm looking at you, Lync Scriptboy) allow you to enter parameters as required for execution.  I figured it was time to add that feature to the Lync Optimizer generated scripts.

So, if you don't want to answer questions as the Lync Optimizer script is running, you can enter answers to those questions as parameters in the command line prior to execution. All parameters are optional. If you don't add the parameter in the command line, the script will ask you during execution as it always has.

An interesting new feature I enabled along with the command line options is the ability to control which sites are included as part of the least-cost/failover routing logic. Previously, if you have lots of sites with Enterprise Voice routes/policies etc. and you choose to enable least-cost/failover routing in the script, your resulting voice policy PSTN usage list would be very long.  This option now allows you to limit which sites to apply least-cost/failover routing to.  Just use the option -LCRSites followed by the ruleset prefix of the sites to apply least-cost failover routing enclosed with quotes.  For example:  Scriptname.ps1 -LeastCostRouting:$TRUE -LCRSites "CA-ON-Toronto, UK-London, SG-Singapore"

If you need to run a bunch of Optimizer-generated scripts, this can be a real timesaver, especially when you have to run the same script multiple times to account for multiple gateways, or applying least-cost routing and that sort of thing.

The Optimizer script also now has the proper syntax for getting help, so you can type HELP and it will spit out all the available options.

The available options are:

SiteID
The numeric identifier associated with the Lync site to apply the script to. If a value is not provided, script will ask during execution. To see a list of sites and the associated site IDs, run Get-CSSite

DialPlanType
Create a site-level or user-level dialplan. Site-level applies to all users at a site, while user-level have to be explicitly applied. Possible values are 'Site' or 'User'. If a value is not provided, script will ask during execution.

LeastCostRouting
Either apply or don't apply least cost routing to the given site. Only applies if multiple Lync Optimizer generated rulesets are detected. If a value is not provided (and multiple rulesets are detected), script will ask during execution.

LCRSites
Apply least-cost/failover routing only to rulesets defined in this list. Separate site names with commas. Input site names exactly the same as the prefix for the desired site (ie CA-ON-Toronto, UK-London etc).

OverwriteSiteVoicePolicy
Overwrite any existing site-level voice policy if values are already there. Only applies if a site-level voice policy exists. If a value is not provided, script will ask during execution, if necessary.

LocationBasedRouting
Apply location-based routing to a selected site. Only applies if Lync network sites have been created. If a value is not provided, script will ask during execution, if it detects Lync network sites.

LBRNetworkSite
The name of the Lync network site to apply location-based routing. Only applies if Lync network sites exist. For a list of Lync network sites, run Get-CSNetworkSite. Works in conjunction with LocationBasedRouting parameter. If a value is not provided, script will ask during execution, if network sites exist, and the user selects location-based routing.

LocalOnly
Creates only local routes and PSTN usages for the selected site. Useful when using a central SIP trunk for all dialing but want to have local-only voice policies for remote sites without their own local PSTN access. Assumes that the users at the remote site are assigned numbers appropriate to their site, and the SIP provider enforces the appropriate local dialing area for those numbers.

MediationPool
The FQDN of a mediation pool to apply the script to. Only applies if multiple mediation pools exist within the selected Lync site. If a value is not provided and multiple mediation pools exist, script will ask during execution.

PSTNGateway
The FQDN of a PSTN gateway/trunk to apply the script to. Only applies if multiple PSTN gateways/trunks are assigned to the selected mediation pool. If a value is not provided and multiple PSTN gateways/trunks exist, script will ask during execution.

ApplicationPool
The FQDN of an application pool to apply the script to. Only applies if multiple application pools exist within the selected Lync site, and call park or premium call blocking is being used. If a value is not provided and multiple application pools exist, and either call park or premium call blocking is being used,  script will ask during execution.

Enter any desired options on the command line using the following syntax:

ScriptName.ps1 -SiteID digit -DialPlanType user/site -LeastCostRouting true/false -LCRSites "sitename1,sitename2,sitename3" -OverwriteSiteVoicePolicy true/false -LocationBasedRouting true/false -LBRNetworkSite NetworkSiteName -MediationPool MedPoolFQDN -PSTNGateway PSTNGatewayFQDN -ApplicationServer AppServerFQDN

Lync 2013 Edge server bug when adding new server pools

I had this happen to me a few weeks ago, and a co-worker had it happen to him today, so I figure it should be mentioned since nobody on the Internet seems to have a proper fix for it.

I installed a new Lync 2013 Standard Edition server in an existing Lync 2013 environment with an established edge server pool.  I created a test account on the new pool and began running through the usual testing scenarios.

All seemed well and fine until we tested federated communications.  An external party could establish an IM or A/V session with the internal user homed on the new pool and see their presence, but the internal user couldn't see presence nor establish a new IM or A/V session with anybody outside the company.

Puzzling to be sure, since everybody else in the company was OK.  I first checked to see if I could telnet to the typical ports required for edge functionality, and all was good. I then logged onto the edge to see if there were any relevant events.

Lo and behold, there was this interesting little warning:

LS Protocol Stack Event 14402
Multiple incoming connections on internal edge from non-internal servers.
In the past 289 minutes the server received 4 incoming connections on internal edge from non-internal servers. The last one was from host newlyncfe.contoso.com.
Cause: This can happen if an internal server is not present in the list of internal servers on the Access Edge Server.
Resolution:
If the server is a valid one, you need to add it to the list of internal servers on the Access Edge Server. If the server is invalid, you may be under an attack from that server.

This left me scratching my head.  How do you add a Lync 2013 server to the "list of internal servers"???  I recalled back to the olden times of OCS, where you had to manually add every valid server name to the edge, but you don't do that anymore, thanks to the wonder of the Topology Builder.

Internet searches brought up several cobweb-covered webpages from the late 2000's, all of them relating to OCS.  No good.  I re-ran Setup on the edge server, hoping that would trigger something and make things all good, but nope.

Finally, I decided to restart the Lync Server Access Edge service (during a period of low activity of course).  Once it restarted, all my troubles went away.  The user on the new front-end was able to initiate sessions to external users, and the 14402 errors on the edge went away.

My next step was going to be blindly restarting other services followed by a last-ditch server reboot. Since that turned out to be unnecessary, if this saves someone from doing a server restart (which would fix the problem too), that's awesome.

So, while Lync normally does an excellent job of detecting new servers and functioning fine without service restarts or server reboots, this is one situation where its not doing that.  It seems the Access Edge service caches the list of valid internal servers at service startup, and doesn't refresh that cache until the next service restart.

Because this seems to be one of the few instances where a service restart is required to maintain connectivity with a new downstream server, I would classify this behaviour as a...

Lync Address Book Weirdness

So, I'm at a fairly large client that's moved from Lync 2010 to 2013.  The server-side migration completed a while ago, and the client migration is in progress.  Enterprise Voice is enabled for a select group of people.

Some people have noticed that when they search for a specific Lync user, the only object that got returned was an object for their Active Directory administrative account that had only a phone number defined (not a Lync-enabled account).  The Lync-enabled "normal" Lync entry didn't show up in the search, even though it did appear in the GALContacts.db file (the Lync locally cached address book file).

The really weird thing was that the object showed up like a phone contact object, but one that had presence working.  The affected user could toggle their presence, and the contact object would dutifully change itself.  However, since it wasn't a Lync object, you couldn't start an IM (but you could call the associated number).

This user's first name isn't actually "Admin" in case you were wondering.

Another weird thing was that the very first time you pulled up that person's name from a search, it would briefly show the proper Lync-enabled account, but switch to the phone contact object view after a half-second or so.

Yet ANOTHER weird thing, was that after viewing the admin account object in Lync a few times, usually after viewing the contact card, the next time I did a search for the same name, only the phone number appeared.  However, I could still look at the contact object for the phone number and it would show the user's email address.

As mentioned previously, the address book had the correct contact information (verified by opening the GALContacts.db file in Notepad....the results ain't pretty, but it works), as well as the information from the AD admin account, but Lync was choosing to show the non-Lync enabled admin account in searches.

When I compared the affected admin accounts with others, I noted that some users showed up properly, but others didn't.  More comparisons showed that for the incorrect information to show up, the following had to be true:

1. Both the Lync-enabled account and the AD admin account had a phone number defined
2. Both the Lync-enabled account and the AD admin account had the same email address defined.

Removing the email address from the admin account, re-generating the Lync address book via Update-CSAddressBook, and re-downloading the updated address book file fixed the issue. 

So, what happened?  When you do a search, the Lync client will look at the address book, your own Outlook contacts, and any social connectors you may have associated with Outlook (like Facebook or LinkedIn).  If the same name shows up in multiple places, Lync will try to consolidate all the information and show only one contact object.

Sometimes, the process goes awry. In this case, it appears that Lync saw the same email address for both the Lync-enabled AD account, and the non-Lync enabled AD admin account, and incorrectly presented the non-Lync enabled account as the single object.

I'm guessing this is a bug, because Lync really should place priority on a Lync-enabled object over a non-Lync one for presentation.