Lync Enterprise Voice Misconceptions #2 - External Access Prefixes

During my travels across the Lync Enterprise Voice landscape, I come across numerous Lync deployments that make some fundamental mistakes in how to manage phone numbers, call routing and other details.  I've covered a lot of these details in parts of my OTHER Enterprise Voice Best Practices, but its obviously a lot to take in, and I've seen enough of the same sort of errors in the wild that prompted me to do this series.

Last week's episode talked about common errors when setting up resiliency/failover routing. This week, we're talking about...

#2 - External Access Prefixes

A lasting legacy of the PBX era is the concept of dialing an external access prefix to make a call to the outside world.  In many deployments, you have to dial a 9 or 8 or something to open a channel to the PSTN.  In some systems, when you press 9 (or whatever the prefix is, I'm going to assume the external access prefix is 9 throughout this article), you can hear the change in dial tone as you actually open a connection to the PSTN.

There are many reasons for using an external access prefix, one of the biggest is to provide a clear demarcation between internal and external calls. When dialing "off-hook" - meaning the user picks up the handset, hears a dialtone then starts dialing the number on the keypad - the PBX needs to know where to send the call based on the digits entered on the keypad.  This might sound obvious, but if you've punched in the digits 2-1-2-3, the PBX needs to know if you want to connect to someone at extension 2123, or if you're just in the process of dialing an external New York number that happens to start with 2123.  If you punched in 2123 and meant to reach extension 2123, then the PBX will connect your call to that extension and everybody's happy. But if you meant to dial New York number 2123334444, then you will be understandably frustrated that you were connected to extension 2123 instead.

The external access prefix deals with this issue in a neat and tidy way. When you dial off-hook, if you enter the external access prefix before dialing your number, then the PBX will not attempt to connect you to an internal extension as you punch in digits.  So, when you dial 92123, the PBX won't attempt to route your call to extension 2123, and will instead wait for the user to enter the remaining digits necessary to complete a call on the PSTN.

In many Lync Enterprise Voice deployments, Lync is connected to the PSTN through an existing PBX which gives Lync both access to the PSTN as well as phones on the PBX. Since Lync's PSTN access is via the PBX, then this means that calls that come from Lync usually needs to have the PBX's external access prefix added in front of PSTN-bound calls.

I've seen many deployments where administrators do one of two things:

1. They normalize dialed numbers in Lync to include the 9
2. They create a normalization rule where users have to dial 9, but they strip it out, leaving a properly formatted E.164 number, using a rule like ^9(\d{10})$ --> +1$1. They then add the 9 using a trunk translation rule before sending to the PBX.

With #1, this breaks Ken's universal E.164 rule, which states: "NORMALIZE EVERY NUMBER TO E.164!!!"  If you don't know what E.164 is, I recommend you check out this post on the topic.  For example, if your normalized number for our earlier example is +912123334444, that number is technically a phone number in Chinchwad, India (which incidentally is now on my list of funny-sounding place names).  91 is the country code for India, and 212 is the area code for Chinchwad.

The ramifications of choosing this path is that you have to adjust everything else in Lync to accommodate what you started with adding the 9 to all normalized numbers.  This means routes, and even phone numbers in Active Directory.  You may argue that this is OK for you, because you have a single site, and you don't have any need to call India. But AD phone numbers can populate users' contact lists on their mobile phones, and they most certainly don't want to be dialing India when they mean to call New York. You could do some additional work to deal with this situation, but it gets messy very fast and doesn't scale beyond a single site.  For example, what if you have a second site that doesn't use 9 for dialing externally?  If you have to route calls from the first site for redundancy, then you'd have to make all sorts of rules in Site B to deal with how you initially set things up in Site A.  As you add sites, this scheme would quickly become unmanageable. 

Case #2 is closer to the correct way to do things. Dialed numbers are ultimately formatted to E.164 standards which is good. Since the PBX ultimately requires numbers are prepended with a 9, we can do this with trunk translation rules. Its trivial to add a 9 to every call that routes out to the PBX. However, you shouldn't have to manually create normalization rules that force users to enter a 9.  If you recall from the beginning of this article, the external access prefix is meant to help people who do off-hook dialing. When dialing from the Lync softclient, there is no off-hook dialing.  You enter the digits, then you press the call button, or ENTER or whatever. Entering a 9 is superfluous in those cases.  However, you do want to ensure a good experience when people do off-hook dialing from a Lync desk phone. How do you support on-hook and off-hook dialing without forcing EVERYONE to enter an external access prefix in Lync?  Use the External Access Prefix option in the Lync dial plan, as shown below.   

This option does several things, firstly it allows people to dial phone numbers in Lync the same way they're accustomed to on the PBX without having to adjust normalization rules.  It also makes Lync work the same way a PBX phone works when dialing off-hook.  The kicker is that you're not forced to enter a 9 when dialing. You either enter a 9 when dialing from the Lync softclient, or you don't. Either way, the call goes through as expected. Same thing for the Lync deskphone. You're not forced to enter a 9, but if you habitually dial off-hook, then you'll want to, or you'll stand a good chance of reaching an internal extension instead of the PSTN number you dialed. 

Rather than rehash what's been said before, have a read of one of my blog posts from waaay back in Lync 2010 days on this topic (don't worry, things haven't changed since then).  It shows how to set your Lync deployment up for extension dialing and external access prefixes the no-muss, no-fuss way. It also shows some examples on how the Lync softclient behaves when using an external access prefix.

You can use this in pure Lync Enterprise Voice deployments, or when integrated with an existing PBX.  In cases where you connect to a PBX, you still have to create a trunk translation rule to add the 9, if necessary. For more information on trunk translation rules, you can read my blog post on the subject, or refer to Technet. 

Tune in next week, where we talk about inter-trunk routing, and how people inadvertently set it up without meaning to.

Fixing Call Issues in a Mitel 3300 - Lync 2013 Deployment

We recently deployed Lync Server 2013 Enterprise Voice for a customer who at the same time replaced their old Mitel PBX with a newer Mitel 3300. The customer wasn't ready to make a full commitment to Lync for voice traffic, but probably 80% of the customer's users are using Lync exclusively for voice.

The Lync infrastructure was connected to the PSTN via the Mitel 3300 PBX using direct SIP.  While things initially seemed to be fine, we were getting numerous reports of sporadic failed calls throughout each day.

Digging through the monitoring server reports (deploying monitoring server reports should be considered ESSENTIAL for any Lync Enterprise Voice deployment), we could see a clear trend for the call failures: Looking at the Failure Distribution Report, we could see the top failed call reason was Gateway side Media negotiation failed.

Clicking on the sessions number hyperlink brought us to the list of calls that failed due to that cause.  Selecting one shows the session detail report for that call:

At the bottom is the Diagnostic Report.  Clicking the Detail link on the server side finally shows the exact reason for the call failure:

10010; source="LyncFE.contoso.com"; reason="Gateway side Media negotiation failed"; component="MediationServer"; ValidationError="SDP Exchange already failed before SetAnswer: Session c line = 0.0.0.0 or ::, call hold not allowed for initial INVITE offer"; GatewayFqdn="10.1.1.2; trunk=NA-OH-Dayton-Mitel"

I did a search through my normal channels and didn't come up with much.  I saw a few references to clients who set the Lync client audio port range to 3, rather than the recommended 20-40 which caused their particular issue, but we were already using 40 as a client port range.

It seemed as though the issue was coming from the Mitel side.  I tried a few things like setting the trunk's RTCPActiveCalls and RTCPCallsonHold values to FALSE on the Lync side, but that had no effect.

We tried to contact the Mitel partner who put in the Mitel system, but they were of very little help.  I asked if I could poke around the Mitel web console to look for a potential fix. Surprisingly, the client said "Sure".  Having never seen a Mitel before, I thought that things were about to go very wrong, or at least we wouldn't get any closer to a solution.

Since the error was related to SDP, we focussed on the SDP Options tab under SIP Peer Profile on the Mitel.  The most promising option was Avoid Signalling Hold to the Peer.  It was promising because the error message from the Lync monitoring reports mentioned both SDP and hold, so.......we set it to Yes, and at the same time we set Enable Mitel Proprietary SDP to No.  Everything else we left alone.

Almost immediately, our problems came to an end. Calls to numbers that would fail every 2nd or 3rd attempt suddenly worked without error every single time. Another side benefit was that we were now getting music on hold when transferring to Mitel call queues, where we weren't before.

I was amazed that we had brought up the issues around both music on hold and the failed calls on numerous occasions, and the Mitel partner could not find a solution, while someone with zero experience with Mitel was able to find a solution in under 10 minutes of poking around.

I was feeling pretty good about fixing that issue, so I tried to solve the other remaining issue where outbound calls would only show the main office number rather than the individual Lync user's DID. However, my beginners luck ran out and I couldn't find a solution.

So, if you're having similar issues with a Mitel - Lync deployment, hopefully this will steer you in the right direction.

Inter-trunk Routing in Lync 2013

Introduction - What is Inter-trunk Routing?

Inter-trunk routing is a new feature in Lync 2013 that allows Lync Server to be the central call processor for a multi-PBX deployment.  In previous versions, Lync was only capable of being the final destination for phone calls.  If an inbound phone call couldn't be matched to a Lync user, the call would fail.  There was no way to have Lync forward calls on to another PBX, which limited the configuration options available to administrators.

Inter-trunk routing gives Lync administrators the ability to take calls in from a variety of sources and forward them on to other non-Lync destinations if there isn't a matching Lync user with the target phone number. 

The Problem

Recently, I had to deploy inter-trunk routing for a Lync deployment that had calls coming in from a variety of sources to a centralized Lync Server 2013 Enterprise Edition deployment:

• a Lync-certified SIP trunk
• a Canada-based T1 line connected to a PSTN gateway
• two US-based Mitel PBXs. 

The inter-trunk routing requirements were very specific: forward four phone numbers entering Lync from the SIP trunk to one of the US-based PBXs.  The four phone numbers were +1 (289) 555-3926, +1 (289) 555-6733, +1 (289) 555-6734 and +1 (289) 555-6780.

The Solution

Having never done Lync inter-trunk routing, I turned to the trusty Internet for answers.  I found lots of information on WHAT Lync inter-trunk routing was all about, but almost nothing on exactly HOW to configure inter-trunk routing in Lync.  I saw references to assigning PSTN usages to trunks, and saw the place in the Lync Control Panel to add it, but nothing but vague allusions to the process of actually implementing it.

UPDATE 10-June-2013: Richard Brynteson posted a more knowledgable deep-dive into inter-trunk routing mere days after I posted this. I made a slight adjustment to my process based on what I read in his blog, so its definitely worth reading.

As usual, when I come across such a situation, I assume that it must be so simple to configure that it didn't occur to anyone to actually document the HOW.  I then assume I must be a complete idiot for not knowing something so plainly obvious to every other Lync administrator on the planet. So, after much flailing around, I figured out a solution that works using inter-trunk routing.

Since I wanted to route only four specific phone numbers between trunks, I needed to make sure that I enabled inter-trunk routing for those numbers and no others.  If I allowed inter-trunk routing for too wide a range of phone numbers, we could get into a situation where calls could enter Lync, pass onto the target PBX and route back into Lync, causing a call loop.  Rather than leave it to the target PBX to ensure a call loop doesn't happen, I figured it best to ensure Lync only pass the specific numbers to the PBX.

I started by editing the site-level voice policy attached to the Lync site where the incoming SIP trunk calls came into Lync.  I created a dedicated PSTN usage/route combination for those four numbers.  The PSTN usage was called NA-ON-Cooksville-289555-MitelForward to keep with the naming convention used by the Lync Dialing Rule Optimizer.  The route was called NA-ON-Cooksville-289555-MitelForward with a route pattern of ^\+1289555(3926|67(33|34|80))$ pointing to the PBX at 1.1.1.1. 

The PSTN usage was placed at the very top of each voice policy's PSTN usage list to ensure that any Lync user that tries to dial those numbers will forward the calls to the desired PBX. For good measure, I also added the PSTN usage to the Global voice policy, although I'm not sure this is necessary.

Next, I assigned the new PSTN usage to the incoming SIP trunk under Trunk Configuration.  If the trunk doesn't exist, add it as a new pool trunk.  I then added a trunk for the PBX and added trunk translation rules to strip the number down to 5 digits, which is what the target PBX is expecting.

Once done, I committed the changes and waited a few minutes for the changes to bake to a nice golden brown.

The moment of truth came when I was able to call one of those four numbers and listen as Lync seamlessly passed the calls onto the Mitel PBX.  Great success!

Conclusion

I'd like to say it was my deep, Yoda-like knowledge of how call routing works in Lync that allowed me to think through the process of determining how inter-trunk routing works. Unfortunately, the truth is that I tried random combinations of things until I stumbled blindly onto the solution.  There's a reason why I chose that picture of the Hoff looking vacant and stunned at the same time as my avatar....it's how I look and feel much of the time.

To summarize the process:

1. Create a PSTN usage/route combination that is specific to the phone numbers required to be forwarded to the PBX.
2. Place the PSTN usage at the top of all voice policies to ensure that both inbound calls from the PSTN and calls made by internal Lync users are routed to the proper destination.
3. Assign the PSTN usage to the incoming trunk.

When complete, the whole call process should function as follows:

1. Inbound call comes into Lync via SIP trunk.
2. Lync normalizes the number into an E.164 phone number
3. Lync does a reverse number lookup to check for a Lync user with the same number
4. If no match in Lync, then it looks for a matching usage/route assigned to a trunk
5. When found, applies any outbound normalization rules and sends the call to the next hop

For a more detailed description on how the call flow will proceed, check out the Mastering Lync blog.  Richard Brynteson did a series on the nitty gritty details behind dialing behaviours in Lync. His post on Inter-trunk dialing is a deep-dive into the details that I don't get into here.

If there's anyone else out there trying to figure out how to implement Lync inter-trunk routing, I hope this helps figure things out.  As always, if you have any questions or can correct any misunderstanding I might have on the process, just leave a comment.

Migrating PBX Users to Lync Enterprise Voice

In all my previous blog entries about Lync Enterprise Voice Best Practices, I've assumed that the Lync deployments described are pure Lync.  There is no mention about connecting to legacy PBXs and how to best setup Lync to integrate with them.  Well, today is the day that we deal with that thorny issue.

Typically, there are two types of PBXs.  The oldest style are purely analog devices and are known in the industry as TDM PBXs.  TDM is short for Time Division Multiplexing - the old analog way of slicing up a telephony pipe to allow multiple calls.  Since Lync is not an analog phone system, you require an IP-PSTN gateway to communicate with TDM PBXs.  There are several gateway vendors out there that provide Lync-compatible gateways.  Some of my favourites include Audiocodes, Dialogic and NET.

The other type of PBX is called an IP-PBX, which is digital in nature and uses TCP/IP for its voice backbone.  Pretty much every PBX sold today is IP-based.  An IP-PBX may use SIP or other proprietary protocol for communications.  If the IP-PBX uses SIP, then there is a fairly good chance it can communicate directly with Lync over TCP/IP.  This is by no means a sure thing, because every vendor interprets the SIP standard differently, and this often means that two different SIP-based telephony systems can't communicate with each other.  In that case, you'll need to use an IP-PSTN gateway to bridge the two.  Even if you can get Lync to talk to the PBX for basic calls, you may encounter issues with transferring or forwarding, which is again due to the different ways that vendors interpret the SIP standard.

Connecting a PBX to Lync
There are two ways to connect an IP-PSTN gateway to a PBX. The most obvious way is to put the gateway behind the PBX (as shown below) connected via a T1 crossover cable. This is the lowest impact solution for the PBX side of things, but it can make routing calls to Lync difficult, because you will be relying on the capabilities of the PBX for such decisions. If you are not well-versed in the operation of the PBX, this will usually require the services of a PBX technician, which can cost both money and time. If you are working with a TDM-PBX, then you are also relying on the availability of a spare PRI card on the PBX. This may not be available due to limited expansion capabilities or availability of hardware.

This may be the only workable solution if your users are assigned extensions, rather than direct numbers (DIDs), because the gateway will always just see the main office number on inbound calls, and won't be able to tell if the call should go to Lync or the PBX.  Also, if you rely on an autoattendant for routing calls, you will need to route all calls to the PBX until you've migrated all users to Lync.

If your users all have DIDs or your PBX doesn't have any expansion slots available for an additional PRI card, then the more preferred solution is to place the IP-PSTN gateway in front of the PBX.  This requires 2 PRI interfaces on the gateway (or more if you have multiple lines coming in to the PBX).  Most gateways allow you to set the interfaces in pass-through mode, so that a failure in the gateway will just pass all calls through as if it weren't even there.

Now inbound routing logic is controlled by the gateway, which is much easier to manage than most PBXs.  Usually, you set a default rule for calls bound to the PBX, and insert rules to forward calls to Lync users as you see fit.  Many gateways provides Active Directory-based routing that can detect if the user is enabled in Lync and route calls accordingly.

Outbound routing from the PBX should be relatively easy as well.  In some cases, you would create a default rule that would route any extension that doesn't have an internal match out of the PBX, and the gateway will route the call to Lync.  Other cases may involve setting up forwarding on the internal extension to a pre-defined extension range, that then gets transferred to the gateway and on to Lync.

Setting Up Lync to Route Calls to a PBX
Once you've got the hardware setup, you need to connect Lync to the PBX or gateway.  This guide won't get into the details of setting up a specific gateway to work with an given PBX.  There are lots of those out there.  Once you've got a working connection, you need to define the IP address of the gateway in your Lync topology document.  You define the gateway under the PSTN gateways node of your Lync site.  A PSTN gateway needs to connect to a mediation pool, which is often collocated with the front-end Lync server role. 

As followers of this blog may know, I am a big proponent of e.164 formatting for phone numbers.  Ideally, all the phone numbers you define for Lync use - from the actual number assigned to an Enterprise Voice enabled user to the phone numbers listed in Active Directory - should be formatted in e.164 format.  If your users have DIDs, then this should be straightforward.  For users that only have extensions, you should use the format +<countrycode><officenumber>;ext=<extension>as explained in my blog post on Internal Extensions.  Resist the temptation to use only the extension as the user's phone number.

As you transition to Lync, you should allow your Lync users to dial others using the methods they are familiar with.  For most, this means dialing by extension.  You will want to allow extension dialing in Lync with the least amount of administrative overhead as you migrate users from the legacy system to Lync.

You achieve this by creating normalization rules that translate extensions to full e.164 phone numbers and outbound translation rules that turn e.164 phone numbers BACK into extensions.  An example with a company whose main number is +1 (212) 555-1111 that uses 3-digit extensions starting with 2 is below:

Normalization Rule: ^(2\d{2})$ --> +12125551111;ext=$1
Outbound Translation Rule: ^\+12125551111;ext=(2\d{2})$ --> $1

Sample scenario:
Dial 234 --> normalize to +12125551111;ext=234 --> strip to 234 --> send to PBX

You may wonder why go through the trouble of normalizing 3-digit extensions to e.164 when we just strip it back down to 3-digits at the gateway.  A long time ago, I evangelized the reasons why you should use e.164 formatting for your user phone numbers.  I won't rehash what was said there, but it suffices to say that e.164 formatting is the way to go in almost every situation.  So, using the above example with +1 (212) 555-1111 as the main office number, assign phone numbers to your Lync users like this:  tel:+12125551111;ext=201.  If a Lync user dials 201, it will normalize to the full e.164 format, find a match in its database and ring the user.

If someone dials an extension that doesn't yet exist in Lync, say 299, it will normalize to e.164, fail to find a match in its database, and then route it out to the PBX after stripping it back down to 3-digits. 

The Lync Dialing Rule Optimizer can help ease the workload involved in setting up this routing logic.  Simply select the Use Extensions checkbox (coming soon), and enter up to 10 extension ranges.  The Optimizer will take care of the rest, including proper ordering and placement of rules.

Migrating PBX Users to Lync
Following this generalized plan makes migrating users to Lync very simple:

1. Enable user in Lync with the proper e.164 phone number.  Lync-initiated calls to their extension should now be intercepted by Lync.
2. Depending on the layout of your legacy PBX, you may just need to simply delete the users extension, and PBX-initiated calls to that number should default to routing out to the gateway.  Other PBXs may require additional work to ensure calls to that specific user route to the gateway.
3. Depending on the gateway, you may need to modify your PSTN-inbound rule to route inbound PSTN calls for that user to Lync.  As mentioned earlier, some gateways can detect the user is enabled on Lync and route the call accordingly.

The first few times I had to go through this exercise, it took me a while to wrap my head around what I now think is the best way to do things.  I hope this will help others get over some of the hurdles of deploying Enterprise Voice and do thing right the first time.

Links to my other posts in my Enterprise Voice Best Practices series:
E.164 Formatting
Normalization
Usages and Routes
Extensions