Response Groups Stop Responding

Our company (Event Zero - makers of the best Skype for Business analytics software out there BTW) relies on Skype for Business response groups for our sales and support queues.  Last week, I noticed that every single one of them on two separate pools were no longer accepting calls.  It wasn't a SIP trunk or mediation issue, because I couldn't get to them by directly entering in their SIP address in the Skype for Business client.  They appeared available (green presence) but would not accept calls. Snooper logs showed they were throwing 480 Temporarily Unavailable errors.

It was especially odd that it happened on two separate S4B pools at roughly the same time.  I tried numerous things, from restarting the RGS service on the affected servers to restarting the servers.  So, yeah, not a lot of tools in my RGS troubleshooting arsenal apparently.

What I found DID work, was to change the Tel URI of one of the workflows to a slightly different number, then changing it back.  Within a few minutes, that particular workflow started working again. 

Rather than doing the same thing to all 20-odd response groups (which would take a LOOONG time because the RGS Workflow web page is so slow), I created a Powershell script to do the same thing.

WARNING: Use script at your own risk. It worked fine for me, but hey I'm not a programmer. Also, this script will use the Description field of the workflow to store the original Tel URI of the response group.  Whatever is there now will get wiped out. You can do this script in other ways, but I was strapped for time and we weren't using the Description field for anything.

$Workflows = Get-CsRgsWorkflow 

Foreach ($WF in $Workflows)
{
 Write-Host "Adding dummy extension to " $WF.Name
 $WF.Description = $WF.LineURI
 $WF.LineURI = $WF.LineURI + ";ext=0"
 Set-CSRgsWorkflow -Instance $WF
}
Foreach ($WF in $Workflows)
{
 Write-Host "Reverting back to original number for " $WF.Name
 $WF.LineURI = $WF.Description
 Set-CSRgsWorkflow -Instance $WF
}

I don't know what caused the problem to start with, but at least this fixed it. Presumably, something happened to "break" the connection between RGS and the associated contact objects, and resetting the Tel URI re-linked them.

Hopefully, this might help others who come across the same thing. If anybody has any insight into why it broke, and why this fix worked, please enlighten me!

Large Scale Extension Dialing in Lync

I was recently working a pretty large Enterprise Voice deployment that had more than a hundred sites located around the world. Every site used DIDs for their users phone numbers, but wanted to be able to reach every other site using a 3-digit site code, plus the last 4 digits of the user's phone number (total 7 digits). This would be pretty easy except for the following wrinkle: they wanted people within any one site to be able to dial just the last 4-digits of other people within that site, rather than the 7-digit "global" extension or full DID. To complicate matters, multiple sites had overlaps with the last 4 digits of their numbers.

Unfortunately, this meant that every single site would require their own dedicated Lync dial plan, with only one rule that is different from the other sites. Management of these sites can be a nightmare, especially when you consider what has to happen when a new site gets added. Not only do you have to add the other sites extension normalization rule to the new site, but every other site has to have their dial plans updated with the new site's extension normalization rule. With many sites, this can become difficult to manage.

Thankfully, Powershell can take a daunting task such as this and make it much easier. I've developed a process to manage large numbers of extension ranges that is scalable and easy to replicate at other sites. Of course, this being me, the core of it is based around Lync Optimizer generated rulesets, and the associated scripts assume you follow the same naming convention. So, without further ado....

For this particular company, I decided on a naming convention for 7-digit extension normalization rules that follow the format CountryID-StateProv-City-7Ext0x (eg. US-TX-Dallas-7Ext01, US-TX-Dallas-7Ext02 etc). 4-digit extension rules for specific sites use the format CountryID-StateProv-City-4Ext0x (eg. US-TX-Dallas-4Ext01).  Every site's dial plan would end up with a large number of 7Ext normalization rules, and a single 4Ext normalization rule specific to that site.

I kept a repository of 7-digit extension dialing rules for all sites, organized alphabetically in a manually created dialplan called ZZ-ExtensionList. This dialplan was only used for the scripts I used later to populate new site lists.

Each site used the Lync Dialing Rule Optimizer to generate rulesets for each site. I used the Extensions entry page to generate the local site's 7Ext0x normalization rule.  For this particular company, since we were using a 3-digit site code and the last 4-digits of the user's DID, the extension entry page looked something like this:

For the above example, the 3-digit site code is 350, and user's phone numbers are in the range +442055662000 to +442055662099.  The # of Ext Digits in DID field tells the Optimizer that only the last 4 digits of the 7-digit extension is actually part of the DID.

I ran the Optimizer generated script against the deployment as per usual, which generated the standard ruleset for the given site. Once complete, I opened the ZZ-ExtensionList dial plan and added the newly generated 7Ext0x rule(s) to the list, making sure to maintain alphabetical order.

I then ran the custom script AddExtensionsToDP.ps1 (see below) with the switch -DialPlan set to the name of the newly added dialplan (ie .\AddExtensionsToDP.ps1 -DialPlan UK-London) This script does two things. First, it adds all the extension dialing rules from ZZ-ExtensionList into the new dialplan. Secondly, it modifies the 7Ext0x rule(s) specific to the new site into a 4Ext0x rule(s). So, for the above example:

Original Pattern/Translation: ^350(20\d{2})$          -->    +44205566$1
New Pattern/Translation:       ^(?:350)?(20\d{2})$  -->    +44205566$1

The new pattern allows users within the site to either dial 20xx or 35020xx for other users within the site.

AddExtensionsToDP.ps1

# This script adds all the extension normalization rules from ZZ-ExtensionList to the selected dialplan # Assumes you've already generated the new dialplan via the Lync Optimizer (www.ucdialplans.com) # Then run UpdateNormRules.ps1 to add the new normalization rule to every other site [CmdletBinding(SupportsShouldProcess = $True)] param( [parameter(ValueFromPipeline = $false, ValueFromPipelineByPropertyName = $true, Mandatory = $true)] [string]$Dialplan ) $NormRuleList = Get-CSVoiceNormalizationRule | Where {$_.Identity -like 'Tag:ZZ-ExtensionList/*' -and $_.Identity -notlike "Tag:ZZ-ExtensionList/$DialPlan-7Ext*"} | Sort-Object Priority ForEach ($NormRule in $NormRuleList) { New-CSVoiceNormalizationRule -Parent $DialPlan -Name $NormRule.Name -Description $NormRule.Description -Pattern $NormRule.Pattern -Translation $NormRule.Translation -Priority $NormRule.Priority -IsInternalExtension:$TRUE } # Replace the 7Ext rule for the target site with its own custom Ext rule $7ExtRules = Get-CsVoiceNormalizationRule | Where {$_.Identity -like "Tag:$DialPlan/$DialPlan-7Ext*"} | Sort-Object Priority -Descending ForEach($7ExtRule in $7ExtRules) { $RuleName = $7ExtRule.Name $NewRuleName = $RuleName -replace "7Ext", "4Ext" $NormPattern = $7ExtRule.Pattern $NewNormPattern = $NormPattern -replace "\^(\d{3})", '^(?:$1)?' New-CSVoiceNormalizationRule -Parent $DialPlan -Name $NewRuleName -Description $7ExtRule.Description -Pattern $NewNormPattern -Translation $7ExtRule.Translation -Priority 0 -IsInternalExtension:$TRUE Remove-CSVoiceNormalizationRule $DialPlan/$RuleName }

Once that script was done, then I ran the UpdateNormRules.ps1 script (shown below) with the -NormRuleBase switch to specify the name of the dial plan created at the beginning. (ie. .\UpdateNormRules.ps1 -NormRuleBase UK-London). This script adds the normalization rules for the new site to all the other site-specific dialplans, in the proper alphabetical order.

UpdateNormRules.ps1

# This script is designed to be run after implementing a new dialplan for a site. # It will insert any extension normalization rule for that site into the dialplan for every other site in alphabetical order # Prior to running this script, insert the new normalization rule(s) in the ZZ-ExtensionList dialplan in the appropriate order. Make sure you name the rule(s) as per the following: # 2-Letter countrycode-State/Prov-City-7Ext0# (where # starts at 1), ie US-TX-Dallas-7Ext01 # # This script will copy the normalization rule to every other dialplan in the same order [CmdletBinding(SupportsShouldProcess = $True)] param( [parameter(ValueFromPipeline = $false, ValueFromPipelineByPropertyName = $true, Mandatory = $true)] [string]$NormRuleBase ) # Find the priority of the first norm rule we want to insert $NormRuleListRev = Get-CSVoiceNormalizationRule | Where {$_.Identity -like "Tag:ZZ-ExtensionList/$NormRuleBase*"} | Sort-Object Priority -Descending ForEach ($NormRulePriority in $NormRuleListRev) { $BasePriority = $NormRulePriority.Priority } $NormRulePrior = (Get-CSVoiceNormalizationRule | Where {$_.Identity -like "Tag:ZZ-ExtensionList/*" -and $_.Priority -eq $BasePriority-1}).Name $NormRulePriorPrefix = $NormRulePrior.substring(0,$NormRulePrior.LastIndexOfAny("-")) $NormRulePriorSuffix = $NormRulePrior.substring($NormRulePrior.Length-5,5) $NormRulePrior = $NormRulePriorPrefix + "*" + $NormRulePriorSuffix $DPList = Get-CsDialPlan | Where {$_.Identity -like '*-*' -and $_.Identity -ne 'Tag:ZZ-ExtensionList' -and $_.Identity -ne "Tag:$NormRuleBase" -and $_.Identity -notlike 'Service:*'} Foreach ($DP in $DPList) { $DPName = $DP.Identity $Priority = (Get-CSVoiceNormalizationRule | Where {$_.Identity -like "$DPName/$NormRulePrior"}).Priority + 1 Write-Host $DP.Identity $Priority If ($Priority -ne $NULL) { ForEach ($NormRule in $NormRuleListRev) { New-CsVoiceNormalizationRule -Name $NormRule.Name -Parent $DPName -Description $NormRule.Description -Pattern $NormRule.Pattern -Translation $NormRule.Translation -IsInternalExtension:$TRUE -Priority $Priority } } }

Once complete, the new site had all the existing site-level extension normalization rules for every other site, and all existing sites also had the new site's extension normalization rule inserted into their dial plans. This ensured that all sites could dial any other site using their site code and extension.

So, to summarize the workflow:

1. Manually create an empty dialplan called ZZ-ExtensionList
2. Generate a Lync Optimizer dialplan for your selected area, populating the Extensions using the appropriate format for your deployment. Make sure the Suffix is set to 7Ext01 (and 02 etc).
3. Run the Lync Optimizer generated ruleset against your deployment.
4. Copy the -7Ext0x rule(s) to the ZZ-ExtensionList dialplan ensuring proper alphabetical order.
5. Run the AddExtensionsToDP.ps1 script to add all the extension rules in ZZ-ExtensionList to the new dialplan.
6. Run the UpdateNormRules.ps1 to update all other dial plans with the newly created extension normalization rule.
7. Repeat steps 2-6 as necessary.

Rinse and repeat for each new dial plan you want to add to your environment. This process along with the attached scripts should make managing large dial plans much simpler. As always, be cautious when running these scripts, as I can't be responsible for any issues this causes.  Always make a backup of your existing Lync EV environment before running these or any scripts provided by strange people who pretend they look like David Hasselhoff.

Automatically Creating Lync Sites and Subnets using AD Sites

To use call admission control or location-based routing in Lync, one of the first steps is to define all the sites and subnets used in the company and to assign the sites to network regions.  In larger companies, this can be a tedious process if this information is not stored in a central, easily accessible location.

One place where network site/subnet information can sometimes be centrally found is within Active Directory itself.  Active Directory and other applications such as Exchange uses site/subnet information to determine where to direct clients for authentication services (among other things).  Ideally, every site and subnet should be defined using the Active Directory Sites and Services snap-in.  If the Active Directory administrator has diligently defined all the sites/subnets, this information can be easily parsed and used in Lync.

I've created a short script that will automate the creation of network sites and subnets using the information stored in AD Sites and Services.  Again, this script is only useful if all the sites/subnets have been defined in AD.  Thanks to Greig Sheridan who fixed the script to deal with AD site names that have characters that are invalid for Lync site names (such as underscore).

Copy the below script into a .PS1 file on a Lync server and run it.  As always, be careful and test this in a lab first.  I can't be responsible for any damage this script will cause.
# Load Active Directory Powershell module if its not already loaded if(-not(Get-Module -name ActiveDirectory)){Import-Module ActiveDirectory} # If a Lync network region doesn't exist, create one and name it the same as the first Lync site # If one or more does exist, select the first one associated with the first Lync site if(-not(Get-CSNetworkRegion)) { $SiteName = (Get-CSSite 1).DisplayName New-CSNetworkRegion $SiteName -CentralSite 1 } else { $SiteID = (Get-CSSite 1).Identity $SiteName = (Get-CSNetworkRegion | Where {$_.CentralSite -eq $SiteID} | Select-Object -First 1).NetworkRegionID } # Get the ConfigurationNamingContext from AD and go to the Sites container $ConfigDN = (Get-ADRootDSE).ConfigurationNamingContext $SiteContainerDN = ("CN=Sites," + $ConfigDN) # Select all the site objects $Sites = Get-ADObject -SearchScope OneLevel -SearchBase $SiteContainerDN -filter {ObjectClass -eq 'Site'} foreach ($Site in $Sites) { # Create Lync network sites based on AD site $SiteObj = Get-ADObject $Site.distinguishedName -properties "siteObjectBL", "description" #Remove dashes from site names and replace with underscore $ADSiteName = $SiteObj.Name $ADSiteName = $ADSiteName -replace "-","_" New-CSNetworkSite -NetworkSiteID $ADSiteName -Description $SiteObj.Description -NetworkRegionID $SiteName # Add each defined AD site subnet to the Lync network site object foreach ($SubnetDN in $SiteObj.siteObjectBL) { $Subnet = Get-ADObject -Identity $SubnetDN -properties "description" $SubnetName = $Subnet.Name #Subnet name is in format 192.168.100.0/24 $SubnetSplit = $SubnetName.Split("/") #Need to split subnet name into IP and mask for command $SubnetID = $SubnetSplit[0] $SubnetMask = $SubnetSplit[1] New-CSNetworkSubnet -SubnetID $SubnetID -MaskBits $SubnetMask -Description $Subnet.Description -NetworkSiteID $ADSiteName } } #Create subnet object with 32-bit mask for all public edge A/V IP addresses $EdgeServerPools = Get-CSService -EdgeServer | Where {$_.SiteID -ne "Site:BackCompatSite"} foreach ($EdgeServerPool in $EdgeServerPools) { $EdgeServers = (Get-CSPool $EdgeServerPool.PoolFQDN).Computers foreach ($EdgeServer in $EdgeServers) { $NetworkInterfaces = Get-CSNetworkInterface -ComputerFQDN $EdgeServer | Where {$_.Interface -eq "External" -and $_.IPAddress -ne "0.0.0.0"} if (($NetworkInterfaces).Count -eq 3) { $NetworkInterface = $NetworkInterfaces | Where {$_.InterfaceNumber -eq 3} } else { $NetworkInterface = $NetworkInterfaces } if ($NetworkInterface.ConfiguredIPAddress -ne $NULL) { $IPAddress = $NetworkInterface.ConfiguredIPAddress } else { $IPAddress = $NetworkInterface.IPAddress } $Description = "A/V external edge IP for " + $EdgeServer New-CSNetworkSubnet -SubnetID $IPAddress.IPAddressToString -MaskBits 32 -Description $Description } }

The script will first check to see if a Lync network region already exists. If so, it will select the first one it sees as the base for all the subnets.  If one doesn't exist, it will create one using the first Lync site as the base. If need be, you can define additional network sites later.

The script will then create all the sites/subnets based on information from AD. All the sites will be assigned to the one network region selected according to the rules in the previous paragraph. Since there is no way for the script to know which site belongs to which region, this will have to be done manually if multiple Lync network regions exist.

IMPORTANT: The script will also create the required subnets for all the Lync edge A/V public IP addresses with a 32-bit subnet mask as per the requirements for call admission control.  You will have to manually assign these subnets to the sites where you deployed the edge servers.

While this script doesn't try to do everything associated with setting up call admission control, it does go a long way towards automating a good chunk of the process. A future revision of the Lync Optimizer will hopefully close the loop and help completely automate the work required to setup location-based routing.

As a final side note, one thing I don't understand is why Lync can't leverage AD Sites & Services for this information directly.  Exchange versions since 2007 have used AD Sites & Services for mail routing, so its not like it hasn't been done before.  Maybe a future version of Lync will add this functionality.  Until then, this script should help those who have well-defined site/subnet information in Active Directory.

Migrating Exchange 2007 UC Custom Prompts to 2010 SP1

If you've migrated from Exchange 2007 to Exchange 2010, you may have experienced some frustration at Microsoft's documentation on how to migrate UC custom prompts (customized audio files for UM autoattendants etc) to Exchange 2010.  A bit of background:  When you recorded a custom greeting for an Exchange 2007 autoattendant or dialplan, the audio file got copied to a fileshare known as the prompt publishing point, a folder that stored all the audio files for a given dialplan.  The Exchange File Distribution Service would copy the audio files to other UM servers in the dialplan.  In Exchange 2010, the audio files are now stored in a system mailbox.  Since there is no automatic synchronization of custom prompts between Exchange 2007 and 2010, some migration process is necessary to copy the 2007 custom prompts to 2010.

A Technet article helpfully explained how to migrate individual custom prompts, but the process was far too tedious to migrate any more than a handful.  I created a simple Powershell script that did the job fairly nicely for a client with several hundred custom prompts, but it still involved multiple steps and had to be run directly on each UM publishing point.  I was going to publish the details here, but I don't need to do that now.

Thankfully, with the release of Exchange 2010 SP1, there is now a very simple way to migrate ALL your custom prompts with a single Powershell script provided by Microsoft: MigrateUMCustomPrompts.ps1.  You can find the script in the Scripts folder on any Exchange 2010 SP1 server installation (or even on the DVD).  Just run the script without any switches and it will copy all custom prompts from all publishing points into Exchange 2010.

A VERY helpful script that hasn't seen much publicity yet.