Upgrading to VCS/Expressway X8.8 and Jabber MRA Broke? Here’s why…

VCS/Expressway X8.8 changes it’s behavior versus prior versions.  8.8 does a reverse lookup of the IP addresses it’s communicating with to make sure it matches the hostname between C and E.

From the Release notes:

DNS entries: Do you have forward and reverse DNS lookups for all infrastructure
systems that the Expressway interacts with? If the Expressway cannot resolve hostnames and IP addresses of systems,your complex deployments (eg.MRA) could stop working as expected after you upgrade.

Oddly enough, the two systems that I’ve been involved in the upgrade to 8.8.1 with, both had the Unified Communications traversal zone with show Active, and hard phones (8800 and DX 650) will register and work properly, but Jabber clients will be unable to login and Jabber will throw an error when trying to login through MRA:

"Unable to Communicate with Server."

Running the debugging logs on Expressway-C you see the following error: 

"Certificate verification failed for host=x.x.x.x, additional info: 
Invalid Hostname expressway-e.domain.com"

The fix is to make sure that Expressway-C can do a reverse DNS lookup on the IP address of Expressway-E. Then flush the DNS cache of C to make sure it re-queries DNS properly.

The debugging log will give you the address and hostname it is trying to do the lookup on.

In a dual-NIC Expressway-E deployment the PTR recrod should point to the private IP address that C talks to.  In a single-NIC NAT hairpin deployment, I’ve seen it talk on the private and public IP.  So check that debug log.

 

 

 

Advertisements

Adventures in Upgrading to CSR 11.0

Now that all of the core CSR 11 components have had a service release under their collective belts, it’s go-time.  I helped a customer upgrade CUCM, IM&P and CUC from 10.5 to release 11.0.

  • CUCM/CUC 11.0(1a)SU1
  • CUCM IM&P 11.0(1)SU1
  • Jabber for Mac and Windows 11.5
  • Latest DX-series/8800 firmware
  • Expressway C/E X8.7.1
  • CWMS 2.6 MR1 Patch 1
  • Security COP to address CSCuy07473 for CUCM 11.0(1)
  • Permanent Licensing Surprises
  • Holding UCCX at 10.6 for now…. (Agent/Supervisor issues)

 

CUCM 11.0(1a)SU1

This is a four node system (pub and 3 subs) running the latest 10.5 SU.  Upgraded the pub during evening hours and told it not to reboot the night before.  Once it had completed the upgrade, I ran the upgrade on the three subs and told them not to reboot.  The maintenance window was the next evening, so we didn’t make any changes during that window.

When trying to reboot the pub to the new version from the GUI it got into an ugly loop.  Switch version reported that an upgrade was still in progress.  Went to the upgrade menu option and it indicated that I had to assume control over the upgrade.  Did so and the log file showed that the upgrade had completed successfully and that the lock files were released.  Went back to switch version and it still indicated that it was in an upgrade…

Bailed on the GUI and issued the version switch from the CLI.  It didn’t complain at all and did the version switch from 10.5 to 11.  It was a faster process than I imagined, taking less than 10 minutes to reboot on 11.  I was a bit concerned if it was going to work given the GUI seeming to be in a loop, but it rebooted just fine.

I rebooted the subs all from the CLI since I didn’t (and now perhaps don’t trust) the GUI switch version.  They all rebooted quickly too and were up and running on v11.0

Important!  Please note that the OVA release notes indicate that the RAM should be upgraded to 6GB for the CUCM and CUC VMs – http://www.cisco.com/web/software/283088407/126036/cucm-11.0.ova.readme.txt.  This was pointed out by a kind reader, which I hadn’t noticed until my CUC was falling apart after the upgrade.  Moving this VM to 6GB was an immediate fix.

CUC 11.0(1a)SU1

This was a textbook upgrade that evening.  I’d prestaged 11.0 like CUCM and the reboot took about 15 minutes.  All was well until I applied the permanent licensing.  Which I’ll cover later

IM&P 11.0(1)SU1

I wasn’t able to pre-stage this upgrade so ran it the evening of the maintenance window.  It’s a fairly small system so it took about an hour to upgrade and reboot.  The reboot seemed excessively long and I was worried, but it came back and workstation Jabber  clients automatically connected.

In conjunction with this upgrade I updated my jabber-update.xml file and push out the latest 11.5(2) version of the Mac and Windows clients.  We also updated all of the user photos on the webserver that houses them to current pictures.  Jabber was hit and miss about actually pulling the new picture.  It seemed that you had to manually view the profile on some users to get it to pull the new picture.

Latest DX/8800 series Firmware

The DX-series firmware has been a bumpy bumpy bumpy road.  It’s finally pretty stable as of 10.2(5)154.  A newer 10.2(5)195 is out so I pushed that out as it has a number of bugfixes.  I also updated the photo location for the DX-series phones and they all now pull the photos correctly from the webserver that houses them.  The super secret URL to put in the Company Photo Directory is this:  http://<webserver ip address>/%%uid%%.jpg

I migrated the DX-es from Anyconnect VPN over to MRA through Expressway that night since this latest ASA Sev 10 Bugfix upgrade has caused an odd cert issue for the DX (not not normal Anyconnect software clients on other platforms).  Remote phone control does work properly from Jabber (that is VPNed in) to the phone that is connected via MRA.

CUCM 11.0 default firmware also had older firmware for the 8800 series phones so I pushed the latest 11.0 version and am anxiously awaiting 11.5 for some really cool upcoming features for the 8800 series.

Expressway C/E X8.7.1

Textbook upgrade.  I love the software that came from Tandberg.

CWMS 2.6MR1 Patch 1

This is still my favorite app to upgrade by miles.  Attach the ISO to the Admin VM in vCenter and press go from the GUI.  An hour or so later after a couple reboots of all the various VMs (Admin, Media, IRP) you kick it back out of maintenance mode and you’re done.

Security COP to address CSCuy07473 for CUCM 11.0(1)

This patch JUST released with the latest security fixes for CiscoSSL (a ciscoized variant of OpenSSL).  Install on each CUCM node and you’re done.  No reboot required.

Permanent Licensing

After upgrading everything to 11.0 everything kicked into 60-day temp license mode as expected.  (Upgrading to CSR 10.5 was bad news when it didn’t do what it was supposed to and CCX ate all of it’s licenses resulting in a P1 case.)

The TAC case for licensing was pretty straightforward.  Had permanent licenses in about a day after providing the contract number that showed SWSS.

I held of installing the permanent licenses until after hours in the event that something would go wrong and take the system down (still nervous after the CCX incident).  Installation went fine with one side issue.

I had complaints about SpeechConnect / voice enabled directory handlers on Unity Connection not working right.  Turns out CUC didn’t like the permanent licenses as far as SpeechConnect.  It had pulled the licenses from ELM/PLM properly and was in compliance, but it took a restart of the Conversation service for it to start doing the voice recognition stuff again.  Rather odd.

Holding at CCX 10.6

Since 10.6 is the last version of CCX to support CAD/CSD and Finesse, I’m working to migrate the contact center over to Finesse.  There are some usability complaints we’re working through.  The users love the idea of a dedicated app that pops when a call comes in as well as the agent-to-agent chat inside CAD.  Getting them to use a web-browser for Finesse has been a challenge.   Once I have those details ironed out we’ll force them into Finesse when we upgrade to CSR 11.5 in the summer.

 

 

 

Deploying Multistream Conferencing with vTS and CUCM

With the release of CE8 code for Cisco video endpoints (like the SX10 (8.1), SX20 (also MX200-G2 and MX300-G2) and SX80-based endpoints like the SX80, MX700 and MX800), and the appropriate infrastructure components, multistream video is a possibility.  Multistream video allows an endpoint to send multiple resolution video streams and have the bridge pass the most appropriate streams to the far-end video units.  The far end video unit would receive a full resolution stream of the active speakers, and then low quality streams of the other participants.  The most useful feature of multistream video is the ability to use both screens of a dual-screen video unit to see remote participants (when doing single-stream transcoded mode, you can only do single screen video, and secondary screen content.)  Multistream also allows for ActiveControl layout, which allows the endpoint to choose the video layout vs. the video bridge determining the layout of the participants (which has rudimentary DTMF layout control).

Components used in my lab configuration:

  • SX10 (CE8.1)
  • MX800 (CE8.0.1)
  • (2) 8845 video phones (used to inject more video streams — these endpoints do not support multistream, they do single stream and receive their layout from the bridge)
  • Conductor XC4.1
  • vTS 4.2(4.23)
  • CUCM 11.0(1)21900-11 (Latest and greatest version is a requirement) -or- VCS X8.7.1

This guide assumes you’ve already setup a Rendezvous (aka MeetMe) number/URI that is routed to Conductor/vTS and you’re able to to normal conference calls.  We’ll modify settings to enable multistream.

Guide to configure endpoints and CUCM SIP Profile – http://www.cisco.com/c/dam/en/us/td/docs/telepresence/infrastructure/solutions/cmrpremises/cmr-premises-deployment-guide-r6-0.pdf

The relevant portion of this configuration is to make sure your SIP trunk to conductor is in a Location that supports full quality video.  I sent the inter-region bandwidth to UNLIMITED in my test system.  Cisco recommends a minimum of 1mpbs per screen, otherwise the vTS bridge may kick that video unit down to single-stream transcoded mode.

Configure the endpoint to support multistream

In CUCM the setting is in the device specific settings, Multistream Mode needs to be set to Auto.  Despite some of the documentation reading otherwise, Auto will attempt to do multistream, there is not actually an On setting.

Configure CUCM

Configure the SIP Profile used by the SIP trunk to Conductor to include the following settings:

  • Allow iX Application Media and Allow multiple codecs in answer SDP are checked on.
  • SDP Transparency Profile is set to Pass all unknown SDP attributes

In System > Service Parameters > Call Manager Service > click advanced > set SIP Maximum Incoming Message Size to 18000.

Configure Conductor

On the Conductor server, under the Conference Template you’re using for your conference, select advanced template parameters and add:

  • Enable iX protocol – True and the box checked
  • Multiscreen layout – ActivePresence and the box checked

No settings on vTS need to be changed, it will automatically do multistream if the endpoints meet the requirements, and CUCM (or VCS) and Conductor are properly configured.

When you join with a multi-stream endpoint you will see the following on vTS Conferences page:

 

vts1

You’ll notice the endpoints that support multistream show Multistream, and the 8845 phone named “Mike White” is Standard because it only supports a single stream.

If we look at the statistics for 5580 (the SX80) you’ll see multiple video streams being sent and received:

vts2

 

Lastly if we look at the call statistics from the video endpoint itself, we see the same information:

endpoint1

 

The touchpanel now shows more details in the layout.  You can see each participant in the conference and the active speaker.

IMG_5313

 

While you can select from several canned layout modes (same typical layouts are you’re used to), this version doesn’t yet support complete drag and drop layout of individual participants where you want them.  If you select a particular participant, you can see information about any of the participants and boot them if you are meeting organizer:

IMG_5314

 

Overall its very cool, and sets the groundwork for much more flexibility in the future with layout control.

 

IMG_5315

Expressway 8.5.2+ MRA Issues

I’ve run into a couple challenges after an upgrade to Expressway/VCS 8.5.2 where MRA for phones quit working.

I found a bug that broke MRA in 8.5.2 (the recommendation has been to downgrade back to 8.5.1).  That bug is shows that it is now fixed in 8.6, so I upgraded to 8.6.1 recently.  MRA started working again, but only on one out of every three login attempts.  It was really weird.  In looking at the logs it showed a bunch of errors:

Home CUCM not available – Unknown CUCM cluster for node sub02

Home CUCM not available – Unknown CUCM cluster for node sub03

The deployment I was working on is a three node (pub and two subs), running split DNS (different internal domain than the external domain name).

After a lot of digging it turns out that there is a change in the way MRA handles CUCM lookup.  When I installed the system added my pub and subs to Expressway-C by IP address.  But it looks like Expressway now attempts to communicate with them via hostname, and not IP as they were defined by me.

Since Expressway is using the domain suffix assigned to MRA (extdomain.com), it is attempting to lookup sub02.extdomain.com and sub03.extdomain.com.  I didn’t have A records for these on my internal DNS server extdomain zone since I’ve never needed to resolve them by the external domain name.

Adding these two records fixed the login issue and it now logins on first attempt like it used to.

Collab Edge MRA for 7800/8800/DX Series Endpoints

Cisco recently posted Expressway (and VCS) X8.5.2 and 10.3.1 firmware for the 8800 and 7800 series phones.  The combination of these products allows these phones to register remotely to CUCM utilizing Collaboration Edge MRA.  (The DX-series (650/70/80) is expected to support MRA in the next release of code due out shortly.)

This functionality isn’t TAC supported yet, and has been released in a “feature preview” form.  I’ve set it up and tested it and it works well for the most part.  However, there is not full feature parity for a phone registered via MRA vs directly registered to CUCM, but for testing and basic calls, it works well.

In order to set it up, make sure your Expressway MRA deployment for Jabber is working properly.  MRA for the 7800/800 series phones uses the same service discovery process that Jabber uses, so if you have Jabber working, you’ll have 95% of the work done.

One important piece of information to know is that the phone firmware trusts 100+ public root CA certificates.  If your Expressway-E server does not have a certificate signed by one of these CA’s it’s not going to work for the phones.

Here’s the basic procedure I followed to make it work:

  1. Installed the 8800 10.3.1 COP file via OSadmin and restarted the TFTP service.
  2. Logged in to Jabber via MRA to ensure the correct functionality of my MRA system and my login credentials.
  3. Defined the phone in CUCM and then connected the phone directly to CUCM so that it would pull the version of firmware that supports MRA.  (My 8851 phone shipped with an older version of code that did not have MRA support.)
  4. Took the phone off of the corporate network to an internet-access only network.
  5. I had initial problems with the phone not attempting MRA lookup after being connected to the internet-only network, so I followed the troubleshooting process of resetting the Network settings on the phone.  It then started to try the MRA process.

Steps the phone follows in MRA registration:

1) Phone attempts normal TFTP registration/_cisco-uds._tcp.domain.com lookup process:

IMG_1456

This fails because the phone has no direct access to CUCM.

2) Firmware now prompts for MRA credentials (These would be the same credentials you use for Jabber MRA login — in my case it is set to use LDAP/AD for authentication):

IMG_1458

Phone now attemps  _collab-edge._tls.domain.com service record lookup (like Jabber does) to discover the Expressway-E/VCS-E host.

3) Phone completes MRA login process

IMG_1459

The phone is now registered and usable.

I’ve read conflicting information about the number of calls supported, and number of lines supported via MRA.  In my experience I have two lines on the phone registered and am able to make two calls per line.  (I’ve not tested more than two calls per line.)  The list of features that may work or not is extensive, so be careful as things like Barge or Intercom may not work yet.

The phone also upgraded code via MRA successfully which is good to know.

IMG_1454

I’ve noticed some oddities with on-hook vs. off-hook dialing.  I know there are some limitations around KPML currently.  In my experience it seems to off-hook dial fine on the primary line, but on a secondary line or when attempting a second call on the primary line you MUST on-hook dial.

Phone registration isn’t supported via TAC yet so feel free to post here and we can collectively attempt to assist.  Remember the most basic step to troubleshoot is to see if your Jabber can successfully login.

Collaboration Edge (MRA) with Split DNS Domains

I recently helped one of our customers work through a split DNS domain Collab Edge MRA deployment that was quite educational.  Here are a few tidbits that I picked up.

Scenario:

Customer’s external (internet facing) DNS domain is:  collabdomain.com
Customer’s internal domain is:  internaldomain.local  (not resolvable externally, only served by the internal DNS server)

Expressway-E lives on the internet so it’s hostname would be vcse.collabdomain.com

Expressway-C, CUCM, IM&P (CUP) and Unity Connection live on the internal domain, so they would be called ucm1.internaldomain.local, cup1.internaldomain.local, ucxn.internaldomain.local.

The users accounts would live in internaldomain.local (presumably the LDAP user would be user@internaldomain.local, etc.)

The external DNS server would serve the public collabdomain.com zone.

The internal DNS server would serve internaldomain.local, as well as an internal version of the collabdomain.com zone.

The challenge with these separate domain names is that when a user goes to login to Jabber via MRA, the service discovery process is going to fail because they are going to type in their IM address as user@internaldomain.local, and there is no _collab-edge._tls.internaldomain.local SRV record availble on the itnernet (nor could there be).

To get around this, the jabber-config.xml file can be modified to include the external domain name that Jabber should try to do service discovery against. 

<Policies>
<VoiceServicesDomain>collabdomain.com</VoiceServicesDomain>
</Policies>

Jabber must first login internally (directly on your corporate network) to pull down the jabber-config.xml that includes this policy.

With this configuration now loaded in Jabber, when the user attempts to login from outside via MRA they will still use their user@internaldomain.local credentials, but Jabber will use the specified VoiceServicesDomain — in this case — collabdomain.com.  It will do the SRV record lookup on _collab-edge._tls.collabdomain.com and now be able to resolve Expressway-E and login.

In addition to the the normal _cisco-uds._tcp.internaldomain.local SRV record, Jabber needs another _cisco-uds._tcp.collabdomain.com SRV record (on the internal DNS server) pointing to ucm1.internaldomain.local for MRA to work with the split domains.The one criticism to this method is requiring the Jabber client to login internally first, which isn’t an ideal scenario for large distributed organizations.  One method that is supposed to work around this is to prime Jabber by installing it generically and launching it from an URL:ciscojabber://provision?ServicesDomain=internaldomain.local&VoiceServicesDomain=collabdomain.com

This could be provided in an email and will supposedly launch Jabber so that it tries the external domain without having to ever login internally first.

 

 

How to get Expressway-C and E licenses

Expressway-E and Expressway-C are free for existing customers who own UCL-Enhanced, CUWL-STD, or CUWL-PRO licenses and have a current UCSS subscription and accompanying ESW contract.

Expressway is not available via PUT however.  Because there are multiple SKUs that are shipped, an $0 (“zero dollar”) order must be placed with your Cisco Reseller.

Attached is a sample BOM to order Expressway-E and Expressway-C version 8.1 – Top level SKU is R-UCL-UCM-LIC-K9

Please see the comments below about PAK fulfillment and registration.  One of the PAKs includes licenses for both Expressway-C and Expressway-E and you must resiger them separately one to each serial number.

 

1.0 R-UCL-UCM-LIC-K9 Top Level SKU For 9.x/10.x User License – eDelivery N/A 21 days No 1 0.00 0.00 0 0.00
1.0.1 CON-ESW-RUCLUCK9 ESSENTIAL SW Top Level SKU For 9. 12 month(s) N/A No 1 0.00 0.00 0 0.00
1.1 EXPWY-VE-C-K9 Cisco Expressway-C Server Virtual Edition N/A 0 days No 1 0.00 0.00 0 0.00
1.1.0.1 CON-ESW-EXPWYVEC ESSENTIAL SW Cisco Expressway-C S 12 month(s) N/A No 1 0.00 0.00 0 0.00
1.2 EXPWY-VE-E-K9 Cisco Expressway-E Server Virtual Edition N/A 0 days No 1 0.00 0.00 0 0.00
1.2.0.1 CON-ESW-EXPWYVEE ESSENTIAL SW Cisco Expressway-E Server Virtual Editi 12 month(s) N/A No 1 0.00 0.00 0 0.00
1.3 SW-EXP-8.X-K9 Software Image for Expressway with Encryption Version X8 N/A 0 days No 1 0.00 0.00 0 0.00
1.4 PC-10X-STANDARD-K9 Prime Collaboration Standard 10.x N/A 21 days Yes 1 0.00 0.00 0 0.00
1.5 LIC-EXP-E-PAK Expressway Series Expressway-E PAK N/A 0 days Yes 1 0.00 0.00 0 0.00
1.6 LIC-EXP-GW Enable GW Feature (H323-SIP) N/A 0 days Yes 2 0.00 0.00 0 0.00
1.7 LIC-EXP-SERIES Enable Expressway Series Feature Set N/A 0 days Yes 2 0.00 0.00 0 0.00
1.8 LIC-SW-EXP-K9 License Key Software Encrypted N/A 0 days Yes 2 0.00 0.00 0 0.00
1.9 LIC-EXP-AN Enable Advanced Networking Option N/A 0 days Yes 1 0.00 0.00 0 0.00
1.10 LIC-EXP-E Enable Expressway-E Feature Set N/A 0 days Yes 1 0.00 0.00 0 0.00
1.11 LIC-EXP-TURN Enable TURN Relay Option N/A 0 days Yes 1 0.00 0.00 0 0.00
SubTotal 0.00
Configset Total 0.00