Upgrading to VCS/Expressway X8.8 and Jabber MRA Broke? Here’s why…

VCS/Expressway X8.8 changes it’s behavior versus prior versions.  8.8 does a reverse lookup of the IP addresses it’s communicating with to make sure it matches the hostname between C and E.

From the Release notes:

DNS entries: Do you have forward and reverse DNS lookups for all infrastructure
systems that the Expressway interacts with? If the Expressway cannot resolve hostnames and IP addresses of systems,your complex deployments (eg.MRA) could stop working as expected after you upgrade.

Oddly enough, the two systems that I’ve been involved in the upgrade to 8.8.1 with, both had the Unified Communications traversal zone with show Active, and hard phones (8800 and DX 650) will register and work properly, but Jabber clients will be unable to login and Jabber will throw an error when trying to login through MRA:

"Unable to Communicate with Server."

Running the debugging logs on Expressway-C you see the following error: 

"Certificate verification failed for host=x.x.x.x, additional info: 
Invalid Hostname expressway-e.domain.com"

The fix is to make sure that Expressway-C can do a reverse DNS lookup on the IP address of Expressway-E. Then flush the DNS cache of C to make sure it re-queries DNS properly.

The debugging log will give you the address and hostname it is trying to do the lookup on.

In a dual-NIC Expressway-E deployment the PTR recrod should point to the private IP address that C talks to.  In a single-NIC NAT hairpin deployment, I’ve seen it talk on the private and public IP.  So check that debug log.




CE 8.2 code on the DX70 and DX80

***UPDATED***  The Devpack, CE8.2.1 and conversion cop are now available on CCO.

CE 8.2.0 firmware for the DX70 and DX80 posted to CCO today.  (But of course we’re still waiting for the Devpack with the QED that will be released in the next week or so.)  The release notes are here.  The Official Conversion Guide is here.  Any instructions in this guide would override what I’ve said.

I’ve been running the beta for the last few weeks and can absolutely say CE code on the DX is MUCH more responsive, stable and usable than the Android-based code.  (That said the last few builds of 10.2(5) android code have been pretty decent.)

CE 8.2 code is absolutely the code to move to if your primary use for the DX is for video calls.



Why move to CE on the DX?

  • Responsiveness.  No more lag!  It’s snappy.
  • Stability.  Seldom, I had the random crash and full-reboot during video calls with Android.  While it was pretty rare, it was super frustrating.
  • Video-centric user interface.  It run CE codec code now and feels like a Cisco codec (like SX10, 20, etc.) now.  It’s all about the video call.
  • Registration to VCS/Expressway
  • Far-end Camera Control
  • OBTP (one button to press) meeting launch and TMS management
  • Fully customizable wallpaper

Why stay on Android?

  • You need Android applications.  CE doesn’t run any apps, period.
  • You use the built-in Cisco Webex and Jabber Android apps on the DX.
  • You need a local web browser.  CE doesn’t have a web browser built in.
  • You are currently using Wi-Fi or Bluetooth.  The CE code doesn’t yet support Wi-Fi or BT.  (That’s coming in a follow-on release.)
  • You need telephony features on the phone like CFwdAll, Shared Lines, Voicemail button, Auto answer.

What happens to Android?

Android will be supported on the DX70 and Dx80 for the life of the product.  Keep running it if you need features that will always only be specific to Android (local web-browser, Android Apps).  The DX650 will remain Android-only.

Converting to CE Code using CUCM

Note: You have a couple options to convert to CE code.   CUCM of course or, as the conversion guide notes, there is a public TFTP server on the internet provided by Cisco to convert a DX using.  As far as CUCM, you can convert the DX either onnet or connected via MRA registration through Expressway.

  1. Upgrade to the latest build of Android code – 10.2(5)207 by installing the COP file on CUCM, restarting TFTP and rebooting your DXes.  (Either way you want to go to this code because of all of the bugfixes.)
  2. Install the latest (Early July 2016) Devpack:
    CUCM 11.0.1:  cmterm-devicepack-
    CUCM 10.5.2: cmterm-devicepack10.5.2.14076-1.cop.sgn
    CUCM 9.1.2: cmterm-devicepack9.1.2.16137-1.cop.sgn
  3. Install the Devpack on CUCM as well as cmterm-synergy-ce8_2_1_no_defaults.cop.sgn (or latest version) so that it gets the Telepresence DX70 and Telepresence DX80 device type QED installed.  Reboot your cluster.  (Unless you’re on CUCM 11.5 which doesn’t need the immediate reboot!!)  Or bug someone you know for the standalone CE 8.2 QED in the meantime.
  4. This devpack should have the CE 8.2 firmware, but if not install the CE 8.2 COP file – mterm-s52040ce8_2_1.k3.cop.sgn (or current); restart TFTP.
  5. In CUCM, change the Phone Load of the existing DX80 to the CE 8.2 phone load name specified in the conversion guide.  For 8.2.0 it is sipdx80.ce821.rel.loads
  6. The DX80 will take a few minutes (10-15) to upgrade to CE 8.2.1.
  7. Take note of the MAC address of the DX80 in the CUCM device, because you are about to DELETE the DX80 device!
  8. Delete the DX80 device from CUCM.
  9. Create a new Telepresence DX80 device in CUCM and paste in the MAC address of the DX80 you just deleted.  Set the appropriate device settings and add an extension/SIP URI to the device.
  10. On the DX80 itself, run through the startup wizard and pick UCM registration or UCM through Expressway (if your endpoint is registering through Expressway).
  11. You’ve now got a DX80 on CE 8.2 code
  12. Enabled Web Access in CUCM device settings so that you can get to the DX80 GUI.
  13. Login to the GUI and set the admin password.  (This step may not be needed, setting the admin username/password was not available in earlier CE betas via the CUCM device setting page.)


Other Notes

  • The Touch 10 doesn’t work on the DX80 or Dx70, you must use the built-in touch screen on the DX.

8800 Series 11.5(1) Firmware – Enhanced Line Mode

11.5 firmware for the 8800 series phones is on CCO now.

It comes with a bunch of cool new features.

Enhanced Line Mode

The coolest and most useful that I’ve seen requests for is the new Enhanced Line Mode.  (I’ll abbreviate it ELM even if there’s overlap with the PLM/ELM acronym.)  ELM allows all 10 buttons on the phones to be used a programmable line keys (PLKs).

The mode we’re used to includes 5 PLKs on the left, and 5 context-sensitive function keys on the right.  I like this mode, having gotten used to it back in the day with the 9900 series phones, but hear customers who need more than 5 PLKs (in particular for admin/receptionists who want more than 5 BLFs or Shared lines).



As you can see in the picture, I can now use all 10 buttons.

While the firmware is out now, there is a Devpack required to enable the ELM feature on the device configuration page.

The release notes indicate that you should get the latest Devpack from CCO, install it and reboot the cluster to enable the ELM setting.  The challenge you’ll hit is that the latest Devpack on CCO as of today (mid-June) doesn’t actually include the QED file that enables the ELM setting.  The Devpack that inclues the QED will release in the next couple weeks.

Look for a Devpack with a late-June/early-July date stamp if you want to turn this feature on.

Enhanced Do Not Disturb

The DND function has been updated to be much more obvious which is nice.


Other features to mention:

  • Wi-Fi Security Enhancements
  • Customized Dial Tone for SIP Phones

See the release notes for more information about the last two.



CUCM 11.5 Released – Warning!

CSR (CUCM/IM&P/CUC) 11.5 has been released!  It includes a lot of cool new features but there’s a warning that needs to be shared for customers who have legacy phones.

New Features of note:

CUCM 11.5

  • “Hitless” Device Pack Installation.  Add new device types without an immediate reboot.  TFTP restart required, and Publisher reboot at next maintenance window.
  • Read-only AXL user role.  (Finally!)
  • PIN Sync to Unity Connection
  • User-customizable  Display Name field searchable in the directory for users who want a nickname or short name.  (e.g. Mike White vs Michael White — both searchable)
  • Directory search for MRA clients/devices
  • UCM Scale Increase with same OVA spec
  • Single SAML SSO Agreement per Cluster
  • Next Gen Encryption (DOD Suite B)
  • VMWare ESXi 6.0 support, Single SAML IdP per Cluster


IM&P (Jabber Server) 11.5

  • Multi-device Messaging.  Finally copies of chats messages are sent to all clients, not the most recently active.  Read notifications sync’d across all devices.
  • Persistent Chat HA and support for MS SQL (in addition to currently-supported Postgres and Oracle) for Persistent Chat functions.

Unity Connection

  • Full video messaging.  In addition to video greetings, video messages can be left in inboxes.  Video playback is supported from the handset, and not Jabber at this time.  (This functionality requires Mediasence as the video engine.)
  • PIN Synchronization between CUC and CUCM
  • New Media Player replaces the old MediaMaster applet
  • Support for Exchange 2016/Outlook 2016, Windows 10, MS Edge Browser
  • Next Gen Encryption (DOD Suite B)
  • VMWare ESXi 6.0 support, Single SAML IdP per Cluster


The warning is located in the release notes.  Certain legacy (i.e. REALLY OLD 12+ year EoS models) phones WILL NOT WORK with CUCM 11.5.

Update: A Field Notice has been posted.

This is a drastic change from previous versions were the phones are no longer supported and may not work properly anymore rather than disabling these models.  For example, I have an old 12SP+ at the office and it would register to CUCM 10, but not really work right.  In the case of 11.5, IT WILL NOT WORK.

Realistically the only models that I still see customers with are the 7935 Conference Station, 7920 Wireless, and 7910.

If you have these endpoints, understand that most have been EoS for over 12 years, and that there is a very aggressive phone trade-in program to get 20-30% extra discount (on top of your normal discount) to go to the new 8800 series phones.

Deprecated Endpoints

As of Cisco Unified Communications Manager Firmware Release 11.5, the following phones are not supported:•    Cisco IP Phone 12 SP+ and related models
•    Cisco IP Phone 30 VIP and related models
•    Cisco Unified IP Phone 7902
•    Cisco Unified IP Phone 7905
•    Cisco Unified IP Phone 7910
•    Cisco Unified IP Phone 7910SW
•    Cisco Unified IP Phone 7912
•    Cisco Unified Wireless IP Phone 7920
•    Cisco Unified IP Conference Station 7935

If you use any of these phone models on an older release of Cisco Unified Communications Manager and you upgrade to Release 11.5, the phone will not work after the upgrade completes.

Here’s an example from my CUCM where I have a 7905 defined.  The phones stays there, but this giant warning appears at the top.


Adventures in Upgrading to CSR 11.0

Now that all of the core CSR 11 components have had a service release under their collective belts, it’s go-time.  I helped a customer upgrade CUCM, IM&P and CUC from 10.5 to release 11.0.

  • CUCM/CUC 11.0(1a)SU1
  • CUCM IM&P 11.0(1)SU1
  • Jabber for Mac and Windows 11.5
  • Latest DX-series/8800 firmware
  • Expressway C/E X8.7.1
  • CWMS 2.6 MR1 Patch 1
  • Security COP to address CSCuy07473 for CUCM 11.0(1)
  • Permanent Licensing Surprises
  • Holding UCCX at 10.6 for now…. (Agent/Supervisor issues)


CUCM 11.0(1a)SU1

This is a four node system (pub and 3 subs) running the latest 10.5 SU.  Upgraded the pub during evening hours and told it not to reboot the night before.  Once it had completed the upgrade, I ran the upgrade on the three subs and told them not to reboot.  The maintenance window was the next evening, so we didn’t make any changes during that window.

When trying to reboot the pub to the new version from the GUI it got into an ugly loop.  Switch version reported that an upgrade was still in progress.  Went to the upgrade menu option and it indicated that I had to assume control over the upgrade.  Did so and the log file showed that the upgrade had completed successfully and that the lock files were released.  Went back to switch version and it still indicated that it was in an upgrade…

Bailed on the GUI and issued the version switch from the CLI.  It didn’t complain at all and did the version switch from 10.5 to 11.  It was a faster process than I imagined, taking less than 10 minutes to reboot on 11.  I was a bit concerned if it was going to work given the GUI seeming to be in a loop, but it rebooted just fine.

I rebooted the subs all from the CLI since I didn’t (and now perhaps don’t trust) the GUI switch version.  They all rebooted quickly too and were up and running on v11.0

Important!  Please note that the OVA release notes indicate that the RAM should be upgraded to 6GB for the CUCM and CUC VMs – http://www.cisco.com/web/software/283088407/126036/cucm-11.0.ova.readme.txt.  This was pointed out by a kind reader, which I hadn’t noticed until my CUC was falling apart after the upgrade.  Moving this VM to 6GB was an immediate fix.

CUC 11.0(1a)SU1

This was a textbook upgrade that evening.  I’d prestaged 11.0 like CUCM and the reboot took about 15 minutes.  All was well until I applied the permanent licensing.  Which I’ll cover later

IM&P 11.0(1)SU1

I wasn’t able to pre-stage this upgrade so ran it the evening of the maintenance window.  It’s a fairly small system so it took about an hour to upgrade and reboot.  The reboot seemed excessively long and I was worried, but it came back and workstation Jabber  clients automatically connected.

In conjunction with this upgrade I updated my jabber-update.xml file and push out the latest 11.5(2) version of the Mac and Windows clients.  We also updated all of the user photos on the webserver that houses them to current pictures.  Jabber was hit and miss about actually pulling the new picture.  It seemed that you had to manually view the profile on some users to get it to pull the new picture.

Latest DX/8800 series Firmware

The DX-series firmware has been a bumpy bumpy bumpy road.  It’s finally pretty stable as of 10.2(5)154.  A newer 10.2(5)195 is out so I pushed that out as it has a number of bugfixes.  I also updated the photo location for the DX-series phones and they all now pull the photos correctly from the webserver that houses them.  The super secret URL to put in the Company Photo Directory is this:  http://<webserver ip address>/%%uid%%.jpg

I migrated the DX-es from Anyconnect VPN over to MRA through Expressway that night since this latest ASA Sev 10 Bugfix upgrade has caused an odd cert issue for the DX (not not normal Anyconnect software clients on other platforms).  Remote phone control does work properly from Jabber (that is VPNed in) to the phone that is connected via MRA.

CUCM 11.0 default firmware also had older firmware for the 8800 series phones so I pushed the latest 11.0 version and am anxiously awaiting 11.5 for some really cool upcoming features for the 8800 series.

Expressway C/E X8.7.1

Textbook upgrade.  I love the software that came from Tandberg.

CWMS 2.6MR1 Patch 1

This is still my favorite app to upgrade by miles.  Attach the ISO to the Admin VM in vCenter and press go from the GUI.  An hour or so later after a couple reboots of all the various VMs (Admin, Media, IRP) you kick it back out of maintenance mode and you’re done.

Security COP to address CSCuy07473 for CUCM 11.0(1)

This patch JUST released with the latest security fixes for CiscoSSL (a ciscoized variant of OpenSSL).  Install on each CUCM node and you’re done.  No reboot required.

Permanent Licensing

After upgrading everything to 11.0 everything kicked into 60-day temp license mode as expected.  (Upgrading to CSR 10.5 was bad news when it didn’t do what it was supposed to and CCX ate all of it’s licenses resulting in a P1 case.)

The TAC case for licensing was pretty straightforward.  Had permanent licenses in about a day after providing the contract number that showed SWSS.

I held of installing the permanent licenses until after hours in the event that something would go wrong and take the system down (still nervous after the CCX incident).  Installation went fine with one side issue.

I had complaints about SpeechConnect / voice enabled directory handlers on Unity Connection not working right.  Turns out CUC didn’t like the permanent licenses as far as SpeechConnect.  It had pulled the licenses from ELM/PLM properly and was in compliance, but it took a restart of the Conversation service for it to start doing the voice recognition stuff again.  Rather odd.

Holding at CCX 10.6

Since 10.6 is the last version of CCX to support CAD/CSD and Finesse, I’m working to migrate the contact center over to Finesse.  There are some usability complaints we’re working through.  The users love the idea of a dedicated app that pops when a call comes in as well as the agent-to-agent chat inside CAD.  Getting them to use a web-browser for Finesse has been a challenge.   Once I have those details ironed out we’ll force them into Finesse when we upgrade to CSR 11.5 in the summer.




Deploying Multistream Conferencing with vTS and CUCM

With the release of CE8 code for Cisco video endpoints (like the SX10 (8.1), SX20 (also MX200-G2 and MX300-G2) and SX80-based endpoints like the SX80, MX700 and MX800), and the appropriate infrastructure components, multistream video is a possibility.  Multistream video allows an endpoint to send multiple resolution video streams and have the bridge pass the most appropriate streams to the far-end video units.  The far end video unit would receive a full resolution stream of the active speakers, and then low quality streams of the other participants.  The most useful feature of multistream video is the ability to use both screens of a dual-screen video unit to see remote participants (when doing single-stream transcoded mode, you can only do single screen video, and secondary screen content.)  Multistream also allows for ActiveControl layout, which allows the endpoint to choose the video layout vs. the video bridge determining the layout of the participants (which has rudimentary DTMF layout control).

Components used in my lab configuration:

  • SX10 (CE8.1)
  • MX800 (CE8.0.1)
  • (2) 8845 video phones (used to inject more video streams — these endpoints do not support multistream, they do single stream and receive their layout from the bridge)
  • Conductor XC4.1
  • vTS 4.2(4.23)
  • CUCM 11.0(1)21900-11 (Latest and greatest version is a requirement) -or- VCS X8.7.1

This guide assumes you’ve already setup a Rendezvous (aka MeetMe) number/URI that is routed to Conductor/vTS and you’re able to to normal conference calls.  We’ll modify settings to enable multistream.

Guide to configure endpoints and CUCM SIP Profile – http://www.cisco.com/c/dam/en/us/td/docs/telepresence/infrastructure/solutions/cmrpremises/cmr-premises-deployment-guide-r6-0.pdf

The relevant portion of this configuration is to make sure your SIP trunk to conductor is in a Location that supports full quality video.  I sent the inter-region bandwidth to UNLIMITED in my test system.  Cisco recommends a minimum of 1mpbs per screen, otherwise the vTS bridge may kick that video unit down to single-stream transcoded mode.

Configure the endpoint to support multistream

In CUCM the setting is in the device specific settings, Multistream Mode needs to be set to Auto.  Despite some of the documentation reading otherwise, Auto will attempt to do multistream, there is not actually an On setting.

Configure CUCM

Configure the SIP Profile used by the SIP trunk to Conductor to include the following settings:

  • Allow iX Application Media and Allow multiple codecs in answer SDP are checked on.
  • SDP Transparency Profile is set to Pass all unknown SDP attributes

In System > Service Parameters > Call Manager Service > click advanced > set SIP Maximum Incoming Message Size to 18000.

Configure Conductor

On the Conductor server, under the Conference Template you’re using for your conference, select advanced template parameters and add:

  • Enable iX protocol – True and the box checked
  • Multiscreen layout – ActivePresence and the box checked

No settings on vTS need to be changed, it will automatically do multistream if the endpoints meet the requirements, and CUCM (or VCS) and Conductor are properly configured.

When you join with a multi-stream endpoint you will see the following on vTS Conferences page:



You’ll notice the endpoints that support multistream show Multistream, and the 8845 phone named “Mike White” is Standard because it only supports a single stream.

If we look at the statistics for 5580 (the SX80) you’ll see multiple video streams being sent and received:



Lastly if we look at the call statistics from the video endpoint itself, we see the same information:



The touchpanel now shows more details in the layout.  You can see each participant in the conference and the active speaker.



While you can select from several canned layout modes (same typical layouts are you’re used to), this version doesn’t yet support complete drag and drop layout of individual participants where you want them.  If you select a particular participant, you can see information about any of the participants and boot them if you are meeting organizer:



Overall its very cool, and sets the groundwork for much more flexibility in the future with layout control.



8800 Handset Firmware 11.0(1)

One common criticism of the 8800 10.x firmware was it’s behavior in truncating Line Text Labels when there was ample space on the display to show more characters.  Version 11.0(1) fixes this:




(Sorry for the mediocre photo.)

The Line Text Label supporting 30 characters (the maximum supported in the LTL field in CUCM) on a single line on the display.  There’s some cool stuff coming in the next release of code which will add even more functionality to the 8800 series.

Firmware 11.0(1) adds the following features:


The highlights being the ability to use the BIB for Barge, and official support for MRA.