Adventures in Upgrading to CSR 11.0

Now that all of the core CSR 11 components have had a service release under their collective belts, it’s go-time.  I helped a customer upgrade CUCM, IM&P and CUC from 10.5 to release 11.0.

  • CUCM/CUC 11.0(1a)SU1
  • CUCM IM&P 11.0(1)SU1
  • Jabber for Mac and Windows 11.5
  • Latest DX-series/8800 firmware
  • Expressway C/E X8.7.1
  • CWMS 2.6 MR1 Patch 1
  • Security COP to address CSCuy07473 for CUCM 11.0(1)
  • Permanent Licensing Surprises
  • Holding UCCX at 10.6 for now…. (Agent/Supervisor issues)

 

CUCM 11.0(1a)SU1

This is a four node system (pub and 3 subs) running the latest 10.5 SU.  Upgraded the pub during evening hours and told it not to reboot the night before.  Once it had completed the upgrade, I ran the upgrade on the three subs and told them not to reboot.  The maintenance window was the next evening, so we didn’t make any changes during that window.

When trying to reboot the pub to the new version from the GUI it got into an ugly loop.  Switch version reported that an upgrade was still in progress.  Went to the upgrade menu option and it indicated that I had to assume control over the upgrade.  Did so and the log file showed that the upgrade had completed successfully and that the lock files were released.  Went back to switch version and it still indicated that it was in an upgrade…

Bailed on the GUI and issued the version switch from the CLI.  It didn’t complain at all and did the version switch from 10.5 to 11.  It was a faster process than I imagined, taking less than 10 minutes to reboot on 11.  I was a bit concerned if it was going to work given the GUI seeming to be in a loop, but it rebooted just fine.

I rebooted the subs all from the CLI since I didn’t (and now perhaps don’t trust) the GUI switch version.  They all rebooted quickly too and were up and running on v11.0

Important!  Please note that the OVA release notes indicate that the RAM should be upgraded to 6GB for the CUCM and CUC VMs – http://www.cisco.com/web/software/283088407/126036/cucm-11.0.ova.readme.txt.  This was pointed out by a kind reader, which I hadn’t noticed until my CUC was falling apart after the upgrade.  Moving this VM to 6GB was an immediate fix.

CUC 11.0(1a)SU1

This was a textbook upgrade that evening.  I’d prestaged 11.0 like CUCM and the reboot took about 15 minutes.  All was well until I applied the permanent licensing.  Which I’ll cover later

IM&P 11.0(1)SU1

I wasn’t able to pre-stage this upgrade so ran it the evening of the maintenance window.  It’s a fairly small system so it took about an hour to upgrade and reboot.  The reboot seemed excessively long and I was worried, but it came back and workstation Jabber  clients automatically connected.

In conjunction with this upgrade I updated my jabber-update.xml file and push out the latest 11.5(2) version of the Mac and Windows clients.  We also updated all of the user photos on the webserver that houses them to current pictures.  Jabber was hit and miss about actually pulling the new picture.  It seemed that you had to manually view the profile on some users to get it to pull the new picture.

Latest DX/8800 series Firmware

The DX-series firmware has been a bumpy bumpy bumpy road.  It’s finally pretty stable as of 10.2(5)154.  A newer 10.2(5)195 is out so I pushed that out as it has a number of bugfixes.  I also updated the photo location for the DX-series phones and they all now pull the photos correctly from the webserver that houses them.  The super secret URL to put in the Company Photo Directory is this:  http://<webserver ip address>/%%uid%%.jpg

I migrated the DX-es from Anyconnect VPN over to MRA through Expressway that night since this latest ASA Sev 10 Bugfix upgrade has caused an odd cert issue for the DX (not not normal Anyconnect software clients on other platforms).  Remote phone control does work properly from Jabber (that is VPNed in) to the phone that is connected via MRA.

CUCM 11.0 default firmware also had older firmware for the 8800 series phones so I pushed the latest 11.0 version and am anxiously awaiting 11.5 for some really cool upcoming features for the 8800 series.

Expressway C/E X8.7.1

Textbook upgrade.  I love the software that came from Tandberg.

CWMS 2.6MR1 Patch 1

This is still my favorite app to upgrade by miles.  Attach the ISO to the Admin VM in vCenter and press go from the GUI.  An hour or so later after a couple reboots of all the various VMs (Admin, Media, IRP) you kick it back out of maintenance mode and you’re done.

Security COP to address CSCuy07473 for CUCM 11.0(1)

This patch JUST released with the latest security fixes for CiscoSSL (a ciscoized variant of OpenSSL).  Install on each CUCM node and you’re done.  No reboot required.

Permanent Licensing

After upgrading everything to 11.0 everything kicked into 60-day temp license mode as expected.  (Upgrading to CSR 10.5 was bad news when it didn’t do what it was supposed to and CCX ate all of it’s licenses resulting in a P1 case.)

The TAC case for licensing was pretty straightforward.  Had permanent licenses in about a day after providing the contract number that showed SWSS.

I held of installing the permanent licenses until after hours in the event that something would go wrong and take the system down (still nervous after the CCX incident).  Installation went fine with one side issue.

I had complaints about SpeechConnect / voice enabled directory handlers on Unity Connection not working right.  Turns out CUC didn’t like the permanent licenses as far as SpeechConnect.  It had pulled the licenses from ELM/PLM properly and was in compliance, but it took a restart of the Conversation service for it to start doing the voice recognition stuff again.  Rather odd.

Holding at CCX 10.6

Since 10.6 is the last version of CCX to support CAD/CSD and Finesse, I’m working to migrate the contact center over to Finesse.  There are some usability complaints we’re working through.  The users love the idea of a dedicated app that pops when a call comes in as well as the agent-to-agent chat inside CAD.  Getting them to use a web-browser for Finesse has been a challenge.   Once I have those details ironed out we’ll force them into Finesse when we upgrade to CSR 11.5 in the summer.

 

 

 

Advertisements

Deploying Multistream Conferencing with vTS and CUCM

With the release of CE8 code for Cisco video endpoints (like the SX10 (8.1), SX20 (also MX200-G2 and MX300-G2) and SX80-based endpoints like the SX80, MX700 and MX800), and the appropriate infrastructure components, multistream video is a possibility.  Multistream video allows an endpoint to send multiple resolution video streams and have the bridge pass the most appropriate streams to the far-end video units.  The far end video unit would receive a full resolution stream of the active speakers, and then low quality streams of the other participants.  The most useful feature of multistream video is the ability to use both screens of a dual-screen video unit to see remote participants (when doing single-stream transcoded mode, you can only do single screen video, and secondary screen content.)  Multistream also allows for ActiveControl layout, which allows the endpoint to choose the video layout vs. the video bridge determining the layout of the participants (which has rudimentary DTMF layout control).

Components used in my lab configuration:

  • SX10 (CE8.1)
  • MX800 (CE8.0.1)
  • (2) 8845 video phones (used to inject more video streams — these endpoints do not support multistream, they do single stream and receive their layout from the bridge)
  • Conductor XC4.1
  • vTS 4.2(4.23)
  • CUCM 11.0(1)21900-11 (Latest and greatest version is a requirement) -or- VCS X8.7.1

This guide assumes you’ve already setup a Rendezvous (aka MeetMe) number/URI that is routed to Conductor/vTS and you’re able to to normal conference calls.  We’ll modify settings to enable multistream.

Guide to configure endpoints and CUCM SIP Profile – http://www.cisco.com/c/dam/en/us/td/docs/telepresence/infrastructure/solutions/cmrpremises/cmr-premises-deployment-guide-r6-0.pdf

The relevant portion of this configuration is to make sure your SIP trunk to conductor is in a Location that supports full quality video.  I sent the inter-region bandwidth to UNLIMITED in my test system.  Cisco recommends a minimum of 1mpbs per screen, otherwise the vTS bridge may kick that video unit down to single-stream transcoded mode.

Configure the endpoint to support multistream

In CUCM the setting is in the device specific settings, Multistream Mode needs to be set to Auto.  Despite some of the documentation reading otherwise, Auto will attempt to do multistream, there is not actually an On setting.

Configure CUCM

Configure the SIP Profile used by the SIP trunk to Conductor to include the following settings:

  • Allow iX Application Media and Allow multiple codecs in answer SDP are checked on.
  • SDP Transparency Profile is set to Pass all unknown SDP attributes

In System > Service Parameters > Call Manager Service > click advanced > set SIP Maximum Incoming Message Size to 18000.

Configure Conductor

On the Conductor server, under the Conference Template you’re using for your conference, select advanced template parameters and add:

  • Enable iX protocol – True and the box checked
  • Multiscreen layout – ActivePresence and the box checked

No settings on vTS need to be changed, it will automatically do multistream if the endpoints meet the requirements, and CUCM (or VCS) and Conductor are properly configured.

When you join with a multi-stream endpoint you will see the following on vTS Conferences page:

 

vts1

You’ll notice the endpoints that support multistream show Multistream, and the 8845 phone named “Mike White” is Standard because it only supports a single stream.

If we look at the statistics for 5580 (the SX80) you’ll see multiple video streams being sent and received:

vts2

 

Lastly if we look at the call statistics from the video endpoint itself, we see the same information:

endpoint1

 

The touchpanel now shows more details in the layout.  You can see each participant in the conference and the active speaker.

IMG_5313

 

While you can select from several canned layout modes (same typical layouts are you’re used to), this version doesn’t yet support complete drag and drop layout of individual participants where you want them.  If you select a particular participant, you can see information about any of the participants and boot them if you are meeting organizer:

IMG_5314

 

Overall its very cool, and sets the groundwork for much more flexibility in the future with layout control.

 

IMG_5315

8800 Handset Firmware 11.0(1)

One common criticism of the 8800 10.x firmware was it’s behavior in truncating Line Text Labels when there was ample space on the display to show more characters.  Version 11.0(1) fixes this:

 

IMG_5212

 

(Sorry for the mediocre photo.)

The Line Text Label supporting 30 characters (the maximum supported in the LTL field in CUCM) on a single line on the display.  There’s some cool stuff coming in the next release of code which will add even more functionality to the 8800 series.

Firmware 11.0(1) adds the following features:

 

The highlights being the ability to use the BIB for Barge, and official support for MRA.

 

DX 10.2(5) Firmware Released

Are you about ready to throw your DX out the window because dialing with the Phone app is an unresponsive nightmare as it tries to do a lookup on every call it’s ever made?  🙂

Well the fix is out.  The Phone app is revamped and actually doesn’t suck.

I’ve been running the 10.2(5) beta firmware for a few weeks and have been much happier with my DX80 and 650.  The final build is now out.

Would you like the DX to stay on your PC input when a call comes in?  It finally does!

Would you like HDMI audio from your PC to actually come out the DX speakers?  You got it!

Release notes are here.

One thing to note for MRA users.  The DX only trusts public CA certs like the 8800 series now.  So if you’ve deployed your Expressway-E with private certs, you’ll bust MRA on the DX if you dont move to public certs.

Catalyst 3850 IOS 16.1.1

Upgraded my 3850 to Denali 16.1.1 today. The new unified web GUI for the whole switch is pretty slick.  There are a ton of new features. See the Release notes here.  Now this is the first release of 16 so I don’t know I’d jump on it for a production network until there is a bug fix rebuild, or at least without checking the open caveats.

I ran into a problem that the APs would register, the radios just wouldn’t come up. Nothing I tried in the GUI (like bouncing the radios between disabled and enabled) would get them to go Operationally UP.

Turns out the secret was to go into the guy and do a “no ap dot11 5ghz shutdown” and a “no ap dot11 24ghz s.”  The radios came right up and everything is working again.

Registering an SX-10 (and TC-based endpoints) through Collab Edge MRA

Recently I worked on an MRA deployment using SX-10, MX-300 and DX-series (650/70/80) endpoints.

I had Expressway-C and E working successfully for 8800 series MRA, but needed to get the TC-based and DX-based endpoints to register.  This turned out to to involve some issues that I wasn’t expecting.

TC-based endpoint registration

There isn’t a lot of documentation  for TC endpoint registration through MRA (since traditionally it’s been registered to VCS through VCS-E).  The best documentation that I could find was here:  http://www.cisco.com/c/en/us/support/docs/unified-communications/expressway/118696-config-cucm-00.html

I began the deployment by registering the MX-300 I use directly to CUCM to make sure I had it successfully working before attempting registration through MRA.  I’d previously had it registered to a VCS-C.

MRA requires TC 7.3 code, so I decided to deploy 7.3.4 since it is the latest bugfix version.  I downloaded and deployed the COP file to CUCM since CUCM will be in control of what version of software the TC endpoint will use once it’s registered to CUCM (like typical phones get firmware).

Note that TC 7.3.3 or greater firmware have different functionality for remote screen monitoring of systems!  TC 7.3.3 introduces the requirement to have an option key for remote system/screen monitoring of the TC endpoint.  You’ll need to work with TAC/Cisco to get option keys cut if you are doing remote screen monitoring before going to 7.3.3+.

Registration to CUCM was straightforward.  I defined the device on CUCM as you would normally define a phone, picking the appropriate SIP profiles.  I didn’t do secure registration as this is optional.   (The documentation above does mention that secure registration is optional, but the example works through a secure registration.)

I made sure to set the device association my end user in CUCM.  This is important for MRA later.

Once the MX-300 was registered and making calls successfully through CUCM, I moved it out to a general internet connection to work on MRA.

On the touchpanel I launched the Provisioning wizard and selected Cisco UCM via Expressway.  After putting in my credentials I was greeted with this error:

IMG_4257 copy

After doing quite a bit of research and looking at the detailed error logs from the MX-300, it turns out that your Expressway-E certificate must also include a SAN for the domain name itself (e.g.  yourdomain.com).  The error actually indicates that it wants a collab-edge.yourdomain.com SAN:

Edge TLS verification failed: Edge domain ‘yourdomain.com’ and corresponding SRVName ‘collab-edge._tls.yourdomain.com’ not found in certificate SAN list.

The challenge I had is that the certificate I’d bought from GoDaddy for Expressway-E (that was working with 8800 MRA) wasn’t a UCC or multi-SAN certificate and you need to have at least the Expressway-E as the CN and the domain as a SAN.

At this point I decided that it was time to move from GoDaddy to DigiCert since they have unlimited resigning of certificates without having to revoke any of them.  This essentially allows you to create as many certificates as you want without having to keep buying more like GoDaddy.  I bought the Wildcard Plus certificate and used it to create a multi-SAN certificate for my Expressway-E.  The CN is always *.yourdomain.com, but you can add a bunch of SANs (like 20 or so.)

I generated the certificate with the following SANs – edge.yourdomain.com, yourdomain.com, collab-edge._tls.yourdomain.com and _collab-edge._tls.yourdomain.com.   One of the partner engineers that I talked to said that he got it working without having to add either collab-edge SAN.  (I’ve not looked into why/when we’ll need the collab-edge SAN and if it is actually be collab-edge as the error indicates, or if it needs the preceding underscore on collab-edge like the SRV records have.)

After applying the certificate to Expressway-E and rebooting it I tried the provisioning wizard on the touchpanel again and was greeted with the SAME ERROR.

It turns out that I hadn’t included the DigiCert root and the DigiCert Intermediate cert in the list of Trusted CAs on the endpoint itself.  The documentation indicates how to install it here – http://www.cisco.com/c/en/us/support/docs/unified-communications/expressway/118696-config-cucm-00.html#anc10  Make sure you have both root and intermediate (if the CA you used signs with an intermediate) on the TC endpoint.

After this the MX-300 registered like a champ and is able to do calls.  I followed this same process to get an SX-10 registered as well.

The partner engineer I talked to said he had to work through a couple issues on his test:  1) Endpoint rejecting the user credentials when running through the provisioning wizard.  Make sure the endpoint is associated with the End User and that the end user has CTI Enabled and CCM End user.   2) Getting an http download error after getting through the initial expressway authentication and that it was caused by the endpoint needing to do a _cisco-uds.yourdomain.com lookup internally to find out where CUCM is to download it’s configuration.   He was in a split domain situation and didn’t have a _cisco-uds record for the external domain on the inside.

I’ll detail the adventure for the DX-series on another post.

 

 

 

 

Expressway 8.5.2+ MRA Issues

I’ve run into a couple challenges after an upgrade to Expressway/VCS 8.5.2 where MRA for phones quit working.

I found a bug that broke MRA in 8.5.2 (the recommendation has been to downgrade back to 8.5.1).  That bug is shows that it is now fixed in 8.6, so I upgraded to 8.6.1 recently.  MRA started working again, but only on one out of every three login attempts.  It was really weird.  In looking at the logs it showed a bunch of errors:

Home CUCM not available – Unknown CUCM cluster for node sub02

Home CUCM not available – Unknown CUCM cluster for node sub03

The deployment I was working on is a three node (pub and two subs), running split DNS (different internal domain than the external domain name).

After a lot of digging it turns out that there is a change in the way MRA handles CUCM lookup.  When I installed the system added my pub and subs to Expressway-C by IP address.  But it looks like Expressway now attempts to communicate with them via hostname, and not IP as they were defined by me.

Since Expressway is using the domain suffix assigned to MRA (extdomain.com), it is attempting to lookup sub02.extdomain.com and sub03.extdomain.com.  I didn’t have A records for these on my internal DNS server extdomain zone since I’ve never needed to resolve them by the external domain name.

Adding these two records fixed the login issue and it now logins on first attempt like it used to.