Adventures in Upgrading to CSR 11.0

Now that all of the core CSR 11 components have had a service release under their collective belts, it’s go-time.  I helped a customer upgrade CUCM, IM&P and CUC from 10.5 to release 11.0.

• CUCM/CUC 11.0(1a)SU1
• CUCM IM&P 11.0(1)SU1
• Jabber for Mac and Windows 11.5
• Latest DX-series/8800 firmware
• Expressway C/E X8.7.1
• CWMS 2.6 MR1 Patch 1
• Security COP to address CSCuy07473 for CUCM 11.0(1)
• Permanent Licensing Surprises
• Holding UCCX at 10.6 for now…. (Agent/Supervisor issues)

 

CUCM 11.0(1a)SU1

This is a four node system (pub and 3 subs) running the latest 10.5 SU.  Upgraded the pub during evening hours and told it not to reboot the night before.  Once it had completed the upgrade, I ran the upgrade on the three subs and told them not to reboot.  The maintenance window was the next evening, so we didn’t make any changes during that window.

When trying to reboot the pub to the new version from the GUI it got into an ugly loop.  Switch version reported that an upgrade was still in progress.  Went to the upgrade menu option and it indicated that I had to assume control over the upgrade.  Did so and the log file showed that the upgrade had completed successfully and that the lock files were released.  Went back to switch version and it still indicated that it was in an upgrade…

Bailed on the GUI and issued the version switch from the CLI.  It didn’t complain at all and did the version switch from 10.5 to 11.  It was a faster process than I imagined, taking less than 10 minutes to reboot on 11.  I was a bit concerned if it was going to work given the GUI seeming to be in a loop, but it rebooted just fine.

I rebooted the subs all from the CLI since I didn’t (and now perhaps don’t trust) the GUI switch version.  They all rebooted quickly too and were up and running on v11.0

Important!  Please note that the OVA release notes indicate that the RAM should be upgraded to 6GB for the CUCM and CUC VMs – http://www.cisco.com/web/software/283088407/126036/cucm-11.0.ova.readme.txt.  This was pointed out by a kind reader, which I hadn’t noticed until my CUC was falling apart after the upgrade.  Moving this VM to 6GB was an immediate fix.

CUC 11.0(1a)SU1

This was a textbook upgrade that evening.  I’d prestaged 11.0 like CUCM and the reboot took about 15 minutes.  All was well until I applied the permanent licensing.  Which I’ll cover later

IM&P 11.0(1)SU1

I wasn’t able to pre-stage this upgrade so ran it the evening of the maintenance window.  It’s a fairly small system so it took about an hour to upgrade and reboot.  The reboot seemed excessively long and I was worried, but it came back and workstation Jabber  clients automatically connected.

In conjunction with this upgrade I updated my jabber-update.xml file and push out the latest 11.5(2) version of the Mac and Windows clients.  We also updated all of the user photos on the webserver that houses them to current pictures.  Jabber was hit and miss about actually pulling the new picture.  It seemed that you had to manually view the profile on some users to get it to pull the new picture.

Latest DX/8800 series Firmware

The DX-series firmware has been a bumpy bumpy bumpy road.  It’s finally pretty stable as of 10.2(5)154.  A newer 10.2(5)195 is out so I pushed that out as it has a number of bugfixes.  I also updated the photo location for the DX-series phones and they all now pull the photos correctly from the webserver that houses them.  The super secret URL to put in the Company Photo Directory is this:  http://<webserver ip address>/%%uid%%.jpg

I migrated the DX-es from Anyconnect VPN over to MRA through Expressway that night since this latest ASA Sev 10 Bugfix upgrade has caused an odd cert issue for the DX (not not normal Anyconnect software clients on other platforms).  Remote phone control does work properly from Jabber (that is VPNed in) to the phone that is connected via MRA.

CUCM 11.0 default firmware also had older firmware for the 8800 series phones so I pushed the latest 11.0 version and am anxiously awaiting 11.5 for some really cool upcoming features for the 8800 series.

Expressway C/E X8.7.1

Textbook upgrade.  I love the software that came from Tandberg.

CWMS 2.6MR1 Patch 1

This is still my favorite app to upgrade by miles.  Attach the ISO to the Admin VM in vCenter and press go from the GUI.  An hour or so later after a couple reboots of all the various VMs (Admin, Media, IRP) you kick it back out of maintenance mode and you’re done.

Security COP to address CSCuy07473 for CUCM 11.0(1)

This patch JUST released with the latest security fixes for CiscoSSL (a ciscoized variant of OpenSSL).  Install on each CUCM node and you’re done.  No reboot required.

Permanent Licensing

After upgrading everything to 11.0 everything kicked into 60-day temp license mode as expected.  (Upgrading to CSR 10.5 was bad news when it didn’t do what it was supposed to and CCX ate all of it’s licenses resulting in a P1 case.)

The TAC case for licensing was pretty straightforward.  Had permanent licenses in about a day after providing the contract number that showed SWSS.

I held of installing the permanent licenses until after hours in the event that something would go wrong and take the system down (still nervous after the CCX incident).  Installation went fine with one side issue.

I had complaints about SpeechConnect / voice enabled directory handlers on Unity Connection not working right.  Turns out CUC didn’t like the permanent licenses as far as SpeechConnect.  It had pulled the licenses from ELM/PLM properly and was in compliance, but it took a restart of the Conversation service for it to start doing the voice recognition stuff again.  Rather odd.

Holding at CCX 10.6

Since 10.6 is the last version of CCX to support CAD/CSD and Finesse, I’m working to migrate the contact center over to Finesse.  There are some usability complaints we’re working through.  The users love the idea of a dedicated app that pops when a call comes in as well as the agent-to-agent chat inside CAD.  Getting them to use a web-browser for Finesse has been a challenge.   Once I have those details ironed out we’ll force them into Finesse when we upgrade to CSR 11.5 in the summer.

 

 

 

UCCX 10.6 – Running Finesse and CAD simultaneously with Mixed Mode

Prior to today, cutting a contact center over from CAD/CSD to Finesse was not an insignificant task, requiring a flash cut of the system.

It is expected that CCX 11.0 will be a Finesse only release, requiring customers to move to Finesse if they want to upgrade to 11 or newer.  The good news is that UCCX 10.6 has been announced to allow CCX to run both CAD and Finesse simultaneously in what Cisco is calling Mixed Mode. 

The COP file is located here – http://software.cisco.com/download/cart.html?imageGuId=5D64ACFE3FA6013D8C3496338255EB4A1E98007F&i=rs

I had a customer on the EFT for it for the last month and it worked very well.  There are some limitations to be aware of and installation will require a couple reboots of your CCX environment.

Installation notes are in the readme here – http://www.cisco.com/web/software/280840578/114618/ReadMe_UCCX_10.6.1_MixedMode_cop.pdf

The release notes mention Appendix A of the Design Guide but I don’t see an updated one posted yet.  The doc or section should have some sort of title like Cisco Finesse Desktop Mixed Mode Deployment when it comes out.  I wouldn’t upgrade until you’ve read ALL the detail in the document, especially for larger contact centers or ones that are doing outbound or chat/email.

Installation

Basically you install the COP file from the CLI and restart, then follow the instructions to configure Finesse using the Finesse Administration portal on CCX.  Once you’ve got Finesse configured you can start to migrate agents and supervisors to it.  Keep in mind that you’re going to want to migrate a team at a time because a supervisor can’t be in Finesse and see their agents in CAD, or vice versa.

Limitations

There are some limitations to be aware of using Mixed Mode, as it won’t support all CCX deployments and features.

Features supported:

• Inbound and Outbound Voice (Direct preview only on both CAD and Finesse) can be migrated from CAD to Finesse in a phased manner
• Supervisory features with few limitations
• Reporting
• Failover
• Recording with few limitations

Features not supported:

• Web chat
• Email
• Predictive and Progressive Outbound

Note: The Chat and Email features will not work in a combination of CAD and Finesse and will work only with Finesse once its activated, hence it is recommended to move all chat and email agents at the same time

Other Limitations to note:

• UCCX On-demand recording is supported only in CAD. Customers will have to use Mediasense or QM for Finesse workflow recording.
• Migration of CAD configuration to Finesse will have to be done manually
• Macros are not supported in Cisco Finesse and cannot be migrated. Customers will have to use the Cisco Finesse workflow engine to accomplish tasks previously done by CAD macros.
• CRM connectors used with CAD will have to be reprogramed to be used with Finesse using API.
• Barge-in, silent monitoring, and intercept are not supported in mixed mode when CAD and Cisco Finesse agents are participants in the same call (for example, in a transfer/conference).
• A team led by one or more supervisors should completely use either CAD or Cisco Finesse for e.g. A supervisor who uses CSD cannot supervise a Cisco Finesse team and vice versa.

Applying Chained SSL Certificates for VOS-based Applications (CUCM, IM&P, UCXN, UCCX, etc.)

One important task in a Jabber (and/or Collab Edge MRA deployment), or a UCCX deployment is to make sure that the clients trust the certificates that are presented by the services.  The current versions of Jabber force validation of certificates, as well as (obviously) web browsers used with Finesse.

I recently worked through applying trusted CA-signed certificates for a UCCX 10.5 Finesse desktop deployment.  The customer had deployed UCCX Finesse without installing valid certificates on the server which was casing a number of problems.

1. Agent web-browser login issues (untrusted certificate)
2. Finesse Gadget’s failing (like Team stats viewer)
3. CUIC report errors (either direct CUIC web interface or gadgets in Finesse)

I’d highly recommend installing trusted certificates before going production with a system to avoid these errors.

Basic flow to deploy trusted SSL certificates

Begin by going to OS Admin and generating a certificate server request (CSR) for the server.  The CSR is then signed by a certificate authority (CA).

To have Jabber or the web browser trust the certificate it must be signed by a trusted public CA like Verisign, GoDaddy, Starfield, Thawte, etc., or by a private CA that’s been added to the trusted list of the computer/mobile device.  (e.g. Private CAs include MS Active Directory Certificate Service for a client that’s part of an AD domain, or an OpenSSL (ahem, if you still trust it…  ;-)) that’s been added to the client’s trust list.)  The most compatible, but more expensive option is to use a public CA to sign the server certificate request (CSR) as it is trusted inherently by the most clients.

One of the challenges we’re seeing from public CA’s is the CSR being signed with an intermediate certificate.  Usually the certificate chain involves the server certificate, an intermediate certificate and a root certificate.  In the past it was much more popular to have the server certificate linked directly to a root certificate.  This has been a problem historically for some servers that don’t know how to deal with intermediate or chained certificates.  Most VOS applications 9.0 and higher support chained certificates, but documentation is non-existent.

After having the CSR signed, you’ll want to request the certificates in Base64 encoded format (usually a .pem or .cer extension which is an ASCII File that has the —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—– markers).  Normally you’ll receive a ZIP file with the server certificate, as well as a root certificate, or in the case of a chained certificate that has an intermediate certificate you may receive a bundle .pem/cer with the intermediate and root certificates.

The server certificate is easy to deal with, as it will typically have the name of the server in the filename and only include that certificate.

Figuring out how to split up the intermediate and root certificates from the bundle .pem/cer can be a challenge.  The file will normally include three certificates, delimited by the —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—– for each certificate.

Typically they are chained in the bundle .pem/cer with the Root certificate first, the Intermediate second.  They may have another certificate at the bottom which is a Class 2 Certificate Authority identifier. This is not needed for the import of the certificates into UCCX.

Windows makes it easy to open the bundle and export just the certificates you want by double-clicking the .cer and looking at the chain and exporting the certificates.   On the Mac, I just take the text file and cut and paste it into separate files for each certificate and then manually open them in Keychain access to figure out which one is which.  Then I name each file appropriately (like root.cer and intermediate.cer).  Now I have all three certificates that I need — root, intermediate and server.

At this point to into OS Admin on the VOS box (UCCX in this example) and import them according to the documentation below.

UCCX 10.5 allows importing of chained certificates but the instructions are incorrect.

The instructions are located here –

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_contact/contact_center/finesse/finesse_1051/installation/guide/CFIN_BK_CA0E68AE_00_cisco-finesse-installation-and-upgrade-1051/CFIN_BK_CA0E68AE_00_cisco-finesse-installation-and-upgrade-1051_chapter_0100.html#CFIN_TK_IB59BABE_00

They indicate the proper order of importing the certificates (1. Root; 2. Intermediate; 3. Server), but step 7b and 8b are incorrect.  It has outdated information indicating that the Root Certificate must be specified as in the screenshot below:

 

This field does not exist in UCCX 10.5, so ignore steps 7b and 8b.  It will automatically chain the certificates.

Once the certificates are uploaded, I bogged down in trying to restart individual services like tomcat and finesse which fixed the client cert errors, but the gadget and CUIC errors persisted.  The eventual fix was a complete server reboot, so I would suggest doing that immediately.

 

Cisco Collaboration System Release 10 (CSR10) Build List

Preliminary build list:

• CUCM 10.0.1.10000-24
• CUC 10.0.1.10000-24
• IM&P (CUP) 10.0.1.10000-26
• UCCX 10.0.1.10000-56