Notes about Upgrading to CSR 11.5

Helped a customer upgrade from 11.0 to CSR 11.5, CUCM 11.5(1)SU1; IM&P 11.5(1)SU1; CUC 11.5(1)SU1.

Unity Connection 11.5

You must apply ciscocm.cuc_11.5SU1_pre_upgrade.cop.sgn before you upgrade to 11.5 because of bugid CSCvb02774.  The install of the patch is straightfoward and does not require a reboot.  I also ran a utils iothrottle disable to make the upgrade run faster (since it was being done after hours.)

If you’re upgrading from 10.x or earlier it is CRITICAL to increase your VM RAM to 6GB.  (This was something I ran into when going to 11.0.  If you leave it at 4GB  it will not function properly at all.)

The upgrade ran normally and took a quite a while for the switch-version to complete.

On a site note, I noticed that the new Unity Connection (CUC) 11.5 .ova files define a 200GB HDD for the bigger VM.  I investigated increasing my HDD from 160 to 200GB, but found out that CUC does NOT support dynamic resize of the HDD.  This will cause the partition to be unaligned and you’ll get to rebuild CUC from scratch.  So leave it at it’s current size.


CUCM 11.5

To save time during the upgrade window, the day before I preloaded the 11.5 ISO on my remote ESXi datastores so that it wouldn’t take forever for the ISO to SFTP over to the remote offices (they have limited bandwidth) , then I attached those ISOs as virtual DVDs to the CUCM servers via vShpere and then launched those upgrades as though they were coming from DVD instead of a remote file server.

The first attempt to launch the upgrade on the Pub failed with the old “common parition doesn’t have enough space” business.  I used RTMT to decrease the Low and High logging watermark to 45 and 40 respectively (and restarted the log partition monitoring service) to create room.

Purge Log Files by Changing the Log Partition Watermarks

  • Another way to create additional disk space is by changing the high and low watermarks on the system. This informs Unified CM of the numbers of log files to purge once the watermark is reached. Use RTMT as follows:
  1. Launch RTMT and log in to the desired cluster.
  2. From the left pane, select Alert Central.
  3. On the right pane, double-click LogPartitionHighWaterMarkExceeded. Change the threshold value to 40.
  4. On the right pane, double-click LogPartitionLowWaterMarkExceeded. Change the threshold value to 45.
  5. This data is polled every five minutes. Allow five to 10 minutes and then check the drive partitions for additional disk space by using one of the methods described above.

http://docwiki.cisco.com/wiki/Unified_CM_L2_Upgrade_Disk_Space_issues#Purge_Log_Files_by_Changing_the_Log_Partition_Watermarks

 

As usual, I ran the Pub first (without switching version), when it completed, I ran the Subs (also without switching versions).

If you’re coming from 11.0, the utils iothrottle disable command is not necessary.  (You can try to run it but CUCM 11.0 tells you it is unneeded.)

I rebooted the Pub and then Subs as normal.


IM&P 11.5

This was also a typical upgrade.  The switch-version took a LONG time for services to come up on the reboot.

 

 

Advertisements

CUCM 11.5 Released – Warning!

CSR (CUCM/IM&P/CUC) 11.5 has been released!  It includes a lot of cool new features but there’s a warning that needs to be shared for customers who have legacy phones.

New Features of note:

CUCM 11.5

  • “Hitless” Device Pack Installation.  Add new device types without an immediate reboot.  TFTP restart required, and Publisher reboot at next maintenance window.
  • Read-only AXL user role.  (Finally!)
  • PIN Sync to Unity Connection
  • User-customizable  Display Name field searchable in the directory for users who want a nickname or short name.  (e.g. Mike White vs Michael White — both searchable)
  • Directory search for MRA clients/devices
  • UCM Scale Increase with same OVA spec
  • Single SAML SSO Agreement per Cluster
  • Next Gen Encryption (DOD Suite B)
  • VMWare ESXi 6.0 support, Single SAML IdP per Cluster

 

IM&P (Jabber Server) 11.5

  • Multi-device Messaging.  Finally copies of chats messages are sent to all clients, not the most recently active.  Read notifications sync’d across all devices.
  • Persistent Chat HA and support for MS SQL (in addition to currently-supported Postgres and Oracle) for Persistent Chat functions.

Unity Connection

  • Full video messaging.  In addition to video greetings, video messages can be left in inboxes.  Video playback is supported from the handset, and not Jabber at this time.  (This functionality requires Mediasence as the video engine.)
  • PIN Synchronization between CUC and CUCM
  • New Media Player replaces the old MediaMaster applet
  • Support for Exchange 2016/Outlook 2016, Windows 10, MS Edge Browser
  • Next Gen Encryption (DOD Suite B)
  • VMWare ESXi 6.0 support, Single SAML IdP per Cluster

 

The warning is located in the release notes.  Certain legacy (i.e. REALLY OLD 12+ year EoS models) phones WILL NOT WORK with CUCM 11.5.

Update: A Field Notice has been posted.

This is a drastic change from previous versions were the phones are no longer supported and may not work properly anymore rather than disabling these models.  For example, I have an old 12SP+ at the office and it would register to CUCM 10, but not really work right.  In the case of 11.5, IT WILL NOT WORK.

Realistically the only models that I still see customers with are the 7935 Conference Station, 7920 Wireless, and 7910.

If you have these endpoints, understand that most have been EoS for over 12 years, and that there is a very aggressive phone trade-in program to get 20-30% extra discount (on top of your normal discount) to go to the new 8800 series phones.

Deprecated Endpoints

As of Cisco Unified Communications Manager Firmware Release 11.5, the following phones are not supported:•    Cisco IP Phone 12 SP+ and related models
•    Cisco IP Phone 30 VIP and related models
•    Cisco Unified IP Phone 7902
•    Cisco Unified IP Phone 7905
•    Cisco Unified IP Phone 7910
•    Cisco Unified IP Phone 7910SW
•    Cisco Unified IP Phone 7912
•    Cisco Unified Wireless IP Phone 7920
•    Cisco Unified IP Conference Station 7935

If you use any of these phone models on an older release of Cisco Unified Communications Manager and you upgrade to Release 11.5, the phone will not work after the upgrade completes.

Here’s an example from my CUCM where I have a 7905 defined.  The phones stays there, but this giant warning appears at the top.

CUCM11.5

Adventures in Upgrading to CSR 11.0

Now that all of the core CSR 11 components have had a service release under their collective belts, it’s go-time.  I helped a customer upgrade CUCM, IM&P and CUC from 10.5 to release 11.0.

  • CUCM/CUC 11.0(1a)SU1
  • CUCM IM&P 11.0(1)SU1
  • Jabber for Mac and Windows 11.5
  • Latest DX-series/8800 firmware
  • Expressway C/E X8.7.1
  • CWMS 2.6 MR1 Patch 1
  • Security COP to address CSCuy07473 for CUCM 11.0(1)
  • Permanent Licensing Surprises
  • Holding UCCX at 10.6 for now…. (Agent/Supervisor issues)

 

CUCM 11.0(1a)SU1

This is a four node system (pub and 3 subs) running the latest 10.5 SU.  Upgraded the pub during evening hours and told it not to reboot the night before.  Once it had completed the upgrade, I ran the upgrade on the three subs and told them not to reboot.  The maintenance window was the next evening, so we didn’t make any changes during that window.

When trying to reboot the pub to the new version from the GUI it got into an ugly loop.  Switch version reported that an upgrade was still in progress.  Went to the upgrade menu option and it indicated that I had to assume control over the upgrade.  Did so and the log file showed that the upgrade had completed successfully and that the lock files were released.  Went back to switch version and it still indicated that it was in an upgrade…

Bailed on the GUI and issued the version switch from the CLI.  It didn’t complain at all and did the version switch from 10.5 to 11.  It was a faster process than I imagined, taking less than 10 minutes to reboot on 11.  I was a bit concerned if it was going to work given the GUI seeming to be in a loop, but it rebooted just fine.

I rebooted the subs all from the CLI since I didn’t (and now perhaps don’t trust) the GUI switch version.  They all rebooted quickly too and were up and running on v11.0

Important!  Please note that the OVA release notes indicate that the RAM should be upgraded to 6GB for the CUCM and CUC VMs – http://www.cisco.com/web/software/283088407/126036/cucm-11.0.ova.readme.txt.  This was pointed out by a kind reader, which I hadn’t noticed until my CUC was falling apart after the upgrade.  Moving this VM to 6GB was an immediate fix.

CUC 11.0(1a)SU1

This was a textbook upgrade that evening.  I’d prestaged 11.0 like CUCM and the reboot took about 15 minutes.  All was well until I applied the permanent licensing.  Which I’ll cover later

IM&P 11.0(1)SU1

I wasn’t able to pre-stage this upgrade so ran it the evening of the maintenance window.  It’s a fairly small system so it took about an hour to upgrade and reboot.  The reboot seemed excessively long and I was worried, but it came back and workstation Jabber  clients automatically connected.

In conjunction with this upgrade I updated my jabber-update.xml file and push out the latest 11.5(2) version of the Mac and Windows clients.  We also updated all of the user photos on the webserver that houses them to current pictures.  Jabber was hit and miss about actually pulling the new picture.  It seemed that you had to manually view the profile on some users to get it to pull the new picture.

Latest DX/8800 series Firmware

The DX-series firmware has been a bumpy bumpy bumpy road.  It’s finally pretty stable as of 10.2(5)154.  A newer 10.2(5)195 is out so I pushed that out as it has a number of bugfixes.  I also updated the photo location for the DX-series phones and they all now pull the photos correctly from the webserver that houses them.  The super secret URL to put in the Company Photo Directory is this:  http://<webserver ip address>/%%uid%%.jpg

I migrated the DX-es from Anyconnect VPN over to MRA through Expressway that night since this latest ASA Sev 10 Bugfix upgrade has caused an odd cert issue for the DX (not not normal Anyconnect software clients on other platforms).  Remote phone control does work properly from Jabber (that is VPNed in) to the phone that is connected via MRA.

CUCM 11.0 default firmware also had older firmware for the 8800 series phones so I pushed the latest 11.0 version and am anxiously awaiting 11.5 for some really cool upcoming features for the 8800 series.

Expressway C/E X8.7.1

Textbook upgrade.  I love the software that came from Tandberg.

CWMS 2.6MR1 Patch 1

This is still my favorite app to upgrade by miles.  Attach the ISO to the Admin VM in vCenter and press go from the GUI.  An hour or so later after a couple reboots of all the various VMs (Admin, Media, IRP) you kick it back out of maintenance mode and you’re done.

Security COP to address CSCuy07473 for CUCM 11.0(1)

This patch JUST released with the latest security fixes for CiscoSSL (a ciscoized variant of OpenSSL).  Install on each CUCM node and you’re done.  No reboot required.

Permanent Licensing

After upgrading everything to 11.0 everything kicked into 60-day temp license mode as expected.  (Upgrading to CSR 10.5 was bad news when it didn’t do what it was supposed to and CCX ate all of it’s licenses resulting in a P1 case.)

The TAC case for licensing was pretty straightforward.  Had permanent licenses in about a day after providing the contract number that showed SWSS.

I held of installing the permanent licenses until after hours in the event that something would go wrong and take the system down (still nervous after the CCX incident).  Installation went fine with one side issue.

I had complaints about SpeechConnect / voice enabled directory handlers on Unity Connection not working right.  Turns out CUC didn’t like the permanent licenses as far as SpeechConnect.  It had pulled the licenses from ELM/PLM properly and was in compliance, but it took a restart of the Conversation service for it to start doing the voice recognition stuff again.  Rather odd.

Holding at CCX 10.6

Since 10.6 is the last version of CCX to support CAD/CSD and Finesse, I’m working to migrate the contact center over to Finesse.  There are some usability complaints we’re working through.  The users love the idea of a dedicated app that pops when a call comes in as well as the agent-to-agent chat inside CAD.  Getting them to use a web-browser for Finesse has been a challenge.   Once I have those details ironed out we’ll force them into Finesse when we upgrade to CSR 11.5 in the summer.

 

 

 

Applying Chained SSL Certificates for VOS-based Applications (CUCM, IM&P, UCXN, UCCX, etc.)

One important task in a Jabber (and/or Collab Edge MRA deployment), or a UCCX deployment is to make sure that the clients trust the certificates that are presented by the services.  The current versions of Jabber force validation of certificates, as well as (obviously) web browsers used with Finesse.

I recently worked through applying trusted CA-signed certificates for a UCCX 10.5 Finesse desktop deployment.  The customer had deployed UCCX Finesse without installing valid certificates on the server which was casing a number of problems.

  1. Agent web-browser login issues (untrusted certificate)
  2. Finesse Gadget’s failing (like Team stats viewer)
  3. CUIC report errors (either direct CUIC web interface or gadgets in Finesse)

I’d highly recommend installing trusted certificates before going production with a system to avoid these errors.

Basic flow to deploy trusted SSL certificates

Begin by going to OS Admin and generating a certificate server request (CSR) for the server.  The CSR is then signed by a certificate authority (CA).

To have Jabber or the web browser trust the certificate it must be signed by a trusted public CA like Verisign, GoDaddy, Starfield, Thawte, etc., or by a private CA that’s been added to the trusted list of the computer/mobile device.  (e.g. Private CAs include MS Active Directory Certificate Service for a client that’s part of an AD domain, or an OpenSSL (ahem, if you still trust it…  ;-)) that’s been added to the client’s trust list.)  The most compatible, but more expensive option is to use a public CA to sign the server certificate request (CSR) as it is trusted inherently by the most clients.

One of the challenges we’re seeing from public CA’s is the CSR being signed with an intermediate certificate.  Usually the certificate chain involves the server certificate, an intermediate certificate and a root certificate.  In the past it was much more popular to have the server certificate linked directly to a root certificate.  This has been a problem historically for some servers that don’t know how to deal with intermediate or chained certificates.  Most VOS applications 9.0 and higher support chained certificates, but documentation is non-existent.

After having the CSR signed, you’ll want to request the certificates in Base64 encoded format (usually a .pem or .cer extension which is an ASCII File that has the —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—– markers).  Normally you’ll receive a ZIP file with the server certificate, as well as a root certificate, or in the case of a chained certificate that has an intermediate certificate you may receive a bundle .pem/cer with the intermediate and root certificates.

The server certificate is easy to deal with, as it will typically have the name of the server in the filename and only include that certificate.

Figuring out how to split up the intermediate and root certificates from the bundle .pem/cer can be a challenge.  The file will normally include three certificates, delimited by the —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—– for each certificate.

Typically they are chained in the bundle .pem/cer with the Root certificate first, the Intermediate second.  They may have another certificate at the bottom which is a Class 2 Certificate Authority identifier. This is not needed for the import of the certificates into UCCX.

Windows makes it easy to open the bundle and export just the certificates you want by double-clicking the .cer and looking at the chain and exporting the certificates.   On the Mac, I just take the text file and cut and paste it into separate files for each certificate and then manually open them in Keychain access to figure out which one is which.  Then I name each file appropriately (like root.cer and intermediate.cer).  Now I have all three certificates that I need — root, intermediate and server.

At this point to into OS Admin on the VOS box (UCCX in this example) and import them according to the documentation below.

UCCX 10.5 allows importing of chained certificates but the instructions are incorrect.

The instructions are located here –

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_contact/contact_center/finesse/finesse_1051/installation/guide/CFIN_BK_CA0E68AE_00_cisco-finesse-installation-and-upgrade-1051/CFIN_BK_CA0E68AE_00_cisco-finesse-installation-and-upgrade-1051_chapter_0100.html#CFIN_TK_IB59BABE_00

They indicate the proper order of importing the certificates (1. Root; 2. Intermediate; 3. Server), but step 7b and 8b are incorrect.  It has outdated information indicating that the Root Certificate must be specified as in the screenshot below:

Image

 

This field does not exist in UCCX 10.5, so ignore steps 7b and 8b.  It will automatically chain the certificates.

Once the certificates are uploaded, I bogged down in trying to restart individual services like tomcat and finesse which fixed the client cert errors, but the gadget and CUIC errors persisted.  The eventual fix was a complete server reboot, so I would suggest doing that immediately.

 

CUCM 9.1(2)SU1 and CUC 9.1(2)SU1 Upgrade Tips

CUCM Publisher

Upgraded from 9.1(2) to 9.1(2)SU1 [9.1.2.11900-12] over the weekend because of the extensive list of bugfixes.  Release notes – http://www.cisco.com/web/software/282074295/108330/cucm-readme-912su1.pdf

Download the update here – http://software.cisco.com/download/release.html?mdfid=284510097&flowid=45900&softwareid=282074295&release=9.1%282%29SU1&relind=AVAILABLE&rellifecycle=&reltype=latest

The publisher failed with the Common partition not having enough space.  This was a common issue in the 8.x to 9.x upgrades.

The fix is to use RTMT to clear out logs and other things in the common partition, or much easier is to use the “Free common space COP file” – ciscocm.free_common_space_v1.0.cop.sgn.cop file from here – http://software.cisco.com/download/release.html?mdfid=284510097&flowid=45900&softwareid=282204704&release=COP-Files&relind=AVAILABLE&rellifecycle=&reltype=latest

Once you install this COP file the common partition should have enough free space to install the upgrade.  I’ve seen this not clear out enough space in one other upgrade, but in the 9.1(2) to 9.1(2)SU1 upgrade it worked like a champ.

Unity Connection

While the Publisher was upgrading, I initiated the Unity Connection (CUC) upgrade to 9.1(2)SU1.  This install started and failed with a very nondescript error:

The solution in my case was documented in several TAC cases: Reboot the CUC server before doing the install.  I did this and the CUC upgrade completed without any issues.

One major bug fix I’ve been waiting for was a unified inbox issue where users on Exchange with very large mailboxes would have CUC quit syncing VM.  SU1 fixes that problem.

CUCM Subscribers

Once the publisher finished and was rebooted the subscribers upgraded without issue.