Notes about Upgrading to CSR 11.5

Helped a customer upgrade from 11.0 to CSR 11.5, CUCM 11.5(1)SU1; IM&P 11.5(1)SU1; CUC 11.5(1)SU1.

Unity Connection 11.5

You must apply ciscocm.cuc_11.5SU1_pre_upgrade.cop.sgn before you upgrade to 11.5 because of bugid CSCvb02774.  The install of the patch is straightfoward and does not require a reboot.  I also ran a utils iothrottle disable to make the upgrade run faster (since it was being done after hours.)

If you’re upgrading from 10.x or earlier it is CRITICAL to increase your VM RAM to 6GB.  (This was something I ran into when going to 11.0.  If you leave it at 4GB  it will not function properly at all.)

The upgrade ran normally and took a quite a while for the switch-version to complete.

On a site note, I noticed that the new Unity Connection (CUC) 11.5 .ova files define a 200GB HDD for the bigger VM.  I investigated increasing my HDD from 160 to 200GB, but found out that CUC does NOT support dynamic resize of the HDD.  This will cause the partition to be unaligned and you’ll get to rebuild CUC from scratch.  So leave it at it’s current size.


CUCM 11.5

To save time during the upgrade window, the day before I preloaded the 11.5 ISO on my remote ESXi datastores so that it wouldn’t take forever for the ISO to SFTP over to the remote offices (they have limited bandwidth) , then I attached those ISOs as virtual DVDs to the CUCM servers via vShpere and then launched those upgrades as though they were coming from DVD instead of a remote file server.

The first attempt to launch the upgrade on the Pub failed with the old “common parition doesn’t have enough space” business.  I used RTMT to decrease the Low and High logging watermark to 45 and 40 respectively (and restarted the log partition monitoring service) to create room.

Purge Log Files by Changing the Log Partition Watermarks

  • Another way to create additional disk space is by changing the high and low watermarks on the system. This informs Unified CM of the numbers of log files to purge once the watermark is reached. Use RTMT as follows:
  1. Launch RTMT and log in to the desired cluster.
  2. From the left pane, select Alert Central.
  3. On the right pane, double-click LogPartitionHighWaterMarkExceeded. Change the threshold value to 40.
  4. On the right pane, double-click LogPartitionLowWaterMarkExceeded. Change the threshold value to 45.
  5. This data is polled every five minutes. Allow five to 10 minutes and then check the drive partitions for additional disk space by using one of the methods described above.

http://docwiki.cisco.com/wiki/Unified_CM_L2_Upgrade_Disk_Space_issues#Purge_Log_Files_by_Changing_the_Log_Partition_Watermarks

 

As usual, I ran the Pub first (without switching version), when it completed, I ran the Subs (also without switching versions).

If you’re coming from 11.0, the utils iothrottle disable command is not necessary.  (You can try to run it but CUCM 11.0 tells you it is unneeded.)

I rebooted the Pub and then Subs as normal.


IM&P 11.5

This was also a typical upgrade.  The switch-version took a LONG time for services to come up on the reboot.

 

 

Upgrading to VCS/Expressway X8.8 and Jabber MRA Broke? Here’s why…

VCS/Expressway X8.8 changes it’s behavior versus prior versions.  8.8 does a reverse lookup of the IP addresses it’s communicating with to make sure it matches the hostname between C and E.

From the Release notes:

DNS entries: Do you have forward and reverse DNS lookups for all infrastructure
systems that the Expressway interacts with? If the Expressway cannot resolve hostnames and IP addresses of systems,your complex deployments (eg.MRA) could stop working as expected after you upgrade.

Oddly enough, the two systems that I’ve been involved in the upgrade to 8.8.1 with, both had the Unified Communications traversal zone with show Active, and hard phones (8800 and DX 650) will register and work properly, but Jabber clients will be unable to login and Jabber will throw an error when trying to login through MRA:

"Unable to Communicate with Server."

Running the debugging logs on Expressway-C you see the following error: 

"Certificate verification failed for host=x.x.x.x, additional info: 
Invalid Hostname expressway-e.domain.com"

The fix is to make sure that Expressway-C can do a reverse DNS lookup on the IP address of Expressway-E. Then flush the DNS cache of C to make sure it re-queries DNS properly.

The debugging log will give you the address and hostname it is trying to do the lookup on.

In a dual-NIC Expressway-E deployment the PTR recrod should point to the private IP address that C talks to.  In a single-NIC NAT hairpin deployment, I’ve seen it talk on the private and public IP.  So check that debug log.

 

 

 

Adventures in Upgrading to CSR 11.0

Now that all of the core CSR 11 components have had a service release under their collective belts, it’s go-time.  I helped a customer upgrade CUCM, IM&P and CUC from 10.5 to release 11.0.

  • CUCM/CUC 11.0(1a)SU1
  • CUCM IM&P 11.0(1)SU1
  • Jabber for Mac and Windows 11.5
  • Latest DX-series/8800 firmware
  • Expressway C/E X8.7.1
  • CWMS 2.6 MR1 Patch 1
  • Security COP to address CSCuy07473 for CUCM 11.0(1)
  • Permanent Licensing Surprises
  • Holding UCCX at 10.6 for now…. (Agent/Supervisor issues)

 

CUCM 11.0(1a)SU1

This is a four node system (pub and 3 subs) running the latest 10.5 SU.  Upgraded the pub during evening hours and told it not to reboot the night before.  Once it had completed the upgrade, I ran the upgrade on the three subs and told them not to reboot.  The maintenance window was the next evening, so we didn’t make any changes during that window.

When trying to reboot the pub to the new version from the GUI it got into an ugly loop.  Switch version reported that an upgrade was still in progress.  Went to the upgrade menu option and it indicated that I had to assume control over the upgrade.  Did so and the log file showed that the upgrade had completed successfully and that the lock files were released.  Went back to switch version and it still indicated that it was in an upgrade…

Bailed on the GUI and issued the version switch from the CLI.  It didn’t complain at all and did the version switch from 10.5 to 11.  It was a faster process than I imagined, taking less than 10 minutes to reboot on 11.  I was a bit concerned if it was going to work given the GUI seeming to be in a loop, but it rebooted just fine.

I rebooted the subs all from the CLI since I didn’t (and now perhaps don’t trust) the GUI switch version.  They all rebooted quickly too and were up and running on v11.0

Important!  Please note that the OVA release notes indicate that the RAM should be upgraded to 6GB for the CUCM and CUC VMs – http://www.cisco.com/web/software/283088407/126036/cucm-11.0.ova.readme.txt.  This was pointed out by a kind reader, which I hadn’t noticed until my CUC was falling apart after the upgrade.  Moving this VM to 6GB was an immediate fix.

CUC 11.0(1a)SU1

This was a textbook upgrade that evening.  I’d prestaged 11.0 like CUCM and the reboot took about 15 minutes.  All was well until I applied the permanent licensing.  Which I’ll cover later

IM&P 11.0(1)SU1

I wasn’t able to pre-stage this upgrade so ran it the evening of the maintenance window.  It’s a fairly small system so it took about an hour to upgrade and reboot.  The reboot seemed excessively long and I was worried, but it came back and workstation Jabber  clients automatically connected.

In conjunction with this upgrade I updated my jabber-update.xml file and push out the latest 11.5(2) version of the Mac and Windows clients.  We also updated all of the user photos on the webserver that houses them to current pictures.  Jabber was hit and miss about actually pulling the new picture.  It seemed that you had to manually view the profile on some users to get it to pull the new picture.

Latest DX/8800 series Firmware

The DX-series firmware has been a bumpy bumpy bumpy road.  It’s finally pretty stable as of 10.2(5)154.  A newer 10.2(5)195 is out so I pushed that out as it has a number of bugfixes.  I also updated the photo location for the DX-series phones and they all now pull the photos correctly from the webserver that houses them.  The super secret URL to put in the Company Photo Directory is this:  http://<webserver ip address>/%%uid%%.jpg

I migrated the DX-es from Anyconnect VPN over to MRA through Expressway that night since this latest ASA Sev 10 Bugfix upgrade has caused an odd cert issue for the DX (not not normal Anyconnect software clients on other platforms).  Remote phone control does work properly from Jabber (that is VPNed in) to the phone that is connected via MRA.

CUCM 11.0 default firmware also had older firmware for the 8800 series phones so I pushed the latest 11.0 version and am anxiously awaiting 11.5 for some really cool upcoming features for the 8800 series.

Expressway C/E X8.7.1

Textbook upgrade.  I love the software that came from Tandberg.

CWMS 2.6MR1 Patch 1

This is still my favorite app to upgrade by miles.  Attach the ISO to the Admin VM in vCenter and press go from the GUI.  An hour or so later after a couple reboots of all the various VMs (Admin, Media, IRP) you kick it back out of maintenance mode and you’re done.

Security COP to address CSCuy07473 for CUCM 11.0(1)

This patch JUST released with the latest security fixes for CiscoSSL (a ciscoized variant of OpenSSL).  Install on each CUCM node and you’re done.  No reboot required.

Permanent Licensing

After upgrading everything to 11.0 everything kicked into 60-day temp license mode as expected.  (Upgrading to CSR 10.5 was bad news when it didn’t do what it was supposed to and CCX ate all of it’s licenses resulting in a P1 case.)

The TAC case for licensing was pretty straightforward.  Had permanent licenses in about a day after providing the contract number that showed SWSS.

I held of installing the permanent licenses until after hours in the event that something would go wrong and take the system down (still nervous after the CCX incident).  Installation went fine with one side issue.

I had complaints about SpeechConnect / voice enabled directory handlers on Unity Connection not working right.  Turns out CUC didn’t like the permanent licenses as far as SpeechConnect.  It had pulled the licenses from ELM/PLM properly and was in compliance, but it took a restart of the Conversation service for it to start doing the voice recognition stuff again.  Rather odd.

Holding at CCX 10.6

Since 10.6 is the last version of CCX to support CAD/CSD and Finesse, I’m working to migrate the contact center over to Finesse.  There are some usability complaints we’re working through.  The users love the idea of a dedicated app that pops when a call comes in as well as the agent-to-agent chat inside CAD.  Getting them to use a web-browser for Finesse has been a challenge.   Once I have those details ironed out we’ll force them into Finesse when we upgrade to CSR 11.5 in the summer.

 

 

 

Collaboration Edge (MRA) with Split DNS Domains

I recently helped one of our customers work through a split DNS domain Collab Edge MRA deployment that was quite educational.  Here are a few tidbits that I picked up.

Scenario:

Customer’s external (internet facing) DNS domain is:  collabdomain.com
Customer’s internal domain is:  internaldomain.local  (not resolvable externally, only served by the internal DNS server)

Expressway-E lives on the internet so it’s hostname would be vcse.collabdomain.com

Expressway-C, CUCM, IM&P (CUP) and Unity Connection live on the internal domain, so they would be called ucm1.internaldomain.local, cup1.internaldomain.local, ucxn.internaldomain.local.

The users accounts would live in internaldomain.local (presumably the LDAP user would be user@internaldomain.local, etc.)

The external DNS server would serve the public collabdomain.com zone.

The internal DNS server would serve internaldomain.local, as well as an internal version of the collabdomain.com zone.

The challenge with these separate domain names is that when a user goes to login to Jabber via MRA, the service discovery process is going to fail because they are going to type in their IM address as user@internaldomain.local, and there is no _collab-edge._tls.internaldomain.local SRV record availble on the itnernet (nor could there be).

To get around this, the jabber-config.xml file can be modified to include the external domain name that Jabber should try to do service discovery against. 

<Policies>
<VoiceServicesDomain>collabdomain.com</VoiceServicesDomain>
</Policies>

Jabber must first login internally (directly on your corporate network) to pull down the jabber-config.xml that includes this policy.

With this configuration now loaded in Jabber, when the user attempts to login from outside via MRA they will still use their user@internaldomain.local credentials, but Jabber will use the specified VoiceServicesDomain — in this case — collabdomain.com.  It will do the SRV record lookup on _collab-edge._tls.collabdomain.com and now be able to resolve Expressway-E and login.

In addition to the the normal _cisco-uds._tcp.internaldomain.local SRV record, Jabber needs another _cisco-uds._tcp.collabdomain.com SRV record (on the internal DNS server) pointing to ucm1.internaldomain.local for MRA to work with the split domains.The one criticism to this method is requiring the Jabber client to login internally first, which isn’t an ideal scenario for large distributed organizations.  One method that is supposed to work around this is to prime Jabber by installing it generically and launching it from an URL:ciscojabber://provision?ServicesDomain=internaldomain.local&VoiceServicesDomain=collabdomain.com

This could be provided in an email and will supposedly launch Jabber so that it tries the external domain without having to ever login internally first.

 

 

Avoiding IM&P (CUP) 10.5 Cluster Upgrade Problems

Upgrade Blues

I recently had a customer upgrade to their cluster to CUCM 10.5(1) as well as their IM&P servers to 10.5(1).  They ran into an situation with the IM&P subscriber server that ended up requiring intervention from TAC to get running.  (I also noticed a number of other customers having this exact same issue.)

The symptom is that after the IM&P subscriber upgrade completes, the administrator screen goes blank when trying to login and the sync service won’t start.

Cause

The issues is caused by a CUCM LDAP sync occurring while the IM&P servers are being upgraded.  This causes all of the users assigned to subscriber IM&P servers to be unable to login.

If your server ends up in this condition, a call to TAC would be the first thing to do so that they can do a remote support session to fix the issue.

Avoiding the Problem in the First Place

Before performing a system upgrade of CUCM and IM&P, change the LDAP sync interval to a sufficiently long period to avoid having a sync occur during your upgrade window.  I’d push it out o a week or so, but make sure it isn’t going to happen during your upgrade.  This is the cause of the issues.

Upgrade CUCM and IM&P as normal

Change your LDAP sync interval back to previous values (many engineers set it to 24 hours).

Applying Chained SSL Certificates for VOS-based Applications (CUCM, IM&P, UCXN, UCCX, etc.)

One important task in a Jabber (and/or Collab Edge MRA deployment), or a UCCX deployment is to make sure that the clients trust the certificates that are presented by the services.  The current versions of Jabber force validation of certificates, as well as (obviously) web browsers used with Finesse.

I recently worked through applying trusted CA-signed certificates for a UCCX 10.5 Finesse desktop deployment.  The customer had deployed UCCX Finesse without installing valid certificates on the server which was casing a number of problems.

  1. Agent web-browser login issues (untrusted certificate)
  2. Finesse Gadget’s failing (like Team stats viewer)
  3. CUIC report errors (either direct CUIC web interface or gadgets in Finesse)

I’d highly recommend installing trusted certificates before going production with a system to avoid these errors.

Basic flow to deploy trusted SSL certificates

Begin by going to OS Admin and generating a certificate server request (CSR) for the server.  The CSR is then signed by a certificate authority (CA).

To have Jabber or the web browser trust the certificate it must be signed by a trusted public CA like Verisign, GoDaddy, Starfield, Thawte, etc., or by a private CA that’s been added to the trusted list of the computer/mobile device.  (e.g. Private CAs include MS Active Directory Certificate Service for a client that’s part of an AD domain, or an OpenSSL (ahem, if you still trust it…  ;-)) that’s been added to the client’s trust list.)  The most compatible, but more expensive option is to use a public CA to sign the server certificate request (CSR) as it is trusted inherently by the most clients.

One of the challenges we’re seeing from public CA’s is the CSR being signed with an intermediate certificate.  Usually the certificate chain involves the server certificate, an intermediate certificate and a root certificate.  In the past it was much more popular to have the server certificate linked directly to a root certificate.  This has been a problem historically for some servers that don’t know how to deal with intermediate or chained certificates.  Most VOS applications 9.0 and higher support chained certificates, but documentation is non-existent.

After having the CSR signed, you’ll want to request the certificates in Base64 encoded format (usually a .pem or .cer extension which is an ASCII File that has the —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—– markers).  Normally you’ll receive a ZIP file with the server certificate, as well as a root certificate, or in the case of a chained certificate that has an intermediate certificate you may receive a bundle .pem/cer with the intermediate and root certificates.

The server certificate is easy to deal with, as it will typically have the name of the server in the filename and only include that certificate.

Figuring out how to split up the intermediate and root certificates from the bundle .pem/cer can be a challenge.  The file will normally include three certificates, delimited by the —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—– for each certificate.

Typically they are chained in the bundle .pem/cer with the Root certificate first, the Intermediate second.  They may have another certificate at the bottom which is a Class 2 Certificate Authority identifier. This is not needed for the import of the certificates into UCCX.

Windows makes it easy to open the bundle and export just the certificates you want by double-clicking the .cer and looking at the chain and exporting the certificates.   On the Mac, I just take the text file and cut and paste it into separate files for each certificate and then manually open them in Keychain access to figure out which one is which.  Then I name each file appropriately (like root.cer and intermediate.cer).  Now I have all three certificates that I need — root, intermediate and server.

At this point to into OS Admin on the VOS box (UCCX in this example) and import them according to the documentation below.

UCCX 10.5 allows importing of chained certificates but the instructions are incorrect.

The instructions are located here –

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_contact/contact_center/finesse/finesse_1051/installation/guide/CFIN_BK_CA0E68AE_00_cisco-finesse-installation-and-upgrade-1051/CFIN_BK_CA0E68AE_00_cisco-finesse-installation-and-upgrade-1051_chapter_0100.html#CFIN_TK_IB59BABE_00

They indicate the proper order of importing the certificates (1. Root; 2. Intermediate; 3. Server), but step 7b and 8b are incorrect.  It has outdated information indicating that the Root Certificate must be specified as in the screenshot below:

Image

 

This field does not exist in UCCX 10.5, so ignore steps 7b and 8b.  It will automatically chain the certificates.

Once the certificates are uploaded, I bogged down in trying to restart individual services like tomcat and finesse which fixed the client cert errors, but the gadget and CUIC errors persisted.  The eventual fix was a complete server reboot, so I would suggest doing that immediately.

 

Jabber for Windows 9.7(0) is available on CCO

NOTE:  Although Jabber 9.7 supports Collab Edge MRA, it will not be officially TAC supported until Expressway 8.1.1 releases in the next few weeks.

 

Grab J4W 9.7 here — http://software.cisco.com/download/release.html?mdfid=284324806&flowid=46406&softwareid=284006014&release=9.7%280%29&relind=AVAILABLE&rellifecycle=&reltype=latest

Release notes are here — http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/jabber/Windows/9_7/JABW_BK_CF8F083D_00_cisco-jabber-for-windows-97.html

Please take a look at the release notes.  While Collab Edge MRA is supported in this client it is not TAC supported until Expressway 8.1.1 releases.  There are some notable feature exceptions to be aware of.

From the release notes:

When using Expressway Mobile and Remote Access to connect to services from outside the corporate firewall, the client does not support:

  • LDAP for contact resolution. Instead, the client must use UDS for contact resolution.
  • File transfer, including screen capture, is not supported with on-premise deployments. File transfer using Expressway Mobile and Remote Access is only supported using WebEx Cloud deployments.
  • Desk phone control mode (CTI), including extension mobility.
  • Extend and Connect. You cannot use the Jabber client to make and receive calls on a non-Cisco IP Phone in the office; to control a non-Cisco IP Phone in the office, such as hold/resume; or control a home or hotel phone when connecting with Expressway Mobile and Remote Access.
  • Dial via Office – Reverse calls.
  • Session persistency. The client cannot recover from disruptions caused by network transitions. For example, if a users start a Cisco Jabber call inside their office and then they walk outside their building and lose Wi-Fi connectivity, the call drops as the client switches to use Expressway Mobile and Remote Access.
  • Cisco WebEx Meetings Server. The client cannot access the Cisco WebEx Meetings Server server, or join or start on-premises Cisco WebEx meetings.
  • Sending problem reports. To work around this issue, users can save the report locally and send the report in another manner.
  • CAPF enrollment.
  • Early Media. Early Media allows the client to exchange data between endpoints before a connection is established. For example, if a user makes a call to a party that is not part of the same organization, and the other party declines or does not answer the call, Early Media ensures that the user hears the busy tone or is sent to voicemail. When using Expressway Mobile and Remote Access, the user does not hear a busy tone if the other party declines or does not answer the call. Instead, the user hears approximately one minute of silence before the call is terminated.
  • Self Care Portal.
  • End-to-end media encryption. Media is not encrypted on the call path between the Cisco VCS Control or Cisco Expressway-C and devices that are registered locally to Cisco Unified Communications Manager. The media path outside of the enterprise is encrypted.