Field Notice – 70379 = Time to Patch CUCM/CUC/CER/CUP 10/11/12!

Please review this recent Field Notice.  There’s a significant issue where a VMTools upgrade can brick your UC VM during a reboot.

Problem Description

After a Unified Communications server node is rebooted for any given reason, it will not boot the operating system (OS) back up. A reboot or power cycle of the server node will not boot the server node back up nor restore its services. The server node will remain offline and Applications Services will not be reinitialized.

The FN includes more details as well as a workaround in the event the system becomes bricked on reboot.

The latest SU’s for the various products include a fix.

CSR 12 Released – Understanding Smart Licensing

CSR 12.0 has been released (CUCM, IM&P, Unity Connection v12.0) and a these version bring a major change for the better to licensing.

No it’s not another change in licensing type (RTU, DLU, UCL, CUWL), but rather it is the ”cloudification” of licenses. Instead of using PLM to pull licenses, version 12 now uses Cisco Smart Licensing. Cisco will now keep track of your licenses for you instead of relying on registering PAKs and dealing with TAC.

Benefits of Smart Licensing:

• One source of truth for licensing. Instead of dealing with PAKs and TAC asking for SO#s during upgrades, the licensing portal will hold all licenses.
• Customer-controlled license pool for an Organization. Licenses can be shared across sub-orgraniations easily.
• Simple upgrades. The licenses are stored in the cloud and entitlement is easily visible there. Waiting for license files is a thing of the past.

Cisco is moving all products to be Smart License enabled. Smart Licensing is not an enforcement mechanism. It’s job is merely to capture and report license ownership and consumption details as sent by a Cisco product. Each product deals with it’s own license enforcement. CUCM/CUC offer a 90-day grace period.

Cisco Smart Software Manager (CSSM) is the customer web portal that enables management all of your Cisco Smart software licenses from one centralized website. With Cisco Smart Software Manager, you organize and view your licenses in groups called virtual accounts and can transfer the licenses between virtual accounts as needed.

More information about Cisco Smart Licensing is found here –https://www.cisco.com/c/en/us/buy/smart-accounts/software-licensing.html

CUCM and all products that use Smart licensing talk to Cisco one of three ways, directly through the internet, via an HTTP/HTTPS proxy, or through the Cisco Software Manager satellite. This is an OVA that you deploy on ESXi which is the central relay for products to communicate back to Cisco for an organization. More information and the download is located here –https://www.cisco.com/c/en/us/buy/smart-accounts/software-manager-satellite.html

How to register CUCM 12 to the Smart Software Manager

Create a Smart Account at software.cisco.com– Administration pulldown, Request Smart Account or if one already exists for your org, Reqest Access to Existing Smart Account.

 

Click on the option Smart Software Licensing > Inventory > Generate New Token

Copy the text of the token to the clipboard which you will eventually paste into CUCM for it to register (see several steps later).

 

Login to the newly upgraded/installed CUCM 12.0

 

In CUCM Admin navigate to: System > Licensing > License Management. You’ll see the following:

If your CUCM has access to the internet to talk to Cisco directly, click on the Register button.

 

Take the token that was generated from the Smart Licensing portal earlier and paste it in and hit Register:

 

If your CUCM does not have direct internet access, you will need to decide how you want CUCM to communicate with Cisco by clicking theedit the Licensing Smart Call Home Transport settings and selecting from the following:

 

If you want to use the Smart Software Manager satellite to communicate requests between your Smart Licensed components (keep in mind that Smart Licensing is the future of licensing for all Cisco products, and currently covers many more products than just Cisco collaboration products), navigate here to download and deploy the OVA – https://software.cisco.com/download/release.html?mdfid=286285506&softwareid=286285517&os=Linux&release=3.1.1&relind=AVAILABLE&rellifecycle=&reltype=latest

Once your system is registered you’ll see a Registration completed successfully message.

The main licensing page will now show registered:

And if you go back to to the Smart Licensing Portal (https://software.cisco.com/#SmartLicensing-Inventory)again you’ll see your product instance:

 

CUCM 11.5(1)SU2 – A critical CUCM upgrade for Jabber for iOS users – iOS Push Notifications (APNs)

Please note that Cisco has published a document covering CUCM IM&P 11.5(SU2) and Push Notifications here.

CUCM 11.5(1)SU2 released last week, and in addition to typical bug fixes it includes a major feature that all customers who use Jabber for iOS (iPad/iPhone) are recommended to deploy before September 2017.

The quick list of new features in CUCM 11.5(1)SU2 (release notes here):

• Cisco Meeting Server 2.x support for CUCM ad-hoc, meet-me, and Conference Now conferences.
• Cisco Spark Remote Device – allow your extension to also ring out to Spark as a soft phone (requires Expressway) without requiring an extra device license (as long as Spark-RD is not their only device).
• CUCM IM&P supports Skype for Business federation.
• CUCM IM&P Roster Cleanup (CLI command to purge contacts from buddy lists for contacts who are no longer present in the system — e.g. Employee leaves the company and should be purged from everyone’s buddy lists.
• CUCM IM&P support for dual MS SQL DB for persistent chat with high availability (instead of just supporting Postgres and Oracle).
• MRA support for Shared Lines on 78xx/88xx (requires Expressway X8.9)
• TLS 1.2 support for syslog.
• CUCM IM&P Apple Push Notification Service support for IM.

I’ll focus on the last feature since it will become the most important of the bunch for anyone running Jabber on iOS.

APNs for iOS IM and Call Notification

It is recommended to upgrade to IM&P 11.5(1)SU2 this summer.  IM&P 11.5(1)SU2 adds support for Apple APNs.  The primary reason for this is to save battery by stopping IM and VoIP apps from continually doing a keep-alive to their service.  When the app is put into the background it will be completely terminated, this requiring APNs to wake up the app to receive IM/VoIP calls.  All APNs notifications are encrypted all the way to the device.

For services like Cisco Spark Messaging, since it is a cloud-based service these back-end changes will be seamless to users.  For customers running WebEx Connect/Messenger instead of CUP/IM&P as the back-end for Jabber, the changes will be handled there.  [Note that for customers using Jabber to Jabber VoIP calls with Connect/Messenger on the back-end will NOT want to turn on APNs today as Jabber will be terminated on background and no calls will ring through unless Jabber is in the foreground.  Keeping APNs off in Jabber will preserve the current behavior.]

APNs allow CUCM IM&P to send a notification to Apple to be pushed to the iOS device running Jabber if it is in the background or not running.  CUCM 11.5(1)SU2, Jabber 11.8(1) and Expressway X8.9.1 are the first versions that will support this method of notification.

This release of CUCM 11.5(1)SU2 enables IM notification via APNs.  A future release of CUCM IM&P is expected to add the Call notification via APNs which is expected to be released in the Summer.

Jabber IM Notification Scenarios

APNs is not required for ALL IM notifications.  It depends on Jabber’s state on the iOS device.

If Jabber is in the foreground, notifications will come directly from IM&P to Jabber as is currently done.  If Jabber is connected via MRA and in the foreground, notifications are relayed through Expressway as is currently done.

If Jabber is in the background or not running, notifications will come from IM&P, be sent to the Cisco Collaboration Cloud relayed to Apple APNs and then to the iOS device.

Once Jabber has been opened and is in the foreground, notifications will come directly from IM&P to Jabber.

Jabber APNs Flow

Because CUCM IM&P needs to be able to talk to Apple to send the notifications, CUCM will need to be able to talk to the Cisco Collaboration Cloud which will relay the notification to Apple.   This may be an architectural change for some customers.

Upon initial release CUCM IM&P’s connection to the Cisco Collaboration Cloud will allow for two connection methods:

1. Direct outbound access (can be through NAT)
2. Connect via a corporate proxy server with authentication.

Unfortunately the initial CUCM IM&P release cannot communicate through Expressway.  This is expected in a subsequent release of Expressway.

In typical customer deployments CUCM IM&P is only allowed to communicate with users on an internal network, or alternately it may be allowed to communicate with Jabber clients via Expressway MRA.  In both of these cases IM&P itself will likely not have NAT or firewall rules setup.  If a proxy server is not available, NAT and outbound firewall rules will need to be configured.

In some customer deployments done in the past CUPS/IM&P would be allowed to federate with other XMPP systems directly.  In this case, firewall rules/NAT are likely setup already and will just require some fine tuning to allow CUCM IM&P to talk out to Cisco for notification relay.   Note: Expressway started supporting XMPP federation proxy a few versions ago, so some customers may be relaying through Expressway and not directly NATing.

If a corporate proxy is available then CUCM IM&P will just need connectivity to the proxy and not directly to the internet.

IM&P will know whether Jabber is in the foreground and will determine if it should send the notification directly or via APNS.

Configuration

CUCM and CUCM IM&P must be at 11.5(1)SU2 or newer.  Jabber must be 11.8(1) or newer.  iOS can be on 10.x and use APNs right now for IM notification.  VoIP call notification via APNs will be deliviered in the future.

CUCM Publisher must have DNS resolution setup and working, then APN Service Enabled under Advacnced Features > Cisco Cloud Onboarding.  Make sure to select “I want Cisco to manage the Cisco Cloud Service CA Certificate required for this trust” if you don’t want ot have to deal with manually importing the CA certs for Cisco Collaboration Cloud communications.

Onboarding will create a unique oAuth token which is automatically distributed to all nodes in the cluster for communication with the Cisco Collaboration Cloud to relay to APNs.

Firewall – Outbound access using TCP 443 to fos-a.wbx2.com, push.webexconnect.com, and idbroker.webex.com

Firewall – Outbound access for iOS devices to connect to Apple.  If iOS devices are allowed out to the internet in your environment then no changes should be required.  If your iOS deployment restricts communication to Apple (e.g. WiFi network that iOS uses is restricted to internal network onl), then outbound ports will have to be opened so that the iOS device can connect to Apple for APNs.  TCP 5223 to 17.0.0.0/8 or on WiFi will fallback to TCP 443 to 17.0.0.0/8.

What does this mean to me?

If you’re running CUCM IM&P (aka CUP/CUPS) on-perm and Jabber for iOS and want IM notifications to still appear when Jabber is not in the foreground you are highly encouraged to CUCM 11.5(1)SU2 and Jabber 11.8(1) or newer before this Fall!  These changes will NOT be back-ported into previous versions of CUCM/IM&P.  I would suggest doing the heavy lifting of getting to SU2 now.   Go through the setup to use APNs for IM notification.  Jabber on iOS 10 will then use APNs for IM notification, and keep-alive API for call notification.

If you have Jabber users connecting via Expressway MRA, you will need to upgrade Expressway to X8.9.1 or newer before Fall.

If you are using SSO today with Jabber (where you are not allowing cached credentials in Jabber) and reauth is required for every re-launch of Jabber, then do not turn on APNs today.  Wait until later this summer for Jabber and CUCM releases which will have a faster logon mechanism before turning on APNs.  Otherwise when your iOS device receives an APN for an incoming call or message users would have to login via SSO and would likely miss the incoming call.

Given the significant effort required to upgrade a system between major versions, I am suggesting everyone begin planning their system upgrade now to get to 11.5(1)SU2.  Then make the easier hop to SUx before the Fall 2017.

 

Notes about Upgrading to CSR 11.5

Helped a customer upgrade from 11.0 to CSR 11.5, CUCM 11.5(1)SU1; IM&P 11.5(1)SU1; CUC 11.5(1)SU1.

Unity Connection 11.5

You must apply ciscocm.cuc_11.5SU1_pre_upgrade.cop.sgn before you upgrade to 11.5 because of bugid CSCvb02774.  The install of the patch is straightfoward and does not require a reboot.  I also ran a utils iothrottle disable to make the upgrade run faster (since it was being done after hours.)

If you’re upgrading from 10.x or earlier it is CRITICAL to increase your VM RAM to 6GB.  (This was something I ran into when going to 11.0.  If you leave it at 4GB  it will not function properly at all.)

The upgrade ran normally and took a quite a while for the switch-version to complete.

On a site note, I noticed that the new Unity Connection (CUC) 11.5 .ova files define a 200GB HDD for the bigger VM.  I investigated increasing my HDD from 160 to 200GB, but found out that CUC does NOT support dynamic resize of the HDD.  This will cause the partition to be unaligned and you’ll get to rebuild CUC from scratch.  So leave it at it’s current size.

CUCM 11.5

To save time during the upgrade window, the day before I preloaded the 11.5 ISO on my remote ESXi datastores so that it wouldn’t take forever for the ISO to SFTP over to the remote offices (they have limited bandwidth) , then I attached those ISOs as virtual DVDs to the CUCM servers via vShpere and then launched those upgrades as though they were coming from DVD instead of a remote file server.

The first attempt to launch the upgrade on the Pub failed with the old “common parition doesn’t have enough space” business.  I used RTMT to decrease the Low and High logging watermark to 45 and 40 respectively (and restarted the log partition monitoring service) to create room.

Purge Log Files by Changing the Log Partition Watermarks

• Another way to create additional disk space is by changing the high and low watermarks on the system. This informs Unified CM of the numbers of log files to purge once the watermark is reached. Use RTMT as follows:

1. Launch RTMT and log in to the desired cluster.
2. From the left pane, select Alert Central.
3. On the right pane, double-click LogPartitionHighWaterMarkExceeded. Change the threshold value to 40.
4. On the right pane, double-click LogPartitionLowWaterMarkExceeded. Change the threshold value to 45.
5. This data is polled every five minutes. Allow five to 10 minutes and then check the drive partitions for additional disk space by using one of the methods described above.

http://docwiki.cisco.com/wiki/Unified_CM_L2_Upgrade_Disk_Space_issues#Purge_Log_Files_by_Changing_the_Log_Partition_Watermarks

 

As usual, I ran the Pub first (without switching version), when it completed, I ran the Subs (also without switching versions).

If you’re coming from 11.0, the utils iothrottle disable command is not necessary.  (You can try to run it but CUCM 11.0 tells you it is unneeded.)

I rebooted the Pub and then Subs as normal.

IM&P 11.5

This was also a typical upgrade.  The switch-version took a LONG time for services to come up on the reboot.

 

 

8800 Series 11.5(1) Firmware – Enhanced Line Mode

11.5 firmware for the 8800 series phones is on CCO now.

It comes with a bunch of cool new features.

Enhanced Line Mode

The coolest and most useful that I’ve seen requests for is the new Enhanced Line Mode.  (I’ll abbreviate it ELM even if there’s overlap with the PLM/ELM acronym.)  ELM allows all 10 buttons on the phones to be used a programmable line keys (PLKs).

The mode we’re used to includes 5 PLKs on the left, and 5 context-sensitive function keys on the right.  I like this mode, having gotten used to it back in the day with the 9900 series phones, but hear customers who need more than 5 PLKs (in particular for admin/receptionists who want more than 5 BLFs or Shared lines).

 

As you can see in the picture, I can now use all 10 buttons.

While the firmware is out now, there is a Devpack required to enable the ELM feature on the device configuration page.

The release notes indicate that you should get the latest Devpack from CCO, install it and reboot the cluster to enable the ELM setting.  The challenge you’ll hit is that the latest Devpack on CCO as of today (mid-June) doesn’t actually include the QED file that enables the ELM setting.  The Devpack that inclues the QED will release in the next couple weeks.

Look for a Devpack with a late-June/early-July date stamp if you want to turn this feature on.

Enhanced Do Not Disturb

The DND function has been updated to be much more obvious which is nice.

Other features to mention:

• Wi-Fi Security Enhancements
• Customized Dial Tone for SIP Phones

See the release notes for more information about the last two.

 

 

8800 Handset Firmware 11.0(1)

One common criticism of the 8800 10.x firmware was it’s behavior in truncating Line Text Labels when there was ample space on the display to show more characters.  Version 11.0(1) fixes this:

 

 

(Sorry for the mediocre photo.)

The Line Text Label supporting 30 characters (the maximum supported in the LTL field in CUCM) on a single line on the display.  There’s some cool stuff coming in the next release of code which will add even more functionality to the 8800 series.

Firmware 11.0(1) adds the following features:

 

• Application Dial Rules
• Audio Voicemail Access from Visual Voicemail
• Barge Enhancements for Cisco IP Phone 8800 Series
• Enhanced Debugging Options
• Mobile and Remote Access Through Expressway
• Problem Report Tool
• User Interface Enhancements

The highlights being the ability to use the BIB for Barge, and official support for MRA.

 

DX 10.2(5) Firmware Released

Are you about ready to throw your DX out the window because dialing with the Phone app is an unresponsive nightmare as it tries to do a lookup on every call it’s ever made?  🙂

Well the fix is out.  The Phone app is revamped and actually doesn’t suck.

I’ve been running the 10.2(5) beta firmware for a few weeks and have been much happier with my DX80 and 650.  The final build is now out.

Would you like the DX to stay on your PC input when a call comes in?  It finally does!

Would you like HDMI audio from your PC to actually come out the DX speakers?  You got it!

Release notes are here.

One thing to note for MRA users.  The DX only trusts public CA certs like the 8800 series now.  So if you’ve deployed your Expressway-E with private certs, you’ll bust MRA on the DX if you dont move to public certs.

Catalyst 3850 IOS 16.1.1

Upgraded my 3850 to Denali 16.1.1 today. The new unified web GUI for the whole switch is pretty slick.  There are a ton of new features. See the Release notes here.  Now this is the first release of 16 so I don’t know I’d jump on it for a production network until there is a bug fix rebuild, or at least without checking the open caveats.

I ran into a problem that the APs would register, the radios just wouldn’t come up. Nothing I tried in the GUI (like bouncing the radios between disabled and enabled) would get them to go Operationally UP.

Turns out the secret was to go into the guy and do a “no ap dot11 5ghz shutdown” and a “no ap dot11 24ghz s.”  The radios came right up and everything is working again.

Registering an SX-10 (and TC-based endpoints) through Collab Edge MRA

Recently I worked on an MRA deployment using SX-10, MX-300 and DX-series (650/70/80) endpoints.

I had Expressway-C and E working successfully for 8800 series MRA, but needed to get the TC-based and DX-based endpoints to register.  This turned out to to involve some issues that I wasn’t expecting.

TC-based endpoint registration

There isn’t a lot of documentation  for TC endpoint registration through MRA (since traditionally it’s been registered to VCS through VCS-E).  The best documentation that I could find was here:  http://www.cisco.com/c/en/us/support/docs/unified-communications/expressway/118696-config-cucm-00.html

I began the deployment by registering the MX-300 I use directly to CUCM to make sure I had it successfully working before attempting registration through MRA.  I’d previously had it registered to a VCS-C.

MRA requires TC 7.3 code, so I decided to deploy 7.3.4 since it is the latest bugfix version.  I downloaded and deployed the COP file to CUCM since CUCM will be in control of what version of software the TC endpoint will use once it’s registered to CUCM (like typical phones get firmware).

Note that TC 7.3.3 or greater firmware have different functionality for remote screen monitoring of systems!  TC 7.3.3 introduces the requirement to have an option key for remote system/screen monitoring of the TC endpoint.  You’ll need to work with TAC/Cisco to get option keys cut if you are doing remote screen monitoring before going to 7.3.3+.

Registration to CUCM was straightforward.  I defined the device on CUCM as you would normally define a phone, picking the appropriate SIP profiles.  I didn’t do secure registration as this is optional.   (The documentation above does mention that secure registration is optional, but the example works through a secure registration.)

I made sure to set the device association my end user in CUCM.  This is important for MRA later.

Once the MX-300 was registered and making calls successfully through CUCM, I moved it out to a general internet connection to work on MRA.

On the touchpanel I launched the Provisioning wizard and selected Cisco UCM via Expressway.  After putting in my credentials I was greeted with this error:

After doing quite a bit of research and looking at the detailed error logs from the MX-300, it turns out that your Expressway-E certificate must also include a SAN for the domain name itself (e.g.  yourdomain.com).  The error actually indicates that it wants a collab-edge.yourdomain.com SAN:

Edge TLS verification failed: Edge domain ‘yourdomain.com’ and corresponding SRVName ‘collab-edge._tls.yourdomain.com’ not found in certificate SAN list.

The challenge I had is that the certificate I’d bought from GoDaddy for Expressway-E (that was working with 8800 MRA) wasn’t a UCC or multi-SAN certificate and you need to have at least the Expressway-E as the CN and the domain as a SAN.

At this point I decided that it was time to move from GoDaddy to DigiCert since they have unlimited resigning of certificates without having to revoke any of them.  This essentially allows you to create as many certificates as you want without having to keep buying more like GoDaddy.  I bought the Wildcard Plus certificate and used it to create a multi-SAN certificate for my Expressway-E.  The CN is always *.yourdomain.com, but you can add a bunch of SANs (like 20 or so.)

I generated the certificate with the following SANs – edge.yourdomain.com, yourdomain.com, collab-edge._tls.yourdomain.com and _collab-edge._tls.yourdomain.com.   One of the partner engineers that I talked to said that he got it working without having to add either collab-edge SAN.  (I’ve not looked into why/when we’ll need the collab-edge SAN and if it is actually be collab-edge as the error indicates, or if it needs the preceding underscore on collab-edge like the SRV records have.)

After applying the certificate to Expressway-E and rebooting it I tried the provisioning wizard on the touchpanel again and was greeted with the SAME ERROR.

It turns out that I hadn’t included the DigiCert root and the DigiCert Intermediate cert in the list of Trusted CAs on the endpoint itself.  The documentation indicates how to install it here – http://www.cisco.com/c/en/us/support/docs/unified-communications/expressway/118696-config-cucm-00.html#anc10  Make sure you have both root and intermediate (if the CA you used signs with an intermediate) on the TC endpoint.

After this the MX-300 registered like a champ and is able to do calls.  I followed this same process to get an SX-10 registered as well.

The partner engineer I talked to said he had to work through a couple issues on his test:  1) Endpoint rejecting the user credentials when running through the provisioning wizard.  Make sure the endpoint is associated with the End User and that the end user has CTI Enabled and CCM End user.   2) Getting an http download error after getting through the initial expressway authentication and that it was caused by the endpoint needing to do a _cisco-uds.yourdomain.com lookup internally to find out where CUCM is to download it’s configuration.   He was in a split domain situation and didn’t have a _cisco-uds record for the external domain on the inside.

I’ll detail the adventure for the DX-series on another post.

 

 

 

 

CCX and Logjam

The latest versions of Firefox and Chrome seem to have fixed the Logjam vulnerability which is causing issues logging into the CCX admin page.

Here’s the workaround:
>> Now for logjam exploit related to the Diffie-Hellman algorithm the workaround in place is as below :
1)    In FireFox, enter “about:config” in the URL field and press enter.
2)     Accept the “This might void your warranty!” warning
3)     In the search field at the top, enter “security.ssl3.dhe_rsa_aes”
4)    Double click each result (128 and 256) to toggle the Value to “false”

After accessing the page and doing what you need, you’ll want to set them back to true or you’ll be vulnerable.

Another option is to keep a copy of the worlds greatest browser ever, Firefox 24, installed for accessing CCX.  😉