IBM CECC – Flawed Platform for ISVs, Developers, Vendors, etc.

Earlier this year, IBM shutdown its IBM PDP (Power Development Platform), originally called IBM VLP (Virtual Loaner Program), and replaced it with IBM CECC (Client Experience Centers Cloud). IBM does loves their acronyms!

I appreciate IBM supplying virtual VMs for developers to use to port their applications to AIX, IBM i and Linux on POWER, so I will try and be polite with my criticism of IBM CECC but IBM is making it really hard for developers to use CECC.

Since this is a (very) long blog post, I’ll get to the point early on and you can continue to read the posting if you want to. My opinion of IBM PDP would be an ‘A-’ (great except for getting LDAP/PAM libraries installed). My opinion of IBM CECC so far is ‘D+’. If you are also a ISV, developer, vendor, etc. using IBM CECC, please lodge your complaints with IBM CECC support, so that IBM will fix the issues especially considering the fact that you can no longer save and restore VMs. It almost makes using CECC pointless because who wants to spend a week setting up the VMs only to have them deleted when the reservation is done.

History: I was introduced to IBM VLP back in 2005 when I took an IBM PartnerWorld course on administration of Linux on POWER at IBM’s head office/training facility in Markham, Ontario, Canada.

At the time, I had my own AIX servers (5.1 & 5.3) and knew next to nothing about IBM i. A couple of years later, I started to use an AIX v6 VM on IBM VLP rather than purchasing my own AIX v6 server.

In late 2008, a customer purchased licenses for MQAUSX for AIX but said they wanted to also secure MQ on IBM i and asked for MQAUSX to be ported to IBM i. I thought, “how hard can it be, I already knew Unix, Linux, Windows and z/OS”. Well, IBM i is truly a very strange beast. I spent several weeks trying to figure it out and was about to quit when I saw a course at Seneca College called “IBM i System Administration”. So, I figured I better take it. It was the best $600 I have spent. By no means am I an IBM i expert but at least now I have a basic understanding of this strange beast and can compile, link and test my applications on IBM i.

The really nice thing about IBM VLP later renamed to IBM PDP is that once you installed the software you need, the VM image can be saved and redeployed in a future reservation. To setup and configure 3 brand new reservations in IBM PDP (AIX, IBM i and Linux on POWER) takes about a week of my time. That is why saving the images and reusing them in future reservations is EXTREMELY important to developers, ISVs, vendors, etc. like me. But in IBM’s infinite wisdom, they no longer offer the ability to save and restore a VM image in CECC. This is truly one of the most developer ‘unfriendly’ things I have ever seen.

Here’s a snippet of the work I do when starting with a blank/default VM image with the goal of building and testing Capitalware products:

Task	AIX	IBM i	Linux on POWER
Upload IBM MQ 9.2 software
Upload Quest Authentication Services software
Upload Centrify’s DirectControl software
Install compiler
Install IBM MQ 9.2
Install LDAP development libraries & modules
Install Quest Authentication Services development libraries & modules
Install Centrify’s DirectControl development libraries & modules
Install PAM development libraries & modules
Create 2 queue managers for different scenarios
Define channels, queues and topics for the 2 queue managers
Create build/staging framework
Create deployment/packaging framework
Upload Capitalware source code
Compile and link all Capitalware products
Perform testing scenarios for the various products
Package products

Legend:
– Green check marks are tasks I had to do
– Orange filled squares are tasks include in the VM image
– Gray filled squares are not applicable tasks
– Red flags are tasks that I could not do and CECC refused to do.

Note: IBM i already has a compiler and LDAP libraries ALREADY installed. It is actually developer friendly!!! Woo Hoo!

The first image I started with was AIX on IBM CECC and quickly discovered it was missing a compiler and LDAP development libraries, so I opened a help desk ticket and requested that they be installed (including a list of LDAP filesets needed). Here is the response I received:

Please use the below link to download and install xlc compiler, which will be available for 60 days of trial.
https://www.ibm.com/us-en/marketplace/xl-cpp-aix-compiler-power
Also, go through the user guide which is available below and search for nfs and mount the ISO which will help you to get the LDAP client packages.
https://www.ibm.com/it-infrastructure/services/cecc-portal/static/docs/CECC-Portal-User-Guide.pdf
And step by step installation guide for LDAP – https://www.ibm.com/support/pages/ldap-aix-step-step-instructions-installing-ldap-client-filesets-aix

Say what?!? I am a developer who spends 99% of their time writing, debugging and testing code. I am not an AIX SysAdmin. I know next to nothing about smit or installp on AIX. They want me to download, install and use a trial version of the compiler for AIX. WTF!!

Oh yeah, when I went to follow the instructions on page 12 to mount the NFS share, I get errors. When I complained to CECC support about getting errors by using the commands from their document, I got the following reply:

We suspect you had done a copy and paste and may have had some residual data when you tried to mount NFS. Here is a successful NFS mount for reference.

Dah! Of course, I copied & pasted the commands, that is how you avoid typos!!

Command in manual:

nfso –o nfs_use_reserved_ports=1

Command in email:

nfso -o nfs_use_reserved_ports=1

Can you tell the difference? I had to clean my glasses then I noticed that the hyphens (‘-’) were different. Whoever created the CECC User Guide was not very careful and changed the hyphen. This is something that DEFINITELY should be fixed in the CECC User Guide.

So, I started smit and it took me probably 5 tries to get the directory correct so that smit would read the package information. I found the LDAP fileset but I also found AIX compiler XLC v9. First, I was surprised then mad because the CECC support made me go off and download (and upload) a trial version of XLC. WTF!! What kind of support is that?

I finished up on AIX then moved on to Linux on POWER. Again, no compiler and no LDAP development libraries (nor PAM development libraries). So, I opened another ticket and requested a compiler and LDAP development libraries. This was the CECC support response:

We would like to inform you that we have installed C compiler on your reservation. To install LDAP development libraries, you should download the rpm package and install it manually. Use below link to download the rpm’s
https://rpmfind.net/linux/rpm2html/search.php?query=openldap2-devel-static&submit=Search+…&system=&arch=

Yeah, a compiler but you want me to hunt and peck for individual rpm packages when the Linux SysAdmin already has the SUSE development DVD or image and could easily use YaST to perform the install which would tell you about all of the required prerequisites. I downloaded the 5 rpms that I know about which required more rpms which I downloaded, which required more rpms which I downloaded, which required more rpms which I downloaded, which required more rpms which I downloaded, etc. and I just gave up. There is only so many hours that you can go around and around wasting time.

I didn’t even bother asking for the PAM development libraries because I know I’m not going to get any support from CECC.

So I finally moved on to IBM i. Surprise, surprise!! It has a compiler installed and as an added bonus, it has the LDAP development libraries already installed. Surprisingly, I had the least amount of problems with my IBM i VM.

On Monday, I extended my 3 reservations for AIX, IBM i and Linux on POWER to the weekend because I could not figure out how to save the 3 VM images for future use. I spent a lot of valuable time setting up these images that I could have spent doing my regular work of writing, debugging and testing code. I opened a ticket regarding about where the option is to save the images. And CECC support responded with:

a decision was made when it was created not to support the save image functionality.

there is no “save image” functionality. We provide “persistent storage” in the form of a NFS share that you can store files on. There is a separate persistent storage user that owns the storage and must be used for copying files to / from it. The automounter is setup to mount it and in the persister user home directory there is a symlink to the mount point.

WTF!!!!!!!! I spent probably 40 hours (a full week) setting up these 3 VM images. What a total waste of time!!

Clearly, IBM has a case of the left hand does know what the right hand is doing!!! (referring to CECC and PartnerWorld) I constantly get emails from IBM PartnerWorld and IBM POWER people about porting and/or testing applications to/on IBM POWER platforms. i.e. AIX, IBM i & Linux. And on the IBM CECC overview page it says:

The first item is “application porting” but it would seem that IBM CECC prefers to frustrate the crap out of developers because I don’t know any ISV, developer, vendor, etc. that wants to spend days installing software every time you need to compile and debug a program.

Capitalware has created and sells 16 programs. At least once a week, I get a bug report for a product. So, how am I suppose to support AIX, IBM i and Linux on POWER, if I have to spend so much time installing software every single time I start an image. It is ridiculous. Why would I even bother supporting AIX, IBM i and Linux on POWER??????

Does IBM PartnerWorld want ISVs, developers, vendors, etc. to use IBM CECC to bring their applications to AIX, IBM i and Linux on POWER or NOT!!! Because clearly, the management at IBM CECC is NOT actually interested in providing a useful platform for ISVs, developers, vendors, etc. who WANT to bring their applications to AIX, IBM i and Linux on POWER.

I’m calling on all ISVs, developers, vendors, etc.. Please lodge your complaints with IBM CECC support, so that IBM will fix the issues and in particular, fix the issue that you can no longer save and restore VMs.

Regards,
Roger Lacroix
Capitalware Inc.