Skip to content

How to evaluate SaaS proposals?

September 23, 2011

There are many ways to complicate a procurement process for a service in general. Apart from these there are some specifics to “Software as a Service” offerings. I’d like to point some of these out in a do or don’t fashion. Let’s start off with the “don’t section” so that I can end on a positive note.

Some of these could have ended up in either section or even are in both sections. I have tried to avoid entries as much as possible.

Don’t do this

  • Go from evaluation straight into usage

It is worth evaluating a technology or a cloud service to better understand it. Just make sure that you do not go from evaluating into productive use with stepping back. You need to run this through a formal analysis and market comparison. You may create a strategic decision by simply turning an evaluation into a live system.

  • Use a cloud service without evaluation on your terms

For classic IT technology or service decisions it is common practice that a project and evaluation is executed. These usually take place on the customers premise and are based on an evaluation plan. With cloud services there is the tendency to drive a different momentum. They grant you a 30 days trial which would run from the first logon onwards. By the time you have made up your mind on what to evaluate, who should evaluate, etc. the 30 days are gone. The argument then is that you can purchase it for a small amount of money and drive the evaluation onwards, but you pay for it. Also you probably get very little support as it is a finished non customizable service in the web. Forget that. You should drive a full and proper evaluation (see Do section below)

  • Treat the SaaS decision like a technology decision based on identical criteria only

How to compare apples to pears without getting fruit salad? That is the key questions. On the one hand you want a fair evaluation of the new and the old world on the other hand some selection criteria from the old world just do not apply to the new world. So how to find out which criteria do still apply and which should be discarded? That is a difficult one you probably need help with. It is also closely connected to the next bullet.

  • Let your administrators and technology people decide on their own

Cloud computing is not a technology direction but a business decision as much as outsourcing is not about bits and bytes but dollars or Euros. Therefore you should ensure that the decision board is not dominated by either the people administering the current system (they would vote against it at all cost) or technology driven people (which would jump on anything new). You need both groups for a proper evaluation but also the IT business management with a broader perspective (the CIO office).

  • Let others create fear in you

There is a common trend visible right now. The providers of classic hosting and outsourcing offerings are openly discrediting cloud offerings. Their main arguments are data privacy and playing to the already existing fear about the US Patriot Act. Do not get fooled by that. They are not playing by the rules here and the only way you can go beyond that is by discarding their stories and evaluate for yourself.

  • Forget to discuss end of contract Ts & Cs

It is important to have a clear understanding of the terms and conditions at the end of contract. You as well as the service provider might assume that you will never leave the service but let’s face it that is an illusion. Getting a clear and documented understanding about grace periods, means of data access, access by third parties, etc. is utterly important. Imagine you have terabytes of data in archives in the cloud. Could, should these be moved? If they are not moved, how can they be accessed? For how long at what cost can they be accessed? This is just one area that is important. If you have high cost of moving off the service you need to add these to the business case!

  • Fall for the common misconceptions about cloud computing

There are so many wrong rumors or pitfalls that could mislead you. Beware of them

Common Misconceptions – Fear, Uncertainty and Doubt

Common Misconceptions – Sales, Sales, Sales

  • Wait and do nothing

Cloud Computing is not something you could wait out. Waiting was already a bad concept when people said they are going to wait for the first Service Pack for Microsoft products. You cannot wait it out. If you have a business challenge, a change or an investment decision on your doorstep you must consider cloud computing. Important: Must consider and not must use! If you do not consider and use it where it drives a benefit, your competition will do it.

Do this

  • Involve the experts / Get help

This suggestion is twofold and both parts are of utmost importance.

External help

Get help from outside much like you would do so if you would evaluate outsourcing. The field of cloud computing is broad and the decisions criteria vary. Either get help from a consulting company or a cloud computing expert (like myself if you allow that much advertisement).

Involve the internal experts

Especially when it comes to legal discussions and risk assessment question better involve your legal department. I have seen to many people acting on half-knowledge about data privacy and surrounding questions. Make sure you have the experts at the table.

  • Trial before evaluation

Sounds strange? Actually it is not so strange at all. Prior running a full-fledged evaluation you might want to task some very few folks to run a trial to determine whether the service would fit at all? It is not worth driving all the effort of an evaluation if basic functionality is missing. So make sure that you know that the service would work for you in general.

  • Evaluate on your terms

Create an evaluation plan and talk to the service provider to jointly drive this evaluation. Involve a partner where the cloud service provider has no local organization but do not limit yourself to the 30 days. For enterprise customers there is flexibility to go beyond these 30 days. To get these you need to have a good evaluation plan and the evaluation groups already set up. If you struggle with the setup ask your service provider for support (technical as well as programmatic). Also ask your service provider for reference customers that you can connect with. You should not only do so with regards to whether they bought the service or not but also their means of evaluation.

  • Understand the SLAs and KPIs in detail

99.9% availability sounds good? You’ll never know until you understand the details. One example: 99.9% measured monthly versus measured annually makes a huge difference in maximum downtime that could occur during one month. Also some companies also count downtimes of 10 minutes or more or only if a certain percentage of users is affected. You see there are many traps when it comes to SLA’s and KPI’s. Make sure you understand them in detail and if in doubt do not fear to ask!

  • Create a tailored RFP

I have seen many just using a standard RFP that they used in the past for outsourcing or technology procurement. This is a guarantee to drive proposal teams crazy or could service or even to opting out of the bidding completely. You need to be aware of the limitations of SaaS and cater for these in the RFP or at least leave room for explanation. I have seen excel list with questions allowing yes or no which were completely unusable for a cloud service. But we were still asked to fill them out. These lists would not help you making a decision either. So beware of your questions they could create value as much as destroy value quite quickly.

As with the misconceptions posts this has the potential to grow into a series. Any feedback is welcomed especially on additional points for future posts. Credit to you is guaranteed.


  1. Thomas permalink

    Great stuff and the author is definitely bringing ‘must think about points’ to the table.

Trackbacks & Pingbacks

  1. Cost Savings only as a driver for the move to the cloud? «

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: