Fabian Schonholz’s Blog

I despise talking to customer service. Most of the companies I have had to call either for information, to report a problem, ask for a refund, get an RMA, or anything related to service, have resulted in horrible experiences. The customer service reps have been rude, they have lacked product knowledge, they have been less than attentive and willing to listen and have made no efforts in trying to find a solution that worked; to make matters worse US based reps can barely speak intelligent English. To the above equation you need to add reps not based on the US. It is not the accent that bothers me - mine is so thick you can cut it with a knife - but the lack of a customer centric culture. The accent just gets further in the way and aggravates the situation.

There are two companies where the experience is 180 degrees in the other direction: USAA and LaCie.

To call USAA’s customer service exceptional is to not do them justice. They are superb. I am not sure how the reps are trained, but I am yet to talk to a rep on the phone and not gotten the help I needed. What impresses me is that when a rep does not know the information, they freely admit it and they are not exactly apologetic. However, they know where to go get the information and who to hand you over to.  The hand over from one rep to another is also fantastic. The first rep introduces you to the second. The second greets you and the first one asks the second if he/she has you. If the answer is positive, the first bids you good bye and now you are talking to the second rep, who had been fully briefed before you started talking, thus, not having to repeat yourself. Of the few times I had to call USAA - they have a great track record - and in those few times the experience has been consistent: Great customer service every time I call.

The experience with LaCie was completely different. I called to complain about an order I had placed where one of the items was back-ordered. My complaint was that I had been charged for the back-ordered item even though it had not been delivered, or so it seemed. The customer service rep, although I was very short, was nice, cordial, composed and quickly turned the situation around by being understanding of how I viewed the situation. He very quickly changed the mood and tone of the call and resulted on a happy customer. I am not sure if that is the experience I will get next time. Regardless, it was a pleasant one. The most important part of this experience was that even though I was in “the wrong”, I was never made to feel that I was wrong.

Few other experiences rival LaCie’s. So few that I can count them with one hand and have change. By at large, my experiences are really crappy and frustrating. The worst experiences are the in-store experiences. Two companies are notorious in my book: CompUSA and Fry’s Electronics. I will not go into details of these experiences because there is nothing to learn from them and in all honesty, I would waste your time describing them. But let’s just say that the reps where less than intelligent; their knowledge of what the products they carry is near nil; their interest in taking care of you is non-existent; their personal hygiene and presentation also lacking. And their vocabulary … well … let’s just say that my children have a better vocabulary than the people I encountered have.

I completely understand that the opportunities for education these people have had are not, to any degree, comparable to mine or my children’s. And I do not blame them for their lacks. I will, on the other hand, make them responsible for it. Who I really blame is the store managers (who probably also do not know any better) or regional managers. I blame people all the way to the top. They are the ones that lack customer focused service and since they lack it, they can not expect their chain of subordinates to react any different than they currently do.

A while back I was recommended I read a book called “Raving Fans” by Blanchard and Bowles. This book is a good example of why USAA and the CS Rep at LaCie are so effective in providing exemplary customer service. I recommend you read it. To think of it, the person that recommended it should read it again. His organization’s customer service is beyond lacking to the point that Fry’s and CompUSA’s in comparison are not too bad.

The customer service landscape today is, based on my experiences, a minefield with a few safe havens. But it needs not to be such a disaster. Traditionally customer service had information issues. In other words, a customer service rep lacked complete and accurate information. And when the data was available, it was hardly ever integrated and presented in such a way that helped the rep. Once upon a time I used to work for Prudential Group Insurance, West Coast Operations. My main responsibilities were to provide technical assistance to customer service reps (CSRs) and help them navigate a series of disconnected mainframe based systems. This was in 1994, ages ago in internet times.

Fast forward to today. CSRs’ operations are no longer, for the most part, mainframe based, and most system have been integrated in such a way that the information is presented in a series of screens that make life much easier to find. To make matters better for CSRs, many of the system offer key-word search to assist finding information more efficiently. A clear example of some of these advances is banking. When you call your bank, in many cases (BofA Credit Card Services for example), you are connected to CSR in India, The Philippines, and other. The rep has access to a great deal more of information regarding your account and transaction history.

Putting the cultural and language elements aside, the first issue starts with security. Some person in some country half across the world has access to some of your most important financial information. But that would not change if the CSR was located here in the US, or for Europe in Europe or however local. What would change is the ability to do background checks. Secondly, what level of encryption in maintained for the connection between the outfit in India, for example, and the data repository in Colorado, again for example? If it was local, then there are regulations that need to be observed and regulatory bodies that conduct audits. Although these regulatory bodies extend their scrutiny to vendors and providers, I am not sure of the level of efficiency and transparency rendered in the above mentioned audits.

However, security, technolgy and data/information even though paramount, the problem remains with my chief complaint: Lack of customer focus on the part of the CSRs. And with the information they have at their disposal and the installed systems providing the information, I am utterly surprised service is still lacking.

In an early blog posts, Building Scalable Web Systems, I discussed very high level some of the needed premises and basis to architecting scalable systems. What the post did not deal with is insurance and Downtime. What is the point of scalability if you have downtime and what is the business continuity plan that maximizes available resources. Also, the post does not deal with success. What happens and what tolerance does the business and market have in the case of massive and rapid adoption. How do you deal with it?

Enter cloud computing and Amazon’s EC2. For those not familiar, EC2 is a cloud environment that provides virtualized hosting services. They provide the hardware infrastructure, the pipes, storage and other services. You provide the application. The promise is that you can scale the hardware need horizontally without having to deal with the hardware itself and its management and upkeep.

The first question is whether I believe it is 100% ready for prime time. You can argue that loads of companies are using it successfully, thus, it is ready. I have talked to some of them to mixed reviews. You can argue that some of the unconfirmed rumors are to be believed because there are indications of truths, thus it is not ready.  Also, I have talked to some people that were not all that happy with EC2. So on and so forth.

The second question is whether it matters or not if it is 100% ready for prime time. And  on the hills of this question, can it be used as a business continuity tool. I will answer both below.

The obvious third is regarding cost. Through all my calculations (and other people’s), EC2 can be more expensive than running your own systems - of course at some external data center. But some of the advantages come around quick adaptability, separation of concerns, system automation and self healing procedures. I will go into more details on this later as well.

Let’s start with the first question: In my opinion EC2 is not 100% ready for prime time. It is a subjective opinion based on my findings and my level of comfort. Part of the decision is based on cost, but mostly on technical merit:

• Full virtualization is not where it needs to be; although there are ways to set up virtualization in the right configuration to make it not only more stable but also better performing. Not knowing EXACTLY how EC2’s virtualization layer works (and I am assuming virtualization) creates a big question mark on how things will truly stand up to friction. For example, it is hard to optimize a virtual machine to run DB servers  that deals with millions of queries a day. Hardware optimization is important with relational DBs.

• Virtual NICs have sort-comings. They collapse under high traffic. The way to overcome this “limitation” is by attaching each virtual NIC with a physical NIC. However, this defeats the purpose of virtualization and limits, the theoretical unlimited number of VMs you can have running on a single server (only as many as you have physical NICs minus 1; you need one NIC for the host Operating System.)

• Let’s not forget performance.  Even though you can create a limitless amount of VMs, the performance of each VM degrades with the provisioning of each new VM on a single server. What I do not know, however, is if there is an optimal number of VMs. In other words, is there a hard limit where before reaching that limit each VM would not change its  performance characteristics regardless of number of active VMs? Not too long ago I ran a virtualized farm. Unfortunately the application I inherited was so horrible that it superseded all problems we had with the environment. So, I can not even begin to answer the last question. Needless to say that the application and environment were replaced.

• But it is not just the DBs that need “specially” optimized hardware. Application servers as well. Maybe not as specialized but a slow processor creates drag. And adding many VMs to spread the load creates more management and more moving parts adding to the risk management factor and what can go wrong.

Continuing answering questions … YES!!! It does 100% matter that they are not ready for prime time. But really, what we need to ask is the degree of how much it matters. How far is EC2 from being 100% ready? I do not know, but they look darn close. By adding granularity to the question we come up with multiple degrees of “how much it matters”. 100%?, 90%?, etc. In the case of EC2, I think it matters less than 20%. They seem that close to being ready - by my definition.

We can define cloud computing in many ways, however, let define it by a behavior: it needs to work like the electric company. Using Bob’s analogy, we do not really know how many generators the electric company has. We just know that we want/need more juice, we plug to the wall and we get more juice. The more juice we use, the more we pay. In the case of EC2, it seems to work that if you need more capacity, you provision a new “machine” and off you go - well, sort of This creates the idea that if you need more juice, plug to the wall and pay at the end what you consume. Not considering cost, it looks like an attractive proposition. But more importantly, think in terms of what it can do for you. Almost instant scalability when you need it and how you needed it.

A little digression …

I do not worry anymore about scalable systems. I know how to build them; I have come up with a methodology and an architecture philosophy and I have repeated the  implementation of the methodology and architecture philosophy with great success. However, while my architectures scale horizontally without much of a inconvenience, the problem of scalability has become an issue of “need” predictability and time for procurement. Now in English: How much traffic will I get and how long does it take to get the hardware and deploy it - I consider real estate and power procurement as part of deploying the hardware.

Over the course of my experience I found that I need 3 running months to predict needs 3 months ahead. I have reduce the problem of CAPEX planning to getting right the initial installation. This initial installation needs to have “enough” capacity to support 3 months of capacity needs. But … what will be the capacity needs on the first three months? On a web based system, it is somewhat unpredictable. Sure, we could plan marketing campaigns designed to “limit” traffic. However, why would you limit and control traffic - there are a great deal of arguments in this area - if you have the potential of being ultra successful.

There is also the argument of cash flow and spending the right amounts of cash on your infrastructure. Funding is a resource and needs to be maximized. Any hardware that is bought today that is not used and needed - Software as well, but to a lesser extent - depreciates and for less cash you can buy something better in the future when the resource is truly needed. Therefore, the initial deployment of hardware becomes not only critical from a capacity point of view but also from a “capital resource” point of view. This is not to suggest, however, that you should not deploy for capacity needs earlier. In other words, stay ahead of the curve. Deploy 3 to 2 months earlier than  needed. What I am suggesting is that you do not need to deploy hardware beyond 3 months or more.

Back to EC2 …

EC2 not being 100% ready creates a problem compounded by the fact that it seems to work and it seems a short ways away from being the real deal. I resolved the problem by thinking, with Bob’s help, of EC2 as an insurance policy and a business continuity plan: I will build my staging environment on EC2, even multiple staging environments.

Let’s define a staging environment as a facsimile of the production environment but scaled down. The facsimile, if at all possible, must contain ALL components.

How to set up an insurance policy and business continuity plan using “the cloud”.

First, let’s look at process and environments. I advocate and implement total separation of environments as part of my Software Development Methodologies. Developers work on their workstation and QA Engineering occurs in isolated environments that in some way represent as accurate as possible production. Staging is the environment where UAT (User Acceptance Testing) occurs and where the build is certified and readied to release. Once it is certified, it is released to production. Staging must be not as accurate as possible, but precisely a facsimile of production. By hosting the staging environment on EC2 - or any such cloud environment for that matter - you can have that precise facsimile at a small cost.

Let’s consider the case of wild success and the fact that it is hard to predict and the capacity needed to “potentiate” success. In this argument I will equate “success” to a “disaster” and how we not only recover from it but also ensure continuity:

If traffic spikes past available capacity, not only does the user experience degrades but  it disappears altogether. In this case, virtually a “disaster” happened since the service becomes unavailable. In this particular disaster, having the right amount of hardware would have prevented it; as we discussed above, however, this is not always easy to determine. Just like in any disaster, the speed of recovery is vital to the continuation and success of the company.  If staging is indeed 100% a scaled down facsimile of production, then on an environment like EC2 scaling up in order to provide “capacity” should be a matter of minutes to just hours and not days. Basically, enough tolerance for the business not to experience a catastrophic downtime. Temporarily moving the production environment from self managed to EC2 provides the company with the necessary time to build out, and potentially better plan, capacity on its facility. Once the “disaster” passes, production can then be moved back from EC2.

In order for this temporary migration to happen seamlessly and effectively a high degree of automation needs to be incorporated into the overall infrastructure from day one. While the last updated staging environment (there can be multiple) will have the latest code and basic configuration, its data will be not current or accurate. Data migration needs to happen on a regular basis, and all staging environments should have, based on the installed release, the latest data set. Not only the data updates must happen automatically, but the discipline of automation, from a “disaster” detection to recovery must be as automated as possible. Once an issue is detected, a single script needs to be run to get the new production environment ready for operations, including needed changes on DNS, load balancing and firewalls. Furthermore, provisioning and  de-provisioning new VMs should also happen as automaticly as possible based on capacity needs.

The last part of this EC2 consideration is cost. It is more expensive than it looks. Once you start racking up the VMs on a per hour basis, racking up traffic at a premium cost and racking up storage, the $0.10 to $0.40 price ranges start to add up. This is cost that you incur every month and that you can not “lease”. So, does it add up to more than what it would cost you to build it and manage it yourself? No, but the costs are comparable, at least in my calculations. Therefore, running on EC2 for 1 to 3 months, even though duplicates the expense for that timeframe, it does not, in theory, break the bank and provides insurance, albeit, at a premium cost.

I have some strong opinions on how technology should be implemented. I do not care to know the secret sauce, but I do want to know in more detail than just general terms how things work. Especially if I am going to bet my company on a platform. The unknowns, the uncertainties based on lack of SLAs and the assumption around virtualization make me a tensed CTO. The result: Not 100% ready and trustworthy to build an company on it. I admit, however, that it is very impressive what they have accomplished, it makes sense, and of the other commercially viable cloud environments (I am not including Google, Yahoo! and MS), EC2 is the only one that, again in my opinion, is worth considering and ultimately using; whether it is for production, or as in my case, as an insurance policy to support unpredicted growth and create a conscientious business continuity plan. With time and maturity, EC2 is a strong solution.

Image courtesy of rengawman.wordpress.com

I like my ducks in a row. Oh yes .. I do indeed. Every time my ducks get out of alignment I react, to some extent, poorly. This is particularly true as I help build companies through technology. Technology is just a business tool and even though it may take center stage as the enabler of a business, it is not the business itself. But that is no excuse to bypass technology best practices.

A word on best practices:

Most people take best practices as a recipe; as a cookbook; as a road to follow. To me best practices is a set of tools that I can use to accomplish particular tasks. There is no particular guide to the practices but the practices themselves as I adapt them to my needs. The same goes for development and project management methodologies. I only adhere to my own. Each problem is different and requires adaptations. It is ridiculous to think that one size can fit all; especially when each task is in the context of varied corporate cultures, projects and business needs.

Back to my ducks …

The whole thing starts with picking the first duck and placing it at the beginning. Then I pick another duck and I scurry to some supposed end and place the “last” duck there. This duck represents where the company may be in a distant future. Call it 5 to 10 years out. It is 100% my conjecture and based on my personal vision of where the business will be in “a period of time”. I based this vision on discussions I have with other stake holders. Will it go there? Who knows. I just like to think of the possibilities and have something to aim at. Does it matter if it does or not? Not at all. The company will experience changes based on the market. The business will go where the market takes it.

Third, I once again scurry around looking for another duck - the right one too - and I place it following the first duck. I turn around, look at the “last” duck and line the first with the second with the last.

It is time for another duck. I rush to find yet another duck, rush to the front and place it, all neatly lined up with the first, second and “last” duck. I go find another duck and I go back to the fourth position. I look back to the “last” duck, I look at the row in front; I look back just to make sure … and … the “last” duck is gone. I mean, nowhere to be found. This is not a real duck; it is my duck. How can it have flown away? Or walked?

I drop the duck in my hand somewhere in position and run back to find the last duck. I look around … I look around … I look around and I finally find it. There it is. But it is not where it is supposed to be. I pick it up and try to figure out where it really belongs. Undoubtably, since it moved, it does not go back to its original place. I figure out its new placement, most likely based on changed assumptions and market forces. And, I have to go to the front and quickly rearrange all the other ducks and align them with the “last” duck. This process happens again and again.

It does not bother me that the “last” duck moved - as a matter of fact I assumed from the beginning that it WILL move; what truly bothers me is that nobody told me before it got moved and then I am expected to auto-magically aligned the other ducks. If the duck had gone “quack quack”, then by just listening I could have quickly rearrange the other ducks on the fly. But these ducks are quiet. They do not make a peep, especially as they are being moved. Or maybe they are being forced to move pointed by a gun under the threat of death if they “quack” ?

I see building companies very much as a process of putting ducks in a row. True, they do not need to be in a perfect line, but the row should have no gaps. The gaps are potential black holes that can drag the whole business into oblivion. Let’s be clear: gaps does not mean not having answers to all the questions. Many of the business or technology questions are answered as we lay down the ducks. Gaps means skipping the full understanding of basic elements in the business. In manufacturing it can mean skipping quality in the automation. In software development, not respecting a project plan. In business development, not having an out in a business relationship. In business in general, not having a solid strategy and not continually contesting it, revising it and analyzing potential risks factors.

Often I am asked “how can you know where the business will be?” As I stated above, I do not know. But I do imagine what the possibilities can be. It is not that hard to look up and try to take a leap of imagination and visualize where the business can be in 3, 5 or even 10 year. It is a dream. It is pure imagination. It is not real. It is a VISION. It is also a goal to aim for and a way to reverse engineer a road-map. Will the business end up there? Most likely not. Most likely it will take detours, it will change and morph, it will reinvent itself. It will struggle to survive (not necessarily in financial terms.) The market dictates where a business goes. And my ducks are witnesses to the detours and changes.

Regardless of the market, the vision needs to be there at the beginning. And the vision needs to adapt to the market. A business starts with imagining an idea. It continues with the fantasy of success. And follows the excitement of victory. In other words: THE VISION. Not vision as in a corporate statement - The Vision and Mission, those are important and necessary because they are internal call to arms and good external communications tools. But the vision as a quest to conquer some uncharted land or defeat some mortal enemy. Will the vision change? Absolutely. The change is what keeps things interesting.

I am a technologist and see technology as a business process, not as an esoteric pursuit of technicality. Indeed, the better the technical solution I come up with, the better I feel and I always strive to produce great technology including novel work when possible; however, as a function of creating value for the company and not for technology sake. I very strongly believe that in the end, if the technology does not answer the business need, for as good and revolutionary it may be, it is worthless.

My ducks, in the end, are just steps in a process to lead an important part of the business to success. Technology is an equal partner to all of the business units. It is normally considered a cost center, but it really is a revenue generator and through automation and operational efficiencies, a direct profit center.

Once a month, on the second Friday of most months, a CTO networking group that I belong to meets. Each meeting starts with a speaker - speaker in the loose term because it is not a fixed format - and then leads to questions and answers by the members and to discussions. This Friday the speaker was an individual that transitioned from being a CTO to being part of a VC. The attendance was not very abundant. However, since it is the path my career is taking and/or wants to take, I was not about to miss the meeting. The presenter brought up some good points and topics of arguments. And, of course, we all engaged readily. One in particular that developed into a full conversation once the speaker left was the initial cost of starting a company.

What Sangam Pant mentioned in terms of cost is that today it is VERY easy to start a company. It only takes $1M. Now … everybody jumped at that except another member and I. First of all, Sangam was using $1M as a figurate amount. Or at least I think he meant it does not take millions upon millions to start a company. Second of all, it is true, it does not take much money. You can use any of many OpenSource projects that are more than adequate for an initial buildout without compromising too much the quality of the architecture and application. You can successfully build on them. Or, you can use a small local team complemented with an outsource team at a fourth of the cost. I prefer the second option. It allows me to add more value to the company by providing a proprietary  code base and create opportunities for the future. Besides, I am more on the side of build than buy for core elements of the business.

The argument from the members was that if you are trying to build a $100M+ company , you need to start with adequate funding. $1M will just not do. My position is that you do not need to start with $10M or $20M. Or even $5M. What you need is to start. The initial problem of a start-up is two fold:

1. You need a product that works. Not juts technically - which is exceedingly important - but also from a business concept point of view. Is there a need in the market? Can a need be developed? Is the market ready for a product like “this”? Can the market mood help a product like “this”? There are a series of market factors that can be put into the conceptual analysis of a product. And many of those factors will help define the product, its marketing strategy and how the market is approached.
2. You need a business model. Now … this is the hard part. You need to devise a way to make money. It does not mean that you have revenues right away and it does not mean that you are profitable right away - although in a previous post I focused on the March To Profitability. Business models do not come by easily. Let’s restate this: An effective fully developed business model that matches the market mood and not only prevails, but gains traction, does not come by easily. A model like that takes time to develop and can only be developed by generating friction against the market.

How much does it take to start addressing these two questions? It changes from project to project, but does not need to be much. If the company is to get to $100M+ you will need growth investment; I absolutely agree that you will need more investment along the way. ALONG THE WAY. But first prove the “model”. Prove that there is a need in the market, or that the market will be somewhat receptive. And first begin to develop your business model and put it through some friction to see how it fairs. For the “first begin to” part of the project a “$1M” ought to be enough. Once you see a small light at the end of the tunnel and it is not an incoming train, the  you can go ask for $5M, $10 or more.

There is also a financial argument to not “taking” in big rounds at first: Valuation and dilution. The more the above questions are answered, and they do get answered over time, the higher the valuation. The higher the valuation the less the dilution. To the cost of a startup you need to add the cost of getting funding. Often enough there are situations where the founders of a company loose control of the helm because they get too diluted to retain a majority; in many of those cases you get there by either poorly negotiating or by asking too much too early and since there was no tangible value but value of an idea, the percentage sold is too large. Either case can be mitigated by having a more robust offering. Even if you negotiated poorly, the odds are in your favor if you have a more developed project well beyond an idea. Thus, the cost of funding is less. And the upside at the end is much larger for all involved.

Going back to the meeting … I have to say that I did not agree with everything the speaker said, his positions and the reasons for them. I have to say that I did not agree with the speaker in every idea or vision that he has on the market and how the market will develop. But the balance of what I got from him is positive. He was not only well spoken, but eloquent in the way he presented his ideas. And whether I agree with him or not, it does not matter. It is whether I could derive value from his experiences, concepts and intelligence.

Sangam, thank you for your time on Friday.