Archive
The competitive advantage of a shared architecture
Why can Facebook, Google, Salesforce and Twitter role out new features every day and regular telecom operators only every 6 months? Although they are dotcoms, they have thousands of employees and a lot of legacy systems as well. However they are able to roll out a new feature every day, if not every hour or minute and large new systems every so many months, weeks or even days.
How do they do it and how can the telecom industry learn from it?
On highscalability, you will find a lot of information on the architectures of large dotcoms. However if you look at different articles you see that each of the larger dotcoms has an architecture that is shared among different products and services, e.g. scaling messages at facebook.
This is the secret sause of the dotcoms. They have built and continuously improved a highly distributed architecture that can handle millions of users and peta bytes of information. On top of this “shared architecture” go the services. New employees are able to quickly create new services because they do not have to worry about scaling data, monitoring the service, deploying/upgrading versions, backing up data, versioning code, etc.
On the other hands operators have no standardized shared architecture. Instead there is a puzzle of different solutions that often use totally different technologies, hardware, etc. Maintenance and upgrades are a nightmare.
Trying to launch any new service requires a massive amount of planning, lots of different skills, expensive investments in third-party licenses and hardware, etc.
How can you do it differently?
Building a private cloud with virtual servers and storage will not resolve operator’s problems. Just virtualizing the puzzle of solutions is not going to do away with complex integrations.
Operators need to make a more bolt move. They need to separate the new from the old. Legacy systems should be kept and isolated. However a new architecture should be built that works in parallel with the legacy systems. This new architecture should focus on launching new services and partner services at dotcom speeds. Everything should be handled as an independent service. Each service should get its own API. A storage services, a billing service, a monitoring service, a provisioning service, an identity service, a datawarehouse service, a deployment service, a mobile shop service, an inventory service, a support service, etc.
All APIs should use a common technology. APIs for third-parties could use REST. APIs for internal high-load usage could use Thrift or Protobuffers. Each API should have two versions, the easy and the low-level version. The easy API offers the most used but in general basic functionality, e.g. sendSMS(from, to, message). The low-level API offers a complete feature set, e.g. sendBinarySMS, sendSMSWithDeliveryConfirmation, etc. This will allow most services to use the easy API but to have access to the advanced functionality when needed.
Loadbalancing when using the services is key. The loadbalancer is the secret for many rolling upgrades in the dotcom world. An application that uses a certain service will use client-based loadbalancing. By having the loadbalancing be able to receive events, it is possible to dynamically add/remove instances of an API, gradually move requests to a new version of the API, etc.
New service developers will now have to focus on building the business logic for the new service and not on data migrations, scaling, monitoring, backups, etc. The service can have completely new ways of billing and charging, a complex deployment workflow, advanced monitoring requirements, large data storage requirements, etc. However it is not the billing or charging system that has to be extended. Neither a centralized EAI. Nor the monitoring system. Instead it is the service that decides what is best for the service via the use of the easy or low-level APIs. By moving the peculiarities of every service into the service and not into generic OSS and BSS systems, these support systems can be drastically simplified.
Operators should try to focus on launching a lot more niche services and opening up their infrastructure to a long-tail of service suppliers. Instead of general services like PBX for SME, operators should think about hotel reservation services, doctor scheduling services, etc. The value of the operator should be in offering a reliable back-office architecture, assuring service quality and managing the support eco-system. The long-tail of service suppliers should be put to work to launch competing niche offerings and let customers decide which one will survive or not.
Next Buzz: Social Enterprise Apps
Social Enterprise Apps are the next buzz. Companies like Salesforce with Chatter, Yammer, Jive, Google with Google+, etc. all want to change the way employees work in 2012 by adopting Facebook and Twitter-like solutions.
At the moment it is too early to tell who will be the winner. Most products however are still just offering only basic features like status messages, connect to colleagues, share documents, etc.
The real interesting features are still to come. Employee driven process creation and management should make it possible for plain humans (not über-programmers) to define and manage company processes and to transfer a world of Excel, Access and other homegrown solutions to the Web and mobile world.
Operators should jump on the social enterprise apps bandwagon because calls and SMS can still be incorporated into this new portfolio of products. However not in the traditional manner. Since everybody has access to a phone, it could be used for quick approvals either by calling in, getting called or sending an SMS. Even faxes could be incorporated. Traditional companies might be more willing to move from paper faxes to online faxes instead of moving from zero to Facebook speed right away.
The key will be the ability to people to define and manage things themselves without needing support from IT or five level of approvals…
The traditional way of innovation is no longer good enough…
Innovation used to be something related to an R&D department that would experiment with new technologies and a marketing department or product management that would ask customers what new features they required. The business team would be killing any innovation that did not present a business case which complied with company rules: e.g. x% margin, €yM revenue in two years, etc.
Why is traditional innovation no longer good enough?
The cost of launching a disruptive innovation that changes a complete industry has come down dramatically. There are many examples: Skype and roaming, Amazon’s Kindle and paper books, P2P and network bandwidth / media revenue, Salesforce and shrink-wrapped software, iPad and Windows PC, iPhone and Nokia, etc.
Disruptive innovations are more frequent than ever and enablers like Cloud Computing, Open Source, Off-shoring, 3D Printing, etc. allow innovators to launch big solutions on a modest budget.
Most traditional innovation is about evolving a current product by adding new features and improving current functionality. Traditional innovation focuses on prototyping new features and products and showing them to potential customers. However process innovations (e.g. Toyota Production System), business innovations (e.g. freemium), marketing innovations (e.g. Intel Inside), disruptive innovation, etc. are often overlooked.
Every one should innovate
More and more companies are convinced that every one in the organization should innovate and not only R&D and product marketing. By putting special innovation processes in place in which employees can share innovation ideas and use collective knowledge to improve them and get funding, innovation becomes more democratic and often more successful. Companies like Google allow employees to focus one day a week on innovation that can be totally unrelated to their day jobs. People vote with their time which project is worth it. Ideas are shared hence collectively the services get better.
Also upper management is no longer looking from above but should innovate by example. Name all big innovative companies and you see that founders are a big part of innovation and participate in it every week: Google (Larry Page and Sergey Brin), Amazon (Jeff Bezos), Apple (former Steve Jobs), Facebook (Mark Zuckerberg), Salesforce (Marc Benoiff), etc.
Daily Innovations instead of Product Releases
The large dotcoms (Google, Facebook, Amazon, etc.) no longer do market research in the traditional way to find out if users like a feature or not. They also no longer focus on major product releases. Instead they focus on incremental innovations on a daily basis. Users request new features via social CRMs and the most voted features get implemented. Often a feature can have multiple implementations. Users are divided into different groups and new features get enabled for subgroups. If a new feature has a positive effect then it survives and gets rolled out to the rest, if not it gets killed or adjusted.
New products no longer get productized from an idea and afterwards customers are searched for it. Instead customer’s pain points result in paper prototypes that get validated and redrawn until they solve the problem. Afterwards real prototypes are made that get launched in beta or even alpha shape towards real users. Beta can already mean that users are paying for it.
Discovering New Innovations
Discovering new innovations is done by combining groups of people with different expertise (marketing, psychology, arts, technical, business, etc.), to understand a new domain and to question a status quo. Most of the time the best innovations are those that remove a status quo and make a painful activity into a joyful activity, e.g. LinkedIn: networking with people and keeping up to date your business network.
After questioning experts and novice users, innovative companies also observe how users use their products. Often heavy users or first-time users are unsatisfied with current products. New ideas are shared inside but also outside of the company with a network of experts as well as people with completely different skills. Afterwards solutions are built based on experimentation. A very important aspect is also being able to transport solutions from other industries. Making associations between unrelated topics and understanding how things are done in a completely different environment can bring new inside…
It is very important that different departments (business, marketing, operations, maintenance, etc.) all work towards launching new innovations and removing obstacles because killing innovation is very easy, making it succeed is not.
Some good books on innovation are: The Innovator’s Dilemma, Nail It then Scale It and The Innovator’s DNA…
How RyanCom would destroy the European telecom business?
Let’s assume a new telecom competitor is entering the European market: RyanCom. Similarities to RyanAir are purely fictual 
1) The Network
Instead of building expensive antenna networks RyanCom would make deals with Cable operators to put Femtocell equipment in cable modems and as such cheaply get coverage in major cities. Everybody that would switch to a Femtocell modem in their home or office would get 6 months of mobile usage for free.
Ryancom would have an agreement with the smallest operator in every country to sublease capacity if a Femtocell is not available.
2) Target User
iPad, Tablet owners, Websurfers, Roamers, etc. Ryancom would allow one data plan for the whole of Europe. Since the bulk of the traffic would go via Femtocell, better access costs could be provided. €5-10-15/month to have €5-10-15 Gigabyte/month.
3) Backoffice systems
All backoffice operating and business systems would be running in a cloud and open source is heavily used. Since there are only a limit number of data plans, there is no need for a billing system. SaaS like Zuora are enough. Google Apps and Salesforce would also be heavily used.
4) Social aspects
Social aspects would be very important. There would be competitions going on for which subscriber can convince more friends to join RyanCom. There is no helpdesk in the traditional manner. There is community support just like GiffGaff.
The End Result
RyanCom would be able to gain young data-intensive and roaming subscribers. They will see RyanCom initially as a second provider for their tablets. Little by little RyanCom could become their first provider when Skype and other applications become common use to make mobile calls.
RyanCom might be a fictional company but operators should be warned that fiction and reality might be just a matter of time…
Changing from Telco Grade to Web 2.0 Grade by fighting telecom myths
Most telecom operators are still thinking that software should be upgraded at most twice a year. Oracle RAC is the only valid database solution. RFQ’s bring innovation. If you pay higher software licenses, the software will have more features and as such will be better.
All of these myths will have to be changed in the coming 12 months if operators want to be stay on top of the game.
Upgrade twice a year
For telecom network equipment, two upgrades a year are fine. However for everything related to services that are offered to consumers or businesses, that means that operators are 180 times less competitive then their direct competition. The large dotcoms like Facebook and Google make software upgrades on a daily basis. 50% of all the files that contain Google software code change every month. Even if “a revolution” would happen and software upgrades would come every month, it would still mean a 30 times lag.
Operators need to start using cloud computing, even if they are private clouds, to deploy their back-office systems. The business needs software solutions to move at market speed. That means that if a new social networking site is hot, then it should be integrated into telecom solution offerings in days. Not in months or a year.
There are many techniques to make deployments more predictable, more frequent and more reliable. Offering extra features or integrations quickly can be done via plugins. You can have a group of early adopters, give feedback. If they don’t survive this feedback, kill them. If they do, scale up quickly.
Oracle RAC
Nothing bad about the quality of Oracle RAC but it is a very expensive solution that needs a lot of man-power to keep on running smoothly. Operators often pay a premium for services that could run equally well on cheaper or Open Source alternatives. Also NOSQL should be embraced.
If the cost of deploying a new service is millions, then only a couple of them will be deployed. By lowering hardware and software costs, innovative projects are more likely to see daylight.
RFQ’s and Innovation
It takes 3 months from idea to finalizing an RFQ document. 1,5 month to get a reply. 1,5 month to do procurement. Half a year in total. Not counting the deployment time which is likely to be another 6 months. The result is that the operator takes 12 months for any “new” system.
Now the question is if that system is really new. Because if an operator was able to define in detail what they want and how they want it, then the technology was probably quite mature to begin with. So operators spend fortunes installing yesterday’s technology 12 months late. Can anybody explain what innovation this is going to bring?
First of all operators should not organize multi-million RFQs for business or end-user solutions. These are likely to come late to market and can only be focused on mass markets.
Instead operators should focus on letting the customer decide what they want by offering a large open eco-system of partners the possibility to offer a very large list of competing services to their customers. The operator should offer open APIs to key assets (charging, numbering plans, call control, network QoS, etc.). As well as offer revenue share and extra services like common marketplaces and support 2.0 (social CRM, helpdesk as a service, etc.). This is called Telecom Platform-as-a-Service or Telco PaaS.
High licenses, more features, better
More features does not mean better. Most people want simplicity, not a long list of features. Easy of use comes at a premium price. Look at Apple’s stock price if you don’t believe it.
It is better to have basic systems that are extremely easy to use with open APIs and plug-ins. A feature by feature comparison will make you choose the most expensive one. However it is hard to put as a feature that the system needs to be easy to use.
In telecom, there is a natural tendency to make things hard. In Web 2.0 the tendency is the opposite. You can see the difference between Nokia and Apple. The Nokia phone would win every feature on feature comparison but the iPhone is winning the market battle…
Instead of organizing an RFP, let end-users and employees play around with early betas or proof-of-concepts. No training, no documentation. Let’s see which solution makes them more productive, the feature rich or the more straight forward. Just ask open APIs and a plugin-mechanism and you will be set…
Operators should act like VCs when it comes to innovation
Operators have an addiction to use RFQ processes. The problems with those processes are:
- They take a long time.
- They are not useful when technology is not well understood.
- They are only applicable to solutions whereby there are multiple providers.
Basically an RFQ will always bring you yesterday’s technology. After integrating this technology, when you launch you will be working with the day-before-yesterday’s technology. Google and others are working on tomorrow’s technology. Launching an innovative service via an RFQ process is as such almost impossible.
If you want to launch innovative, and new revenue generating, services that bring you back into the game, then the first thing that you need to accept is that RFQ’s are not your answer. So what is the alternative?
Innovation is about trying and accepting failure as a natural step towards success. You need to do everything to quickly detect failure so you do not spend too much effort on such solutions. How can you do this?
Forget asking marketing experts or even doing market studies, etc. Do you know any expert that would have predicted the success of Facebook?
The answer is to form small teams, called tiger teams. 1 functional/sales expert and 5 technical experts is a nice size. These teams should be isolated from the rest of the organization. Ideally the teams are mixed with members from partners or suppliers that know about a specific domain.
Additionally the tiger teams should have access to some innovation framework that helps them use some of the internal assets without having to do real deep integrations.
Finally the tiger teams should have access to part of your website,e.g. labs.operatorname.com, where they can launch their creations and get immediate feedback from beta users.
Operators should invest in these teams as would a VC. You get a little bit of money in the first round. Just enough to get me a prototype in 1 to 3 months. Afterwards we look for 1 to 2 months to see if there are beta users that like the solution. If not, we kill it.
If the solution is interesting for end-users or businesses then the next round of spending should come. However ideally operators start working together with other operators that are not competing and jointly invest. This way, your tiger team might not have given any result this month but the tiger team of your neighbouring operator might. By jointly investing, the cost to get the first commercial version is a lot lower.
The tiger team should not switch off the beta and start working on version 1.0. Instead it should use a social CRM to get feedback from early adopters. The roadmap should be steered by the community of early adopters. The service should with their guidance go from alpha to 1.0 in a maximum of 6 months.
As soon as the service is live, then it should be extended to all the other operators that also invested in it. Via Cloud Computing, it would be possible that this service is only installed in one place and offered to multiple operators with a minimum integration. Partners could even take over all costs of operations in exchange of a revenue share.
Once version 1.0 is reached, it is time to focus on offering open APIs and plugins so the community can start extending the solution and not necessarly the operators.
FCC’s net neutrality short-term heavy pain, long-term gain???
The newest FCC ruling about net neutrality will have far reaching consequences for VoIP. US operators will not be allowed to block VoIP from competing providers. This means that Skype, Google Voice, etc. will have free access to the operator’s networks.
It is unclear if Europe will follow the US. Not being able to block VoIP will mean that voice and SMS revenues will decline faster than before. Phone tariffs will no longer be able to include statements like “VoIP is not allowed or will mean the user accepts a more expensive tariff plan”.
However is allowing universal VoIP really bad in the long-term?
In case the FCC’s net neutrality would not have included this statement and operators would have continued having a “monopoly” on mobile VoIP what would have happened? Probably nothing major. And it is exactly this state of not being forced to move on and bring out innovative services to substitute declining voice and SMS revenues that is dangerous. Dotcoms are innovating at crazy speeds. Sooner or later they would find ways around a legal “monopoly”. However with the new rules the operator’s management is no longer exclusively thinking about rolling out LTE or not.
In Europe VoIP should also be declared open for general usage because otherwise there is a major risk that European operators lag behind in the mid-term and can never recuperate again…
