Telecom operators have always focused on two aspects: ARPU and time-to-market. In the latest technology craze – Cloud Computing – a lot of telecom operators are seeing a new golden grail. Those that can see further than SaaS marketplaces and moving their hosting to IaaS are only the happy few. Since Cloud Computing = SaaS + PaaS + IaaS, it is normal that operators start talking about Telco PaaS. However Telco PaaS is a lot more than combining telco assets with an IT PaaS.

IT PaaS are aimed at quickly launching applications. The IT grail is to launch thousands of applications to find the one that everybody likes and become an overnight millionair. Telecom operators might be easily fooled that opening their telecom assets to IT PaaS developers will bring that one application that will turn the telecom sector around: “The Killer App”.

Unfortunately a telecom application is more than an SDK+app server in the Cloud that can do VoIP. The reason why companies pay a plus to telecom operators is “trust”. “Trust” that you can pick up the phone and call somebody. “Trust” that if something is not working that you can call their call center. “Trust” that tomorrow they will offer you the service.

All this “Trust” has to do with the way operators have their backoffice systems and processes set-up. Having thousands of developers creating applications that mix telecom with SaaS might give some nice innovation. However telecom operators are not prepared to handle thousands of end-users that find bugs in a long list of applications and start calling call-centers massively. End-user expectations are different for telecom applications then for IT applications. This definitely has to do with the price they pay for them.

What should a Telco PaaS offer?

More than a fixed feature set, the most important changes for an operator that wants to offer a Telco PaaS might be internal. Operators will need a large shift in thinking to be able to accept some of the new realities:

1. Telco PaaS services need to be launched in weeks not years.
2. Telco PaaS services will be buggy, unstable and fail.
3. Telco PaaS services can not be supported via a call center.
4. There are no Telco PaaS standards and there are likely not to be any until it is too late.
5. Telecom can not be greedy

Launching in weeks instead of years

If IT PaaS is bringing something then it is speed of development. Telco PaaS needs the same type of speed. In practice this means that REST and JSON should be the operator’s vocabulary, not SIP and CDRs. Telco Assets need to be exposed to non-telco programmers. Developer communities need to be created. Marketplaces that allow developers to sell their creations by the click of a button and not to worry about complex charging and billing.

Unstability, bugs and failure

Not every IT programmer is a genius. There are probably quite a few geniuses. Instead you need to expect that people will do things incorrectly, by error but also on purpose. Application virtualization and sandboxing are key to make sure “mistakes” don’t bring down the whole platform. On the other hand customers need to be segmented. There are customers that can see further than the bugs and see the potential. These are called visionaries or early adopters. It is critical that operators allow these early adopters to play around with buggy services. However it is equally important that the majority of users know that the sandbox might contain buggy apps and that the call center is not the place they can find help.

No support via the call center

All sandbox applications can not be supported via a call center. Agents will not know anything about these thousands of apps but neither should they. The only one that knows is the one that developed the service. He or she should get the right tools to quickly understand which bugs or feature requests are important, e.g. via a social CRM. The operator should monitor those promising apps that are ready to graduate the sandbox. They should be place in an incubation program. Incubation will see if the applications can go mainstream and will professionalize support, availability and reliability.

No Telco PaaS standards

In the dotcom world the first one that creates a solution, creates the standard. In Telco PaaS this will be the same. If we do not know how users will use a Telco PaaS, how can we expect that standardization bodies will define the correct standards. However in disruptive innovation, unlike technology evolution, first movers have an ever lasting advantage.

Telecom Greediness

If you have a monopoly, you can be greedy. No competitor will offer anybody a better deal. However in Cloud Computing, greediness will kill your best innovation. It is better to get a small percentage, signup or usage fee per application when you have thousands of applications then to get nothing at all. Many cents can make billions. Think Adwords…

Ask a Telecom architect how you create a telecom network application, often dubbed as value-added services. He or she will focus on SIP/SS7 standards, service delivery platforms, etc.

The future of cloud-based telecom network apps, let’s call them tapps, is going a totally different direction. For the former telecom architect they probably like open source solutions like Mobicents that allows you to create SIP-based applications on Java. The Asterisks and other types of VoIP application servers are other alternatives.

However for a new generation of Web-based programmers this is all too complex. These are the programmers that like Javascript, Ajax, JSON,  PHP, Ruby, etc.

The majority of them will be fine with whatever Twilio or Tropo offer via easy to use REST APIs or embedded in their favorite scripting language. Which cloud-based application needs more than calling, SMS, answering the phone, getting feedback from the user, telling the use what to do, putting multiple users in a conference, transcribing what the user does, forwarding a call, etc.? 95% of the functionality is covered with a handful of REST APIs.

For business developers that are more used to Java, they can also use Java APIs to access for instance Twilio. To be able to cheaply launch an application and scale it afterwards they could deploy it on Google App Engine. A new alternative has just come around from Amazon: Elastic Beanstalk. A developer can write their app and deploy it on Beanstalk. They no longer have to worry about monitoring, scaling, opening firewalls, etc.

Other alternatives are to extend Cloud-ready telecom applications via plug-ins. An example here could be Twilio’s OpenVBX in which you can easily add new plug-ins.

The conclusion is that 2011 will be the year in which Web 2.0 and the Cloud meet the Telephone 2.0. However the Telephone 2.0 will unlikely pass through Bluevia and other operator initiatives given the fact that they are running about two years late and are very scattered, slow-moving initiatives.

Operators should embrace the new reality and try to help these new applications find new users. The Appstore brought a new eco-system to life. Millions of small and medium-sized Telephone 2.0 applications are waiting to be discovered by Billions of users. Remember that not everybody can pay an expensive mobile with an expensive data plan. However there are billions that can pay for cheap call and SMS-based applications. We need to help the billions find those tapps that are useful to them…

With the world looking more at XML, SOAP and REST these days, it is perhaps  anti-natural to think binary again. However with Protocol Buffers [Protobuf], Thrift, Avro and BSON being used by the large dotcoms, thinking binary feels modern again…

How can we apply binary to telecom? Binary SIP?

SIP is a protocol for handling sessions for voice, video and instant messaging. It is a dialect of XML. For a SIP session to be set-up a lot of communication is required between different parties. What if that communication is substituted by a binary protocol based for instance on protocol buffers? Google’s protocol buffers can dramatically reduce network loads and parsing, even between 10 to a 100 times compared to regular XML.

What would be the advantages:

• Latency – faster parsing and smaller network traffic reduces latency which is key in real-time communication.
• Performance – faster parsing and lower load means that more can be done for less. One server can handle more clients.
• Scalability – distributing the handling of SIP sessions over more machines becomes easier if each transaction can be handled faster.

Disadvantages:

• No easy debugging – SIP can be human ready hence debugging is “easier”. However in practice tools could be written that allow binary debugging.
• Syncing client & server – clients and server libraries need to be in sync otherwise parsing can not be handled. Protocol buffers ignores extensions that are unknown so there is some freedom for an old client to connect to a newer server or vice-versa.
• Firewalls/Existing equipment – a new binary protocol can not be interchanged with existing equipment. A SIP to binary SIP proxy is necessary.

It would be interesting to see if a binary SIP prototype joined with the latest NOSQL data stores can compete with commercial SIP/IMS equipment in scalability, latency and performance.

Any operator that has not started a project on Cloud Computing is late. The typical data center at an operator is filled with servers that are under utilized e.g. application servers and database servers are running at 30% of memory, disk and CPU. Just by doing step one of getting to Cloud Computing: virtualization, operators are able to save substantially in the cost of hardware, electricity, maintenance, etc. Virtualization means decoupling software from hardware. This allows to run multiple operating systems on one server.

However this would only be focusing on the tip of the iceberg. Cloud Computing is so much more…

Private Clouds

Automatic Scaling

Let´s first focus on the internal systems of an operator. After solutions have been virtualized, then you are able to scale them to more or less servers. The first step is to automate this process. If you have an application server cluster, do you need 8 nodes all the time? You probably only need them the week before Christmas or during some other peak period. So the ideal is to be able to measure the load and to automate the deployment of more or less cluster nodes based on load. The same can be done with the database. During the night you have 2 nodes. In the morning 3. During the day 4. During peak moments 8. In the evening 3 again. You could save massive amounts of money if application servers and databases can be scaled in this way. You ideally also are able to pay licenses based on what you really use and not on your maximum number of nodes during a yearly peak.

Redesigning Applications and Data

Both Amazon and Google found out that if they redesign their applications then they can get even more gains than pure virtualization. Amazon´s S3 service is a clear example. However internally they started with services like Dynamo on which S3 is build. The first step is to build general data stores. Multiple applications should be using a common data store instead of needing a separate database cluster each.

Unlike popular believe in the IT world, the dotcoms are not filling their data centers with Oracle RAC clusters. The dotcoms are designing special purpose data stores. The data volumes any market-leading dotcom has to deal with are so massive that a SQL database can not keep up. SQL databases are very good at running efficient queries on structural data or making sure transactions are consistent. However they fail when data is unstructured, write operations are massive or data volumes grow with terabytes every data.

Relational Data

So for all low-volume applications that need transactional data and read more than they write, you could still use a unified Oracle RAC cluster to serve multiple applications. An alternative approach are the data stores that have been build by Amazon (Relational Database Service or SimpleDB) or Google´s App Engine (Datastore with JDO).

What other alternatives are there?

Read Mostly Data

Data that needs to be read a lot and is not updated frequently can get an enormous performance and scalability boost by using an in-memory data store. The dotcom standard is memcached. Facebook (800 servers and 28TB) and Twitter are addicted to memcached.

Documents, Images & Videos

Binary and media files are best stored outside of a database. In small numbers they are often stored on a file system. However they occupy a lot of disk as well as network bandwidth when moved around. The ideal is a document store with a content-delivery network or CDN as a front-end. Amazon´s S3 and CloudFront are examples. Storing them in a compressed format, e.g. LZO can save valuable space. Also transcoding into different formats, e.g. thumbnails or preview can help save network bandwidth.

Unstructed Realtime Data

Data that is unstructured and needs to be stored and accessed in real-time in high volumes are best stored in special purpose data stores. You can write a book about the latest NoSQL solutions. Write an email to maarten at telruptive dot com if you are interested.

Analytics Data

Twitter has described most extensively how they use all the unstructured data they get from their logs and other sources. They use technology from Facebook to stream it into a high-available file-system from Yahoo. There they run massive parallel map-reduce operations to get to know a lot more about what users are doing and who is influencing who, etc.

Social Graph

The social graph is about who knows who and what kind of relationship you have. This data is best stored in graph data stores.

Collective Intelligence

Again a chapter by itself but dotcoms are also heavy users of collective intelligence which often means dedicate systems.

Accessing Data

Instead of stove pipes with data, the dotcoms are making data accessible to all their applications. Either via search interfaces, web technology to access data (e.g. REST and JSON) or efficient binary interfaces (Thrift and Protocol Buffers).

Messaging and Notification

Amazon is having a simple queue service and a simple notification service to make sure applications communicate in a uniform matter.

Applications

If applications have access to all the above services then the architecture of an application is simplified enormously. Most of the famous dotcoms don´t use middleware. They prefer the SOA principle. However unlike the IT SOA solutions, a dotcom would take an application and make it into a chain of reusable services. Let´s take an IVR application as an example. There would be a service to do voice recognition. Another one for voice transcription. Another one for text-to-speech. A transcoding service to transcode between different media formats (e.g. high-quality voice and low-phone-quality voice). And so on. Each service has independent load-balancing and can be scaled separately. Services can be re-used between applications. An application is very short because it just need to define which services need to work together and how.

Application Deployment

The dotcoms deploy new features on a daily and even hourly basis. This means that all application deployment is fully automated. When a new feature is deployed it does not necessarily overwrite an existing feature. It is possible that a new functionality has been solved in 5 different approaches. Dotcoms would split the total user base and let small parts of users try out the different approaches. Depending on the user´s feedback they would take the preferred approach and slowly scale up from 1% to 100%. If they detect that the feature has a performance problem or a bug then they would be able to roll-back or decrease the load, fix it and deploy gradually again.

The Network, OSS and BSS

There is a substantial effort needed to redesign a network to be cloud-aware. Some components need latencies lower than 10 milli-seconds (e.g. antennas), hence most of this logic will have to be processed locally. However all systems that can live with 100 milli-seconds latencies benefit from a cloud make-over.

Especially in the area of OSS and BSS there is room for optimizing applications and making them cloud-aware. Global services like a network inventory service, a user profile service, a device profile service, etc. would mean simpler applications and less data duplication.

Opening the Cloud

So the network and IT infrastructure is being redesigned to allow for faster innovation and lower costs. However Cloud Computing can also be used to increment revenues.

Being a Cloud Infrastructure Provider

Many IT consultancies and software/hardware vendors will tell an operator that they could be a Cloud infrastructure provider. On slides this really looks nice. However unless an operator is not using the cloud computing principles for their own systems as described in the first part, they are lacking substantial knowledge about how to manage such an infrastructure. Without this knowledge it would be hard to have a very optimized solution and as such be price competitive with the existing players.

Being a Cloud Platform Provider

Although closer to the operator´s core competencies, being a cloud platform provider would still be for those operators that are Cloud experts. A Cloud platform provider would allow others to use the infrastructure services to create applications on top. The complexity lies in the fact that malicious users try to break the platform which could have a very negative effect on the infrastructure if not handled correctly.

Being a Cloud Service Provider

This is the default option most operators should explore first before moving into the other areas. Being a service provider also has a roadmap:

Reselling SaaS

The easiest step is to be the storefront and to resell IT applications from others, e.g. cloud backup storage, security solutions, etc.

Offering Telco SaaS

The next step would be to offer specific telecom applications. Applications that are build for the operator or even better applications that can be build by others based on the operator´s assets. An example would be a PBX in the Cloud.

Open Market for SaaS

Building all telecom applications yourself is hard. Attracting others to do it for you is easier. However just putting a “Net App Store” and an SDK on the web will not get you to dominate the market. Only an open market with a large eco-system of companies and developers can generate large quantities of “Net Apps”. If you are thinking about building an open market, why don´t we talk first. Send an email to maarten at telruptive dot com.

Asset exposure is a hot topic in the telecom industry. Everybody is talking about opening up telecom assets for others to create new services. However I haven´t seen any successful asset exposure by the established operators yet. Why is this? Let´s look at the four major European players.

Vodafone calls network asset APIs,” enablers” and is offering mobile payment and location services. More is announced. Then again there is also betavine, oneapi and vodafone network apis. And we are not talking yet about widgets and mashups.

Telefonica has the movilforum, O2 litmus and the movistar developers platform.  A list of separate APIs is available with exotic names like “Manage Post Pay Bolt On’s API”.

“Orange developer” gets you in Google to all you need to develop. Unlike Vodafone and Telefonica.  A long list of APIs is available, including the personal profile api and the personal richprofile api.

“T-Mobile developer” also worked. Unfortunately there was no information for network assets API only mobile apps development.

There is one red line in this short test: “being a telecom developer is a hard job”.

Finding the sites with development information is hard. You would think that “operatorname +developer” should be enough to Google you there.

Understanding which API to take is even harder.  I don´t want to imagine writing an application with the APIs.

It is clear that European telecom operators would like to have an active developers community but they would need to understand some basic principles:

• out of sight, out of mind
• 80-20
• simplificationality
• eat your own dog food
• make one millionaire
• collective
• compete against the dots

Out of sight, out of mind

If I can´t find your API in the first 5 Google items when I look for developer and your company name, then I will probably never write an application for you. SEO, search engine optimization, is the first mandatory step when a developer portal is launched. Your portal can be the best in the world but if I can´t find it then I can´t use it.

80-20

80% and even 95% of all applications only need 10% to 20% of the functionality. The telecom industry has made a habit of “over-standardizing” APIs. A simple example: sending an SMS. What does an application developer wants to do in 95% of the cases? Send “text A” to “number B”. Only 5% of the applications need a receipt that you actually read the SMS. Even below 0,01% would like the operator logo, the internal port for message delivery in the device, etc. to be available. So the trick is to have a very basic high-level API and if required a more detailed low-level API.

Simplificationality

Why use “simplificationality” if you can use “simple”? Developers want to keep things simple. Programming is already hard enough. Standards bodies in the IT industry have designed the XML and SOAP standard to share unstructured data between as many systems as possible. Developers are neglecting both and are migrating to JSON and REST. A JSON document is a lot shorter and holds the same info. A REST API you can call from the command line. There is an interesting startup that has taken this simple principal and applied it correctly:

Eat your own dog food

A lot of startups are using the slogan: “eat your own dog food”. If you create something for others to use, then the first user should be you. Amazon created their web services stack for internal use and afterwards opened it up to the world. Until your services are not written with your API, then you should not open it up for others. After writing some services with your own API, you can be sure that it is fully functional, well documented and no longer has awkward glitches.

Make one millionaire

The economy is moved by the invisible hand which in human language means: “people are selfish”. I am prepared to suffer and write an application with your API as long as I have a change to become an overnight millionaire. Apple´s App Store has made for sure somebody a millionaire. As soon as there is one, there will be a million others trying. Make sure you share your revenue is based on the value contribution (Apple sells => 30%, developers invents and writes => 70%). If people feel they have an honest chance of becoming rich, then they will go the extra mile.

Collective

99.99% of all the apps that developers will write with your APIs will be rubbish.  However as long as users are able to easily find the other 0.01%, the operator will make money will all 100%. For users to be able to find a needle in a haystack, you need collective intelligence. People need to be able to benefit from viral marketing, recommendation engines, top 25, etc. in order for them not to have to waste their time. Early adopters will try out the bulk of the apps but they are only a minority. The rest of the mortals need to be helped. If I am looking for the coolest game, then it should be first in the list of games.  Apps needs to be automatically clustered, categorized, recommended, etc.

Compete against the dots

An operator´s competition for asset exposure is not your-next-door-operator but a dotcom that offers a way around your assets. For years operators sat on their location assets without exploiting them. Google came and wiped most of the value out with its Latitude, Maps and Goggles. Copy the dotcom technology and ways of doing things.