Charles – Chuck – Butler, a colleague at Canonical, wrote a very nice blog post explaining the basics of Big Data. It does not only explain them but it also allows anybody to set up Big Data solutions in minutes via Juju. Really recommended reading:
This is a good example of the power of cloud orchestration. Some expert creates charms and bundles them with Juju and afterwards anybody can easily deploy, integrate and scale this Big Data solution in minutes.
Samuel Cozannet, another colleague, used some of these components to create an open source Tweet sentiment analysis solution that can be deployed in 14 minutes and includes autoscaling, a dashboard, Hadoop, Storm, Kafka, etc. He presented it on the OpenStack Developer Summit in Paris and will be providing instructions for everybody to set it up shortly.
Internet of Things, or IoT in short, is going to be the biggest revolution of this century. However the engineering challenges are massive. Billions of low-energy sensors will have to be invented, produced, integrated, operated and maintained. These sensors will talk to millions of gateways over new network protocols. The gateways will communicate via next-generation network devices and software-defined networks to public and private clouds. These clouds will run scale out big data solutions that will dwarf even the biggest supercomputers today. Billions of containers will be running massive micro-services IoT business solutions. One mistake and a hacker from a country you don’t even know to locate can bring down a whole house, a whole company, or worse a whole country.
The business side is looking even less clear. The open source IoT tsunami that is coming means asking for licenses is suicide. Open source hardware and 3D printing result in razor thin hardware margins. Privacy will be central when technology is so central to people’s home and businesses. If it is free then you are the product will become harder if a dot com can know what you are doing in bed, with whom and even if you liked it. Business SLAs will be stricter than ever because customers will expect answers in sub-seconds however human operators will no longer be able to find and fix problems because of the overall system complexities. Roll-outs will be global even if telecom operator’s networks don’t cross borders and electricity plugs and voltage are super local. Ask a customer what IoT solution they want and they will answer you what is IoT.
In these circumstances you might be surprised that I will be focusing on solving these challenges and Telruptive readers will be the first to know when some are resolved…
This blog post is not about the technical details around LXC, LXD, Docker, Kubernetes, etc. It focuses on the different use cases LXD and Docker are solving and should help non-experts understand them.
Canonical demoed a prototype of LXD last week at ODS. Several journalists incorrectly understood that LXD is a competitor of Docker. The truth is that LXD is trying to solve a completely different use case than Docker. Ubuntu 14.04 was the first operating system to provide commercial support for Docker. Six times more Docker images are powered by Ubuntu than all other operating systems combined. Ubuntu loves Docker.
Different Use Cases?
Docker is focused on being the universal container for applications. Developers love Docker because it allows them to prototype quickly solutions and share them with others. Docker is best compared to an onion. All internal layers are read-only and only the last layer is writeable. This means that people can quickly reuse Docker containers that were made by others and add their own layers on top if desired. Afterwards you upload your “personalised onion” to the Docker hub hence others can benefit from your work. Docker is ideal for augmenting developer productivity and showing of innovations.
Canonical is the company behind Ubuntu and Ubuntu powers 64% of all OpenStack’s in production, the fastest growing open source project in the world. In OpenStack, like in VMWare or on AWS, you run a hypervisor on a host operating system and then install a guest operating system on top. Because you have two layers of operating systems you can host on one server many applications on multiple operating systems at the same time. This is greatly optimising resource usage over non-virtualization. However because you need to duplicate operating systems you are wasting a lot of resources. Now ideally you could put Docker directly inside OpenStack and run all applications from inside containers. The problem with this is that Docker does not give an administrator the possibility to remotely log into the container and just add some monitoring, backup, etc. and other normal activities administrators do to guarantee SLAs. In comes LXD. LXD is building on top of a container technology called LXC which was used by Docker before. However LXD allows you to have access to a virtual server, just like you would have in case of a hypervisor. The big difference is that LXD does not require operating systems to be duplicated. Instead it partitions the host operating system and assures fair and secure usage between different applications that run inside different containers. The result is that the same server can pack many more applications and startup as well as migrations of applications between different servers becomes extremely fast. This idea is not new. Mainframes already had containers. Solaris had containers. LXD just makes sure that your favourite private cloud has containers that are easy to manage.
Can a hypervisor, Docker and LXD coexist?
Yes. The hypervisor could make sure Windows runs on top of an Ubuntu host [linux containers can not support Windows on top]. Docker containers can host some next generation scale out solution that is either purpose build for Docker or has made changes to support some of the new paradigms Docker introduces. LXD will be best for all your standard Linux workloads that you just want to move as is. No need to update the applications or the tools that get integrated into them.
Since LXD has an Apache licence and is available on Github, it is very likely that the future will actually evolve into a world where LXD and Docker advantages get combined in some shape or form. Hopefully with new innovations being added as well. That is the power of open source innovation and exactly the reason why Canonical has shared LXD with the world…
I just saw Eric Dishman’s TED session on “Health care should be a team sport“. I love the idea of providing people with chronicle illness the means to be diagnosed and treated remotely and use big data to learn of a large group of patients with similar issues. Personally this would mean that when my sons have breathing problems we would not have to drag them in the middle of the night to a hospital where they are exposed to many viruses. Instead by measuring their oxygen level and listening to their longs a personalized remote diagnose could be made and some nebulizers or other things administered. At scale all equipment would probably cost less than £200 because Maplin already sells the nebulizer and oxygen level meter for a combined £110. Add another £90 at worst for a stethoscope that can be connected via bluetooth to a smartphone. Now via Hangout a doctor could remotely diagnose the results and even in the future a computer programme. All results of millions of patients would be collected in order to improve treatment. So no need for an expensive hospital in London with a receptionist, nurse and doctor dedicating 2 hours. By just avoiding one hospital night, the whole system would be enormously profitable. Additionally Ubuntu’s Juju can be used to set up all the big data and diagnostic software in minutes in any cloud or server on any place in the world. If other open source solutions are used then the total solution would be in reach for any developing country. There are probably more than one developer whose kids are asthmatic, and would happily contribute time. It sounds like an ideal Gates Foundation or Kickstarter project. If you think you can help please reach out to me because this is not work for me, this is personal engagement.
It is not often that one is responsible for cloud [and Big Data and IoT] strategy in a company of 600 people and you get told by the OpenStack foundation that your solution went from 55% market share to 64% while competitors like RedHat, HP, VMWare, etc. are spending hundreds [or more] of times more on marketing and engineering than you. Now I would love to claim responsibility for it but I would be lying. My mentors, Mark Shuttleworth and Simon Wardley, have laid the foundations years before I joined the company. But Ubuntu and Canonical, the company behind it, are the poster child example of why promoting chief financial officers into strategic roles in the last ten years was a terrible idea. Bean counters are about to inflict potentially irreparable damage onto iconic hardware and legacy software vendors. The reason is really easy: disruptive innovation. The innovator’s dilemma explained it years ago already. When some initial inferior technology comes along like Cloud Computing and OpenStack, then existing vendors will not get any demand from existing customers. Only when technology matures will customers start defecting en masse. But then already other companies have years of a head-start. Add to it that Ubuntu OpenStack is not only the most innovative solutions but also wants to be the most flexible [see our Autopilot, OIL, MAAS and Juju for more details] and the cheapest. So if you are on a quarter-based projected revenue track and you find out that your competitor is doing those three things extremely well, then it might be time to brush up those skills and experiences on your CV. Regarding the future, let me just tell you that the best is still to come :-)
Deep belief networks have made it possible to train computers to predict if a sentence is positive, negative or neutral. Most sentiment analysis captures headlines because tweets can be analysed. However are there business applications beyond social networking analytics?
Here are five examples:
1) Investment banking – reading complex reports
The financial industry is shaving off microseconds for high-frequency trading. However these algorithms assume that they can predict what a single big trade will be like. What if super computers would analyse any governmental report, news feed, etc. in real-time at a fraction of the time a human can do this. Initially these algorithms could get the most import data in front of analysts but there is no reason why automatic algorithms would not be able to make trades. There could be algorithms that look for natural disasters. Others that look at the sentiment of national bank reports.
2) Telecom: detecting defects and reading complaints
What do you do when call quality is bad? You send an SMS to the other person with your message plus some insult about your mobile provider. If your bill is too high, then you call their call centre or open a complaint on the website. Computers can more efficiently detect patterns in this behaviour than humans and can raise alerts before large groups of customers start to complain on Twitter.
3) IT: log processing and intrusion detection
Often strange user behaviour can be detected by analysing the commands that are introduced on a command line. Are they neutral, positive or negative? A hacker that is trying to exploit a bug and afterwards enters into log files to destroy their tracks could be caught because their commands are highly negative.
4) Retail: product reviews
What if a customer starts leaving bad reviews? Or even worse average reviews because they feel bad about a certain feature or services but not about the overall experience. Would you rather have a computer tell you in advance or wait until a crowd gathers enough tweets?
5) Politics: election sentiment
Real-time dashboards with sentiments for different candidates by analysing all written press. Find out what voters feel strong about.
The cloud is revolutionising IT. However there are two sides to every story: the winners and the losers. Who are they going to be and why? If you can’t wait here are the losers: HP, Oracle, Dell, SAP, RedHat, Infosys, VMWare, EMC, Cisco, etc. Survivors: IBM, Accenture, Intel, Apple, etc. Winners: Amazon, Salesforce, Google, CSC, Workday, Canonical, Metaswitch, Microsoft, ARM, ODMs.
Now the question is why and is this list written in stone?
What has cloud changed?
If you are working in a hardware business (storage, networking, etc. is also included) then cloud computing is a value destroyer. You have an organisation that is assuming small, medium and large enterprises have and always will run their own data centre. As such you have been blown out of the water by the fact that cloud has changed this fundamental rule. All of a sudden Amazon, Google and Facebook go and buy specialised webscale hardware from your suppliers, the ODMs. Facebook all of a sudden open sources hardware, networking, rack and data centre designs and makes it that anybody can compete with you. Cloud is all about scale out and open source hence commodity storage, software defined networks and network virtualisation functions are converting your portfolio in commodity products. If you are an enterprise software vendor then you always assumed that companies will buy an instance of your product, customise it and manage it themselves. You did not expect that software can be offered as a service and that one platform can offer individual solutions to millions of enterprises. You also did not expect that software can be sold by the hour instead of licensed forever. If you are an outsourcing company then you assume that companies that have invested in customising Siebel will want you to run this forever and not move to Salesforce.
Reviewing the losers
HP’s Cloud Strategy
HP has been living from printers and hardware. Meg rightfully has taken the decision to separate the cashcow, stop subsidising other less profitable divisions and let it be milked till it dies. The other group will focus on Cloud, Big Data, etc. However HP Cloud is more expensive and slower moving than any of the big three so economies of scale will push it into niche areas or make it die. HP’s OpenStack is a product that came 2-3 years late to the market. A market as we will see later that is about to be commoditised. HP’s Big Data strategy? Overpay for Vertica and Autonomy and focus your marketing around the lawsuits with former owners, not any unique selling proposition. Also Big Data can only be sold if you have an open source solution that people can test. Big Data customers are small startups that quickly have become large dotcoms. Most enterprises would not know what to do with Hadoop even if they could download it for free [YES you can actually download it for free!!!].
Oracle’s Cloud Strategy
Oracle has been denying Cloud existed until their most laggard customer started asking questions. Until very recently you could only buy Oracle databases by the hour from Amazon. Oracle has been milking the enterprise software market for years and paying surprise visits to audit your usage of their database and send you an unexpected bill. Recently they have started to cloud-wash [and Big Data wash] their software portfolio but Salesforce and Workday already are too far ahead to catch them. A good Christmas book Larry could buy from Amazon would be “The Innovator’s Dilemma“.
Dell’s Cloud Strategy
Go to the main Dell page and you will not find the word Big Data or Cloud. I rest my case.
SAP’s Cloud Strategy
Workday is working hard on making SAP irrelevant. Salesforce overtook Siebel. Workday is likely to do the same with SAP. People don’t want to manage their ERP themselves.
RedHat’s Cloud Strategy
[I work for their biggest competitor] RedHat salesperson to its customers: There are three versions. Fedora if you need innovation but don’t want support. CentOS if you want free but no security updates. RHEL is expensive and old but with support. Compare this to Canonical. There is only one Ubuntu, it is innovative, free to use and if you want support you can buy it extra.
For Cloud the story is that RedHat is three times cheaper than VMWare and your old stuff can be made to work as long as you want it according to a prescribed recipe. Compare this with an innovator that wants to completely commoditise OpenStack [ten times cheaper] and bring the most innovative and flexible solution [any SDN, any storage, any hypervisor, etc.] that instantly solves your problems [deploy different flavours of OpenStack in minutes without needing any help].
Infosys or any outsourcing company
If the data centre is going away then the first thing that will go away is that CRM solution we bought in the 90′s from a company that no longer exists.
For the company that brought virtualisation into the enterprise it is hard to admit that by putting a rest API in front of it, you don’t need their solution in each enterprise any more.
Commodity storage means that scale out storage can be offered at a fraction of the price of a regular EMC SAN solution. However the big killer is Amazon’s S3 that can give you unlimited storage in minutes without worries.
A Cisco router is an extremely expensive device that is hard to manage and build on top of proprietary hardware, a proprietary OS and proprietary software. What do you think will happen in a world where cheap ASIC + commodity CPU, general purpose OS and many thousands of network apps from an app store become available? Or worse, a network will no longer need many physical boxes because most of it is virtualised.
What does a cloud loser mean?
A cloud loser means that their existing cash cows will be crunched by disruptive innovations. Does this mean that losers will disappear or can not recuperate? Some might disappear. However if smart executives in these losing companies would be given the freedom to bring to market new solutions that build on top of the new reality then they might come out stronger. IBM has shown they were able to do so many times.
Let’s look at the cloud survivors.
IBM has shown over and over again that it can reinvent itself. It sold its x86 servers in order to show its employees and the world that the future is no longer there. In the past it bought PWC’s consultancy which will keep on reinventing new service offerings for customers that are lost in the cloud.
Just like PWC’s consultancy arm within IBM, Accenture will have consultants that help people make the transition from data centre to the cloud. Accenture will not be leading the revolution but will be a “me-to” player that can put more people faster than others.
X86 is not going to die soon. The cloud just means others will be buying it. Intel will keep on trying to innovate in software and go nowhere [e.g. Intel's Hadoop was going to eat the world] but at least its processors will keep it above the water.
Apple knows what consumers want but they still need to prove they understand enterprises. Having a locked-in world is fine for consumers but enterprises don’t like it. Either they come up with a creative solution or the billions will not keep on growing.
What does a cloud survivor mean?
A cloud survivor means that the key cash cows will not be killed by the cloud. It does not give a guarantee that the company will grow. It just means that in this revolution, the eye of the tornado rushed over your neighbours house, not yours. You can still have lots of collateral damage…
IaaS = Amazon. No further words needed. Amazon will extend Gov Cloud into Health Cloud, Bank Cloud, Energy Cloud, etc. and remove the main laggard’s argument: “for legal & security reasons I can’t move to the cloud”. Amazon currently has 40-50 Anything-as-a-Service offerings in 36 months they will have 500.
PaaS & SaaS = Salesforce. Salesforce will become more than a CRM on steroids, it will be the world’s business solutions platform. If there is no business solution for it on Salesforce then it is not a business problem worth solving. They are likely to buy competitors like Workday.
Google is the king of the consumer cloud. Google Apps has taken the SME market by storm. Enterprise cloud is not going anywhere soon however. Google was too late with IaaS and is not solving on-premise transitional problems unlike its competitors. With Kubernetes Google will re-educate the current star programmers and over time will revolutionise the way software is written and managed and might win in the long run. Google’s cloud future will be decided in 5-10 years. They invented most of it and showed the world 5 years later in a paper.
CSC has moved away from being a bodyshop to having several strategic important products for cloud orchestration and big data. They have a long-term future focus, employing cloud visionaries like Simon Wardley, that few others match. You don’t win a cloud war in the next quarter. It took Simon 4 years to take Ubuntu from 0% to 70% on public clouds.
What Salesforce did to Oracle’s Siebel, Workday is doing to SAP. Companies that have bought into Salesforce will easily switch to Workday in phase 2.
Since RedHat is probably reading this blog post, I can’t be explicit. But a company of 600 people that controls up to 70% of the operating systems on public clouds, more than 50% of OpenStack, brings out a new server OS every 6 months, a phone OS in the next months, a desktop every 6 months, a complete cloud solution every 6 months, can convert bare-metal into virtual-like cloud resources in minutes, enables anybody to deploy/integrate/scale any software on any cloud or bare-metal server [Intel, IBM Power 8, ARM 64] and is on a mission to completely commoditise cloud infrastructure via open source solutions in 2015 deserves to make it to the list.
Metaswitch has been developing network software for the big network guys for years. These big network guys would put it in a box and sell it extremely expensive. In a world of commodity hardware, open source and scale out, Clearwater and Calico have catapulted Metaswitch to the list of most innovative telecom supplier. Telecom providers will be like cloud providers, they will go to the ODM that really knows how things work and will ignore the OEM that just puts a brand on the box. The Cloud still needs WAN networks. Google Fibre will not rule the world in one day. Telecom operators will have to spend their billions with somebody.
If you are into Windows you will be on Azure and it will be business as usual for Microsoft.
In an ODM dominated world, ARM processors are likely to move from smart phones into network and into cloud.
Nobody knows them but they are the ones designing everybody’s hardware. Over time Amazon, Google and Microsoft might make their own hardware but for the foreseeable future they will keep on buying it “en masse” from ODMs.
What does a cloud winner mean?
Billions and fame for some, large take-overs or IPOs for others. But the cloud war is not over yet. It is not because the first battles were won that enemies can’t invent new weapons or join forces. So the war is not over, it is just beginning. History is written today…