April 3, 2011 • 10:53 0
Videos and slides from the CloudOps Summit in Frankfurt are available online.
Here is the video of our Elastic Detector pitch.
Don’t hesitate to contact us if you have questions or if you are interested in a free trial.
Filed under: Internals, Presentations, CloudOps Summit, Elastic Detector
March 30, 2011 • 14:57 0
There are two terms that are referred to significantly often in discussions about cloud computing, its drivers, and its impact. The first term is “DevOps“ – a combination of the terms development and operations. It refers to the fact that the tasks of developers and system administrators get increasingly closer in a cloud-based IT world where infrastructure resources become programmable fostering application centric deployment and agile development processes. System administrators are supposed to write sophisticated scripts to automate large parts of operations and think as a developer. (Interesting Links: [here][here] and [here])
The other term is “IT Consumerization” – it refers to the observation that applications, tools, and technologies from the consumer world find their way into the enterprise. This movement has several drivers: employees that are getting more and more mobile are necessarily forced to access their data from different locations and devices (laptops, mobile phones, PCs). As a consequence, enterprise IT infrastructures become ubiquitous and heterogenous: the former one-size-fits approach of IT departments to centralize administration, management, and security of every PC, is no longer feasible today with the number of increasing devices and accelerated technological progress. Thus, employees are given more and more control about what devices and tools they can pick (BYOT – “Bring You Own Technology”). This movement opened the door into the enterprise for SaaS tools like GMail or Salesforce – but also for cloud infrastructure services such as Amazon EC2: quickly need a demo-machine? need some machines for load-testing? need to share some really big files? Amazon EC2 offers the immediate solution to it – without following the lengthy processes of the IT department that may result in rejection of the demand or a purchase with a delivery that takes several weeks. Speed and simplicity play an important role here. (Interesting Links: [here][here] and [here])
While people assume that both are just two sides of the same medal, I find they are somehow conflicting movements. The DevOps movement requires highly skilled IT workers that combine the competences of developers and system administrators and that are able to write sophisticated automation scripts. IT Consumerization means a shift from classical heavy-weight tools (such as HP OpenView, for example) to a broad variety of simpler tools (mostly SaaS tools) that focus on specific use-cases, have a much smaller feature set than classical tools, and are far easier to use. Those tools (let’s cite Pingdom for monitoring as an example, but also the EC2 Management Console) take away a lot of the burden of administrators, extremely simplify their work, and thus even allow less-skilled people to manage a big part of the IT needs of a company.
Is there an error in my reasoning? Where’s the breakup? Feedback welcome!
Filed under: Cloud Computing, Discussions, Devops, IT Consumerization
March 23, 2011 • 16:30 0
Last week I attended the CloudOps Summit in Frankfurt. The motto of this conference was “Run the Cloud” and the central idea to show how cloud computing is already used today, how hands-on solutions and architectures look like, how cloud systems are operated, and what tools are already available.
While web-startups almost immediately understand the advantages of public cloud infrastructures such as Amazon EC2, Rackspace or GoGrid and already use those intensively to avoid up-front investments into hardware, scale their infrastructure dynamically to their needs, and benefit from a pay-per-usage model, established enterprises are much more hesitant in adoption – mostly due to security concerns, fear of vendor lock-in, the costs for migrating their legacy data, or the constraints of remaining compatible with existing software. This leads to the funny situation that the big ones listen carefully to learn from the new and small ones.
The conference started with a couple of short 6-minute “lightning talks” and was followed by parallel tracks about architecture, management, operations, and presentations of startups.
Some highlights from the Lightening talks
Jean-Paul Schmetz explained in his keynote that cloud computing means that everything becomes software in the cloud – storage, memory, CPU – they all have become resources that can be created and destroyed programatically on demand. Hardware are fixed assets that requires planning, budgeting, and thus accurate predictions of the future, something very hard to achieve in a world of constantly-changing requirements and needs.
Chris Boos sees in cloud computing the big opportunity for system administrators to ged rid of the boring part of operations and maintenance and to concentrate on the interesting and challenging tasks of creating new things. Cloud computing and its inherent need for automation actually liberate the rare IT experts and revalue their skills.
Nicolas Plögert from Sharewise showed how his company outsourced almost all non-critical business processes to more than a dozen of web-services – communication, billing, customer relations management to name a few.
Florian von Kurnatkowski told us that even the automative industrie wants to make their internal network (ENX) more flexible by transforming it into a cloud infrastructure.
Startups
In the startup tracks (in which I presented Elastic Detector, our cloud security monitoring service) there were a couple of interesting products around cloud infrastructures:
ScaleUp builds software that helps providers to build their own public clouds. There focus is on account management, provisioning, and managing the “point of purchase”, i.e. the spot where providers and consumers meet.
Scalarium provides a SaaS product that allows to deploy and scale web-application for Amazon EC2.
CloudSafe allows to store and share critical documents in the cloud. All data is encrypted and different access models are supported.
CentralStationCRM is a CRM SaaS product targeting small companies that are over-whelmed by the complexity of products like Salesforce.
SemYou aims to combine the simplicity of an app-store with the flexibility of SaaS applications. There goal is that users can activate any kind of web-application with a single click on their computer that will run transparently in the cloud.
Impossible Software allows to create “dynamic videos” where logos and brands can be integrated in video templates.
Thanks to the organizers for their great work and looking forward to another CloudOps Summit in Frankfurt next year! All presentations are available behind this link.
Filed under: Cloud Computing, Discussions, Solutions, CloudOps Summit
February 9, 2011 • 16:41 1
Previous [Part 3]
Cloud Computing – Centralized or Decentralized?
After this long excurse into history, let’s come back to our initial question: is cloud-computing centralized or decentralized? Well, the answer is: both! Consider a simple web-application: parts of it is running decentralized in your browser (Ajax). The data may be stored in a single data-center – centralized, but the database is replicated on different virtual machines – decentralized. The web-application may make use of other services – decentralized, but provides its features via the same URL to thousands of users – centralized.
Does the question even matter?
The terms centralization and decentralization have always been misused in the history of computers to simplify any form of change triggered by technological progress in a fad kind of way (something similar happened in organisation theory). But for me the history of computers does not show alternance between centralized and decentralized architectures, it shows that architectures have gotten more complex and differentiated fostering more specialization and abstractions, more dedicated software and hardware components, sophisticated layers, and specific solutions. Some are centralized, others are decentralized – the term does not make sense anymore. Discussing on the pros and cons of centralization can at best be done for individual components (like databases).
Cloud Computing is the answer to the increasing demand – of entreprises as well as of consumers – for ubiquitous information in a mobile world. It reflects new forms of communication and collaboration and is far beyond the discussion on pros and cons of decentralized architectures.
This post is part of a series. Click on the links below to read the other parts.
[Part 1] The first computers
[Part 2] Closer to the users
[Part 3] Home computers and the Internet
[Part 4] Centralized or Decentralized?
Filed under: Cloud Computing, Discussions, architecture, Centralized, Decentralized
February 4, 2011 • 12:07 2
Previous [Part 2]
Home Computers and The Internet
Another technological breakthrough , the micro-processor in the early 70ties (the first commercially available micr0-process was the 4004 from Intel) changed the computer industrie in a way it never happened before. The Altair 8800 was a micro-computer construction set for around 500$ that laid the ground for computer pioneers in the USA and a democratization of access to computers. The first operating system CP/M and programming languages like BASIC created a thriving eco-system around micro-computers with applications outside the classical enterprise domain like gaming, music and graphics, learning vocabulary or maths. The computer found its way into the home – and becomes smaller, more powerful, better graphics: Apple 2, PET 2001, ZX81/Commodore 64, MacIntosh, and of course the triumph of the PC and Microsoft that somehow inherited the PC monopoly from IBM. In the entreprise, we see the raise of Mini-Computers and DEC – also for researches and smaller enterprises – that died with the raise of the PC.
Already In the 70s, computer started to be connected (remember the ARPANET project) and laid the basis for the Internet and the World Wide Web that began to conquer the world in the 90 of the last century. The first web-applications (E-Commerce, what else?) started to pop-up in the mid 90ies. Given the today definitions of cloud computing and its different flavors IaaS, PaaS, and SaaS, web-applications were the first form of cloud computing. Amazon’s web-shop, Yahoo’s directory service, Google’s search engine, Flickr’s photo sharing site, WordPress’ blogging-tool, Google’s Gmail and Docs are milestones in the history of web-applications – and what we call SaaS today. In 2006, Amazon launched S3 and EC2, services that allow to allocate computing ressources – storage and virtual servers – via the web. That means, not only applications, but complete data-centers can run somewhere else in the world. The term cloud computing was born – motivated by the little clouds used in network diagrams to describe a network like the Internet transparently.
Continue [Part 4]
Filed under: AWS, Discussions, architecture, Centralized, Decentralized
February 3, 2011 • 15:35 3
Previous [Part 1].
Closer To The Users
In the 50s, the innovation of mechanical relays (instead of vacuum tubes) made computers (e.g. the UNIVACs) already much smaller and affordable for other governmental organizations like the US Census Bureau. The first computer language FLOW-MATIC appeared. IBM entered the market with their IBM 701. In 1956, around 100 machines are installed all over the United States. IBM invented the first hard drive system (RAMAC) that laid the foundation of modern hard disk systems and that replaced the awkward punchcard system.
IBMs System/360 started the era of mainframe computers with a modular architecture that allowed to support different configurations and to replace the processor unit for upgrades. In the mid-sixties, IBM sells around 1000 machines per month. Computers became interesting for the first business applications. Still they are only feasible for large enterprises and military applications. Software and hardware were bundled, the machines not even programmable by others.
The miniaturisation of circuits, the usage of transistors and integrated allowed to enter the era of mini-computers in the 60ties and the rising of DEC. The PDP-8 is considered the first commercial micro-computer. It still had the size of two large refrigerators, yet it could be started up by a single person, didn’t develop much heat and didn’t require a cooled location so that it could be situated closed to the people that used it. Most users were researchers and larger enterprises. The computer got closer to its users. And it got smaller and smaller: the PDP-8e in the early 70ties could be put on a table already. The prices went down to under 2000$.
Continue [Part 3]
Filed under: Cloud Computing, Discussions, architecture, Centralized, Decentralized
February 2, 2011 • 11:25 2
The question if an architecture should be centralized or decentralized is one of the recurring questions in systems design, but also object of discussion in organizational and political theory. In most enterprises and especially IT departments, technical and organisational aspects fall together and even influence each other. To understand the possible impact of cloud computing on the structure of organisations, we would like to figure out if cloud computing is actually following a centralized or a decentralized architecture. Let’s therefore start with a journey through history – the history of cloud computing is nothing less than the history of computers and computer networks – and their alternation between centralization and decentralization.
The First Computers in Use
The first effectively used computer (ENIAC) occupied the space of a large building (around 300 m2), was targeted for very specialized military applications, and all instructions hardcoded and controlled by 6000 switches that could be manually controlled. The next computer (EDVAC) became more flexible, controllable and less error-prone with the use of punchcards. The next one (MIT’s Whirlwind) was the first with a rudimental user interface (a kind of radar screen). Until then, computers are large, extremely expensive, and specialized, and only one organization in the world could use it: military.
Giant monolytic building blocks on a fixed location. The first computers are without any doubt centralized.
Continue [Part 2]
Filed under: Cloud Computing, Discussions, architecture, Centralized, Decentralized
January 28, 2011 • 09:49 0
We already wrote about it recently: 2011 could be the year that sees the rising of PaaS. SaaS providers open up their platforms for third party providers and IaaS providers provide added services that simplify usage and hide low-level abstractions. The first group moves move down in the cloud-stack, the second one move up – both will meet in a more and more crowded middle.
The latest move of thought- and market-leader AWS confirms this prediction. Last week they released a service called “Elastic Beanstalk“, a clear PaaS offer that enables users to deploy Java web-applications in a Tomcat environment (apparently, other stacks like Ruby on Rails are in the pipe) by uploading their code via a graphical user-interface. AWS starts the application, provides transparent scaling, load-balancing, and fault-tolerance if needed. In so far, it is a direct competitor of Google Apps Engine, but with the difference that it still gives full control to users on OS level. That is, users can deploy and customize whatever they want on the servers and thus have all the flexibility they need, e.g. start cron-jobs to periodically normalize data in the database, use a file system to store their data instead of a database, or produce chart-images in the background – all things not possible with Google Apps engine today. Beanstalk leverages the power of the EC2 offer to address the biggest limitation of existing PaaS offers.
The service seems to be dedicated to SaaS providers, Web-Startups, E-Commerce sites that are often forced to care for all that awkward low-level stuff themselved and often don’t have the resources to do it correctly. Who else will use Beanstalk? Let me know your ideas in the comments.
Filed under: AWS, Elastic Beanstalk
January 12, 2011 • 11:45 0
What is the difference between our elastic monitoring/security tool Elastic Detector and the monitoring feature of EC2 called Amazon CloudWatch – aren’t both monitoring tools? The answer will illustrate what we understand by elastic monitoring and elastic security, and explain the difference between monitoring and detection.
Amazon CloudWatch allows to measure performance metrics of individual resources (instances and volumes). The information can be retrieved via a specific API, but also visualized as a graph within the EC2 console. The metrics currently supported are CPU, Disk I/O and Network I/O. While the API allows to retrieve information aggregated by instance-type or image-type, the console currently only displays information per resource. The API also allows to define alerts in case a metric exceeds a certain threshold.
Elastic Detector also collects performance data and even uses the CloudWatch API to retrieve them. It also sends notifications for alerts, but there are fundamental differences:
Therefore, detection is a layer on top of monitoring that uses the data produced by the monitoring layer to improve visibility, detect problems, and analyze anomalies. The ultimate goal of detection is to give users full visibility over the whole infrastructure to quickly discover problems, distinguish relevant from irrelevant information, and provide them with the tools and the context to track down the origin of those problems to eventually solve them.
Filed under: AWS, Elastic Security, AWS Cloudfront, Elastic Detector
January 7, 2011 • 10:16 1
We recently posted this article that describes how to generate Amazon EC2 read-only credentials for third-party providers. Amazon released a little helper tool this week, a Policy Generator, that facilitates building quite complex policies. The AWS blog very well describes the necessary steps to produce a policy in this post.
Following our example, the result would look like
{
"Id":"Policy1294332686329",
"Statement":[{
"Sid":"Stmt1294332597167",
"Action":["cloudwatch:GetMetricStatistics",
"cloudwatch:ListMetrics"
],
"Effect":"Allow",
"Resource":"*"
},
{
"Sid":"Stmt1294332680637",
"Action":["ec2:DescribeInstanceAttribute",
"ec2:DescribeInstances",
"ec2:DescribeSecurityGroups"
],
"Effect":"Allow",
"Resource":"*"
}
]
}
This code must be copy/pasted into a file (e.g. policy_file.txt) and then applied by the following command:
I hope that the AWS Policy Generator will be extended to allow also the generation of credentials linked to a policy. That way, the most embarrassing part for many people would disappear: the download and installation part of the command line tools.
Filed under: AWS, AWS Policy Generator, IAM
Cloudy_Scripts:
Tools for programming Infrastructure Clouds
Elastic Detector
Elastic Security for Amazon EC2
Matthias Jung
Sergio Loureiro
Frederic Donnat
Recent Comments