Elastic Security

Icon

Security for the Cloud

February 19, 2010 • 12:44 0

Programming the Cloud: Cloudy_Scripts

Scripting is indispensable to automate IT infrastructures. IT automation gets even more important when resources are run in cloud infrastructures due to higher frequency of changes: usecases include scaling up to cope with peaks of load, scaling down to more efficiently use resources, or temporary usage for developing, testing, or rolling out marketing campaigns. There are a couple of commercial (e.g. RightScale) and open-source solutions (e.g. Chef) with quite impressive capabilities, which also require some significant effort to learn correct usage.

However, we found that sometimes we only need to run a script once. For example, we wanted to encrypt an EBS storage for an instance or create a bootable instance from a snapshot. We packed our scripts into an open-source project (written in Ruby) and wrote an easy-to-use web-application called Cloudy_Scripts that presents the necessary input information in a form and provides status updates and log-messages to track progress. Cloudy_Scripts does not require registration, credentials for the cloud providers are not stored anywhere. The scripts basically use the cloud provider API and ssh-wrapped bash-commands to fulfill their goal. We start with two scripts for now.

Please let us know if Cloudy_Scripts is helpful for you, if there are other scripts you would like to be added, if you detect bugs or flaws, or if you want to participate in the open-source project.

Filed under: Internals, Solutions,

January 19, 2010 • 11:12 0

Privacy in Hostile Environments?

Foto: Mykl Roventine

Mark Zuckerberg says that the age of privacy is over. Those who feel socially invulnerable and are totally confortable to give control over their personal data into the hand of American startup companies might skip the rest of this article. Those who believe that privacy will remain a precious asset and precondition of freedom and democracy in the future, might start to worry if the rising use of web-services and the ubiquity of access to all kind of potentially confidential information concerning their life or the company they work for might become a threat important enough to abstain from using those kind of applications in certain contexts.

What options do exist besides full trust in the cloud providers and negociating appropriate SLAs? Is it possible and technically feasible to manage privacy within the web without relying on the web-application providers like Google, Salesforce or Facebook to respect privacy concerns and implement the necessary measures to avoid unsollicited access to private data and abuse from outside and inside the service platform?

Actually, the question is not a new one and the common response to it is the use of proxies that intercept confidential data and replace them with anonymized data. This must happen in a completely transparent way to not break the system. A prominent example is the use of anonymization proxies (e.g. Proxify) to hide IP addresses to ISPs. However, protecting data that is stored on the system of a SaaS provider is much more sophisticated.

Here are some ideas how such a solution can be realized to anonymize specific data used in web-applications.

Network: Van Jacobson proposes a solution on the network level by switching from the current location based architecture to a new paradigm called  Content Centric Networking that uses content objects as the principal abstraction and that allows to build in security features on the data level. This idea probably will remain an idea for a long time since it would represent a revolution that requires replacing robust and well-understood equipment on a running system all over the world.

Database: Another idea is to use a proxy between web-application and database, which would need to be deployed on the premises of the SaaS provider. The proxy intercepts SQL queries between application server and database, identifies confidential data and replaces them. Some advanced Database Firewalls are able to identify user-based data-streams and match them against their firewall rules. The advantage of this approach is that it works generically for all web-applications without changing any line of code. However, the architecture is quite complex and has several open questions: how to manage keys (must be stored outside the SaaS provider)? Whom and how to specify, which kind of data to be protected?

Browser: Yet another possibility would be to let the proxy work at the client side, for example, as a browser plugin that either intercepts Javascript calls on the application level or HTTP requests/responses to anonymize data. The biggest question here is if it is possible to have a generic solution that works for all kind of web-applications and that doesn’t harm the integrity of the application.

But maybe the answer will not be a technical but a political one? The awareness for data privacy is growing strongly, at least in Europe. France ponders a Right-To-Forget law. Will the solution be at the end in the hands of politicians and judges?

Filed under: Privacy, SaaS, Solutions

January 7, 2010 • 16:15 0

Data Remanence in the Cloud

Foto: S.Bär

Any critical data must not only be protected against unauthorized access and distribution, but also securely deleted at the end of its life-cycle. For organizations storing information related to health, financial or defense it is mandatory to ensure that no data is left on disks from where it is exposed to the risk of being recovered by malicious users. This problem is generally referred to Data Remanence.

When you have full control of your file servers, you would use tools like this that overwrite the corresponding sectors on the disk several times to literally destroy any physical trace of a file. But how would you do this in the cloud?

The technique of overwriting file sectors does not work without the collaboration of the cloud provider. You are not given access to the physical device, but only to higher level abstractions like file-systems (e.g. Amazon EBS) or key-value based APIs (e.g. Amazon S3). In SaaS/Paas environments, access only happens on the data level. Until cloud providers start paying attention to this issue (I am not aware of even a single provider) and offer secure deletion as a feature of there services, there is only one solution that works already today at least on IaaS platforms: strongly encrypt your data and keep the key at a safe place, i.e. outside the cloud where your data is stored. Secure deletion then becomes nothing more than destroying the key.

Filed under: Solutions

November 18, 2009 • 11:44 0

Alternatives to AMIs?

Foto: ilovecode

Some weeks ago, I discussed the dangers of using Amazon Machine Images (AMIs) from third-party providers and suggested that a web-service combining profound security scanning with community feedback could be a solution to reach a good level of  transparency and trust. But are AMIs really the right abstraction for a software marketplace? Can they replace installers? Is searching for an AMI that fits the need of a customer and then launching it, the way that people want to install software? Are AMIs the determining abstraction of cloud IT?

I see two important limitations to the role of the AMI:

  • The first is the problem of sharing resources on application level: starting an AMI for every application is costly, the customer needs to pay for every running instance that is in most cases overprovisioned. At the other hand, you loose all advantages of simplicity when you deploy, maintain, and upgrade several applications on the same instance. There’s probably no AMI that deploys your preferred CRM plus your preferred bug-tracking tool plus WordPress – you need to create it yourself (and run into new questions, e.g. how to maintain and upgrade your setup).
  • The second is that launching is just not enough: instances must also be configured, e.g. with IP addresses, application specific parameters, or to achieve persistency. An AMI is not just a simpler software installer or even a DVD-like digital asset that can be played with one click, it is only one brick of a service or an application among others and it rarely works out-of-the-box.

If AMIs are not the right entity for a cloud infrastructure marketplace, which could be the right one then? One possibility could be a marketplace for “cloud deployment services” where a deployment service is basically a programming routine that takes input from the user and performs the complete setup, i.e. starting AMIs, attaching persistent storages, configuring static IP addresses, applying account information, connecting application-servers to database, encrypting storage, etc.

Such deployment services would clearly solve problem number two (configuration), but could also be enabled to reuse existing resources and thus solve the problem number one (resource sharing). At the other hand, they may become so specific that they are often needed only once. Any thoughts?

Filed under: AWS, Discussions, Solutions

November 5, 2009 • 16:45 0

Software Updates on EC2

Foto: Lady-Bug

Let’s imagine I am a small software vendor that uses Amazon EC2 for his complete IT infrastructure. Let’s say I have around 20 virtual machines on Ubuntu Linux running all my applications. There are internal applications, i.e. applications that only my company uses – like a CRM, code-repository, a document-management tool,  a project-management tool, a billing-system or a mail-server. And there are external applications that are shared with the outside such as a file-server to download documents and software, a bug-tracking tool, and a SaaS version of my products. Almost any software is also using a database instance in the backend. Of course, there are also a couple of management tools running and scripted processes for backup, integrity checks, or auditing.

So far so good. Now a new version of Ubuntu comes out that fixes a potential security leak. In addition, I want to use the latest release of MySQL for performance reasons. And there’s this new version of my bugtracking system with the long-awaited new feature. In the classical IT world, I would start software updates (e.g. for Ubuntu) or download and install the latest version of MySQL and my bug-tracking tool. How would I do it in the EC2 Cloud?

I could try it the same way. But I might run into problems like this – a new OS version might require an update of the kernel, which is not possible with EC2. So I do need to start up a new instance from an AMI containing the last Ubuntu version, reinstall and configure my software, reimport my data, and copy and reactivate my scripts. When I choose to do it like that, I must also be prepared for the (rare) case of corrupted instances or unintentional termination of an instance. In such cases, I need to do the whole process from scratch. RightScale helps to automate those kind of updates, since it allows to write and assign reusable scripts to AMIs that are executed on startup or termination of an instance.

With VMWare, I would install the new software and create a snapshot after the upgrade, which allows me to switch back immediately to a certain system state at any time I want. This feature is not available for EC2, but can be somehow simulated: the process of creating a VM snapshot corresponds to the process of bundling a new AMI. Everytime I need to upgrade, I would do my updates on the running instance (or better: a clone of that instance based on the same AMI) and then create a new AMI out of it. Elastic-Server is a service that helps you to do that more easily. Afterwards, I need to shut-down my instance based on the old AMI and start a new one based on the new AMI. However, that only works for application and service software upgrades and for OS upgrades without kernel modifications. For kernel upgrades, I still need to do the whole installation process from scratch…

Ideally, I would like to separate my AMI from the installation information of the software. What about a solution like this: I start from a clean OS-only image and store all modifications on persistent storage i.e. EBS (Elastic Block Storage). On my AMI, there is an installation script executed at startup that checks the current setup against the target setup stored on EBS.  Installing MySQL? Store the whole bin+lib+config on EBS. The installation script will check if it needs to apply modifications and gets them directly from the EBS. Upgrading MySQL means simply updating the target state on EBS. Upgrading the OS means: using another AMI (that includes the installation script – that’s the catch).

I will try to formalize those solutions and add them to the list of EC2 Patterns. Every input is welcome (especially, I am still searching for good pattern names)!

Filed under: AWS, Solutions

October 20, 2009 • 09:28 3

EC2 Design Patterns (2): Trusted Gateway

Second pattern from our series about EC2 Design Patterns.

Intent

Give API access to an EC2 instance created from a publically available virtual image (AMI) without the help of external services or applications.

Motivation

The motivation is the same as for the External Console pattern: a EC2 server instance needs access to the EC2 API for reasons of configuration, automation, or monitoring of other instances. Any access to the API requires the EC2 credentials. Those credentials allow complete control over the whole platform of an Amazon account and must therefore be stored and communicated in a secure and reliable manner.  Amazon EC2 provides a console via a web-interface that allows to deploy and startup Amazon Machine Images, but it does not allow to pass AWS credentials to the upstarting image.

Since the image is public and can be instantiated by many different accounts, the credentials cannot be stored on the image itself.

Applicability

Use TrustedGateway when

  • you want to apply an AMI that needs access to the EC2 API
  • you want to facilitate deployments and especially minimize manual interventions for the user who deploys the image  (e.g. avoid starting the image using API command line tools, edit configuration files on the image, etc.)
  • you want no other external service to manage your credentials. All credentials should be managed within EC2

Participants

  • Trusted Gateway: EC2 instance that needs the EC2 credentials for configuration or monitoring purpose
  • Controlled Instance(s): virtual instances resulting from AMI instantiation and supposed to be controlled by the Trusted Gateway
  • Admin: a person or organism that deploys the AMI and manages AMI instances
  • Trusted EBS: An instance of Amazon’s elastic block storage that is encrypted and used by the Trusted Gateway

Collaborations

  • Initial Startup
    • The Admin deploys the Trusted Gateway as any other AMI
    • The Admin connects securely (SSH/HTTPS) to the Trusted Gateway and passes the EC2 credentials to the Trusted Gateway
    • The Trusted Gateway stores the EC2 credentials encrypted with any locally stored key-pair
    • The Trusted Gateway connects to the EC2 API and retrieves all relevant information about Controlled Instances to perform his tasks
    • The Trusted Gateway uses the EC2 credentials to encrypt the Trusted EBS storage and store all relevant information needed to perform his tasks
  • Reboot
    • The Trusted Gateway retrieves and decrypts the EC2 credentials stored locally
    • The Trusted Gateway connects to the EC2 API and retrieves all relevant information about Controlled Instances to perform its tasks
  • Recover
    • When the Trusted Gateway instance was completely destroyed (due to termination or a fatal AWS error), a new instance without the stored EC2 credentials is instantiated by the Admin
    • The Admin connects securely (SSH/HTTPS) to the Trusted Gateway, passes the EC2 credentials to the Trusted Gateway, and selects the EBS that was used before
    • The Trusted Gateway stores the EC2 credentials encrypted with any locally stored key-pair
    • The Trusted Gateway recovers by using the information on the Trusted EBS and continues his tasks

Consequences

The Trusted Gateway offers the following benefits:

  • No external service is needed to manage the platform and the security credentials; thus no additional security risk
  • User does not need to intervene when the Trusted Gateway is rebooted (that is in normal operation)

At the other hand, it may have the following drawbacks:

  • Initially and for recovery, the user needs to provide the credentials
  • No strong security: the credentials are stored on the instance and encrypted with a key on the machine itself (security by obscurity)

Known Uses

  • An Amazon User wrote a Script to mount EBS at startup that includes that pattern
  • Shlomo Swidler describes a couple of techniques of passing EC2 credentials securely to instances including this pattern (“How to Put AWS Credentials on EC2 Instance”, point 6)

Note: this is work in progress. Any feedback, questions, or corrections is welcome and might be integrated in an updated version. If you use the pattern, let me know!

Filed under: AWS, Discussions, Solutions

October 15, 2009 • 09:03 1

AMI Marketplace?

Foto: Digital Cat

Foto: Digital Cat

SensePost showed in a video how easy it was to make Amazon EC2 users to instantiate malicious images. After thousands of automated trials, they managed to register their own AMI with a small ID and thus have it a prominent place in the Amazon AMI catalogue. Several thousand EC2 Users instantiated it within a short time-interval without any verification.

Amazon does not provide any certification or validation of images. It doesn’t even provide an outbound filtering option in its firewall feature (security groups) that would limit the impact of malicious instances starting any kind of attacks. Hoff suggests to instantiate only images you created yourself. But that would also mean another ugly and awkward little thing to do before being able to start, another technical obstacle to use the cloud. And it doesn’t protect you necessarily against glitches in the official and generally trustworthy images of Amazon itself, neither.

One solution to his problem could be an AMI Marketplace, a simple web-portal that would provide relevant information about any AMI. That information would be retrieved by running scanning software that identifies server-processes on that AMI, open ports, scheduled tasks, integrity checks on kernel files and packages, and that generates warnings about potential viruses/rootkits/malware. In addition, user generated information like ratings and comments could be added – users of the same AMI could be easily connected. Such a portal would solve a couple of today’s issues with AMIs:

  • Lack of Integrity: the problem described above. How can I be sure that the AMI I want to use does not contain any malware? How can I be sure that the AMI contains really what its description promises?
  • Lack of Trust: How can I trust small and unknown providers? The portal could help microISVs to build a reputation in the AMI market
  • Lack of Transparency: How can I find the appropriate image for my needs? Many images contain similar software, misleading names, or insufficient subscriptions (Test: try to search for images yourself on Amazon’s AMI Catalogue).
  • Lack of Support: Often other users are the best sources to discuss issues concerning a certain AMI. How can I reach them and get feedback from them?

Isn’t there a business opportunity? Well, maybe. If the problem would be so pressing, Amazon itself would have probably already released such a portal. I suspect that the people who instantiated SensePost’s malicious AMI were simply playing around a bit with the EC2 platform. Any sensible IT administrator intending to do something serious with EC2 would either get an AMI directly from Amazon or other well-known and trusted providers like Alestic or create an image himself based on an original and official AMI created by Amazon. Maybe AMIs are not considered as a kind of DVD that simply needs to be plugged in and played, but just as a basis to work with and install other software on top. Will there be such an AMI market? What do you think?

P.S. Here is a web-service that provides more transparency than the Amazon AMI Catalogue, but still not addressing most of the problems I noted above.

Filed under: AWS, Discussions, Solutions

September 21, 2009 • 10:28 1

Amazon VPC Brief Analysis

Some weeks ago, Amazon Web services announced VPC (Virtual Private Cloud) in a move to address security requirements for enterprise customers and to provide the missing link for hybrid deployments (although some questions remain especially concerning the technology behind their offer). Since we were recently suggesting a list of requirements for a cloud VPN, we want to take Amazon’s announcement as a reason to compare and match VPC features with this list.

The overall usecase Amazon is addressing is Communication between the internal network and the cloud. Here is the list (*):

Clientless: VPC uses IPSec which is supported by the majority of security gateways, so no need for the installation of a client VPN.

Centralized management: VPC configuration is provided by the Amazon API (although not yet integrated in the Amazon Console). Existing VPN Monitoring tools already used in the internal infrastructure should also be operational in the private part of the cloud.

Authentication and authorization features : Even if integration with security groups is not yet provided, they can be expected soon. Concerning authentication the method provided is IKE Security Association using Pre-Shared Keys. Role based access control is not provided by Amazon.

Integration with endpoint security: VPC targets the security of communication, not providing endpoint security. However, enterprises may deploy existing endpoint security products within the AMIs in the VPC.

Advanced logging and reporting: In our opinion, this is the Achilles’ heel of AWS – and VPC is no better. No information is provided at the network and firewall level.

Support of different communication methods and devices: We do not know yet if  multicast will one day be supported in EC2 and VPC. Concerning devices, Amazon announces that “We also plan to support Software VPNs in the near future.”

High availability: Only one VPC can be configured per AWS account for the moment. No elastic load balancing is available so it is up to the customers to construct their HA solution.

Static addressing: Today it is possible to specify a subnet, but the IP address is randomly picked within the subnet. You cannot use elastic IPs. These restrictions are expected to be dropped by amazon in the roadmap.

Conclusion: Even though there are a couple of requirements where VPC falls short, VPC is an important first step towards IaaS security and it will help customers to confidently move to the cloud. It lays the ground on which customers can built upon and extend their security architecture into the public cloud.

(*) Green: works out of the box. Yellow: works partly or can be achieved with additional reasonable efforts of the customer. Red: not fulfilled.

Filed under: AWS, Discussions, Solutions

August 20, 2009 • 10:49 7

EC2 Design Patterns (1): ExternalConsole

First pattern from our series about EC2 Design Patterns.

Intent

Give API access to an EC2 instance created from a publically available virtual image (AMI).

Motivation

Sometimes it is necessary that a server deployed on the EC2 platform has access to the EC2 API itself. Examples include access to persistent storage, instantiate other machine images to scale an application, reassign IP addresses, monitor other instances, change security groups, create key pairs, create snapshots for backup, and more. Any access to the API requires the EC2 credentials. Those credentials allow complete control over the whole platform of an Amazon account and must therefore be stored and communicated in a secure and reliable manner.  Amazon EC2 provides a console via a web-interface that allows to deploy and startup Amazon Machine Images, but it does not allow to pass AWS credentials to the upstarting image.

Since the image is public and can be instantiated by many different accounts, the credentials cannot be stored on the image itself. Credentials can be passed as so called user data when an image is started via the API.

Applicability

Use ExternalConsole when

  • you want to apply an AMI that needs access to the EC2 API
  • you want to facilitate deployments and especially avoid any manual interventions for the user who deploys the image  (e.g. avoid starting the image using API command line tools, edit configuration files on the image, etc.)

Participants

  • External Console: piece of software running outside Amazon EC2 with access to the EC2 credentials (by asking the user to enter them or by retrieving it from a database, for example). The main task of the external console is to deploy a virtual image on EC2 while passing the AWS credentials to it.
  • AMI: Amazon Machine Image, e.g. a virtual image that needs access to the EC2 API.
  • AMI Instance: the actual virtual machine instance resulting from deploying the AMI. The AMI Instance
  • Image Provider: a person or organism that create the AMI
  • Image User: a person or organism that deploys the AMI and manages AMI instances

Collaborations

  • The Image User that wants to deploy an image connects to the External Console and instructs it to deploy a certain AMI
  • The External Console retrieves the AWS credentials (e.g. from a database), connects to the EC2 API, and deploys the selected AMI while passing the AWS credentials to the upstarting AMI instance
  • The AMI instance gets the AWS credentials as user data and stores them on local storage. It uses the credentials to perform API methods (e.g. to have access to persistent storage). Upon a reboot, it retrieves the credentials from its storage

Consequences

The ExternalConsole offers the following benefits:

  • From the perspective of the image provider: flexibility to provide abstraction to the image users from all possible EC2 dependencies. The External Console component is under the control of the software provider and can be adapted transparently for the end-user to any API modifications or new features
  • From the perspective of the image user: ease of use. The credentials must be entered only once

At the other hand, it may have the following drawbacks:

  • From the perspective of the image provider: additional costs to implement and run software outside EC2
  • From the perspective of the image user, the AWS Credentials are given into the hand of an external provider and its measures to secure the access to the credentials

Known Uses

A couple of companies specialiced on managing EC2 deployments, life-cycle of images, and ease of configuration by providing an alternative web-interface to Amazon EC2 Console. All those companies have in common to store the AWS credentials on behalf of the user in their own database and use it for all operations involving the EC2 API. A list of those companies include RightScale, Enstratus, CloudKick, or Kaavo. Their approach uses the ExternalConsole pattern.

Note: this is work in progress. Any feedback, questions, or corrections is welcome and might be integrated in an updated version. If you use this pattern, let me know!

Filed under: Discussions, Solutions,

August 20, 2009 • 10:47 3

EC2 Design Patterns

Design Patterns

Design patterns are an established tool in the design and development of software systems. The basic idea behind is to identify common solutions to well-described and contextualized problems. Ideally, they allow to reuse working solutions to problems in similar contexts and to facilitate communication in a domain via a pattern language. The idea emerged from the work of an architect (Christoph Alexander) and was applied to software development in the late eightees before it gained a lot of traction with the popularity of object oriented programming.

Design patterns for EC2

When I started looking on EC2 – the API, the various existing tools, and the architectural pieces and features Amazon’s cloud infrastructure platform provides – I felt in the same situation as I was when I started developing communication systems in Java. Many other people have already be confronted with the same problem space, seeking for help in forums, and finally cobbling together their own solutions. Unfortunately, it is not easy to benefit from those other people’s work, because their ideas are dispersed over blogs and forums and are not available in a structured form using a common domain language. It is not always obvious to know if someone tried to solve the same problem if the beast gets a different name every time. That’s why I think it would make sense to start collecting solutions on EC2 and transform them into reusable design patterns.

You can find a list of Patterns below. We adhere to the conventions by the GoF. We would be happy to integrate in our series design patterns that you have come across during you work with EC2. You can leave a comment below or use the contact form for a private message here.

Collection of EC2 Patterns

  • ExternalConsole: Give API access to an EC2 instance created from a publically available virtual image (AMI)
  • TrustedGateway: Give API access to an EC2 instance created from a publically available virtual image (AMI) without the help of an external service

Filed under: AWS, Solutions

Twitter Updates

Follow

Get every new post delivered to your Inbox.