Log in

entries friends calendar profile Previous Previous Next Next
Jef"I am the pusher robot"Spaleta - Does EC2 have security best practises?
ramblings of the self-elected Fedora party whip
Does EC2 have security best practises?
After being attacked by an EC2 server running an http://alestic.com/ built image.. it got me to thinking. What are the EC2 security best practises?

Has anything changed at all since Russell Coker wrote about this?  http://etbe.coker.com.au/2008/10/13/ec2-security/

Russell goes into some detail concerning EC2 approach to kernels specifically. But I have some more basic questions about the security culture populating the EC2 customer space. I think EC2 may have very big problem with breeding a culture of insecurity.  EC2 lowers the barrier of getting a virtual host up and running.... but doesn't not balance that lower barrier with mandated security practises. 

Out of all the shared AMIs on EC2 how many have ssh enabled root with password authing out of the box? How many people running those AMIs or derived private AMIs have punched a hole in the AWS firewall to allow ssh access from anywhere?  Are the AWS admins tracking how many EC2 instances are attempting to ssh password auth against other EC2 instances ? How widespread is ssh password attack attempts internal to EC2?  Some aggregate AWS firewall log datamining should be able to give a picture of that.

How many of the public AMIs come out of the box with known vulnerabilities?  Public AMIs sit on a shelf, and may not get vulnerability updates for months at a time..if ever.  Are people doing the same with private AMIs derived from those shared AMI's..not patching them for security for months at a time? How long does a vulnerability persist in the cloud? Longer than traditional IT shops or remote hosting setups?
Does the EC2 pricing model discourage users from aggressively applying security updates?

Do AMI cataloguing have any sort of "heritage" or "series" metric so that you can tell how individually registered AMIs relate to each other? If a publicly shared AMI like alestic's  Gutsy image reaches EOL (this month)...do all the private derived images based on that AMI reach EOL as well or do they live on without access to security updates? How long do they live on? Forever?  The fact that Amazon is still offering public AMIs for linux distributions that have gone EOL indicates the answer to this..is the wrong answer. It's really not cool to continue to offer AMIs that are EOL and won't be receiving security updates. This is a problem.

How many people are running updated AMI that do correct for vulnerabilities when an updated AMI becomes available?  When public AMI's are revved for security, are the people running private AMI's derived from that public AMI encouraged to rev their private AMI as well or do people patch forward manually?  Does the EC2 pricing model encourage or discourage frequent revving of AMIs to include security updates?
How frequent is frequent enough? Once a quarter? Once a month? alestic seems to be doing it once a quarter, but are people picking up the updates are rebasing their private images from those updated shared AMIs?

How many AMIs use selinux policy as a layer of protection out of the box?  Is Amazon doing anything in terms of making it easy find the AMIs which do support selinux out of the box? Are they making it possible to re-bundle instances with functional selinux policies?  Selinux has demonstrated track record of mitigating vulnerabilities..when used. 

8 comments or Leave a comment
From: dmalcolm Date: April 13th, 2009 06:53 pm (UTC) (Link)
Typo: Russell's surname is "Coker", not "Cooker", BTW

The problem with computer security, as in some many other fields, is that economic incentives aren't set up correctly.

With some tweaking, economics may favour security here: the owner of a vm is being charged per-unit for that vm's activities. So if you're being charged e.g. for the CPU and network bandwidth costs of the denial of service attacks that a vm you're "responsible for" is making, then you have a financial incentive to ensuring that the vm is doing what you want it to, not what a botnet controller wants it to. Of course, it all depends on the numbers. If the cost to the vm creator of an unsecured vm is lower than the cost to some victim (or of externally-born costs) that doesn't solve anything.
jspaleta From: jspaleta Date: April 13th, 2009 08:01 pm (UTC) (Link)
Fixed his name...

I think there is also a cost associated with bundling new AMIs with security updates I believe. But I'd be glad to be shown I'm wrong on that. I'm not sure the cpu/data throughput costs that EC2 is charging clients is going to be an incentive to actually keep things updated.

For hosted systems that are doing reasonable amounts of data throughput.. the cost of moving "real" data around in the cloud to and from dedicated storage might overwhlem the incurred cost of parasitic bandwidth usage of a bot on that host. We'd have to see workload patterns to get a feel for that. Amazon might have to supply some very detailed reporting concerning outbound and inbound traffic. I wonder, do hosts get charged for the bandwidth of incoming ssh password attack attempts which fail? I'm sure there are more than enough examples of that sort of activity to ballpark the parasitic cost of that sort of attack. What's the cost of 2000 failed ssh login attempts? 1/100 of a penny?

Data access internal to the EC2 segment you are own is free of charge. Smart attackers who primarily attack other ec2 hosts aren't even going to show up on bandwidth charges...just cpu. How many ec2 internal ssh dictionary attack attempts does one cpu hour buy you?

From: dmalcolm Date: April 13th, 2009 08:19 pm (UTC) (Link)
Yes, looks like there are some very real asymmetries in:
- the damage an intruder can do to 3rd parties vs the cost incurred to the nominal owner of the vm during that attack
- the cost to the owner to keep the system updated vs the cost of letting the box get rooted
and that a pure economic model isn't good enough.

It seems that the cloud provider needs to provide a system for handling complaints from 3rd parties. If a box has been rooted, and is attacking 3rd parties and is shut down, that will impact the nominal owner of the vm. Presumably that will inconvenience the nominal owner enough that he/she has an incentive to keep the vm secure.

Economics _might_ help defeat the spam case: if you find you're paying the electricity bill of a spammer, you're probably incentivized to secure the vm.

From: ehammond Date: April 13th, 2009 09:40 pm (UTC) (Link)

EC2 and security


Lots of great questions and I encourage folks to keep raising and exploring these issues. Since security depends in part on sysadmins doing their job (and knowing what their job is), I think we're always going to have issues, especially since many less experienced folks don't realize they are sysadmins. Here are partial responses to some of the points you brought up:

- Though you bring up a lot of great security issues, I'm not sure many of them are that much more prevalent in environments like EC2 than they are in other data centers around the world. How many non-EC2 sysadmins are keeping their running servers up to date with the latest security patches?

- You ask: "If a publicly shared AMI like alestic's Gutsy image reaches EOL (this month)...do all the private derived images based on that AMI reach EOL as well or do they live on without access to security updates? How long do they live on?"

=> Perhaps I wasn't entirely clear in the wording: It's not the image that is reaching EOL, but Ubuntu 7.10 Gutsy itself. There will be no further security updates released for Gutsy after the 18th, so I recommend not using it.

=> The Gutsy AMI and any other AMIs based on it have no time bombs built in. They will continue to be available and run just like any distribution CD. You just *shouldn't* use them.

- You ask: "When public AMI's are revved for security, are the people running private AMI's derived from that public AMI encouraged to rev their private AMI as well or do people patch forward manually?"

=> I think different people take different approaches to this. Both methods are valid, though I think folks should be doing normal updates more often than new AMIs are released.

- You ask: "Does the EC2 pricing model encourage or discourage frequent revving of AMIs to include security updates?"

=> Not as such. It's financially inexpensive to store multiple AMIs. However, it tends to be expensive in terms of human labor to build and customize AMIs, which is one reason I encourage folks to write scripts to automate it so that it can be done easily in the future.

- You ask: "Do AMI cataloguing have any sort of "heritage" or "series" metric so that you can tell how individually registered AMIs relate to each other?"

=> Amazon's AMI catalog on http:/ec2ami.notlong.com hasn't changed much over time and definitely needs an update. I've tried to document the relationship of the base AMIs I built on pages like http://ec2hardy.notlong.com but this is a lot of manual effort to maintain (See the "History" section).

- One point which you didn't mention but which does relate to security on EC2: It is not possible for normal users to patch kernels when security updates come out. We can only run the kernels provided by Amazon or trusted partners like Red Hat and Canonical. As far as I can tell, Amazon does not have a practice of releasing new kernels when security patches come out. I don't even think it would be possible for them to do so in such a way as to affect existing AMIs which default to the old kernel.

Also, since I happen to build the AMIs listed on http://alestic.com, I thought I'd answer a few questions about them:

- The images on http://alestic.com follow the standard Amazon EC2 security approaches with the root password disabled, only allowing ssh access using the keypair specified at launch.

- The images include the latest security patches as of the day they were built.

- Though it's impossible to "update" an existing image on EC2 and keep the same AMI id, I do release regular updated images with all security patches, and encourage folks to always start with the latest release (and still perform further updates after running it).

- I choose to not delete older images because there may still be companies depending on them. Even though we might think that they are wrong to start with old software, deleting the image would be like Canonical or Red Hat building a time bomb into their software where the entire OS shut down and would not reboot when the release reached EOL.

Eric Hammond
jspaleta From: jspaleta Date: April 13th, 2009 11:41 pm (UTC) (Link)
I think you take a very cavalier approach to image building. Encouraging companies to install and maintain network facing systems that can no longer receive security updates is an extremely bad cultural norm.

By continuing to make images available well past EOL and with no alternative source of security updates identifies you are helping to sustain bad practises.

It's not about adding a time bomb.. its about being honest about the role of security updates. No one should be "relying" on you or your image if you can't keep it secure past EOL. Refusing to decommission it, cost everyone in terms of security. You have to take responsibility for the fact that you are making it easier for people to choose not to upgrade to a supported release. Suck it up and be responsible.

Hell man, even Canonical drops its entire update tree from archive.ubuntu.com after a release goes EOL. They don't make it easy for you to patch forward a Feisty install. If you happen to find mirrors still carrying any of the updates today, good for you. Want to take bets on how long Canonical keeps the Gutsy update repositories around?

-jef"all of this started because I was ssh password attacked from a host running one of your images..but im not bitter"spaleta
From: ehammond Date: April 14th, 2009 04:49 am (UTC) (Link)

AMI updates


You say, "I think you take a very cavalier approach to image building". I appreciate your directness and can understand your position, but the way that I think about it is that I take a very serious approach to sustaining stable images which companies might be depending on to run their business.

Systems built on EC2 are different from systems running in the historically standard situations. In normal systems, the OS is installed once and it basically sits there for months or years with upgrades (hopefully) being applied. If a mirror happens to go down accidentally or because the provider decided to remove support for a particular release, then the server will probably keep running with no interruption.

In EC2, it is common to build an architecture which scales with demand. This is often done by firing off one or a hundred instances of a particular AMI, with each server perhaps installing and configuring some software before it goes off and does its job.

If that AMI is removed, then the entire business based on that architecture can go down because new servers will stop running.

I would love to get some tools from Amazon which let me know which images are being run actively by EC2 users. I would also like a way to be able to send notices to customers who are using specific images to give them security alerts and notices of available upgrades.

If these features were added, I would feel much better about giving notice to people using old images, waiting for them to stop using them, and then decommissioning the images.

As it stands, I'm providing products which I spend my own time creating and my own money sustaining but I have no way of knowing exactly how popular they are or of contacting my customers.

I agree with much of what you say as a valid point of view, and admit that I have chosen to support possibly bad decisions of my customers (though some may have very valid reasons and their systems may not even be network facing). However, I would like to offer a couple points in my defense:

1. I regularly release updated AMIs and encourage folks to use the latest one. In the 14 months that I have supported Gutsy, I have released 20 Gutsy AMI updates. I don't think you'll find any other AMI publisher who releases updates on public AMis as often as I have.

2. I have created a community around the images I have built and post announcements every time a new update is released or when a release is reaching end of life.

I do take security seriously and am trying to make the best choices given the EC2 AMI publishing environment and my customer focused priorities.

Ok, enough defending :) I do take your comments seriously and will think about them as things go forward and as I talk with Canonical about their plans for the official Ubuntu images.

BTW, I am extremely curious about how you found out what image a remote EC2 instance was running if it wasn't one of your own instances. It seems like this must have been information provided to you by Amazon. Do you happen to know the specific AMI id?

Eric Hammond
From: (Anonymous) Date: April 14th, 2009 09:07 am (UTC) (Link)

What's the difference?

What's the difference between a virtual EC2 server and a real server, except that they can conveniently be labelled under the EC2 umbrella? Surely this is a generic problem, not an EC2 problem?
From: (Anonymous) Date: November 23rd, 2009 08:47 am (UTC) (Link)

Quotation of Plato

Entire ignorance is not so terrible or extreme an evil, and is far from being the greatest of all; too much cleverness and too much learning, accompanied with ill bringing-up, are far more fatal.
Quotation of Plato
8 comments or Leave a comment