Comments on Agile Testing: Experiences deploying a large-scale infrastructure in Amazon EC2

Way to go Grig. Saw the post on HighScalability! H...

2016-03-26T05:36:02.237-07:00

Way to go Grig. Saw the post on HighScalability! Hope all is well we miss you over here!;-)

Nati -- I did look at the Gigaspaces products a wh...

2016-03-26T05:32:13.664-07:00

Nati -- I did look at the Gigaspaces products a while ago. We preferred to roll our own, because we have certain needs that might not be fulfilled by a generic management platform as yours. But I'll keep an eye on your offerings.

Great Article! Thank you!

2010-04-22T05:27:30.697-07:00

Great Article! Thank you!

Super Great article. Well written. You inspired me...

2009-08-25T18:59:39.395-07:00

Super Great article. Well written. You inspired me to get into blogging :)

Whatever you do, make more time to blog!!!

Shlomo -- we are attaching EBS volumes to instance...

2009-08-11T11:52:09.627-07:00

Shlomo -- we are attaching EBS volumes to instances at instance creation time, from a management node. When the instance starts up, it knows via user data what volume to mount (device name) and where to mount it (local directory).

Grig

I like the idea of never needing to SSH into an in...

2009-08-10T10:41:32.612-07:00

I like the idea of never needing to SSH into an instance. But if the instance needs to be able to attach its own EBS drives and access S3 then you need to put your EC2 credentials on the instance somehow.

Do you burn them into the AMI?
Do you pass them in via the User-Data?
Do you keep them encrypted in the AMI and pass in a decryption key in the User-Data?

Awesome article. Thank You.

2009-06-04T08:27:20.262-07:00

Awesome article. Thank You.

Grig - great story. "I will blog separately abo...

2009-05-22T08:55:17.145-07:00

Grig -

great story.

"I will blog separately about how exactly slack works for us"

really keen that story too ;)

-- Martin

for EC2 monitoring I will suggest http://www.monit...

2009-05-12T06:35:00.000-07:00

for EC2 monitoring I will suggest http://www.monitis.com, you can get up and running monitoring in a couple of minutes.

This is interesting. But keep in mind that a hos...

2009-05-05T22:40:00.000-07:00

This is interesting.

But keep in mind that a hosting service like Amazon can have some pretty large consequences especially as you reach larger scales. Just because its cheap when your small doesnt mean there isnt a more cost effective solution as you grown. Keep in mind Economies of Scale. Not to mention many other hidden costs.

Take a look at this article today from SaaSblogs.com. If Salesforce.com were just single tenant running in a hosted EC2 environment.

http://www.saasblogs.com/2009/05/05/what-if-salesforcecom-werent-multi-tenant/

Good stuff - especially because it's based on hand...

2009-04-25T21:29:00.000-07:00

Good stuff - especially because it's based on hands-on experience. Thanks for sharing.

Frederic -- while I'm happy to blog about technolo...

2009-04-21T16:09:00.000-07:00

Frederic -- while I'm happy to blog about technologies that we use at OpenX, I don't want to go into details about specific business decisions we've made (such as which exact portions of our infrastructure we're hosting in EC2, or what our peak usage is, etc.).

I prefer to blog here rather than write an article in a newspaper. Thanks for inquiring though.

Grig

Is this system built to host the hosted part of op...

2009-04-21T15:24:00.000-07:00

Is this system built to host the hosted part of openX.

What is the lowest number of instance you are running and how much instance max do you start at peak load.

I'm looking for big player in Switzerland that moved to the cloud for an economical article in a swiss newspaper. Maybe I will find some, but in case I won't find any, are interested?

Frederic -- we don't currently scale up and down c...

2009-04-21T15:05:00.000-07:00

Frederic -- we don't currently scale up and down continuously, based on load average. We are working on coming up with an algorithm that makes sense for us, which will include load average, but also other important metrics such as overall latency etc.

For now, we are scripting the bulk termination and deployment of instances, but that operation is triggered manually based on peak vs. non-peak traffic (which usually corresponds to day of week). This is currently sufficient for our needs, because it still automatically modifies the various pieces of infrastructure that are involved, such as load balancers.

Grig, I think you just begin to work with the clou...

2009-04-21T14:51:00.000-07:00

Grig, I think you just begin to work with the cloud and regaring how much money you are going to save, this will come after few months.

Can you share how you manage to start/stop instances based on the peak. Do you use LA like scalr.net does or do you use some other algorithm?

I like the cloud every day more and more, but some economical journalists are asking us how much a big player could save and as I have never used anything else as the cloud to host web services, I cannot really compare. This is why Amazon points me to your blog post. You will probably able to talk about the calculable and uncalcalculable effets of migrating to the cloud in a near future.

Frederic -- to answer your first question, we didn...

2009-04-21T14:43:00.000-07:00

Frederic -- to answer your first question, we didn't want to use a 3rd party product such as RightScale, scalr.net or GigaSpaces because we have certain requirements that would be hard to achieve using a more general 3rd party tool. We preferred to roll our own infrastructure deployment tools, which give us the very fine grained level of control we need in terms of automated 'elastic' deployments.

As for the financials, I can't comment too much except to say that we are saving money by going to EC2. The fact that we're paying by the hour allows us to save money during off-peak hours by terminating instances that are not needed. Of course, to achieve this you need to have a completely automated way of terminating and deploying instances of different types (web servers, db server, haproxy load balancers etc). This works very well for us. So in this case, extreme flexibility and scalability actually contribute towards financial savings.

Another thing I'd like to say is that after you enjoyed the benefits of deploying EC2 instances automatically, whenever you need them, you find it very restrictive to work in a traditional hosting environment. It's like floating in the clouds versus carrying a heavy terrestrial weight ;-)

Grig

Grig, I follow the openX project for a long time (...

2009-04-21T14:18:00.000-07:00

Grig, I follow the openX project for a long time (since it was called phpadsnew ;-) and I'm very happy you are using Amazon too. A european Manager of Amazon send me the link to your blog post and after reading it I was wondering why you didn't use scalr.net to manage what you achieved with openX and Amazon. We are also heavy user of Amazon WS and scalr.net too and this tool saved us a lot of time ;-)

I have a second question regarding the Amazon experience in regards to your old way of hosting openX. Can you share any info regarding the economic facts. We are a startup and we never used anything else than Amazon and cloud computing, but I'm looking for info about big player like you who were hosting openX traditionnally and are moving to the cloud for flexibility, scalability and maybe to save some money. I really would like to hear about this last thing.

Dan -- we do run all our DB servers with the DB fi...

2009-04-15T18:07:00.000-07:00

Dan -- we do run all our DB servers with the DB files on an EBS. I thought I made that point in my post...

Grig

Why don't you use Amazon's Elastic Block Store sto...

2009-04-15T18:03:00.000-07:00

Why don't you use Amazon's Elastic Block Store store guarentee you DB exists if all your DB instances die for some reason? Is it a cost issues? Or you trust that you will always have an DB instance running.

Good article,.... i love this one, stay update yea...

2009-04-15T07:43:00.000-07:00

Good article,.... i love this one, stay update yeah !!!

Python rocks! Thanks for such helpful information ...

2009-04-14T11:38:00.000-07:00

Python rocks!
Thanks for such helpful information on better using Amazon EC2!! :)

You should check out OpSource. We handle all of t...

2009-04-13T16:48:00.000-07:00

You should check out OpSource.
We handle all of this for you and behind a firewall.

Load balancing, firewalls, multiple servers, millions of users, 24x7 end user support, app management...the list goes on. And we handle some of the largest and most robust applications on the web today. We deal with this everyday.

Why do it yourself?
Leverage expertise and get rid of the down time.

Nati -- I did look at the Gigaspaces products a wh...

2009-04-11T08:19:00.000-07:00

Kent -- thanks for the pointers to resources, I'll...

2009-04-11T08:18:00.000-07:00

Kent -- thanks for the pointers to resources, I'll definitely check out the MySQL-related ones.

Grig

Take a look at this article as you might want to r...

2009-04-10T14:42:00.000-07:00

Take a look at this article as you might want to rethink round robin with your HAProxy. link