Amazon new distributed in-memory caching offering

Amazon is now offering the ability to add distributed in memory caching to your applications deployed in the EC2 cloud. 

This looks like a drop in replacement for the very popular open source cache implementation memcached (they have memcached protocol compatibility). It is quite interesting how Amazon leverages the in house statistics that it collects on applications being deployed on its infrastructure to decide on what new service to offer – case in point caching and many others like the RDS service too.  They do continue their march up the stack – into applications and PAAS. The boundaries between PAAS and IAAS vendors are surely blurring.

Some interesting points:

• Bringing management, monitoring and automation to caching as well – via new cache management AWS API. I am sure (if they have not already done so)– they would be including support for this in the cloud formation templates
• Interesting security angle – security groups for accessing cache – This seems to be a clear value add over vanilla memcached
• They support sharding of keys to cluster nodes –in order to dynamically resize your cluster, the client library should use a consistent hash function – this is pretty standard in the memcached world as well
• Lots of interesting cache metrics would be available from cloud watch – which makes it easy to integrate with the AWS Autoscaling service – e.g. if a cache cluster is running @ > 85 % of memory utilization then it’s time to launch another cluster node as part of the cluster
• They refer to a caching engine – hopefully they will add support for other caching engine implementations more powerful key-value store like redis (far more feature rich compared to memcached)

Ruby and cloud platforms

There is a growing trend towards utilizing the Ruby ecosystem for developing enterprise products of the scale and complexity of cloud management platforms. Most famously cloudfoundry, the polyglot PAAS platform is built on ruby – and there is evidence that the Rackspace cloud uses Ruby extensively for data center automation and they state that this helps them compete effectively with the much larger Amazon.

http://rubyconfindia.org/2011/presentations/munjal-RubyConfIndia2011.pdf

The advantages of using ruby in this context (For the ruby community these are well known!):

• Faster, agile development with fewer developers required, more flexible design
• Program to domain via DSL’s (ruby makes it easy to develop DSL’s) instead of force fitting languages to domain
• Ease of refactoring & integration with third party systems
• Huge community and lots of gems (packages) available for almost any conceivable requirement
• DSL’s also enabled the field engineers to tailor business logic instead of solely relying on the developers

An update

I had a discussion with a colleague on the reasons for the growing popularity of ruby in the enterprise products space (as compared to Java/Groovy/Grails) and here are some random observations in no particular order attempting to understand the trend :

• Once ruby had proven itself out there in the web scale environments – with  some high profile success stories – architects  & decision makers in the enterprise world have started looking at it seriously
• People who regularly code in Ruby are averse to trying out anything as verbose as Java – just the productivity / ease of use etc is unmatched – even compared to the groovy / grails world
• You would be surprised to know how rich / extensive the ruby supported/wrapped libraries are. This includes many wrapped C/C++ libraries which can AFAIK be easily incorporated into ruby
• Even if we can use large popular java libraries in groovy/grails – unless they are wrapped in the easy to use groovy/ruby style – it does not add to the agility, concise code, better mapping to domain etc.
• Popularity zoomed after the rails revolution – ruby got a huge boost from the popularity of the framework – no java web framework comes close to the simplicity of rails – and grails is just a copy !
• Some of it probably boils down to the hackers – they identify better with ruby than java (or java extensions) as java has always been considered an enterprise / boring language
• The size of community and some very capable people behind it has also led to great momentum of producing useful gems – almost everything you can think of is available as a gem
• Rise of rails and growth of agile as a movement has also coincided and been mutually beneficial