Category: python

• What's New in Euca2ools 3, Part 2: A Developer's Perspective

  3 Apr 2013 • gholms • Category: eucalyptus, python

  The upcoming version of euca2ools, version 3, completely reworks the command line suite to make it both easier to write and easier to use. Part 1 of this series discussed the user-facing changes version 3 has to offer, and today we’re going to take a look at how things improve on the developer’s side of the fence.

  A change in philosophy: declarative programming

  The developer is very much in the driver’s seat in version 1 of euca2ools. To use a car analogy, the developer directly controls the code’s direction, speed, and gearbox manually. Version 2 adds a cruise control by centralizing a lot of boilerplate code in the form of boto’s roboto module. Version 3 opts to let the developer give the requestbuilder framework a destination, step aside completely, and let it do the driving for the boring parts of the trip.

  Requestbuilder offers a set of base classes and a domain-specific language based on python’s standard argparse library that allows the developer to say exactly how something should look at the command line in addition to how it should look when given to the server all in the same place.

  What makes this so powerful is that it lets anybody with a service’s documentation and knowledge of how to use argparse write a command line tool quickly and painlessly. For instance, it took me around a day to write highly-customized command line tools for every operation Amazon’s Elastic Load Balancing service supports. Here’s the code from one of them:

  class CreateLBCookieStickinessPolicy(ELBRequest):
      DESCRIPTION = ('Create a new stickiness policy for a load balancer, '
                     'whereby the load balancer automatically generates cookies '
                     'that it uses to route requests from each user to the same '
                     'back end instance. This type of policy can only be '
                     'associated with HTTP or HTTPS listeners.')
      ARGS = [Arg('LoadBalancerName', metavar='ELB',
                  help='name of the load balancer to modify (required)'),
              Arg('-e', '--expiration-period', dest='CookieExpirationPeriod',
                  metavar='SECONDS', type=int, required=True,
                  help='''time period after which cookies should be considered
                  stale (default: user's session length) (required)'''),
              Arg('-p', '--policy-name', dest='PolicyName', metavar='POLICY',
                  required=True, help='name of the new policy (required)')]
  

  The framework hands everything inside each Arg in this code to argparse to gather input from the command line and then send the results directly to the web server using whatever name argparse gives the input it gets. For instance, whatever a user supplies using the -e option ends up getting sent to the server as a CookieExpirationPeriod parameter. With a small amount of practice it becomes quite easy to write a bunch of commands this way very quickly.

  One request, one command

  Euca2ools are built around a “one request, one command” tenet. This means that, in general, there is a dedicated command for each thing a web service can do. This philosophy naturally lends itself to the tight coupling between command line options and what gets sent to the server discussed earlier, but it also lends itself to reversing the usual relationship between web services and web service requests. Whereas one typically writes an object that represents the service and uses methods on it to send requests, in euca2ools it is the commands, and thus the requests, which are the first-class citizens. Each command that represents a request instead points to a service, rather than the other way around.

  The way this works in practice is by defining a base class for each service and a base class that all methods which use that service share:

  class CloudWatch(requestbuilder.service.BaseService):
      NAME = 'monitoring'
      DESCRIPTION = 'Instance monitoring service'
      API_VERSION = '2010-08-01'
      AUTH_CLASS = requestbuilder.auth.QuerySigV2Auth
      URL_ENVVAR = 'AWS_CLOUDWATCH_URL'
  
      ARGS = [MutuallyExclusiveArgList(
                  Arg('--region', dest='userregion', metavar='USER@REGION',
                      route_to=SERVICE, help='''name of the region and/or user
                      in config files to use to connect to the service'''),
                  Arg('-U', '--url', metavar='URL', route_to=SERVICE,
                      help='instance monitoring service endpoint URL'))]
  
  class CloudWatchRequest(requestbuilder.request.AWSQueryRequest):
      SERVICE_CLASS = CloudWatch
  

  Services can supply their own command line options in the same way as requests. After it gathers options from the command line, requestbuilder uses route_to to choose where to send it. This also provides a convenient way to tell the framework not to send an option to the server at all when a command needs to process it specially: just use route_to=None.

  Convention over configuration

  The oft-quoted programming paradigm for frameworks is just as true for euca2ools 3 as it is elsewhere. Want to make a command print something? Just write a print_result method. The result from the server gets passed in as a dictionary.

  class TerminateInstances(EucalyptusRequest):
      DESCRIPTION = 'Terminate one or more instances'
      ARGS = [Arg('InstanceId', metavar='INSTANCE', nargs='+',
                  help='ID(s) of the instance(s) to terminate')]
      LIST_TAGS = ['instancesSet']
  
      def print_result(self, result):
          for instance in result.get('instancesSet', []):
              print self.tabify(('INSTANCE', instance.get('instanceId'),
                                 instance.get('previousState', {}).get('name'),
                                 instance.get('currentState', {}).get('name')))
  

  Want to make a request do fancier preparations than argparse can do on its own? Just write a preprocess method that takes things from self.args and adds things to self.params to be sent to the server.

  class DescribeSecurityGroups(EucalyptusRequest):
      DESCRIPTION = ('Show information about security groups\n\nNote that '
                     'filters are matched on literal strings only, so '
                     '"--filter ip-permission.from-port=22" will *not* match a '
                     'group with a port range of 20 to 30.')
      ARGS = [Arg('group', metavar='GROUP', nargs='*', route_to=None,
                  default=[], help='limit results to specific security groups')]
      ...
      def preprocess(self):
          for group in self.args['group']:
              if group.startswith('sg-'):
                  self.params.setdefault('GroupId', [])
                  self.params['GroupId'].append(group)
              else:
                  self.params.setdefault('GroupName', [])
                  self.params['GroupName'].append(group)
  

  There are also a few other methods one can plug in, such as postprocess, and, for especially early-running code, configure. Expect documentation for requestbuilder that covers this in detail in the future.

  Scratching the surface

  The examples above cover only a fraction of what is possible with euca2ools 3’s new infrastructure. While you can look forward to some more advanced uses of it in later blog posts, you can also take a look at the current euca2ools code in development to see some of the interesting things one can do with it. Today’s pre-release of that code carries with it commands for all three of AWS’s “triangle” services: Auto Scaling, CloudWatch, and Elastic Load Balancing. Continuing what seems to have become a euca2ools tradition, just look for the commands that start with euscale (pronounced “you scale”) euwatch (“you watch”), and eulb (“you’ll be”).

  Packages for Fedora and RHEL 6 are available here. If you’re using another OS or want to build the code yourself you can simply clone euca2ools’s git repository’s requestbuilder branch. Requestbuilder itself is available on PyPI and GitHub. As always, I encourage you to test this code against AWS and Eucalyptus 3.3 and let me know what you think on the euca-users mailing list. If you encounter bugs, please file them in the project’s bug tracker.

  ---

• How to Override a Class Method in Python

  28 Jan 2013 • gholms • Category: python

  A class method in python differs from an instance method in a couple important ways:

  1. It binds to a class rather than an instance (hence its name). Thus, its first argument is a class, often called cls rather than the usual self.
  2. It can be called on both an instance of a class and the class itself.

  In general, they behave similarly, but one area in which they can differ is when we go to override the class method:

  class Spam(object):
      @classmethod
      def parrot(cls, message):
          print cls.__name__, "says:", message
  
  class Eggs(Spam):
      @classmethod
      def parrot(cls, message):
          Spam.parrot(cls, message)
  

  This code is broken because Spam.parrot is already bound to the Spam class. This means Spam.parrot’s cls argument will be the Spam class rather than the Eggs class that we wanted, so Spam.parrot will end up being called in the wrong context. Even worse, everything Eggs.parrot passed to it, including cls, ends up getting passed as regular arguments, resulting in disaster.

  >>>> Spam.parrot("Hello, world!")
  Spam says: Hello, world!
  >>>> Eggs.parrot("Hello, world!")
  Traceback (most recent call last):
    File "<console>", line 1, in <module>
    File "<console>", line 4, in parrot
  TypeError: parrot() takes exactly 2 arguments (3 given)
  

  To chain up to the Spam class’s implementation we need to use a super object, which will delegate things the way we want.

  class Eggs(Spam):
      @classmethod
      def parrot(cls, message):
          super(Eggs, cls).parrot(message)
  

  There’s a shortcut for the last line if you’re using python 3:

        super().parrot(message)
  

  The super object functions as a proxy that delegates method calls to a class higher up in the Eggs class’s hierarchy, and in this case it is critical in ensuring that it gets called in the right context.

  >>>> Spam.parrot("Hello, world!")
  Spam says: Hello, world!
  >>>> Eggs.parrot("Hello, world!")
  Eggs says: Hello, world!
  

  If you haven’t used super() before, here’s what it’s doing:

  1. The python interpreter looks for Eggs in cls.__mro__, a tuple that represents the order in which the interpreter tries to match method and attribute names to classes.
  2. The interpreter checks the class dictionary for the next class that follows Eggs in that list that contains "parrot".
  3. The super object returns that version of "parrot", bound to cls, using the attribute-fetching __get__(cls) method.
  4. When Eggs.parrot calls this bound method, cls gets passed to Spam.parrot in place of the Spam class.

  In general I tend to stick with the older-style syntax for chaining method calls, but this is one case where super() is simply indispensable.

  ---

• Euca2ools: Past, Present, and Future

  14 Apr 2012 • gholms • Category: eucalyptus, python

  For those who don’t know, I work on the euca2ools suite of command line tools for interacting with Eucalyptus and Amazon Web Services clouds on Launchpad. As of late the project has stagnated somewhat, due in part to the sheer number of different tools it includes. Nearly every command one can send to a server that uses Amazon’s APIs should have at least one corresponding command line tool, making development of euca2ools’ code repetitive and error-prone.

  Today this is going to end.

  But before we get to that part, let’s chronicle how euca2ools got to where they are today.

  The Past

  Early euca2ools versions employed the popular boto Python library to do their heavy lifting. Each tool of this sort triggers a long chain of events:

  • The tool translates data from the command line into its internal data structures.
  • The tool translates its internal data into the form that boto expects and then hands it off to boto.
  • Boto translates the data into the form that the server expects and then sends it to the server.
  • When the server responds, boto translates its response into a packaged form that is useful for programming and returns it to the tool.
  • The tool immediately tears that version back apart and translates it into a text-based form that can go back to the command line.

  Things shouldn’t be this convoluted. Not in Python.

  The Present

  Tackling this problem involved coming up with ways to simplify not only the code, but also the process through which they are written. This led to two major changes, upon which all of the current euca2ools code is built.

  “eucacommand”

  The first step was consolidating all of the code involved in performing the first step of this process — reading data from the command line — into one location. Each tool then simply needed to describe what it expected to receive from the command line, and the shared code would take care of the rest. For example, let’s look at part of an older command, euca-create-volume:

  class CreateVolume(EucaCommand):
      Description = 'Creates a volume in a specified availability zone.'
      Options = [Param(name='size', short_name='s', long_name='size',
                       optional=True, ptype='integer',
                       doc='size of the volume (in GiB).'),
                 Param(name='snapshot', long_name='snapshot',
                       optional=True, ptype='string',
                       doc="""snapshot id to create the volume from.
                       Either size or snapshot can be specified (not both)."""),
                 Param(name='zone', short_name='z', long_name='zone',
                       optional=False, ptype='string',
                       doc='availability zone to create the volume in')]
  

  Because there are three Params the shared code library reads three bits of info from the command line and hands them to the command’s code, which then hands them to boto, and so on.

  This methodology forms the basis for all of the current euca2ools that begin with “euca”.

  Roboto

  For a euca2ools command line tool to be useful it has to gather data from the command line, send these data to the server, and return data from the server to the user. A little-known boto sub-project written by boto developer (and former euca2ools developer) Mitch Garnaat, roboto, takes this statement literally and opts to let tools work at a lower level: instead of translating data from the command line into an intermediate format to send to boto, tools send these data directly to the server in the form that the server expects. The effect of this is that of essentially removing boto from the euca2ools code base altogether. By removing boto from the path that data have to take to get from the command line to the server and back, roboto makes tool writing and debugging simpler because there is less code to walk through and understand.

  Roboto is the basis for all of the current euca2ools that begin with “euare”.

  The Future

  That is the state of the code today. Where do we go from here? While roboto allows one to create command line tools with a minimal amount of effort, it has several rough edges which prevented it from taking off and which make it sub-optimal for building out the hundreds of commands that the euca2ools suite will soon need to cover:

  • User-unfriendly — When a user types something wrong or forgets to include something, roboto’s messages are often uselessly terse and unhelpful.
  • A steeper learning curve than necessary — Roboto contains a large amount of custom code dedicated to fetching information from the command line. This steepens the learning curve for people who want to contribute code or fix bugs.
  • Too much hardcoding — Roboto assumes that all tools do certain things, such as ascertaining what keys they should use to access the cloud, the same way.
  • Still more work than it has to be — Though it makes writing tools simpler, roboto still hands each tool a bucket of information and expects the tool to pick out the bits the server needs and send them onward.

  Enter requestbuilder

  Requestbuilder is a new Python library that attempts to rethink the way roboto works in a way that is more familiar to the typical Python developer and requires less custom code to run. The easiest way to illustrate this is with an example.

  A command line tool embodies a specific request to the server, so each such tool defines a Request that describes how it works:

  class ListUsers(EuareRequest):
      Description = 'List the users who start with a specific path'
      Args = [Arg('-p', '--prefix', dest='PathPrefix', metavar='PREFIX',
                  help='list only users whose paths begin with a prefix'),
              Arg('--max-items', type=int, dest='MaxItems',
                  help='limit the number of results')]
  
      def main(self):
          return self.send()
  
      def print_result(self, result):
          for user in result['Users']:
              print user['Arn']
  

  Those familiar with Python’s argparse library will recognize the code inside Arg(...), because requestbuilder does away with roboto’s custom code for reading things off the command line and instead lets argparse do the work. This cuts down on the amount of code we need to maintain, makes tool writing easier for developers who are already familiar with the Python standard library, and makes command line-related error messages much more user-friendly.

  When the tool starts running, requestbuilder uses data from the command line to fill in a dictionary called args and runs the tool’s main method, whose job is to process this information and fill in the portions of the request that will be sent to the server: params, headers, and post_data, and then run the send method to send it all to the server and retrieve a response. Attaching each of these sets of data to the request instead of passing them around between methods allows one to send a request, tweak it, and send the tweaked version as well.

  Why doesn’t the code above fill any of these things in? Since most of the data that comes off the command line goes directly to the server, when a tool runs send requestbuilder will automatically fill in params from the contents of args so the tool doesn’t have to: whatever the user supplied with --prefix at the command line gets sent to the server with the name PathPrefix, and so forth.

  But what if something should not be sent to the server? While data from the command line go into params to be sent to the server by default, one can tell requestbuilder to send a particular bit of data elsewhere instead:

  Arg('--debug', action='store_true', route_to=None)
  

  None instructs requestbuilder to leave the “debug” flag alone and not attempt to send it anywhere. Data can also go elsewhere, such as to the connection that gets set up as the tool contacts the server:

  Arg('-I', '--access-key-id', dest='aws_access_key_id', route_to=CONNECTION)
  

  Astute readers will note that I haven’t described what EuareRequest in the earlier example does, so here is the code for that:

  class EuareRequest(BaseRequest):
      ServiceClass = Euare
      Args = [Arg('--delegate', dest='DelegateAccount', metavar='ACCOUNT',
                  help='''[Eucalyptus extension] run this command as another
                          account (only usable by cloud administrators)''')]
  

  Requestbuilder makes tool writers’ jobs easier by allowing one type of request to inherit its command line options from another type of request and then supply their own by simply listing more of them. This is a little different from the way Python usually works; Requestbuilder does some magic behind the scenes to make this possible. As a result, everything common to commands that access the EUARE service (Eucalyptus’s equivalent of Amazon’s IAM service) can go into one place to be shared with others.

  The final piece of information requestbuilder needs is a ServiceClass, which describes the web service that the tool connects to. A service class is another simple bit of code that looks like this:

  class Euare(BaseService):
      Description = 'Eucalyptus User, Authorization and Reporting Environment'
      APIVersion = '2010-05-08'
      EnvURL = 'EUARE_URL'
  

  The net gain from all this is a smaller, but much more flexible code base that should be able to scale better than anything we have had before. Requestbuilder’s use of Python’s argparse library also makes tools much more informative to users than ever before.

  How You Can Help

  We’re developing requestbuilder on GitHub as a project under the boto organization. We’re going to start rewriting euca2ools, one by one, improving requestbuilder to support new things as we go. It’s still early on, so if you have ideas to share or you’re interested in helping develop this code, now is your chance!

  We’re also moving development of euca2ools itself to GitHub. This will make it easier to work on euca2ools and requestbuilder in parallel. It will also make it easier to share code with the rest of the boto community.

  If you’re interested in getting involved, join us on the #boto or #eucalyptus IRC channels on Freenode. You can also send e-mail to Eucalyptus’s community list.

  ---