Spelunking Nova – flags and services

I’ve been doing a lot of spelunking into the nova codebase, digging around and trying to learn some of the under pinnings. Some of these pieces were a bit confusing to me, so I’m stashing them up here for Google to find and share with others in the future.

Before I dive into the gritty details, it’s worth getting a high level overview so that some of this (hopefully) makes sense. OpenStack’s service architecture is made up of services that all talk with each other to get things done. nova-network, nova-scheduler, etc. There’s a lot of underpinning in the nova codebase to make those services relatively easy to write and work together – I was mostly curious about how they passed messages back and forth. As I dove in, the two pieces that stood out as needing to be understood first were the unified service framework in nova and configuration using flags (which it heavily depends upon).

Configuration – using the flags

The configuration for nova services – global or specific to a service – are all done with configuration files that can be over-ridden on the command line, taking advantage of python gflags to make it all work nice. I didn’t know much about the flags system, so I dug around in the python-gflags project. They have the documentation for how to use gFlags in the code itself: http://python-gflags.googlecode.com/svn/trunk/gflags.py.

To summarize it up:

Python modules in the codebase can define and use flags, and there is a general nova flags file that holds the cross-service (common) configuration settings. Nova defaults to looking for it’s configuration in a nova.conf file in the local directory. Failing that it looks for the nova.conf file in /etc/nova/nova.conf. Where it looks for the configuration file can be overridden (typically on the command line) by (--flagfile) and a location to a config file. The code that makes this happen nova.utils.default_flagfile().

To use the configuration from within code, you typically instantiate the global flags, add any flag definitions (with default values) that you care to add, and then use ’em! Here’s a code snippet example:

from nova import flags
# import the nova wrapper around python gflags
#  .. there's some interesting wrapping for taking in arguments
#     and passing along extras values to your code
#  .. and it's where the global flag definitions reside
FLAGS = flags.FLAGS
# get the global instance
#  .. this attempts to read the /etc/nova/nova.conf for flags
# 
# You can define an additional flag here if you needed to...
flags.DEFINE_string('my_flag', 'default_value', 
        'human readable description of your flag')
# there's also flags.DEFINE_bool, flags.DEFINE_integer and more...
# 
# And then you can use the flags
#  .. the flags you defined show up as attributes 
#     on that FLAGS object
print FLAGS.my_ip

If you were happening to write a script that took in flags and worked with them for a command-line script, you might do something like:

from nova import flags
form nova import utils
utils.default_flagfile()
flags.FLAGS(sys.argv)
GLOBAL_FLAGS = flags.FLAGS
# ... and on to the rest of your code

There is some good end-user documentation on how to find the flags. The gist is – if you want to know what flags are there, the easiest way is to hand in the flag “–help” or “–shorthelp” from the command line. That is how the gFlags library is set up to tell you about the flags.

Update:

After a little digging down a side passage, I noticed that service.py had some debugging code in it that iterated through all the set flags. You iterate directly on FLAGS (treating it as an iterable thing) and use FLAGS.get() to retrieve the set values.

    logging.debug(_('Full set of FLAGS:'))
    for flag in FLAGS:
        flag_get = FLAGS.get(flag, None)
        logging.debug('%(flag)s : %(flag_get)s' % locals())

Services

There are two types of services in Nova: system services and web services. The code to use and launch them is basically the same, and Nova has this all bundled into a general service architecture and code base. The reason that configuration is so important is that the nova service framework has a convention of knowing how to run a service based on flags from the framework.

Here’s a bit of example code of a service to illustrate what I’m talking about.

nova-exampleservice:

import eventlet
eventlet.monkey_patch()

import sys
from nova import flags
from nova import service
from nova import utils

if __name__ == '__main__':
    utils.default_flagfile()
    flags.FLAGS(sys.argv)
    service.serve()
    service.wait()

The convention starts off by using the name of the script invoked – in this case “nova-exampleservice”. The scripts in bin/ (like nova-network) use this mechanism. This convention can be overridden, of course, but it does make things pretty straightforward once you know the convention. The key to this convention is that the code in nova.service looks in the configuration for a class to instantiate (expected to be a subclass of nova.manager.Manager) named after the service that was just invoked. (this convention is in code under the nova.service.create() method)

For our example of nova-exampleservice, the service is going to look in the configuration for exampleservice_manager, expecting the value to be a class that it can load that will be a subclass of nova.manager.Manager and will be responsible for running the service.

This code is invoked from service.serve() from our example above. Again, it looks for the flag “exampleservice_manager” and try to load that class to do the work.

An updated example that sets a default manager that will attempt to load the class mymodule.exampleservice.ExampleServiceManager by default:

nova-exampleservice:

import eventlet
eventlet.monkey_patch()

import sys
from nova import flags
from nova import service
from nova import utils

if __name__ == '__main__':
    utils.default_flagfile()
    flags.FLAGS(sys.argv)
    flags.DEFINE_string('exampleservice_manager',
            'mymodule.exampleservice.ExampleServiceManager',
            'Default manager for the nova-exampleservice')
    service.serve()
    service.wait()

The manager class has two classes that you override to get your stuff done:

• init_host
• periodic_tasks

There are also some conventions around adding methods to your manager and invoking them using the service framework’s RPC mechanism, which I’ll dig into with another post.

Ref: Nova Developer Documentation
Ref: OpenStack Compute (Nova) Administration Manual
Ref: Openstack Wiki: Unified Service Architecture