Spelunking Nova – flags and services

I’ve been doing a lot of spelunking into the nova codebase, digging around and trying to learn some of the under pinnings. Some of these pieces were a bit confusing to me, so I’m stashing them up here for Google to find and share with others in the future.

Before I dive into the gritty details, it’s worth getting a high level overview so that some of this (hopefully) makes sense. OpenStack’s service architecture is made up of services that all talk with each other to get things done. nova-network, nova-scheduler, etc. There’s a lot of underpinning in the nova codebase to make those services relatively easy to write and work together – I was mostly curious about how they passed messages back and forth. As I dove in, the two pieces that stood out as needing to be understood first were the unified service framework in nova and configuration using flags (which it heavily depends upon).

Configuration – using the flags

The configuration for nova services – global or specific to a service – are all done with configuration files that can be over-ridden on the command line, taking advantage of python gflags to make it all work nice. I didn’t know much about the flags system, so I dug around in the python-gflags project. They have the documentation for how to use gFlags in the code itself: http://python-gflags.googlecode.com/svn/trunk/gflags.py.

To summarize it up:

Python modules in the codebase can define and use flags, and there is a general nova flags file that holds the cross-service (common) configuration settings. Nova defaults to looking for it’s configuration in a nova.conf file in the local directory. Failing that it looks for the nova.conf file in /etc/nova/nova.conf. Where it looks for the configuration file can be overridden (typically on the command line) by (--flagfile) and a location to a config file. The code that makes this happen nova.utils.default_flagfile().

To use the configuration from within code, you typically instantiate the global flags, add any flag definitions (with default values) that you care to add, and then use ’em! Here’s a code snippet example:

from nova import flags
# import the nova wrapper around python gflags
#  .. there's some interesting wrapping for taking in arguments
#     and passing along extras values to your code
#  .. and it's where the global flag definitions reside
FLAGS = flags.FLAGS
# get the global instance
#  .. this attempts to read the /etc/nova/nova.conf for flags
# 
# You can define an additional flag here if you needed to...
flags.DEFINE_string('my_flag', 'default_value', 
        'human readable description of your flag')
# there's also flags.DEFINE_bool, flags.DEFINE_integer and more...
# 
# And then you can use the flags
#  .. the flags you defined show up as attributes 
#     on that FLAGS object
print FLAGS.my_ip

If you were happening to write a script that took in flags and worked with them for a command-line script, you might do something like:

from nova import flags
form nova import utils
utils.default_flagfile()
flags.FLAGS(sys.argv)
GLOBAL_FLAGS = flags.FLAGS
# ... and on to the rest of your code

There is some good end-user documentation on how to find the flags. The gist is – if you want to know what flags are there, the easiest way is to hand in the flag “–help” or “–shorthelp” from the command line. That is how the gFlags library is set up to tell you about the flags.

Update:

After a little digging down a side passage, I noticed that service.py had some debugging code in it that iterated through all the set flags. You iterate directly on FLAGS (treating it as an iterable thing) and use FLAGS.get() to retrieve the set values.

    logging.debug(_('Full set of FLAGS:'))
    for flag in FLAGS:
        flag_get = FLAGS.get(flag, None)
        logging.debug('%(flag)s : %(flag_get)s' % locals())

Services

There are two types of services in Nova: system services and web services. The code to use and launch them is basically the same, and Nova has this all bundled into a general service architecture and code base. The reason that configuration is so important is that the nova service framework has a convention of knowing how to run a service based on flags from the framework.

Here’s a bit of example code of a service to illustrate what I’m talking about.

nova-exampleservice:

import eventlet
eventlet.monkey_patch()

import sys
from nova import flags
from nova import service
from nova import utils

if __name__ == '__main__':
    utils.default_flagfile()
    flags.FLAGS(sys.argv)
    service.serve()
    service.wait()

The convention starts off by using the name of the script invoked – in this case “nova-exampleservice”. The scripts in bin/ (like nova-network) use this mechanism. This convention can be overridden, of course, but it does make things pretty straightforward once you know the convention. The key to this convention is that the code in nova.service looks in the configuration for a class to instantiate (expected to be a subclass of nova.manager.Manager) named after the service that was just invoked. (this convention is in code under the nova.service.create() method)

For our example of nova-exampleservice, the service is going to look in the configuration for exampleservice_manager, expecting the value to be a class that it can load that will be a subclass of nova.manager.Manager and will be responsible for running the service.

This code is invoked from service.serve() from our example above. Again, it looks for the flag “exampleservice_manager” and try to load that class to do the work.

An updated example that sets a default manager that will attempt to load the class mymodule.exampleservice.ExampleServiceManager by default:

nova-exampleservice:

import eventlet
eventlet.monkey_patch()

import sys
from nova import flags
from nova import service
from nova import utils

if __name__ == '__main__':
    utils.default_flagfile()
    flags.FLAGS(sys.argv)
    flags.DEFINE_string('exampleservice_manager',
            'mymodule.exampleservice.ExampleServiceManager',
            'Default manager for the nova-exampleservice')
    service.serve()
    service.wait()

The manager class has two classes that you override to get your stuff done:

  • init_host
  • periodic_tasks

There are also some conventions around adding methods to your manager and invoking them using the service framework’s RPC mechanism, which I’ll dig into with another post.

Ref: Nova Developer Documentation
Ref: OpenStack Compute (Nova) Administration Manual
Ref: Openstack Wiki: Unified Service Architecture

5 thoughts on “Spelunking Nova – flags and services

  1. I am a java person and I have downloaded the source code of Open Stack and have installed it . I have played with virtualization before and so I have a fair idea. I also understand RabbitMQ etc.
    The question I have for you is how to start understanding the code base . Where / how do I start . Is there an IDE to start with ? If not is there a place to start poking around in the code. Where do you recommend I start with ?

    Like

    • Hey Hari,

      First, almost all of the code in the OpenStack project (and everything in the core) is in python – so getting some base familiarity with python is the initial step. There isnt an IDE or specific editor set up to dig around – many of the folks I know working on the project use vim or emacs. I use textmate on my mac to read the code, and almost all of the development is done on Ubuntu based linux distributions – so there are bash scripts to do “all in one” installs and such to get it working.

      I recommend getting it all running in an “all in one” setup and then reading and tracing code to understand whats on the inside. Well, that is what Im doing anyway. You can run all of the nova stack, or all of the swift stack (compute and storage respectively) within a VM for the purposes of seeing what it does and light functional testing.

      The libraries that the code uses are the next challenge – Im going to write more on the service framework, eventlet, and the “carrot” library which enables the use of AMQP/RabbitMQ for communications in another post. There is also the IRC channel (#openstack and #openstack-dev on FreeNode), LaunchPad answers, OpenStack forums, and the mailing list as avenues to ask questions. You will hae much better luck in those venues with specific questions over broader questions.

      Hope this helps get you started into OpenStack!

      Like

  2. Another snippit that folks may find useful when making command-line tools that interact with nova’s database is:

    #!/usr/bin/env python
    import gettext
    from nova import context
    from nova import db
    from nova import flags
    from nova import log as logging
    from nova import utils
    import sys
    
    gettext.install('nova', unicode=1)
    logging.setup()
    
    utils.default_flagfile()   
    non_flag_args = flags.FLAGS(sys.argv)
    
    # your code here...
    

    Like

  3. Spoke too soon… It might be useful to add a python search path. This chunk was also useful.

    # If ../nova/__init__.py exists, add ../ to Python search path, so that
    # it will override what happens to be installed in /usr/(local/)lib/python...
    import os
    import sys
    possible_topdir = os.path.normpath(os.path.join(os.path.abspath(sys.argv[0]),
                                       os.pardir,
                                       os.pardir))
    if os.path.exists(os.path.join(possible_topdir, 'nova', '__init__.py')):
        sys.path.insert(0, possible_topdir)
    

    This is what my template looks like now:

    #!/usr/bin/env python
    import gettext
    
    # If ../nova/__init__.py exists, add ../ to Python search path, so that
    # it will override what happens to be installed in /usr/(local/)lib/python...
    import os
    import sys
    possible_topdir = os.path.normpath(os.path.join(os.path.abspath(sys.argv[0]),
                                       os.pardir,
                                       os.pardir))
    if os.path.exists(os.path.join(possible_topdir, 'nova', '__init__.py')):
        sys.path.insert(0, possible_topdir)
    
    from nova import context
    from nova import db
    from nova import flags
    from nova import log as logging
    from nova import utils
    
    gettext.install('nova', unicode=1)
    logging.setup()
    
    utils.default_flagfile()
    non_flag_args = flags.FLAGS(sys.argv)
    
    # code here...
    

    Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s