Service Framework

Warning

This section is incomplete.

Mitogen includes a simple framework for implementing services exposed to other contexts, with some built-in subclasses to capture common designs. This is a work in progress, and new functionality will be added as common usage patterns emerge.

Overview

Service

  • User-supplied class with explicitly exposed methods.
  • May be auto-imported/constructed in a child from a parent simply by calling it
  • Identified in calls by its canonical name (e.g. mypkg.mymod.MyClass) by default, but may use any naming scheme the configured activator understands.
  • Children receive refusals if the class is not already activated by a aprent
  • Has an associated Select instance which may be dynamically loaded with receivers over time, on_message_received() invoked if any receiver becomes ready.

Invoker

  • Abstracts mechanism for calling a service method and verifying permissions.
  • Built-in ‘service.Invoker’: concurrent execution of all methods on the thread pool.
  • Built-in ‘service.SerializedInvoker’: serialization of all calls on a single thread borrowed from the pool while any request is pending.
  • Built-in ‘service.DeduplicatingInvoker’: requests are aggregated by distinct (method, kwargs) key, only one such method ever executes, return value is cached and broadcast to all request waiters. Waiters do not block additional pool threads.

Activator

  • Abstracts mechanism for activating a service and verifying activation permission.
  • Built-in activator looks for service by fully.qualified.ClassName using Python import mechanism, and only permits parents to trigger activation.

Pool

  • Manages a fixed-size thread pool, mapping of service name to Invoker, and an aggregate Select over every activate service’s Selects.
  • Constructed automatically in children in response to the first CALL_SERVICE message sent to them by a parent.
  • Must be constructed manually in parent context.
  • Has close() and add() methods.

Example

import mitogen
import mitogen.service


class FileService(mitogen.service.Service):
    """
    Simple file server, for demonstration purposes only! Use of this in
    real code would be a security vulnerability as it would permit children
    to read arbitrary files from the master's disk.
    """
    handle = 500
    required_args = {
        'path': str
    }

    def dispatch(self, args, msg):
        with open(args['path'], 'r') as fp:
            return fp.read()


def download_file(context, path):
    s = mitogen.service.call(context, FileService.handle, {
        'path': path
    })

    with open(path, 'w') as fp:
        fp.write(s)


@mitogen.core.takes_econtext
def download_some_files(paths, econtext):
    for path in paths:
        download_file(econtext.master, path)


@mitogen.main()
def main(router):
    pool = mitogen.service.Pool(router, size=1, services=[
        FileService(router),
    ])

    remote = router.ssh(hostname='k3')
    remote.call(download_some_files, [
        '/etc/passwd',
        '/etc/hosts',
    ])
    pool.stop()

Reference

class mitogen.service.Policy

Base security policy.

class mitogen.service.AllowParents
class mitogen.service.AllowAny
mitogen.service.arg_spec(spec)

Annotate a method as requiring arguments with a specific type. This only validates required arguments. For optional arguments, write a manual check within the function.

@mitogen.service.arg_spec({
    'path': str
})
def fetch_path(self, path, optional=None):
    ...
Parameters:spec (dict) – Mapping from argument name to expected type.
mitogen.service.expose(policy)

Annotate a method to permit access to contexts matching an authorization policy. The annotation may be specified multiple times. Methods lacking any authorization policy are not accessible.

@mitogen.service.expose(policy=mitogen.service.AllowParents())
def unsafe_operation(self):
    ...
Parameters:policy (mitogen.service.Policy) – The policy to require.
mitogen.service.Service(router)
class mitogen.service.Invoker(service)
class mitogen.service.SerializedInvoker(**kwargs)
class mitogen.service.DeduplicatingInvoker(service)

A service that deduplicates and caches expensive responses. Requests are deduplicated according to a customizable key, and the single expensive response is broadcast to all requestors.

A side effect of this class is that processing of the single response is always serialized according to the result of key_from_request().

Only one pool thread is blocked during generation of the response, regardless of the number of requestors.

class mitogen.service.Service(router)
NO_REPLY = <object object>

Sentinel object to suppress reply generation, since returning None will trigger a response message containing the pickled None.

on_message(recv, msg)

Called when a message arrives on any of select’s registered receivers.

on_shutdown()

Called by Pool.shutdown() once the last worker thread has exitted.

class mitogen.service.Pool(router, services, size=1)

Manage a pool of at least one thread that will be used to process messages for a collection of services.

Internally this is implemented by subscribing every Service’s mitogen.core.Receiver using a single mitogen.select.Select, then arranging for every thread to consume messages delivered to that select.

In this way the threads are fairly shared by all available services, and no resources are dedicated to a single idle service.

There is no penalty for exposing large numbers of services; the list of exposed services could even be generated dynamically in response to your program’s configuration or its input data.

Parameters:
  • router (mitogen.core.Router) – Router to listen for CALL_SERVICE messages on.
  • services (list) – Initial list of services to register.
activator_class

alias of Activator