Mitogen is a Python library for writing distributed self-replicating programs.


Be aware this is prerelease code, and that comprehensive automated tests are presently absent. Router.enable_debug() is your friend. If have a use for this software, please drop an e-mail so expectations and bug fixes can be managed sensibly.


There is no requirement for installing packages, copying files around, writing shell snippets, upfront configuration, or providing any secondary link to a remote machine aside from an SSH connection. Due to its origins for use in managing potentially damaged infrastructure, the remote machine need not even have free disk space or a writeable filesystem.

It is not intended as a generic RPC framework; the goal is to provide a robust and efficient low-level API on which tools like Salt, Ansible, or Fabric can be built, and while the API is quite friendly and comparable to Fabric, ultimately it is not intended for direct use by consumer software.

The focus is to centralize and perfect the intricate dance required to run Python code safely and efficiently on a remote machine, while avoiding temporary files or large chunks of error-prone shell scripts, and supporting common privilege escalation techniques like sudo, potentially in combination with exotic connection methods such as WMI, telnet, or console-over-IPMI.

Automatic Bootstrap

Mitogen’s main feature is enabling your Python program to bootstrap and communicate with new copies of itself under its control running on remote machines, using only an existing installed Python interpreter and SSH client, something that by default can be found on almost all contemporary machines in the wild. To accomplish bootstrap, Mitogen uses a single 400 byte SSH command line and 8KB of its own source code sent to stdin of the remote SSH connection.

$ python ../
SSH command size: 593
Preamble size: 11655 (11.38KiB)
                  Original          Minimized           Compressed     
mitogen.parent  43934 42.9KiB  29681 29.0KiB 67.6%   7850  7.7KiB 17.9%
mitogen.ssh      9183  9.0KiB   5750  5.6KiB 62.6%   1931  1.9KiB 21.0%
mitogen.sudo     6646  6.5KiB   3406  3.3KiB 51.2%   1210  1.2KiB 18.2%   4453  4.3KiB   2182  2.1KiB 49.0%    804  0.8KiB 18.1%
mitogen.service 32424 31.7KiB  17736 17.3KiB 54.7%   4766  4.7KiB 14.7%
mitogen.fakessh 15544 15.2KiB   7964  7.8KiB 51.2%   2584  2.5KiB 16.6%
mitogen.master  23449 22.9KiB  14774 14.4KiB 63.0%   4366  4.3KiB 18.6%

Once bootstrapped, the remote process is configured with a customizable argv[0], readily visible to system administrators of the remote machine using the UNIX ps command:

20051 ?        Ss     0:00  \_ sshd: dmw [priv]
20053 ?        S      0:00  |   \_ sshd: dmw@notty
20054 ?        Ssl    0:00  |       \_ /usr/bin/python(mitogen:dmw@Eldil.home:22476)
20103 ?        S      0:00  |           \_ tar zxvf myapp.tar.gz

The example context was started by UID dmw on host Eldil.home, process ID 22476.

IO Multiplexer

The bootstrap includes a compact IO multiplexer (like Twisted or asyncio) that allows it to perform work in the background while executing your program’s code. For example, the remote context can be used to connect to a new user on the remote machine using sudo, or as an intermediary for extending the program’s domain of control outward to other machines, enabling your program to manipulate machines behind a firewall, or enable its data plane to cohere to your network topology.

bastion_host = router.ssh(

docker_host = router.ssh(

sudo_account = router.sudo(
    password='sudo password',

internal_box = router.docker(
), './')

The multiplexer also ensures the remote process is terminated if your Python program crashes, communication is lost, or the application code running in the context has hung.

Module Forwarder

Slaves are configured with a custom PEP-302 importer that forwards requests for unknown Python modules back to the host program. When your program asks a context to execute code from an unknown module, all requisite modules are transferred automatically and imported entirely in RAM without need for further configuration.

import myapp.mypkg.mymodule

# myapp/, myapp/mypkg/, and myapp/mypkg/
# are transferred automatically.

As the forwarder reuses the import mechanism, it should integrate cleanly with any tool such as py2exe that correctly implement the protocols in PEP-302, allowing truly single file applications to run across multiple machines without further effort.

Common sources of import latency and bandwidth consumption are mitigated:

  • Modules need only be uploaded once per directly connected context. Subsequent requests for modules from children of that context will be served by the child itself.
  • Imports by threads within a context triggering a load are deduplicated and joined with any identical requests triggered by other threads in the same context and children in the context’s subtree.
  • No roundtrip is required for negative responses due to Python 2’s import statement semantics: children have a list of submodules belonging to a package, and ignore requests for submodules that did not exist on the master.
  • Imports are extracted from each module, compared to those found in memory, and recursively preloaded into children requesting that module, minimizing round-trips to one per package nesting level. For example, django.db.models only requires 3 round-trips to transfer 456KiB, representing 1.7MiB of uncompressed source split across 148 modules.

SSH Client Emulation


Support is included for starting subprocesses with a modified environment, that cause their attempt to use SSH to be redirected back into the host program. In this way tools like rsync, git, sftp, and scp can efficiently reuse the host program’s existing connection to the remote machine, including any firewall/user account hopping in use, with no additional configuration.

Scenarios that were not previously possible with these tools are enabled, such as running sftp and rsync over a sudo session, to an account the user cannot otherwise directly log into, including in restrictive environments that for example enforce an interactive TTY and account password.

bastion = router.ssh(hostname='')
webserver = router.ssh(via=bastion, hostname='webserver')
webapp = router.sudo(via=webserver, username='webapp')
fileserver = router.ssh(via=bastion, hostname='fileserver')

# Transparently tunnelled over fileserver -> .. -> sudo.webapp link, webapp, [
    'rsync', 'appdata', 'appserver:appdata'

Message Routing


Slaves may communicate autonomously without direct interaction with the master, allowing a wide variety of complex data and control flows to be expressed using the links between the processes.

Logging Forwarder

The bootstrap configures the remote process’s Python logging package to forward all logs back to the local process, enabling management of program logs in one location.

18:15:29 D mitogen.ctx.k3: mitogen: Importer.find_module('mitogen.zlib')
18:15:29 D mitogen.ctx.k3: mitogen: _dispatch_calls((1002L, False, 'posix', None, 'system', ('ls -l /proc/self/fd',), {}))

Stdio Forwarder

To ease porting of crusty old infrastructure scripts to Python, the bootstrap redirects stdio for itself and any child processes back into the logging framework. This allows use of functions as basic as os.system(‘hostname; uptime’) without further need to capture or manage output.

18:17:28 D mitogen.ctx.k3: mitogen: _dispatch_calls((1002L, False, 'posix', None, 'system', ('hostname; uptime',), {}))
18:17:56 I mitogen.ctx.k3: stdout: k3
18:17:56 I mitogen.ctx.k3: stdout: 17:37:10 up 562 days,  2:25,  5 users,  load average: 1.24, 1.13, 1.14

Detached Subtrees


Contexts may detach from and outlive the running program, while maintaining communication with descendents in their subtree. This enables persistent background tasks that reuse Mitogen features.

def become_monitoring_master(children, econtext):

    while True:
        for child in children:
            if > 0.9:
                alert_operator('Child is too busy! ' + str(child))

dc1.call_async(become_monitoring_master, children)

Blocking Code Friendly

Within each process, a private thread runs the I/O multiplexer, leaving the main thread and any additional application threads free to perform useful work.

While Mitogen is internally asynchronous, it hides this asynchrony from consumer code. This is since writing asynchronous code is mostly a foreign concept to the target application of managing infrastructure. It should be possible to rewrite a shell script in Python without significant restructuring, or mind-bending feats of comprehension to understand control flow.


# Install our application.

tar zxvf app.tar.gz


def install_app():
    Install our application.
    os.system('tar zxvf app.tar.gz')

Or even:, 'tar zxvf app.tar.gz')

Exceptions raised by function calls are propagated back to the parent program, and timeouts can be configured to ensure failed calls do not block progress of the parent.

Scatter/Gather Calls

Functions may be invoked asynchronously, with results returned as they become available.

def usage(path):
    return sum((os.path.getsize(os.path.join(dirpath, name))
                for dirpath, dirnames, filenames in os.walk(path)
                for name in dirnames + filenames), 0)

total = 0
for msg in Select(c.call_async(usage, '/tmp') for c in contexts):
    usage = msg.unpickle()
    print 'Context %s /tmp usage: %d' % (recv.context, usage)
    total += usage

print 'Total /tmp usage across all contexts: %d' % (total,)

Single File Programs

Programs that are self-contained within a single Python script are supported. External contexts are configured such that any attempt to execute a function from the main Python script will correctly cause that script to be imported as usual into the slave process.

#!/usr/bin/env python
Install our application on a remote machine.

Usage: <hostname>

    <hostname>  Hostname to install to.
import os
import sys

import mitogen

def install_app():
    os.system('tar zxvf my_app.tar.gz')

def main(broker):
    if len(sys.argv) != 2:
        print __doc__

    context = mitogen.ssh.connect(broker, sys.argv[1])

if __name__ == '__main__' and mitogen.is_master:
    import mitogen.utils

Event-driven IO

Code running in a remote context can be connected to a Channel. Channels are used to send data asynchronously back to the parent, without further need for the parent to poll for changes. This is useful for monitoring systems managing a large fleet of machines, or to alert the parent of unexpected state changes.

def tail_log_file(channel, path='/var/log/messages'):
    Forward new lines in a log file to the parent.
    size = os.path.getsize(path)

        new_size = os.path.getsize(path)
        if new_size == size:
        elif new_size < size:
            size = 0

        fp = open(path, 'r')
        channel.send( - size))
        size = new_size


The package is written using syntax compatible all the way back to Python 2.4 released November 2004, making it suitable for managing a fleet of potentially ancient corporate hardware. For example Mitogen can be used out of the box against Red Hat Enterprise Linux 5, released in 2007.

Support for Python 3 is included using 2to3 triggered during, however such a Mitogen install does not support communicating with older 2.x systems. A future revision will support full cross 2/3 compatibility.

Zero Dependencies

Mitogen is implemented entirely using the standard library functionality and interfaces that were available in Python 2.4.