Release Notes

Known Issues

Mitogen For Ansible

  • The Ansible 2.7 reboot module may require a pre_reboot_delay on systemd hosts, as insufficient time exists for the reboot command’s exit status to be reported before necessary processes are torn down.

  • On OS X when a SSH password is specified and the default connection type of smart is used, Ansible may select the Paramiko plug-in rather than Mitogen. If you specify a password on OS X, ensure connection: ssh appears in your playbook, ansible.cfg, or as -c ssh on the command-line.

  • The raw action executes as a regular Mitogen connection, which requires Python on the target, precluding its use for installing Python. This will be addressed in a future 0.2 release. For now, simply mix Mitogen and vanilla Ansible strategies in your playbook:

    - hosts: web-servers
      strategy: linear
      tasks:
      - name: Install Python if necessary.
        raw: test -e /usr/bin/python || apt install -y python-minimal
    
    - hosts: web-servers
      strategy: mitogen_linear
      roles:
      - nginx
      - initech_app
      - y2k_fix
    
  • Performance does not scale linearly with target count. This requires significant additional work, as major bottlenecks exist in the surrounding Ansible code. Performance-related bug reports for any scenario remain welcome with open arms.
  • Performance on Python 3 is significantly worse than on Python 2. While this has not yet been investigated, at least some of the regression appears to be part of the core library, and should therefore be straightforward to fix as part of 0.2.x.
  • Module Replacer style Ansible modules are not supported.
  • Actions are single-threaded for each (host, user account) combination, including actions that execute on the local machine. Playbooks may experience slowdown compared to vanilla Ansible if they employ long-running local_action or delegate_to tasks delegating many target hosts to a single machine and user account.
  • Connection Delegation remains in preview and has bugs around how it infers connections. Connection establishment will remain single-threaded for the 0.2 series, however connection inference bugs will be addressed in a future 0.2 release.
  • Connection Delegation does not support automatic tunnelling of SSH-dependent actions, such as the synchronize module. This will be addressed in the 0.3 series.

Core Library

  • Serialization is still based on pickle. While there is high confidence remote code execution is impossible in Mitogen’s configuration, an untrusted context may at least trigger disproportionately high memory usage injecting small messages (“billion laughs attack”). Replacement is an important future priority, but not critical for an initial release.
  • Child processes are not reliably reaped, leading to a pileup of zombie processes when a program makes many short-lived connections in a single invocation. This does not impact Mitogen for Ansible, however it limits the usefulness of the core library. A future 0.2 release will address it.
  • Some races remain around mitogen.core.Broker destruction, disconnection and corresponding file descriptor closure. These are only problematic in situations where child process reaping is also problematic.
  • The fakessh component does not shut down correctly and requires flow control added to the design. While minimal fixes are possible, due to the absence of flow control the original design is functionally incomplete.
  • The multi-threaded Service Framework remains in a state of design flux and should be considered obsolete, despite heavy use in Mitogen for Ansible. A future replacement may be integrated more tightly with, or entirely replace the RPC dispatcher on the main thread.
  • Documentation is in a state of disrepair. This will be improved over the 0.2 series.

v0.2.4 (2018-??-??)

Mitogen for Ansible

Enhancements

  • #76, #351, #352: disconnect propagation has improved, allowing Ansible to cancel waits for responses from abruptly disconnected targets. This ensures a task will gracefully fail rather than hang, for example on network failure or EC2 instance maintenance.
  • #369: Connection.reset() is implemented, allowing meta: reset_connection to shut down the remote interpreter as documented, and improving support for the reboot module.
  • 09aa27a6: the mitogen_host_pinned strategy wraps the host_pinned strategy introduced in Ansible 2.7.

Fixes

  • #323, #333: work around a Windows Subsystem for Linux bug that caused tracebacks to appear during shutdown.
  • #334: the SSH method tilde-expands private key paths using Ansible’s logic. Previously the path was passed unmodified to SSH, which expanded it using pwd.getpwnam(). This differs from os.path.expanduser(), which uses the HOME environment variable if it is set, causing behaviour to diverge when Ansible was invoked across user accounts via sudo.
  • #364: file transfers from controllers running Python 2.7.2 or earlier could be interrupted due to a forking bug in the tempfile module.
  • #370: the Ansible reboot module is supported.
  • #373: the LXC and LXD methods print a useful hint on failure, as no useful error is normally logged to the console by these tools.
  • #391: file transfer from 2.x controllers to 3.x targets was broken due to a regression caused by refactoring, and compounded by #426.
  • #400: work around a threading bug in the AWX display callback when running with high verbosity setting.
  • #409: the setns method was silently broken due to missing tests. Basic coverage was added to prevent a recurrence.
  • #409: the LXC and LXD methods support mitogen_lxc_path and mitogen_lxc_attach_path variables to control the location of third pary utilities.
  • #410: the sudo method supports the SELinux --type and --role options.
  • #420: if a Connection was constructed in the Ansible top-level process, for example while executing meta: reset_connection, resources could become undesirably shared in subsequent children.
  • #362, #435: the previous fix for slow Python 2.x subprocess creation on Red Hat caused newly spawned children to have a reduced open files limit. A more intrusive fix has been added to directly address the problem without modifying the subprocess environment.
  • #397, #454: the previous approach to handling modern Ansible temporary file cleanup was too aggressive, and could trigger early finalization of Cython-based extension modules, leading to segmentation faults.

Core Library

  • #76: routing records the destination context IDs ever received on each stream, and when disconnection occurs, propagates mitogen.core.DEL_ROUTE messages towards every stream that ever communicated with the disappearing peer, rather than simply towards parents. Conversations between nodes anywhere in the tree receive mitogen.core.DEL_ROUTE when either participant disconnects, allowing receivers to wake with mitogen.core.ChannelError, even when one participant is not a parent of the other.
  • #109, 57504ba6: newer Python 3 releases explicitly populate sys.meta_path with importer internals, causing Mitogen to install itself at the end of the importer chain rather than the front.
  • #387, #413: dead messages include an optional reason in their body. This is used to cause mitogen.core.ChannelError to report far more useful diagnostics at the point the error occurs that previously would have been buried in debug log output from an unrelated context.
  • #399, #437: ignore a DeprecationWarning to avoid failure of the su method on Python 3.7.
  • #405: if an oversized message is rejected, and it has a reply_to set, a dead message is returned to the sender. This ensures function calls exceeding the configured maximum size crash rather than hang.
  • #406: mitogen.core.Broker did not call mitogen.core.Poller.close() during shutdown, leaking the underlying poller FD in masters and parents.
  • #406: connections could leak FDs when a child process failed to start.
  • #288, #406, #417: connections could leave FD wrapper objects that had not been closed lying around to be closed during garbage collection, causing reused FD numbers to be closed at random moments.
  • #411: the SSH method typed “y” rather than the requisite “yes” when check_host_keys=”accept” was configured. This would lead to connection timeouts due to the hung response.
  • #416: around 1.4KiB of memory was leaked on every RPC, due to a list of strong references keeping alive any handler ever registered for disconnect notification.
  • #418: the mitogen.parent.iter_read() helper would leak poller FDs, because execution of its finally block was delayed on Python 3. Now callers explicitly close the generator when finished.
  • #422: the fork method could fail to start if sys.stdout was opened in block buffered mode, and buffered data was pending in the parent prior to fork.
  • #438: a descriptive error is logged when stream corruption is detected.
  • #439: descriptive errors are raised when attempting to invoke unsupported function types.
  • #453: the loggers used in children for standard IO redirection have propagation disabled, preventing accidental reconfiguration of the logging package in a child from setting up a feedback loop.
  • #456: a descriptive error is logged when mitogen.core.Broker.defer() is called after the broker has shut down, preventing new messages being enqueued that will never be sent, and subsequently producing a program hang.
  • 16ca111e: handle OpenSSH 7.5 permission denied prompts when ~/.ssh/config rewrites are present.
  • 9ec360c2: a new mitogen.core.Broker.defer_sync() utility function is provided.
  • f20e0bba: mitogen.service.FileService.register_prefix() permits granting unprivileged access to whole filesystem subtrees, rather than single files at a time.
  • 8f85ee03: mitogen.core.Router.myself() returns a mitogen.core.Context referring to the current process.
  • 824c7931: exceptions raised by the import hook were updated to include probable reasons for a failure.

Thanks!

Mitogen would not be possible without the support of users. A huge thanks for bug reports, testing, features and fixes in this release contributed by Andreas Krüger, Berend De Schouwer, Brian Candler, Duane Zamrok, Guy Knights, Jiří Vávra, Jonathan Rosser, Johan Beisser, Josh Smift, Mehdi, Michael DeHaan, Mohammed Naser, @syntonym, and @yodatak.

v0.2.3 (2018-10-23)

Mitogen for Ansible

Enhancements

  • #315, #392: Ansible 2.6 and 2.7 are supported.
  • #321, #336: temporary file handling was simplified, undoing earlier damage caused by compatibility fixes, improving 2.6 compatibility, and avoiding two network roundtrips for every related action (assemble, aws_s3, copy, patch, script, template, unarchive, uri). See Temporary Files for a complete description.
  • #376, #377: the kubectl connection type is now supported. Contributed by Yannig Perré.
  • 084c0ac0: avoid a roundtrip in copy and template due to an unfortunate default.
  • 7458dfae: avoid a roundtrip when transferring files smaller than 124KiB. Copy and template actions are now 2-RTT, reducing runtime for a 20-iteration template loop over a 250 ms link from 30 seconds to 10 seconds compared to v0.2.2, down from 120 seconds compared to vanilla.
  • #337: To avoid a scaling limitation, a PTY is no longer allocated for an SSH connection unless the configuration specifies a password.
  • d62e6e2a: many-target runs executed the dependency scanner redundantly due to missing synchronization, wasting significant runtime in the connection multiplexer. In one case work was reduced by 95%, which may manifest as faster runs.
  • 5189408e: threads are cooperatively scheduled, minimizing GIL contention, and reducing context switching by around 90%. This manifests as an overall improvement, but is easily noticeable on short many-target runs, where startup overhead dominates runtime.
  • The faulthandler module is automatically activated if it is installed, simplifying debugging of hangs. See Diagnosing Hangs for details.
  • The MITOGEN_DUMP_THREAD_STACKS environment variable’s value now indicates the number of seconds between stack dumps. See Diagnosing Hangs for details.

Fixes

  • #251, #340: Connection Delegation could establish connections to the wrong target when delegate_to: is present.
  • #291: when Mitogen had previously been installed using pip or setuptools, the globally installed version could conflict with a newer version bundled with an extension that had been installed using the documented steps. Now the bundled library always overrides over any system-installed copy.
  • #324: plays with a custom module_utils would fail due to fallout from the Python 3 port and related tests being disabled.
  • #331: the connection multiplexer subprocess always exits before the main Ansible process, ensuring logs generated by it do not overwrite the user’s prompt when -vvv is enabled.
  • #332: support a new sys.excepthook()-based module exit mechanism added in Ansible 2.6.
  • #338: compatibility: changes to /etc/environment and ~/.pam_environment made by a task are reflected in the runtime environment of subsequent tasks. See Process Environment Emulation for a complete description.
  • #343: the sudo --login option is supported.
  • #344: connections no longer fail when the controller’s login username contains slashes.
  • #345: the IdentitiesOnly yes option is no longer supplied to OpenSSH by default, better matching Ansible’s behaviour.
  • #355: tasks configured to run in an isolated forked subprocess were forked from the wrong parent context. This meant built-in modules overridden via a custom module_utils search path may not have had any effect.
  • #362: to work around a slow algorithm in the subprocess module, the maximum number of open files in processes running on the target is capped to 512, reducing the work required to start a subprocess by >2000x in default CentOS configurations.
  • #397: recent Mitogen master versions could fail to clean up temporary directories in a number of circumstances, and newer Ansibles moved to using atexit to effect temporary directory cleanup in some circumstances.
  • b9112a9c, 2c287801: OpenSSH 7.5 permission denied prompts are now recognized. Contributed by Alex Willmer.
  • A missing check caused an exception traceback to appear when using the ansible command-line tool with a missing or misspelled module name.
  • Ansible since >=2.7 began importing __main__ from ansible.module_utils.basic, causing an error during execution, due to the controller being configured to refuse network imports outside the ansible.* namespace. Update the target implementation to construct a stub __main__ module to satisfy the otherwise seemingly vestigial import.

Core Library

  • A new mitogen.parent.CallChain class abstracts safe pipelining of related function calls to a target context, cancelling the chain if an exception occurs.
  • #305: fix a long-standing minor race relating to the logging framework, where no route for Message.. would frequently appear during startup.
  • #313: mitogen.parent.Context.call() was documented as capable of accepting static methods. While possible on Python 2.x the result is ugly, and in every case it should be trivial to replace with a classmethod. The documentation was fixed.
  • #337: to avoid a scaling limitation, a PTY is no longer allocated for each OpenSSH client if it can be avoided. PTYs are only allocated if a password is supplied, or when host_key_checking=accept. This is since Linux has a default of 4096 PTYs (kernel.pty.max), while OS X has a default of 127 and an absolute maximum of 999 (kern.tty.ptmx_max).
  • #339: the LXD connection method was erroneously executing LXC Classic commands.
  • #345: the SSH connection method allows optionally disabling IdentitiesOnly yes.
  • #356: if the master Python process does not have sys.executable set, the default Python interpreter used for new children on the local machine defaults to "/usr/bin/python".
  • #366, #380: attempts by children to import __main__ where the main program module lacks an execution guard are refused, and an error is logged. This prevents a common and highly confusing error when prototyping new scripts.
  • #371: the LXC connection method uses a more compatible method to establish an non-interactive session. Contributed by Brian Candler.
  • af2ded66: add mitogen.fork.on_fork() to allow non-Mitogen managed process forks to clean up Mitogen resources in the child.
  • d6784242: the setns method always resets HOME, SHELL, LOGNAME and USER environment variables to an account in the target container, defaulting to root.
  • 830966bf: the UNIX listener no longer crashes if the peer process disappears in the middle of connection setup.

Thanks!

Mitogen would not be possible without the support of users. A huge thanks for bug reports, testing, features and fixes in this release contributed by Alex Russu, Alex Willmer, atoom, Berend De Schouwer, Brian Candler, Dan Quackenbush, dsgnr, Jesse London, John McGrath, Jonathan Rosser, Josh Smift, Luca Nunzi, Orion Poplawski, Peter V. Saveliev, Pierre-Henry Muller, Pierre-Louis Bonicoli, Prateek Jain, RedheatWei, Rick Box, nikitakazantsev12, Tawana Musewe, Timo Beckers, and Yannig Perré.

v0.2.2 (2018-07-26)

Mitogen for Ansible

  • #291: ansible_*_interpreter variables are parsed using a restrictive shell-like syntax, supporting a common idiom where ansible_python_interpreter is set to /usr/bin/env python.
  • #299: fix the network_cli connection type when the Mitogen strategy is active. Mitogen cannot help network device connections, however it should still be possible to use device connections while Mitogen is active.
  • #301: variables like $HOME in the remote_tmp setting are evaluated correctly.
  • #303: the Doas become method is supported. Contributed by Mike Walker.
  • #309: fix a regression to process environment cleanup, caused by the change in v0.2.1 to run local tasks with the correct environment.
  • #317: respect the verbosity setting when writing to Ansible’s log_path, if it is enabled. Child log filtering was also incorrect, causing the master to needlessly wake many times. This nets a 3.5% runtime improvement running against the local machine.
  • The mitogen_ssh_debug_level variable is supported, permitting SSH debug output to be included in Mitogen’s -vvv output when both are specified.

Core Library

  • #291: the python_path parameter may specify an argument vector prefix rather than a string program path.
  • #300: the broker could crash on OS X during shutdown due to scheduled kqueue filter changes for descriptors that were closed before the IO loop resumes. As a temporary workaround, kqueue’s bulk change feature is not used.
  • #303: the Doas become method is now supported. Contributed by Mike Walker.
  • #307: SSH login banner output containing the word ‘password’ is no longer confused for a password prompt.
  • #319: SSH connections would fail immediately on Windows Subsystem for Linux, due to use of TCSAFLUSH with termios.tcsetattr(). The flag is omitted if WSL is detected.
  • #320: The OS X poller could spuriously wake up due to ignoring an error bit set on events returned by the kernel, manifesting as a failure to read from an unrelated descriptor.
  • #342: The network_cli connection type would fail due to a missing internal SSH plugin method.
  • Standard IO forwarding accidentally configured the replacement stdout and stderr write descriptors as non-blocking, causing subprocesses that generate more output than kernel buffer space existed to throw errors. The write ends are now configured as blocking.
  • When mitogen.core.enable_profiling() is active, mitogen.service threads are profiled just like other threads.
  • The ssh_debug_level parameter is supported, permitting SSH debug output to be redirected to a Mitogen logger when specified.
  • Debug logs containing command lines are printed with the minimal quoting and escaping required.

Thanks!

Mitogen would not be possible without the support of users. A huge thanks for the bug reports and pull requests in this release contributed by Alex Russu, Andy Freeland, Ayaz Ahmed Khan, Colin McCarthy, Dan Quackenbush, Duane Zamrok, Gonzalo Servat, Guy Knights, Josh Smift, Mark Janssen, Mike Walker, Orion Poplawski, falbanese, Tawana Musewe, and Zach Swanson.

v0.2.1 (2018-07-10)

Mitogen for Ansible

  • #297: compatibility: local actions set their working directory to that of their defining playbook, and inherit a process environment as if they were executed as a subprocess of the forked task worker.

v0.2.0 (2018-07-09)

Mitogen 0.2.x is the inaugural feature-frozen branch eligible for fixes only, except for problem areas listed as in-scope below. While stable from a development perspective, it should still be considered “beta” at least for the initial releases.

In Scope

  • Python 3.x performance improvements
  • Subprocess reaping improvements
  • Major documentation improvements
  • PyPI/packaging improvements
  • Test suite improvements
  • Replacement CI system to handle every supported OS
  • Minor deviations from vanilla Ansible behaviour
  • Ansible raw action support

The goal is a tick/tock model where even-numbered series are a maturation of the previous unstable series, and unstable series are released on PyPI with --pre enabled. The API and user visible behaviour should remain unchanged within a stable series.

Mitogen for Ansible

  • Support for Ansible 2.3 - 2.7.x and any mixture of Python 2.6, 2.7 or 3.6 on controller and target nodes.
  • Drop-in support for many Ansible connection types.
  • Preview of Connection Delegation feature.
  • Built-in file transfer compatible with connection delegation.

Core Library

  • Synchronous connection establishment via OpenSSH, sudo, su, Docker, LXC and FreeBSD Jails, local subprocesses and os.fork(). Parallel connection setup is possible using multiple threads. Connections may be used from one or many threads after establishment.
  • UNIX masters and children, with Linux, MacOS, FreeBSD, NetBSD, OpenBSD and Windows Subsystem for Linux explicitly supported.
  • Automatic tests covering Python 2.6, 2.7 and 3.6 on Linux only.