News

New in Open Source: Python-to-Go, Atom in Orbit, DeepMind Lab, More

• By David Ramel
• January 5, 2017

As everybody settles back into the grind after the holidays, here's a list of notable new open source projects that might have been missed amid all the reveling.

Grumpy: Go Running Python! (or a Python-to-Go Source Code Transcompiler and Runtime)
This project from Google, just announced yesterday, is hot -- it's the top trending project on GitHub today, having amassed more than 3,400 stars in just one day.

Its description reads:

Grumpy is a Python to Go source code transcompiler and runtime that is intended to be a near drop in replacement for CPython 2.7. The key difference is that it compiles Python source code to Go source code which is then compiled to native code, rather than to bytecode. This means that Grumpy has no VM. The compiled Go source code is a series of calls to the Grumpy runtime, a Go library serving a similar purpose to the Python C API (although the C API is not directly supported).

In a blog post yesterday, Project Lead Dylan Trotter, a YouTube engineer, said Grumpy was developed because the YouTube front-end team found it difficult to get concurrent workloads performant on CPython 2.7. The team investigated other Python runtimes but found nothing that didn't solve the problem without introducing other issues.

"So we asked ourselves a crazy question: 'What if we were to implement an alternative runtime optimized for real-time serving?'" Trotter said. "Once we started going down the rabbit hole, Go seemed like an obvious choice of platform since its operational characteristics align well with our use case (e.g. lightweight threads). We wanted first-class language interoperability and Go's powerful runtime type reflection system made this straightforward. Python in Go felt very natural, and so Grumpy was born.

"Grumpy is an experimental Python runtime for Go. It translates Python code into Go programs, and those transpiled programs run seamlessly within the Go runtime. We needed to support a large existing Python codebase, so it was important to have a high degree of compatibility with CPython (quirks and all). The goal is for Grumpy to be a drop-in replacement runtime for any pure-Python project."

The project is also hot on Hacker News (HN), where it has accumulated 1,338 points and generated 419 comments in just one day. Much of the discussion concerned the support of Python 2.7, even though the project began some time after Python 3.x was released (Python 3.6 was released Dec. 8). "Basically, we needed to support a large existing Python 2.7 codebase," Trotter explained in the comments section of HN, pointing to this discussion.

"The goal of this project is to produce a version of Atom that runs in Chrome from Atom's source that's as faithful to the desktop version as possible," the GitHub description reads.

Facebook, an active open source contributor, provided more details on Atom in Orbit in a blog post last week that detailed the company's "Favorite hacks of 2016."

"A few engineers set out to produce a version of Atom that runs in the browser as a step toward supporting remote development," the post reads. "While Web apps in general offer several advantages for engineers -- for example, automatic backups, remote access and server-side processing -- Atom on the desktop still enjoyed several features that weren't available in the browser, such as synchronous access to the filesystem, access to local resources, natively implemented dependencies, access to native APIs, and unrestricted access to the Internet.

"During the hackathon, the engineers found workarounds to these obstacles and contributed a set of scripts that repackages Atom's source so that it runs as a Web app. This hack is still ongoing -- you can follow progress, try it out or contribute to the code on GitHub."

With some "clever engineering," the Facebook team created workarounds to address various features found in the desktop version of Atom that weren't directly transferable to running verbatim Atom code in a browser, such as synchronous access to the filesystem via the fs module; all resources being available locally and assumed to be cheap to access; natively implemented dependencies; unrestricted access to the Internet; and access to native APIs.

Facebook set up a test page for developers to play around with the ongoing hack.

containerd (Docker Core Container Runtime)
Container kingpin Docker Inc. announced last month it was spinning out conatinerd, a core component of its Docker Engine, and donating it as a community project.

"While Docker Engine is a complete container platform for end users and includes the Docker API, Docker commands and services, containerd is a component that provides the industry with an open, stable and extensible base for building non-Docker products and container solutions," Docker said. "Leading cloud providers Alibaba Cloud, Amazon Web Services (AWS), Google, IBM and Microsoft have committed to providing maintainers and contributors to the project."

It's now on GitHub under an Apache 2.0 license.

The project description reads:

containerd is an industry-standard container runtime with an emphasis on simplicity, robustness and portability. It is available as a daemon for Linux and Windows, which can manage the complete container lifecycle of its host system: image transfer and storage, container execution and supervision, low-level storage and network attachments [and so on].

containerd is designed to be embedded into a larger system, rather than being used directly by developers or end-users.

Its listed features include:

• OCI Image Spec support
• OCI Runtime Spec support
• Image push and pull support
• Container runtime and lifecycle support
• Network primitives for creation, modification and deletion of interfaces
• Management of network namespaces containers to join existing namespaces
• Multi-tenant support with CAS storage for global images

"This is the result of months of close collaboration and input from thought leaders in the Docker community," said Docker exec Solomon Hykes in a statement. "containerd will unlock a whole new phase of innovation and growth across the entire ecosystem, which in turn will benefit every Docker developer and customer.

"Docker's focus has always been on solving users' problems first and then spinning out the plumbing projects that address those challenges along the way. We are excited by the support that the containerd project is getting from the leaders in the industry and we know their backing of resources will fuel the growth of this collaborative project."

That support includes contributions from 50 developers so far, says GitHub, which also lists 11 releases, six branches, 733 commits, 1,142 starts and 182 forks.

DeepMind Lab
DeepMind Technologies Ltd., an artificial intelligence (AI) specialist, says its mission is "to push the boundaries of AI, developing systems that can learn to solve any complex problem without needing to be taught how." To that end, it open sourced its flagship platform, DeepMind Lab.

"DeepMind Lab is a fully 3D game-like platform tailored for agent-based AI research," the company said in a blog post last month. "It is observed from a first-person viewpoint, through the eyes of the simulated agent. Scenes are rendered with rich science fiction-style visuals. The available actions allow agents to look around and move in 3D. The agent's 'body' is a floating orb. It levitates and moves by activating thrusters opposite its desired direction of movement, and it has a camera that moves around the main sphere as a ball-in-socket joint tracking the rotational look actions.

"Example tasks include collecting fruit, navigating in mazes, traversing dangerous passages while avoiding falling off cliffs, bouncing through space using launch pads to move between platforms, playing laser tag, and quickly learning and remembering random procedurally generated environments."

The DeepMind Lab GitHub project description says it's "A customizable 3D platform for agent-based AI research" and notes it was based on id Software's Quake III Arena via ioquake3 and other open source software.

Only two developers have contributed to the project, which has garnered 2,971 stars and generated one release and 531 forks. It runs on Linux and requires several dependencies, including various external software libraries, Python 2.7 (or possibly other versions), Lua 5.1, OpenGL and others.

An academic paper provides more details on the technology.