Interactive Manta Jobs with mlogin(1)

It is 6am on Tuesday the 25th of June, 2013 — at least, it is in US/Pacific — which means we at Joyent are finally lifting the covers off of our new product: Manta. Manta is a brand new system that spans the twin pillars of Object Storage and Compute to provide a revolutionary new way of operating on data in the cloud. You can read more about it in a write-up from Mark Cavage on the Joyent blog.

In this post I’m going to cover one of the pieces I recently added to Manta — namely: the Manta Interactive Session Engine. While Manta was primarily designed to run batch-style jobs across a large number of input objects, this subsystem allows you to run an interactive UNIX program in the environment of a Manta compute job and control it from your terminal.

The client utility that allows you to run interactive jobs is mlogin(1). To crib, briefly, from its manual page:

mlogin  allows you  to spawn an  interactive job  in
Manta.  Once running, your terminal will be attached
to the remote process running in the job via a shell
session tunneled  through HTTPS, similar  in concept
to SSH.

Interactive  sessions are a great way to debug a new
job  script in-situ,  or to  experience and  explore
the  compute zone environment hands on.  It can also
become  part   of  a  workflow   using   interactive
terminal utilities  on large Manta  objects  without
the need  to download  or transfer  the data -- e.g.
the use of mdb (the Modular Debugger) on crash dumps
and core files.

To get started with mlogin, you’ll need to sign up for a Joyent Public Cloud account, which gets you access to the Manta Storage service as well as our IaaS cloud offering. Once you sign in to the portal with your new account, you’ll find personalised instructions for adding your SSH key and downloading and installing the Manta command-line tools, the source for which is available at joyent/node-manta on Github) and includes mlogin.

Once you have the command-line tools installed and you’ve been able to use mput to upload a file into Manta, you can take a swing at mlogin. The simplest possible interactive job is what we call a Reduce task with no input objects. You can spawn one of these by just typing mlogin! Let’s do that now, and while we’re there we can take a look at the environment inside a Manta compute job.

As you can see, the default mlogin invocation attaches you to a bash shell running in the Manta compute environment — a regular SmartOS zone. Some environment variables (starting with MANTA_) are provided that you can use in the job programs you write.

Let’s try something a bit more complicated: an interactive job that uses an input object, also known as a Map task. Here at Joyent we write a lot of software in Javascript using the node.js platform. As part of debugging this software we often take core dumps of the running program, which we are then able to analyse post-mortem with mdb(1): the illumos Modular Debugger. We have recently taken to uploading those core dumps into Manta, where we can now (via the magic of mlogin) run mdb(1) in-situ on the dump without copying it to another system!

In this screenshot I have started a Manta job on an uploaded core file, and invoked mdb(1) on it. In a Map task, the MANTA_INPUT_FILE environment variable contains the path to where the system has mounted (read-only) your input object. I have then used ::findjsobjects, as described in Bryan Cantrill’s blog post on finding node.js memory leaks.

One of the more gratifying parts of building mlogin(1) was the realisation that apart from the WebSockets engine for forwarding the live shell traffic to and from the client, the rest of the subsystem was implemented in terms of existing Manta primitives. I was able to pass configuration into the interactive job through the use of a regular Manta object, the job itself is a regular Manta compute job, and the forwarding agent that runs in the job is delivered into the Manta compute zone as a regular Asset. You can see more details of the implementation in the source, and I have high hopes that our user community will start producing their own innovative tools on top of Manta that I haven’t even begun to imagine!

We at Joyent Engineering encourage you to check out Manta today, and we look forward to hearing your feedback!