Interactive Manta Jobs with mlogin(1)
It is 6am on Tuesday the 25th of June, 2013 — at least, it is in US/Pacific — which means we at Joyent are finally lifting the covers off of our new product: Manta. Manta is a brand new system that spans the twin pillars of Object Storage and Compute to provide a revolutionary new way of operating on data in the cloud. You can read more about it in a write-up from Mark Cavage on the Joyent blog.
In this post I'm going to cover one of the pieces I recently added to Manta — namely: the Manta Interactive Session Engine. While Manta was primarily designed to run batch-style jobs across a large number of input objects, this subsystem allows you to run an interactive UNIX program in the environment of a Manta compute job and control it from your terminal.
The client utility that allows you to run interactive jobs is mlogin(1). To crib, briefly, from its manual page:
mlogin allows you to spawn an interactive job in Manta. Once running, your terminal will be attached to the remote process running in the job via a shell session tunneled through HTTPS, similar in concept to SSH. Interactive sessions are a great way to debug a new job script in-situ, or to experience and explore the compute zone environment hands on. It can also become part of a workflow using interactive terminal utilities on large Manta objects without the need to download or transfer the data -- e.g. the use of mdb (the Modular Debugger) on crash dumps and core files.
To get started with
mlogin, you'll need to sign up for a Joyent Public Cloud account, which gets you access to the Manta Storage service as well as our IaaS cloud offering. Once you sign in to the portal with your new account, you'll find personalised instructions for adding your SSH key and downloading and installing the Manta command-line tools, the source for which is available at joyent/node-manta on Github) and includes
Once you have the command-line tools installed and you've been able to use
mput to upload a file into Manta, you can take a swing at
mlogin. The simplest possible interactive job is what we call a Reduce task with no input objects. You can spawn one of these by just typing
mlogin! Let's do that now, and while we're there we can take a look at the environment inside a Manta compute job.
As you can see, the default
mlogin invocation attaches you to a
bash shell running in the Manta compute environment — a regular SmartOS zone. Some environment variables (starting with
MANTA_) are provided that you can use in the job programs you write.
mdb(1) in-situ on the dump without copying it to another system!
In this screenshot I have started a Manta job on an uploaded core file, and invoked
mdb(1) on it. In a Map task, the
MANTA_INPUT_FILE environment variable contains the path to where the system has mounted (read-only) your input object. I have then used
::findjsobjects, as described in Bryan Cantrill's blog post on finding node.js memory leaks.
One of the more gratifying parts of building
mlogin(1) was the realisation that apart from the WebSockets engine for forwarding the live shell traffic to and from the client, the rest of the subsystem was implemented in terms of existing Manta primitives. I was able to pass configuration into the interactive job through the use of a regular Manta object, the job itself is a regular Manta compute job, and the forwarding agent that runs in the job is delivered into the Manta compute zone as a regular Asset. You can see more details of the implementation in the source, and I have high hopes that our user community will start producing their own innovative tools on top of Manta that I haven't even begun to imagine!
We at Joyent Engineering encourage you to check out Manta today, and we look forward to hearing your feedback!