Tech —

Microsoft’s “Ink to Code” turns drawings into user interfaces

Early prototype is being developed as a "Garage" project.

Microsoft Ink to Code.

Sketching out rough ideas—traditionally on the back of a napkin in the US, backs of envelopes being preferred in the UK—is a common and important part of the design process, with the familiar pen or pencil and paper being favored for rough mockups and outlines. Ink to Code, a new Microsoft Garage project, hopes to turn those rough sketches into usable, working code.

Ink to Code is currently itself a rough prototype of an app. The basic premise is simple: designers can sketch out the bare bones of application interfaces with a stylus, and Ink to Code will turn those sketches into real code, specifically the XML markup used for Universal Windows Platform apps and Xamarin apps for Android. It uses the Windows 10 Ink APIs to recognize the objects that have been drawn, converting handwriting into text and boxes into screens, buttons, text boxes, and image placeholders.

Currently, the app is only an early prototype—a basic proof of concept rather than a fully fledged development tool. It recognizes only a few interface elements, along with rules and guidelines for aligning things, and it produces only basic code with no functional parts. The company has various ideas of how to develop it further but wants to hear from real designers and developers where to focus its development efforts. Some obvious directions are supporting a greater range of interface elements and producing more functional code; one can easily imagine how it could, for example, identify login screens and automatically plumb in authentication workflows.

Code to Ink.
Enlarge / Code to Ink.
Microsoft

The app was built as Garage project. The Garage is Microsoft's program to give its developers the ability to create new, innovative applications that tread new ground for the company. These projects aren't fully supported products with large development teams, but rather public experiments to test out ideas and solicit feedback from users. Garage apps don't have to be fully developed, complete applications and instead can be more of a minimal viable product to prove the basic concepts. Ink to Code was built by a team of interns at Microsoft's New England Research and Development ("NERD") site in Cambridge, Massachussets.

A recent success story from the Garage is Microsoft's Android homescreen/app launcher, the Microsoft Launcher. This started life as a Garage product called Arrow Launcher. With a growing user base and feature list, it has transitioned into a "real" Microsoft application with a full development team and a greater strategic significance: it's being used as a core element of Microsoft's cross-device features, such as "continue on PC."

Listing image by Danny Hope

Channel Ars Technica