Dr Hailin Jin and Concept Canvas

Dr Hailin Jin Hogwarts has their fictitious wizards … but did you know Adobe’s real-life Wizarding World is filled with Graphic Gurus and Digital Scientists? One such weaver of Adobe magic is Dr. Hailin Jin … Principal Scientist on the near supernatural Concept Canvas search engine.

I recently interviewed the incredible Dr. Hailin Jin, who gave me a behind the scenes look into this amazing new search engine concept (which he previewed at Adobe MAX 2016).

Dee Marie: Thank you Dr. Jin for taking time out of your busy schedule to introduce our readers to Concept Canvas. To start off, what is the Concept Canvas search engine?

Dr. Hailin Jin: Concept Canvas is a novel way of finding images based on spatial relationships of concepts. If a designer is looking for an image of a person with a brown dog for a project, searching “person and brown dog” will bring up countless photos that may not orient the human and animal exactly as they would like in the image. The combination of text and image-based search capabilities in Concept Canvas saves designers time in finding the perfect image that fits with the layout of their content.

Preview of the Concept Canvas interface within Adobe Photoshop

Preview of the Concept Canvas interface within Adobe Photoshop searching Adobe Stock images

Dee Marie: How specifically does Concept Canvas differ from other search engines?

There are two dominant image search paradigms: text-based search and image-based search. Text-based image search is the traditional experience of searching images on a search engine with a text input. The challenge with text-based image search systems is refining your queries according to the desired appearance, causing people to go through several pages of search results to find what they want.

Image-based search systems, based on the latest Deep Learning technologies, can find images that are visually similar to the query image. But you need to have a query image to start with which is not always available.

Concept Canvas is a new search paradigm that falls between text-based and image-based search. It combines the power of using text in text-based systems and the ability to define visual appearance in image-based systems.

Dee Marie: Please, give our readers a behind-the-scenes look into the creation of Concept Canvas.

Dr. Hailin Jin: I first came up with this idea in late 2015 during a meeting with Jon Brandt, Senior Principal Scientist, Adobe Research, when we were discussing technology he developed that searches images with combinations of image snippets. It was a great technology with one caveat: the user would have to provide image snippets, which can be difficult in some cases.

I thought, “Why not use text?” Everybody can express what they want but not necessarily find images that fit the idea. I presented my idea to Adobe Research in early 2016 and received a lot of great feedback. I decided to pursue this idea as an internship project and hired Long Mai from the Portland State University as my intern. The project started in June 2016, and the team consisted of Long Mai, Jon Brandt; Zhe Lin, Principal Scientist, Adobe Research; Chen Fang, Research Scientist, Adobe Research and myself. By the end of August 2016 we developed a prototype search system that works for five million Adobe Stock images.

We demonstrated the technology as part of the sneaks demo at Adobe MAX 2016 in November to more than 10,000 attendees. We also submitted a technical paper describing the details of the technology to CVPR 2017, the premier conference in computer vision. While I came up with the idea for the technology, my intern Long Mai had the opportunity to execute most of the work.

Dee Marie: At last year’s Adobe MAX you previewed the Concept Canvas in conjunction with Adobe Stock. Will the Concept Canvas search engine function in other apps, such as Adobe Bridge … where artists can utilize the magic of Concept Canvas within their personal computer image files?

Dr. Hailin Jin: The technology can be used anywhere people need to find images, including Adobe Stock and beyond. For instance, you may want to use it in Photoshop CC or Lightroom CC to search your personal albums.

It works great on mobile devices too. On a mobile device, it is natural to interact with a 2D canvas (the touch screen) with gestures where you can easily move concepts around.

Dee Marie: Going back to Adobe MAX 2016, what was your experience like, and how did people react to Concept Canvas?

Dr. Hailin Jin: Doing a live demo in front of 10,000 people can be intimidating. Even though I did the DeepFont demo in 2015, I was still a bit nervous. However, everything went according to plan, and I received a lot of good feedback from the attendees. At the MAX Bash following the presentation there were people coming to me saying congratulations. It was a fantastic experience!

Dee Marie: Congratulations on a successful presentation. Now something personal … describe your journey from receiving a Ph.D. in Electrical Engineering, to becoming an Adobe Principal Scientist.

Dr. Hailin Jin: During my postdoctoral period in 2003, I discovered Adobe was looking for computer vision researchers. I joined Adobe in 2004 as a Research Scientist. I became a Senior Research Scientist in 2007 and a Principal Scientist in 2013.

Dee Marie: What are the duties of an Adobe Principal Scientist?

Dr. Hailin Jin: Principal Scientist is a very senior technical position at Adobe, and the duties vary according to the organization the person is in. At Adobe Research, I am responsible for research and advanced engineering in the development of current or future Adobe products and technologies. I sometimes lead cross functional working groups or initiatives or provide consulting and advice to groups at Adobe. I am also responsible for mentoring and managing junior technical contributors.

Dee Marie: What was your postdoctoral research in Computer Science geared towards?

Dr. Hailin Jin: Both my doctoral and postdoctoral research are in computer vision, an interdisciplinary field that focuses on how computers can be made to understand the world from images and video. I worked on problems such as reconstructing the 3D shape of an object from multiple images, inferring 3D motion of a moving camera from images, and estimating lighting and illumination from images. I implemented the algorithms (which involves coding) to make sure the algorithms are working.

Dee Marie: Over the years, what has been your favorite Adobe project, that you were a part of bringing to fruition?

Dr. Hailin Jin: That is a difficult question; I worked on multiple projects and technologies. Every one of those is intellectually fulfilling and has provided me the opportunity to work with many talented people. If I had to pick one, I would say it is DeepFont technology, which took almost three years from the start of the project to its first appearance in Photoshop.

Dee Marie: Now, for the most important question … everyone wants to know … what is the approximat release date for Concept Canvas?

Dr. Hailin Jin: Concept Canvas was demonstrated at Adobe MAX as a “Sneak” – a preview of technology in development in Adobe Labs. Each year at Adobe MAX, we showcase some of the most innovative projects from the Adobe Labs—which may or may not make their way into our lineup of tools.

Oh, that is a Sneak Tease! With such an innovative search engine concept, we, at YURdigital, eagerly look forward to, and urge Adobe, to make Concept Canvas a part of its essential tools. Thank you again Dr. Jin, for being so generous with your time.


Dee Marie, awarding winning author and freelance journalistDee Marie is an awarding winning author and freelance journalist. She has previously served as Managing Editor and Editor-in-Chief of an international printed CGI magazine. She invites you to visit her on Facebook, Twitter, and her Sons of Avalon website.