English / Français
drozBot, the portraitist robot
drozBot is a robot that draws with fast and smooth gestures similar to those of an artist. From a photo, it creates a portrait of the person by tracing a series of lines in China ink on a sheet of paper.
Guided by artificial intelligence and a movement model inspired by human handwriting, it can transform an image into an abstract drawing. The result is surprising: expressive, dynamic, and natural strokes, which give the impression that the robot knows how to draw rather than execute a series of commands.
drozBot uses technologies developed in the Robot Learning & Interaction group at Idiap. The goal is to quickly generate portraits with three main objectives:
- To recognize the person who was drawn
- To recognize the distinctive style defined by the artist
- To recognize the quality of the dynamic strokes based on the final portrait
The latter goal is achieved by combining approaches from several scientific disciplines, blending robotics, optimal control, and motion science. As with calligraphy, even if a person only looks at the final portrait, a successful sketch must reflect the fluidity and skill of the movement that enabled the portrait to be created.
This aspect of motion generation is at the heart of the research work of Idiap's Robot Learning & Interaction group. The group focuses on human-robot collaboration and movement representations to facilitate the transfer of manipulation tasks, for various applications ranging from industry to human assistance.
International Collaboration: Between Art, Science & Technology
In collaboration with various international researchers from the University of London, Goldsmiths College (Daniel Berio and Frederic Fol Leymarie) or the University of Konstanz (Michael Stroh and Oliver Deussen), within the framework of projects such as EACVA (Embodied Agents in Contemporary Visual Art), the Robot Learning & Interaction group at Idiap explores how to extend human-robot collaboration to fields related to art and creativity. The robot is seen here as a new tool for artists to explore new methods of creation.
In the same way that an artist decides which brushes, pens, or pencils to use, as well as additional guides such as using a ruler to draw straighter lines or a compass to draw regular arcs, the robot can be used as a new type of guide and assistance.
To achieve this, our research explores which interfaces can make it easy to teach a robot the assistance the artist wants. For example, in the form of a style that is shown to the robot (learning by demonstration), which the robot then has to repeat for new content or on new parts of the canvas.
In this research vision, our group is interested in the development of intuitive technologies so that a robot can learn by imitation, combined with interfaces to be able to easily reprogram these machines, without requiring knowledge of robotics or computer language ("no-code" approach).
Interdisciplinary Collaboration at Idiap
drozBot revolves around various research themes at Idiap on artificial intelligence, for example drawing inspiration from research results on human-robot social interaction (Emmanuel Senft), on biometrics (Sébastien Marcel), on perception (Jean-Marc Odobez), or on semantic and symbolic representations for neural network models (André Freitas, Paola Merlo).
Smooth Movements and Gestures
In artistic fields, deep learning approaches, with generative models, have democratized (for better or worse) the generation of images from textual descriptions. These AI models generate images in the form of pixels. A robot, on the other hand, generates paths and trajectories, with control at the level of its joints. Moreover, unlike digital art, the robot must face the real world, with geometric and physical constraints, for example, having to compensate for the effects of gravity when moving its arm holding an ink-soaked brush.
The scientific challenge is to define mathematical representations that mimic the capabilities of human movement, with fluid and rapid gestures for an aesthetic rendering of China ink drawings. For this, the movement to be generated is a crucial aspect, controlling both the orientation and trajectory of the brush, as well as the pressure applied on the paper.
AI and Robotics
To help the robot plan the series of strokes it should generate, other AI tools are used to automatically detect, from the image coming from a camera, what the pose of the person is in the image, and where the most important elements of the face are located, such as the eyes, mouth, and nose. Other elements of the image, such as the background, hair, or glasses, are detected in a similar way. This information helps the robot determine what concentration and precision of strokes are required in the portrait, focusing on the most important elements and producing a quicker sketch for less important elements of the drawing.
The robot's planned strokes must then be transformed into smooth and fast movements to mimic the artist's gestures, skillfully handling the brush and deftly using an inkwell to maintain the ideal amount of ink.
Scientific Publications:
Contact:
Sylvain Calinon, Senior Research Scientist at Idiap, heading the Robot Learning & Interaction group. Email: sylvain.calinon@idiap.ch
drozBot is the result of a collaboration between Guillaume Clivaz, Jérémy Maceiras, Tobias Löw, Daniel Berio, and Sylvain Calinon.
This project is supported by the Swiss National Science Foundation (SNSF) as part of an Agora project with Phänomena.