Can I teach a robot to replicate a line art
Raghav BV, Subham Kumar, Vinay P Namboodiri

(PDF) (arXiv) (code coming soon…)

How to cite

@inproceedings{RKN19,
        title = {Can {I} teach a robot to replicate a
                  line art},
        booktitle = {2020 IEEE Winter Conference on Applications of
                  Computer Vision (WACV)},
        author = {Venkataramaiyer, Raghav B. and Kumar, Subham and
                  Namboodiri, Vinay P},
        year = {2020},
        pages = {1922-1930},
        doi = {10.1109/WACV45572.2020.9093434}
}

Abstract

Line art is arguably one of the fundamental and versatile modes of expression. We propose a pipeline for a robot to look at a grayscale line art and redraw it. The key novel elements of our pipeline are: a) we propose a novel task of mimicking line drawings, b) to solve the pipeline we modify the Quick-draw dataset to obtain supervised training for converting a line drawing into a series of strokes c) we propose a multi-stage segmentation and graph interpretation pipeline for solving the problem. The resultant method has also been deployed on a CNC plotter as well as a robotic arm. We have trained several variations of the proposed methods and evaluate these on a dataset obtained from Quick-draw. Through the best methods we observe an accuracy of around 98% for this task, which is a significant improvement over the baseline architecture we adapted from. This therefore allows for deployment of the method on robots for replicating line art in a reliable manner. We also show that while the rule-based vectorization methods do suffice for simple drawings, it fails for more complicated sketches, unlike our method which generalizes well to more complicated distributions.

The Network Architecture

We reinterpret the U-Net architecture with modular blocks for downsampling, upsampling and bottom, as a 4-tuple \((k_1\, k_2\, r\, d)\) for easy parameterization and architecture search.

Figure 2: Reinterpretation of the U-Net architecture.

Dataset

We take the Quick-draw dataset and synthesise the input as sketches, and output as their segmentation mask.

Figure 3: Top. Samples from the Quick-draw dataset; Bottom. Rendered true labels.

Qualitative Results

Here is a list of qualitative results. First we compare the quality of our segmentation model against a popular vectorization method.

Figure 4: Comparison of our segmentation (left column) against Favreau’s vectorization (right column). Top to Bottom. A simple line sketch from Favreau’s examples; A simple line sketch from Quick-draw dataset; A complicated hatch pattern with zoomed in details.

Then we show the results of two of our graph interpretation models. The former model uses a set of fixed global parameters, and the latter updates parameters for every edge using feedback from comparison with the input image.

Figure 5: The top row shows the input. The subsequent rows show reconstruction using a graph interpreted from segmentation mask, with values of mask width and threshold \((\beta,\, \tau)\) set as \((2,\, 0.3)\), \((3,\, 0.22)\), \((5,\, 0.2)\), \((7,\, 0.15)\) respectively downwards.

Figure 6: The results of feedback loop in graph interpretation. Rows top to bottom. the input image; naive graph interpretation; interpretation after \(N = 5\) updates; and after \(N = 10\) updates.