Live 64: Understanding the Virtual Sketching framework

In yesterday's live stream—Live 64—we continued learning about how to vectorize sketches with machine learning (using the Virtual Sketching framework by Simo-Serra et al).

There was little action as we spent time reading the paper and trying to understand how the model works. In short, the framework consists of four distinct steps—aligned cropping, stroke generation, differentiable rendering, and differentiable pasting—that I summarized from the Line Drawing Generation Framework section of the publication.¹

• Aligned cropping crops patches from the input image, according to a window that moves in time t, and resamples to images of fixed size.
• Stroke generation takes the cropped patches as inputs to predict the parameters of the next stroke. This module follows an encoder-decoder architecture. The encoder is a convolutional neural network (CNN) that models image-level information. The decoder is a recurrent neural network (RNN) that takes in image features and outputs stroke parameters. When the paper mentions stroke parameters, it's referring to Bézier curves.
• Differentiable rendering is a neural renderer that approximates stroke images based on the Bézier curve stroke parameters.
• Differentiable pasting pastes the rendered strokes in the full-resolution canvas.

I forked the source code at nonoesp/virtual_sketching and plan to make small modifications to improve certain utilities and understand the underlying logic.

My goal is to (a) obtain coordinates of the vector output, render it myself, and create input/output workflows, (b) understand the model's architecture and how it learns during training, and (c) understand differential rendering and pasting.

It would definitely be more productive for me to study some of these topics offline and share my findings live. But I haven't put the time to do that over the last weeks and might have to continue working on this only during live streams.

If this is something that interests you, please let me know on Twitter or, even better, on the Discord community.

Thanks for watching.

See you next week!