In the previous article I showed how to stylize an animation sequence using the painting “The Starry Night” by Vincent Van Gogh.

The problems is that between frame by frame it “blinks” does not maintain fluidity, that is because each frame is processed separately and the computer uses its “creativity” to fill in the image space that does not have a match, to solve this I use the EbSynth neural network this can maintain quite the fluidity between frames if we help it a little.

How do we help you? Giving you a guiding degrade, this is like a map of where to put each color so you don’t use random results when creating the frame.


ebsynth -style hand_paint.png -guide frame1.png frame2.png -weight 2 -guide map.png map.png -weight 1 -output result_02.png


Here the “map.png” is the same in both the first and the second frame, this can generate some problems of “washing” the reference drawing, but it can be fixed by modifying the weight (-weight) of the guides, adjust the parameters until the result is optimal.

Test on an animation sequence

Download the Example

wget -c https://download.tuxfamily.org/4232/wordpress/2017/05/4232.cf-e2.gif

Extract the frames

ffmpeg -i 4232.cf-e2.gif w.%04d.png

Create Map

convert \( xc:#000000 xc:#FF0000 +append \) \( xc:#00FF00 xc:#FFFF00 +append \) -append -size 500x281 xc:white +swap -fx 'v.p{i/(w-1),j/(h-1)}' map.png

Paint the first frame with G’MIC

gmic w.0001.png _fx_stylize starrynight +fx_stylize 1,6,0,0,0.5,2,3,0.5,0.1,3,3,0,0.7,1,0,1,0,5,5,7,1,30,1,2,1.85,0 output[2] result_w.0001.png

Create the sequence with EbSynth


ls w.*png | while read A; do
ebsynth -style result_w.0001.png -guide w.0001.png $A -weight 2 -guide map.png map.png -weight 1 -output eb_$A
done


Make video


ffmpeg -y -i eb_w.%04d.png -r 4 eb_result.mkv


Origin

Result

Now the fluidity is maintained, the frames are drawings in png and can be modified by hand, instead of the Van Gogh painting we can paint the first frame as we think of.