In the previous article I showed how to stylize an animation sequence using the painting “The Starry Night” by Vincent Van Gogh.
The problems is that between frame by frame it “blinks” does not maintain fluidity, that is because each frame is processed separately and the computer uses its “creativity” to fill in the image space that does not have a match, to solve this I use the EbSynth neural network this can maintain quite the fluidity between frames if we help it a little.
How do we help you? Giving you a guiding degrade, this is like a map of where to put each color so you don’t use random results when creating the frame.
ebsynth -style hand_paint.png -guide frame1.png frame2.png -weight 2 -guide map.png map.png -weight 1 -output result_02.png
Here the “map.png” is the same in both the first and the second frame, this can generate some problems of “washing” the reference drawing, but it can be fixed by modifying the weight (-weight) of the guides, adjust the parameters until the result is optimal.
Test on an animation sequence
Download the Example
wget -c https://download.tuxfamily.org/4232/wordpress/2017/05/4232.cf-e2.gif
Extract the frames
ffmpeg -i 4232.cf-e2.gif w.%04d.png
Create Map
convert \( xc:#000000 xc:#FF0000 +append \) \( xc:#00FF00 xc:#FFFF00 +append \) -append -size 500x281 xc:white +swap -fx 'v.p{i/(w-1),j/(h-1)}' map.png
Paint the first frame with G’MIC
gmic w.0001.png _fx_stylize starrynight +fx_stylize 1,6,0,0,0.5,2,3,0.5,0.1,3,3,0,0.7,1,0,1,0,5,5,7,1,30,1,2,1.85,0 output[2] result_w.0001.png
Create the sequence with EbSynth
ls w.*png | while read A; do
ebsynth -style result_w.0001.png -guide w.0001.png $A -weight 2 -guide map.png map.png -weight 1 -output eb_$A
done
Make video
ffmpeg -y -i eb_w.%04d.png -r 4 eb_result.mkv
Origin
Result
Now the fluidity is maintained, the frames are drawings in png and can be modified by hand, instead of the Van Gogh painting we can paint the first frame as we think of.