Using AI for colouring process?

[mikdog]

So, I enjoy animating but one of the processes I abhor is the colouring process. It's not creative at all, just a slog.
I thought it was just me until I saw Tonino's video title lol:
https://youtu.be/CNqdWW4GtrE

I wanted to know if there were a way to eg colour the first few frames, or colour some key frames, and let an AI do the rest based on the reference images?
Surely AI's at a smart-enough level now where it can identify 'oh this is the eye, oh this is the skin, this is the pants' etc. and colour matte fills accordingly?

Would be amazing to not have to do the laborious process every time of colouring, especially when it should be evident even to even a 3 year old what parts of the image are what.

Wondering if anyone has any ideas how we can eg get ChatGPT or some kind of AI model to work for us in TVPaint? If anything I think doing the boring bits is a good use of AI.

Thank you.

[slowtiger]

1. CTG layers got that covered already.

2. You don't need any steenking AI to do this. The technique is well established for about 30 years now, I've used it in Animo and Toons.

[Peter Wassink]

So, I enjoy animating but one of the processes I abhor is the colouring process. It's not creative at all, just a slog.
I thought it was just me until I saw Tonino's video title lol:
https://youtu.be/CNqdWW4GtrE
...
Thank you.

i agree.
And, like i saw Tonino mention in the video, i have experimented quite a bit with EB-synth, but found its not very suitable for regular 2D (its more useful with life action footage.)
so
i 'm curious too.

[Peter Wassink]

1. CTG layers got that covered already.

Am i missing something?
CTG won't color your animation on the basis of a first frame.

[mikdog]

1. CTG layers got that covered already.

Am i missing something?
CTG won't color your animation on the basis of a first frame.

I'm curious too! Maybe I underestimated the power of CTG layers? Wouldn't be the first time I've misunderstood the function of a feature.

[slowtiger]

To be sure I just re-read the manual, it's mentioned here: https://doc.tvpaint.com/docs/colorize-t ... s/examples

[Peter Wassink]

Thats a no.
This example from the manual is a runcycle that happens to allow just a few strategically placed squiggle lines to color all frames correctly.
Its actually a bit of a cheat and not a real life coloring example.
Its good to be aware that this is a way to use it, just be aware that most animation will still require more traditional approach even in CTG.

CTG is a dynamic color system, though its pretty good at deciding where to put the boundaries between colors based on the line drawing, it doesn't color intelligently.
You still need to make sure there is a squiggle correctly placed on each frame.
If you are lucky, or place them cleverly you will be able to re-use and minimise the number of squiggles, but its still you that will have to judge and do this, its not automated.

[mikdog]

Mmm… I also glanced through the manual and didn’t see mention of AI, I think CTG is more of an algorithm that doesn’t understand what it’s filling or what is on-screen. It’s still a great tool and I use it to save time, but it’s not quite Ai in the sense of being able to interpret the image in any meaningful way, and like Peter mentioned, still requires a large degree of user input.

However, if you show a child a character and ask it point out the eyes, pants etc it will and can then colour an image accurately, even as that character moves and turns etc. which CTG currently can’t do.

I feel that this is a huge opportunity to create a tool to fill this gap that if often rote slog work which is why colouring is sometimes farmed out to juniors, because almost anybody can do it by following the rules.

Ai is good at following rules and from what I can gather, is now pretty good at identifying parts of an image.

In my mind, that 1+1 is just waiting to equal 2 with an easy to use tool where the user has a reference pic of their character coloured, maybe in a 4 pose turnaround that the Ai can reference, and based on that, the Ai can colour all the uncoloured linework without the user needing to lay down strokes or identify parts of the image, maybe only later in a clean-up pass in case there are mistakes.

I’m guessing just a matter of time before someone creates it, I’ll be stoked to try it!

[Jet]

I’m guessing just a matter of time before someone creates it, I’ll be stoked to try it!

Already coming, see here: https://olm-rd.github.io/animestyle-col ... n-project/

You can try Cadmium, but some users have issues with accessing server: https://cadmium.app/

OpenToonz can also reference the last painted cel with onion skin paint, but it won't work if you use separation lines (lines that get autopainted), basically no shadow lines etc., and more useful for animation that is on 1's.

People are also working on full inbetween and paint methods, as you can read here: https://github.com/Doubiiu/ToonCrafter

This is created with 2 images and a "prompt", but it can be guided using sketches:


Like or hate AI, TVPaint etc. need to start looking at incorporating it, because others will, and as the original poster says, cel painting isn't a job anyone wants to do, or inbetweening for that matter. There may still be jobs for cel painting and inbetweening, but their tools will utilize AI, without a doubt, soon.

[mikdog]

Thanks for these links.
Yeh. Got a fairly big short film coming up and I'm not particularly looking forward to all the manual-labour rote/slog-work colouring that I really think is low-level and low-creative-level stuff an AI could take care of.
The linework animation I can get through pretty quickly and though the CTG method for colouring drawings can save some time, it often takes multiple passes to get all the bits as the computer seems to have little understanding of what shapes it's 'looking' at. Is a massive time-sink.
That 'Reference-based Sketch Colourization' of ToonCrafter looks pretty neat, thinking something like that could be what I'm after - just colour one frame and let the AI interpolate the rest, it seems to have an understanding of how the big shapes move.
https://github.com/Doubiiu/ToonCrafter? ... lorization

I'm a little sour on how I've seen AI being used as a replacement for creativity, but for basic stuff like this, I think it's more of a time-saver.

[mikdog]

I’m guessing just a matter of time before someone creates it, I’ll be stoked to try it!

Already coming, see here: https://olm-rd.github.io/animestyle-col ... n-project/

You can try Cadmium, but some users have issues with accessing server: https://cadmium.app/

OpenToonz can also reference the last painted cel with onion skin paint, but it won't work if you use separation lines (lines that get autopainted), basically no shadow lines etc., and more useful for animation that is on 1's.

People are also working on full inbetween and paint methods, as you can read here: https://github.com/Doubiiu/ToonCrafter

This is created with 2 images and a "prompt", but it can be guided using sketches:


Like or hate AI, TVPaint etc. need to start looking at incorporating it, because others will, and as the original poster says, cel painting isn't a job anyone wants to do, or inbetweening for that matter. There may still be jobs for cel painting and inbetweening, but their tools will utilize AI, without a doubt, soon.

Ooh, and I see Cadmium has a Mac version too, that didn't trigger a security warning on my comp.
Keen to test this out when I get to it, and will report back if/when so, thanks for the link.

[mikdog]

Hehe - yeh, Cadmium didn't really work well for me, bit of a garbled mess of colours.
Might give it another go with something simpler, defs didn't work as cleanly as the example GIFs they show on their site.
Finding a lot of the AI stuff *looks* neat but is currently unusable in a real sense for 2D animation.