Non-destructive mask over a layer

[it_ross]

Hello,

I'm trying to see if there is a way to mask or otherwise obscure part of a layer, such as a character, without outright erasing any part of that layer. An example of this would be when I want to have part of a background be in front of a character, such as a bush. The only way I know how to do this is to make a new layer with a cutout of the part of the background that I want to have in the foreground, but there are some instances where this doesn't work well, such as if the background isn't at full opacity.

I made a GIF demonstrating the kind of function I'm imagining:

I made this GIF by drawing and coloring in a character on one single layer and then going frame-by-frame and erasing it with various shapes. If the character were moving, or if I wanted to use another animated character as the "mask", this method would not work, or it would be very destructive to the underlying animation (and if the character were colored in with a CTG layer, then this method would be unusable altogether).

I'm wondering if there is anything in TVPaint akin to masking in photoshop. I saw another thread here basically stating that there isn't, but then I do not know how one could possibly create the GIF above without literally erasing the character. I also know about stenciling, but that is still a restrictive and destructive process and doesn't work in all instances, especially if you've already drawn the character.

The particular reason why I'm trying to figure this out is because I'm trying to make two animated layers overlap with one another, such that part of layer A is IN FRONT OF layer B, while another part of layer A is BEHIND layer B. An example of this would be two characters on two separate layers hugging, for which I drew an extremely simplified example:

WAWG_10-3.png (63.21 KiB) Viewed 4053 times

The character on the left has one arm in the foreground and another arm in the background. If these characters are on separate layers, I would like to know how to keep the left character on the top layer, while having the character on the right obscure the background arm, if that makes sense. I figure that masking, as described above, would be the answer.

Thank you.

[D.T. Nethery]

Hello,

I'm trying to see if there is a way to mask or otherwise obscure part of a layer, such as a character, without outright erasing any part of that layer. An example of this would be when I want to have part of a background be in front of a character, such as a bush. The only way I know how to do this is to make a new layer with a cutout of the part of the background that I want to have in the foreground

That is the most straightforward way to do it. The Bush is an overlay element that is on top of the Character layer and the Background layer.

but there are some instances where this doesn't work well, such as if the background isn't at full opacity.

I'm not sure why the background not being at full opacity would have any effect on the bush overlay ? Three layers:

Layer 1 - Bush Overlay 100% opacity
Layer 2 - Character 100% opacity
Layer 3 - Background (at whatever opacity you prefer; in the screengrab I have it set to 40%)

(click on the images to see them full size)


It's really the same concept as when we worked with cel layers in the pre-computer days .

I'm wondering if there is anything in TVPaint akin to masking in photoshop. I saw another thread here basically stating that there isn't, but then I do not know how one could possibly create the GIF above without literally erasing the character. I also know about stenciling, but that is still a restrictive and destructive process and doesn't work in all instances, especially if you've already drawn the character.

There is not masking like in Photoshop. It has been requested , but not yet implemented. I think it would be great to have the same sort of masking that can be done in Photoshop and After Effects in TVPaint , but for what you are describing I don't think you need masking to get the results you want. Just arranging the layers in a logical way will do the trick.

The particular reason why I'm trying to figure this out is because I'm trying to make two animated layers overlap with one another, such that part of layer A is IN FRONT OF layer B, while another part of layer A is BEHIND layer B. An example of this would be two characters on two separate layers hugging, for which I drew an extremely simplified example:

The character on the left has one arm in the foreground and another arm in the background. If these characters are on separate layers, I would like to know how to keep the left character on the top layer, while having the character on the right obscure the background arm, if that makes sense. I figure that masking, as described above, would be the answer.

Thank you.

Draw Character A on the top layer over Character B. This top layer includes Character A's foreground arm that goes in front of Character B.
Draw Character A's far arm on a layer underneath Character B.
Background layer is underneath all.

Layer 1 - Character A (including foreground arm)
Layer 2 Character B
Layer 3 - Character A's far arm (behind Char. B)
Layer 4 - Background

.

[it_ross]

Thank you for the response.

I'm not sure why the background not being at full opacity would have any effect on the bush overlay ?

The bush may have been a poor example. A better example is if the character is standing on grass and I want to have some of the grass be in front of the character. If the background layer is not at full opacity, then whatever overlay I use for the grass won't match it. Even if I set the overlay's opacity to match the background, you'd be able to see the character's feet through the grass:

As for the characters hugging, the method you describe works for when there is a clear border separating the left character's body from the rear arm, but I'm not sure how to implement this solution if there is no such border in the design:

[Svengali]

Investigate the SPARE BUFFER...

In the documentation under Drawing Tools Advanced Functions, a section on Clipboard and Spare.

The discussion on the Spare Buffer might give you some alternative ideas on how to answer your questions...
Any image from a layer frame can be moved to the Spare Buffer where it exists but hidden. Once the Spare Buffer is loaded, you can then paint parts of it back into the same frame on a different layer or other frame using the MERGE drawing mode with the Airbrush tool .
It can be helpful if you turning on Stencil or Inverted Stencil for the frame you are painting over when Merging from the Spare Buffer.

A search for "Spare Buffer" on the forum will turn up more than a few discussions about ways of using the Spare Buffer...
sven

[slowtiger]

Having a half-transparent background can cause trouble. But you can easily create an opaque overlay like this:
- duplicate transparent layer
- create a layer under it
- fill it completely with white (or whatever your BG colour is)
- activate mask in duplicated BG layer and reverse it
- hit delete on the white layer
This is not perfect (you'll faintly see the borders of covered objects), but good enough for most cases. It works because your "transparent BG" really is only "transparent colour on white".

Most of the time I save myself this trouble and create completely opaque elements anyway. As for grass, a different style makes it easier: have a completely green shape on one layer and cover it with grass strokes in different greens on another, I use a rough pencil stroke for these. A duplicate of this second layer makes a perfect overlay.

[D.T. Nethery]

Thank you for the response.

I'm not sure why the background not being at full opacity would have any effect on the bush overlay ?

The bush may have been a poor example. A better example is if the character is standing on grass and I want to have some of the grass be in front of the character. If the background layer is not at full opacity, then whatever overlay I use for the grass won't match it.

Even if I set the overlay's opacity to match the background, you'd be able to see the character's feet through the grass:

If your intention is to have the grass and sky be a lighter color, why not just paint the background with the lighter colors at 100% opacity to begin with , rather than paint it darker and then lower the opacity to make the background colors appear lighter ? Then the grass overlay around the character's feet could be the exact same green as the grass on the background.

As for the characters hugging, the method you describe works for when there is a clear border separating the left character's body from the rear arm, but I'm not sure how to implement this solution if there is no such border in the design:
WAWG_10-3_5.png

The simplest approach in this case would be to draw both characters on the same layer.

In the situation you describe here with two characters touching, how would masking as in Photoshop solve the problem of the overlapping parts ? (to be clear: I'm not disagreeing with you or saying that masking a la Photoshop wouldn't be useful, but I genuinely don't understand what you're envisioning in terms of using masking in this case, so I'm trying to understand what result you would like to achieve.) It seems to me that it would be easier to draw both characters together on the same layer.


To use a more complex example, if you had a scene with two characters in close contact like this , with different parts of their bodies touching and overlapping , it would be unnecessarily complicated to split up the different ovelapping areas as separate layers.

.

[it_ross]

There are a lot of helpful techniques being described here. I am looking into the Spare Buffer option that was proposed.

Having a half-transparent background can cause trouble. But you can easily create an opaque overlay like this:
- duplicate transparent layer
- create a layer under it
- fill it completely with white (or whatever your BG colour is)
- activate mask in duplicated BG layer and reverse it
- hit delete on the white layer
This is not perfect (you'll faintly see the borders of covered objects), but good enough for most cases. It works because your "transparent BG" really is only "transparent colour on white".

I am slightly unsure of how this process works, but I might give it a try.

why not just paint the background with the lighter colors at 100% opacity to begin with

As for this, there might be situations where that is not an option (a project where I am not the one creating the background), or the background needs to be semi-transparent, such as if the background is a window and there is another animated character behind it.

how would masking as in Photoshop solve the problem of the overlapping parts

The way I was thinking about it, I would use a mask to obscure the portion of the left character that is supposed to be covered by the right character. If the left character has their arms around the right character, I would but the left character on the top layer and I would use a mask to cover up their rear arm as it is supposed to be blocked by the right character. At least, that's the masking solution I thought of.

I 100% agree that in the case of two characters embracing like in the screenshot, it would make way more sense from the get-go to animate them on the same layer. I just have some very specific use cases where I wanted to see if photoshop-esque masks could help me, so I've tried to come up with some simple examples. The replies here often have a solution that helps in the example but doesn't apply to my real use case lol , so I'll try to be as specific as possible:

Imagine you have an animated character in front of a background, where the background is also moving, and you want to cover part of the character with an invisible mask. Like in my original screenshot, with the person standing in some clouds with a hole moving through them, but in this case the clouds are all moving in different directions, so one cannot simply take a chunk of the background and move it to the foreground.

The actual thing that I am trying to animate is a close-up of a person planting a sapling in some dirt, lowering it into a hole. As the sapling is lowered, it gets obscured by the edge of the hole closest to the camera. The obvious solution would be to cut out a chunk of the dirt background and put it as the top layer, the problem is that the background dirt is actually made up of many layers that are in motion. Here is an illustration:

[slowtiger]

In such a case, with several layers/objects in movement, I first render this chunk as a video, then import, then cut a a part of this to cover the plant. OR I create a simple mask of the covering part and use it to cut off the parts of the plant that should be invisible, after the plant insertion is animated. Nothing complicated.

What you describe is a situation where storyboard/production designer should be much more aware of the technical process. The best problem is the one you can avoid completely.

[Peter Wassink]

....If the character were moving, or if I wanted to use another animated character as the "mask", this method would not work, or it would be very destructive to the underlying animation (and if the character were colored in with a CTG layer, then this method would be unusable altogether).

in the ctg layer you can indicate which layers the coloring should obey as border for the coloring process, which means, if you activate the right layers, the CTG is perfectly usable.

There are many solutions in tvpaint (as usual) to solve your animated grass mask problem.

Here is an easy one, that does not involve using the layer stencils:
you could pick up a selection of the backround that you need to mask, using the FreeHand cutbrush set to "display"
make sure you pick it up as an animbrush (select the range of the animation and click yes when the "cut out anim brush?"comes up
then on a new layer on top choose the "stamp in place" option in the tool menu. Voila