Here is how to create refined masks for furry things:
- Create a new document with a white background (>File >New >Create)
- Enable the >Curvature Pen tool, choose black as the foreground color and set the mode to >Shape on the options bar
- Create a vertical line on the canvas by clicking in the top area and in the bottom area; click on the top point again to close the path
- Create an anchor point at the center of the line and drag out to the left, creating a circle segment
- Set another anchor point at the center of the line and drag left slightly less to create a shape that resembles a strand of hair
- Go to >edit, choose >define brush preset, give it a name and press >ok
As a starting point (adjust as needed):
- Adjust the brush settings by clicking on the brush settings icon on the options bar
- Increase >Spacing to e.g. 65%
- Enable >Shape Dynamics and increase >Size Jitter to 12%
- Add >Angle Jitter of 1% while setting >Control to >Direction
- Enable >Scattering and increase >Scatter to 20% and >Count to 6
- Save by clicking on the >Fly-out Menu choose >New Brush Preset and enter a name
To apply on a mask:
- Select your mask and contract it to create room for painting with the fur brush by selecting >Filter >Other > Minimum
- Select a radius (e.g. 8 Pixels) that fits the resolution of your image
- Select white as the foreground color, zoom-in to the edge and start painting (Note: use [ or ] to adjust the brush and "left arrow" or "right arrow" to adjust the angle of the brush
- To match "out of focus fur" go to >Filter >Blur >Gaussian Blur and select a radius that best matches the image blur
- To correct for color casts, add a new layer, clip it to the layer below with >Ctr Alt G, select a soft round brush hold the >Alt key and select the correct color of the fur to paint on the edge
First seen in a Video Tutorial (chapter "Custom Brushes for Masking" at timestamp 1:50) shared by Jesús Ramirez on the Photoshop Training Channel