ControlNet

How to install and use ControlNet

• The HED model has been superseded by the Soft Edge model. Soft Edge is the better option, and HED is on the list for legacy only.

• Tested to be fully compatible with graphics cards that have 6GB VRAM and up.

• Not currently compatible with Inpaint/Outpaint

• Not compatible with Remove Background when used on the same Gen

• Inpaint, Tile, and Shuffle models aren't working in Artroom yet.

What is ControlNet?

ControlNet is a method for copying the features of an image, much more accurately than a regular image to image process. First the ControlNet model makes a 'preprocessed image' of a starting image, and uses that to guide your regular model Each ControlNet model has its own style for preprocessing. It will take some practice and experimentation to find the right one.

Below are some examples of each model, what the preprocessor does, and several output images. Just note that these are just some good examples of what it can do, and may not always be perfect.

These five work well with just about any image you use with them, with no special requirements for using.

Depth

Attempts to predict the distance of objects from the viewer. Good for getting the general shape of an image, without copying specific details.

Normal

This model attempts to figure out what way a subject is facing. Like Depth, it creates a general shape without copying details.

Pose

Creates a wire-frame of the subject in an image. Copies the exact pose without any other details from the original image. Now with better face and hands.

Soft Edge

Traces an outline of the image. Similar to canny, but is less strict when copying the smaller details.

Canny

The most commonly used ControlNet model. Creates a fine line to trace an image. Good for copying the subject in an image very accurately.

These next five are a bit more specific on how they want to be used, and will be explained in each tab.

MLSD

Used to copy the general shape of the scenery. Does not pick up on subjects very well.

Lineart

Used with lineart/monochrome images, adding color and details to them. Doesn't work well with regular colored images. Can sometimes work better with inverted colors if the outline is too strong.

Lineart Anime

Used with lineart/monochrome images, adding color and details to them. Doesn't usually work well with regular colored images or non anime models. Can sometimes work better with inverted colors if the outline is too strong.

Scribble

Takes a crude drawing made in msPaint and roughly follows it. This model has more freedom to deviate from the base image than others, and can lead to some interesting results.

Instruct Pix2Pix

The listed prompts are the only prompts used on that image

This model s meant to take the base image and, using plain English, change specific details of it. Like, "make her hair blue" or "change the season to winter" As simple as it sounds, this can be one of the more difficult models to get working right. Inpaint may be a better option for more controlled changes. The Paint Tab

These last ones are the old version 1.0 ControlNet models. Leaving them here as a reference.

Canny

Pose

Depth

HED

This was replaced by Soft Edge

Normal

Scribble

Using ControlNet

Pose kits and Saved Preprocessed Images