Comment on page
ControlNet
How to install and use ControlNet
- The HED model has been superseded by the Soft Edge model. Soft Edge is the better option, and HED is on the list for legacy only.
- Tested to be fully compatible with graphics cards that have 6GB VRAM and up.
- Not currently compatible with Inpaint/Outpaint
- Not compatible with Remove Background when used on the same Gen
- Inpaint, Tile, and Shuffle models aren't working in Artroom yet.
ControlNet is a method for copying the features of an image, much more accurately than a regular image to image process. First the ControlNet model makes a 'preprocessed image' of a starting image, and uses that to guide your regular model Each ControlNet model has its own style for preprocessing. It will take some practice and experimentation to find the right one.
Below are some examples of each model, what the preprocessor does, and several output images.
Just note that these are just some good examples of what it can do, and may not always be perfect.
These five work well with just about any image you use with them, with no special requirements for using.
Depth
Normal
Pose
Soft Edge
Canny
Attempts to predict the distance of objects from the viewer. Good for getting the general shape of an image, without copying specific details.
This model attempts to figure out what way a subject is facing. Like Depth, it creates a general shape without copying details.
Creates a wire-frame of the subject in an image. Copies the exact pose without any other details from the original image.
Now with better face and hands.
Traces an outline of the image. Similar to canny, but is less strict when copying the smaller details.
The most commonly used ControlNet model. Creates a fine line to trace an image. Good for copying the subject in an image very accurately.
These next five are a bit more specific on how they want to be used, and will be explained in each tab.
MLSD
Lineart
Lineart Anime
Scribble
Instruct Pix2Pix
Used to copy the general shape of the scenery. Does not pick up on subjects very well.
Used with lineart/monochrome images, adding color and details to them.
Doesn't work well with regular colored images.
Can sometimes work better with inverted colors if the outline is too strong.
Used with lineart/monochrome images, adding color and details to them.
Doesn't usually work well with regular colored images or non anime models.
Can sometimes work better with inverted colors if the outline is too strong.
Takes a crude drawing made in msPaint and roughly follows it. This model has more freedom to deviate from the base image than others, and can lead to some interesting results.
The listed prompts are the only prompts used on that image
This model s meant to take the base image and, using plain English, change specific details of it. Like, "make her hair blue" or "change the season to winter"
As simple as it sounds, this can be one of the more difficult models to get working right.
Inpaint may be a better option for more controlled changes. The Paint Tab
These last ones are the old version 1.0 ControlNet models. Leaving them here as a reference.
Canny
Pose
Depth
HED
Normal
Scribble
This was replaced by Soft Edge
Last modified 5mo ago