Just use the GIMP Animation Package (GAP); it is particular well suited for processing video frames.

mplayer test.mkv -vo png Run gimp
Open the first frame "000001.png"
Perform "Video->Frames Modify" and select the Apply Filter on Layer(s) function and click on OK. Choose the "plug-in-sobel" filter, and click on the Apply Constant button. You will be prompted for the filters settings to be used (and since you are using PNG files, you will also be presented with an export prompt).

Quoting Dark Shadow :

I would like to know how to use GIMP in the following script since ImageMagick's edge detection sucks compared to Gimps defaults with Sobel. Also it would be nice to not have it write the file at every step.

I don't care either way if the whole thing is moved to Gimp as long as the edge detection is.

mplayer test.mkv -vo png for img in `ls *.png`;
do convert -fx G $img $img && convert -edge 2 $img $img && convert -negate $img $img;
done;

Quoting Dark Shadow :

Now I need to know the best way to do the other steps on all frames like the sobel filter. Since when I just do something like "colours > desaturate" and "colours > invert" it only does one frame, and I could not find them anywhere under the new video menu.

GAP is only able to execute *plug-ins* registered with the Procedural DataBase (PDB). The two functions you mention are not actually plug-ins, but core functions of GIMP. Similarly for the Color Levels, Hue/Saturation, Curves tool, et cetera. Though these are all registered in the PDB, they are not directly accessible from the GAP's Apply Filter function (this is owing to the fact they do not support the "LAST VALUES" framework that plug-ins use to employ the same settings from one call to the next).

Nonetheless, GAP provides some wrapper functions for some of these core routines and tools which provide equivalent functionality (though perhaps not quite as intuitively). These can be found near the bottom of the list of PDB functions with names starting with "plug-in-wr-".

In order to desaturate your frames, you should use 'plug-in-wr-huesat' with a Saturation setting of "-100".

In order to invert the colors in your frames, use 'plug-in-wr-color-levels' and set the Low Output value to "255", and the High Output value to "0" (you are thus inverting the output).