Properly normalized the optical flow field before warping and after warping based on width and height. Now, because the range of the values are between -1 and 1 (usually much smaller), the flow doesn't get corrupted by the grid_sample for 3D or the warpPerspective for 2D anymore. So, I was able to remove all workarounds and just fix the abs to rel and rel to abs functions.
added consistency flow masks
- there is now an option to use flow consistency masks and an attached option for consistency mask blur, defaulted to 2.
- if you save extra frames, it also save consistency masks now
- you can see the effect on the flow in the flow outputs as well
- it doesn't work as great with cadence because you see afterimages, but if you up the blur it can be a little better.
fixed Frames to Video
- made the ffmpeg routine that Frames to Video uses able to take image files other than png. If png, it includes the -vcodec png as normal. But, if anything else it includes vcodec libx264, which works for jpgs. (jpgs don't work if using vcodec png, so I made it switchable). I haven't tested it with other filetypes, but I bet it works with others too. The png vcodec was specific to png.
- also added two more lines of instruction for how to use the file string.
- I also changed a few ransac functions for future use. They work as normal, but now have a switching behavior if passed depth. But, I'm not passing depth to them for now.
- a few minor code var name edits in hybrid video to align code better (mostly changed matrices to M, as is often convention
- commented a bunch of unused imports in render.py
- I'll leave it up to someone else to delete them after it's verified that everything works fine with them commented. I searched and didn't find them in that file. VSCode showed them as gray automatically, but I also verified.
Discovered that RAFT wasn't actually working due to an issue with the function that got the flows. There was a missing "elif". So, the RAFT flow would get calculated and stored in the variable 'r', but then 'r' would always be overwritten by the default Farneback at the end. We were fooling ourselves into thinking that was RAFT, when in actuality the RAFT flow is invalid and causes an error if actually used.
- Changed function call for flow methods so that this can never happen. Now, each case returns directly.
- Added to deprecation utils for now. We can remove the RAFT to Farneback conversion when we get RAFT working
Realignment of the way I handled frame indexes in motion routines to align everything more clearly
Major improvement to motion using prev_img during cadence!
- added a prev_img during cadence so that there is a prev_img to refer to for the flow
Fixed color matching issue with first frame on Image and Video Init modes
- first frame color match can't be done beforehand, so it's done afterwards. But, that normally makes for a very bad first frame. So, I added a redo for it to clean up the color matched image on first frame.
Major improvement to RANSAC
- switched to use SIFT for feature matching instead of Lucas-Kanade
- changed all border_mode to REFLECT_101, which matched how optical flow handled it, removed all the excess silly border_mode translations. This works much better.
I figured out a hack to make optical flow cadence work in 2D. To do optical flow cadence, I have to warp the flow field. But, the 2D animation warping function, usually used on images, would mess with the values of the flow (as if they were colors). So, I scaled them down by 1000 going in and scale them back up doing out, and it eliminates the effect it had which would make the image wobble around.
The same scaling actually messes with 3D optical flow cadence, so I leave that working at the normal scale factor.
I also made one modification to 3D optical flow cadence where it temporarily changes the sampling mehod (used by 3d warping function) to 'nearest' just for the flow warping, then restores it to it's previous value. This should help to minimize any pixel effects from warping.
Overlay mask working properly with static mask or video mask.
- made new masks.py file that we can use to consolidate other mask functions from main render code
- Overlay mask NOW WORKS WITH CADENCE. I made the function switchable between PIL rgb and np bgr and inject the function during cadence and for normal saves
- I didn't use old code. I just made this work myself.
Fixed one-off issue with hybrid video where the first frame was duplicated. (it became obvious when doing overlay video masking)
re-commented a few lines, removed some whitespace
new option to use init image as hybrid video
- you can use all normal compositing stuff with that mode.
- you can't use optical flow, because it's not a video - but - if there are already video frames in the directory, well - you should be able to still use flow and ransac modes, although I haven't tested that
changed Optical flow redo to a selection of optical flow type with None as default
- added Redo flow factor schedule
- made optical flow redo look nicer by warping the same image that it is using to compare to prev_img, then it goes in for final generation
changed Optical flow cadence to a selection of optical flow type with None as default
- added cadence flow factor schedule
Added flow guidance from last flow
- no UI change - it just works by default with hybrid motion optical flow types. It just uses the last flow as guidance, so it's a minor difference, but a little better temporal coherence on flow
changed hybrid composite to a selection with None as default
- now can set hybrid compositing to Normal (After Motion), Before Motion, or After Generation!
- Adds DIS Fine mode, and sets it as the new default
- Optical flow cadence and optical flow redo generation will now use DIS Fine, for pixel accuracy
- Adds flow guidance to hybrid flow. No setting needed, it just happens automatically, assisting the flow calculation by feeding in the last flow.
- Adds flow_factor_schedule for hybrid flow
- reduced some code for bordermode by setting a variable for the cv2 border mode translated from the user setting
- replaced Image,Resampling.LANCZOS with PIL.Image.LANCZOS
- had to add import PIL
- had heard of people on colab encountering this issue. Should fix both hybrid video and image/video color matching.
- tested, works fine locally, so I see no reason not to change
optional checkbox to include optical flow in normal cadence
-starts by getting flow between turbo_prev_image and turbo_next_image.
- after getting the flow, it warps the turbo_next_image with negative flow to match prev
- then, normal image warping during cadence happens
- then it warps the flow field to match the animation, first converting it to relative for warp, then back to absolute flow
- I calculate the flow division for each tween value and divide that by two.
- it then applies that tweened flow to the prev and next image
This update fixes remap's border behavior when applying optical flow.
- I extended the function so it extends borders and recrops.
- obeys wrap/replicate setting, except that wrap is replaced with reflect101
Removed optical flow options that don't work unless you run contrib version of opencv (SF and DenseRLOF)
- functions remain for possible future use if I can figure out how to run both opencv's.
Added 'DIS Medium' optical flow, which a dense flow, similar to the two types that we can't run.
Added checkbox control for hybrid_motion_use_prev_img
- makes all motion, whether ransac or flow, compare against previous rendered frame rather than previous video frame
Added color_coherence_force_grayscale
- forces grayscale before and after generation, so that color coherence mode can still be used (like color matching against video), but the image will be forced to grayscale. (NOTE: actual image file is still RGB)
The current code will always extract all of the input video frames.
This PR adds two new input parameters for extracting an input video:
extract from frame, and extract to frame.
Now we can only test frames 0 to 50, 50 to 100, etc.
better handling of wrong file/folder names, added function to delete all imgs from a folder (shorter code); now users can generate only the masks again even without having the 'generate_hybrid_frames' button selected
hithereai
kabachuha
reallybigname