turns out ddim_eta is a global general setting param now, not passed directly as a p.ddim_eta any more. So i made it a schedule.
Todo: only show in ui if ddim is selected.
Properly normalized the optical flow field before warping and after warping based on width and height. Now, because the range of the values are between -1 and 1 (usually much smaller), the flow doesn't get corrupted by the grid_sample for 3D or the warpPerspective for 2D anymore. So, I was able to remove all workarounds and just fix the abs to rel and rel to abs functions.
removed excess options, removed dev code, cleaned up code, ready for production.
- removed depth tensor autocontrast options
- made equalization a part of the normalization process (not an optional arg now)
- removed my mechanism for 0-based normalization. Decided it wasn't any better. Maybe worse... mostly similar.
- cracked open a bottle of champagne on this function's hull
also added a file for the first steps of making an auto-navigation module to be used in the future inside the transform_image_3d function! It generates a rotation matrix based on greatest or least depth in the tensor, instead of using animation keys.
makes progress advance smoothly during cadence rather than skipping ahead suddenly after cadence completes
left line commented for state.current_image, since I'm not sure what that variable actually does. It doesn't update the preview when I try it, even when turning up the speed of preview updates in auto1111. But, if we decide to make a preview mechanism or need that state var for cadence, it can just be uncommented. It matches the format of the one in the main non-cadence loop.
added consistency flow masks
- there is now an option to use flow consistency masks and an attached option for consistency mask blur, defaulted to 2.
- if you save extra frames, it also save consistency masks now
- you can see the effect on the flow in the flow outputs as well
- it doesn't work as great with cadence because you see afterimages, but if you up the blur it can be a little better.
fixed Frames to Video
- made the ffmpeg routine that Frames to Video uses able to take image files other than png. If png, it includes the -vcodec png as normal. But, if anything else it includes vcodec libx264, which works for jpgs. (jpgs don't work if using vcodec png, so I made it switchable). I haven't tested it with other filetypes, but I bet it works with others too. The png vcodec was specific to png.
- also added two more lines of instruction for how to use the file string.
- I also changed a few ransac functions for future use. They work as normal, but now have a switching behavior if passed depth. But, I'm not passing depth to them for now.
- a few minor code var name edits in hybrid video to align code better (mostly changed matrices to M, as is often convention
- commented a bunch of unused imports in render.py
- I'll leave it up to someone else to delete them after it's verified that everything works fine with them commented. I searched and didn't find them in that file. VSCode showed them as gray automatically, but I also verified.
It wasn't working anyway, so I removed that mechanism and restored it to the previous behavior of just color matching after generation when using Image of Video Input.
Also cleaned up some code and added console reporting about Redo cycles.
Discovered that RAFT wasn't actually working due to an issue with the function that got the flows. There was a missing "elif". So, the RAFT flow would get calculated and stored in the variable 'r', but then 'r' would always be overwritten by the default Farneback at the end. We were fooling ourselves into thinking that was RAFT, when in actuality the RAFT flow is invalid and causes an error if actually used.
- Changed function call for flow methods so that this can never happen. Now, each case returns directly.
- Added to deprecation utils for now. We can remove the RAFT to Farneback conversion when we get RAFT working
Realignment of the way I handled frame indexes in motion routines to align everything more clearly
Major improvement to motion using prev_img during cadence!
- added a prev_img during cadence so that there is a prev_img to refer to for the flow
Fixed color matching issue with first frame on Image and Video Init modes
- first frame color match can't be done beforehand, so it's done afterwards. But, that normally makes for a very bad first frame. So, I added a redo for it to clean up the color matched image on first frame.
Major improvement to RANSAC
- switched to use SIFT for feature matching instead of Lucas-Kanade
- changed all border_mode to REFLECT_101, which matched how optical flow handled it, removed all the excess silly border_mode translations. This works much better.
I figured out a hack to make optical flow cadence work in 2D. To do optical flow cadence, I have to warp the flow field. But, the 2D animation warping function, usually used on images, would mess with the values of the flow (as if they were colors). So, I scaled them down by 1000 going in and scale them back up doing out, and it eliminates the effect it had which would make the image wobble around.
The same scaling actually messes with 3D optical flow cadence, so I leave that working at the normal scale factor.
I also made one modification to 3D optical flow cadence where it temporarily changes the sampling mehod (used by 3d warping function) to 'nearest' just for the flow warping, then restores it to it's previous value. This should help to minimize any pixel effects from warping.
After thorough testing of generation directly, I verified that it can handle tile sizes of 8. If this tile size is not met, animation does now work correctly because the images coming out of generation don't match the specified dimensions, making the prev_img wrong, which makes animation warping wrong.
Previous tile size of 64 was legacy, from notebook, and old auto1111 I believe. But, the pipeline can handle divisors of 8, verified. However, still not sure if the images produced are as good as when you use the 64 tile size, which is the size of the latent representation.
In any case, there was no limit on this before, and with these changes, it will assure that animation is always accurate. Suggest leaving slider at increments of 64. But, now if they select a dimension manually, it will be properly forced to tile size of 8, to ensure proper sizing through the engine.
Overlay mask working properly with static mask or video mask.
- made new masks.py file that we can use to consolidate other mask functions from main render code
- Overlay mask NOW WORKS WITH CADENCE. I made the function switchable between PIL rgb and np bgr and inject the function during cadence and for normal saves
- I didn't use old code. I just made this work myself.
Fixed one-off issue with hybrid video where the first frame was duplicated. (it became obvious when doing overlay video masking)
re-commented a few lines, removed some whitespace
This was an afterthought, and after further testing, I don't think it's working quite right. I'm not sure why, but removing for now. Will re-examine later.
I accidentally had color matching against video on instead of LAB at first when I was testing, so I didn't see it. Maybe I just need to convert to BGR?
Anyway, this restores the previous behavior of color matching, where if it resumes, it just matches against the prev_img.
Now requires at least two actual frames to resume, but it works.
So,
at cadence 1, you need 2 frames
at cadence 3, you need 3 frames
at cadence 10, you need 10 frames
etc.
It gets the first and last frames, reads them, and feeds them to cadence cycle properly for turbo_prev_img AND turbo_next_img, then it recreates the last frame cycle.
It automatically accounts for cadence interruptions, like if you ran out of disk space during cadence or it got interrupted. (or you delete the last few frames or whatever)
Basically, I can't make it fail now, no matter how many times I resume, or how many random frames I delete going backwards before resuming.
Try it out!
I did consider saving out any frames that were recreated, but then realized that has it's own issues with possible duplicates as well, and after testing, I just don't think it's necessary either. It works, every time, as far as I can tell.
new option to use init image as hybrid video
- you can use all normal compositing stuff with that mode.
- you can't use optical flow, because it's not a video - but - if there are already video frames in the directory, well - you should be able to still use flow and ransac modes, although I haven't tested that
changed Optical flow redo to a selection of optical flow type with None as default
- added Redo flow factor schedule
- made optical flow redo look nicer by warping the same image that it is using to compare to prev_img, then it goes in for final generation
changed Optical flow cadence to a selection of optical flow type with None as default
- added cadence flow factor schedule
Added flow guidance from last flow
- no UI change - it just works by default with hybrid motion optical flow types. It just uses the last flow as guidance, so it's a minor difference, but a little better temporal coherence on flow
changed hybrid composite to a selection with None as default
- now can set hybrid compositing to Normal (After Motion), Before Motion, or After Generation!
- Uses the same variable, optical_flow_cadence but now uses "None" for disabled and has the three optical flow methods.
- Improved reporting of cadence frames to include optical flow cadence type if one is selected
My most embarrassing commit.. I tested a bunch of different numbers of redos... except for 1... which caused an error and made me realize I had a one off problem... Tested 0, 1, 2, 3... working properly now. 1 redo yields 2 generations... as it should.
After testing it became obvious that part of the magic of the redo was letting the colors drift during redo... so, I just color matched on the way out, and it maintains it's cool factor, while maintaining colors!
Added slider for diffusion redo.
- llterally does a diffusion multiple extra times, feeding it back into itself. It is suggested to turn steps down if you turn redo up.
- while redoing, seed is randomized and restored
- fixed Optical Flow Redo so that it also randomizes and restores the seed, making it less muddy.
- Adds DIS Fine mode, and sets it as the new default
- Optical flow cadence and optical flow redo generation will now use DIS Fine, for pixel accuracy
- Adds flow guidance to hybrid flow. No setting needed, it just happens automatically, assisting the flow calculation by feeding in the last flow.
- Adds flow_factor_schedule for hybrid flow
- reduced some code for bordermode by setting a variable for the cv2 border mode translated from the user setting
- replaced Image,Resampling.LANCZOS with PIL.Image.LANCZOS
- had to add import PIL
- had heard of people on colab encountering this issue. Should fix both hybrid video and image/video color matching.
- tested, works fine locally, so I see no reason not to change
Follow up to strength 0 color match reset... prevents color match sample being replaced when strength goes to 0 if using color coherence Video or color coherence Image.
Option works in 2D/3D mode
Does a disposable generation before the actual generation to get the flow between previous image and the generation. Then, it discards that generation and warps the init_sample based on the captured flow before the actual generation happens.
- Takes twice as long (obviously)
- It can introduce smoothing, especially over time, but also adds a type of artistic consistency between frames.
- can be mixed and matched with any other modes, optical flow cadence, video, etc...
You can now color match against an image!
- the one caveat is that normal color matching is done with the prev_img. There is no prev_img for the 1st frame, so I had to force the color match on the output rather than the input. It makes the first frame look sharpened at times, since the histogram is forced on it.
I also fixed video color coherence
- I realized that it was never color matching on the first frame for video. You usually just wouldn't notice, since you're using the video.
- I also realized it was one frame off, fixed.
Once I tried some different images I realized that the color balance was RGB/BGR swapped on the color matches. Ooops! This fixes it. Perfect color matching after strength 0.
I wasn't thinking about it quite right before, and I was cancelling the wrong cadence section based on strength 0. This repairs it.
It now looks at the proper strength key (from the tween frame_start_idx) to see if strength is 0 and skip optical flow cadence if it is.
Also, fixes and condenses the reporting of whether it's a cadence frame or an optical flow cadence by combining the info with the tween reporting.
I made cadence skip optical flow on frames with strength 0. This won't actually help you unless your cadence start is aligned perfectly with your strength 0.
Made strength 0 reset color match, as if it were the first frame in an animation.
I plan to make a scene change schedule, at which point I can add a strength threshold value for scene changes and replace this code.
I'll have to do some more work to track down a method to warp the flow properly in 2D mode. For now, optical flow cadence is a 3D only thing.
Made it say Optical flow cadence (3D only)
Made it hidden by default
Made it show when you switch to 3D mode
Made it not do the routines on the back end if not in 3D mode
reallybigname
hithereai
kabachuha
rewbs
Joe Toch
Jason Burbage
reallybigname