Adding support for more config.toml values

README.md

@@ -51,16 +51,6 @@ The GUI allows you to set the training parameters and generate and run the requi
       - [Software Updates](#software-updates)
       - [Recommendations for Users](#recommendations-for-users)
     - [2024/03/13 (v23.0.11)](#20240313-v23011)
-    - [2024/03/13 (v23.0.9)](#20240313-v2309)
-    - [2024/03/12 (v23.0.8)](#20240312-v2308)
-    - [2024/03/12 (v23.0.7)](#20240312-v2307)
-    - [2024/03/11 (v23.0.6)](#20240311-v2306)
-    - [2024/03/11 (v23.0.5)](#20240311-v2305)
-    - [2024/03/10 (v23.0.4)](#20240310-v2304)
-    - [2024/03/10 (v23.0.3)](#20240310-v2303)
-    - [2024/03/10 (v23.0.2)](#20240310-v2302)
-    - [2024/03/09 (v23.0.1)](#20240309-v2301)
-    - [2024/03/02 (v23.0.0)](#20240302-v2300)
 
 ## 🦒 Colab
 
@@ -345,6 +335,8 @@ To use the default configuration file, follow these steps:
 
 This approach allows you to easily adjust the configuration to suit your specific needs to open the desired default folders for each type of folder/file input supported in the GUI.
 
+You can specify the path to your config.toml (or any other name you like) when running the GUI. For instance: ./gui.bat --config c:\my_config.toml
+
 ## LoRA
 
 To train a LoRA, you can currently use the `train_network.py` code. You can create a LoRA network by using the all-in-one GUI.
@@ -387,10 +379,13 @@ If you encounter an X error related to the page file, you may need to increase t
 If you encounter an error indicating that the module `tkinter` is not found, try reinstalling Python 3.10 on your system.
 
 ### LORA Training on TESLA V100 - GPU Utilization Issue
+
 #### Issue Summary
+
 When training LORA on a TESLA V100, users reported low GPU utilization. Additionally, there was difficulty in specifying GPUs other than the default for training.
 
 #### Potential Solutions
+
 - **GPU Selection:** Users can specify GPU IDs in the setup configuration to select the desired GPUs for training.
 - **Improving GPU Load:** Utilizing `adamW8bit` optimizer and increasing the batch size can help achieve 70-80% GPU utilization without exceeding GPU memory limits.
 
@@ -421,8 +416,8 @@ ControlNet dataset is used to specify the mask. The mask images should be the RG
   - DeepSpeed is supported. PR [#1101](https://github.com/kohya-ss/sd-scripts/pull/1101)  and [#1139](https://github.com/kohya-ss/sd-scripts/pull/1139) Thanks to BootsofLagrangian! See PR [#1101](https://github.com/kohya-ss/sd-scripts/pull/1101) for details.
   - The masked loss is supported in each training script. PR [#1207](https://github.com/kohya-ss/sd-scripts/pull/1207) See [Masked loss](#masked-loss) for details.
   - Some features are added to the dataset subset settings.
-    - `secondary_separator` is added to specify the tag separator that is not the target of shuffling or dropping. 
+    - `secondary_separator` is added to specify the tag separator that is not the target of shuffling or dropping.
-      - Specify `secondary_separator=";;;"`. When you specify `secondary_separator`, the part is not shuffled or dropped. 
+      - Specify `secondary_separator=";;;"`. When you specify `secondary_separator`, the part is not shuffled or dropped.
     - `enable_wildcard` is added. When set to `true`, the wildcard notation `{aaa|bbb|ccc}` can be used. The multi-line caption is also enabled.
     - `keep_tokens_separator` is updated to be used twice in the caption. When you specify `keep_tokens_separator="|||"`, the part divided by the second `|||` is not shuffled or dropped and remains at the end.
     - The existing features `caption_prefix` and `caption_suffix` can be used together. `caption_prefix` and `caption_suffix` are processed first, and then `enable_wildcard`, `keep_tokens_separator`, shuffling and dropping, and `secondary_separator` are processed in order.
@@ -436,11 +431,11 @@ ControlNet dataset is used to specify the mask. The mask images should be the RG
   - The options `--sample_every_n_epochs` and `--sample_every_n_steps` in each training script now display a warning and ignore them when a number less than or equal to `0` is specified. Thanks to S-Del for raising the issue.
   - The [English version of the dataset settings documentation](./docs/config_README-en.md) is added. PR [#1175](https://github.com/kohya-ss/sd-scripts/pull/1175) Thanks to darkstorm2150!
 
-- Add GUI support for the new parameters listed above.
+- Added GUI support for the new parameters listed above.
-- Move accelerate launch parameters to new `Accelerate launch` accordion above `Model` accordion.
+- Moved accelerate launch parameters to a new `Accelerate launch` accordion above the `Model` accordion.
-- Add support for `Debiased Estimation loss` to Dreambooth settings.
+- Added support for `Debiased Estimation loss` to Dreambooth settings.
-- Add support for "Dataset Preparation" defaults via the config.toml file.
+- Added support for "Dataset Preparation" defaults via the config.toml file.
-- Add field to allow for the input of extra accelerate launch arguments.
+- Added a field to allow for the input of extra accelerate launch arguments.
 
 ### 2024/03/21 (v23.0.15)
 
@@ -491,50 +486,3 @@ ControlNet dataset is used to specify the mask. The mask images should be the RG
 
 - Increase icon size.
 - More setup fixes.
-
-### 2024/03/13 (v23.0.9)
-
-- Reworked how setup can be run to improve Stability Matrix support.
-- Added support for huggingface-based vea path.
-
-### 2024/03/12 (v23.0.8)
-
-- Add the ability to create output and logs folder if it does not exist
-
-### 2024/03/12 (v23.0.7)
-
-- Fixed minor issues related to functions and file paths.
-
-### 2024/03/11 (v23.0.6)
-
-- Fixed an issue with PYTHON paths that have "spaces" in them.
-
-### 2024/03/11 (v23.0.5)
-
-- Updated python module verification.
-- Removed cudnn module installation in Windows.
-
-### 2024/03/10 (v23.0.4)
-
-- Updated bitsandbytes to 0.43.0.
-- Added packaging to runpod setup.
-
-### 2024/03/10 (v23.0.3)
-
-- Fixed a bug with setup.
-- Enforced proper python version before running the GUI to prevent issues with execution of the GUI.
-
-### 2024/03/10 (v23.0.2)
-
-- Improved validation of the path provided by users before running training.
-
-### 2024/03/09 (v23.0.1)
-
-- Updated bitsandbytes module to 0.43.0 as it provides native Windows support.
-- Minor fixes to the code.
-
-### 2024/03/02 (v23.0.0)
-
-- Used sd-scripts release [0.8.4](https://github.com/kohya-ss/sd-scripts/releases/tag/v0.8.4) post commit [fccbee27277d65a8dcbdeeb81787ed4116b92e0b](https://github.com/kohya-ss/sd-scripts/commit/fccbee27277d65a8dcbdeeb81787ed4116b92e0b).
-- Major code refactoring thanks to @wkpark. This will make updating sd-scripts cleaner by keeping sd-scripts files separate from the GUI files. This will also make configuration more streamlined with fewer tabs and more accordion elements. Hope you like the new style.
-- This new release is implementing a significant structure change, moving all of the sd-scripts written by kohya under a folder called sd-scripts in the root of this project. This folder is a submodule that will be populated during setup or GUI execution.

config example.toml

@@ -2,23 +2,81 @@
 # Edit the values to suit your needs
 
 # Default folders location
-models_dir = "./models"           # Pretrained model name or path
+[model]
-train_data_dir = "./data"         # Image folder (containing training images subfolders) / Image folder (containing training images)
+models_dir = "./models"                    # Pretrained model name or path
-output_dir = "./outputs"          # Output directory for trained model
+output_name = "new model"                  # Trained model output name
-reg_data_dir = "./data/reg"       # Regularisation directory
+train_data_dir = "./data"                  # Image folder (containing training images subfolders) / Image folder (containing training images)
-logging_dir = "./logs"            # Logging directory
+dataset_config = "./test.toml"             # Dataset config file (Optional. Select the toml configuration file to use for the dataset)
-config_dir = "./presets"          # Load/Save Config file
+training_comment = "Some training comment" # Training comment
-log_tracker_config_dir = "./logs" # Log tracker configs directory
+save_model_as = "safetensors"              # Save model as (ckpt, safetensors, diffusers, diffusers_safetensors)
-state_dir = "./outputs"           # Resume from saved training state
+save_precision = "bf16"                    # Save model precision (fp16, bf16, float)
-vae_dir = "./models/vae"          # VAEs folder path
 
-# Example custom folder location
+[folders]
-# models_dir = "e:/models"    # Pretrained model name or path
+output_dir = "./outputs"    # Output directory for trained model
+reg_data_dir = "./data/reg" # Regularisation directory
+logging_dir = "./logs"      # Logging directory
+
+[configuration]
+config_dir = "./presets" # Load/Save Config file
+
+[advanced]
+adaptive_noise_scale = 0                  # Adaptive noise scale
+additional_parameters = ""                # Additional parameters
+bucket_no_upscale = true                  # Don't upscale bucket resolution
+bucket_reso_steps = 64                    # Bucket resolution steps
+caption_dropout_every_n_epochs = 0        # Caption dropout every n epochs
+caption_dropout_rate = 0                  # Caption dropout rate
+color_aug = false                         # Color augmentation
+clip_skip = 1                             # Clip skip
+debiased_estimation_loss = false          # Debiased estimation loss
+flip_aug = false                          # Flip augmentation
+fp8_base = false                          # FP8 base training (experimental)
+full_bf16 = false                         # Full bf16 training (experimental)
+full_fp16 = false                         # Full fp16 training (experimental)
+gradient_accumulation_steps = 1           # Gradient accumulation steps
+gradient_checkpointing = false            # Gradient checkpointing
+ip_noise_gamma = 0                        # IP noise gamma
+ip_noise_gamma_random_strength = false    # IP noise gamma random strength (true, false)
+keep_tokens = 0                           # Keep tokens
+log_tracker_config_dir = "./logs"         # Log tracker configs directory
+log_tracker_name = ""                     # Log tracker name
+masked_loss = false                       # Masked loss
+max_data_loader_n_workers = "0"           # Max data loader n workers (string)
+max_timestep = 1000                       # Max timestep
+max_token_length = "150"                  # Max token length ("75", "150", "225")
+mem_eff_attn = false                      # Memory efficient attention
+min_snr_gamma = 0                         # Min SNR gamma
+min_timestep = 0                          # Min timestep
+multires_noise_iterations = 0             # Multires noise iterations
+multires_noise_discount = 0               # Multires noise discount
+no_token_padding = false                  # Disable token padding
+noise_offset = 0                          # Noise offset
+noise_offset_random_strength = false      # Noise offset random strength (true, false)
+noise_offset_type = "Original"            # Noise offset type ("Original", "Multires")
+persistent_data_loader_workers = false    # Persistent data loader workers
+prior_loss_weight = 1.0                   # Prior loss weight
+random_crop = false                       # Random crop
+save_every_n_steps = 0                    # Save every n steps
+save_last_n_steps = 0                     # Save last n steps
+save_last_n_steps_state = 0               # Save last n steps state
+save_state = false                        # Save state
+save_state_on_train_end = false           # Save state on train end
+scale_v_pred_loss_like_noise_pred = false # Scale v pred loss like noise pred
+shuffle_caption = false                   # Shuffle captions
+state_dir = "./outputs"                   # Resume from saved training state
+use_wandb = false                         # Use wandb
+vae_batch_size = 0                        # VAE batch size
+vae_dir = "./models/vae"                  # VAEs folder path
+v_pred_like_loss = 0                      # V pred like loss weight
+wandb_api_key = ""                        # Wandb api key
+wandb_run_name = ""                       # Wandb run name
+weighted_captions = false                 # Weighted captions
+xformers = "xformers"                     # CrossAttention (none, sdp, xformers)
 
 # This next section can be used to set default values for the Dataset Preparation section
 # The "Destination training direcroty" field will be equal to "train_data_dir" as specified above
-# [dataset_preparation]
+[dataset_preparation]
-# instance_prompt = "instance"
+instance_prompt = "instance"
-# class_prompt = "class"
+class_prompt = "class"
-# images_folder = "/some/folder/where/images/are"
+images_folder = "/some/folder/where/images/are"
-# reg_images_folder = "/some/folder/where/reg/images/are"
+reg_images_folder = "/some/folder/where/reg/images/are"

kohya_gui/class_advanced_training.py

@@ -46,10 +46,10 @@ class AdvancedTraining:
         self.config = config
 
         # Determine the current directories for VAE and output, falling back to defaults if not specified.
-        self.current_vae_dir = self.config.get("vae_dir", "./models/vae")
+        self.current_vae_dir = self.config.get("advanced.vae_dir", "./models/vae")
-        self.current_state_dir = self.config.get("state_dir", "./outputs")
+        self.current_state_dir = self.config.get("advanced.state_dir", "./outputs")
         self.current_log_tracker_config_dir = self.config.get(
-            "log_tracker_config_dir", "./logs"
+            "advanced.log_tracker_config_dir", "./logs"
         )
 
         # Define the behavior for changing noise offset type.
@@ -75,19 +75,19 @@ class AdvancedTraining:
             # Exclude token padding option for LoRA training type.
             if training_type != "lora":
                 self.no_token_padding = gr.Checkbox(
-                    label="No token padding", value=False
+                    label="No token padding", value=self.config.get("advanced.no_token_padding", False)
                 )
             self.gradient_accumulation_steps = gr.Slider(
                 label="Gradient accumulate steps",
                 info="Number of updates steps to accumulate before performing a backward/update pass",
-                value="1",
+                value=self.config.get("advanced.gradient_accumulation_steps", 1),
                 minimum=1,
                 maximum=120,
                 step=1,
             )
-            self.weighted_captions = gr.Checkbox(label="Weighted captions", value=False)
+            self.weighted_captions = gr.Checkbox(label="Weighted captions", value=self.config.get("advanced.weighted_captions", False))
         with gr.Group(), gr.Row(visible=not finetuning):
-            self.prior_loss_weight = gr.Number(label="Prior loss weight", value=1.0)
+            self.prior_loss_weight = gr.Number(label="Prior loss weight", value=self.config.get("advanced.prior_loss_weight", 1.0))
 
             def list_vae_files(path):
                 self.current_vae_dir = path if not path == "" else "."
@@ -96,14 +96,14 @@ class AdvancedTraining:
             self.vae = gr.Dropdown(
                 label="VAE (Optional: Path to checkpoint of vae for training)",
                 interactive=True,
-                choices=[""] + list_vae_files(self.current_vae_dir),
+                choices=[self.config.get("advanced.vae_dir", "")] + list_vae_files(self.current_vae_dir),
-                value="",
+                value=self.config.get("advanced.vae_dir", ""),
                 allow_custom_value=True,
             )
             create_refresh_button(
                 self.vae,
                 lambda: None,
-                lambda: {"choices": [""] + list_vae_files(self.current_vae_dir)},
+                lambda: {"choices": [self.config.get("advanced.vae_dir", "")] + list_vae_files(self.current_vae_dir)},
                 "open_folder_small",
             )
             self.vae_button = gr.Button(
@@ -116,7 +116,7 @@ class AdvancedTraining:
             )
 
             self.vae.change(
-                fn=lambda path: gr.Dropdown(choices=[""] + list_vae_files(path)),
+                fn=lambda path: gr.Dropdown(choices=[self.config.get("advanced.vae_dir", "")] + list_vae_files(path)),
                 inputs=self.vae,
                 outputs=self.vae,
                 show_progress=False,
@@ -126,23 +126,24 @@ class AdvancedTraining:
             self.additional_parameters = gr.Textbox(
                 label="Additional parameters",
                 placeholder='(Optional) Use to provide additional parameters not handled by the GUI. Eg: --some_parameters "value"',
+                value=self.config.get("advanced.additional_parameters", ""),
             )
         with gr.Row():
             self.save_every_n_steps = gr.Number(
                 label="Save every N steps",
-                value=0,
+                value=self.config.get("advanced.save_every_n_steps", 0),
                 precision=0,
                 info="(Optional) The model is saved every specified steps",
             )
             self.save_last_n_steps = gr.Number(
                 label="Save last N steps",
-                value=0,
+                value=self.config.get("advanced.save_last_n_steps", 0),
                 precision=0,
                 info="(Optional) Save only the specified number of models (old models will be deleted)",
             )
             self.save_last_n_steps_state = gr.Number(
                 label="Save last N steps state",
-                value=0,
+                value=self.config.get("advanced.save_last_n_steps_state", 0),
                 precision=0,
                 info="(Optional) Save only the specified number of states (old models will be deleted)",
             )
@@ -161,10 +162,10 @@ class AdvancedTraining:
                 ), gr.Checkbox(interactive=full_bf16_active)
 
             self.keep_tokens = gr.Slider(
-                label="Keep n tokens", value="0", minimum=0, maximum=32, step=1
+                label="Keep n tokens", value=self.config.get("advanced.keep_tokens", 0), minimum=0, maximum=32, step=1
             )
             self.clip_skip = gr.Slider(
-                label="Clip skip", value="1", minimum=1, maximum=12, step=1
+                label="Clip skip", value=self.config.get("advanced.clip_skip", 1), minimum=1, maximum=12, step=1
             )
             self.max_token_length = gr.Dropdown(
                 label="Max Token Length",
@@ -173,7 +174,7 @@ class AdvancedTraining:
                     "150",
                     "225",
                 ],
-                value="75",
+                value=self.config.get("advanced.max_token_length", "75"),
             )
 
         with gr.Row():
@@ -181,15 +182,15 @@ class AdvancedTraining:
                 self.fp8_base = gr.Checkbox(
                     label="fp8 base training (experimental)",
                     info="U-Net and Text Encoder can be trained with fp8 (experimental)",
-                    value=False,
+                    value=self.config.get("advanced.fp8_base", False),
                 )
             self.full_fp16 = gr.Checkbox(
                 label="Full fp16 training (experimental)",
-                value=False,
+                value=self.config.get("advanced.full_fp16", False),
             )
             self.full_bf16 = gr.Checkbox(
                 label="Full bf16 training (experimental)",
-                value=False,
+                value=self.config.get("advanced.full_bf16", False),
                 info="Required bitsandbytes >= 0.36.0",
             )
 
@@ -206,45 +207,45 @@ class AdvancedTraining:
 
         with gr.Row():
             self.gradient_checkpointing = gr.Checkbox(
-                label="Gradient checkpointing", value=False
+                label="Gradient checkpointing", value=self.config.get("advanced.gradient_checkpointing", False)
             )
-            self.shuffle_caption = gr.Checkbox(label="Shuffle caption", value=False)
+            self.shuffle_caption = gr.Checkbox(label="Shuffle caption", value=self.config.get("advanced.shuffle_caption", False))
             self.persistent_data_loader_workers = gr.Checkbox(
-                label="Persistent data loader", value=False
+                label="Persistent data loader", value=self.config.get("advanced.persistent_data_loader_workers", False)
             )
             self.mem_eff_attn = gr.Checkbox(
-                label="Memory efficient attention", value=False
+                label="Memory efficient attention", value=self.config.get("advanced.mem_eff_attn", False)
             )
         with gr.Row():
             self.xformers = gr.Dropdown(
                 label="CrossAttention",
                 choices=["none", "sdpa", "xformers"],
-                value="xformers",
+                value=self.config.get("advanced.xformers", "xformers"),
             )
             self.color_aug = gr.Checkbox(
                 label="Color augmentation",
-                value=False,
+                value=self.config.get("advanced.color_aug", False),
                 info="Enable weak color augmentation",
             )
             self.flip_aug = gr.Checkbox(
                 label="Flip augmentation",
-                value=False,
+                value=getattr(self.config, "advanced.flip_aug", False),
                 info="Enable horizontal flip augmentation",
             )
             self.masked_loss = gr.Checkbox(
                 label="Masked loss",
-                value=False,
+                value=self.config.get("advanced.masked_loss", False),
                 info="Apply mask for calculating loss. conditioning_data_dir is required for dataset",
             )
         with gr.Row():
             self.scale_v_pred_loss_like_noise_pred = gr.Checkbox(
                 label="Scale v prediction loss",
-                value=False,
+                value=self.config.get("advanced.scale_v_pred_loss_like_noise_pred", False),
                 info="Only for SD v2 models. By scaling the loss according to the time step, the weights of global noise prediction and local noise prediction become the same, and the improvement of details may be expected.",
             )
             self.min_snr_gamma = gr.Slider(
                 label="Min SNR gamma",
-                value=0,
+                value=self.config.get("advanced.min_snr_gamma", 0),
                 minimum=0,
                 maximum=20,
                 step=1,
@@ -252,26 +253,26 @@ class AdvancedTraining:
             )
             self.debiased_estimation_loss = gr.Checkbox(
                 label="Debiased Estimation loss",
-                value=False,
+                value=self.config.get("advanced.debiased_estimation_loss", False),
                 info="Automates the processing of noise, allowing for faster model fitting, as well as balancing out color issues. Do not use if Min SNR gamma is specified.",
             )
         with gr.Row():
             # self.sdpa = gr.Checkbox(label='Use sdpa', value=False, info='Use sdpa for CrossAttention')
             self.bucket_no_upscale = gr.Checkbox(
-                label="Don't upscale bucket resolution", value=True
+                label="Don't upscale bucket resolution", value=self.config.get("advanced.bucket_no_upscale", True)
             )
             self.bucket_reso_steps = gr.Slider(
                 label="Bucket resolution steps",
-                value=64,
+                value=self.config.get("advanced.bucket_reso_steps", 64),
                 minimum=1,
                 maximum=128,
             )
             self.random_crop = gr.Checkbox(
-                label="Random crop instead of center crop", value=False
+                label="Random crop instead of center crop", value=self.config.get("advanced.random_crop", False)
             )
             self.v_pred_like_loss = gr.Slider(
                 label="V Pred like loss",
-                value=0,
+                value=self.config.get("advanced.v_pred_like_loss", 0),
                 minimum=0,
                 maximum=1,
                 step=0.01,
@@ -281,7 +282,7 @@ class AdvancedTraining:
         with gr.Row():
             self.min_timestep = gr.Slider(
                 label="Min Timestep",
-                value=0,
+                value=self.config.get("advanced.min_timestep", 0),
                 step=1,
                 minimum=0,
                 maximum=1000,
@@ -289,7 +290,7 @@ class AdvancedTraining:
             )
             self.max_timestep = gr.Slider(
                 label="Max Timestep",
-                value=1000,
+                value=self.config.get("advanced.max_timestep", 1000),
                 step=1,
                 minimum=0,
                 maximum=1000,
@@ -303,13 +304,13 @@ class AdvancedTraining:
                     "Original",
                     "Multires",
                 ],
-                value="Original",
+                value=self.config.get("advanced.noise_offset_type", "Original"),
                 scale=1,
             )
             with gr.Row(visible=True) as self.noise_offset_original:
                 self.noise_offset = gr.Slider(
                     label="Noise offset",
-                    value=0,
+                    value=self.config.get("advanced.noise_offset", 0),
                     minimum=0,
                     maximum=1,
                     step=0.01,
@@ -317,12 +318,12 @@ class AdvancedTraining:
                 )
                 self.noise_offset_random_strength = gr.Checkbox(
                     label="Noise offset random strength",
-                    value=False,
+                    value=self.config.get("advanced.noise_offset_random_strength", False),
                     info="Use random strength between 0~noise_offset for noise offset",
                 )
                 self.adaptive_noise_scale = gr.Slider(
                     label="Adaptive noise scale",
-                    value=0,
+                    value=self.config.get("advanced.adaptive_noise_scale", 0),
                     minimum=-1,
                     maximum=1,
                     step=0.001,
@@ -331,7 +332,7 @@ class AdvancedTraining:
             with gr.Row(visible=False) as self.noise_offset_multires:
                 self.multires_noise_iterations = gr.Slider(
                     label="Multires noise iterations",
-                    value=0,
+                    value=self.config.get("advanced.multires_noise_iterations", 0),
                     minimum=0,
                     maximum=64,
                     step=1,
@@ -339,7 +340,7 @@ class AdvancedTraining:
                 )
                 self.multires_noise_discount = gr.Slider(
                     label="Multires noise discount",
-                    value=0,
+                    value=self.config.get("advanced.multires_noise_discount", 0),
                     minimum=0,
                     maximum=1,
                     step=0.01,
@@ -348,7 +349,7 @@ class AdvancedTraining:
             with gr.Row(visible=True):
                 self.ip_noise_gamma = gr.Slider(
                     label="IP noise gamma",
-                    value=0,
+                    value=self.config.get("advanced.ip_noise_gamma", 0),
                     minimum=0,
                     maximum=1,
                     step=0.01,
@@ -356,7 +357,7 @@ class AdvancedTraining:
                 )
                 self.ip_noise_gamma_random_strength = gr.Checkbox(
                     label="IP noise gamma random strength",
-                    value=False,
+                    value=self.config.get("advanced.ip_noise_gamma_random_strength", False),
                     info="Use random strength between 0~ip_noise_gamma for input perturbation noise",
                 )
             self.noise_offset_type.change(
@@ -369,19 +370,19 @@ class AdvancedTraining:
             )
         with gr.Row():
             self.caption_dropout_every_n_epochs = gr.Number(
-                label="Dropout caption every n epochs", value=0
+                label="Dropout caption every n epochs", value=self.config.get("advanced.caption_dropout_every_n_epochs", 0),
             )
             self.caption_dropout_rate = gr.Slider(
-                label="Rate of caption dropout", value=0, minimum=0, maximum=1
+                label="Rate of caption dropout", value=self.config.get("advanced.caption_dropout_rate", 0), minimum=0, maximum=1
             )
             self.vae_batch_size = gr.Slider(
-                label="VAE batch size", minimum=0, maximum=32, value=0, step=1
+                label="VAE batch size", minimum=0, maximum=32, value=self.config.get("advanced.vae_batch_size", 0), step=1
             )
         with gr.Group(), gr.Row():
-            self.save_state = gr.Checkbox(label="Save training state", value=False)
+            self.save_state = gr.Checkbox(label="Save training state", value=self.config.get("advanced.save_state", False))
 
             self.save_state_on_train_end = gr.Checkbox(
-                label="Save training state at end of training", value=False
+                label="Save training state at end of training", value=self.config.get("advanced.save_state_on_train_end", False)
             )
 
             def list_state_dirs(path):
@@ -390,15 +391,15 @@ class AdvancedTraining:
 
             self.resume = gr.Dropdown(
                 label='Resume from saved training state (path to "last-state" state folder)',
-                choices=[""] + list_state_dirs(self.current_state_dir),
+                choices=[self.config.get("advanced.state_dir", "")] + list_state_dirs(self.current_state_dir),
-                value="",
+                value=self.config.get("advanced.state_dir", ""),
                 interactive=True,
                 allow_custom_value=True,
             )
             create_refresh_button(
                 self.resume,
                 lambda: None,
-                lambda: {"choices": [""] + list_state_dirs(self.current_state_dir)},
+                lambda: {"choices": [self.config.get("advanced.state_dir", "")] + list_state_dirs(self.current_state_dir)},
                 "open_folder_small",
             )
             self.resume_button = gr.Button(
@@ -410,7 +411,7 @@ class AdvancedTraining:
                 show_progress=False,
             )
             self.resume.change(
-                fn=lambda path: gr.Dropdown(choices=[""] + list_state_dirs(path)),
+                fn=lambda path: gr.Dropdown(choices=[self.config.get("advanced.state_dir", "")] + list_state_dirs(path)),
                 inputs=self.resume,
                 outputs=self.resume,
                 show_progress=False,
@@ -418,23 +419,23 @@ class AdvancedTraining:
             self.max_data_loader_n_workers = gr.Textbox(
                 label="Max num workers for DataLoader",
                 placeholder="(Optional) Override number of epoch. Default: 8",
-                value="0",
+                value=self.config.get("advanced.max_data_loader_n_workers", "0"),
             )
         with gr.Row():
             self.use_wandb = gr.Checkbox(
                 label="WANDB Logging",
-                value=False,
+                value=self.config.get("advanced.use_wandb", False),
                 info="If unchecked, tensorboard will be used as the default for logging.",
             )
             self.wandb_api_key = gr.Textbox(
                 label="WANDB API Key",
-                value="",
+                value=self.config.get("advanced.wandb_api_key", ""),
                 placeholder="(Optional)",
                 info="Users can obtain and/or generate an api key in the their user settings on the website: https://wandb.ai/login",
             )
             self.wandb_run_name = gr.Textbox(
                 label="WANDB run name",
-                value="",
+                value=self.config.get("advanced.wandb_run_name", ""),
                 placeholder="(Optional)",
                 info="The name of the specific wandb session",
             )
@@ -446,15 +447,15 @@ class AdvancedTraining:
 
             self.log_tracker_name = gr.Textbox(
                 label="Log tracker name",
-                value="",
+                value=self.config.get("advanced.log_tracker_name", ""),
                 placeholder="(Optional)",
                 info="Name of tracker to use for logging, default is script-specific default name",
             )
             self.log_tracker_config = gr.Dropdown(
                 label="Log tracker config",
-                choices=[""]
+                choices=[self.config.get("log_tracker_config_dir", "")]
                 + list_log_tracker_config_files(self.current_log_tracker_config_dir),
-                value="",
+                value=self.config.get("log_tracker_config_dir", ""),
                 info="Path to tracker config file to use for logging",
                 interactive=True,
                 allow_custom_value=True,
@@ -463,7 +464,7 @@ class AdvancedTraining:
                 self.log_tracker_config,
                 lambda: None,
                 lambda: {
-                    "choices": [""]
+                    "choices": [self.config.get("log_tracker_config_dir", "")]
                     + list_log_tracker_config_files(self.current_log_tracker_config_dir)
                 },
                 "open_folder_small",
@@ -478,7 +479,7 @@ class AdvancedTraining:
             )
             self.log_tracker_config.change(
                 fn=lambda path: gr.Dropdown(
-                    choices=[""] + list_log_tracker_config_files(path)
+                    choices=[self.config.get("log_tracker_config_dir", "")] + list_log_tracker_config_files(path)
                 ),
                 inputs=self.log_tracker_config,
                 outputs=self.log_tracker_config,

kohya_gui/class_configuration_file.py

@@ -60,8 +60,8 @@ class ConfigurationFile:
                 # Dropdown for selecting or entering the name of a configuration file.
                 self.config_file_name = gr.Dropdown(
                     label="Load/Save Config file",
-                    choices=[""] + self.list_config_dir(self.current_config_dir),
+                    choices=[self.config.get("config_dir", "")] + self.list_config_dir(self.current_config_dir),
-                    value="",
+                    value=self.config.get("config_dir", ""),
                     interactive=True,
                     allow_custom_value=True,
                 )

kohya_gui/class_folders.py

@@ -103,8 +103,8 @@ class Folders:
             # Output directory dropdown
             self.output_dir = gr.Dropdown(
                 label="Output directory for trained model",
-                choices=[""] + self.list_output_dirs(self.current_output_dir),
+                choices=[self.config.get("folders.output_dir", "")] + self.list_output_dirs(self.current_output_dir),
-                value="",
+                value=self.config.get("folders.output_dir", ""),
                 interactive=True,
                 allow_custom_value=True,
             )
@@ -138,8 +138,8 @@ class Folders:
                     if not self.finetune
                     else "Train config directory (Optional. where config files will be saved)"
                 ),
-                choices=[""] + self.list_reg_data_dirs(self.current_reg_data_dir),
+                choices=[self.config.get("folders.reg_data_dir", "")] + self.list_reg_data_dirs(self.current_reg_data_dir),
-                value="",
+                value=self.config.get("folders.reg_data_dir", ""),
                 interactive=True,
                 allow_custom_value=True,
             )
@@ -169,8 +169,8 @@ class Folders:
             # Logging directory dropdown
             self.logging_dir = gr.Dropdown(
                 label="Logging directory (Optional. to enable logging and output Tensorboard log)",
-                choices=[""] + self.list_logging_dirs(self.current_logging_dir),
+                choices=[self.config.get("folders.logging_dir", "")] + self.list_logging_dirs(self.current_logging_dir),
-                value="",
+                value=self.config.get("folders.logging_dir", ""),
                 interactive=True,
                 allow_custom_value=True,
             )

kohya_gui/class_source_model.py

@@ -57,13 +57,13 @@ class SourceModel:
 
         # Set default directories if not provided
         self.current_models_dir = self.config.get(
-            "models_dir", os.path.join(scriptdir, "models")
+            "model.models_dir", os.path.join(scriptdir, "models")
         )
         self.current_train_data_dir = self.config.get(
-            "train_data_dir", os.path.join(scriptdir, "data")
+            "model.train_data_dir", os.path.join(scriptdir, "data")
         )
         self.current_dataset_config_dir = self.config.get(
-            "dataset_config_dir", os.path.join(scriptdir, "dataset_config")
+            "model.dataset_config", os.path.join(scriptdir, "dataset_config")
         )
 
         model_checkpoints = list(
@@ -82,7 +82,7 @@ class SourceModel:
 
         def list_train_data_dirs(path):
             self.current_train_data_dir = path if not path == "" else "."
-            return list(list_dirs(path))
+            return list(list_dirs(self.current_train_data_dir))
 
         def list_dataset_config_dirs(path: str) -> list:
             """
@@ -109,7 +109,7 @@ class SourceModel:
                         self.pretrained_model_name_or_path = gr.Dropdown(
                             label="Pretrained model name or path",
                             choices=default_models + model_checkpoints,
-                            value="runwayml/stable-diffusion-v1-5",
+                            value=self.config.get("model.models_dir", "runwayml/stable-diffusion-v1-5"),
                             allow_custom_value=True,
                             visible=True,
                             min_width=100,
@@ -150,7 +150,7 @@ class SourceModel:
                         self.output_name = gr.Textbox(
                             label="Trained Model output name",
                             placeholder="(Name of the model to output)",
-                            value="last",
+                            value=self.config.get("model.output_name", "last"),
                             interactive=True,
                         )
                 with gr.Row():
@@ -163,7 +163,7 @@ class SourceModel:
                             ),
                             choices=[""]
                             + list_train_data_dirs(self.current_train_data_dir),
-                            value="",
+                            value=self.config.get("model.train_data_dir", ""),
                             interactive=True,
                             allow_custom_value=True,
                         )
@@ -191,9 +191,9 @@ class SourceModel:
                         # Toml directory dropdown
                         self.dataset_config = gr.Dropdown(
                             label="Dataset config file (Optional. Select the toml configuration file to use for the dataset)",
-                            choices=[""]
+                            choices=[self.config.get("model.dataset_config", "")]
                             + list_dataset_config_dirs(self.current_dataset_config_dir),
-                            value="",
+                            value=self.config.get("model.dataset_config", ""),
                             interactive=True,
                             allow_custom_value=True,
                         )
@@ -264,18 +264,19 @@ class SourceModel:
                         label="Training comment",
                         placeholder="(Optional) Add training comment to be included in metadata",
                         interactive=True,
+                        value=self.config.get("model.training_comment", ""),
                     )
 
                 with gr.Row():
                     self.save_model_as = gr.Radio(
                         save_model_as_choices,
                         label="Save trained model as",
-                        value="safetensors",
+                        value=self.config.get("model.save_model_as", "safetensors"),
                     )
                     self.save_precision = gr.Radio(
                         save_precision_choices,
                         label="Save precision",
-                        value="fp16",
+                        value=self.config.get("model.save_precision", "fp16"),
                     )
 
                 self.pretrained_model_name_or_path.change(