# Output Format Design
## Purpose
This document describes how `diffusion-cli` should let users choose the
final image file format and compression quality.
The current generation path produces RGB image tensors and writes or
returns PNG images. PNG is a good default because it is lossless and is
widely supported, but it can be inconveniently large. Users should be
able to request smaller formats such as JPEG, WebP, or AVIF without
changing the model inference path.
The design keeps inference and compression separate:
- Model inference still produces image tensors.
- The internal no-loss handoff format remains PNG when an external
compressor is needed.
- A post-processing step invokes ImageMagick only when the requested
final format requires it.
- CLI and web UI users receive the final requested format.
This separation matters because image generation and image compression
have different failure modes. A model failure means inference did not
complete. A compression failure means inference completed, but the final
encoding could not be produced. The user should see a clear error for
the exact failing stage.
## Goals
The feature must:
- Add configurable output format and quality values.
- Support config file defaults.
- Support CLI overrides for one-shot generation.
- Expose format and quality controls in the web UI.
- Keep PNG as the default output format.
- Avoid requiring ImageMagick for default PNG output.
- Use ImageMagick for non-PNG formats.
- Surface ImageMagick failures clearly to CLI and web UI users.
- Return correct MIME types from first-party UI generation responses.
- Preserve existing base64 response shapes while encoding the final
compressed image bytes.
- Add focused tests for validation, command construction, errors, and
UI request handling.
The feature should not:
- Change the diffusion sampling algorithm.
- Change VAE decode behavior.
- Add image-to-image support.
- Add upload support.
- Add an async conversion job queue.
- Add a frontend build system.
- Replace Pillow for the existing PNG path.
- Let arbitrary user strings become output command arguments without
validation.
## External References
The implementation should be checked against these references:
- ImageMagick command line usage:
[ImageMagick command-line processing](https://imagemagick.org/script/command-line-processing.php).
- ImageMagick quality option:
[ImageMagick command-line options: quality](https://imagemagick.org/script/command-line-options.php#quality).
- Python subprocess safety and error handling:
[subprocess documentation](https://docs.python.org/3/library/subprocess.html).
- Python temporary file handling:
[tempfile documentation](https://docs.python.org/3/library/tempfile.html).
- MIME type registration background:
[IANA media types](https://www.iana.org/assignments/media-types/media-types.xhtml).
- HTTP response semantics:
[RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html).
## Terminology
### Output Format
The output format is the final image encoding given to the user.
Examples:
- `png`
- `jpg`
- `webp`
- `avif`
The output format is not the same thing as the model output. The model
does not directly create JPEG or AVIF files. The model creates numeric
image data, and the image I/O layer encodes that data.
### Extension Name
The extension name is the suffix used for output file names.
For example, if the output path is:
```text
image.png
```
and the requested extension is:
```text
jpg
```
then the final output path should be:
```text
image.jpg
```
The extension name must be normalized before use. This avoids treating
`JPG`, `.jpg`, and `jpeg` as different internal formats.
### Quality
Quality is an integer passed to ImageMagick's `-quality` option.
For lossy formats, higher values usually mean larger files and fewer
visible compression artifacts. For some formats and ImageMagick builds,
the exact interpretation can vary. The application should therefore
describe this setting as an encoder quality value rather than promising
a specific file size.
### Final Image
The final image is the image after optional compression.
For PNG output, the final image is produced directly by the existing
Pillow path. For non-PNG output, the final image is produced by
ImageMagick from a temporary PNG input.
## User-Facing Behavior
### Default Behavior
Existing users should not need ImageMagick unless they opt into a
non-PNG format.
With no new config and no new CLI options:
```bash
uv run diffusion-cli --prompt "a ceramic mug"
```
the command should continue to write:
```text
output.png
```
This behavior avoids turning ImageMagick into a hard runtime dependency
for the default case.
### CLI Behavior
Add two generation options:
```text
--output-extension EXTENSION
--output-quality QUALITY
```
`--output-extension`
: Selects the final file extension and image format. The value should
accept either a bare extension or a dotted extension. For example,
`jpg` and `.jpg` should both be valid.
`--output-quality`
: Selects the ImageMagick quality value for non-PNG formats. The value
must be an integer between 1 and 100.
Example:
```bash
uv run diffusion-cli \
--prompt "a ceramic mug" \
--output output.png \
--output-extension jpg \
--output-quality 85
```
The command should write:
```text
output.jpg
```
The user-provided output path still controls the directory and base
name. The selected extension controls the suffix. This means users can
keep existing config such as `output = "output.png"` and only change
the format setting.
### Config File Behavior
Add two keys under `[generation]`:
```toml
[generation]
output_extension = "jpg"
output_quality = 85
```
These values should follow the same precedence model as other
generation defaults:
1. Command line option.
2. Config file value.
3. Built-in default.
The built-in defaults should be:
```text
output_extension = "png"
output_quality = 95
```
The exact default quality does not matter for PNG when PNG bypasses
ImageMagick. It does matter as the default for users who select a
non-PNG format without setting quality. A value of `95` preserves high
visual quality while still allowing formats such as JPEG, WebP, and
AVIF to compress better than PNG in many cases.
### Web UI Behavior
The web UI should expose format and quality next to the other generation
settings.
The controls should be:
- A `select` for output format.
- A numeric input or slider for quality.
The format options should be visible labels backed by normalized values:
```text
PNG -> png
JPEG -> jpg
WebP -> webp
AVIF -> avif
```
The UI should send these fields to `/ui/api/txt2img`:
```json
{
"prompt": "a ceramic mug",
"output_extension": "jpg",
"output_quality": 85
}
```
The UI defaults endpoint should include:
```json
{
"output_extension": "png",
"output_quality": 95,
"output_formats": [
{"value": "png", "label": "PNG"},
{"value": "jpg", "label": "JPEG"},
{"value": "webp", "label": "WebP"},
{"value": "avif", "label": "AVIF"}
]
}
```
The generated image response should use the final MIME type:
```json
{
"images": [
{
"mime_type": "image/jpeg",
"data": "...base64...",
"seed": 123
}
]
}
```
The browser already builds data URLs from `mime_type` and `data`, so the
gallery can keep using the same display flow.
## Supported Formats
The first implementation should use an explicit allowlist:
```text
png
jpg
jpeg
webp
avif
```
Internally, normalize this list to canonical extensions:
```text
png -> png
jpg -> jpg
jpeg -> jpg
webp -> webp
avif -> avif
```
The application should reject unsupported extensions before running
ImageMagick. This gives the user a predictable validation error and
prevents arbitrary strings from becoming output file suffixes.
The initial MIME type mapping should be:
```text
png -> image/png
jpg -> image/jpeg
webp -> image/webp
avif -> image/avif
```
The mapping should live in Python code rather than relying only on
`mimetypes.guess_type()`. A small explicit mapping is easier to test and
keeps behavior stable across platforms.
## Data Model Changes
### GenerationDefaults
Add optional config defaults:
```python
output_extension: str | None = None
output_quality: int | None = None
```
These are optional because a config file may omit either key.
### ImageGenerationRequest
Add optional per-request values:
```python
output_extension: str | None = None
output_quality: int | None = None
```
These values are request-level because the web UI and CLI can override
config defaults for a single generation.
### GenerationConfig
Add validated final values:
```python
output_extension: str
output_quality: int
output_mime_type: str
```
The extension should already be normalized in `GenerationConfig`. Code
outside the config layer should not need to understand aliases such as
`jpeg`.
The MIME type can either be stored directly or computed by a helper.
Storing it on the config is acceptable because it is derived from a
validated extension and is useful at response boundaries.
### GeneratedImage
`GeneratedImage` already has:
```python
data: bytes
format: str
seed: int
```
The `format` field should contain the normalized final extension. The UI
route can derive MIME type from this value, or the service can include
MIME type if the response model is expanded later.
## Validation
### Extension Validation
Validation should:
1. Require a string.
2. Strip surrounding whitespace.
3. Remove one leading dot if present.
4. Convert to lowercase.
5. Map aliases to canonical values.
6. Reject unsupported values.
Examples:
```text
"jpg" -> "jpg"
".jpg" -> "jpg"
"JPEG" -> "jpg"
" webp" -> "webp"
"gif" -> error
"" -> error
```
The error should mention the accepted values:
```text
Output extension must be one of png, jpg, webp, avif: got gif
```
### Quality Validation
Validation should:
1. Require an integer.
2. Reject booleans.
3. Require the range `1..100`.
Examples:
```text
85 -> valid
1 -> valid
100 -> valid
0 -> error
101 -> error
true -> error
"85" -> error in TOML and JSON parsing layers
```
The error should be:
```text
Output quality must be between 1 and 100: got 0
```
## Image I/O Design
### Current Boundary
The current `image_io.py` module already owns:
- Batch output path calculation.
- Tensor-to-uint8 conversion.
- File saving.
- Byte encoding for HTTP responses.
This is the right module for final image format handling because it is
already the boundary between tensors and encoded images.
### Output Path Calculation
Batch naming should keep the existing behavior of inserting the batch
index before the suffix:
```text
output.jpg
output_0000.jpg
output_0001.jpg
```
The output suffix should be determined by the normalized output
extension, not by the suffix on the configured output path.
Recommended helper:
```python
def outputPathWithExtension(output: Path, extension: str) -> Path:
"""Return output with the normalized image extension."""
```
For `output = Path("result.png")` and `extension = "webp"`, the helper
returns:
```text
result.webp
```
For `output = Path("result")` and `extension = "jpg"`, the helper
returns:
```text
result.jpg
```
### PNG Path
For `png`, keep using Pillow directly:
```python
Image.fromarray(..., mode="RGB").save(path)
```
This preserves current behavior and keeps the default path independent
from ImageMagick.
### Non-PNG File Path
For non-PNG file output:
1. Convert the tensor image to an RGB uint8 array.
2. Save that array to a temporary PNG file.
3. Run ImageMagick:
```text
magick TEMP.png -quality QUALITY OUTPUT.EXT
```
4. Return the final output path.
5. Clean up the temporary PNG.
The temporary PNG should be created in a temporary directory. This
avoids leaving intermediate files next to user output and avoids
collisions between concurrent conversions.
Use `subprocess.run()` with a list of arguments:
```python
subprocess.run(
[
"magick",
str(temp_png),
"-quality",
str(quality),
str(output_path),
],
check=True,
capture_output=True,
text=True,
)
```
Do not use `shell=True`. A list of arguments avoids shell parsing and
prevents spaces or special characters in paths from changing command
meaning.
### Non-PNG Byte Path
For web UI responses:
1. Convert the tensor image to an RGB uint8 array.
2. Encode the array to temporary PNG bytes or a temporary PNG file.
3. Run ImageMagick to write the requested final format to a temporary
output file.
4. Read the final output bytes.
5. Return those bytes in `GeneratedImage`.
Using temporary files for both input and output is simpler than piping
binary data into and out of ImageMagick. It also matches the CLI command
shape and keeps error handling consistent.
## ImageMagick Error Handling
### Missing Executable
If `magick` is missing, `subprocess.run()` raises `FileNotFoundError`.
The application should catch it and raise `DiffusionCliError`:
```text
ImageMagick executable not found: magick
```
This tells the user the dependency is missing.
### Failed Conversion
If ImageMagick exits non-zero, `subprocess.run(..., check=True)` raises
`subprocess.CalledProcessError`.
The application should include the most useful available diagnostic:
1. `stderr`, if present.
2. `stdout`, if present.
3. A generic message with the exit code.
Example final error:
```text
ImageMagick failed to write avif output: no encode delegate for this
image format
```
This is important for AVIF because ImageMagick support depends on the
local build and installed delegates.
### Empty Output
After ImageMagick succeeds, the code should verify that the final output
file exists and is a regular file.
If the file is missing, raise:
```text
ImageMagick did not create output file: /path/to/output.avif
```
This check catches unusual cases where a command exits successfully but
does not produce the expected file.
## Service Design
### GenerationService.generateImages
`generateImages()` should return final-format bytes.
Current behavior:
```text
tensor -> PNG bytes -> GeneratedImage(format="png")
```
New behavior:
```text
tensor -> configured output bytes -> GeneratedImage(format=extension)
```
The service already builds a full `GenerationConfig` before calling the
model stack. That config should include `output_extension` and
`output_quality`, so the service can pass validated values to
`encodeImages()`.
### GenerationService.generateToFiles
`generateToFiles()` should write final-format files.
Current behavior:
```text
tensor -> save PNG files -> list[Path]
```
New behavior:
```text
tensor -> save configured format files -> list[Path]
```
The returned paths must be the final paths. If the user requested
`jpg`, the printed CLI path should end in `.jpg`.
## API Behavior
### First-Party UI API
`POST /ui/api/txt2img` should accept:
```json
{
"output_extension": "jpg",
"output_quality": 85
}
```
The route should validate JSON scalar types before constructing the
internal request:
- `output_extension` must be a string when present.
- `output_quality` must be an integer when present.
The config layer should perform final normalization and range checking.
The UI route may also reject obviously invalid scalar types early so the
user gets a `400 bad_request` instead of a later generation error.
On ImageMagick failure, the response should be:
```json
{
"error": "generation_failed",
"message": "ImageMagick failed to write avif output: ..."
}
```
Use status code `500` because inference may have succeeded, but the
server failed to complete the requested operation.
### Base64 API Responses
All API responses that return base64 image data should base64 encode the
final compressed image bytes.
For example, if the selected output extension is `jpg`, the generation
flow should produce JPEG bytes first, and then encode those JPEG bytes
as base64. The base64 string should not imply PNG. It is only a text
transport encoding for whatever image bytes were produced.
The first-party UI response already includes `mime_type`, so it can tell
the browser exactly what the base64 data contains:
```json
{
"images": [
{
"mime_type": "image/jpeg",
"data": "...base64...",
"seed": 123
}
]
}
```
Compatibility responses such as `/sdapi/v1/txt2img` may not have a MIME
type field in their existing response shape. For now, they should still
base64 encode the compressed bytes and assume the consumer can handle
the image format. This keeps one consistent output-format path across
CLI, UI, and compatibility APIs.
### SillyTavern Compatibility API
The compatibility API should keep returning base64 strings in the same
shape it returns today.
The first implementation may add parsing for output format fields only
if a compatibility client sends them, but it does not need to invent a
new compatibility-specific response shape. If the service returns JPEG,
WebP, or AVIF bytes, the route should base64 encode those bytes directly
and return them in the existing `images` array.
This assumes the consumer can handle the compressed image format. If a
specific client later proves it only accepts PNG, that client-specific
behavior should be handled in that API profile deliberately.
## Frontend Design
### Form State
Add fields to `INITIAL_FORM`:
```javascript
output_extension: "png",
output_quality: 95,
```
When defaults load, merge server-provided values the same way as other
generation defaults.
### Controls
Add a format select:
```text
Format: [PNG | JPEG | WebP | AVIF]
```
Add a quality input:
```text
Quality: [95]
```
The quality input should use:
```text
min = 1
max = 100
step = 1
```
The quality control can remain enabled for PNG for implementation
simplicity, but the UI may disable it when PNG is selected because PNG
does not use ImageMagick in the first implementation. If it is disabled
for PNG, the request should still include the current value or omit it;
the backend default will handle either case.
### Gallery
The current gallery uses:
```javascript
"data:" + image.mime_type + ";base64," + image.data
```
That behavior should remain. The only required change is that
`mime_type` must come from the backend's final format.
## Security and Robustness
### Avoid Shell Invocation
The ImageMagick command must be run with argument lists and
`shell=False`, which is the default for `subprocess.run()`.
This matters because output paths and config values are user-controlled.
Even though the extension is validated, paths may contain spaces or
characters that have meaning in a shell. Passing an argument list avoids
shell interpretation.
### Validate Before Command Construction
Extension and quality should be validated before any temporary file or
subprocess work starts.
This keeps bad input cheap to reject and makes error messages
deterministic.
### Temporary Files
Temporary files should live in a temporary directory created by
`tempfile.TemporaryDirectory()`.
The temporary directory should be scoped to one conversion or one batch
save operation. When the context manager exits, intermediates are
removed automatically.
### Concurrency
Server generation is already serialized by `GenerationService`'s lock.
Temporary directories still matter because future code may relax the
lock or add more server concurrency. Unique temporary directories avoid
file name collisions.
## Implementation Plan
### Step 1: Add Config Constants and Validation
Add constants:
```python
DEFAULT_OUTPUT_EXTENSION = "png"
DEFAULT_OUTPUT_QUALITY = 95
OUTPUT_EXTENSION_ALIASES = {
"png": "png",
"jpg": "jpg",
"jpeg": "jpg",
"webp": "webp",
"avif": "avif",
}
OUTPUT_MIME_TYPES = {
"png": "image/png",
"jpg": "image/jpeg",
"webp": "image/webp",
"avif": "image/avif",
}
```
Add helpers:
```python
def normalizeOutputExtension(value: str) -> str:
"""Return a supported canonical output extension."""
def validateOutputQuality(value: int) -> int:
"""Return a valid output quality value."""
def outputMimeType(extension: str) -> str:
"""Return the MIME type for a normalized output extension."""
```
These helpers should live in `config.py` unless the project later grows
a dedicated output-format module.
### Step 2: Extend TOML Loading
Add `output_extension` and `output_quality` to
`GENERATION_CONFIG_KEYS`.
Parse them with existing scalar helpers:
- `_optionalString()` for `output_extension`
- `_optionalInt()` for `output_quality`
Do not normalize TOML values during raw loading. Normalization belongs
in the config build step where CLI, UI, and config values are merged.
### Step 3: Extend Request and Config Dataclasses
Add the fields described in the data model section.
Update `buildGenerationConfigFromRequest()` to coalesce:
```python
raw_extension = coalesce(
request.output_extension,
generation.output_extension,
DEFAULT_OUTPUT_EXTENSION,
)
raw_quality = coalesce(
request.output_quality,
generation.output_quality,
DEFAULT_OUTPUT_QUALITY,
)
```
Then validate:
```python
output_extension = normalizeOutputExtension(raw_extension)
output_quality = validateOutputQuality(raw_quality)
```
### Step 4: Extend CLI Parser
Add parser options with no argparse defaults:
```python
parser.add_argument("--output-extension")
parser.add_argument("--output-quality", type=int)
```
Update `buildGenerationConfig()` to pass the parsed values into
`ImageGenerationRequest`.
### Step 5: Add Image I/O Conversion
Update `saveImages()` and `encodeImages()` to accept:
```python
output_extension: str = "png"
output_quality: int = 95
```
For PNG, keep existing behavior.
For non-PNG, call a helper that writes a temporary PNG and runs
ImageMagick.
Recommended helper names:
```python
def convertPngFile(
png_path: Path,
output_path: Path,
output_extension: str,
output_quality: int,
) -> None:
"""Convert one PNG file to the requested output format."""
def encodeArray(
image_array: np.ndarray,
output_extension: str,
output_quality: int,
) -> bytes:
"""Encode one RGB image array to final image bytes."""
```
### Step 6: Update Generation Service
Pass config output fields to image I/O:
```python
encoded_images = encodeImages(
images,
config.output_extension,
config.output_quality,
)
```
Return:
```python
GeneratedImage(
data=data,
format=config.output_extension,
seed=config.seed,
)
```
Update file output similarly:
```python
saveImages(
images,
config.output,
config.output_extension,
config.output_quality,
)
```
### Step 7: Update UI Routes
Add JSON parsing for:
```text
output_extension
output_quality
```
Include defaults and format options in `/ui/api/defaults`.
Use the image format to set MIME type in `/ui/api/txt2img`:
```python
"mime_type": outputMimeType(image.format)
```
### Step 8: Update Frontend
Add form state, controls, and request fields.
The UI should not hard-code MIME behavior. It should continue to trust
the backend response's `mime_type`.
### Step 9: Update Documentation
Update `README.md` with:
- Config keys.
- CLI examples.
- Note that ImageMagick is required for non-PNG formats.
- Note that AVIF support depends on the local ImageMagick build.
## Testing Plan
### Config Tests
Add tests that verify:
- Missing config uses `png` and `95`.
- TOML `output_extension = "jpeg"` normalizes to `jpg`.
- CLI `--output-extension webp` overrides config `jpg`.
- Invalid extension fails clearly.
- Invalid quality fails clearly.
- Boolean quality is rejected.
- UI defaults include output format and quality.
### CLI Tests
Add tests that verify:
- Parser leaves new configurable defaults as `None`.
- Parser accepts explicit `--output-extension`.
- Parser accepts explicit `--output-quality`.
### Image I/O Tests
Add tests that verify:
- `saveImages()` still writes PNG by default.
- Non-PNG output changes the file suffix.
- ImageMagick command uses argument lists in the expected order.
- Missing `magick` raises `DiffusionCliError`.
- Non-zero ImageMagick exit raises `DiffusionCliError` with stderr.
- `encodeImages()` returns final-format bytes for non-PNG by reading
the converted output file.
The ImageMagick tests should mock `subprocess.run()` rather than require
ImageMagick in the test environment.
### Generation Service Tests
Add tests that verify:
- `generateImages()` passes output extension and quality to
`encodeImages()`.
- Returned `GeneratedImage.format` matches the configured extension.
- `generateToFiles()` passes output extension and quality to
`saveImages()`.
### UI Route Tests
Add tests that verify:
- Defaults endpoint includes format options.
- UI request includes output format values in the internal request.
- Invalid `output_extension` returns `400`.
- Invalid `output_quality` returns `400`.
- Response MIME type changes when the generated image format is `jpg`.
- ImageMagick-related `DiffusionCliError` returns `500` with the error
message.
### Frontend Tests
The project currently has lightweight static frontend coverage through
served asset tests. The first implementation can keep this level and
verify that `app.js` contains the new request fields through existing
route tests.
If browser-level tests are added later, cover:
- Changing the format select changes request JSON.
- Changing quality changes request JSON.
- A JPEG response displays in the gallery through a data URL.
## Edge Cases
### Output Path Has No Suffix
If the output path is:
```text
result
```
and the extension is `jpg`, write:
```text
result.jpg
```
### Output Path Has Different Suffix
If the output path is:
```text
result.png
```
and the extension is `avif`, write:
```text
result.avif
```
The configured path's suffix should not override the explicit format
setting.
### Batch Output
If batch size is 2, output path is `result.png`, and extension is `webp`,
write:
```text
result_0000.webp
result_0001.webp
```
### AVIF Unsupported Locally
If the local ImageMagick build lacks AVIF support, the user should see
the stderr message from ImageMagick. The application should not attempt
to detect delegate support ahead of time in the first implementation.
Running the conversion is simpler and produces the most accurate local
diagnostic.
### PNG Quality
The first implementation should ignore `output_quality` for PNG because
PNG remains on the existing Pillow path.
If PNG compression tuning becomes important later, add a separate
setting with PNG-specific semantics. Do not overload JPEG-style quality
without documenting the difference.
## Open Questions
### Should Compatibility APIs Force PNG?
No for the first implementation.
The compatibility endpoint currently returns base64 image strings
without MIME metadata. The first implementation should base64 encode the
compressed image bytes and assume the consumer can handle the actual
format. This keeps behavior simple and avoids special-casing output
format by route.
If a specific consumer later proves it assumes PNG, the relevant API
profile can force PNG or add a profile-specific compatibility option.
### Should Quality Default Be 90 or 95?
This design recommends `95` because it favors visual quality for image
generation output. Users who want smaller files can lower it in config
or the UI.
`90` would produce smaller files by default for lossy formats, but it
would also make the first opt-in conversion more visibly lossy. Since
the feature exposes quality directly, the higher default is the more
conservative choice.
### Should ImageMagick Be Checked At Server Startup?
No for the first implementation.
The server can run normally for PNG output without ImageMagick. Checking
at startup would create a warning or failure for users who never request
non-PNG output. The conversion path should report the missing executable
only when it is needed.
## Acceptance Criteria
The feature is complete when:
- Existing PNG generation still works without ImageMagick installed.
- CLI users can request `jpg`, `webp`, and `avif` output.
- Config users can set default output format and quality.
- Web UI users can select output format and quality.
- UI image responses include correct MIME types.
- ImageMagick missing or conversion failures are surfaced as clear
user-facing errors.
- Tests cover validation, command construction, route parsing, and
service integration.