Integer overflows in the damage shaper

[YaLTeR]

I've no idea how the user ended up with a 1920×35791423 px large window (apparently was TF2) or how their GPU succeeded in allocating dmabufs this big, but I guess it wouldn't hurt adding checks or saturating ops into the shaper.

 9月 13 16:19:02 yun niri[642501]: 2025-09-13T19:19:02.041265Z DEBUG niri::pw_utils: pw stream: got format = VideoInfoRaw { format: VideoFormat::BGRA, flags: VideoFlags(0x4), modifier: 144115188080057091, size: spa_rectangle { width: 1920, height: 35791423 }, framerate: spa_fraction { num: 0, denom: 1 }, max_framerate: spa_fraction { num: 60, denom: 1 }, views: 0, interlace_mode: 0, pixel_aspect_ratio: spa_fraction { num: 0, denom: 0 }, multiview_mode: 0, multiview_flags: 0, chroma_site: 0, color_range: 0, color_matrix: 0, transfer_function: 0, color_primaries: 0 } stream_id=2
 9月 13 16:19:02 yun niri[642501]: 2025-09-13T19:19:02.041279Z DEBUG niri::pw_utils: pw stream: moving to ready state stream_id=2
 9月 13 16:19:02 yun niri[642501]: 2025-09-13T19:19:02.061657Z DEBUG niri::pw_utils: pw stream: state changed: Paused -> Streaming stream_id=2
 9月 13 16:19:02 yun niri[642501]: thread 'main' panicked at /build/cargo-vendor-dir/smithay-0.7.0/src/backend/renderer/damage/shaper.rs:84:37:
 9月 13 16:19:02 yun niri[642501]: attempt to multiply with overflow
 9月 13 16:19:02 yun niri[642501]: stack backtrace:
 9月 13 16:19:02 yun niri[642501]:    0: __rustc::rust_begin_unwind
 9月 13 16:19:02 yun niri[642501]:    1: core::panicking::panic_fmt
 9月 13 16:19:02 yun niri[642501]:    2: core::panicking::panic_const::panic_const_mul_overflow
 9月 13 16:19:02 yun niri[642501]:    3: smithay::backend::renderer::damage::shaper::DamageShaper<_>::shape_damage_impl
 9月 13 16:19:02 yun niri[642501]:    4: smithay::backend::renderer::damage::OutputDamageTracker::damage_output
 9月 13 16:19:02 yun niri[642501]:    5: niri::pw_utils::Cast::dequeue_buffer_and_render
 9月 13 16:19:02 yun niri[642501]:    6: niri::niri::Niri::render_windows_for_screen_cast
 9月 13 16:19:02 yun niri[642501]:    7: niri::backend::Backend::with_primary_renderer
 9月 13 16:19:02 yun niri[642501]:    8: niri::niri::Niri::redraw_queued_outputs
 9月 13 16:19:02 yun niri[642501]:    9: niri::niri::State::refresh_and_flush_clients
 9月 13 16:19:02 yun niri[642501]:   10: calloop::loop_logic::EventLoop<Data>::run
 9月 13 16:19:02 yun niri[642501]:   11: niri::main

This particular overflow is here:

smithay/src/backend/renderer/damage/shaper.rs

Lines 83 to 89 in eb45814

	// Damage the current bounding box when there's a damage rect covering near all the area.
	if max_damage_area as f32 / (damage_bbox.size.w * damage_bbox.size.h) as f32
	> MAX_DAMAGE_TO_DAMAGE_BBOX_RATIO
	{
	self.out_damage.push(damage_bbox);
	return;
	}