Recovery after panic

[farmaazon]

Do you have any ideas how to try to recover app after a panic?

Of course I know the simplest solution of having kind of a watchdog script restoring entire process, but sometimes it's practical to read "last valid state" from memory.

So here is how I've tried:

// various imports 

#[derive(Default, Deref)]
pub struct Snapshot(Cell<Option<Game>>);

thread_local! {
    pub static CRASH_INFO: Cell<Option<String>> = Default::default();
}

fn initialize(
    snapshot: Rc<Snapshot>,
    crash_info: Option<String>,
) -> Result<(ui::Main, Rc<Handler>)> {
    let ui = ui::Main::new()?;
    let weak_ui = ui.as_weak();
    // Some initialization stuff
    if let Some(panic) = crash_info {
        ui.global::<ui::Error>().invoke_report_error(
            format!("Application crashed!. Recovered last known valid state.\n\n {panic}").into(),
        );
    }
    let handler = Handler::initialize(ui.clone_strong(), snapshot.take(), snapshot);
    ui.show()?;
    Ok((ui, handler))
}

fn main() {
    if let Err(error) = simple_logger::SimpleLogger::new().init() {
        eprintln!("Failed to initialize logger. {error}. Probably no logs will be reported.");
    }

    let std_panic_hook = panic::take_hook();
    panic::set_hook(Box::new(move |panic| {
        std_panic_hook(panic);
        CRASH_INFO.with(|info| info.set(Some(panic.to_string())));
    }));

    let snapshot = Rc::new(Snapshot::default());
    loop {
        let last_crash = CRASH_INFO.with(|info| info.take());
        let (ui, _handler) =
            initialize(snapshot.clone(), last_crash).expect("Failed to initialize application");
        let loop_result = panic::catch_unwind(slint::run_event_loop);
        ui.hide().expect("Failed to tear down application");
        match loop_result {
            Ok(result) => {
                result.expect("Failed to run event loop");
                break;
            }
            Err(_) => {
                log::error!("Trying to restore application after crash");
            }
        }
    }
}

The I preferred restoring all slint structures (generated in ui module) to avoid any possibility of inconsistent state left after panic. Unfortunately, the second run always fails:

1. winit backend panics when calling show, saying that creating multiple event loops is not supported.
2. qt running event loop again repeatedly panics with message 'Recursion in timer code'.

I guess what I do is not quite supported. My question is: do you have any idea how can achieve something similar?

[hunger]

I doubt this can be done in a way that is even vaguely reliable. First off you can make rust just abort on panic and not even try to unwind the stack. Secondly it is not flly documented waht happens in edge cases during unwinding (like triggering a second panic during the unwind process, etc.).

Even if you manage to rn all the necessary Drop traits for everythign: One typical reason to panic is running out of memory. Rebuilding the entire UI from scratch in that case does not sound like a good idea -- as that will just panic again when trying to allocate memory for the UI elements.

I think the only reliable way to recover a process after a panic is from another process.

[farmaazon]

Thanks for the answer!

Regarding reliability: I wanted a simple approach, to just be able to recover where possible. I haven't aimed to catch every possible panic. I think it would work enough for my needs if not for some statics vars left in a broken state I cannot clean up.

For the case with allocating memory, I just assumed most of those allocations happens before I run event loop, so the app will just crash "normally", without entering infinite loop. I was also ready to give up recovery when panic happens after ~2 seconds or so.