ies restart with covariance reinflation

[cnicol-gwlogic]

When restarting an ies run that uses covariance reinflation, I have noticed that the mean-shifted ensemble size is reduced to the size of the restart obs ensemble - which will be smaller than the iteration 0 parameter ensemble due to failed model runs / realisations. I am using the following keywords / inputs for the restart:

ies_parameter_ensemble         pest/cal09_ies_v10r1.x.manualmeanshift.par.jcb <--I made this so I had the full iteration 0 parameter set of 1000 realisations, but mean-shifted to current best mean param set.
ies_observation_ensemble       pest/cal09_ies_v10r1.obs+noise.jcb <--this is my initial run's obs/noise. I just wanted noise to be identical.
ies_restart_observation_ensemble pest/cal09_ies_v10r2a.0.obs.backup.jcb <--this is say ~900 realz of model "obs" outputs for the restart, so upgrades can begin right away. (my base real failed randomly, but I knew it should not have, so reran and made this new jcb with base obs in it)

Is there a reason why the original full iteration 0 parameter ensemble can't be mean-shifted, thereby reinflating to the full original ies_num_realz? It seems a waste / problematic to reduce it unnecessarily. Or am I using the restart keywords incorrectly?

I've tried a few different restart keyword arrangements, and this mean-shifted ensemble size cull seems to be a consistent behaviour. Eg, spec'ing ies_restart_parameter_ensemble (at say ~900 realz), and ies_parameter_ensemble (at the full 1000 realz). I guess in the case above I should have just ditched ies_restart_observation_ensemble and bitten that rerun bullet to get the full 1000 realz.

Tips appreciated.

[jtwhite79]

As of v5.2.17, ies will mean shift the full prior parameter ensemble, so even runs that have failed during the first cycle of iterations get another chance (Ive found that a lot more runs will succeed after the mean shift than survived the initial prior). But I think your case, since ~100 runs are missing from the restart ensemble, then ies doesnt know what the sim values for those ~100 failed runs, so it has to try to align the par ensemble with the restart ensemble so that you can jump straight to the upgrade calcs. I guess it would be possible to have ies try to run those missing 100 reals...just have to add that functionality.

[cnicol-gwlogic]

Thanks for the details. No need for that added functionality as those runs would typically just fail again (sans random crap that I should really resolve); I've just restarted it without the restart obs ensemble.

To be clear, I meant is there a reason in my case above (with n_iter_mean(3)) why on the next reinflation at iter4 it does not mean-shift the full 1000 realz - it is limited in my case to the size / realz of the restart_obs_ensemble (even in 5.2.17). I'm ok with the reduced ensemble in iters 1-3 (failed runs are failed runs) - but I thought it could use the mean from iter3 to shift ies_parameter_ensemble for iter4, not a trimmed down version of that.

Anyway, so it sounds like the gist is:

• don't use ies_restart_observation_ensemble if you want the full prior reinflate set,
• make sure you provide the full original prior or a full mean-shifted param ensemble via ies_parameter_ensemble, and/or the full mean-shifted param ensemble via ies_restart_parameter_ensemble.

I have noticed that too re: less fails after the first mean shift; makes sense hey. I'm using 5.217 / playing with some of those new reinflate options; excellent ideas - ta.

[jtwhite79]

I'm keen any ideas or suggestions with reinflation...

[cnicol-gwlogic]

Since you asked: is this resizing (down) pe_base (I think the original prior parameter ensemble) on provision of a reduced size restart parameter ensemble? I'm thinking that is where my reduced-ensemble size reinflation issue might come from if so (which I clarified above in an update to my comment). Can we avoid that if this is the case?

[jtwhite79]

I see what you are wanting to do now. Sorry I'm slow...I can make that happen.

[cnicol-gwlogic]

haha - more like garbled messaging on my part. No stress.