fix: Argo workflows joins failing with wide foreach flows

[saikonen]

Partial remedy to the issue with large foreach flows input-paths being repeated in the ARGO_TEMPLATE multiple times, exceeding the max size for it.

closes #1538

[saikonen]

I think we should proceed with alternative solution where we pass state through the configured datastore, instead of relying on Argo Workflow parameters, as even the fix in this PR does not solve the scaling issues completely due to the way Argo materializes the parameter values in the ARGO_TEMPLATE env var, and the input parameters for a foreach join being arbitrarily large.

The current fix relies on reading the template json from environment, but as there are proposals such as argoproj/argo-workflows#12325 already in the works, this might break with future versions of Argo.

[shrinandj]

This looks good to me!

[rohanrebello]

I've tested that this PR works for up to 3000-way foreach's on AWS

[tslott]

Can this be merged?

[saikonen]

have to close this PR as a dead end due to the direction that Argo Workflows took with the new handling of oversized ARGO_TEMPLATE where they offload it to disk as a ConfigMap, making it not parseable in the way I had in mind in this.

This PR being only a partial fix at best lead me to a different approach, one which should be in review now and hopefully a proper solution to the issue at hand: #1704