Recently created workspaces are lost when machine is drastically closed

[lethevimlet]

Recently created workspaces are lost when machine is drastically closed or docker che-server container is killed, if never stopped properly at least once after the workspace creation.
Although docker workspace containers are created and persist after reboot eclipse che dashboard is unable to recover them if you don't stop che properly at least once after the workspace creation.

Reproduction Steps:

Using Ubuntu 16.04 and Docker 1.12.1

1. Install eclipse che docker container as described here https://www.eclipse.org/che/getting-started/download/
2. Access dashboard and create a new workspace
3. Immediately after workspace has been created and run for the fist time, reboot the machine, or docker kill che-server-container-id
4. docker start che-server-container-id and access the dashboard, you will find no workspace attached to the running che-server.

Expected behavior:

Eclipse Che should be able to restore workspaces and write the configuration to workspaces.json

Observed behavior:

You can check that workspace container was created as expected with docker ps -a or by listing ls /home/user/che/workspaces
If you check /home/user/che/storage/workspaces.json is empty.

If I repeat the the process and before rebooting I do a che stop, everything gets saved as expected, in fact from then the workspaces that where created before the che stop are correctly displayed no matter if I force close the machine or kill che container, although the problem persists for the recently created workspaces.

Che version: codenvy/che-server:latest
OS and version: Ubuntu Server x64 16.0.4 LTS
Docker version: 1.12.1
Che install: Docker

Additional information:

I tried to run che-server container as root and as a user in docker group
I found that this does not occurs when deploying eclipse che server as standalone server, without being in a docker container.

[ddementieva]

@josevimlet It's an expected behavior. Che server should be correctly shut down. When you kill Che server container Tomcat that is running inside fails to save data.

[lethevimlet]

Humm, so lets say I create a project file inside a workspace, having che server stopped at least once before, wait a few seconds so auto-save do its thing and then I force close the machine.
That file will get restored without problem next time I start eclipse che as expected, but this behaviour does not extend to workspace creation itself, it makes little sense to me and I'm not sure why tomcat shouldn't be able to handle a simple json file write properly since I'm not closing the machine asap, I'm giving minutes to handle it before force closing, but I believe there must be a good reason behind.

Is there a way I can set an autosave workspace or something similar, so I don't have to manually close and start my workspaces to ensure I don't lost my setting, I know I can add a workspace with the same name and It will get reattached but I find thins annoying at least.

[ghost]

@josevimlet you do not need to manually stop a workspace. But there's a need to correctly stop the server itself.

[lethevimlet]

@eivantsov sorry that's what I meant, stopping che server container at least once, I updated the comment so it reflects that, thanks for pointing that out ^^

[ghost]

@josevimlet if you use Che cli you may correctly stop the container. Or stop it using docker cli.

[lethevimlet]

@eivantsov That's does not really address the issue, still is manually stoping the che-server to enforce workspaces.json getting saved, and accessing the machine console as root or docker user which is exactly the oposite your are looking when working on a web based IDE.

Doesn't having to ssh to the machine to save workspace really defeats the point of having a web environment on the first place?

Don't get me wrong I love the idea of working on a web app, but there's a real issue to address with that workspaces.json file save behaviour and the solution should not be manually saving it by stopping the service, workspace.json file should be saved/updated each time a workspace is created or modified.

[ghost]

@josevimlet what's your Che version?

Currently we have an in memory DB (no jsons) and the info whould be saved immediately.

[TylerJewell]

@eivantsov - the request is legitimate. It seems that the events that trigger an update / persistence of the in-memory configuration to disk is not aligned with user expectations.

[ghost]

@TylerJewell What I am trying to say is that the current Che does not have this issue.

[TylerJewell]

Oh I see - you are saying that the JPA merge in 5.0.0-M3 eliminates this problem? If so, then that is great - I didn't know the JPA change simplified this scenario.

[ghost]

@TylerJewell I haven't checked but asked around. DB should be updated immediately after workspace creation.

[lethevimlet]

Are those changes available on the latest nightly as a docker container? I would love to give it a test.

[TylerJewell]

Yeah, nightly is on M5-SNAPSHOT now.

[lethevimlet]

According to docs:
docker run --rm -t -v /var/run/docker.sock:/var/run/docker.sock eclipse/che:nightly start
should pull the latest nightly but I won't find the docker image, how can I do get running the latest che server nightly so I can test if the issue persist.
Is there a separated nightly repo?

Thanks in advance.

[TylerJewell]

That pulls the nightly of the che-launcher. It does not pull the latest version of Che. Please, look at setting CHE_VERSION environment variable. The way you set it is slightly different if you are using the docker syntax vs. the CLI.

Docker: https://eclipse-che.readme.io/docs/usage-docker#versions-and-nightly-builds
CLI: https://eclipse-che.readme.io/docs/configuration#configure-che-using-the-cli

[lethevimlet]

thanks ill have a look, and test it asap.

[lethevimlet]

Tested the nightly, the good news seems that in fact workspaces.json gets saved as expected, which fixes the issue, but the problem is the nightly crashes when creating the workspace.
So, ill have to wait until the JPA changes (5.0 M3) are merged on the latest stable release, any chances of seeing this on 4.7.1 or 4.7.2 or ill have to wait until 5.0?

[TylerJewell]

If you are sering an issueplease provide a complete list of reproduction steps. Our internal systems are showing everything as stable.

It is not possible to backport any finctionality to the 4.x line.

[lethevimlet]

Ok, I'll make a clean install so I can discard that any of the old config files are messing with nightly and test workspace creation, If everything goes as expected I'll write it here and If I see any problem i'll open a detailed issue for that.

[JamesDrummond]

@josevimlet You should wait until #2670 is fixed before using the nightly again.

[TylerJewell]

Yes, @JamesDrummond is correct - there is a confirmed issue uncovered today with the latest M3 release where the new JPA storage is not being done in the proper place, so workspaces are not being recovered.

[lethevimlet]

Tested the latest nightly and everything seems to be working fine ^_^

[TylerJewell]

Whew!