The `dockstore tool launch` failed with a WDL to run a docker container on Docker on Fedora 36

junaruga · September 6, 2022, 6:02pm

I installed the Docker on Fedora 36, as I couldn’t run the dockstore cli with Podman: Dockstore-cli: Support Podman (alternatrive Docker) or workaround? . The Docker installation steps are here.

Then I can run the Docker container (e.g. ubuntu) from my host Fedora.

$ which docker
/bin/docker

$ docker version
Client:
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.18.4
 Git commit:        aa7e414
 Built:             Tue Jul 19 21:40:23 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.4
  Git commit:       f756502
  Built:            Tue Jul 19 21:40:23 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        
 runc:
  Version:          1.1.3
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit: 

$ docker run --rm -t docker.io/ubuntu:latest cat /etc/os-release | grep ID
VERSION_ID="22.04"
ID=ubuntu
ID_LIKE=debian
$ echo $?
0

But the dockstore tool launch ... with a WDL file failed.
Here is the full log. Here is the used WDL file. Any idea to fix this issue? Thanks

$ java --version
openjdk 17.0.4 2022-07-19
OpenJDK Runtime Environment (Red_Hat-17.0.4.0.8-1.fc36) (build 17.0.4+8)
OpenJDK 64-Bit Server VM (Red_Hat-17.0.4.0.8-1.fc36) (build 17.0.4+8, mixed mode, sharing)

$ dockstore --version
Dockstore version 1.13.0-rc.0
The latest stable version is 1.12.0
You are currently on the latest unstable version. If you wish to upgrade to the latest stable version, please use the following command:
   dockstore --upgrade-stable

...
dockstore tool launch \
	--local-entry "bio-diversity-genomics-garg.wdl" \
	--json "bio-diversity-genomics-garg.inputs.json"
...
19:51:12.382 [main] ERROR io.dockstore.common.Utilities - problems running command: java -jar /home/jaruga/.dockstore/libraries/cromwell-77.jar run --inputs /tmp/foo4850486516087678844json /home/jaruga/git/GargGroup/BioDivGenomics/bio-diversity-genomics-garg.wdl
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
	at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
	at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
	at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
	at java.base/java.lang.Thread.run(Thread.java:833)
19:51:12.386 [main] ERROR io.dockstore.client.cli.nested.BaseLanguageClient - Problem running launcherCromwell: 
java.lang.IllegalStateException: problems running command: {}java -jar /home/jaruga/.dockstore/libraries/cromwell-77.jar run --inputs /tmp/foo4850486516087678844json /home/jaruga/git/GargGroup/BioDivGenomics/bio-diversity-genomics-garg.wdl
	at io.dockstore.common.Utilities.executeCommand(Utilities.java:192)
	at io.dockstore.common.Utilities.executeCommand(Utilities.java:107)
	at io.dockstore.client.cli.nested.BaseLauncher.executeEntry(BaseLauncher.java:121)
	at io.dockstore.client.cli.nested.BaseLanguageClient.commonExecutionCode(BaseLanguageClient.java:429)
	at io.github.collaboratory.wdl.WDLClient.executeEntry(WDLClient.java:144)
	at io.dockstore.client.cli.nested.BaseLanguageClient.launchPipeline(BaseLanguageClient.java:204)
	at io.github.collaboratory.wdl.WDLClient.launch(WDLClient.java:72)
	at io.dockstore.client.cli.nested.AbstractEntryClient.launchWdl(AbstractEntryClient.java:1470)
	at io.dockstore.client.cli.nested.AbstractEntryClient.checkEntryFile(AbstractEntryClient.java:913)
	at io.dockstore.client.cli.nested.AbstractEntryClient.launch(AbstractEntryClient.java:1378)
	at io.dockstore.client.cli.nested.AbstractEntryClient.processEntryCommands(AbstractEntryClient.java:312)
	at io.dockstore.client.cli.Client.run(Client.java:737)
	at io.dockstore.client.cli.Client.main(Client.java:633)
Caused by: org.apache.commons.exec.ExecuteException: problems running command: {}java -jar /home/jaruga/.dockstore/libraries/cromwell-77.jar run --inputs /tmp/foo4850486516087678844json /home/jaruga/git/GargGroup/BioDivGenomics/bio-diversity-genomics-garg.wdl (Exit value: 1)
	at io.dockstore.common.Utilities.executeCommand(Utilities.java:188)
	... 12 common frames omitted
Cromwell exit code: 1
...

junaruga · September 6, 2022, 8:29pm

I found the cause of the error. This error happens when the SE Linux is enforcing on Fedora 36.

Edited: Here is the upstream issue ticket I opened.

github.com/broadinstitute/cromwell

The `java -jar cromwell-77.jar run foo.wdl` with `docker: ..` fails on enabled SE Linux.

opened 09:03PM - 06 Sep 22 UTC

junaruga

## My environment OS: Fedora 36 SE Linux mode: enforcing ``` $ java --version openjdk 17.0.4 2022-07-19 OpenJDK Runtime Environment (Red_Hat-17.0.4.0.8-1.fc36) (build 17.0.4+8) OpenJDK 64-Bit Server VM (Red_Hat-17.0.4.0.8-1.fc36) (build 17.0.4+8, mixed mode, sharing) $ java -jar /home/jaruga/.dockstore/libraries/cromwell-77.jar --version cromwell 77 $ docker --version Docker version 20.10.17, build aa7e414 $ sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: enforcing Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Memory protection checking: actual (secure) Max kernel policy version: 33 ``` ## Summary I prepared the reproducer [here](https://github.com/junaruga/dockstore-cli-docker-test). The command fails with the WDL file including Docker container when the SE Linux is enablerd. ``` $ java -jar /path/to/cromwell-77.jar run /path/to/hello-docker.wdl ... Workflow 3d200d74-77eb-45d8-8fe6-b41b14aab6f6 transitioned to state Failed ``` Then run the `script.submit` file again. The `docker run ...` exists with "Permission denied". ``` $ bash -x cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script.submit + rm -f /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid + docker run --cidfile /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid -i --entrypoint /bin/bash -v /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:delegated ubuntu@sha256:2d7ecc9c5e08953d586a6e50c29b91479a48f69ac1ba1f9dc0420d18a728dfc5 /cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script /bin/bash: /cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script: Permission denied +++ cat /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid ++ docker wait 021d88f941ea0a813e0ea756700f59a065a6aac90cc8628cd062f4047b3ea84d + rc=126 ++ cat /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid + docker rm 021d88f941ea0a813e0ea756700f59a065a6aac90cc8628cd062f4047b3ea84d 021d88f941ea0a813e0ea756700f59a065a6aac90cc8628cd062f4047b3ea84d + exit 126 ``` This is related to the docker run volume mount option requiring `:z` or `:Z`. > https://docs.docker.com/engine/reference/commandline/run/ > Labeling systems like SELinux require that proper labels are placed on volume content mounted into a container. Without a label, the security system might prevent the processes running inside the container from using the content. By default, Docker does not change the labels set by the OS. > > To change the label in the container context, you can add either of two suffixes :z or :Z to the volume mount. These suffixes tell Docker to relabel file objects on the shared volumes. The z option tells Docker that two containers share the volume content. As a result, Docker labels the content with a shared content label. Shared volume labels allow all containers to read/write content. The Z option tells Docker to label the content with a private unshared label. Only the current container can use a private volume. Now I replaced the suffixes ":delegated" with `:z`. ``` $ git diff diff --git a/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script.submit b/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script.submit index 063b374..f37a650 100644 --- a/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script.submit +++ b/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script.submit @@ -7,7 +7,7 @@ docker run \ -i \ \ --entrypoint /bin/bash \ - -v /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:delegated \ + -v /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:z \ ubuntu@sha256:2d7ecc9c5e08953d586a6e50c29b91479a48f69ac1ba1f9dc0420d18a728dfc5 /cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script # get the return code (working even if the container was detached) ``` And the command above succeeded. ``` $ bash -x cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script.submit + rm -f /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid + docker run --cidfile /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid -i --entrypoint /bin/bash -v /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:z ubuntu@sha256:2d7ecc9c5e08953d586a6e50c29b91479a48f69ac1ba1f9dc0420d18a728dfc5 /cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script Hello Docker +++ cat /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid ++ docker wait 56df02a84010fae2573793c374e4ae3c506dc5c9a759169d09c378d0dcdfaa0d + rc=0 ++ cat /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid + docker rm 56df02a84010fae2573793c374e4ae3c506dc5c9a759169d09c378d0dcdfaa0d 56df02a84010fae2573793c374e4ae3c506dc5c9a759169d09c378d0dcdfaa0d + exit 0 ``` ## Workaround I was able to suppress this error by disabling SE Linux on Fedora 36. But this is not an ideal way. Because I think SE Linux is enabled (Current mode: enforcing) as a default on RPM-based distributions such as Fedora, RHEL, CentOS stream and etc for security reasons. ``` $ sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: enforcing Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Memory protection checking: actual (secure) Max kernel policy version: 33 ``` Disabled like this. ``` $ sudo setenforce 0 $ sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: permissive Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Memory protection checking: actual (secure) Max kernel policy version: 33 ``` I guess you can try [SE Linux on Ubuntu](https://wiki.ubuntu.com/SELinux) to reproduce this error.