The `dockstore tool launch` failed with a WDL to run a docker container on Docker on Fedora 36

junaruga · September 6, 2022, 8:29pm

I found the cause of the error. This error happens when the SE Linux is enforcing on Fedora 36.

Edited: Here is the upstream issue ticket I opened.

github.com/broadinstitute/cromwell

The `java -jar cromwell-77.jar run foo.wdl` with `docker: ..` fails on enabled SE Linux.

opened 09:03PM - 06 Sep 22 UTC

closed 03:44PM - 13 Sep 23 UTC

junaruga

## My environment OS: Fedora 36 SE Linux mode: enforcing ``` $ java --ve…rsion openjdk 17.0.4 2022-07-19 OpenJDK Runtime Environment (Red_Hat-17.0.4.0.8-1.fc36) (build 17.0.4+8) OpenJDK 64-Bit Server VM (Red_Hat-17.0.4.0.8-1.fc36) (build 17.0.4+8, mixed mode, sharing) $ java -jar /home/jaruga/.dockstore/libraries/cromwell-77.jar --version cromwell 77 $ docker --version Docker version 20.10.17, build aa7e414 $ sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: enforcing Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Memory protection checking: actual (secure) Max kernel policy version: 33 ``` ## Summary I prepared the reproducer [here](https://github.com/junaruga/dockstore-cli-docker-test). The command fails with the WDL file including Docker container when the SE Linux is enablerd. ``` $ java -jar /path/to/cromwell-77.jar run /path/to/hello-docker.wdl ... Workflow 3d200d74-77eb-45d8-8fe6-b41b14aab6f6 transitioned to state Failed ``` Then run the `script.submit` file again. The `docker run ...` exists with "Permission denied". ``` $ bash -x cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script.submit + rm -f /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid + docker run --cidfile /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid -i --entrypoint /bin/bash -v /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:delegated ubuntu@sha256:2d7ecc9c5e08953d586a6e50c29b91479a48f69ac1ba1f9dc0420d18a728dfc5 /cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script /bin/bash: /cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script: Permission denied +++ cat /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid ++ docker wait 021d88f941ea0a813e0ea756700f59a065a6aac90cc8628cd062f4047b3ea84d + rc=126 ++ cat /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid + docker rm 021d88f941ea0a813e0ea756700f59a065a6aac90cc8628cd062f4047b3ea84d 021d88f941ea0a813e0ea756700f59a065a6aac90cc8628cd062f4047b3ea84d + exit 126 ``` This is related to the docker run volume mount option requiring `:z` or `:Z`. > https://docs.docker.com/engine/reference/commandline/run/ > Labeling systems like SELinux require that proper labels are placed on volume content mounted into a container. Without a label, the security system might prevent the processes running inside the container from using the content. By default, Docker does not change the labels set by the OS. > > To change the label in the container context, you can add either of two suffixes :z or :Z to the volume mount. These suffixes tell Docker to relabel file objects on the shared volumes. The z option tells Docker that two containers share the volume content. As a result, Docker labels the content with a shared content label. Shared volume labels allow all containers to read/write content. The Z option tells Docker to label the content with a private unshared label. Only the current container can use a private volume. Now I replaced the suffixes ":delegated" with `:z`. ``` $ git diff diff --git a/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script.submit b/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script.submit index 063b374..f37a650 100644 --- a/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script.submit +++ b/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script.submit @@ -7,7 +7,7 @@ docker run \ -i \ \ --entrypoint /bin/bash \ - -v /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:delegated \ + -v /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:z \ ubuntu@sha256:2d7ecc9c5e08953d586a6e50c29b91479a48f69ac1ba1f9dc0420d18a728dfc5 /cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script # get the return code (working even if the container was detached) ``` And the command above succeeded. ``` $ bash -x cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script.submit + rm -f /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid + docker run --cidfile /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid -i --entrypoint /bin/bash -v /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting:z ubuntu@sha256:2d7ecc9c5e08953d586a6e50c29b91479a48f69ac1ba1f9dc0420d18a728dfc5 /cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/script Hello Docker +++ cat /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid ++ docker wait 56df02a84010fae2573793c374e4ae3c506dc5c9a759169d09c378d0dcdfaa0d + rc=0 ++ cat /home/jaruga/git/dockstore-cli-docker-test/cromwell-executions/HelloWorldWithDocker/3d200d74-77eb-45d8-8fe6-b41b14aab6f6/call-WriteGreeting/execution/docker_cid + docker rm 56df02a84010fae2573793c374e4ae3c506dc5c9a759169d09c378d0dcdfaa0d 56df02a84010fae2573793c374e4ae3c506dc5c9a759169d09c378d0dcdfaa0d + exit 0 ``` ## Workaround I was able to suppress this error by disabling SE Linux on Fedora 36. But this is not an ideal way. Because I think SE Linux is enabled (Current mode: enforcing) as a default on RPM-based distributions such as Fedora, RHEL, CentOS stream and etc for security reasons. ``` $ sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: enforcing Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Memory protection checking: actual (secure) Max kernel policy version: 33 ``` Disabled like this. ``` $ sudo setenforce 0 $ sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: permissive Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Memory protection checking: actual (secure) Max kernel policy version: 33 ``` I think you can try [SE Linux on Ubuntu](https://wiki.ubuntu.com/SELinux) to reproduce this error if you only have Debian-based Linux.