CWL error when using OxoG: Tool definition failed validation

Tools and Workflows
1

cd /mnt/1TB/work/DockerTest-Miguel/tests/OxoG/DO52621/ && dockstore workflow launch --entry ICGC-TCGA-PanCancer/OxoG-Dockstore-Tools/pcawg-oxog-full-workflow:develop --json Dockstore.json
problems running command: cwltool --non-strict --print-pre /tmp/1502088182535-0/oxog_varbam_annotate_wf.cwl
stderr for command:
/usr/local/bin/cwltool 1.0.20161202203310
Resolved ‘/tmp/1502088182535-0/oxog_varbam_annotate_wf.cwl’ to 'file:///tmp/1502088182535-0/oxog_varbam_annotate_wf.cwl’
Tool definition failed validation:
While checking field steps
While checking object file:///tmp/1502088182535-0/oxog_varbam_annotate_wf.cwl#run_variant_bam_normal
Field run contains undefined reference to file:///tmp/1502088182535-0/Variantbam-for-dockstore/variantbam.cwl

java.lang.RuntimeException: problems running command: cwltool --non-strict --print-pre /tmp/1502088182535-0/oxog_varbam_annotate_w

2

solomon can probably shed more light on this but in the meantime, it looks like the file variantbam.cwl is just plain missing.
Can you tell me what you see when you ran git submodule update --init --recursive (and then tree)?

3

The message
Field run contains undefined reference to file:///tmp/1502088182535-0/Variantbam-for-dockstore/variantbam.cwl
makes it sound like the variantbam CWL is missing. In the directory where you pulled the code from github, could you tell me what is in the subdirectory Variantbam-for-dockstore? It should not be empty. If it is empty, you will need to clone the submodule reference. Try the steps that @dyuen mentioned above.

4

Hi,

I’m not sure where we are with this. Do you mean @solomon that I should try @dyuen steps? they don’t really apply to me I think; I’m using dockstore, I wouldn’t know when or how to update the git submodule. Can you guys try to resolve this and create a working dockstore entry? I’m wouldn’t know how to go about it!

Best

Miguel

5

Hi,
I’ve discussed with @SolomonShorser-OICR and I think the issue is that the OxoG entry uses git submodules, Dockstore doesn’t yet understand how to follow git submodule references, I’ve created an issue at https://github.com/ga4gh/dockstore/issues/868 to add this feature. In the meantime, there are two things I’d like to try.

  1. Clone the underlying repo and pull in submodules
    git clone https://github.com/ICGC-TCGA-PanCancer/OxoG-Dockstore-Tools.git
    cd OxoG-Dockstore-Tools
    git submodule update --init --recursive
  2. Launch locally with
    dockstore workflow launch --local-entry oxog_varbam_annotate_wf.cwl --json Dockstore.json
  3. If this fails, you can also try cwltool directly
    cwltool oxog_varbam_annotate_wf.cwl Dockstore.json

I think Solomon has provided you with a test JSON which has not been added directly yet

6

I’ll try this next week.

7

Hi,

What I got from Solomon was a Dockstore.json file but not the input files. I’ve tried to use the Dockstore.json file I produced for my test, but I got an error that I think is unrelated

ubuntu@ip-10-90-2-246:~/DockerTest-Miguel/scm/OxoG-Dockstore-Tools$ cwltool oxog_varbam_annotate_wf.cwl ../../tests/OxoG/DO52621/Dockstore.json 
/usr/local/bin/cwltool 1.0.20161202203310
Resolved 'oxog_varbam_annotate_wf.cwl' to 'file:///mnt/1TB/work/DockerTest-Miguel/scm/OxoG-Dockstore-Tools/oxog_varbam_annotate_wf.cwl'
Tool definition failed validation:
Validation error in object file:///mnt/1TB/work/DockerTest-Miguel/scm/OxoG-Dockstore-Tools/oxog_varbam_annotate_wf.cwl
  Could not validate `Workflow` because
    Workflow record could not validate extension field `dct:creator` because it is not recognized and strict is True.  Did you include a $schemas section?
8

This message should not break the workflow. Could you please try running with the --non-strict option? I can’t recall if dockstore CLI supports --non-strict, but cwltool does so if using the dockstore command doesn’t work, calling cwltool directly should work.

Another option that will be needed is --relax-path-checks - some of the reference files have filenames with characters that cwltool does not like. This option should fix that issue.

9

Oh right, one more thing: You will need reference files! The bundle to use is here (it’s big):

https://personal.broadinstitute.org/gsaksena/public_full8.tar.gz

It contains the reference file that is referred to by “refFile” and “refDataDir” in the input JSON

    "refFile": {
        "path": "/datastore/data/oxog_ref_data/full_refdata/public/Homo_sapiens_assembly19.fasta",
        "class": "File"
    }

    "refDataDir": {
        "class":"Directory",
        "path":"/datastore/oxog_refdata",
        "location":"/datastore/oxog_refdata"
    }