Managing workflow versions if repository contains multiple workflows

Dockstore
1

Hi there,

we would like to create a github repository that will host multiple workflows. I understand that versions are typically taken from branches or github tags, but would it be possible to read the workflow version either form the workflow document itself, or to indicate the version of a workflow in the .dockstore.yml file ?

Thanks for your help,
Marius

2

Hi,
For this issue weā€™re testing a new feature detailed at https://github.com/dockstore/dockstore/issues/3695

Please let us know whether that would work for you, thanks!

3

Thanks @dyuen,

that might work, as I understand it we could skip creating versions for some tags or branches on a per workflow basis, which is cool. That does seems quite clunky if you have to exclude everything for all workflows except the one weā€™re now going to create/update.

I think what would be ideal for us if we could list something like

workflows:
- name: 'COVID-19 variation analysis on Illumina metagenomic data'
  primaryDescriptorPath: workflows/sars-cov-2-variant-calling/workflow.ga
  version: 0.1.1
- name: 'Some unrelated workflow'
  primaryDescriptorPath: workflows/topic/another_path.ga
  version: 1.1

But maybe the github app workflows isnā€™t the right one for this scenario and we should POST zip files as in https://docs.dockstore.org/en/develop/advanced-topics/posting-zips.html?
If thatā€™s the case I have a follow-up since that works fine for wdl example but fails for Galaxy workflows.
Thatā€™s the .dockstore.yml (which does look different from the rest of the manual, maybe this needs to be updated ?)

dockstoreVersion: 1.0
class: workflow
primaryDescriptor: workflow.ga

zipping that up:

zip firstversion.zip workflow.ga .dockstore.yml

trying the post:

 curl -X POST "https://dockstore.org/api/workflows/hostedEntry/${WORKFLOW_ID}" -H "accept: application/json" -H "Authorization: Bearer ${DOCKSTORE_TOKEN}" -H "Content-Type: multipart/form-data" -F "file=@firstversion.zip;type=application/zip"

Error reading request%

WORKFLOW_ID here is 15818, which is an empty Galaxy Galaxy workflow I created in the UI (I also created one via the API, same result).

I guess we could have a single repository to keep contribution simple and create or update otherwise read-only repos from the master branch. I think Iā€™m going to go that route but it does seem like a bit of a hassle. On the upside that would mean we run tests before upload.

4

that might work, as I understand it we could skip creating versions for some tags or branches on a per workflow basis, which is cool.

I think we actually went the other way during implementation, we decided that a branch or tag would need to be included to be updated. Hopefully thatā€™s a bit less clunky

  • if filters: is not included, donā€™t filter out anything
  • if filters: is included, only include branchs/tags matching the specified filters

Thatā€™s the .dockstore.yml (which does look different from the rest of the manual, maybe this needs to be updated ?)

Yes, I think that .dockstore.yml is marked as version 1.0 whereas the current format is 1.2. Weā€™ll get that updated. I also would not recommend that API because weā€™ve marked that (at the top of the article) as deprecated

5

OK, yes, I think this is a bit easier to use, but version management does still seem to be a bit unfortunate
as compared to listing explicit workflow versions (or a one-workflow-per repo setup).

Correct me if Iā€™m wrong, but I guess this would be the workflow weā€™d have to use:
Create a tag with version 1.0 and include each workflow that reached that version. When another workflow reaches version 1.0 Iā€™d have to drop the 1.0 tag and re-create it with the additional workflow included.

Maybe explicit versioning in .dockstore.yml doesnā€™t work conceptually because dockstore isnā€™t hosting the workflows if they are registered through the github application process ?

Weā€™re discussing the layout now, just trying to see which repo strategy we will choose ultimately (https://docs.dockstore.org/en/develop/advanced-topics/best-practices/best-practices-dockstore.html#structuring-and-organizing-your-git-repositories is very helpful in that regard)

6

I think the reason why weā€™re not including versions in the .dockstore.yml is that we go off of the tags and branches we receive from github. If we also saw version numbers in the .dockstore.yml then it could get complex to resolve disagreements if that makes sense. Instead, we see what tags and branches we receive and the proposed feature (implemented in our development server) decides which ones to keep.

For your case, it does sound like one workflow would need to ā€œskipā€ tags if a tag 1.0 was applicable to Workflow A but was not applicable to workflow B. Then a tag 2.0 might be applicable to Workflow B only.

1 Like