Dockstore automatic sync stopped working for new tags

Dockstore
1

Our repository GitHub - broadinstitute/gatk: Official code repository for GATK versions 4 and up has had a .dockstore.yml that synced perfectly fine for many years (gatk/.dockstore.yml at master · broadinstitute/gatk · GitHub), and the app is fully authorized on that repo. Then at some point, new tagged releases stopped syncing for our published workflows. Branches still sync fine and are 100% up-to-date, but new tags don’t show up for our published workflows when we do a release. What could be causing this? Any suggestions on how I could fix it with minimal disruption to users of our workflows?

Thanks,
David

2

Hi,

I took a look, I think this was partly intentional and partly a miscommunication.
It looks like this PR Added filter for only master branch by bshifaw · Pull Request #7217 · broadinstitute/gatk · GitHub filtered the .dockstore.yml so that we only pay attention to updates on the master branch (intentional), not other branches or tags (the miscommunication).

So looking at Dockstore or Dockstore , there’s a gap between late 2021 and May 2023. Looking at branches like “tb_recalibrate_gq” it looks like that filter setting took a while to migrate to the various branches which is why we didn’t stop processing updates for branches that we not master instantly after the PR in May 14, 2021, but only after that .dockstore.yml had spread to all branches with updates.

Looking at the examples ( Templates — Dockstore documentation ) , if the intent was to only process tags and the master branch, something like the following may be worth a try

filters:
    branches:
        - master
    tags:
        - /.*/
1 Like
3

Thanks dyuen for this incredibly helpful reply! That certainly explains it – we’ve opened a PR to implement your suggested fix (added tags to dockstore.yaml by tmelman · Pull Request #8323 · broadinstitute/gatk · GitHub). Do you know of an easy way to fill in the missing tags from 2021-2023 for our workflows?

Regards,
David

4

No problem,

I may need to brainstorm with the team but there are two problems, one which is solvable by using the API:

a) We basically need to workaround the github app machinery since the events are long and buried. There is an API call Swagger UI that could be called the 8 or 9(do tags in this repo always occur on master?) releases since the original PR was merged.

b) Once we re-create the event and call the API, Dockstore will dutifully pull the .dockstore.yml from that version and ignore it since the .dockstore.yml at the time said to ignore it.

Here, I think we need some brainstorming for options.
One thought, for some of the releases like 4.2.6.1, would creating a 4.2.6.2 with the new .dockstore.yml be viable?

5

Hi dyuen,

Ah yes, (b) certainly poses a problem. I think rather than amend all of our releases from the past ~2 years, we’ll wait to see if any of the users of our pipelines actually request one of the missing versions, and amend them on an as-needed basis. Thanks again for your help with this!

Regards,
David

6

Sounds reasonable.
Let us know when the next release goes out and we can keep an eye out to see whether it gets released properly on Dockstore.

1 Like