I’ve tried recently a few times to refresh github.com/broadinstitute/viral-pipelines/assemble_refbased (Dockstore is about a week out of date) but, after a few minutes of spinning, I get a relatively non-descript 504 error (image below). status.dockstore.org indicates no problems that it is aware of. Is there a way to get more information on what is erroring?
1 Like
Hey Daniel,
I’m sorry that the error isn’t more informative. What is happening is that the refresh is taking so long that it is reaching the maximum time allowed for a request, so the request is erroring with a timeout.
We are currently working on ways to optimize the refresh so this won’t happen. Unfortunately, there isn’t a good solution right now for the current version of Dockstore. All I can think of is deleting some older versions of your workflow, which obviously isn’t a good idea.
We have a new release coming out early July which would have two possible solutions. I think the GitHub App approach is better long term, but they would both work.
GitHub Apps
This is a new process for getting content onto Dockstore. It requires some initial setup, but the end result is that you no longer need to refresh on Dockstore. Dockstore will automatically pull content from GitHub.
The general process would be
- install our GitHub App on your repository
- add a configuration file to the root of your repository (/.dockstore.yml). We recommend adding to your default branch so that any new branches will also contain the config file. Commonly this is the master or develop branch
After that, any push event that occurs on that repository on a branch with the configuration file present will be automatically added to Dockstore.
Our documentation for this is still in development, but once it is finalized I can post a link here.
Refresh Version
This is a bit of a hacky approach, but it will work when the new release is made. We are adding the ability to refresh specific versions of a workflow. In the UI, this can only be done for existing versions. Via the API you can refresh versions that are not yet on Dockstore. This requires knowing the ID of the workflow, which you can get using the network tab for the public Dockstore page of your workflow. In your case this is 12671. Note the below link won’t work until we make a new release in early July.
https://dockstore.org/api/static/swagger-ui/index.html#/workflows/refreshVersion
You’ll need to click on the lock button on the top right and paste your Dockstore token (which you can find in your account page).
Eventually we plan to surface this feature via the website, but for now it is only via the API.
I recommend the GitHub Apps approach. It will save you from having to deal with manually refreshing on Dockstore when a change is made. Your Dockstore workflows will be automatically kept up to date.
I hope this is helpful. Please let me know if you have any questions.
-Andrew Duncan
Thanks @agduncan94 – I’ve certainly been looking forward to v1.9 since last month when I tried it out a bit on dev. The GitHub Apps approach works quite smoothly, I look forward to seeing it in production soon. Are you also suggesting that the Apps hook doesn’t cause a refresh of all versions, just the specifically pushed one?
When you say “deleting some older versions”, what does that mean? Git doesn’t really have a way of deleting history. Or are you referring to “versions” in a dockstore sense? I am looking for a workaround that I can utilize prior to 1.9.
Anytime there is a push event on GitHub (ex. make a release, make a tag, make a new branch, push to an existing branch) for a branch/tag that contains a .dockstore.yml, Dockstore will update/add a version of the workflow for the corresponding branch/tag that was changed on GitHub. This will not trigger the refresh of other versions on Dockstore, as they have not changed.
The end result is versions are refreshed individually as they are changed on GitHub, and not in batches with the refresh button on Dockstore.
For the “deleting some older versions”, I just meant if you delete some older releases on GitHub and then refresh on Dockstore, these releases will be removed from Dockstore. This means Dockstore doesn’t need to refresh as many versions so it will take less time for the refresh. Hopefully this would result in the refresh taking less time than the timeout, but there is no guarantee.
I’ll try and brainstorm to see if I or anyone else on the team can figure out a temporary fix for before 1.9 release.