Modifying flow parameters on a rerun

[lukef16]

Hi All,

I was wondering if there is a way to modify flow parameters when doing a rerun.

My use case is that I have one big flow that contains many tasks, and I use the submit 'wait for' functionality to control dependancies. In addition I set persist_result=True in the task decorator to ensure that I don't re run failed tasks.

The input to my main flow is a large dictionary that will control the behaviour (inputs and dependancies) of all the tasks to run in my flow.

For example (This is a dummy example to paint a picture):

{ "tasks": [ { "id": 1, "name": "maths", "args": 1, "dependency": [""] }, { "id": 2, "name": "maths", "args": 100, "dependency": [""] }, { "id": 3, "name": "save_data", "args": "/file/location/data.txt", "dependency": ["1", "2"] } ] }

In the example above, my flow first runs two tasks in parallel, both will run for a long period of time. Then my final task (depends on tasks 1 and 2) will save the data to a given file location. Now, say the "save_data" task fails because the file already exisits and I don't want to override the data. I'd like to rerun the flow run and modify the task inputs to save the file to a different location. However, I don't want to rerun the completed 'maths' tasks.

I know I can copy the flow run into a new run, modify the input flow parameters and rerun, but I think this will re run all of the tasks.

For this use case, do you have any suggestions on how best to use prefect to deal with my requirements?

Many thanks,

Luke

[lukef16]

I've just done a bit more research, what are your opinions on this potential solution.

At the end of every task, I create an artifact to store infomation about the task, naming it with the task name and unique flow run id ( ie 1234). Then if I do want to re-run a failed task in the flow. I just copy my flow run (lets say flow-1234 to flow-1234-v2).

I will just need to make sure that any task first checks if a corresponding artifact already exisits, if it does then we move to the next task. If not the task will actually do something.

Not an ideal solution but this may work?