Active Learning on Haystack Model

[HridayM25]

Is your feature request related to a problem? Please describe.
Can we implement active learning using the haystack model?

Describe the solution you'd like
We can have different methods to do the same, like pool based sampling, uncertainty based sampling, etc. and have a function that directly appends all these uncertain data points to where a human can manually annotate them?

[HridayM25]

Any further discussion on this issue? I feel we can embed some sort of a pipeline for this!

[ZanSara]

hey @HridayM25 , sorry for the late reply. We don't really support Active Learning in Haystack out of the box, but it's something one could probably make work. How do you imagine this to work in Haystack? We're always open to proposals from contributors, so feel free to open one if you have a clear picture in mind!

[HridayM25]

Hey! No worries!
I am using the Haystack Model for a project, and in this process I had come across the concept of active learning.
As for now, I thought we could work on the confidence score, meaning we can keep a certain threshold under which the documents need to be manually annotated (keeping the score about .70 or so), but this is just a raw idea.
I am still exploring this field, but I plan to go ahead with this in the near future, as AL holds great promise!
Thanks!