Question answering is the task of giving a direct answer to a question asked in natural language. In its classic extractive form, the model reads a passage and points to the span of text that answers the question, a setup made popular by the SQuAD dataset. Open-domain systems go a step further and first have to find the right documents in a large collection.
That search step is the idea behind retrieval-augmented generation (RAG): instead of relying only on what a language model memorised during training, the system retrieves relevant text and gives it to the model together with the question. The lessons begin with a university lecture on question answering, then explain why RAG is useful, and finish with a hands-on series that walks through the retrieval and generation steps.
Watch the lessons in order, then continue on to Evaluation, Bias & Deployment.
IBM researcher Marina Danilevsky explains how adding a retrieval step lets a language model use current, sourced information instead of relying only on its training data.
Video 6minThe first video in LangChain’s RAG series introduces RAG as a way to connect language models to private or recent data, and previews the indexing, retrieval and generation steps.
Video 5minFocuses on the retrieval step: searching an index of documents to find the passages most relevant to a question.
Video 5minFocuses on the generation step: building a prompt from the retrieved passages and the question, then passing it to a language model to produce the answer.
Video 6min