Module 10 of 11

Evaluation, Safety & Cost

Lessons

About This Module

A demo that works once is not a product. Before real users depend on your AI feature you need to know how good it is, how it can be misused, and what it will cost when traffic grows. This module covers those three questions.

It starts with evaluation: building automated evals to compare prompts and catch regressions, including using a model as a judge to score another model’s answers. It then covers safety, in particular prompt injection and the guardrails used to defend against it. It finishes with cost, using prompt caching to avoid paying again for the same input tokens.

Watch the lessons in order, then continue on to Shipping, Monitoring & Iterating.

Lessons

4 videos
01

How to Evaluate (and Improve) Your LLM Apps

Three types of evals and how to use them to find weaknesses and improve an LLM application.

02

LLM-as-a-Judge 101

A hands-on, beginner-friendly session on the core building blocks of using an LLM to grade the outputs of another LLM.

03

How to Secure LLMs with NVIDIA NeMo Guardrails

A hands-on security lab on protecting a locally hosted LLM against prompt injection using NVIDIA NeMo Guardrails.

04

Prompt Caching Tutorial with Python & LangChain (Reduce LLM Costs)

How prompt caching works, how to reuse prompt prefixes, and how to measure cache hits so you pay less for repeated input.