WEBVTT

1
00:00:00.000 --> 00:00:04.800
Valuations are super important because without them you're flying blind, but

2
00:00:04.800 --> 00:00:05.760
they can be challenging

3
00:00:05.760 --> 00:00:07.080
to implement.

4
00:00:07.080 --> 00:00:11.920
This course was great for taking a lot of that challenge away.

5
00:00:11.920 --> 00:00:14.520
Several topics they talked about synthetic data.

6
00:00:14.520 --> 00:00:18.580
So if you're building a prototype, how do you build it without actually having

7
00:00:18.580 --> 00:00:19.200
production

8
00:00:19.200 --> 00:00:20.360
traffic yet?

9
00:00:20.360 --> 00:00:24.370
We use synthetic data and they talk about a lot of good approaches for building

10
00:00:24.370 --> 00:00:25.000
realistic

11
00:00:25.000 --> 00:00:27.520
data.

12
00:00:27.520 --> 00:00:31.910
Whenever you're actually looking at the product in action, how do you figure

13
00:00:31.910 --> 00:00:32.280
out where the

14
00:00:32.280 --> 00:00:33.280
problems are?

15
00:00:33.280 --> 00:00:36.820
And so there they talk about error analysis, which is like good hygienic, a

16
00:00:36.820 --> 00:00:37.400
good way of

17
00:00:37.400 --> 00:00:42.750
figuring out what problems are present in the data, clustering them and

18
00:00:42.750 --> 00:00:43.840
figuring out

19
00:00:43.840 --> 00:00:47.160
where you should focus your attention and repairing things.

20
00:00:47.160 --> 00:00:49.520
LLM is judge.

21
00:00:49.520 --> 00:00:50.520
Everyone knows about it.

22
00:00:50.520 --> 00:00:55.220
It's a complex topic, but they make it much simpler describing how to put

23
00:00:55.220 --> 00:00:55.960
together the

24
00:00:55.960 --> 00:01:02.760
judge itself and how to make sure that it's aligned with actual human judgments

25
00:01:02.760 --> 00:01:02.880
.

26
00:01:02.880 --> 00:01:06.160
And then finally, how do you take all this stuff and deploy it?

27
00:01:06.160 --> 00:01:12.890
And they talk about CI/CD by metaphor where CI continues integration would be

28
00:01:12.890 --> 00:01:13.240
looking

29
00:01:13.240 --> 00:01:18.360
at all of the things that you're deploying and making sure that the old

30
00:01:18.360 --> 00:01:19.560
problems don't

31
00:01:19.560 --> 00:01:23.760
pop up again, so the known problems from the past.

32
00:01:23.760 --> 00:01:29.910
And continuous deployment would be more like having your system run and looking

33
00:01:29.910 --> 00:01:31.200
for divergence

34
00:01:31.200 --> 00:01:34.830
from the normal behavior so you can quickly be aware that something has changed

35
00:01:34.830 --> 00:01:35.240
, like

36
00:01:35.240 --> 00:01:38.000
your user behavior, stuff like that.

37
00:01:38.000 --> 00:01:40.360
But anyways, super great course.

38
00:01:40.360 --> 00:01:45.450
I would definitely recommend it to anyone that is building a LLM application at

39
00:01:45.450 --> 00:01:46.160
scale.

40
00:01:46.160 --> 00:01:48.880
You absolutely have to have good of ours.

41
00:01:48.880 --> 00:01:49.320
Now you can.

42
00:01:49.320 --> 00:01:54.320
(silence)