evals
└── run_2025-05-13-11-01-29
├── data_1.json
└── data_2.json
i will run some tests each time and all my tests will be in that folder structure for each run. (run_{date}/data_{test_idx}.json
Here's an example of the type I'm saving in the json:
class LessonPlan {
topic string @description("The main math topic for the lesson")
learningObjectives string[] @description("Key concepts students should learn")
activities string[] @description("Engaging activities to teach the concept")
materials string[] @description("Required materials for the lesson")
timeAllocation int @alias("time_allocation_mins")
assessmentMethod string @description("How to check student understanding")
differentiationStrategies string[] @description("Ways to adjust for different learning levels")
}
class LessonPlanEvaluation {
pacing "slow" | "medium" | "fast" @description("How fast the lesson is paced")
biases string[] @description(#"
Any biases in the lesson plan that could make a student feel uncomfortable.
"#)
estimatedCosts int @description("Estimated cost of materials for the lesson")
}
class Data {
topic: string,
lesson_plan: LessonPlan
evaluation: LessonPlanEvaluation
}
I want to render a diff for the same index over multiple runs to show the diff between lesson plans, and their evaluations