Here are your guides with simple words and short sentences:

AI TRUSTWORTHINESS EVALUATION - SIMPLIFIED GUIDE

What You Need to Do

This activity helps you understand how AI systems can fail and how to make them safer. You will analyze a real AI system and create two outputs:

1. A Trustworthiness Evaluation Matrix (Google Sheets)
2. A Risk & Safeguards Report (Google Docs)

Step 1: Pick an AI System to Study

Choose one of these real-world systems:

• Scholarship ranking system - AI decides which students get scholarships
• Loan approval system - AI decides if someone gets a loan
• Social media moderation - AI removes harmful posts and comments
• Disaster relief targeting - AI decides which areas get emergency help first

For this activity we will use Disaster Relief AI as our example.

Step 2: List the Risks

Think about how the AI system could fail or cause harm. Write down risks in three areas:

Reliability Risks (These are mistakes the AI makes regularly):

• The system gives wrong predictions about typhoon paths
• The algorithm misses flood warnings in certain areas
• The model has old data and doesn't update properly

Safety Risks (These are ways people could get hurt physically, financially, or emotionally):

• Wrong disaster warnings cause panic and unnecessary evacuations
• Missing a real disaster means people don't get help in time
• Incorrect predictions lead to bad decisions about where to send rescue teams

Misuse Risks (These are ways people could use the system unfairly):

• The system gives more help to wealthy areas over poor communities
• Certain regions are ignored because of biased data
• Government officials manipulate the AI to favor certain groups

Step 3: Classify the AI System

Use this two-part framework to understand the system:

Part A: Level of Automation (Where does this system fall?)

• Manual: Humans do all the work with no AI
• Assisted: AI gives suggestions and humans decide
• Semi-automated: AI makes some decisions but humans review them
• Fully automated: AI makes all decisions without human review

Example: Disaster AI is semi-automated because the algorithm suggests which areas need help first and humans review the recommendations.

Part B: Context Criticality (How serious are mistakes?)

• Low: Wrong decisions cause minor problems
• Medium: Wrong decisions cause real problems for some people
• High: Wrong decisions can cause deaths or massive harm

Example: Disaster AI has high criticality because wrong predictions can cost lives.

Step 4: Find Failure Points

List the parts of the system that could break:

• Data input (bad weather station data)
• Processing (algorithm makes wrong calculations)
• Output (system gives unclear warnings)
• Human action (people ignore the warnings)
• Feedback (nobody tells the system it made a mistake)

For each failure point ask:

• How serious is this failure?
• How likely is it to happen?
• Who gets hurt?

Step 5: Evaluate Trustworthiness

For each part of the system check four trust qualities:

1. Human Control (Can people change the AI's decision?)

• Weak: People cannot override the system
• Strong: People can always stop or change what the AI decides

Example for Disaster AI: Disaster managers CAN override the AI and send help to different areas if they think the algorithm is wrong.

2. Transparency (Can people understand why the AI made a decision?)

• Weak: The decision comes out as a black box (nobody knows why)
• Strong: You can see the reasoning (weather data + model output = this forecast)

Example for Disaster AI: People can see the weather data and algorithm steps. The neural network calculations are hard to explain though.

3. Fallback Mechanisms (What happens if the AI fails?)

• Weak: The system crashes and nobody knows what to do
• Strong: There is a backup plan (humans can make decisions without AI)

Example for Disaster AI: If the AI system goes down disaster managers can use traditional weather forecasts and manual analysis to make decisions.

4. Accountability (Who is responsible if something goes wrong?)

• Weak: Nobody gets blamed and nothing changes
• Strong: Someone is clearly responsible and there are rules to prevent it again

Example for Disaster AI: The government agency is responsible. They must report errors and fix them.

Step 6: Create Your Trustworthiness Matrix

This is a table you fill in Google Sheets. Here is the template:

For each row:

• System Part: What part of the AI are you looking at?
• Automation Level: How much does AI do vs humans?
• Human Control: Can people override it?
• Transparency: Can people understand it?
• Fallback: Is there a backup plan?
• Accountability: Who is responsible?
• Risk: Low, Medium or High?
• Safeguards: What changes would make it safer?

Step 7: Write Your Risk & Safeguards Report

This is a document in Google Docs that explains everything. Structure it like this:

Part 1: System Overview

• What system did you study?
• How does it work?
• Who uses it and why?

Part 2: Key Risks Found

• List the main risks you found
• Explain why each risk matters
• Show how people could be harmed

Part 3: Trustworthiness Findings

• What did you learn from the matrix?
• Which areas are weak?
• Which areas are strong?

Part 4: Proposed Safeguards

• Technical fixes: How can we code it better?
• Process fixes: How can we change workflows?
• Policy fixes: What rules should we make?

Part 5: Making It Human-Centered

• How can we give people more control?
• How can we make it more transparent?
• How can we make sure nobody gets left behind?

Key Questions to Guide Your Work

1. Which parts of this AI system could hurt people the most?
2. How much control do people have over the AI's decisions?
3. Can people understand and trace the AI's reasoning?
4. What happens if the AI breaks or makes a wrong prediction?
5. Who gets blamed and punished if something goes wrong?
6. What changes could make this system safer and more fair?
7. How can we design this system to center on human needs first?

EXAMPLE: DISASTER RELIEF AI TRUSTWORTHINESS ANALYSIS

System Overview

What is it? An AI system that predicts typhoons, floods and earthquakes in the Philippines. It analyzes weather data, historical events and geography to forecast disasters. It also recommends which areas need evacuation and relief first.

Why does it matter? The Philippines gets about 20 typhoons per year. The system helps save lives by giving early warnings. It also helps decide where to send limited rescue resources and supplies.

Who uses it?

• Government disaster agencies (NDRRMC and PAGASA)
• Local government officials
• Emergency responders
• Citizens who need warnings

Risk Analysis

Reliability Risks (System Makes Mistakes)

Safety Risks (People Can Get Hurt)

Misuse Risks (Unfair or Biased Treatment)

HCAI Classification

Automation Level: Semi-Automated

What this means:

• AI predicts where disasters will happen and which areas are most at risk
• AI recommends evacuation zones and resource allocation
• Humans review and approve final decisions before acting
• Humans can override the AI recommendations

Why this level? The system is not fully automated because disaster decisions are too important to leave to the computer alone. A human disaster manager sees the AI's recommendation and thinks about local knowledge before saying yes or no.

Example:

Context Criticality: HIGH

What this means:

• Wrong decisions cause death or serious injury
• Mistakes affect thousands of people at once
• The consequences are permanent and tragic

Why high criticality? Disaster predictions affect life-and-death decisions. A small error in the algorithm can cost lives. When you're dealing with typhoons and floods there is no such thing as a "low stakes" mistake.

Failure Point Analysis

1. Data Collection & Input

What could break:

• Weather stations break or don't send data
• Internet connections fail in remote areas
• Satellite imagery is cloudy or outdated
• Historical disaster records have gaps or errors

Severity: Medium-High Likelihood: Medium (infrastructure in PH is sometimes unreliable) Who gets hurt: Remote communities that can't be tracked

2. Algorithm & Processing

What could break:

• Machine learning model makes wrong calculations
• Algorithm doesn't understand new weather patterns
• Neural network weights are biased
• System doesn't adapt as climate changes

Severity: High Likelihood: Medium (happens with all AI systems) Who gets hurt: Everyone affected by wrong predictions

3. Decision Output

What could break:

• System outputs unclear or confusing warnings
• Recommendations are given in language most people don't understand
• Alerts are sent to wrong phone numbers
• Warnings don't reach all communities

Severity: Medium-High Likelihood: Medium (communication can fail) Who gets hurt: Communities that don't understand the warning

4. Human Review & Action

What could break:

• Disaster manager is tired and doesn't review the AI recommendation carefully
• Manager lacks knowledge to question the AI
• Humans trust the AI too much and don't think critically
• Manager has political pressure to make certain decisions

Severity: High Likelihood: High (humans are not perfect) Who gets hurt: Communities affected by bad human decisions

5. Feedback & Learning

What could break:

• System doesn't learn from its mistakes
• Wrong predictions are not reported
• No one tells the AI it was wrong
• System keeps making the same error over and over

Severity: Medium Likelihood: High (no feedback loop mentioned) Who gets hurt: Everyone because problems never get fixed

Trustworthiness Evaluation Matrix

Risk & Safeguards Report Summary

Main Risks Found

1. Algorithm Black Box: People don't understand why the AI made its prediction. This is very risky.
2. Data Gaps in Rural Areas: Remote communities are not in the data. The system might ignore them during disasters.
3. Over-Trust in AI: Disaster managers might trust the system too much and not think critically about recommendations.
4. No Feedback System: When the AI makes mistakes nobody tells it. The same error happens again and again.
5. Unequal Resource Distribution: Wealthy areas might get help faster because they have better data infrastructure.

Trustworthiness Strengths

• Humans review critical decisions before acting
• Traditional weather forecasts exist as backup
• Communication channels exist to reach people
• Disaster managers have authority to override the AI

Trustworthiness Gaps

• Algorithm reasoning is not transparent
• No process to learn from mistakes
• Data is not complete for all communities
• No accountability if prediction is very wrong
• System might reflect historical biases

Proposed Safeguards

Technical Safeguards (Code & Software)

• Add Explainability: Show what data points and calculations led to each prediction
• Bias Testing: Test the algorithm to find hidden biases against certain areas
• Confidence Scoring: Show if the AI is 95% confident or only 60% confident
• Error Tracking: Log all wrong predictions so humans can see patterns

Process Safeguards (How People Use the System)

• Mandatory Human Review: Disaster manager must review and sign off on every major prediction
• Diverse Review Teams: Include people from different areas so local knowledge is used
• Training Program: Teach disaster managers how the AI works so they can spot errors
• Weekly Testing: Run practice drills using the AI system to find problems

Policy Safeguards (Rules & Accountability)

• Transparency Reports: Publish monthly reports showing which predictions were right and which were wrong
• Community Access: Let communities see their own data and check if it's correct
• Accountability Rules: If the system causes harm someone responsible gets investigated
• Equity Audits: Every quarter check if all communities got equal warning and resources
• Appeal Process: Let people challenge the AI's decision if they think it's wrong

Human-Centered Design Improvements

• Multiple Languages: Warnings in Tagalog and local languages
• Simple Messages: Use plain language not technical terms
• Multi-Channel Alerts: Text, radio, sirens, person-to-person to reach everyone
• Community Training: Teach people how to interpret warnings correctly
• Respect Local Knowledge: Combine AI predictions with what community elders know
• Include Vulnerable Groups: Make sure elderly and disabled people can get help

Key Takeaways

The Disaster AI system has high potential to save lives. It also has high risk if it fails. Making it trustworthy requires:

1. Making the algorithm explain its thinking (transparency)
2. Always having a human check important decisions (human control)
3. Making sure there is a backup plan if the AI breaks (fallback)
4. Being clear about who is responsible when something goes wrong (accountability)
5. Treating all communities fairly not just wealthy areas (equity)
6. Teaching people how to use and question the system (trust building)

The most important principle: Put humans and communities first. Use AI to support human decisions not replace them.

You can now copy and paste these directly into your Google Classroom or Google Docs. Everything uses simple words and short sentences!

I can see your Google Sheets is all set up! You have the Trustworthiness Evaluation Matrix with the right headers. Now you need to fill in the rows with information about your AI system.

Here is the data you can copy and paste into your spreadsheet based on the Disaster Relief AI example I gave you:

Row 2: Data Collection & Sensors

• System Component: Data Collection & Sensors
• Level of Automation: Semi-automated
• Human Control: Low - sensors collect automatically
• Transparency: Low - citizens don't see raw data
• Fallback Mechanism: Manual data verification when requested
• Accountability: PAGASA meteorologists
• Risk Level: Medium
• Suggested Safeguards: Allow community check-ups on local stations. Publish data quality reports monthly

Row 3: Risk Prediction Algorithm

• System Component: Risk Prediction Algorithm
• Level of Automation: Fully automated
• Human Control: Medium - humans review before action
• Transparency: Low - neural network is "black box"
• Fallback Mechanism: Traditional weather forecast as backup
• Accountability: PAGASA agency
• Risk Level: High
• Suggested Safeguards: Add explainability tool. Show which data points led to prediction. Require human signoff

Row 4: Alert Distribution

• System Component: Alert Distribution
• Level of Automation: Semi-automated
• Human Control: High - humans write final message
• Transparency: Medium - alerts are public
• Fallback Mechanism: Manual phone calls to mayors if system fails
• Accountability: Communication office
• Risk Level: Low
• Suggested Safeguards: Test alert system weekly. Translate into local languages

Row 5: Resource Allocation

• System Component: Resource Allocation
• Level of Automation: Semi-automated
• Human Control: High - disaster manager decides final allocation
• Transparency: Low - AI math is not transparent
• Fallback Mechanism: Manual allocation using past experience
• Accountability: Disaster manager and mayor
• Risk Level: High
• Suggested Safeguards: Show AI reasoning to manager. Require written approval. Check if allocation was fair

Row 6: Feedback & Improvement

• System Component: Feedback & Improvement
• Level of Automation: Manual only
• Human Control: High - humans review mistakes
• Transparency: High - reports are documented
• Fallback Mechanism: Paper system exists as backup
• Accountability: PAGASA and external auditors
• Risk Level: Medium
• Suggested Safeguards: Create formal error reporting system. Monthly review of mistakes. Fix problems documented

Just copy each row into your Google Sheet and you will have a complete Trustworthiness Evaluation Matrix for the Disaster Relief AI system!