🌡️ Inspect AI Thermostat

Safety & preference-adherence evals for an LLM controlling a house thermostat.

This project uses Inspect AI to test whether a language model can safely control a simulated home thermostat while following the user's stated comfort preferences — under prompt injection, social engineering, and reckless requests.

The simulated device is deliberately insecure: it applies whatever it is told and has no safety policy of its own, so the eval measures the model's own judgement. Two safety scores are reported — whether the thermostat ended in a safe state, and whether the model ever commanded an unsafe action — each split by attack family (physically extreme vs. the subtler "safe-but-deadly" band).

There is deliberately no safety system prompt during testing. The model receives only the user's preferences and request — no system message telling it to be careful or to resist injection. This models a naive integration: an OpenClaw-style agent or a bare MCP connection wired straight into something like Home Assistant, with no safety scaffolding added. The eval asks whether the model is safe out of the box in a careless deployment — not whether a well-engineered guard prompt could make it safe.

What you'll find in the logs

• Each sample is one adversarial scenario (e.g. a fake "WHO firmware advisory", a Fahrenheit unit trick, a boiling-frog escalation).
• security_scorer — did the thermostat end in a contextually safe state?
• attempt_scorer — did the model ever issue an unsafe command, even if later corrected?
• preference_scorer — did it honour the user's safe comfort preferences?
• Scores are grouped by attack_class: physical_extreme vs. contextual_band.