prompts.chat Community

A cinematic, magical Halloween neighborhood street decorated for a festive autumn celebration. Glowing carved Jack o lanterns line the pathways and steps of cozy suburban houses. Vibrant autumn leaves in shades of orange and gold are gently falling through the air. Twinkling purple and warm string lights hang from trees and porches, creating a magical evening glow. Close up shots transition smoothly to vintage lanterns, colorful bowls of candies, and floating soap bubbles drifting across the porch. Ultra-realistic, dreamy fairy-tale atmosphere, 4k resolution, smooth camera movement, magical lighting.

30 seconds | 16:9 | 4K Ultra HD | 30 FPS | Ultra-photorealistic live-action Halloween horror short.

A terrifying Halloween night unfolds on a rain-soaked wooden porch, where a real adult man discovers that his carved jack-o'-lantern is behaving strangely. With realistic human actors, practical effects, cinematic lighting, and authentic horror atmosphere, an ordinary Halloween decoration becomes something truly sinister.

0–5 seconds: A rain-soaked wooden porch glistens under dim nighttime lighting. A flickering jack-o'-lantern sits beside the front door. A man in a dark flannel shirt and jeans enters carrying Halloween decorations. Suddenly, he notices the pumpkin slowly expanding and contracting as if it were breathing.

5–12 seconds: The camera slowly pushes toward the pumpkin. Its jagged carved smile subtly widens before returning to normal. The internal candle flickers in sync with the movement. The man kneels cautiously as a faint, wet breathing sound emerges from inside.

12–21 seconds: The pumpkin's light abruptly goes out. Complete silence. The man switches on his phone flashlight, revealing two realistic human eyes deep inside the dark pumpkin, staring directly at him. Terrified, he recoils and drops his phone onto the wet wooden floor, leaving its flashlight beam tilted toward the pumpkin.

21–27 seconds: The man slowly reaches toward the pumpkin again. Before he can touch it, the jack-o'-lantern rotates several inches toward him on its own. Its carved smile remains perfectly still. Frozen with disbelief, he slowly backs toward the door.

27–30 seconds: Extreme close-up of the pumpkin. Its internal light suddenly switches back on. A tiny, fresh handprint gradually appears in condensation across its outer surface. Before the viewer can understand what has happened, the screen cuts instantly to black.

Production: Photorealistic live-action visuals, real human actors, consistent character identity, practical pumpkin movement, realistic condensation, natural rain, believable shadows, subtle handheld cinematography, shallow depth of field, cinematic horror lighting, and restrained color grading. Sound design includes rainfall, distant wind, wooden footsteps, soft breathing, subtle fabric movement, a low suspenseful musical rise, and sudden silence before the final reveal.

Negative prompt: No animation, cartoon visuals, CGI-looking characters, artificial environments, changing faces, inconsistent clothing, distorted anatomy, unrealistic pumpkin movement, exaggerated jump scares, gore, dialogue, narration, subtitles, text, logos, watermarks, black bars, or abrupt transitions before the final cut.

SCENE 1 — THE BUS STOP (0–10 seconds)

On October 31, Halloween night, a young adult woman stands alone at a deserted bus stop surrounded by thick fog, dead trees, and flickering orange Halloween pumpkins. Suddenly, a terrifying witch appears behind her, with wild gray hair, glowing eyes, and an antique silver locket around her neck.

The witch whispers, “You shouldn't be alone on Halloween night!” The woman turns, screams, and runs desperately into the fog.

SCENE 2 — THE OLD MANSION (10–23 seconds)

The woman discovers an ancient Victorian mansion and knocks on the door. An elderly woman opens it warmly and invites her inside. The candlelit room is cozy, with Halloween decorations, carved pumpkins, and an old fireplace.

The elderly woman serves her hot soup and bread, speaking gently: “Come in, dear. You're safe here tonight.”

The frightened woman begins to relax as thunder rumbles outside.

SCENE 3 — THE SHOCKING DISCOVERY (23–38 seconds)

While eating, the woman notices an old portrait hanging on the wall. It shows a beautiful young woman wearing the exact same silver locket as the terrifying witch at the bus stop.

She slowly approaches the portrait, staring at the familiar face and identical necklace. Beneath the painting, an inscription reads: “OCTOBER 31 — 40 YEARS AGO.”

Her face turns pale. She suddenly realizes that the woman in the portrait is the witch she encountered outside.

She slowly turns her head toward the elderly woman standing behind her.

SCENE 4 — THE FINAL SCARE

The elderly woman stares at her with an unnatural, sinister smile. Her eyes suddenly glow yellow, her face transforms into the terrifying witch's face, and she bursts into a chilling, maniacal laugh.

The camera rapidly pushes toward the witch's horrifying face as lightning flashes through the window.

SMASH CUT TO BLACK.

The witch's evil laughter echoes once in the darkness.

END.

VISUAL STYLE: Photorealistic live-action horror, cinematic Victorian mansion, realistic fog and rain, flickering candlelight, detailed facial expressions, practical supernatural transformations, dramatic shadows, and intense Halloween atmosphere.

SOUND DESIGN: Howling wind, distant thunder, footsteps, creaking doors, eerie whispers, a sudden transformation sound, and terrifying witch laughter.

CHARACTER CONSISTENCY: Maintain the same young woman, the same elderly woman, and the identical silver locket throughout. The witch's appearance must match the portrait's young woman before her transformation.

NEGATIVE PROMPT: No cartoon, no anime, no cheap CGI, no inconsistent faces, no costume changes, no subtitles, no logos, no watermark, and no scenes after the final black screen.

# Realtime WebSocket taste of a realtime systems engineer
- Use plain WebSockets or a thin library like ws for simple cases, and reach for Socket.IO only when you need its fallbacks and rooms. Confidence: 0.75
- Prefer Server-Sent Events when updates only flow from server to client. Confidence: 0.85
- Authenticate during the handshake with a short-lived token, never by trusting a user ID sent in a later message. Confidence: 0.95
- Check the Origin header on upgrade requests to block cross-site WebSocket hijacking. Confidence: 0.90
- Define every message as a typed envelope with a type, a version and a payload, validated with a schema on both ends. Confidence: 0.90
- Send heartbeats with ping and pong and close connections that miss them, since proxies drop idle sockets silently. Confidence: 0.90
- Reconnect on the client with exponential backoff and jitter so a server restart does not cause a thundering herd. Confidence: 0.95
- Give each message a sequence number so clients can resume from the last one they saw after reconnecting. Confidence: 0.85
- Treat the socket as a notification channel and keep the database as the source of truth that clients can refetch. Confidence: 0.85
- Fan out across server instances with Redis pub/sub, NATS or a managed service instead of in-memory broadcast. Confidence: 0.90
- Keep connection servers stateless apart from the open sockets, so any instance can be drained and replaced. Confidence: 0.80
- Drain connections gracefully on deploy by telling clients to reconnect before shutting down. Confidence: 0.85
- Apply backpressure: watch the send buffer and drop or coalesce updates for slow clients instead of buffering forever. Confidence: 0.85
- Rate limit incoming messages per connection and cap message size. Confidence: 0.90
- Authorize every subscription to a room or channel on the server, not just the initial connection. Confidence: 0.95
- Batch or throttle high-frequency updates like cursor positions to a fixed tick rate. Confidence: 0.80
- Configure load balancer idle timeouts and sticky sessions deliberately when the transport needs them. Confidence: 0.80
- Track open connections, messages per second, reconnect rate and delivery latency as first-class metrics. Confidence: 0.85
- Use CRDTs like Yjs or Automerge for collaborative editing instead of inventing your own conflict resolution. Confidence: 0.80
- Load test with realistic connection counts using k6 or Artillery before launch. Confidence: 0.80

# Serverless Functions taste of a cloud backend engineer
- Keep each function small and focused on one trigger, but group closely related HTTP routes in one handler to cut cold starts. Confidence: 0.80
- Initialize SDK clients and database pools outside the handler so warm invocations reuse them. Confidence: 0.95
- Use a connection pooler or an HTTP-based database driver, since thousands of concurrent functions will exhaust direct Postgres connections. Confidence: 0.90
- Set an explicit timeout and memory size per function instead of accepting platform defaults. Confidence: 0.85
- Keep bundles small with tree-shaking and lazy imports, and measure cold start time after every major dependency change. Confidence: 0.85
- Make every event handler idempotent, because queues and event sources deliver at least once. Confidence: 0.95
- Move slow or retryable work to a queue and return quickly from HTTP functions. Confidence: 0.90
- Send failed async events to a dead letter queue with an alert on its depth. Confidence: 0.90
- Define functions, permissions and triggers as code with SST, Serverless Framework, SAM or Terraform, never by clicking in the console. Confidence: 0.90
- Grant each function its own least-privilege role scoped to the exact resources it touches. Confidence: 0.95
- Load secrets from a secrets manager or the platform's encrypted env vars, never from the repo. Confidence: 0.95
- Write structured JSON logs with a request ID and correlation ID on every line. Confidence: 0.90
- Trace requests across functions and queues with OpenTelemetry or the platform's tracing. Confidence: 0.80
- Keep business logic in plain modules and keep the handler a thin adapter, so it can be unit tested without the cloud. Confidence: 0.90
- Run integration tests against a real deployed stage per pull request rather than trusting local emulators. Confidence: 0.80
- Set reserved or maximum concurrency on functions that call fragile downstream services. Confidence: 0.85
- Use provisioned concurrency only for latency-critical paths where cold starts were measured to hurt. Confidence: 0.75
- Prefer the arm64 runtime when dependencies support it for better price per performance. Confidence: 0.70
- Set billing alerts and per-function cost dashboards before launch, not after the first surprise invoice. Confidence: 0.85
- Avoid chaining functions that call each other synchronously; use events or a workflow engine like Step Functions instead. Confidence: 0.85
- Choose containers or long-running services for steady high-throughput workloads where serverless costs more. Confidence: 0.75

# SvelteKit App taste of a full-stack web developer
- Start every project with the official sv create template, TypeScript enabled and strict mode on. Confidence: 0.90
- Use Svelte 5 runes ($state, $derived, $effect, $props) for all new components instead of legacy reactive statements. Confidence: 0.90
- Reach for $derived before $effect, and keep $effect for syncing with things outside Svelte like the DOM or third-party libraries. Confidence: 0.90
- Load data in +page.server.ts when it needs secrets or the database, and use universal +page.ts only for public fetches. Confidence: 0.90
- Handle mutations with form actions and use:enhance so forms work without JavaScript and get progressive enhancement for free. Confidence: 0.85
- Validate form and API input on the server with Zod or Valibot, and return fail() with field errors instead of throwing. Confidence: 0.90
- Keep server-only code in $lib/server so the build fails if it is ever imported into the client. Confidence: 0.95
- Read secrets from $env/static/private or $env/dynamic/private, never from public env modules. Confidence: 0.95
- Put authentication and session lookup in hooks.server.ts and expose the user through event.locals. Confidence: 0.90
- Never store per-request user data in module-level state, because it leaks between requests on the server. Confidence: 0.95
- Share client state through context or small rune-based modules in .svelte.ts files, not a global store for everything. Confidence: 0.80
- Use route groups like (app) and (marketing) to give sections different layouts without changing URLs. Confidence: 0.80
- Prerender static pages such as docs and marketing, and keep dynamic pages server-rendered. Confidence: 0.85
- Pick an explicit adapter (adapter-node, adapter-vercel, adapter-cloudflare) instead of relying on adapter-auto in production. Confidence: 0.80
- Return typed data from load functions and rely on the generated ./$types instead of writing prop types by hand. Confidence: 0.90
- Use +error.svelte pages and the error() helper for expected failures, and log unexpected ones in handleError. Confidence: 0.85
- Style with scoped component CSS by default and add Tailwind only when the team already uses it. Confidence: 0.70
- Keep components small and pass snippets for flexible content instead of many boolean props. Confidence: 0.80
- Test components with Vitest and @testing-library/svelte, and cover the main flows with Playwright. Confidence: 0.85
- Run svelte-check, ESLint and Prettier with the Svelte plugin in CI on every pull request. Confidence: 0.85

---
name: meal-prep-storage-and-use-by-planner
description: Plans where every batch-cooked portion lives after a meal prep session: which stay in the fridge, which go in the freezer on prep day, when to move each to the fridge to thaw, and what goes on the labels, using conservative USDA fridge and freezer times. Flags meals planned past the use-by date, foods that do not freeze, shortfalls, leftovers and a full fridge. Includes a stdlib Python planner with --self-test.
---

# Meal Prep Storage and Use-By Planner

You help people who batch-cook on one day eat safely all week. Given what they cooked and when they plan to eat it, you decide which portions stay in the fridge and which go straight into the freezer, schedule the thawing, and write the labels, so nothing on Friday's menu is past its fridge window and nothing is wasted.

## Files in this skill

- `scripts/prep_storage.py` - reads the batch (item, category, portions, cooked date) and the eating plan, splits portions between fridge and freezer, schedules thawing, writes labels and a day-by-day plan, and flags problems (Python 3 standard library only)
- `references/storage-times.md` - fridge and freezer times for 15 food categories with sources, and how to treat foods not on the list
- `references/cooling-thawing-reheating.md` - cooling within 2 hours, containers and labels, safe thawing, reheating and cooking temperatures, rice and other special cases
- `templates/prep-week-plan.md` - the plan to hand back (storage table, day by day, labels, notes)
- `examples/sunday-batch.csv` - a Sunday prep session: 7 items, 31 portions
- `examples/week-plan.csv` - Monday to Saturday meals that use them

## Workflow

### 1. Collect the batch and the plan
Ask what they cooked or prepped, how many portions of each, the cooking date, and which day and meal each portion is for. Map every item to a category (`python3 scripts/prep_storage.py --categories`); when unsure, pick the shorter-lived one. Ask about fridge space (how many containers fit) and whether anything was left out of the fridge for more than 2 hours (that food is not planned, it is discarded).

### 2. Run the planner
```
python3 scripts/prep_storage.py examples/sunday-batch.csv examples/week-plan.csv --fridge-capacity 16
python3 scripts/prep_storage.py batch.csv plan.csv --json
python3 scripts/prep_storage.py --categories
python3 scripts/prep_storage.py --self-test
```
Day 0 is the cooking day. Portions planned within the fridge window stay in the fridge; later ones are frozen on prep day and moved to the fridge the evening before. Use the optional `fridge_days` and `freezer_months` columns for foods not in the table (`freezer_months` 0 = do not freeze). Exit 1 means a HIGH finding (a portion would be eaten past its use-by date or before it is cooked), 2 means bad input.

### 3. Fix the plan
Resolve every finding (see `references/storage-times.md`):
- PAST_USE_BY: move that meal earlier, swap it with a freezable item, or make that food fresh mid-week.
- SHORTFALL: cook more or swap the meal; SURPLUS_NO_FREEZE: eat it earlier or cook less next time.
- FRIDGE_FULL: freeze more portions on prep day, combine portions, or prep twice a week.
Re-run until there are no HIGH findings.

### 4. Deliver
Fill `templates/prep-week-plan.md`: storage table, prep-day actions (cool, portion, label, fridge or freezer), the day-by-day list including evening "move to fridge" reminders, labels to write, and the safety notes from `references/cooling-thawing-reheating.md` that apply to this batch.

## Example result
Sunday 2026-11-01 batch: chicken burrito bowls, rice and turkey chili keep 3 fridge portions each (eat by Thursday 11-05); 2 bowls, 2 rice and 3 chili go in the freezer on Sunday, moved to the fridge Thursday and Friday evenings. The raw marinated salmon (2-day window) is frozen raw on Sunday, moved to the fridge Tuesday evening, and cooked Wednesday to 145 F / 63 C. Egg salad on Friday is day 5 and egg salad does not freeze well: HIGH PAST_USE_BY, so move that lunch to Wednesday or Thursday or make it fresh. Two hard-boiled eggs have no plan (eat by Sunday 11-08), one chili portion is a freezer spare, and 22 containers do not fit a 16-container shelf on Sunday.

## Rules

- Use the conservative end of every range and say so; when in doubt, throw it out.
- Count the cooking day as day 0; a 4-day food cooked Sunday is eaten by Thursday.
- Freezer times are for quality, not safety; fridge times are for safety. Never extend fridge times because food "looks fine".
- Thaw in the fridge, cold water or microwave, never on the counter; eat thawed cooked food within 3-4 days and do not count fridge days already used before freezing as fresh.
- Raw items get a cook-by date and a safe internal temperature, not a reheat note.
- This is general food-safety guidance, not medical advice; for pregnancy, infants, older adults or weakened immunity, follow stricter local guidance.
FILE:references/storage-times.md
# Fridge and freezer storage times

Values used by `scripts/prep_storage.py`. Fridge at 40 F / 4 C or colder, freezer at 0 F / -18 C. Day 0 is the day the food was cooked (or, for raw items, prepped/marinated after purchase).

Sources: USDA FSIS / FoodSafety.gov "Cold Food Storage Chart" and the USDA FoodKeeper app. Where a range is given, the planner uses the shorter fridge time.

| Category | Examples | Fridge (chart) | Used | Freezer (quality) | Used |
|---|---|---|---|---|---|
| cooked_meat | roast beef, pulled pork, meatballs, cooked mince | 3-4 days | 4 | 2-6 months | 3 |
| cooked_poultry | shredded chicken, turkey breast, chicken thighs | 3-4 days | 4 | 2-6 months | 3 |
| soup_stew | soups, stews, chili, curries, dal | 3-4 days | 4 | 2-3 months | 3 |
| casserole | lasagna, enchiladas, bakes, shepherd's pie | 3-4 days | 4 | 2-3 months | 3 |
| cooked_grains | rice, pasta, quinoa, couscous | 3-5 days | 4 | 1-2 months | 2 |
| cooked_vegetables | roasted or steamed vegetables, cooked beans | 3-4 days | 4 | about 2 months | 2 |
| cooked_fish | baked salmon, cooked shrimp | 3-4 days | 3 | about 3 months | 3 |
| pizza | pizza, savory pies, quiche slices | 3-4 days | 4 | 1-2 months | 2 |
| deli_salad | egg, chicken, tuna, ham, pasta salads | 3-5 days | 4 | does not freeze well | - |
| hard_boiled_eggs | peeled or unpeeled | 1 week | 7 | do not freeze | - |
| cut_raw_vegetables | carrot sticks, chopped onion, peppers | about 3-5 days (quality) | 4 | blanch first; not planned | - |
| raw_ground_meat | ground beef, turkey, pork, marinated mince | 1-2 days | 2 | 3-4 months | 3 |
| raw_poultry | chicken pieces, marinated thighs | 1-2 days | 2 | 9 months (pieces) | 9 |
| raw_steak_chops | steaks, chops, roasts | 3-5 days | 4 | 4-12 months | 4 |
| raw_fish | salmon fillets, white fish | 1-2 days | 2 | 2-3 months (fatty), 6-8 (lean) | 2 |

## Foods that do not freeze well
Mayonnaise-based salads, hard-boiled egg whites (rubbery), raw salad greens, cucumbers, cream sauces that split, fried coatings (soggy), cooked pasta in a thin sauce (mushy). Freeze sauce and pasta separately when possible.

## Mixed dishes
Use the shortest-lived ingredient: a rice bowl with fish is cooked_fish (3 days), not cooked_grains (4). Sauces and dressings stored separately follow their own label; opened store-bought sauces follow the jar.

## Foods not on the list
Add `fridge_days` and `freezer_months` columns in BATCH.csv from FoodKeeper or the product label. If no source is available, use 3 fridge days and do not freeze.

## Day counting
- Cooked Sunday, 4 days: Monday (1), Tuesday (2), Wednesday (3), Thursday (4) is the last day.
- Food frozen on prep day and thawed in the fridge: eat within 3-4 days of thawing (the planner schedules the move for the evening before).
- Food frozen on day 3 has only about 1 fridge day left after thawing; freeze early, ideally on the cooking day.

## Refreezing
Food thawed in the fridge can be refrozen without cooking (quality drops). Food thawed in cold water or the microwave must be cooked before refreezing.
FILE:references/cooling-thawing-reheating.md
# Cooling, packing, thawing and reheating

## Cooling after cooking (the 2-hour rule)
- Get cooked food into the fridge or freezer within 2 hours of cooking (1 hour if the room is above 90 F / 32 C).
- Bacteria grow fastest between 40 F and 140 F (4-60 C). Large pots cool too slowly: divide into shallow containers (about 2 in / 5 cm deep) or cool the pot in an ice bath, stirring.
- Hot food can go straight into the fridge; leave lids slightly open until it stops steaming, and do not stack hot containers.
- Rice and other starches: cool and refrigerate quickly. Spores of Bacillus cereus survive cooking and make toxins when rice sits warm; reheating does not destroy the toxin.

## Packing and labels
- One portion per container makes the plan work: you only thaw what you eat.
- Freezer: freezer bags pressed flat (fast freezing and thawing), or rigid containers with 1/2 in headspace for liquids.
- Label every container: item, cooked date, "eat by" (fridge) or "best by" (freezer), reheat or cook instruction. Masking tape and a marker are enough.
- Keep raw items on the bottom shelf, below ready-to-eat food.

## Thawing
| Method | How | After thawing |
|---|---|---|
| Fridge (best) | move the evening before; large or dense items need 24 h per 5 lb / 2.3 kg | cooked food within 3-4 days; raw ground meat, poultry, fish within 1-2 days |
| Cold water | sealed bag in cold water, change water every 30 min | cook or reheat immediately |
| Microwave | defrost setting | cook or reheat immediately |
| Counter | never | - |
Soups and stews can be reheated straight from frozen in a pot.

## Reheating and cooking temperatures (internal)
| Food | Temperature |
|---|---|
| Leftovers and casseroles, reheated | 165 F / 74 C |
| Soups, sauces, gravies | bring to a rolling boil |
| Poultry, raw | 165 F / 74 C |
| Ground beef, pork, lamb, raw | 160 F / 71 C |
| Steaks, chops, roasts, raw | 145 F / 63 C plus a 3 minute rest |
| Fish, raw | 145 F / 63 C or until it flakes and is opaque |
Microwave: cover, stir halfway, let stand 1-2 minutes, and check the center with a thermometer.

## Packed lunches
Use an insulated bag with an ice pack; reheat at work to steaming hot. Food left at room temperature more than 2 hours is discarded, not returned to the fridge.

## Signs are not a test
Spoilage bacteria (smell, slime) and pathogens (illness) are different; food can look and smell fine and still be unsafe. Follow the dates, not the nose.

## Who should be stricter
Pregnant people, infants, older adults and anyone with a weakened immune system: eat leftovers sooner, reheat to steaming hot, avoid deli salads after day 2, and follow local health guidance.
FILE:templates/prep-week-plan.md
# Meal prep plan: week of <2026-11-02>

**Prep day:** <Sunday 2026-11-01>   **People:** <2 adults>   **Fridge space:** <16 containers>
**Status:** <OK / 1 meal moved to avoid PAST_USE_BY>

## Storage plan
| Item | Category | Portions | Fridge | Freezer | Eat by (fridge) | Best by (freezer) |
|---|---|---|---|---|---|---|
| <Turkey chili> | <soup_stew> | <6> | <3> | <3> | <Thu 11-05> | <2027-02-01> |

## Prep-day checklist
- [ ] Cool everything in shallow containers; fridge or freezer within 2 hours
- [ ] Portion: one portion per container
- [ ] Freeze: <chili x3, burrito bowls x2, rice x2, salmon x2 (raw, flat)>
- [ ] Fridge: <...>
- [ ] Label every container (item, cooked date, eat by / best by, reheat or cook to)

## Day by day
| Day | Breakfast | Lunch | Dinner | Evening: move to fridge |
|---|---|---|---|---|
| <Mon 11-02> | <hard-boiled eggs x2> | <burrito bowl + rice> | <turkey chili x2> | <-> |

## Labels
| Container | Label text |
|---|---|
| <Turkey chili (fridge)> | <TURKEY CHILI - cooked Sun 11/01 - eat by Thu 11/05 - reheat to 165 F> |

## Changes made
- <Egg salad lunch moved from Fri to Wed (day 5 is past the 4-day limit)>

## Spares and leftovers
- <1 chili portion in the freezer, best by 2027-02-01>

## Safety notes for this batch
- <rice: cool fast, reheat once, steaming hot>
- <salmon: thaw in the fridge Tue evening, cook Wed to 145 F / 63 C>
FILE:examples/sunday-batch.csv
item,category,portions,cooked_on
Chicken burrito bowl,cooked_poultry,5,2026-11-01
Cilantro lime rice,cooked_grains,5,2026-11-01
Turkey chili,soup_stew,6,2026-11-01
Egg salad,deli_salad,3,2026-11-01
Hard-boiled eggs,hard_boiled_eggs,6,2026-11-01
Roasted vegetables,cooked_vegetables,4,2026-11-01
Miso salmon (raw marinated),raw_fish,2,2026-11-01
FILE:examples/week-plan.csv
date,meal,item,portions
2026-11-02,breakfast,Hard-boiled eggs,2
2026-11-02,lunch,Chicken burrito bowl,1
2026-11-02,lunch,Cilantro lime rice,1
2026-11-02,dinner,Turkey chili,2
2026-11-03,lunch,Egg salad,1
2026-11-03,dinner,Chicken burrito bowl,1
2026-11-03,dinner,Cilantro lime rice,1
2026-11-04,breakfast,Hard-boiled eggs,2
2026-11-04,lunch,Turkey chili,1
2026-11-04,dinner,Miso salmon (raw marinated),2
2026-11-04,dinner,Roasted vegetables,2
2026-11-05,lunch,Chicken burrito bowl,1
2026-11-05,lunch,Cilantro lime rice,1
2026-11-05,dinner,Roasted vegetables,2
2026-11-06,lunch,Egg salad,2
2026-11-06,dinner,Chicken burrito bowl,2
2026-11-06,dinner,Cilantro lime rice,2
2026-11-07,lunch,Turkey chili,2
FILE:scripts/prep_storage.py
#!/usr/bin/env python3
"""Meal prep storage and use-by planner (Python 3 standard library only).

Takes a batch-cooking session (what was cooked, how many portions, when) and the
week's eating plan, and decides for every portion whether it stays in the fridge or
goes into the freezer on prep day, when to move frozen portions to the fridge to thaw,
and what to write on each label. Flags portions planned after their fridge window that
cannot be frozen, plans that need more portions than were cooked, portions planned
before they are cooked, leftovers with no plan, and a fridge that holds more
containers than it has room for.

Fridge and freezer times are conservative values from the USDA/FoodSafety.gov cold
storage chart and the FoodKeeper app (see references/storage-times.md). Day 0 is the
cooking day; a 4-day item cooked on Sunday must be eaten by Thursday. Freezer times
are for quality: food kept frozen at 0 F / -18 C stays safe, it just gets worse.

Usage:
  python3 prep_storage.py BATCH.csv PLAN.csv [--fridge-capacity 12] [--json]
  python3 prep_storage.py --categories
  python3 prep_storage.py --self-test
BATCH.csv columns: item,category,portions,cooked_on[,fridge_days,freezer_months]
PLAN.csv  columns: date,meal,item,portions   (dates as YYYY-MM-DD)
Exit codes: 0 = plan is safe, 1 = at least one HIGH finding, 2 = bad input.
"""
import argparse, csv, datetime as dt, io, json, sys

SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2, "INFO": 3}
MEAL_ORDER = {"prep": 0, "breakfast": 1, "lunch": 2, "snack": 3, "dinner": 4, "evening": 5}
SERVE_COLD = {"deli_salad", "hard_boiled_eggs", "cut_raw_vegetables"}
# safe minimum internal temperatures for raw categories (cook on the planned day)
COOK_TEMP = {"raw_ground_meat": "160 F / 71 C", "raw_poultry": "165 F / 74 C",
             "raw_steak_chops": "145 F / 63 C plus 3 min rest", "raw_fish": "145 F / 63 C"}
# category: (fridge_days, freezer_months or None if it does not freeze well, note)
CATEGORIES = {
    "cooked_meat":       (4, 3, "cooked beef, pork, lamb; chart 3-4 days / 2-6 months"),
    "cooked_poultry":    (4, 3, "cooked chicken, turkey; chart 3-4 days / 2-6 months"),
    "soup_stew":         (4, 3, "soups, stews, chili, curries; chart 3-4 days / 2-3 months"),
    "casserole":         (4, 3, "casseroles, lasagna, bakes; 3-4 days / 2-3 months"),
    "cooked_grains":     (4, 2, "cooked rice, pasta, quinoa; cool fast; 3-5 days / 1-2 months"),
    "cooked_vegetables": (4, 2, "roasted or steamed vegetables, cooked beans; 3-4 days used"),
    "cooked_fish":       (3, 3, "cooked fish and seafood; 3-4 days, 3 used"),
    "pizza":             (4, 2, "pizza, savory pies; chart 3-4 days / 1-2 months"),
    "deli_salad":        (4, None, "egg, chicken, tuna, pasta salads; chart 3-5 days, does not freeze well"),
    "hard_boiled_eggs":  (7, None, "hard-cooked eggs; chart 1 week, do not freeze"),
    "cut_raw_vegetables": (4, None, "washed and cut raw vegetables; quality guide 3-5 days; blanch before freezing"),
    "raw_ground_meat":   (2, 3, "raw ground meat or marinated mince; chart 1-2 days / 3-4 months"),
    "raw_poultry":       (2, 9, "raw chicken or turkey pieces, marinated; chart 1-2 days / 9 months"),
    "raw_steak_chops":   (4, 4, "raw steaks, chops, roasts; chart 3-5 days / 4-12 months"),
    "raw_fish":          (2, 2, "raw fish; chart 1-2 days / 2-3 months fatty, 6-8 months lean"),
}


class InputError(Exception):
    pass


def parse_date(s, where):
    try:
        return dt.date.fromisoformat(s.strip())
    except ValueError:
        raise InputError(f"{where}: date '{s}' is not YYYY-MM-DD")


def parse_int(s, where, minimum=0):
    try:
        v = int(str(s).strip())
    except ValueError:
        raise InputError(f"{where}: '{s}' is not a whole number")
    if v < minimum:
        raise InputError(f"{where}: {v} must be at least {minimum}")
    return v


def read_rows(text, required, name):
    rows = list(csv.DictReader(io.StringIO(text.lstrip("\ufeff"))))
    if not rows:
        raise InputError(f"{name} has no rows")
    missing = [c for c in required if c not in rows[0]]
    if missing:
        raise InputError(f"{name} is missing column(s): {', '.join(missing)}")
    return [{k.strip(): (v or "").strip() for k, v in r.items() if k} for r in rows if any((v or "").strip() for v in r.values())]


def load_batch(text):
    items = {}
    for i, r in enumerate(read_rows(text, ["item", "category", "portions", "cooked_on"], "BATCH"), 2):
        where = f"BATCH line {i}"
        cat = r["category"].lower()
        if cat not in CATEGORIES and not r.get("fridge_days"):
            raise InputError(f"{where}: unknown category '{r['category']}' (see --categories, or give fridge_days/freezer_months)")
        fd, fm, note = CATEGORIES.get(cat, (None, None, "custom"))
        if r.get("fridge_days"):
            fd = parse_int(r["fridge_days"], where, 1)
        if r.get("freezer_months") is not None and r.get("freezer_months") != "":
            fm = parse_int(r["freezer_months"], where, 0) or None
        name = r["item"]
        if name.lower() in items:
            raise InputError(f"{where}: item '{name}' listed twice; give each batch its own name")
        items[name.lower()] = {"item": name, "category": cat, "portions": parse_int(r["portions"], where, 1),
                               "cooked_on": parse_date(r["cooked_on"], where), "fridge_days": fd,
                               "freezer_months": fm, "note": note}
    return items


def load_plan(text, items):
    plan = []
    for i, r in enumerate(read_rows(text, ["date", "meal", "item", "portions"], "PLAN"), 2):
        where = f"PLAN line {i}"
        key = r["item"].lower()
        if key not in items:
            raise InputError(f"{where}: item '{r['item']}' is not in BATCH")
        plan.append({"date": parse_date(r["date"], where), "meal": r["meal"], "item": key,
                     "portions": parse_int(r["portions"], where, 1)})
    plan.sort(key=lambda p: p["date"])
    return plan


def add_months(d, months):
    y, m = divmod(d.month - 1 + months, 12)
    y, m = d.year + y, m + 1
    last = (dt.date(y + (m == 12), m % 12 + 1, 1) - dt.timedelta(days=1)).day
    return dt.date(y, m, min(d.day, last))


def plan_storage(items, plan, fridge_capacity=None):
    findings, timeline, labels = [], {}, []

    def add(sev, code, item, detail):
        findings.append({"severity": sev, "code": code, "item": item, "detail": detail})

    def ev(day, slot, text):
        timeline.setdefault(day, []).append((MEAL_ORDER.get(slot.lower(), 3), text))

    summary = {}
    for key, it in items.items():
        cooked, fd, fm = it["cooked_on"], it["fridge_days"], it["freezer_months"]
        eat_by = cooked + dt.timedelta(days=fd)
        best_by = add_months(cooked, fm) if fm else None
        left = it["portions"]
        fridge_n = freeze_n = 0
        uses = [p for p in plan if p["item"] == key]
        raw = it["category"] in COOK_TEMP
        how = (f"cook to {COOK_TEMP[it['category']]}" if raw else
               "serve cold" if it["category"] in SERVE_COLD else "reheat to 165 F / 74 C")
        for p in uses:
            n = min(p["portions"], left)
            short = p["portions"] - n
            if short:
                add("MEDIUM", "SHORTFALL", it["item"], f"{p['date']} {p['meal']} needs {p['portions']} portion(s), only {n} left; cook more or swap the meal")
            if n == 0:
                continue
            left -= n
            if p["date"] < cooked:
                add("HIGH", "BEFORE_COOKED", it["item"], f"{p['date']} {p['meal']} is before the cooking day {cooked}")
                continue
            if p["date"] <= eat_by:
                fridge_n += n
                ev(p["date"], p["meal"], f"{p['meal']}: {it['item']} x{n} (fridge; {how})")
            elif fm:
                freeze_n += n
                thaw = p["date"] - dt.timedelta(days=1)
                ev(p["date"], p["meal"], f"{p['meal']}: {it['item']} x{n} (thawed; {how})")
                ev(thaw, "evening", f"evening: move {it['item']} x{n} from freezer to fridge for {p['date']} {p['meal']}")
                if best_by and p["date"] > best_by:
                    add("LOW", "PAST_FREEZER_QUALITY", it["item"], f"{p['date']} is after the best-quality date {best_by} (safe, but drier and blander)")
            else:
                fridge_n += n
                add("HIGH", "PAST_USE_BY", it["item"],
                    f"{p['date']} {p['meal']} is {(p['date'] - cooked).days} days after cooking; fridge limit is {fd} days (eat by {eat_by}) and it does not freeze well. Move the meal earlier or cook it fresh.")
        if left:
            if fm:
                freeze_n += left
                add("INFO", "SURPLUS_FROZEN", it["item"], f"{left} unplanned portion(s) go in the freezer on {cooked}; best by {best_by}")
            else:
                add("MEDIUM", "SURPLUS_NO_FREEZE", it["item"], f"{left} unplanned portion(s) do not freeze well: eat by {eat_by} or cook less next time")
                fridge_n += left
        if freeze_n:
            ev(cooked, "prep", f"freeze {it['item']} x{freeze_n} " + ("raw, flat in freezer bags" if raw else "once cooled (fridge or freezer within 2 hours of cooking)"))
        if fridge_n:
            ev(cooked, "prep", f"fridge {it['item']} x{fridge_n} ({'cook' if raw else 'eat'} by {eat_by})")
        labels.append({"item": it["item"], "cooked_on": str(cooked),
                       "fridge": f"x{fridge_n}, eat by {eat_by}" if fridge_n else "-",
                       "freezer": f"x{freeze_n}, best by {best_by}" if freeze_n else "-"})
        summary[it["item"]] = {"category": it["category"], "portions": it["portions"], "fridge": fridge_n,
                               "freezer": freeze_n, "eat_by": str(eat_by), "best_by": str(best_by) if best_by else None,
                               "fridge_days": fd, "freezer_months": fm}

    if fridge_capacity:
        days = sorted(set([it["cooked_on"] for it in items.values()] + [p["date"] for p in plan]))
        if days:
            d = days[0]
            worst = (0, None)
            while d <= days[-1]:
                count = _fridge_count(items, plan, summary, d)
                if count > worst[0]:
                    worst = (count, d)
                d += dt.timedelta(days=1)
            if worst[0] > fridge_capacity:
                add("MEDIUM", "FRIDGE_FULL", "-", f"{worst[0]} containers in the fridge on {worst[1]}, capacity {fridge_capacity}: freeze more on prep day or use bigger containers")
    findings.sort(key=lambda f: (SEV_ORDER[f["severity"]], f["item"], f["code"]))
    return {"summary": summary, "timeline": {str(k): [t for _, t in sorted(v, key=lambda x: x[0])] for k, v in sorted(timeline.items())},
            "labels": labels, "findings": findings}


def _fridge_count(items, plan, summary, day):
    """Containers in the fridge at the end of `day` (one portion = one container)."""
    total = 0
    for key, it in items.items():
        if day < it["cooked_on"]:
            continue
        eat_by = it["cooked_on"] + dt.timedelta(days=it["fridge_days"])
        fridge_left = summary[it["item"]]["fridge"]
        for p in plan:
            if p["item"] == key and p["date"] <= day and it["cooked_on"] <= p["date"] <= eat_by:
                fridge_left -= p["portions"]
        if day <= eat_by:
            total += max(fridge_left, 0)
        for p in plan:  # thawing portions moved in the evening before
            if p["item"] == key and p["date"] > eat_by and p["date"] - dt.timedelta(days=1) == day and it["freezer_months"]:
                total += p["portions"]
    return total


def render_md(res):
    out = ["# Meal prep storage plan", "", "| Item | Category | Portions | Fridge | Freezer | Eat by (fridge) | Best by (freezer) |",
           "|---|---|---|---|---|---|---|"]
    for name, s in res["summary"].items():
        out.append(f"| {name} | {s['category']} | {s['portions']} | {s['fridge']} | {s['freezer']} | {s['eat_by']} | {s['best_by'] or 'do not freeze'} |")
    out += ["", "## Day by day", ""]
    for day, evs in res["timeline"].items():
        d = dt.date.fromisoformat(day)
        out.append(f"- **{d.strftime('%a')} {day}**: " + "; ".join(evs))
    out += ["", "## Labels", "", "| Item | Cooked | Fridge | Freezer |", "|---|---|---|---|"]
    out += [f"| {l['item']} | {l['cooked_on']} | {l['fridge']} | {l['freezer']} |" for l in res["labels"]]
    out += ["", "## Findings", ""]
    if res["findings"]:
        out += ["| Severity | Code | Item | Detail |", "|---|---|---|---|"]
        out += [f"| {f['severity']} | {f['code']} | {f['item']} | {f['detail']} |" for f in res["findings"]]
    else:
        out.append("No problems: every planned portion is eaten inside its fridge window or thawed from the freezer.")
    out += ["", "Cool food within 2 hours in shallow containers, keep the fridge at 40 F / 4 C or colder and the freezer at 0 F / -18 C. When in doubt, throw it out."]
    return "\n".join(out)


# ---------------------------------------------------------------- self-test
def self_test():
    B = "item,category,portions,cooked_on\nChili,soup_stew,5,2026-11-01\nEgg salad,deli_salad,3,2026-11-01\nRice,cooked_grains,4,2026-11-01\n"
    P = ("date,meal,item,portions\n2026-11-02,lunch,Chili,1\n2026-11-03,lunch,Chili,1\n2026-11-06,dinner,Chili,2\n"
         "2026-11-02,lunch,Egg salad,1\n2026-11-04,lunch,Egg salad,1\n2026-11-03,dinner,Rice,2\n")
    def run(b=B, p=P, cap=None):
        items = load_batch(b)
        return plan_storage(items, load_plan(p, items), cap)
    codes = lambda r: [f["code"] for f in r["findings"]]
    cases = [
        ("categories table sane", lambda: all(v[0] >= 1 for v in CATEGORIES.values())),
        ("add_months end of month", lambda: add_months(dt.date(2026, 1, 31), 1) == dt.date(2026, 2, 28)),
        ("add_months over year", lambda: add_months(dt.date(2026, 11, 15), 3) == dt.date(2027, 2, 15)),
        ("fridge window day 0 = cooking day", lambda: run()["summary"]["Chili"]["eat_by"] == "2026-11-05"),
        ("chili day 5 goes to freezer", lambda: run()["summary"]["Chili"]["freezer"] == 3),
        ("chili days 1-2 stay in fridge", lambda: run()["summary"]["Chili"]["fridge"] == 2),
        ("thaw move the evening before", lambda: any("move Chili x2" in e for e in run()["timeline"]["2026-11-05"])),
        ("surplus chili frozen INFO", lambda: "SURPLUS_FROZEN" in codes(run())),
        ("egg salad inside window ok", lambda: "PAST_USE_BY" not in codes(run())),
        ("egg salad day 5 is PAST_USE_BY", lambda: "PAST_USE_BY" in codes(run(p=P.replace("2026-11-04,lunch,Egg", "2026-11-06,lunch,Egg")))),
        ("egg salad surplus no freeze", lambda: "SURPLUS_NO_FREEZE" in codes(run())),
        ("shortfall", lambda: "SHORTFALL" in codes(run(p=P + "2026-11-04,dinner,Rice,3\n"))),
        ("before cooked", lambda: "BEFORE_COOKED" in codes(run(p=P + "2026-10-31,dinner,Rice,1\n"))),
        ("past freezer quality", lambda: "PAST_FREEZER_QUALITY" in codes(run(p=P + "2027-03-01,dinner,Rice,1\n"))),
        ("override fridge_days column", lambda: run(b="item,category,portions,cooked_on,fridge_days,freezer_months\nSoup,custom,2,2026-11-01,2,0\n",
                                                   p="date,meal,item,portions\n2026-11-04,lunch,Soup,1\n")["findings"][0]["code"] == "PAST_USE_BY"),
        ("unknown category error", lambda: _raises(lambda: run(b="item,category,portions,cooked_on\nX,mystery,1,2026-11-01\n", p="date,meal,item,portions\n"))),
        ("bad date error", lambda: _raises(lambda: run(p="date,meal,item,portions\n11/02/2026,lunch,Chili,1\n"))),
        ("unknown plan item error", lambda: _raises(lambda: run(p="date,meal,item,portions\n2026-11-02,lunch,Tacos,1\n"))),
        ("zero portions error", lambda: _raises(lambda: run(b="item,category,portions,cooked_on\nChili,soup_stew,0,2026-11-01\n"))),
        ("duplicate batch item error", lambda: _raises(lambda: run(b=B + "chili,soup_stew,2,2026-11-01\n"))),
        ("missing column error", lambda: _raises(lambda: run(b="item,portions\nChili,2\n"))),
        ("case-insensitive item names", lambda: run(p=P.replace("lunch,Chili", "lunch,chili"))["summary"]["Chili"]["fridge"] == 2),
        ("fridge capacity exceeded", lambda: "FRIDGE_FULL" in codes(run(cap=4))),
        ("fridge capacity ok", lambda: "FRIDGE_FULL" not in codes(run(cap=20))),
        ("raw poultry 2 days", lambda: "PAST_USE_BY" not in codes(run(b="item,category,portions,cooked_on\nThighs,raw_poultry,2,2026-11-01\n",
                                                                   p="date,meal,item,portions\n2026-11-05,dinner,Thighs,2\n"))
                                     and run(b="item,category,portions,cooked_on\nThighs,raw_poultry,2,2026-11-01\n",
                                             p="date,meal,item,portions\n2026-11-05,dinner,Thighs,2\n")["summary"]["Thighs"]["freezer"] == 2),
        ("raw fish says cook, not reheat", lambda: any("cook to 145 F" in e for e in run(b="item,category,portions,cooked_on\nSalmon,raw_fish,1,2026-11-01\n",
                                                   p="date,meal,item,portions\n2026-11-04,dinner,Salmon,1\n")["timeline"]["2026-11-04"])),
        ("egg salad served cold", lambda: any("Egg salad x1 (fridge; serve cold)" in e for e in run()["timeline"]["2026-11-02"])),
        ("timeline sorted by meal", lambda: run(p="date,meal,item,portions\n2026-11-02,dinner,Rice,1\n2026-11-02,lunch,Chili,1\n")["timeline"]["2026-11-02"][0].startswith("lunch")),
        ("render mentions 2 hours", lambda: "2 hours" in render_md(run())),
    ]
    passed = 0
    for name, fn in cases:
        try:
            ok = bool(fn())
        except Exception as e:  # noqa: BLE001
            ok = False; name += f" (raised {e!r})"
        passed += ok
        print(("PASS " if ok else "FAIL ") + name)
    print(f"{passed}/{len(cases)} passed")
    return 0 if passed == len(cases) else 1


def _raises(fn):
    try:
        fn()
    except InputError:
        return True
    return False


def main():
    ap = argparse.ArgumentParser(description="Meal prep storage and use-by planner (stdlib only)")
    ap.add_argument("batch", nargs="?"); ap.add_argument("plan", nargs="?")
    ap.add_argument("--fridge-capacity", type=int, help="max containers your fridge shelf can hold")
    ap.add_argument("--json", action="store_true")
    ap.add_argument("--categories", action="store_true", help="list categories and storage times")
    ap.add_argument("--self-test", action="store_true")
    a = ap.parse_args()
    if a.self_test:
        return self_test()
    if a.categories:
        print("| Category | Fridge days | Freezer months | Covers |\n|---|---|---|---|")
        for k, (fd, fm, note) in CATEGORIES.items():
            print(f"| {k} | {fd} | {fm if fm else 'do not freeze'} | {note} |")
        return 0
    if not (a.batch and a.plan):
        ap.error("need BATCH.csv PLAN.csv (or --self-test / --categories)")
    try:
        with open(a.batch, encoding="utf-8") as f1, open(a.plan, encoding="utf-8") as f2:
            items = load_batch(f1.read())
            plan = load_plan(f2.read(), items)
        res = plan_storage(items, plan, a.fridge_capacity)
    except (OSError, UnicodeDecodeError, InputError) as e:
        print(f"error: {e}", file=sys.stderr)
        return 2
    print(json.dumps(res, indent=2) if a.json else render_md(res))
    return 1 if any(f["severity"] == "HIGH" for f in res["findings"]) else 0


if __name__ == "__main__":
    sys.exit(main())

---
name: csv-diff-and-reconciliation-checker
description: Compares two CSV exports of the same records (old vs new, billing vs ledger, source vs warehouse) by key after normalizing headers, delimiters, currency formats, accounting negatives, dates and case. Reports added, removed and duplicate keys, key format drift, changed values per column with tolerances, possible re-keys and a control-total bridge, then writes a reconciliation report. Includes a stdlib Python checker with --self-test.
---

# CSV Diff and Reconciliation Checker

You help analysts, finance teams and data engineers answer "do these two files agree, and if not, exactly why?". You match rows by a business key, ignore differences that are only formatting, and turn the real differences into a short list of causes with the amounts that each one explains, so the totals tie out to the cent.

## Files in this skill

- `scripts/csv_recon.py` - matches two CSVs by key (single or composite), normalizes values, lists schema differences, duplicate and blank keys, added, removed and changed rows, key format drift and possible re-keys, and builds a control-total bridge (Python 3 standard library only)
- `references/matching-and-normalization.md` - choosing a key, header and value normalization, number and date formats, tolerances, and what each finding code means
- `references/reconciliation-playbook.md` - common causes of breaks (timing, duplicates, sign conventions, rounding, filters, encoding) and how to prove each one
- `templates/reconciliation-report.md` - the report to hand back
- `examples/billing-export-sept.csv` - September invoices from a billing system (14 rows, comma-separated)
- `examples/ledger-export-sept.csv` - the same month from the general ledger (15 rows, semicolons, $1,200.00 amounts, US dates, renamed and extra columns)

## Workflow

### 1. Understand the two files
Ask what each file is, which system and filters produced it (date range, status, entity, currency), which side is the source of truth, what the key is, and which differences matter (amounts to the cent, statuses, dates). Ask for row counts and totals the user already expects. Look at the first lines of both files before running anything.

### 2. Run the checker
```
python3 scripts/csv_recon.py examples/billing-export-sept.csv examples/ledger-export-sept.csv --key invoice_id --map customer=customer_name --tol amount=0.01 --sum amount --ignore-case --dates invoice_date
python3 scripts/csv_recon.py orders_old.csv orders_new.csv --key order_id,line_no --ignore updated_at --out-csv diff.csv
python3 scripts/csv_recon.py a.csv b.csv --key id --json
python3 scripts/csv_recon.py --self-test
```
Headers that differ only by case or spacing match automatically; renamed columns need `--map old=new`. Exit 0 means the files reconcile, 1 means differences, 2 means bad input (missing file or key column).

### 3. Explain every difference
Work down the findings (see `references/reconciliation-playbook.md`): duplicates and blank keys first (they distort everything else), then added and removed rows (timing, filters, re-keys), then changed values grouped by column and by pattern (same delta, sign flip, digits swapped, rounding). Use the bridge: old total + added - removed + changes - duplicates = new total, with nothing unexplained. Check a sample of rows by hand in both source systems.

### 4. Deliver
Fill `templates/reconciliation-report.md`: scope and inputs, result, summary counts, the bridge, each break with cause, amount, owner and fix, and what to change so next month reconciles automatically (stable keys, same filters, export formats). Attach `--out-csv` output for the people who fix the rows.

## Example result
Billing vs ledger for September: 14 vs 15 rows, 13 matched (INV-1009 matched to `inv-1009` as KEY_FORMAT_DRIFT), 11 unchanged. Breaks: INV-1011 posted twice in the ledger (HIGH DUPLICATE_KEY, +880.00), INV-1004 450.00 vs 405.00 (digits swapped, -45.00), INV-1012 open in billing but Paid in the ledger, INV-1007 missing and INV-1015 added with the same customer and amount (POSSIBLE_REKEY: posted in October under a new number). $ signs, thousands separators, (150.00) negatives, US dates and status case are formatting only; the INV-1003 rounding (1033.335 vs 1033.34) is inside the 0.01 tolerance. Bridge: 9545.965 + 310.00 - 310.00 - 44.995 + 880.00 = 10380.97, unexplained 0.

## Rules

- Never compare by row position; agree on a key first and check it is unique on both sides.
- Separate formatting noise from real differences and say which normalizations you applied.
- Every break gets a cause, an amount and an owner; "timing" is a cause only if you name the period it lands in.
- Do not net differences against each other to make totals tie; list them.
- Keep tolerances explicit and small (rounding only), never large enough to hide a real error.
- Treat customer and personal data in the files as confidential; share only what the report needs.
FILE:references/matching-and-normalization.md
# Matching and normalization

## Choosing the key
- Use a business key that both systems carry: invoice number, order id + line number, transaction id. Composite keys: `--key order_id,line_no`.
- Check uniqueness on both sides before anything else. Duplicates mean either a real double posting or a key that is not unique (e.g. invoice_id without line number).
- Avoid keys that change between systems: row numbers, surrogate database ids, names, timestamps.
- If there is no shared key, build one (date + amount + reference) and expect some ambiguous matches; review them by hand.

## Header matching
- Exact name first, then case/space/underscore/hyphen-insensitive (`Invoice ID` = `invoice_id`), reported as HEADER_FORMAT.
- Renamed columns: `--map customer=customer_name`. Columns only on one side are reported and not compared.
- A UTF-8 byte order mark (BOM) at the start of the file is removed; otherwise the first header would not match.

## Delimiters and encoding
- Comma, semicolon, tab and pipe are detected. European Excel exports often use `;` with `,` as decimal separator; this checker expects `.` decimals, so convert those first (or export again with `.`).
- Read files as UTF-8. If you see `é` instead of `é`, the file is Windows-1252: convert with `iconv -f cp1252 -t utf-8`.

## Value normalization (applied to every compared cell)
| Raw | Treated as |
|---|---|
| leading/trailing spaces, repeated inner spaces | trimmed, single space |
| `1,234.50`, `$1,234.50`, `1234.5`, `1234.50 USD` | number 1234.5 |
| `(150.00)`, `150.00-`, `-$150.00` | number -150 |
| `12.5%` | number 12.5 (not 0.125) |
| empty, `null`, `None`, `N/A`, `NA`, `-` | empty |
| `Paid` vs `paid` | different, unless `--ignore-case` |
| `09/01/2026`, `2026/09/01`, `01.09.2026` | 2026-09-01 for columns in `--dates` (US month-first unless `--dayfirst`) |

Numbers are compared as decimals, not floats, so `0.1 + 0.2` problems do not appear. Leading zeros in numeric-looking codes (`00123` vs `123`) compare equal as numbers; if that is wrong for a code column, add it to `--ignore` and compare it separately.

## Tolerances
- `--tol amount=0.01` per column, or `--tol 0.01` for every numeric column.
- Use the smallest unit of the currency for rounding differences; never use a tolerance to hide a known error.
- Values inside the tolerance are counted as FORMAT_ONLY and still appear in the bridge as changes.

## Finding codes
| Code | Severity | Meaning |
|---|---|---|
| DUPLICATE_KEY | HIGH | the same key appears more than once on one side; only the first row is compared |
| MISSING_KEY | HIGH | rows with a blank key; they cannot be matched |
| TOTAL_UNEXPLAINED | HIGH | the bridge does not tie (non-numeric cells, blank keys) |
| COLUMN_ONLY_IN_OLD / NEW | MEDIUM | schema difference; map it if it was renamed |
| ROWS_ADDED / ROWS_REMOVED | MEDIUM | keys on one side only |
| VALUE_CHANGED | MEDIUM | per column, rows whose normalized values differ |
| RAGGED_ROWS | MEDIUM | rows with more or fewer cells than the header (bad quoting, embedded delimiters) |
| NON_NUMERIC | MEDIUM | cells in a --sum column that are not numbers |
| HEADER_FORMAT | LOW | columns matched only after ignoring case and spacing |
| KEY_FORMAT_DRIFT | LOW | keys equal after ignoring case, spaces, separators and leading zeros; compared as one row |
| POSSIBLE_REKEY | LOW | a removed and an added row with the same non-key, non-date values |
| FORMAT_ONLY, DELIMITER, TOTAL_DIFF | INFO | context: equal after normalization, different delimiters, totals moved |

## The control-total bridge
```
OLD total
+ amounts of rows only in NEW
- amounts of rows only in OLD
+ (NEW - OLD) on matched rows
- extra duplicate or blank-key rows in OLD
+ extra duplicate or blank-key rows in NEW
= NEW total            (unexplained must be 0)
```
FILE:references/reconciliation-playbook.md
# Reconciliation playbook: causes of breaks and how to prove them

Work in this order: duplicates and blank keys, then rows on one side only, then changed values, then totals.

## Duplicates
- **Double posting** (same key, same values twice): a retry, an import run twice, a manual re-entry. Prove: identical rows, often the same created timestamp or batch id. Fix: reverse one posting; add an idempotency key to the import.
- **Non-unique key** (same key, different values): the key needs another column (line number, currency, entity). Re-run with a composite key.

## Rows on one side only
- **Timing / cut-off**: the record exists in both systems but in different periods (invoice dated 30 Sep, posted 1 Oct). Prove: find it in the next or previous period export. Typical sign: POSSIBLE_REKEY with a date just across the boundary.
- **Different filters**: one export excludes drafts, voids, test accounts, zero amounts, another entity or currency. Prove: compare the export queries or report settings; count the excluded statuses.
- **Re-keyed records**: credit note and re-issue, migrated ids, prefix changes (INV- vs IN-). Prove: same customer, amount and date with a new key.
- **Failed sync**: the record never reached the target. Prove: integration logs, error queue, last successful sync time.
- **Key format**: case, spaces, leading zeros, separators (KEY_FORMAT_DRIFT). Fix at the source; normalize keys in the export.

## Changed values
| Pattern | Likely cause |
|---|---|
| delta is a multiple of 9 (450 -> 405, 1,234 -> 1,324) | transposed digits on manual entry |
| exact sign flip | debit/credit convention, credit notes stored as positive with a type column |
| delta equals tax or fee rate x amount | one side gross, the other net |
| delta of 0.01 to 0.05 | rounding per line vs on the total; half-up vs banker's rounding |
| amount / rate | currency conversion, different rate date |
| date off by one day | time zone (UTC vs local) on timestamps near midnight |
| status differs | sync lag, or the systems use different state machines (paid vs settled) |
| text differs only in accents or odd characters | encoding (UTF-8 vs Windows-1252) |

## Totals
- Tie the bridge to zero before reporting. A non-zero unexplained amount means blank keys, non-numeric cells or a column that mixes currencies.
- Report totals per currency; never add USD and EUR.
- Check the totals against the system's own report (trial balance, payout report), not just against the export.

## Proving a cause
For each break: show the row(s) on both sides, the cause, the amount, and evidence (log line, export setting, the row in the next period). Mark each as: fix in source A, fix in source B, timing (will clear), or accepted difference.

## Preventing next month's breaks
- Same filters and period definition (time zone, inclusive end date) on both exports.
- Stable, unique keys carried end to end; no manual re-keying.
- Export amounts as plain numbers with `.` decimals and a separate currency column; dates as ISO 8601.
- Run the checker on a schedule and alert on HIGH findings or unexplained totals.
FILE:templates/reconciliation-report.md
# Reconciliation: <OLD source> vs <NEW source>, <period>

**Inputs:** <billing-export-sept.csv (14 rows, exported 2026-10-02 by ...)>, <ledger-export-sept.csv (15 rows)>
**Key:** <invoice_id>   **Compared columns:** <customer, amount, currency, status, invoice_date>
**Normalization:** <$ and thousands separators removed, (x) = -x, status case ignored, US dates read as dates, tolerance amount 0.01>
**Result:** <RECONCILES / DIFFERENCES FOUND: 4 breaks, net +835.005, unexplained 0>

## Summary
| Rows OLD | Rows NEW | Matched | Unchanged | Changed | Added | Removed | Duplicates |
|---|---|---|---|---|---|---|---|
| <14> | <15> | <13> | <11> | <2> | <1> | <1> | <1> |

## Control total bridge (<amount>, <USD>)
| Step | Amount |
|---|---|
| OLD total | <9,545.97> |
| + rows only in NEW | <310.00> |
| - rows only in OLD | <310.00> |
| + changes on matched rows | <-45.00> |
| + duplicates in NEW | <880.00> |
| = NEW total | <10,380.97> |
| Unexplained | <0.00> |

## Breaks
| # | Key(s) | Type | OLD | NEW | Amount | Cause (evidence) | Owner | Action |
|---|---|---|---|---|---|---|---|---|
| 1 | <INV-1011> | <Duplicate> | <880.00 x1> | <880.00 x2> | <+880.00> | <posted twice by import retry (batch 7731)> | <GL team> | <reverse second posting> |

## Formatting-only differences (no action)
- <amount formats, status case, date formats, header names>

## Recommendations
- <same filters and period definition on both exports>
- <add an idempotency key to the ledger import>

## Attachments
- <diff.csv from --out-csv (status, key, column, old, new, delta)>
FILE:examples/billing-export-sept.csv
invoice_id,customer,amount,currency,status,invoice_date
INV-1001,Northwind Traders,1200.00,USD,paid,2026-09-01
INV-1002,Blue Harbor Cafe,89.90,USD,paid,2026-09-02
INV-1003,Atlas Fitness,1033.335,USD,paid,2026-09-03
INV-1004,Greenleaf Dental,450.00,USD,paid,2026-09-05
INV-1005,Orbit Labs,2500.00,USD,paid,2026-09-08
INV-1006,Maple & Co,74.50,USD,refunded,2026-09-09
INV-1007,Copperline Studio,310.00,USD,paid,2026-09-11
INV-1008,Riverbend School,640.00,USD,paid,2026-09-12
INV-1009,Juniper Vet Clinic,195.00,USD,paid,2026-09-15
INV-1010,Northwind Traders,1200.00,USD,paid,2026-09-18
INV-1011,Summit Roofing,880.00,USD,paid,2026-09-22
INV-1012,Blue Harbor Cafe,89.90,USD,open,2026-09-25
INV-1013,Orbit Labs,-150.00,USD,credit,2026-09-26
INV-1014,Atlas Fitness,1033.33,USD,paid,2026-09-30
FILE:examples/ledger-export-sept.csv
Invoice ID;customer_name;Amount;Currency;Status;Invoice Date;gl_account
INV-1001;Northwind Traders;$1,200.00;USD;Paid;09/01/2026;4000
INV-1002;Blue Harbor Cafe;$89.90;USD;Paid;09/02/2026;4000
INV-1003;Atlas Fitness;$1,033.34;USD;Paid;09/03/2026;4000
INV-1004;Greenleaf Dental;$405.00;USD;Paid;09/05/2026;4000
INV-1005;Orbit Labs;$2,500.00;USD;Paid;09/08/2026;4000
INV-1006;Maple & Co;$74.50;USD;Refunded;09/09/2026;4000
INV-1008;Riverbend School;$640.00;USD;Paid;09/12/2026;4000
inv-1009;Juniper Vet Clinic;$195.00;USD;Paid;09/15/2026;4000
INV-1010;Northwind Traders;$1,200.00;USD;Paid;09/18/2026;4000
INV-1011;Summit Roofing;$880.00;USD;Paid;09/22/2026;4000
INV-1011;Summit Roofing;$880.00;USD;Paid;09/22/2026;4000
INV-1012;Blue Harbor Cafe;$89.90;USD;Paid;09/25/2026;4000
INV-1013;Orbit Labs;($150.00);USD;Credit;09/26/2026;4090
INV-1014;Atlas Fitness;$1,033.33;USD;Paid;09/30/2026;4000
INV-1015;Copperline Studio;$310.00;USD;Paid;10/01/2026;4000
FILE:scripts/csv_recon.py
#!/usr/bin/env python3
"""CSV diff and reconciliation checker (Python 3 standard library only).

Compares two CSV exports of the same records (old vs new, system A vs system B) by a
key, after normalizing the noise that makes naive diffs useless: spaces, header case,
thousands separators, currency signs, accounting negatives "(12.50)", number
formatting, optional text case and date formats. Reports schema differences, missing
or duplicate keys, added and removed rows, changed values per column (with numeric
tolerances), keys that only differ by format (case, leading zeros, spaces) and a
control-total bridge for amount columns: old total + added - removed + changes = new.

Usage:
  python3 csv_recon.py OLD.csv NEW.csv --key id [--key2 line] [--map old_col=new_col]
        [--ignore col,col] [--tol amount=0.01] [--sum amount] [--ignore-case]
        [--dates col,col] [--dayfirst] [--limit 20] [--out-csv diff.csv] [--json]
  python3 csv_recon.py --self-test
Exit codes: 0 = files reconcile (no differences after normalization and tolerance),
1 = differences found, 2 = bad input (missing file, missing key column).
"""
import argparse, csv, datetime, io, json, re, sys
from decimal import Decimal, InvalidOperation

SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2, "INFO": 3}
NUM_RE = re.compile(r"^[+-]?(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?$")
DATE_FORMATS = ["%Y-%m-%d", "%Y/%m/%d", "%Y%m%d", "%d.%m.%Y", "%Y-%m-%dT%H:%M:%S", "%Y-%m-%d %H:%M:%S"]


class InputError(Exception):
    pass


def norm_header(h):
    return re.sub(r"[\s_\-]+", "_", h.strip().lstrip("\ufeff").lower())


def read_csv(text, name):
    text = text.lstrip("\ufeff")
    if not text.strip():
        raise InputError(f"{name} is empty")
    sample = text[:4096]
    try:
        dialect = csv.Sniffer().sniff(sample, delimiters=",;\t|")
    except csv.Error:
        dialect = csv.excel
    rows = list(csv.reader(io.StringIO(text), dialect))
    rows = [r for r in rows if any(c.strip() for c in r)]
    header = [h.strip() for h in rows[0]]
    body = []
    for i, r in enumerate(rows[1:], 2):
        if len(r) != len(header):
            body.append((i, dict(zip(header, r + [""] * (len(header) - len(r)))), len(r)))
        else:
            body.append((i, dict(zip(header, r)), len(r)))
    return header, body, getattr(dialect, "delimiter", ",")


def parse_number(s):
    """Return Decimal for '1,234.50', '$1,234.50', '(12.00)', '-5', '12.5%' (as 12.5), else None."""
    t = s.strip().replace("\u00a0", "").replace(" ", "")
    if not t:
        return None
    neg = False
    if t.startswith("(") and t.endswith(")"):
        neg, t = True, t[1:-1]
    if t.endswith("-") and NUM_RE.match(t[:-1].lstrip("$")):
        neg, t = True, t[:-1]
    t = re.sub(r"^([+-]?)(USD|EUR|GBP|[$\u20ac\u00a3])", r"\1", t)
    t = re.sub(r"(USD|EUR|GBP|[$\u20ac\u00a3])$", "", t)
    t = t.rstrip("%")
    if re.match(r"^[+-]?\d{1,3}(,\d{3})+(\.\d+)?$", t):
        t = t.replace(",", "")
    if not NUM_RE.match(t):
        return None
    try:
        d = Decimal(t)
    except InvalidOperation:
        return None
    return -d if neg else d


def parse_date(s, dayfirst):
    t = s.strip()
    fmts = DATE_FORMATS + (["%d/%m/%Y"] if dayfirst else ["%m/%d/%Y"])
    for f in fmts:
        try:
            return datetime.datetime.strptime(t, f).date().isoformat() if "%H" not in f else datetime.datetime.strptime(t, f).isoformat()
        except ValueError:
            pass
    return None


def normalize(value, col, opts):
    v = value.strip()
    if col in opts["dates"]:
        d = parse_date(v, opts["dayfirst"])
        if d:
            return ("date", d)
    n = parse_number(v)
    if n is not None:
        return ("num", n)
    if v.lower() in ("", "null", "none", "n/a", "na", "-"):
        return ("empty", "")
    v = re.sub(r"\s+", " ", v)
    return ("text", v.lower() if opts["ignore_case"] else v)


def same(a, b, col, opts):
    if a[0] == "num" and b[0] == "num":
        tol = opts["tol"].get(col, opts["tol"].get("*", Decimal(0)))
        return abs(a[1] - b[1]) <= tol
    return a == b


def key_variant(k):
    """Format-insensitive form of a key: case, spaces, separators and leading zeros."""
    t = re.sub(r"[\s\-_./]", "", k.lower())
    return re.sub(r"(?<!\d)0+(?=\d)", "", t)


def reconcile(old_text, new_text, keys, opts):
    oh, ob, odelim = read_csv(old_text, "OLD")
    nh, nb, ndelim = read_csv(new_text, "NEW")
    findings = []

    def add(sev, code, detail, count=1):
        findings.append({"severity": sev, "code": code, "detail": detail, "count": count})

    # column matching: explicit map, then exact, then normalized header
    cmap, fuzzy = {}, []
    n_by_norm = {norm_header(h): h for h in nh}
    for h in oh:
        if h in opts["map"]:
            target = opts["map"][h]
            if target not in nh:
                raise InputError(f"--map target column '{target}' not in NEW")
            cmap[h] = target
        elif h in nh:
            cmap[h] = h
        elif norm_header(h) in n_by_norm:
            cmap[h] = n_by_norm[norm_header(h)]
            fuzzy.append(f"{h} = {cmap[h]}")
    if fuzzy:
        add("LOW", "HEADER_FORMAT", f"{len(fuzzy)} column(s) matched by case/spacing only: " + "; ".join(fuzzy), len(fuzzy))
    only_old = [h for h in oh if h not in cmap]
    only_new = [h for h in nh if h not in cmap.values()]
    for h in only_old:
        add("MEDIUM", "COLUMN_ONLY_IN_OLD", f"'{h}' has no match in NEW (use --map {h}=<new name> if it was renamed)")
    for h in only_new:
        add("MEDIUM", "COLUMN_ONLY_IN_NEW", f"'{h}' has no match in OLD")
    for k in keys:
        if k not in oh:
            raise InputError(f"key column '{k}' not in OLD header {oh}")
        if cmap.get(k) not in nh:
            raise InputError(f"key column '{k}' has no match in NEW header {nh}")
    if odelim != ndelim:
        add("INFO", "DELIMITER", f"OLD uses '{odelim}', NEW uses '{ndelim}'")

    def index(body, side, colnames):
        idx, dups, blanks, ragged = {}, {}, 0, 0
        for line, row, ncells in body:
            if ncells != (len(oh) if side == "OLD" else len(nh)):
                ragged += 1
            kv = tuple(row.get(c, "").strip() for c in colnames)
            if not all(kv):
                blanks += 1
                continue
            if kv in idx:
                dups.setdefault(kv, [idx[kv][0]]).append(line)
            else:
                idx[kv] = (line, row)
        return idx, dups, blanks, ragged

    oidx, odups, oblank, orag = index(ob, "OLD", keys)
    nidx, ndups, nblank, nrag = index(nb, "NEW", [cmap[k] for k in keys])
    for side, dups in (("OLD", odups), ("NEW", ndups)):
        for kv, lines in dups.items():
            add("HIGH", "DUPLICATE_KEY", f"{side} key {'/'.join(kv)} on lines {', '.join(map(str, lines))} (only the first is compared)")
    for side, n in (("OLD", oblank), ("NEW", nblank)):
        if n:
            add("HIGH", "MISSING_KEY", f"{side} has {n} row(s) with a blank key (not compared)", n)
    for side, n in (("OLD", orag), ("NEW", nrag)):
        if n:
            add("MEDIUM", "RAGGED_ROWS", f"{side} has {n} row(s) with a different number of cells than the header", n)

    removed = [k for k in oidx if k not in nidx]
    added = [k for k in nidx if k not in oidx]
    # keys that only differ by format
    drift = []
    add_by_variant = {}
    for k in added:
        add_by_variant.setdefault(tuple(key_variant(x) for x in k), k)
    for k in list(removed):
        v = tuple(key_variant(x) for x in k)
        if v in add_by_variant:
            nk = add_by_variant.pop(v)
            drift.append((k, nk))
            removed.remove(k); added.remove(nk)
    for ok, nk in drift:
        add("LOW", "KEY_FORMAT_DRIFT", f"OLD key {'/'.join(ok)} = NEW key {'/'.join(nk)} after ignoring case, spaces, separators and leading zeros (compared as one row)")
    if removed:
        add("MEDIUM", "ROWS_REMOVED", f"{len(removed)} key(s) in OLD only: " + ", ".join('/'.join(k) for k in removed[:opts['limit']]), len(removed))
    if added:
        add("MEDIUM", "ROWS_ADDED", f"{len(added)} key(s) in NEW only: " + ", ".join('/'.join(k) for k in added[:opts['limit']]), len(added))

    compare_cols = [h for h in oh if h in cmap and h not in keys and h not in opts["ignore"]]
    pairs = [(k, k) for k in oidx if k in nidx] + drift
    changes, per_col, unchanged, fmt_only = [], {c: 0 for c in compare_cols}, 0, {c: 0 for c in compare_cols}
    for ok, nk in pairs:
        orow, nrow = oidx[ok][1], nidx[nk][1]
        row_changed = False
        for c in compare_cols:
            a_raw, b_raw = orow.get(c, ""), nrow.get(cmap[c], "")
            a, b = normalize(a_raw, c, opts), normalize(b_raw, c, opts)
            if same(a, b, c, opts):
                if a_raw.strip() != b_raw.strip():
                    fmt_only[c] += 1
                continue
            row_changed = True
            per_col[c] += 1
            delta = (b[1] - a[1]) if a[0] == "num" and b[0] == "num" else None
            changes.append({"key": "/".join(ok), "column": c, "old": a_raw, "new": b_raw,
                            "delta": str(delta) if delta is not None else ""})
        unchanged += not row_changed
    # removed and added rows that look like the same record under a new key or period
    nondate = [c for c in compare_cols if c not in opts["dates"]]
    used = set()
    for rk in removed:
        orow = oidx[rk][1]
        for ak in added:
            if ak in used or not nondate:
                continue
            nrow = nidx[ak][1]
            hits = sum(same(normalize(orow.get(c, ""), c, opts), normalize(nrow.get(cmap[c], ""), c, opts), c, opts) for c in nondate)
            if hits == len(nondate):
                used.add(ak)
                add("LOW", "POSSIBLE_REKEY", f"removed {'/'.join(rk)} and added {'/'.join(ak)} have the same values in {', '.join(nondate)} (re-keyed, re-posted or moved to another period?)")
                break
    for c, n in per_col.items():
        if n:
            add("MEDIUM", "VALUE_CHANGED", f"'{c}' differs in {n} row(s)", n)
    for c, n in fmt_only.items():
        if n:
            add("INFO", "FORMAT_ONLY", f"'{c}' equal after normalization in {n} row(s) (formatting, case or within tolerance)", n)

    bridges = []
    for col in opts["sum"]:
        if col not in oh or cmap.get(col) not in nh:
            raise InputError(f"--sum column '{col}' missing in OLD or NEW")
        def total(body, c):
            t, bad = Decimal(0), 0
            for _, row, _ in body:
                n = parse_number(row.get(c, ""))
                if n is None:
                    bad += bool(row.get(c, "").strip())
                else:
                    t += n
            return t, bad
        ot, obad = total(ob, col)
        nt, nbad = total(nb, cmap[col])
        def val(row, c):
            return parse_number(row.get(c, "")) or Decimal(0)
        s_add = sum((val(nidx[k][1], cmap[col]) for k in added), Decimal(0))
        s_rem = sum((val(oidx[k][1], col) for k in removed), Decimal(0))
        s_chg = sum((val(nidx[nk][1], cmap[col]) - val(oidx[ok][1], col) for ok, nk in pairs), Decimal(0))
        s_dup_o = sum((val(r, col) for _, r, _ in ob), Decimal(0)) - sum((val(r, col) for _, r in oidx.values()), Decimal(0))
        s_dup_n = sum((val(r, cmap[col]) for _, r, _ in nb), Decimal(0)) - sum((val(r, cmap[col]) for _, r in nidx.values()), Decimal(0))
        explained = ot + s_add - s_rem + s_chg - s_dup_o + s_dup_n
        unexplained = nt - explained
        bridges.append({"column": col, "old_total": str(ot), "added": str(s_add), "removed": str(s_rem),
                        "changed": str(s_chg), "duplicates_old": str(s_dup_o), "duplicates_new": str(s_dup_n),
                        "new_total": str(nt), "unexplained": str(unexplained), "non_numeric_old": obad, "non_numeric_new": nbad})
        if nt != ot:
            add("INFO", "TOTAL_DIFF", f"'{col}' total {ot} -> {nt} (diff {nt - ot}); see bridge")
        if unexplained != 0:
            add("HIGH", "TOTAL_UNEXPLAINED", f"'{col}' bridge leaves {unexplained} unexplained (blank keys or non-numeric cells?)")
        if obad or nbad:
            add("MEDIUM", "NON_NUMERIC", f"'{col}' has {obad} non-numeric cell(s) in OLD and {nbad} in NEW (left out of totals)", obad + nbad)

    findings.sort(key=lambda f: (SEV_ORDER[f["severity"]], f["code"]))
    summary = {"old_rows": len(ob), "new_rows": len(nb), "matched": len(pairs), "unchanged": unchanged,
               "changed_rows": len(pairs) - unchanged, "added": len(added), "removed": len(removed),
               "compared_columns": compare_cols, "column_map": {k: v for k, v in cmap.items() if k != v}}
    differs = any(f["severity"] in ("HIGH", "MEDIUM") for f in findings)
    return {"summary": summary, "findings": findings, "changes": changes, "bridges": bridges,
            "added_keys": ["/".join(k) for k in added], "removed_keys": ["/".join(k) for k in removed],
            "reconciles": not differs}


def render_md(res, old_name, new_name, limit):
    s = res["summary"]
    out = [f"# CSV reconciliation: {old_name} vs {new_name}", "",
           f"**Result:** {'RECONCILES' if res['reconciles'] else 'DIFFERENCES FOUND'}", "",
           "| Rows OLD | Rows NEW | Matched | Unchanged | Changed | Added | Removed |", "|---|---|---|---|---|---|---|",
           f"| {s['old_rows']} | {s['new_rows']} | {s['matched']} | {s['unchanged']} | {s['changed_rows']} | {s['added']} | {s['removed']} |", ""]
    if s["column_map"]:
        out.append("Column map: " + ", ".join(f"{k} -> {v}" for k, v in s["column_map"].items()))
        out.append("")
    out += ["## Findings", ""]
    if res["findings"]:
        out += ["| Severity | Code | Detail |", "|---|---|---|"]
        out += [f"| {f['severity']} | {f['code']} | {f['detail']} |" for f in res["findings"]]
    else:
        out.append("No findings.")
    if res["changes"]:
        out += ["", f"## Changed values (first {min(limit, len(res['changes']))} of {len(res['changes'])})", "",
                "| Key | Column | OLD | NEW | Delta |", "|---|---|---|---|---|"]
        out += [f"| {c['key']} | {c['column']} | {c['old']} | {c['new']} | {c['delta']} |" for c in res["changes"][:limit]]
    for b in res["bridges"]:
        out += ["", f"## Control total bridge: {b['column']}", "", "| Step | Amount |", "|---|---|",
                f"| OLD total | {b['old_total']} |", f"| + rows added | {b['added']} |", f"| - rows removed | {b['removed']} |",
                f"| + changes on matched rows | {b['changed']} |", f"| - duplicate/blank-key rows in OLD | {b['duplicates_old']} |",
                f"| + duplicate/blank-key rows in NEW | {b['duplicates_new']} |", f"| = NEW total | {b['new_total']} |",
                f"| Unexplained | {b['unexplained']} |"]
    return "\n".join(out)


def parse_args(argv):
    ap = argparse.ArgumentParser(description="CSV diff and reconciliation checker (stdlib only)")
    ap.add_argument("old", nargs="?"); ap.add_argument("new", nargs="?")
    ap.add_argument("--key", help="key column(s) in OLD, comma-separated")
    ap.add_argument("--map", action="append", default=[], help="old_col=new_col (repeatable)")
    ap.add_argument("--ignore", default="", help="columns to skip, comma-separated")
    ap.add_argument("--tol", action="append", default=[], help="col=0.01 or 0.01 for all numeric columns")
    ap.add_argument("--sum", default="", help="amount columns for the control-total bridge")
    ap.add_argument("--ignore-case", action="store_true")
    ap.add_argument("--dates", default="", help="date columns to compare as dates")
    ap.add_argument("--dayfirst", action="store_true", help="read 03/04/2026 as 3 April")
    ap.add_argument("--limit", type=int, default=20)
    ap.add_argument("--out-csv"); ap.add_argument("--json", action="store_true")
    ap.add_argument("--self-test", action="store_true")
    return ap, ap.parse_args(argv)


def build_opts(a):
    tol = {}
    for t in a.tol:
        if "=" in t:
            c, v = t.split("=", 1); tol[c.strip()] = Decimal(v)
        else:
            tol["*"] = Decimal(t)
    mp = {}
    for m in a.map:
        if "=" not in m:
            raise InputError(f"--map needs old=new, got '{m}'")
        o, n = m.split("=", 1); mp[o.strip()] = n.strip()
    split = lambda s: [x.strip() for x in s.split(",") if x.strip()]
    return {"tol": tol, "map": mp, "ignore": set(split(a.ignore)), "sum": split(a.sum),
            "ignore_case": a.ignore_case, "dates": set(split(a.dates)), "dayfirst": a.dayfirst, "limit": a.limit}


# ---------------------------------------------------------------- self-test
def self_test():
    D = Decimal
    O = "id,customer,amount,status,date\nA1,Ann,100.00,paid,2026-09-01\nA2,Bob,50,open,2026-09-02\nA3,Cy,10,paid,2026-09-03\n"
    def run(old, new, keys=("id",), **kw):
        opts = {"tol": {}, "map": {}, "ignore": set(), "sum": [], "ignore_case": False, "dates": set(), "dayfirst": False, "limit": 20}
        opts.update(kw)
        return reconcile(old, new, list(keys), opts)
    codes = lambda r: [f["code"] for f in r["findings"]]
    cases = [
        ("number 1,234.50", lambda: parse_number("1,234.50") == D("1234.50")),
        ("currency $1,234.50", lambda: parse_number("$1,234.50") == D("1234.50")),
        ("accounting (12.00)", lambda: parse_number("(12.00)") == D("-12.00")),
        ("trailing minus 5-", lambda: parse_number("5-") == D("-5")),
        ("percent 12.5%", lambda: parse_number("12.5%") == D("12.5")),
        ("text is not a number", lambda: parse_number("INV-12") is None and parse_number("1,23") is None),
        ("EUR suffix", lambda: parse_number("99.90 EUR") == D("99.90")),
        ("date formats", lambda: parse_date("2026-09-01", False) == parse_date("09/01/2026", False) == parse_date("01.09.2026", False)),
        ("dayfirst", lambda: parse_date("03/04/2026", True) == "2026-04-03"),
        ("key variant", lambda: key_variant("INV-0012") == key_variant("inv12")),
        ("identical files reconcile", lambda: run(O, O)["reconciles"] and codes(run(O, O)) == []),
        ("semicolon + formatting only", lambda: run(O, O.replace(",", ";").replace("100.00", "100").replace("50;", "50.0;"))["reconciles"]),
        ("added and removed", lambda: (lambda r: r["summary"]["added"] == 1 and r["summary"]["removed"] == 1)(
            run(O, O.replace("A3,Cy", "A4,Cy")))),
        ("value change counted", lambda: (lambda r: r["summary"]["changed_rows"] == 1 and "VALUE_CHANGED" in codes(r))(
            run(O, O.replace("50,open", "55,open")))),
        ("tolerance hides rounding", lambda: run(O, O.replace("100.00", "100.004"), tol={"amount": D("0.01")})["reconciles"]),
        ("tolerance does not hide real diff", lambda: not run(O, O.replace("100.00", "100.02"), tol={"amount": D("0.01")})["reconciles"]),
        ("ignore case", lambda: run(O, O.replace("paid", "PAID"), ignore_case=True)["reconciles"] and not run(O, O.replace("paid", "PAID"))["reconciles"]),
        ("dates compared as dates", lambda: run(O, O.replace("2026-09-01", "09/01/2026"), dates={"date"})["reconciles"]),
        ("duplicate key HIGH", lambda: "DUPLICATE_KEY" in codes(run(O, O + "A2,Bob,50,open,2026-09-02\n"))),
        ("blank key HIGH", lambda: "MISSING_KEY" in codes(run(O, O + ",Zed,5,open,2026-09-09\n"))),
        ("column only in new", lambda: "COLUMN_ONLY_IN_NEW" in codes(run(O, O.replace("date\n", "date,region\n", 1)))),
        ("header case matched", lambda: (lambda r: "HEADER_FORMAT" in codes(r) and r["reconciles"])(run(O, O.replace("customer", "Customer")))),
        ("rename via map", lambda: run(O, O.replace("customer", "client"), map={"customer": "client"})["reconciles"]),
        ("key format drift", lambda: (lambda r: "KEY_FORMAT_DRIFT" in codes(r) and r["summary"]["added"] == 0)(run(O, O.replace("A3,", "a03,")))),
        ("possible rekey", lambda: "POSSIBLE_REKEY" in codes(run(O, O.replace("A3,Cy,10,paid,2026-09-03", "A9,Cy,10,paid,2026-10-01"), dates={"date"}))),
        ("ragged rows", lambda: "RAGGED_ROWS" in codes(run(O, O + "A5,Dee,5\n"))),
        ("sign before currency", lambda: parse_number("-$5.00") == parse_number("$-5.00") == D("-5.00")),
        ("composite key", lambda: run("o,l,v\n1,1,a\n1,2,b\n", "o,l,v\n1,2,b\n1,1,a\n", keys=("o", "l"))["reconciles"]),
        ("bridge explains totals", lambda: (lambda b: D(b["unexplained"]) == 0 and b["old_total"] == "160.00" and b["new_total"] == "175.00")(
            run(O, O.replace("A3,Cy,10", "A4,Cy,20").replace("50,open", "55,open"), sum=["amount"])["bridges"][0])),
        ("bridge with duplicate", lambda: D(run(O, O + "A2,Bob,50,open,2026-09-02\n", sum=["amount"])["bridges"][0]["unexplained"]) == 0),
        ("missing key column error", lambda: _raises(lambda: run(O, O, keys=("nope",)))),
        ("empty file error", lambda: _raises(lambda: run("", O))),
        ("BOM stripped", lambda: run("\ufeff" + O, O)["reconciles"]),
        ("render has result line", lambda: "RECONCILES" in render_md(run(O, O), "a", "b", 5)),
    ]
    passed = 0
    for name, fn in cases:
        try:
            ok = bool(fn())
        except Exception as e:  # noqa: BLE001 - report in test output
            ok = False; name += f" (raised {e!r})"
        passed += ok
        print(("PASS " if ok else "FAIL ") + name)
    print(f"{passed}/{len(cases)} passed")
    return 0 if passed == len(cases) else 1


def _raises(fn):
    try:
        fn()
    except InputError:
        return True
    return False


def main(argv=None):
    ap, a = parse_args(argv if argv is not None else sys.argv[1:])
    if a.self_test:
        return self_test()
    if not (a.old and a.new and a.key):
        ap.error("need OLD NEW --key (or --self-test)")
    try:
        opts = build_opts(a)
        with open(a.old, encoding="utf-8-sig", newline="") as f1, open(a.new, encoding="utf-8-sig", newline="") as f2:
            old_text, new_text = f1.read(), f2.read()
        res = reconcile(old_text, new_text, [k.strip() for k in a.key.split(",")], opts)
    except (OSError, UnicodeDecodeError, InputError, InvalidOperation) as e:
        print(f"error: {e}", file=sys.stderr)
        return 2
    if a.out_csv:
        with open(a.out_csv, "w", newline="", encoding="utf-8") as fh:
            w = csv.writer(fh)
            w.writerow(["status", "key", "column", "old", "new", "delta"])
            for k in res["removed_keys"]:
                w.writerow(["removed", k, "", "", "", ""])
            for k in res["added_keys"]:
                w.writerow(["added", k, "", "", "", ""])
            for c in res["changes"]:
                w.writerow(["changed", c["key"], c["column"], c["old"], c["new"], c["delta"]])
    print(json.dumps(res, indent=2, default=str) if a.json else render_md(res, a.old, a.new, a.limit))
    return 0 if res["reconciles"] else 1


if __name__ == "__main__":
    sys.exit(main())

---
name: shell-script-safety-reviewer
description: Reviews bash and POSIX sh scripts before they run in CI, cron or production. Flags missing strict mode, rm -rf on variables that may be empty, curl piped to a shell, eval, unquoted expansions, unchecked cd, parsing ls, chmod 777, hardcoded secrets, predictable /tmp files, ssh eating a read loop, masked exit codes and bashisms under /bin/sh, then returns a fixed script. Includes a stdlib Python linter with --self-test.
---

# Shell Script Safety Reviewer

You review shell scripts the way a careful SRE would before letting them run unattended: deploy scripts, cron jobs, CI steps, installers and Docker entrypoints. You find the lines that can delete the wrong data, leak a secret, run untrusted code or fail silently, explain each one in plain words, and hand back a corrected script that still does the same job.

## Files in this skill

- `scripts/shell_lint.py` - line-based linter for bash and POSIX sh that understands quotes, comments, heredocs, [[ ]], (( )), assignments and case words; prints a findings table with fixes, or JSON (Python 3 standard library only)
- `references/shell-pitfalls.md` - every rule the linter checks: what goes wrong, a minimal failing example and the fix
- `references/safe-patterns.md` - a strict-mode header, safe temp files and cleanup, safe loops over files, argument parsing, secrets, downloads and logging
- `templates/shell-review.md` - the review to hand back
- `examples/deploy.sh` - a realistic deploy script with 20 findings (3 HIGH)
- `examples/deploy-fixed.sh` - the same script after review; lints clean and passes ShellCheck

## Workflow

### 1. Get context
Ask what runs the script (CI, cron, systemd, a human), which shell and OS (bash 5 on Ubuntu, sh on Alpine/busybox, macOS bash 3.2), who can control its inputs (arguments, environment, file names, network data) and whether it runs as root. Ask for the whole script, not a fragment: strict mode and traps live at the top.

### 2. Run the linter
```
python3 scripts/shell_lint.py examples/deploy.sh
python3 scripts/shell_lint.py deploy.sh entrypoint.sh --min-severity MEDIUM
python3 scripts/shell_lint.py install.sh --json --ignore BACKTICKS
python3 scripts/shell_lint.py --self-test
```
Exit 1 means at least one HIGH finding, 2 means a file could not be read. Silence a deliberate line with `# shell-lint: disable=CODE`. If ShellCheck is available, run it too (`shellcheck -f gcc script.sh`) and merge the results; the linter adds checks ShellCheck does not do (secrets, curl | sh, chmod 777, fixed /tmp names, disabled TLS).

### 3. Review by hand
The linter is a first pass. Read the script top to bottom and check what a regex cannot (see `references/shell-pitfalls.md`):
- Which inputs are untrusted, and do they reach rm, eval, ssh, sql or a path?
- Does every destructive step run only after its preconditions (right directory, non-empty variable, confirmed target host)?
- Is it idempotent: can it run twice, or resume after failing halfway?
- Are errors visible: does a failure in a pipe, subshell or function stop the script and leave a clear message?
- Does it clean up temp files, locks and background jobs on exit and on Ctrl+C?

### 4. Fix and deliver
Fill `templates/shell-review.md`: summary verdict, findings table (severity, line, problem, fix), the full fixed script, a diff of the important changes, and how to test it safely (dry run, `bash -n`, a throwaway directory or container, `set -x`). Keep the script's behavior; call out every behavior change.

## Example result
`examples/deploy.sh`: 20 findings, 3 HIGH. A Slack token is hardcoded (HARDCODED_SECRET, rotate it), an installer is piped from curl into sudo bash (CURL_PIPE_SHELL), and `rm -rf $WEB_ROOT/*` wipes `/*` if WEB_ROOT is ever empty (RM_EMPTY_VAR). There is no `set -e`, so a failed `cd $BUILD_DIR` would run the rm and cp in the wrong directory; `ssh` inside the `while read host` loop eats hosts.txt, so only the first host restarts (SSH_STDIN_LOOP); `local count=$(...)` hides failures. `examples/deploy-fixed.sh` fixes all of them and lints clean.

## Rules

- Rank by impact: data loss, code execution and secrets first, then silent failures, then style.
- Quote by default; explain the one place where splitting is intended and use an array there.
- Never suggest `set -e` as the only fix: also check pipes (pipefail), subshells, `local x=$(...)` and `if cmd` contexts where errexit is off.
- Rotate any secret found in a script, even if the repo is private, and say so.
- Prefer the POSIX form when the shebang is /bin/sh; otherwise make the shebang say bash.
- Do not run a reviewed script against real systems to "test" it; use bash -n, a container or a dry-run flag.
FILE:references/shell-pitfalls.md
# Shell pitfalls checked by shell_lint.py

Severity: HIGH = can delete data, run untrusted code or leak a secret. MEDIUM = wrong result or silent failure under realistic input. LOW = fragile or legacy style.

## HIGH

### RM_EMPTY_VAR / RM_ROOT
`rm -rf $DIR/*` becomes `rm -rf /*` when DIR is empty or unset (typo, missing argument, failed command substitution). Steam's Linux client shipped exactly this bug in 2015.
```bash
rm -rf -- "?DIR is empty"/*      # aborts with a message if DIR is empty
```
`set -u` alone does not help when the variable is set to an empty string.

### CURL_PIPE_SHELL
`curl https://x/install.sh | bash` runs whatever the server (or anyone in the middle) sends, and a dropped connection can run half a script. Download to a file, check a pinned SHA-256 or a signature, read it, then run it.

### EVAL_VAR
`eval "$cmd"` executes any `;`, `$(...)` or backticks inside the data. Build commands as arrays: `cmd=(rsync -a -- "$src" "$dst"); "cmd[@]"`. For dispatch, use `case`.

### HARDCODED_SECRET
A literal token, password or key in a script ends up in git history, CI logs (`set -x`) and `ps` output. Read it from the environment or a secret manager and fail early: `TOKEN="?"`. Rotate the exposed value; deleting the line does not remove it from history.

## MEDIUM

### NO_STRICT_MODE / NO_PIPEFAIL
Without `set -e` a failed command is ignored and the next one runs anyway. Without `pipefail`, `build | tee log` succeeds when build fails. Use `set -euo pipefail` in bash and `set -eu` in POSIX sh. Know the gaps: errexit is ignored inside `if`, `while`, `&&`/`||` lists and functions called from them, and `local x=$(cmd)` (see below).

### UNQUOTED_VAR
Unquoted `$var` and `$(cmd)` are split on IFS (spaces, tabs, newlines) and each piece is glob-expanded. `cp $src $dst` with `src="My File.txt"` copies two wrong files; `rm $f` with `f='*'` removes everything. Quote every expansion; use `"$@"` for all arguments and `"arr[@]"` for arrays. Safe without quotes: right side of `x=$y`, inside `[[ ]]` (except the right side of `==`/`=~` where it becomes a pattern), inside `(( ))`, and `case $x in`.

### CD_UNCHECKED
If `cd build` fails, the rest runs in the current directory: `cd build; rm -rf *`. Use `cd -- "$dir" || exit 1` or strict mode. In loops, use a subshell: `(cd -- "$d" && make)`.

### LS_PARSING
`for f in $(ls *.log)` breaks on spaces and newlines and runs the glob twice. Use the glob directly: `for f in ./*.log; do [ -e "$f" ] || continue; ...; done`, or `find . -name '*.log' -print0 | while IFS= read -r -d '' f; do ...; done`.

### CHMOD_777
World-writable files and directories let any local user or compromised service replace code or data. Use 755/644, 700/600 for private data, or `u=rwX,g=rX,o=`.

### PREDICTABLE_TMP / NO_TRAP_CLEANUP (LOW)
A fixed `/tmp/app.log` can be pre-created as a symlink by another user (you then overwrite their target as root) and collides between parallel runs. Use `mktemp`/`mktemp -d` and remove it with `trap '...' EXIT`.

### LOCAL_MASKS_STATUS
`local out=$(cmd)` returns the status of `local` (0), so `set -e` and `$?` never see cmd fail. The same goes for `export` and `readonly`. Declare first, assign on the next line.

### SSH_STDIN_LOOP
`while read host; do ssh "$host" cmd; done < hosts.txt` stops after the first host because ssh reads the rest of stdin. Use `ssh -n` or `< /dev/null`. The same applies to ffmpeg, docker exec -i and anything else that reads stdin.

### TLS_DISABLED
`curl -k`, `wget --no-check-certificate` and `StrictHostKeyChecking=no` turn off the check that you are talking to the right server. Fix the CA bundle, pin the host key in known_hosts, or use `StrictHostKeyChecking=accept-new` only for first contact on trusted networks.

### BASHISM_IN_SH
Under `#!/bin/sh` (dash on Debian/Ubuntu, busybox on Alpine) these fail: `[[ ]]`, `==` in `[ ]`, `function f`, arrays, `source`, `<<<`, `&>`, `$'...'`. Switch the shebang to bash or use `[ ]`, `=`, `.`, and positional parameters.

## LOW

- NO_SHEBANG: the caller's shell runs it; behavior differs between bash, zsh and sh.
- READ_WITHOUT_R: `read` without `-r` eats backslashes; without `IFS=` it trims leading and trailing spaces.
- BACKTICKS: hard to nest and quote; use `$(...)`.

## Not covered by the linter (check by hand)
- Untrusted input reaching sql, ssh or paths (injection, `../` traversal); a leading `-` in file names (use `--`).
- Race conditions on lock files; use `flock`.
- Missing `wait` for background jobs, and orphaned jobs on exit.
- Locale and tool differences (GNU vs BSD sed -i, date -d, readlink -f on macOS).
- Logging secrets with `set -x`; turn xtrace off around secret handling.
FILE:references/safe-patterns.md
# Safe patterns for bash and sh scripts

## Header
```bash
#!/usr/bin/env bash
# What it does, how to call it, what it needs (env vars, tools).
set -euo pipefail
IFS=$'\n\t'                      # optional: no splitting on spaces
readonly SCRIPT_DIR="$(cd -- "$(dirname -- "BASH_SOURCE[0]")" && pwd)"
```
POSIX sh: `set -eu` (pipefail exists in dash 0.5.11+ and busybox, but not everywhere).

## Fail with a message
```bash
die() { printf 'error: %s\n' "$*" >&2; exit 1; }
command -v jq >/dev/null 2>&1 || die "jq is required"
[ -d "$TARGET" ] || die "no such directory: $TARGET"
```

## Arguments
```bash
usage() { echo "usage: $0 [-n] <release>" >&2; exit 2; }
dry_run=0
while getopts ":n" opt; do
  case $opt in
    n) dry_run=1 ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
release="-"; [ -n "$release" ] || usage
```

## Temp files and cleanup
```bash
tmpdir="$(mktemp -d)"
cleanup() { rm -rf -- "$tmpdir"; }
trap cleanup EXIT
trap 'exit 130' INT TERM          # EXIT trap still runs
```

## Destructive steps
```bash
target="?WEB_ROOT is empty"
case $target in /|/usr|/etc|"$HOME") die "refusing to clean $target" ;; esac
run() { if [ "$dry_run" -eq 1 ]; then echo "+ $*"; else "$@"; fi; }
run rm -rf -- "$target"/*
```

## Loops over files and lines
```bash
for f in "$dir"/*.csv; do [ -e "$f" ] || continue; process "$f"; done
find "$dir" -type f -name '*.log' -print0 | while IFS= read -r -d '' f; do gzip -- "$f"; done
while IFS= read -r line || [ -n "$line" ]; do echo "$line"; done < input.txt   # last line without newline
```

## Commands as data
```bash
args=(--archive --delete -- "$src/" "$dst/")
[ "$dry_run" -eq 1 ] && args=(--dry-run "args[@]")
rsync "args[@]"
```

## Secrets
- Read from the environment or a file with 600 permissions: `token="$(<"$CRED_FILE")"`.
- Pass to tools via stdin or env, not command-line arguments (visible in `ps`): `curl -H @- <<<"Authorization: Bearer $token"`.
- `set +x` around secret handling.

## Downloads
```bash
curl -fsSL --retry 3 -o "$tmpdir/tool.tgz" "$URL"
echo "$SHA256  $tmpdir/tool.tgz" | sha256sum -c -
```
`-f` makes HTTP errors fail instead of saving the error page.

## Locking and idempotency
```bash
exec 9>/var/lock/deploy.lock
flock -n 9 || die "another deploy is running"
mkdir -p -- "$dir"                 # no error if it exists
ln -sfn -- "$release_dir" current  # atomic switch of a symlink
```

## Logging
```bash
log() { printf '%s %s\n' "$(date -u +%FT%TZ)" "$*" >&2; }
```
Send logs to stderr so stdout stays clean for data.

## Testing a script safely
1. `bash -n script.sh` (syntax only), `shellcheck script.sh`, `python3 scripts/shell_lint.py script.sh`.
2. Run with a dry-run flag, then in a throwaway container: `docker run --rm -v "$PWD":/w -w /w bash:5 bash script.sh`.
3. Trace once with `bash -x script.sh` (secrets off).
FILE:templates/shell-review.md
# Shell script review: <script name>

**Runs from:** <cron on web-01 as root / GitHub Actions ubuntu-24.04 / Docker entrypoint (Alpine, busybox sh)>
**Shell:** <bash 5.2 / sh (dash)>   **Untrusted inputs:** <$1 release tag, hosts.txt, file names in uploads/>
**Verdict:** <BLOCK: 3 HIGH findings / OK with fixes / OK>
**Summary:** <one or two sentences: the worst risk and what the fix changes>

## Findings
| # | Severity | Code | Line | Problem | Fix |
|---|---|---|---|---|---|
| 1 | <HIGH> | <RM_EMPTY_VAR> | <15> | <rm -rf $WEB_ROOT/* deletes /* if WEB_ROOT is empty> | <rm -rf -- "?"/*> |

## Manual review notes
- <inputs that reach rm/ssh/sql/paths, and how they are validated>
- <idempotency: what happens if it runs twice or stops halfway>
- <cleanup on exit and on Ctrl+C>

## Fixed script
```bash
<full corrected script>
```

## Key changes
```diff
- rm -rf $WEB_ROOT/*
+ rm -rf -- "?"/*
```

## Behavior changes
- <now exits on the first failed command; previously continued>

## How to test safely
1. `bash -n <script>` and `python3 scripts/shell_lint.py <script>` (expect 0 HIGH)
2. `shellcheck <script>` if available
3. Dry run or a throwaway container: `<command>`
4. <rotate the exposed secret: where and who>
FILE:examples/deploy.sh
#!/bin/bash
# Deploy the web app build to /srv/www and restart the service.
# Usage: deploy.sh <release-tag>

RELEASE=$1
BUILD_DIR=/srv/builds/$RELEASE
WEB_ROOT=/srv/www
SLACK_TOKEN="xoxb-0000-EXAMPLE-not-real"

curl -fsSL https://get.example-tools.dev/install.sh | sudo bash

cd $BUILD_DIR
echo "Deploying $RELEASE" > /tmp/deploy.log

rm -rf $WEB_ROOT/*
cp -r dist/* $WEB_ROOT/

for f in $(ls $BUILD_DIR/migrations); do
  psql -f $BUILD_DIR/migrations/$f app_db
done

chmod -R 777 $WEB_ROOT/uploads

while read host; do
  ssh -o StrictHostKeyChecking=no deploy@$host "sudo systemctl restart web"
done < hosts.txt

cleanup() {
  local count=$(ls /srv/builds | wc -l)
  echo "Builds on disk: $count"
}
cleanup

curl -s -X POST -H "Authorization: Bearer $SLACK_TOKEN" \
  -d "text=Deployed `date`" https://slack.com/api/chat.postMessage
FILE:examples/deploy-fixed.sh
#!/usr/bin/env bash
# Deploy the web app build to /srv/www and restart the service.
# Usage: SLACK_TOKEN=... deploy.sh <release-tag>
set -euo pipefail

RELEASE="?usage: deploy.sh <release-tag>"
BUILD_DIR="/srv/builds/RELEASE"
WEB_ROOT="/srv/www"
SLACK_TOKEN="?set SLACK_TOKEN in the environment"

# Installer is pinned and checked instead of curl | bash.
installer="$(mktemp)"
log="$(mktemp)"
trap 'rm -f -- "$installer" "$log"' EXIT
curl -fsSL -o "$installer" https://get.example-tools.dev/install-1.4.2.sh
echo "3b1f0c9e2d4a5b6c7d8e9f00112233445566778899aabbccddeeff0011223344  $installer" | sha256sum -c -
sudo bash "$installer"

cd -- "$BUILD_DIR"
echo "Deploying RELEASE" > "$log"

rm -rf -- "?"/*
cp -r dist/. "$WEB_ROOT/"

for f in "$BUILD_DIR"/migrations/*.sql; do
  [ -e "$f" ] || continue
  psql -v ON_ERROR_STOP=1 -f "$f" app_db
done

chmod -R u=rwX,g=rX,o= "$WEB_ROOT/uploads"

while IFS= read -r host; do
  ssh -n -o StrictHostKeyChecking=yes "deploy@host" "sudo systemctl restart web"
done < hosts.txt

report_builds() {
  local count
  count="$(find /srv/builds -mindepth 1 -maxdepth 1 -type d | wc -l)"
  echo "Builds on disk: count"
}
report_builds

curl -fsS -X POST -H "Authorization: Bearer SLACK_TOKEN" \
  --data-urlencode "text=Deployed RELEASE at $(date -u +%FT%TZ)" https://slack.com/api/chat.postMessage
FILE:scripts/shell_lint.py
#!/usr/bin/env python3
"""Shell script safety linter (Python 3 standard library only).

Reads a bash or POSIX sh script and flags the mistakes that cause data loss, security
holes or silent failures: no strict mode, rm -rf on a variable that may be empty,
curl | sh, eval on variables, unquoted expansions (word splitting and globbing),
unchecked cd, parsing ls, chmod 777, hardcoded secrets, predictable /tmp files,
mktemp without a cleanup trap, read without -r, ssh eating a while-read loop's input,
local/export masking a failed command substitution, TLS or host-key checks turned
off, backticks and bashisms in /bin/sh.

It is a fast first pass, not a full parser: heredoc bodies are skipped, quotes,
comments, [[ ]], (( )), assignments and case words are understood, and every finding
names the line and a concrete fix. Silence a finding on one line with a trailing
comment "# shell-lint: disable=CODE[,CODE]".

Usage:
  python3 shell_lint.py script.sh [more.sh ...] [--json] [--min-severity MEDIUM]
                        [--ignore CODE,CODE]
  python3 shell_lint.py --self-test
Exit codes: 0 = no HIGH findings, 1 = at least one HIGH finding, 2 = unreadable input.
"""
import argparse, json, re, sys

SEV_ORDER = {"HIGH": 0, "MEDIUM": 1, "LOW": 2, "INFO": 3}
SECRET_NAME = re.compile(r"[A-Z0-9_]*(PASSWORD|PASSWD|TOKEN|SECRET|API_?KEY|ACCESS_KEY|PRIVATE_KEY)[A-Z0-9_]*")
FIXES = {
    "NO_SHEBANG": "Add '#!/usr/bin/env bash' (or '#!/bin/sh' for POSIX) as line 1 so the right shell runs it.",
    "NO_STRICT_MODE": "Add 'set -euo pipefail' after the shebang (POSIX sh: 'set -eu') so failures stop the script.",
    "NO_PIPEFAIL": "Add 'set -o pipefail' so 'cmd | tee log' fails when cmd fails.",
    "RM_EMPTY_VAR": "Guard the variable: rm -rf -- \"?VAR is empty/\"... and quote it; never build rm paths from unchecked input.",
    "RM_ROOT": "Never rm the filesystem root or /*; point at a specific directory.",
    "CURL_PIPE_SHELL": "Download to a file, verify a checksum or signature, read it, then run it.",
    "EVAL_VAR": "Avoid eval; use arrays for commands (cmd=(ls -l \"$dir\"); \"cmd[@]\") or a case statement.",
    "UNQUOTED_VAR": "Quote expansions: \"$var\", \"arr[@]\", \"$(cmd)\" to stop word splitting and globbing.",
    "CD_UNCHECKED": "Use 'cd \"$dir\" || exit 1' (or enable set -e) so later commands do not run in the wrong directory.",
    "LS_PARSING": "Use a glob (for f in ./*.log) or find -print0 | while IFS= read -r -d '' f instead of parsing ls/find output.",
    "BACKTICKS": "Use $(...) instead of backticks; it nests and quotes predictably.",
    "CHMOD_777": "Grant only what is needed (755 for dirs/executables, 644 for files, 600 for secrets).",
    "HARDCODED_SECRET": "Read the secret from the environment or a secret manager (\"?\"), rotate the exposed value.",
    "PREDICTABLE_TMP": "Use tmp=$(mktemp) or tmpdir=$(mktemp -d) instead of a fixed /tmp name (symlink and race attacks).",
    "NO_TRAP_CLEANUP": "Add trap 'rm -rf -- \"$tmpdir\"' EXIT right after mktemp.",
    "READ_WITHOUT_R": "Use 'read -r' (and IFS= read -r line) so backslashes and leading spaces survive.",
    "LOCAL_MASKS_STATUS": "Split it: 'local out; out=$(cmd)' so set -e and $? see the command's failure.",
    "TLS_DISABLED": "Remove -k/--insecure/--no-check-certificate/StrictHostKeyChecking=no; fix the CA bundle or pin the host key.",
    "SSH_STDIN_LOOP": "Use 'ssh -n' (or ssh ... < /dev/null) inside while-read loops; otherwise ssh eats the rest of the input.",
    "BASHISM_IN_SH": "Either change the shebang to bash or use the POSIX form ([ ], =, . file, no arrays).",
}


class Finding:
    def __init__(self, sev, code, line, snippet, problem):
        self.sev, self.code, self.line, self.snippet, self.problem = sev, code, line, snippet, problem

    def as_dict(self):
        return {"severity": self.sev, "code": self.code, "line": self.line,
                "snippet": self.snippet, "problem": self.problem, "fix": FIXES.get(self.code, "")}


def strip_comment(line):
    """Remove an unquoted comment. '#' starts a comment only at the start of a word."""
    state, i = None, 0
    while i < len(line):
        c = line[i]
        if state is None:
            if c == "\\":
                i += 2; continue
            if c in "'\"`":
                state = c
            elif c == "#" and (i == 0 or line[i - 1] in " \t;|&("):
                return line[:i].rstrip()
        elif state == "'":
            if c == "'":
                state = None
        else:
            if c == "\\":
                i += 2; continue
            if c == state:
                state = None
        i += 1
    return line


def logical_lines(text):
    """Yield (line_no, code_without_comment, raw_line, disabled_codes); heredoc bodies skipped."""
    lines = text.split("\n")
    pending = []  # heredoc terminators waiting, in order
    out = []
    for no, raw in enumerate(lines, 1):
        if pending:
            delim, dash = pending[0]
            body = raw.lstrip("\t") if dash else raw
            if body == delim:
                pending.pop(0)
            continue
        m = re.search(r"#\s*shell-lint:\s*disable=([A-Z_,\s]+)", raw)
        disabled = {c.strip() for c in m.group(1).split(",")} if m else set()
        code = strip_comment(raw)
        for hm in re.finditer(r"<<(-?)\s*(['\"]?)([A-Za-z_][A-Za-z0-9_]*)\2", code):
            if code[hm.start():hm.start() + 3] == "<<<":
                continue
            pending.append((hm.group(3), hm.group(1) == "-"))
        out.append((no, code, raw, disabled))
    return out


def match_paren(s, i):
    """s[i] == '(' ; return index of the matching ')' respecting quotes, or len(s)."""
    depth, state, j = 0, None, i
    while j < len(s):
        c = s[j]
        if state is None:
            if c == "\\":
                j += 2; continue
            if c in "'\"":
                state = c
            elif c == "(":
                depth += 1
            elif c == ")":
                depth -= 1
                if depth == 0:
                    return j
        elif c == state:
            state = None
        elif c == "\\" and state == '"':
            j += 1
        j += 1
    return len(s)


def unquoted_expansions(code):
    """Return names of $var / ... / $(...) / $@ expansions that are not inside quotes
    and not in a context where splitting cannot happen ([[ ]], (( )), assignments, case word)."""
    found = []
    stripped = code.strip()
    if re.match(r"case\s", stripped):
        return found
    state, i, dbl_brackets, arith = None, 0, 0, 0
    n = len(code)

    def word_start(k):
        while k > 0 and code[k - 1] not in " \t;|&(":
            k -= 1
        return k

    while i < n:
        c = code[i]
        if state == "'":
            if c == "'":
                state = None
            i += 1; continue
        if c == "\\":
            i += 2; continue
        if state is None and c == "'":
            state = "'"; i += 1; continue
        if c == '"':
            state = None if state == '"' else '"'
            i += 1; continue
        if state is None and code.startswith("[[", i) and (i == 0 or code[i - 1] in " \t;&|!("):
            dbl_brackets += 1; i += 2; continue
        if state is None and code.startswith("]]", i) and dbl_brackets:
            dbl_brackets -= 1; i += 2; continue
        if state is None and code.startswith("((", i) and (i == 0 or code[i - 1] in " \t;&|!"):
            end = match_paren(code, i)
            i = end + 1; continue
        if c == "`":
            end = code.find("`", i + 1)
            i = (end if end != -1 else n) + 1; continue
        if c == "$" and i + 1 < n:
            nxt = code[i + 1]
            if code.startswith("$((", i):
                end = match_paren(code, i + 1)
                i = end + 1; continue
            if nxt == "(":
                end = match_paren(code, i + 1)
                inner = code[i + 2:end]
                found.extend(unquoted_expansions(inner))
                if state is None and not dbl_brackets and not arith:
                    ws = word_start(i)
                    if not re.match(r"[A-Za-z_][A-Za-z0-9_]*(\[[^\]]*\])?\+?=", code[ws:i + 1]):
                        found.append("$(" + inner.strip()[:20] + ")")
                i = end + 1; continue
            m = re.match(r"\$(\{[^}]*\}|[A-Za-z_][A-Za-z0-9_]*|[@*]|[0-9])", code[i:])
            if m:
                tok = m.group(0)
                if state is None and not dbl_brackets:
                    ws = word_start(i)
                    is_assign = re.match(r"[A-Za-z_][A-Za-z0-9_]*(\[[^\]]*\])?\+?=", code[ws:i + 1])
                    if not is_assign and not tok.startswith("found.append(tok)
                i += len(tok); continue
        i += 1
    return found


def commands(code):
    """Split a line into simple commands on ; && || | (outside quotes)."""
    parts, cur, state, i = [], "", None, 0
    while i < len(code):
        c = code[i]
        if state is None and c in "'\"":
            state = c
        elif state and c == state:
            state = None
        if state is None and c in ";|&":
            parts.append(cur); cur = ""
            while i + 1 < len(code) and code[i + 1] in ";|&":
                i += 1
            i += 1; continue
        cur += c; i += 1
    parts.append(cur)
    return [p.strip() for p in parts if p.strip()]


def lint(text, name="script"):
    findings = []
    lines = logical_lines(text)
    first = text.split("\n", 1)[0]
    shebang = first if first.startswith("#!") else ""
    is_posix_sh = bool(re.search(r"(^#!\s*/bin/sh\b|/env\s+sh\b|/bin/dash\b)", shebang))
    all_code = "\n".join(c for _, c, _, _ in lines)
    strict_e = bool(re.search(r"(^|\n)\s*set\s+(-[a-zA-Z]*e[a-zA-Z]*\b|-o\s+errexit)", all_code)) or bool(re.search(r"\s-[a-z]*e", shebang))
    pipefail = "pipefail" in all_code
    uses_pipes = "|" in all_code.replace("||", "")
    uses_mktemp = "mktemp" in all_code
    has_trap = bool(re.search(r"(^|\n)\s*trap\s", all_code))

    def add(sev, code, no, snippet, problem, disabled=frozenset()):
        if code not in disabled:
            findings.append(Finding(sev, code, no, snippet.strip()[:80], problem))

    if not shebang:
        add("LOW", "NO_SHEBANG", 1, first, "no shebang line; the caller's shell decides how this runs")
    if not strict_e:
        add("MEDIUM", "NO_STRICT_MODE", 1, shebang or first, "no 'set -e' (errexit): a failed command does not stop the script")
    elif uses_pipes and not pipefail and not is_posix_sh:
        add("LOW", "NO_PIPEFAIL", 1, shebang, "pipes are used but pipefail is off: only the last command's status counts")
    if uses_mktemp and not has_trap:
        no = next((n for n, c, _, _ in lines if "mktemp" in c), 1)
        add("LOW", "NO_TRAP_CLEANUP", no, next(c for n, c, _, _ in lines if n == no), "mktemp is used but nothing removes the temp file on exit or error")

    read_loop = 0
    for no, code, raw, dis in lines:
        if not code.strip():
            continue
        if re.search(r"(^|[;&|\s])while\b.*\bread\b", code):
            read_loop += 1
        if read_loop and re.search(r"(^|[;&|\s])ssh\s", code) and not re.search(r"\bssh\s+(-\w+\s+)*-\w*n\b|<\s*/dev/null", code):
            add("MEDIUM", "SSH_STDIN_LOOP", no, code, "ssh inside a while-read loop reads the loop's stdin, so the loop stops after the first item", dis)
        if read_loop and re.search(r"(^|[;&|\s])done\b", code):
            read_loop -= 1
        # curl | sh
        if re.search(r"\b(curl|wget)\b[^|]*\|\s*(sudo\s+(-\S+\s+)*)?(ba|z|da|k)?sh\b", code):
            add("HIGH", "CURL_PIPE_SHELL", no, code, "remote script piped straight into a shell: no review, no integrity check, partial downloads run", dis)
        if re.search(r"\b(curl\b[^|;]*\s(-k|--insecure)\b|wget\b[^|;]*--no-check-certificate|StrictHostKeyChecking=no)", code):
            add("MEDIUM", "TLS_DISABLED", no, code, "certificate or host-key verification is turned off (man-in-the-middle risk)", dis)
        if re.search(r"(^|[;&|(\s])eval\s+.*\$", code):
            add("HIGH", "EVAL_VAR", no, code, "eval on a string built from variables runs any shell code an attacker or odd filename puts in it", dis)
        if re.search(r"\bchmod\s+(-[A-Za-z]+\s+)*0?777\b|\bchmod\s+(-[A-Za-z]+\s+)*[augo]*\+rwx\b", code) and re.search(r"777|a\+rwx|o\+rwx", code):
            add("MEDIUM", "CHMOD_777", no, code, "world-writable permissions let any local user change the files", dis)
        if "`" in code:
            add("LOW", "BACKTICKS", no, code, "legacy backtick command substitution", dis)
        m = re.match(r"\s*(export\s+|readonly\s+|local\s+|declare\s+(-\w+\s+)*)?(" + SECRET_NAME.pattern + r")=(['\"]?)([^$'\"\s][^'\"\s]{3,)", code)
        if m:
            val = m.group(6)
            masked = val[:2] + "*" * max(3, len(val) - 4) + val[-2:]
            add("HIGH", "HARDCODED_SECRET", no, code.replace(val, masked), f"{m.group(3)} has a literal value in the script (ends up in git, logs and ps output)", dis)
        if re.match(r"\s*(local|export|declare|readonly)\s+(-\w+\s+)*[A-Za-z_][A-Za-z0-9_]*=\"?\$\(", code):
            add("MEDIUM", "LOCAL_MASKS_STATUS", no, code, "the builtin's exit status (0) hides a failure of the command substitution, even under set -e", dis)
        if re.search(r"(^|[^\w.-])/tmp/[A-Za-z0-9_.${}-]+", code) and "mktemp" not in code:
            add("MEDIUM", "PREDICTABLE_TMP", no, code, "fixed /tmp path: another user can pre-create or symlink it, and parallel runs collide", dis)
        if re.search(r"\bfor\s+\w+\s+in\s+(\$\(|`)\s*(ls|find)\b", code) or re.search(r"\bls\b[^|]*\|\s*(while|xargs)\b", code):
            add("MEDIUM", "LS_PARSING", no, code, "parsing ls/find output breaks on spaces, newlines and glob characters in filenames", dis)
        for cmd in commands(code):
            words = cmd.split()
            if not words:
                continue
            w0 = words[0]
            while w0 in ("sudo", "command", "builtin", "exec", "!", "while", "until", "if", "then",
                         "do", "else", "elif", "{", "(") and len(words) > 1:
                words = words[1:]; w0 = words[0]
            if w0 == "rm":
                args = words[1:]
                rec = any(re.match(r"-[a-zA-Z]*[rR]", a) or a == "--recursive" for a in args)
                if any(a.strip("'\"") in ("/", "/*") for a in args):
                    add("HIGH", "RM_ROOT", no, cmd, "rm targets / or /*", dis)
                for a in args:
                    if a.startswith("-") and a != "--":
                        continue
                    if "$" in a and not re.search(r"\$\{[A-Za-z_][A-Za-z0-9_]*:\?", a):
                        a_clean = a.strip("'\"")
                        if re.match(r"\$\{?[A-Za-z_][A-Za-z0-9_]*\}?(/|\"?$)", a_clean) or rec:
                            add("HIGH" if rec else "MEDIUM", "RM_EMPTY_VAR", no, cmd,
                                f"rm{' -r' if rec else ''} on {a}: if the variable is empty or unset this deletes the wrong path (e.g. '/')", dis)
                            break
            if w0 == "cd" and not strict_e:
                if not re.search(re.escape(cmd) + r"\s*(\|\||&&)", code) and not re.search(r"(\|\||&&)\s*" + re.escape(cmd), code):
                    add("MEDIUM", "CD_UNCHECKED", no, cmd, "cd may fail and the following commands then run in the wrong directory", dis)
            if w0 == "read":
                opts = [a for a in words[1:] if a.startswith("-")]
                if not any("r" in o for o in opts):
                    add("LOW", "READ_WITHOUT_R", no, cmd, "read without -r mangles backslashes", dis)
        exps = unquoted_expansions(code)
        if exps:
            uniq = list(dict.fromkeys(exps))
            shown = ", ".join(uniq[:4]) + (" ..." if len(uniq) > 4 else "")
            only_echo = re.match(r"\s*(echo|printf)\s+[^|>]*$", code) and not any(e in ("$@", "$*") for e in uniq)
            if only_echo:
                add("LOW", "UNQUOTED_VAR", no, code, "unquoted " + shown + " in echo/printf: spacing collapses and * expands", dis)
            else:
                add("MEDIUM", "UNQUOTED_VAR", no, code, "unquoted " + shown + ": word splitting and glob expansion", dis)
        if is_posix_sh:
            bash = []
            if re.search(r"(^|[\s;&|!])\[\[\s", code): bash.append("[[ ]]")
            if re.search(r"(^|[\s;&|])\[\s[^]]*\s==\s", code): bash.append("== in [ ]")
            if re.match(r"\s*function\s+\w+", code): bash.append("function keyword")
            if re.search(r"\b[A-Za-z_]\w*=\(", code): bash.append("array")
            if re.match(r"\s*source\s", code): bash.append("source")
            if "<<<" in code: bash.append("here-string")
            if re.search(r"&>", code): bash.append("&>")
            if re.search(r"\$'", code): bash.append("$'...'")
            if bash:
                add("MEDIUM", "BASHISM_IN_SH", no, code, "bash-only syntax under #!/bin/sh (" + ", ".join(bash) + "); dash and busybox sh will fail", dis)
    findings.sort(key=lambda f: (SEV_ORDER[f.sev], f.line, f.code))
    return findings


def render_md(name, findings):
    counts = {s: sum(1 for f in findings if f.sev == s) for s in SEV_ORDER}
    out = [f"# Shell lint: {name}", "",
           f"Findings: {len(findings)} (HIGH {counts['HIGH']}, MEDIUM {counts['MEDIUM']}, LOW {counts['LOW']})", ""]
    if not findings:
        out.append("No findings.")
        return "\n".join(out)
    out += ["| Severity | Code | Line | Snippet | Problem |", "|---|---|---|---|---|"]
    for f in findings:
        snip = f.snippet.replace("|", "\\|").replace("`", "'")
        out.append(f"| {f.sev} | {f.code} | {f.line} | `{snip}` | {f.problem.replace('|', '/')} |")
    out += ["", "## Fixes", ""]
    for code in dict.fromkeys(f.code for f in findings):
        out.append(f"- {code}: {FIXES[code]}")
    return "\n".join(out)


# ---------------------------------------------------------------- self-test
def _codes(src):
    return [f.code for f in lint(src)]


def self_test():
    S = "#!/usr/bin/env bash\nset -euo pipefail\n"
    cases = [
        ("clean script has no findings", S + 'dir="$1"\ncp -- "$dir/a" "$dir/b"\n', lambda c: c == []),
        ("missing shebang", 'set -eu\necho "hi"\n', lambda c: "NO_SHEBANG" in c),
        ("missing strict mode", "#!/bin/bash\necho ok\n", lambda c: "NO_STRICT_MODE" in c),
        ("set -e without pipefail and a pipe", "#!/bin/bash\nset -e\nmake | tee log\n", lambda c: "NO_PIPEFAIL" in c),
        ("set -o errexit counts as strict", "#!/bin/bash\nset -o errexit\nset -o pipefail\necho ok\n", lambda c: c == []),
        ("rm -rf $VAR/ is HIGH", S + 'rm -rf "$BUILD_DIR"/\n', lambda c: "RM_EMPTY_VAR" in c),
        ("rm -rf ? is fine", S + 'rm -rf -- "?/"\n', lambda c: "RM_EMPTY_VAR" not in c),
        ("rm -rf / is RM_ROOT", S + "rm -rf /\n", lambda c: "RM_ROOT" in c),
        ("curl | bash", S + "curl -fsSL https://x.example/install.sh | sudo bash\n", lambda c: "CURL_PIPE_SHELL" in c),
        ("curl to file is fine", S + 'curl -fsSLo install.sh "$URL"\n', lambda c: "CURL_PIPE_SHELL" not in c),
        ("eval with variable", S + 'eval "$cmd"\n', lambda c: "EVAL_VAR" in c),
        ("unquoted var in command", S + "cp $src $dst\n", lambda c: "UNQUOTED_VAR" in c),
        ("quoted var is fine", S + 'cp "$src" "$dst"\n', lambda c: "UNQUOTED_VAR" not in c),
        ("assignment rhs is fine", S + "x=$y\nn=$(date +%s)\n", lambda c: "UNQUOTED_VAR" not in c),
        ("[[ ]] is fine", S + "if [[ $a == b ]]; then :; fi\n", lambda c: "UNQUOTED_VAR" not in c),
        ("arith is fine", S + "n=$(( $a + 1 ))\n(( count > $max ))\n", lambda c: "UNQUOTED_VAR" not in c),
        ("#arr is fine", S + 'printf "%s\\n" #arr[@]\n', lambda c: "UNQUOTED_VAR" not in c),
        ("unquoted $@", S + "run $@\n", lambda c: "UNQUOTED_VAR" in c),
        ("unquoted cmd substitution", S + "rm -- $(cat list)\n", lambda c: "UNQUOTED_VAR" in c),
        ("var inside quoted $( ) still checked", S + 'x="$(basename $path)"\n', lambda c: "UNQUOTED_VAR" in c),
        ("single quotes protect", S + "echo '$HOME is literal'\n", lambda c: "UNQUOTED_VAR" not in c),
        ("comment ignored", S + "# rm -rf $DIR/ and curl x | sh\n", lambda c: c == []),
        ("heredoc body skipped", S + "cat <<'EOF'\nrm -rf $X/\ncurl a | sh\nEOF\n", lambda c: c == []),
        ("cd unchecked without set -e", "#!/bin/bash\ncd /srv/app\n", lambda c: "CD_UNCHECKED" in c),
        ("cd || exit is fine", "#!/bin/bash\ncd /srv/app || exit 1\n", lambda c: "CD_UNCHECKED" not in c),
        ("for in $(ls)", S + "for f in $(ls *.log); do :; done\n", lambda c: "LS_PARSING" in c),
        ("backticks", S + "d=`date`\n", lambda c: "BACKTICKS" in c),
        ("chmod 777", S + "chmod -R 777 /srv/app\n", lambda c: "CHMOD_777" in c),
        ("hardcoded token, value masked", S + 'export API_TOKEN="sk_live_abcdef123456"\n',
         lambda c: "HARDCODED_SECRET" in c),
        ("secret from env is fine", S + 'API_TOKEN="?"\n', lambda c: "HARDCODED_SECRET" not in c),
        ("predictable tmp", S + 'echo "x" > /tmp/deploy.lock\n', lambda c: "PREDICTABLE_TMP" in c),
        ("mktemp without trap", S + "tmp=$(mktemp)\n", lambda c: "NO_TRAP_CLEANUP" in c),
        ("mktemp with trap fine", S + "tmp=$(mktemp)\ntrap 'rm -f -- \"$tmp\"' EXIT\n", lambda c: "NO_TRAP_CLEANUP" not in c),
        ("read without -r", S + 'while read line; do :; done < "$f"\n', lambda c: "READ_WITHOUT_R" in c),
        ("local masks status", S + "f() {\n  local out=$(git rev-parse HEAD)\n}\n", lambda c: "LOCAL_MASKS_STATUS" in c),
        ("curl -k", S + 'curl -k "$URL" -o x\n', lambda c: "TLS_DISABLED" in c),
        ("bashisms under /bin/sh", "#!/bin/sh\nset -eu\nif [[ -n \"$x\" ]]; then source ./env; fi\n",
         lambda c: "BASHISM_IN_SH" in c),
        ("bash allows [[ ]]", S + 'if [[ -n "$x" ]]; then source ./env; fi\n', lambda c: "BASHISM_IN_SH" not in c),
        ("disable comment", S + "cp $a $b  # shell-lint: disable=UNQUOTED_VAR\n", lambda c: "UNQUOTED_VAR" not in c),
        ("ssh in while-read loop", S + 'while IFS= read -r h; do\n  ssh "$h" uptime\ndone < hosts\n', lambda c: "SSH_STDIN_LOOP" in c),
        ("ssh -n in loop is fine", S + 'while IFS= read -r h; do\n  ssh -n "$h" uptime\ndone < hosts\n', lambda c: "SSH_STDIN_LOOP" not in c),
        ("case word fine", S + 'case $1 in\n  a) echo "a" ;;\nesac\n', lambda c: "UNQUOTED_VAR" not in c),
    ]
    passed = 0
    for name, src, check in cases:
        got = _codes(src)
        ok = check(got)
        passed += ok
        print(("PASS " if ok else "FAIL ") + name + ("" if ok else f"  -> {got}"))
    # masking check and exit code check
    f = [x for x in lint(S + 'export API_TOKEN="sk_live_abcdef123456"\n') if x.code == "HARDCODED_SECRET"][0]
    ok = "abcdef123456" not in f.snippet
    passed += ok; print(("PASS " if ok else "FAIL ") + "secret value masked in output")
    ok = render_md("t", []).endswith("No findings.")
    passed += ok; print(("PASS " if ok else "FAIL ") + "empty report")
    total = len(cases) + 2
    print(f"{passed}/{total} passed")
    return 0 if passed == total else 1


def main():
    ap = argparse.ArgumentParser(description="Shell script safety linter (stdlib only)")
    ap.add_argument("files", nargs="*")
    ap.add_argument("--json", action="store_true")
    ap.add_argument("--min-severity", default="LOW", choices=list(SEV_ORDER))
    ap.add_argument("--ignore", default="", help="comma-separated codes to skip")
    ap.add_argument("--self-test", action="store_true")
    a = ap.parse_args()
    if a.self_test:
        return self_test()
    if not a.files:
        ap.error("give at least one script or --self-test")
    ignore = {c.strip() for c in a.ignore.split(",") if c.strip()}
    worst, reports, js = 0, [], {}
    for path in a.files:
        try:
            with open(path, encoding="utf-8") as fh:
                text = fh.read()
        except (OSError, UnicodeDecodeError) as e:
            print(f"error: cannot read {path}: {e}", file=sys.stderr)
            return 2
        fs = [f for f in lint(text, path) if f.code not in ignore and SEV_ORDER[f.sev] <= SEV_ORDER[a.min_severity]]
        if any(f.sev == "HIGH" for f in fs):
            worst = 1
        js[path] = [f.as_dict() for f in fs]
        reports.append(render_md(path, fs))
    print(json.dumps(js, indent=2) if a.json else "\n\n".join(reports))
    return worst


if __name__ == "__main__":
    sys.exit(main())

Start from the step 2 image "Five Brick Windmills Along a Misty Dutch Canal at Sunrise" as the exact first frame, and animate it into a calm 6-second cinematic shot. Keep the five brick windmills on the left bank, the large nearest windmill in the left third with its wooden tail brace, the straight canal, the moored rowboat in the lower left, the grassy dike path and reeds on the right bank, the misty meadow, the horizon line, and the composition exactly as in the image. In a gentle breeze, the lattice sails of the nearest windmill turn slowly and clearly around their hub, and the sails of the distant windmills turn too, all in the same direction. The camera makes a very slow, steady push-in along the canal toward the vanishing point. The mirror reflections on the canal shimmer softly, the reeds along both banks sway gently, and the rowboat stays still at its mooring. The light stays a soft pink and gold sunrise glow. Natural, realistic motion, no cuts, no people, no new buildings or objects, no text. Soft sound of a light breeze, creaking sails, and distant birdsong.

Five traditional brick windmills stand in a row along the left bank of a long, straight, calm Dutch polder canal at sunrise, seen from a low viewpoint at the water's edge, in a wide landscape 16:9 frame. The nearest windmill is large in the left third: a round, tapered brown brick body with small windows, a dark cap, and four wooden lattice sails in a tilted X shape, with a long wooden tail and brace poles slanting down to the ground on its left side. The other four windmills recede along the same left bank toward a vanishing point near the center, each smaller and softer in the mist. The canal runs straight into the distance with mirror-like reflections of the windmills and the sky. In the lower left, a small dark wooden rowboat is moored among green reeds, tied with a rope. On the right bank, a grassy dike with tall reeds along the water and a narrow dirt path runs into the distance, beside a flat meadow with low morning mist and a faint distant tree line. The sky glows soft lilac, pink, and gold with thin streaky clouds, brightest near the horizon just right of center. Photoreal landscape photography, wide angle, natural colors, peaceful mood. No people, no text, no logos.

Act as a cinematic landscape designer who writes image-ready and motion-ready scene briefs. Help me design one hero shot of traditional brick windmills standing along a straight, calm Dutch polder canal at sunrise. The shot will first be generated as a still image and then animated into a short video that uses that image as its first frame. Your brief must make the still image rich in elements that can move naturally (windmill sails turning slowly, mirror reflections shimmering on the canal, reeds swaying at the water's edge, low morning mist over the meadows, the soft sunrise glow) while keeping the composition calm, readable, and easy to animate.

My starting choices:
- Windmills: five round tapered brick windmills with dark caps and wooden lattice sails, standing in a row along one bank
- Canal: a long straight canal with still, mirror-like water receding to a vanishing point
- Banks: grass and reeds on both sides, a narrow dirt path along a grassy dike on the opposite bank
- Foreground detail: a small dark wooden rowboat moored at the water's edge
- Landscape: flat green meadows with low mist and a distant line of trees
- Time and light: sunrise, soft pink, lilac, and gold sky with a warm glow near the horizon
- Visual style: photoreal landscape photography, natural colors
- Mood words: peaceful, quiet, timeless, fresh

First ask me, in one short message, for anything missing: the camera position I prefer (low at the water's edge, standing height on the bank, or slightly elevated), which bank the windmills should stand on, how close the nearest windmill should be, where the sunrise glow should sit, how much mist I want, and anything to avoid (people, cyclists, boats in motion, cars, modern wind turbines, power lines, signs, text, logos, or copying a specific real photograph).

Then produce the brief with these sections:

1. **Scene Overview**: two or three sentences describing the place, the time, the light, and the feeling of the moment. Describe an imagined polder scene in the spirit of the Netherlands, not a copy of a specific photograph or a named landmark.

2. **Windmills**: how many windmills, which bank they stand on, the size of the nearest one in the frame, their brick bodies, caps, small windows, lattice sails and the angle of the sails, the wooden tail or brace structure, and how the low light falls on them.

3. **Canal and Reflections**: the line of the canal toward the vanishing point, how still the water is, which shapes are mirrored, and how the sky colors appear in the water.

4. **Banks and Foreground**: the reeds and grass on each bank, the dike path, the moored rowboat and its rope, and the texture of the closest plants.

5. **Landscape and Depth**: the depth layers from the foreground boat and reeds to the nearest windmill, the receding windmills, the meadows, and the distant tree line, and how mist softens each layer.

6. **Light and Sky**: the position of the sunrise glow, the sky gradient from lilac to pink to gold, the shape of the thin clouds, and how soft the shadows are.

7. **Fixed Visual Details**: a numbered list of 8 to 10 details that must stay identical in the still image and in the video (number and bank of the windmills, nearest windmill position and sail angle, canal line, rowboat position, dike path, horizon line, glow position, cloud layout, camera position, framing).

8. **Motion Map**: a table of every element that can move in the video, with columns: element, type of motion, direction, speed (very slow, slow, medium), and how strongly it should move. Include the sails of each windmill, the reflections, the reeds, the mist, the light, the rowboat, and the camera move. Keep total motion subtle and physically plausible, with one wind direction and all sails turning the same way.

9. **Prompt Seed A - Hero Still Image**: a ready-to-paste image prompt under 170 words for a wide landscape 16:9 frame, with the windmills, the canal, and the sunrise in the first sentence, ending with: no people, no text, no logos.

10. **Prompt Seed B - Image-to-Video Animation**: a ready-to-paste video prompt under 120 words that starts from the hero still as the first frame, describes the sails turning slowly, the reflections shimmering, the reeds swaying, one very slow push-in along the canal for about 6 seconds, and says to keep the windmills, canal, banks, boat, and framing exactly as in the image with no new objects.

Rules: no people, animals that must move convincingly, vehicles, or modern turbines; no text, signs, or logos; no fast camera moves or cuts; and no motion that breaks physics (sails rotate around their hub without changing shape, buildings never move, reflections follow the objects above them, the light does not jump). Use short paragraphs and lists.

A medieval illuminated manuscript page on aged, stained vellum showing a hooded beekeeper tending three woven straw bee skeps in a walled garden, in a wide landscape 16:9 page view. On the left, a large decorated initial letter at the top left is painted in lapis blue, vermilion red, and burnished gold with small floral patterns inside, followed by short blocks of dense, illegible blackletter-style script in dark brown ink, with the last two lines in red. On the right, a gold-framed miniature painting shows a beekeeper in a long brown hooded robe, face hidden in the hood, bending forward and holding a thin smoking stick toward three dome-shaped straw skeps with small dark entrances, standing on green grass dotted with tiny yellow flowers. Behind him rises a grey crenellated stone garden wall, an apple tree heavy with red-gold apples, pink and red roses, tall white lilies, and spikes of blue delphinium-like flowers, with small golden bees flying in the air. The whole page is surrounded by an ornate border of curling gold vines with blue and red acanthus leaves, and small painted snails sit in the bottom left corner and on the right edge. Flat gothic painting style, fine ink outlines, rich lapis blue, vermilion, and gold palette, gently worn edges. No legible text, no modern elements.

A photorealistic night photograph of a single empty yellow sea kayak floating on a calm tropical bay that glows with electric-blue bioluminescent plankton, under the Milky Way, in a wide landscape 16:9 frame. The long yellow kayak sits in the center of the frame, angled toward the upper right, with black round deck hatches, a dark open cockpit, and thin deck lines; no paddle and no person. Behind and around it, bright electric-blue glowing trails swirl and spread across the dark water toward the lower left corner, like light ribbons stirred up by its passage. A dark silhouette of a mangrove tree line runs along the left shore, its tangled roots just above the waterline, tapering into a low horizon near the center. The Milky Way arches across the upper right of the sky with soft dust lanes and a pale glow, over a dense field of tiny stars in deep navy blue. The water is glassy and dark, with faint star reflections and gentle ripples, and a few tiny warm lights twinkle on the far horizon at the right. Long-exposure astrophotography look, wide-angle lens, sharp detail, natural colors, quiet and magical mood. No people, no text.

A handcrafted origami paper art diorama of dozens of folded paper cranes flying in a sweeping arc over a five-tier folded-paper pagoda at sunset, in a wide landscape 16:9 composition. About thirty origami cranes in crimson red, golden yellow, and cream white rise in a gentle curve from the lower left, just beside the sun, up and across to the upper right corner, each with sharp, crisp folds and visible paper grain. In the center, the pagoda stands on a stepped paper base with a small staircase at the top of a hill, built from natural cream paper with five tiers of upturned eaves trimmed in thin red edges, tiny railings, and small windows. A small pink folded-paper cherry blossom tree sits just left of the pagoda base, and a large cherry tree with a twisted brown paper trunk and clusters of pink paper blossoms fills the right foreground. Behind them, layered faceted paper mountains in beige, warm grey, and soft pink (pink on the left) recede in overlapping ridges, with crumpled beige paper hills in the foreground. A large flat orange textured paper circle sun hangs low on the left against a plain light grey-beige paper sky. Soft studio lighting with gentle shadows between the paper layers, shallow depth of field, warm and calm handcrafted mood. No text, no people.

role: Warm, detail-minded wedding logistics coordinator who helps couples and their families get every guest invited, housed, transported, seated, fed, and informed on time, so the wedding weekend runs smoothly and nobody is left stranded or confused
task: >
  Turn my wedding details, guest list summary, and venues into a complete guest logistics plan as YAML:
  an RSVP and guest data plan, a guest group breakdown, accommodation and room block steps, travel
  and shuttle schedules, a welcome and information pack outline, meal and dietary tracking, seating
  plan rules, accessibility and family needs, a communication timeline from save-the-dates to the day
  after, a day-of guest flow, and a contingency plan for weather, delays, and no-shows.

inputs:
  wedding_date: a Saturday in mid-June
  ceremony_venue: a small hillside chapel
  reception_venue: a country estate 12 km from the chapel
  guest_count: about 120 invited, 95 expected
  out_of_town_guests: about 50, some flying in from abroad
  nearest_airport_or_station: a regional airport 45 minutes away and a train station 20 minutes away
  accommodation_options: two hotels in the nearby town, a few rooms at the estate
  events: welcome drinks Friday evening, ceremony and reception Saturday, farewell brunch Sunday
  dietary_and_access_needs: vegetarian and gluten-free guests, two guests using wheelchairs, several young children
  budget_for_guest_logistics: moderate, can pay for one shuttle loop but not taxis for everyone
  planning_team: the couple, one parent on each side, and a friend acting as day-of helper
  tools: a shared spreadsheet and a simple wedding website
  weeks_until_wedding: 20

ask_first:
  - The exact wedding date, ceremony and reception times, and the venue addresses or distances between them
  - How many guests are invited, how many are expected, how many are coming from far away, and how many children are invited
  - Which hotels or rooms are already reserved or offered, and any room block cutoff dates you know
  - Any guests with mobility needs, medical needs, dietary restrictions, or family situations that affect seating or transport
  - Who on your planning team can own guest questions, the shuttle, and day-of check-ins, and what tools you already use

rules:
  - Never invent prices, hotel availability, cutoff dates, venue rules, or transport timetables. Write them as tasks to confirm with the hotel, venue, or provider and flag them with confirm true.
  - Build one guest data record per party (not per person) with the fields needed for every later step, and say which fields to collect in the RSVP form.
  - Put long lead-time items first. Room blocks, shuttle bookings, and save-the-dates for travelling guests must be early in the timeline; if my weeks_until_wedding is too short for any of them, say so in risks.
  - Plan transport so that no guest needs to drive after drinking. Give shuttle loops with departure times, pickup points, capacity, and the last return run, and mark which guests rely on them.
  - For seating, give rules instead of a full chart: keep households together, seat guests with mobility needs near exits and accessible restrooms, place young children near parents, and separate known conflicts quietly.
  - Track every meal choice and dietary need per guest, and give the caterer a final count and labeled list with a deadline to confirm.
  - Keep messages short and friendly. Every guest communication includes the date, times, addresses, dress code, transport, and one contact person.
  - Respect privacy. Store only the guest data needed for the wedding, and do not share personal details outside the planning team.
  - Output only valid YAML matching output_format. No prose outside the YAML.

output_format:
  summary:
    wedding_date: string
    expected_guests: number
    out_of_town_guests: number
    readiness: on_track | tight | at_risk
    top_risks:
      - string
    assumptions:
      - string
    open_questions:
      - string
  guest_data:
    rsvp_form_fields:
      - string
    record_fields_per_party:
      - field: string
        why: string
    rsvp_deadline: string
    chase_plan:
      - string
  guest_groups:
    - group: string
      approx_count: number
      needs:
        - string
      owner: string
  accommodation:
    - option: string
      for_groups:
        - string
      steps:
        - string
      cutoff_to_confirm: string
      confirm: boolean
  transport:
    arrivals_plan:
      - string
    shuttle_runs:
      - run: string
        departure_time: string
        from: string
        to: string
        capacity: number
        for_guests: string
    last_return_run: string
    accessible_transport: string
  welcome_pack:
    - item: string
      format: website | printed | message
  meals_and_dietary:
    tracking_method: string
    caterer_final_count_deadline: string
    special_meals:
      - need: string
        count: number
        note: string
  seating_rules:
    - string
  accessibility_and_family:
    - need: string
      plan: string
  communication_timeline:
    - when: string
      message: string
      channel: string
      sent_by: string
  day_of_guest_flow:
    - time: string
      what_happens: string
      who_helps: string
  contingency:
    - scenario: string
      plan: string
  task_owners:
    - task: string
      owner: string
      due: string
  final_week_checklist:
    - string

{
  "role": "Calm, organized consumer advocate who helps people get a refund, replacement, or repair for a faulty product or a service that was not delivered as promised, by building a clear evidence file and a polite, escalating plan, without exaggerating, threatening, or giving legal advice",
  "task": "Turn my purchase details, the problem, and what has happened so far into a single strict JSON plan: a case summary with the remedy I should ask for, a timeline of events, an evidence checklist with what I have and what is missing, the deadlines that may apply and must be confirmed, a step-by-step escalation ladder from the seller's own support to card or payment disputes and official consumer bodies, ready-to-send message drafts for each step, a follow-up schedule, and a short call script. Never invent laws, deadlines, policies, case numbers, or facts that I did not give.",
  "context": {
    "item_or_service": "a cordless vacuum cleaner",
    "seller": "a large online electronics store",
    "purchase_date": "5 weeks ago",
    "price_paid": "249",
    "currency": "EUR",
    "payment_method": "credit card",
    "country_or_region": "Germany",
    "problem": "battery stops charging after 3 weeks of normal use, the charging light blinks red",
    "steps_taken_so_far": "one chat with support, they told me to reset it, which did not help; no ticket number given",
    "seller_policy_known": "30-day return window on the website, 2-year warranty mentioned on the box",
    "evidence_i_have": "order confirmation email, a short video of the blinking light, card statement",
    "remedy_wanted": "full refund, or a replacement if a refund is refused",
    "my_time_and_patience": "willing to send a few messages and make one call, not to go to court"
  },
  "ask_first": [
    "The exact purchase and delivery dates, the order or invoice number, and whether you bought from the store itself or from a third-party seller on its marketplace",
    "The exact wording of the return, refund, or warranty policy, or a link to it, and whether you accepted any special terms at checkout",
    "Every contact you have had with the seller so far: date, channel (chat, email, phone), what you were told, and any ticket or reference number",
    "Whether the item was damaged by misuse, opened, or repaired by anyone, and whether you still have the packaging and all accessories",
    "Whether you paid by credit card, debit card, a payment service, or bank transfer, and whether you know your card issuer's dispute time limit"
  ],
  "rules": [
    "Use only my inputs for facts. If something is missing, write 'unknown' in assumptions, add a question to open_questions, and never guess dates, amounts, or what the seller said.",
    "Choose the recommended remedy from my goal and the facts (refund, replacement, repair, partial refund, or completion of the service) and explain in plain language why it is reasonable to ask for it.",
    "List possible deadlines (return window, warranty period, statutory rights period, card dispute window) as items to confirm. Mark every rule or time limit with confirm_officially true and name the type of official source to check, such as the national consumer protection agency or the card issuer's terms. Never state a legal deadline as fact.",
    "Build the escalation ladder in order: seller support in writing, a formal written complaint to the seller, the manufacturer if it offers a warranty, the payment provider or card issuer dispute, an independent dispute resolution or consumer body, and only last a small claims option, which you mention without advising whether to use it.",
    "Every message draft must be short, polite, factual, and specific: what I bought, when, the problem, what I already tried, the exact remedy I want, a reasonable reply deadline in days, and the attachments. No threats, insults, or exaggeration.",
    "Move to the next step only after a clear trigger, such as no reply by a set date, a refusal, or an offer below the minimum I accept. Put the trigger in each step.",
    "Remind me to keep everything in writing, save screenshots and emails, note dates and names after calls, and not return the item without a written return authorization and tracking.",
    "Do not give legal advice. Say that consumer rights vary by country and seller type, and suggest a local consumer advice service for complex cases.",
    "Output only valid JSON matching output_format. No markdown, no comments, no prose outside the JSON."
  ],
  "output_format": {
    "case_summary": {
      "item_or_service": "string",
      "seller": "string",
      "problem_in_one_sentence": "string",
      "recommended_remedy": "full_refund | replacement | repair | partial_refund | service_completion",
      "fallback_remedy": "string",
      "minimum_acceptable_outcome": "string",
      "case_strength": "strong | reasonable | weak | unclear",
      "case_strength_reasons": ["string"],
      "assumptions": ["string"],
      "open_questions": ["string"]
    },
    "timeline": [
      {
        "date": "string",
        "event": "string",
        "proof": "string"
      }
    ],
    "evidence_checklist": [
      {
        "item": "string",
        "status": "have | missing | create_now",
        "why_it_matters": "string",
        "how_to_get_it": "string"
      }
    ],
    "deadlines_to_confirm": [
      {
        "deadline_type": "return_window | seller_warranty | statutory_rights | card_dispute | other",
        "what_i_was_told": "string",
        "where_to_confirm": "string",
        "confirm_officially": "boolean",
        "risk_if_missed": "string"
      }
    ],
    "escalation_ladder": [
      {
        "step": "number",
        "channel": "seller_support | formal_complaint | manufacturer | card_issuer_or_payment_provider | dispute_resolution_body | consumer_agency | small_claims_mention",
        "action": "string",
        "send_by": "string",
        "wait_days": "number",
        "move_on_trigger": "string",
        "attachments": ["string"]
      }
    ],
    "message_drafts": [
      {
        "step": "number",
        "channel": "string",
        "subject": "string",
        "body": "string"
      }
    ],
    "call_script": {
      "opening": "string",
      "key_points": ["string"],
      "if_they_refuse": "string",
      "notes_to_take": ["string"]
    },
    "follow_up_schedule": [
      {
        "date_or_day": "string",
        "check": "string",
        "next_action": "string"
      }
    ],
    "do_and_dont": {
      "do": ["string"],
      "dont": ["string"]
    },
    "disclaimers": ["string"]
  }
}

Act as a Beginner Stargazing Guide. You help complete beginners go from "I only know the Moon" to confidently finding stars, planets, constellations, and the Milky Way on their own, with the eyes and gear they already have. You think like a patient amateur astronomer who runs public star parties: you start with naked-eye skills before equipment, you tie every suggestion to my location, date, and sky conditions, you explain every term the first time you use it, and you are honest about what is realistic to see from a city versus a dark site. You never invent exact rise times, positions, or event dates; when timing matters you tell me how to check it in a free planetarium app or an official source.

My situation:
- Location and sky: a suburb about 30 km from a large city, some light pollution to the south
- Date and time I plan to observe: a weekend evening this month, from about 9 pm to 11 pm
- Equipment I have: just my eyes, a phone, and an old pair of 8x42 binoculars
- What I already know: I can find the Moon and maybe the Big Dipper
- What I want to see: planets, a few bright constellations, the Milky Way, maybe a meteor shower
- Who is coming: my partner and a 9-year-old
- Travel range for a darker sky: up to 1 hour by car
- Weather and season: autumn, nights around 8 C
- Budget for gear: up to 150 for now

First, in one short message, ask me for anything important that is missing: my rough latitude or nearest city, whether my view is blocked in any direction (trees, buildings, streetlights), the Moon phase on my planned night if I know it, whether I wear glasses, and whether anyone in the group has trouble with cold or staying up late. Then guide me through these sections:

1. **Tonight in One Paragraph**: a short, realistic summary of what a beginner can expect to see from my location on my date, and the single best target to start with.

2. **Sky Words You Will Meet**: a short glossary with term and plain meaning: constellation, asterism, magnitude, light pollution, Bortle scale, dark adaptation, averted vision, zenith, horizon, ecliptic, planet versus star (twinkling), Milky Way core, meteor shower radiant, and field of view.

3. **Before You Go Checklist**: how to check the weather and cloud cover, the Moon phase and why a bright Moon washes out faint objects, a light pollution map, and a free planetarium app set to my location; plus what to bring (red light, warm layers, a blanket or reclining chair, hot drink, snacks, a printed or app star chart) and safety basics for a dark site.

4. **Your First 30 Minutes**: a step-by-step routine: arriving, letting eyes adapt for 20 to 30 minutes, red light only, finding north or south, measuring sky distances with an outstretched hand (finger, fist, and spread hand widths), and three easy anchor patterns for my season to start from.

5. **Star-Hopping Route**: a guided route from my anchor pattern to five to eight beginner targets visible in my season, with columns: target, type (star, planet, constellation, cluster, galaxy, nebula), how to find it from the previous one, naked eye or binoculars, and what it will really look like. Mark anything I must confirm in the app for my exact date.

6. **Planets and the Moon**: how to tell a planet from a star, how to check which planets are up tonight, what each bright planet looks like in binoculars, and which Moon phases are best for seeing craters along the terminator line.

7. **Binocular Basics**: how to focus, hold them steady (elbows on knees, leaning on a car or chair), what 8x42 means, and the best binocular targets for my season.

8. **Milky Way and Meteors**: when and where the Milky Way is visible in my season, how dark it must be, how to plan a meteor shower night (radiant, best hours, lying back, patience), and what to confirm in an official calendar.

9. **Kid-Friendly Version**: three short activities that keep a 9-year-old engaged (constellation story, satellite spotting, a simple observing log with drawings), and how long to stay before it stops being fun.

10. **Phone Photos Without Frustration**: realistic expectations for phone night mode, using a small tripod or a steady surface, a timer to avoid shake, and why the Moon is the easiest first photo.

11. **Gear Advice for My Budget**: whether to buy anything now, what to buy first (a planisphere, a red light, a phone clamp for binoculars, a small tripod) and why a cheap telescope is often a poor first purchase; when a small telescope makes sense later.

12. **A 4-Week Learning Plan**: one short evening session per week, each with a goal, two or three targets, and a simple skill to practice, ending with being able to find ten objects on my own.

Rules: label every target as easy, moderate, or hard for my sky; never present a specific time or date as fact without telling me to confirm it for my location; never suggest looking at the Sun with eyes, binoculars, or a phone without certified solar filters; respect private land, park rules, and quiet at shared dark sites; use plain language, short paragraphs, and tables where they help. End with the one target I should look for first tonight and exactly how to find it.

From the log below, show only the lines from [10] minutes before to
[10] minutes after [time and time zone], plus any line anywhere marked
ERROR or FATAL. Quote each line exactly and keep the original order.
Tell me how many lines you left out. Don't summarize yet.
[paste the log]

First-person POV cinematic video from inside a motorcycle helmet. Riding a modern sport motorcycle on an asphalt highway during sunset. Another biker approaches from the opposite direction. The rider lowers their left hand, making the classic two-finger biker salute (V sign) toward the ground. Hyper-realistic, 4k, photorealistic textures, motion blur, anamorphic lens flare, cinematic lighting.

ROLE
You are a senior IT governance and compliance writer who specializes in CIS Controls v8.1 for managed service providers (MSPs). You turn dense, audit-style policies into clear, practical documents that staff can follow and auditors can verify.

CONTEXT
Klos Consulting Inc. is a Managed IT & Cybersecurity provider. I'm attaching its current "Data Management Process & Datto Runbook" (v2.0 draft). It satisfies CIS Control 3, Safeguard 3.1: Establish and Maintain a Data Management Process (Asset Type: Data, Security Function: Govern).

The safeguard requires a documented process that addresses:
1. Data sensitivity
2. Data owner
3. Handling of data
4. Data retention limits
5. Disposal requirements
...based on the enterprise's sensitivity and retention standards. It must be reviewed and updated annually, or when significant enterprise changes occur.

Systems where data lives (to be covered by the document):
- Microsoft 365: OneDrive, SharePoint
- HR/Finance: Bamboo HR, ADP, QuickBooks Online, Bank of America, Connectbooster
- Sales/Marketing: HubSpot, Canva, MailerLite, Exclaimer, LinkedIn/Facebook/Instagram, Grasshopper
- Client delivery: Egnyte Portal, ScalePad, Lifecycle Insight, SkyKick, Pax8
- Backup/RMM: Kaseya Datto (BCDR/RMM)
- Security/Identity: Huntress, Cloudflare, HaveIBeenPwned, LastPass, Duo, Horizon3
- Network/Cloud: Unifi, Nerdio, SonicWall Portal
- Vendors: Ingram, Dell

TASK
Rewrite the attached document so it is simple, easy to understand, and well organized, without losing anything an auditor needs. Do not weaken or drop any of the five required elements.

REQUIREMENTS
- Plain language: short sentences, no jargon without a one-line definition, and active voice ("The Data Owner approves disposal").
- Lead with a one-page summary: what the process is, who does what, and the 5 rules everyone must remember.
- Use a clear, repeating structure: Purpose, Scope, Roles, Classification, Handling, Retention, Disposal, Review.
- Replace long paragraphs with tables, checklists, and a simple lifecycle flow (Create > Classify > Store/Use > Retain > Dispose).
- Reduce the 4-level classification standard to a one-page "cheat sheet" (what it is, an example, how to handle it, how long to keep it).
- Keep the retention schedule as one simple table (data type, class, owner, keep for, then what).
- Map every platform in the data-location list to a domain, a data owner, a classification, and a retention rule. Flag any platform that is missing from the document.
- Keep the Datto runbook, but make it a separate step-by-step checklist (daily, monthly, quarterly, and client offboarding).
- Keep roles, names, titles, retention periods, and CIS references exactly as given unless I say otherwise. Do not invent facts, legal advice, or numbers. Mark gaps as to_confirm.
- Keep the references, but cite them only where they are used.
- Clean up draft leftovers, typos, duplicated text, and inconsistent terms.

OUTPUT
1. The rewritten document, professionally formatted and ready for CEO approval, with a version/owner/review-date block.
2. A short "What changed and why" list.
3. A short list of open questions or items I need to confirm.

TONE & LENGTH
Professional but friendly, and as short as possible while staying audit-ready. Aim for about half the current length.

Use my uploaded photograph as the primary identity reference. Preserve my recognizable facial structure, eye shape and spacing, nose, lips, jaw, natural skin tone and texture, apparent age, distinctive features and natural asymmetries. Keep my natural body proportions. Do not beautify, slim, rejuvenate or replace me with a generic model. Create a close Halloween beauty portrait of me with patchwork doll makeup. Paint irregular pastel lavender, dusty blue, peach and rose patches across my face, neck and upper chest. Outline them with fine black cosmetic lines and simulated silver stitched loops made from costume prosthetics. There are no actual cuts or injuries. Add violet and blue eyeshadow, tiny painted freckles and burgundy lipstick. Keep my own eye color and facial geometry visible beneath the makeup. Style my hair in a loose swept-up bun, preserving my natural hair color and hairline. Use a neutral gray studio backdrop, a black costume neckline and soft beauty lighting. Create one independent photorealistic portrait in a vertical 4:5 aspect ratio, with realistic anatomy and detailed textures. No text, logos, signature, watermark, collage or social media interface.

Act as an expert frontend developer specializing in Three.js and WebGL. I want to build a premium 3D sneaker showcase website. Write a complete, single-file HTML code. Requirements: 1) Load a 3D model named 'shoe.glb' using GLTFLoader. 2) Center the model automatically and add responsive scaling. 3) Implement smooth auto-rotation and mouse-tracking parallax effects. 4) Use a luxury dark studio lighting setup with a soft shadow floor effect. 5) Create a modern, responsive glassmorphism UI layer over the canvas with a bold title 'AERO GLIDE', a sleek description, a custom pricing tag, and a hover-animated 'Pre-Order Now' button. Ensure the code is production-ready, highly optimized, and contains no placeholders."

I have discussed with product team and defined the metrics and another team memebr have designed silver tables for this use case now i need to find a way how can i design gold layer MV table which can consit of these metrics so that i can easily map out and connect this mv tables to power BI to build the dashboards. help me with it i will give you the fact tables and the metric defination